Spring Boot Actuator & Observability

# CLAUDE.md — Spring Boot Actuator & Observability

## Actuator

- Add `spring-boot-starter-actuator`. It's mandatory in production.
- Default exposed endpoints: `/actuator/health`, `/actuator/info`. Everything else is opt-in.
- Expose only what's needed: `management.endpoints.web.exposure.include=health,info,metrics,prometheus`.
- **Never** expose `env`, `heapdump`, `threaddump`, or `loggers` to public traffic. Lock them behind admin auth or a separate management port.

## Health checks

- Boot composes `/actuator/health` from `HealthIndicator` beans (DB, Redis, etc.).
- Implement custom `HealthIndicator` for app-specific dependencies:
  ```java
  @Component
  class StripeHealthIndicator implements HealthIndicator { public Health health() { ... } }
  ```
- For Kubernetes, enable probes split: `management.endpoint.health.probes.enabled=true`.
  - `/actuator/health/liveness` — process is alive (don't depend on DB here)
  - `/actuator/health/readiness` — can serve traffic (DB + cache reachable)

## Metrics with Micrometer

- Micrometer is auto-configured. Add a Prometheus registry: `io.micrometer:micrometer-registry-prometheus`.
- Tag every metric — high cardinality kills Prometheus. Tag by **service**, **endpoint**, **status** — never by user_id.
- Standard metrics from Boot: HTTP request timings, JDBC pool, JVM, Tomcat. Custom metrics:
  ```java
  Timer timer = Timer.builder("checkout.duration").tag("flow", "card").register(meterRegistry);
  ```

## Tracing

- **Micrometer Tracing** with OpenTelemetry exporter. Add `micrometer-tracing-bridge-otel` + `opentelemetry-exporter-otlp`.
- Sampling: 10% in production, 100% in dev. Configure with `management.tracing.sampling.probability`.
- Propagate `traceparent` across HTTP, message queues, and async hops. Spring instruments most clients automatically.

## Logging

- Structured JSON logs. Use **logstash-logback-encoder** or Boot's built-in JSON encoder (Boot 3.4+).
- Add `trace_id` and `span_id` to every log line via MDC. Spring populates them when tracing is enabled.
- One log statement per significant decision. Don't log every method entry.

## Profiling

- **Spring Boot Profiler** (Boot 3.5+) for prod-safe sampling.
- For deeper diagnostics, **JFR** (Java Flight Recorder) — built into the JVM, very low overhead.
- `/actuator/heapdump` on demand for OOM investigation. Lock behind admin auth.

## Alerting

- Define SLOs: `availability`, `p99 latency`, `error rate`. Alert when burned.
- Don't alert on every spike. Multi-window, multi-burn-rate alerts (Google SRE book) are the right pattern.
- Page on user-impacting symptoms, not on every CPU spike.

## Production checklist

- Actuator on a separate port (`management.server.port=8081`) so internal-only endpoints don't share routing with public traffic.
- Prometheus scraping `/actuator/prometheus`.
- Logs shipped to Loki/Datadog/your aggregator.
- Traces shipped to Tempo/Honeycomb/your tracer.

## Don't

- Don't expose unauthenticated `/actuator/env` — leaks every config value, including resolved secrets.
- Don't set `management.endpoints.web.exposure.include=*` in production.
- Don't tag metrics by `user_id` or any high-cardinality value. Your bill will explode.
- Don't bypass tracing for "performance" — it costs <1% and saves debugging hours.