# Observability

> **Conformance:** L1 for OB-01..OB-04 (every project); L2 for
> OB-05..OB-08 (every project that runs as a service / has
> users). The L2 rules apply once you can be paged.
> **Audience:** Anyone wiring metrics, traces, or alerts on a
> federation project; SRE / on-call rotations consuming the
> output; AI agents deciding what an SLI breach means.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **Scope.** Observability = three pillars (logs, metrics,
> traces). [`LOGGING.md`](LOGGING.md) governs the first pillar;
> this doc governs the other two **and** the umbrella discipline
> (SLI / SLO / alert) that ties all three together. Without an
> umbrella, every consumer invents its own metric taxonomy and
> alert philosophy — by the time a federation crosses three
> services in production, the on-call rotation has three
> mental models to maintain. Codify it once.

---

## Rules box (read this first)

| ID | Rule |
|---|---|
| <a id="OB-01"></a>**OB-01** | **Observability MUST be the three-pillars shape** — logs (LG-* governs), metrics (this doc), traces (this doc). Every consumer MUST emit logs. Consumers that run as a service MUST emit metrics + traces. CLIs and library shapes MAY skip metrics + traces (they have no continuous lifecycle to observe). |
| <a id="OB-02"></a>**OB-02** | **Every metric name MUST follow the federation convention `mgf_<domain>_<unit>_<aggregation>`** — for example `mgf_http_request_duration_seconds_bucket`, `mgf_db_pool_connections_in_use`. The `mgf_` prefix isolates federation metrics from third-party metrics in shared dashboards; the `_<unit>` segment matches Prometheus convention; the `_<aggregation>` segment is the metric type (`_total`, `_bucket`, `_sum`, `_count`, or empty for gauge). |
| <a id="OB-03"></a>**OB-03** | **Every metric MUST carry the canonical label set** — `app` (the consumer's `app_name`), `version` (the consumer's `app_version`), `env` (e.g., `prod`, `staging`, `dev`). Custom labels MAY be added per metric; the canonical three are non-optional. Without them, a single dashboard cannot disambiguate the same metric coming from two consumers or two versions. |
| <a id="OB-04"></a>**OB-04** | **Traces MUST be OpenTelemetry-shaped.** Spans MUST be operation-named in verb-object form (`db.query`, `http.request`, `vault.unlock`); span attributes MUST follow OTel semantic conventions where they exist (`http.method`, `db.system`, `net.peer.name`). The federation's `mgf.common.observability.otel_setup` wires the SDK; consumers don't roll their own. |
| <a id="OB-05"></a>**OB-05** | **Every customer-facing operation MUST have a defined SLI** (Service Level Indicator) — a single number that says "is this operation healthy right now?" Examples: "95th percentile request latency over 5 min," "fraction of requests returning 5xx over 5 min." SLIs live in `docs/operability/SLI.md` (one row per SLI: name, definition, data source, target). |
| <a id="OB-06"></a>**OB-06** | **Every SLI SHOULD have a target SLO** (Service Level Objective) — the threshold below/above which the operator considers the service unhealthy. SLOs are operator-chosen, not framework-imposed; the rule is just that they MUST be written down (so an alert can fire against them). |
| <a id="OB-07"></a>**OB-07** | **Every alert MUST be actionable.** The alert's description MUST answer "what does the operator do?" in one sentence; the runbook link MUST exist (`docs/operability/runbooks/<alert-name>.md`). Pages with no runbook = noise, and noisy pages cost trust faster than missing pages cost incidents. |
| <a id="OB-08"></a>**OB-08** | **SLI / SLO / alert definitions MUST be versioned alongside the code they describe** — same repo, same branch, same review process. Drift between code and observability config (the code emits `request_count` but the alert still queries `requests_total`) is a defect: the alert misses real incidents, the operator distrusts the alert next time, and the incident escalates further than it should. |
| <a id="OB-09"></a>**OB-09** | **The structured `LogRecord` schema (`mgf.common.observability.LogRecord`) MUST evolve additively + non-breakingly.** It is the federation's canonical cross-language log shape (the logging analog of `TestRecord`), so it is built to grow without a redesign: (1) ad-hoc/new data goes in `fields` (structured context) or `labels` (tags) — **no schema change**; (2) promoting to a typed top-level key is additive — an *optional*, defaulted field + a `LOG_SCHEMA_VERSION` bump (old readers ignore it, new readers default it for old records); (3) readers MUST tolerate **and losslessly preserve** unknown keys from a newer schema (round-trip via `LogRecord.extra`) so an older relay never strips newer producers' data; (4) **renaming / repurposing / removing / retyping** an existing key is the only breaking move and REQUIRES a MAJOR `schema_version` bump. Consumers MUST NOT pin to the absence of a key. Rationale: logging is the federation's primary window into running systems + the spine of centralized collection — its schema must extend forever without breaking the apps and pipelines built on it. |
| <a id="OB-10"></a>**OB-10** | **Every runtime emits the `LogRecord` wire shape byte-identically, pinned against one shared golden.** The schema is cross-language (Python + Zig + TS), so a single canonical fixture + manifest (`mgf/common/observability/schema/log_record_fixture.jsonl` + `log_record_schema.json`, generated by `tools/gen_log_record_schema.py`) is the source of truth, and **each language ships a conformance test that loads that golden and asserts its emitter reproduces every line byte-for-byte** (`clients/{typescript,zig}/`, Python in `test_log_record_conformance.py`). The wire is: keys in exactly the manifest's `emit_order` then any unknown (`extra`) keys appended; compact separators (no spaces); **raw UTF-8** (`ensure_ascii=false` — no `\uXXXX` escaping, so non-Python emitters match natively); core keys always present; optionals omitted when empty/null/zero. Adding a typed field MUST update the manifest + bump `LOG_SCHEMA_VERSION` (the Python pin test fails otherwise). **This doc OWNS the canonical wire** — the `LogRecord` schema (OB-09), the `emit_order` manifest, and the golden are the single source of truth for every pillar; LOGGING is the first pillar and its `JsonFormatter` ([`LOGGING.md`](LOGGING.md) §5.3) is the **Python reference emitter** of this wire, not a second, independent record model. Any emitter — the reference `JsonFormatter`, the Zig/TS clients, or a consumer's own — is conformant iff it reproduces the golden byte-for-byte; the `JsonFormatter`'s flat top-level keys + promoted `extra={…}` are the Python projection of OB-09's `fields` / `labels` / `extra` model, ordered by `emit_order`. Rationale: "unified logging" is only unified if a Zig daemon, a Python service, and an SSR frontend produce the *same bytes* a single tail/stream/shipper can read without per-language special-casing — which holds only if there is exactly one wire schema and the per-language formatters reference it rather than re-inventing field names. |
| <a id="OB-11"></a>**OB-11** | **Per-app logging statistics (`mgf.common.observability.LogStatsCollector`) MUST be cheap, bounded, and fail-safe.** Apps SHOULD expose *how much* they log (total, per-level, per-component, overall + recent-window rate, async-overflow drops) alongside the *what/where/how* of `describe_logging`. The collector is a `logging.Handler` and therefore: (1) **cheap** — every increment is O(1) under a short lock, with no formatting or I/O in `emit`; (2) **bounded** — the per-component map is capped (`max_components`, overflow folded into `__other__`) so a logger-name explosion cannot grow memory without limit; (3) **fail-safe** — `emit` MUST NOT raise into the caller (a stats handler that breaks logging is worse than no stats). `snapshot()` returns a frozen, JSON-able `LogStats` (versioned `schema_version`) for the per-app web surface (`/logs/info`) and the centralized shipper. Rationale: statistics are part of the unified-logging window into a running system, but observing the system must never perturb or endanger it. |

---

## 1. Why the three-pillars umbrella matters (OB-01)

Each pillar answers a different question:

| Pillar | Question | Cost shape |
|---|---|---|
| **Logs** | "What happened?" | Cheap to emit, expensive to retain. Querying is text-grep. |
| **Metrics** | "How often / how slow / how many?" | Cheap to emit, cheap to retain (aggregated). Querying is statistical. |
| **Traces** | "Why is *this specific request* slow?" | Expensive to emit (per-request), expensive to retain. Querying is causal-chain. |

A consumer with logs only can't answer "what's the p99 latency
over the last hour?" without a heroic log-grep + summarisation.
A consumer with metrics only can't answer "why was request
12345 slow?" without traces. A consumer with traces only pays
for the volume but can't trend or alert without metrics.

The federation's discipline: every running service emits all
three, with the cornerstone library doing the wiring (LG-*
configures logging; `mgf.common.observability.otel_setup`
configures metrics + traces; consumer adds the metric
declarations + trace spans).

---

## 2. Metric naming (OB-02) — examples

```
mgf_http_request_duration_seconds_bucket{le="0.05"}
mgf_http_request_duration_seconds_count
mgf_http_request_duration_seconds_sum
mgf_http_requests_total{method="GET",status="200"}
mgf_db_pool_connections_in_use
mgf_db_query_duration_seconds_bucket{le="0.01"}
mgf_vault_unlock_attempts_total{result="success"}
mgf_vault_unlock_attempts_total{result="failure"}
mgf_worker_queue_depth
mgf_crash_reports_emitted_total{sink="local"}
mgf_crash_reports_emitted_total{sink="remote"}
```

The pattern:

- **Prefix** `mgf_` — federation namespace.
- **Domain** `http`, `db`, `vault`, `worker`, `crash_reports`,
  etc. — the subsystem.
- **Unit** `seconds`, `connections`, `attempts`, `bytes`, etc.
- **Aggregation suffix** `_bucket` (histogram), `_count`,
  `_sum`, `_total` (counter), or empty (gauge).

The canonical three labels (OB-03) attach to every series:

```
{app="hrb-app", version="0.4.1", env="prod", method="GET", status="200"}
```

A Grafana dashboard that filters on `app="hrb-app"` shows just
that consumer's metrics; one that filters on `env="prod"` shows
just production. Without these labels, the dashboard can't
disambiguate.

---

## 3. Trace shape (OB-04)

Spans are operation-named:

```
GET /api/users               ← bad (HTTP path)
http.request                 ← good (operation verb-object)

select_user_by_id            ← bad (function name leak)
db.query                     ← good
```

Span attributes follow OTel semantic conventions:

```
http.method        = "GET"
http.status_code   = 200
http.route         = "/api/users/{id}"
db.system          = "postgresql"
db.operation       = "SELECT"
db.statement       = "SELECT * FROM users WHERE id = $1"   ← parameterized; literals bound out (SC-09 / LG-04 redaction applies to any sensitive value)
net.peer.name      = "db.internal"
```

The federation's `mgf.common.observability.otel_setup` wires
the SDK + the exporter + the default sampler. Consumers add
spans via the standard OTel API (`tracer.start_as_current_span(
"http.request")`) or via the framework integrations
(`mgf-fastapi` ships RequestIdMiddleware that creates the
top-level span automatically).

---

## 4. The SLI file shape (OB-05)

`docs/operability/SLI.md`:

```markdown
# Service Level Indicators

Every SLI here MUST have an SLO (OB-06) and an alert (OB-07).
The SLO target is operator-chosen; revisit quarterly.

## sli_http_latency_p95
**What it measures**: 95th percentile of `mgf_http_request_duration_seconds_bucket`
                      over the last 5 minutes, across all routes.
**Data source**: Prometheus / Grafana metric query
                 `histogram_quantile(0.95, sum by (le) (rate(mgf_http_request_duration_seconds_bucket[5m])))`.
**SLO**: ≤ 250 ms.
**Why this target**: User-perceived "fast" is sub-300 ms;
                     250 ms leaves headroom for network + render.
**Alert**: alert_http_latency_p95_breach (page on 5-minute breach).
**Runbook**: docs/operability/runbooks/alert_http_latency_p95_breach.md

## sli_http_error_rate
**What it measures**: fraction of `mgf_http_requests_total{status=~"5.."}` 
                      over the last 5 minutes.
**Data source**: Prometheus / Grafana metric query
                 `sum(rate(mgf_http_requests_total{status=~"5.."}[5m])) / sum(rate(mgf_http_requests_total[5m]))`.
**SLO**: ≤ 0.1% (one in a thousand requests).
**Why this target**: ...
**Alert**: alert_http_error_rate_breach.
**Runbook**: docs/operability/runbooks/alert_http_error_rate_breach.md
```

A consumer with no SLIs (a CLI tool, a library) has no SLI.md
— that's fine; OB-05 applies only to consumers with customer-
facing operations. A consumer with one SLO but no SLI.md
fails OB-05.

---

## 5. The runbook shape (OB-07)

`docs/operability/runbooks/alert_http_latency_p95_breach.md`:

```markdown
# Runbook: HTTP latency p95 breach

**What fired**: 95th-percentile request latency exceeded 250 ms
for 5 consecutive minutes.

**First action** (within 5 min): check the Grafana dashboard
`http-latency-breakdown` and identify the slowest route. If one
route dominates, that's your suspect.

**Common causes**:
- Database connection pool saturation → check
  `mgf_db_pool_connections_in_use` vs configured `pool_size`.
- Downstream API slowness → check
  `mgf_http_outbound_request_duration_seconds` for the suspect
  upstream.
- Recent deploy → revert if the latency-baseline correlates
  with the deploy time.

**Escalate if**: the breach persists for >15 minutes after
first action.

**Owner**: @bassam (primary), @oncall (backup).
```

The runbook is short. Three sections (what fired / first
action / common causes) and the operator can act. A runbook
with no "first action" is decoration.

---

## 6. Versioning observability with the code (OB-08)

```
mgf-fastapi/
├── src/mgf/fastapi/
├── tests/
└── docs/
    ├── operability/
    │   ├── SLI.md                              ← SLIs
    │   └── runbooks/
    │       ├── alert_http_latency_p95_breach.md
    │       └── alert_http_error_rate_breach.md
    └── alerts/
        └── prometheus-alerts.yml               ← versioned alert rules
```

Same git repo, same branch, same PR review. If a developer
renames `mgf_http_requests_total` to `mgf_http_request_count`
(NEVER do this — it's a breaking observability change), the
Prometheus alert rule in `docs/alerts/` MUST move in the same
PR. CI catches the drift if the alert rule references a metric
name that doesn't exist anywhere in the code (a simple grep is
enough).

---

## 7. The L1/L2 split rationale

OB-01..OB-04 are L1 because they cost nothing to follow from
day one: the metric naming convention, the canonical label
set, and the trace shape are all decisions made once at
project birth. Getting them wrong is expensive to undo (every
dashboard, every alert, every downstream consumer of the
metric stream has to migrate).

OB-05..OB-08 are L2 because they require an operational
context (a service to monitor, an on-call rotation to page) —
a library or CLI doesn't have those. Once a service ships, the
L2 rules are non-negotiable; before that, they don't apply.

---

## 8. Self-review checklist

For every project:

- [ ] Logs emitted via `mgf.common.observability.logging_setup` (LG-* applies).
- [ ] If a service: metrics emitted, naming `mgf_<domain>_<unit>_<aggregation>` (OB-02).
- [ ] If a service: canonical labels `app`/`version`/`env` on every metric (OB-03).
- [ ] If a service: traces wired via `mgf.common.observability.otel_setup`; spans operation-named (OB-04).

For every service:

- [ ] `docs/operability/SLI.md` exists with one row per customer-facing operation (OB-05).
- [ ] Each SLI has an SLO target written down (OB-06).
- [ ] Each SLO has an alert wired (OB-07).
- [ ] Each alert has a runbook (OB-07).
- [ ] Alert rules + runbooks are versioned alongside the code that emits the underlying metrics (OB-08).

---

## 9. Mechanical enforcement

| Rule | Enforcement |
|---|---|
| OB-01 | `mgf-fed app-doctor` (service-kind only) checks that the OTel SDK is wired. Today: review-only. |
| OB-02 | A custom check could lint metric registrations against the `mgf_*` regex; not currently shipped. Review-only. |
| OB-03 | The federation's `MetricsRegistry` (when shipped) MAY auto-attach the canonical labels from `bootstrap()`'s identity (app_name, app_version, env). Today: review-only. |
| OB-04 | Review-only. ruff has no OTel-shape rule. |
| OB-05 | Review-only. `mgf-fed catch-up` audit MAY add a "service kind + missing SLI.md" finding; not currently shipped. |
| OB-06 | Review-only. |
| OB-07 | Review-only. A future `mgf-fed catch-up` audit could grep alert rules for matching runbook files; not currently shipped. |
| OB-08 | Review-only. CI grep for alert-rule metric names against code-emitted metric names is the easiest mechanical check; not currently shipped. |
| OB-09 | **Enforced.** `test_log_record_conformance.py` pins the manifest to the live dataclass + parser — adding/removing/retyping a typed field fails CI until the manifest + `LOG_SCHEMA_VERSION` are updated. |
| OB-10 | **Enforced.** Per-language conformance tests (`test_log_record_conformance.py`, `clients/typescript/conformance.test.ts`, `clients/zig/log_record.zig` test block) assert byte-for-byte reproduction of the shared golden; the generator (`tools/gen_log_record_schema.py`) + pin test prevent golden drift. (Zig runs in mgf-hrb CI — no Zig toolchain here.) |
| OB-11 | **Enforced (unit).** `test_stats.py` pins the cheap/bounded/fail-safe properties: O(1) counts, component cardinality cap → `__other__`, `emit` swallows internal errors, JSON-able snapshot. Review-only for whether a given service actually enables stats. |

Most of the L2 rules are review-enforced today; OB-03 has the
cleanest path to automation (auto-attach via `bootstrap`) and
is the natural first mechanical check when the federation
ships its own metrics-registry abstraction.
