# Logging

> **Conformance:** L1 / L2 / L3 (see `README.md`)
> **Audience:** Human engineers AND AI coding agents.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **📦 Note.** `setup_logging`, `JsonFormatter`, `TextFormatter`, `Redactor`, `setup_otel`, `operation_span`, `install_excepthook`, `install_threading_excepthook`, `report_now`, and the crash-reporter sinks live in **`mgf.common.observability`**. The OpenTelemetry SDK + Sentry + httpx-instrumentation deps are pulled in by the **`mgf-common[observability]`** extra. Consumers import these directly rather than re-implementing them.

---

## Rules box (read this first)

| ID | Rule | Applies to |
|---|---|---|
| <a id="LG-01"></a>**LG-01** | Every module MUST acquire its logger via `logging.getLogger(__name__)` (or the equivalent canonical hierarchical name). No bare `logging.info(...)`. | all |
| <a id="LG-02"></a>**LG-02** | The application MUST configure logging in **exactly one place per entry point** (CLI `main`, GUI `app.run`, scheduler service entry, etc.) and MUST emit JSON when stdout is not a TTY (so log aggregators can parse it). Human-readable text is the TTY default. | all |
| <a id="LG-03"></a>**LG-03** | Every log record MUST carry the OpenTelemetry **trace_id** and **span_id** when a span is active. The standard formatter adds these automatically. | all |
| <a id="LG-04"></a>**LG-04** | Sensitive fields (passwords, tokens, private keys, session cookies, IP addresses on request) MUST be redacted by the formatter before any sink sees them. The redaction set MUST be configurable. | all |
| <a id="LG-05"></a>**LG-05** | ERROR-level logs MUST include the exception via `LOG.exception(...)` or `exc_info=True`. Never just the stringified exception in the message. | all |
| <a id="LG-06"></a>**LG-06** | Logs MUST reach an aggregator the operator can search. The aggregator MAY be: a **rotating local file** (desktop / single-server, no platform aggregator), the process's **stdout/stderr** (cloud-native / container; the runtime captures + forwards to the platform aggregator per 12-factor §XI), or a **remote sink** (OTLP / syslog / HTTP webhook; opt-in). A "console only, no aggregation" configuration is non-compliant in production. **Revised in v0.18.0** (closes FEEDBACK LOG-01 — the original rotating-file mandate didn't fit cloud-native consumers). | all |
| <a id="LG-07"></a>**LG-07** | GUI apps MUST ship a **log-viewer dialog** that reads the rotating file sink so users can view logs without a terminal. | gui-app |
| <a id="LG-08"></a>**LG-08** | Log messages MUST use lazy %-style formatting (`LOG.info("processing %s", x)`), NOT f-strings. Lazy formatting is skipped when the level is filtered. | all |
| <a id="LG-09"></a>**LG-09** | Structured fields MUST be passed via `extra={...}`, NOT concatenated into the message. The JSON formatter promotes them to top-level keys. | all |
| <a id="LG-10"></a>**LG-10** | Headless services (systemd timers, daemons, mobile background tasks) MUST configure logging at their entry point too, with `JournalHandler` (Linux) / equivalent OS sink in addition to the file sink. | service |
| <a id="LG-11"></a>**LG-11** | Operation start/end MUST be logged with a stable `operation` name field at INFO for any function ≥100 ms OR any function that crosses a process / network / storage boundary, so flows are reconstructable from logs alone (without needing the trace backend up). Short pure-Python helpers are exempt. | all |
| <a id="LG-12"></a>**LG-12** | Log emission MUST be cheap (≤100 µs per record). Heavy formatting / IO MUST happen in the handler, not the call site. | all |
| <a id="LG-13"></a>**LG-13** | High-frequency log sites (inner loops, per-event callbacks, per-progress-tick) MUST use rate-limiting or sampling so a runaway condition does not flood every sink. The canonical shapes are in §15. | all |
| <a id="LG-14"></a>**LG-14** | Custom correlation context (request ID, user ID, lab name) MUST propagate through `contextvars.ContextVar`, not thread-local storage or function arguments. The standard formatter promotes registered contextvars to log fields. See §16. | all |
| <a id="LG-15"></a>**LG-15** | Every INFO+ record at a notable lifecycle / security / boundary event MUST carry a stable `event` field in `<subject>.<action>` form (e.g. `vault.read`, `bootstrap.replace`, `ssh.connect.failed`). Event names MUST be documented in `docs/reference/EVENTS.md` and MUST NOT be renamed without a deprecation cycle — they are alert-target identifiers, treat them like public-API names. See §4.5. | all |
| <a id="LG-16"></a>**LG-16** | Every WARNING and ERROR record MUST answer three questions: **WHAT** (past-tense declarative message — "connection refused", not "connecting..."), **WHY** (structured context as `extra={}` fields — never concatenated into the message), and **HOW** (when remediation is known, an actionable hint in the message OR a `hint` field, per the **EH-09** style). A record that names a failure with no context and no recovery path is a defect. See §4.6. | all |
| <a id="LG-17"></a>**LG-17** | Every ERROR record SHOULD carry a stable `fingerprint` field so observability tools (Sentry / Datadog / OTLP-attached APM) group occurrences of the same logical failure together. The default fingerprint is the exception's fully-qualified type name; the formatter computes this automatically when `exc_info` is set. Sites where args are part of identity (rate-limit-per-endpoint, per-tenant resource errors) MUST override via the `fingerprint` extra field. See §4.7. | all |

### Per-kind compliance matrix

The `Applies to` column is machine-readable. `catch-up` and
future audit tooling filter by the project's detected kind so
non-applicable rules don't surface as violations. Closes FEEDBACK
LOG-02 (LG-07's GUI-only scope) + provides the contract pattern
ASPIRE-02 prescribes for kind-aware standards.

| Kind | Applicable LG-* rules |
|---|---|
| **all** | LG-01, LG-02, LG-03, LG-04, LG-05, LG-06, LG-08, LG-09, LG-11, LG-12, LG-13, LG-14 |
| **gui-app** | All-applicable + **LG-07** (log-viewer dialog) |
| **service** | All-applicable + **LG-10** (headless / journald sink) |
| **fastapi**, **django** | All-applicable. LG-06's *stdout-aggregator* branch typically applies (cloud-native deploy); see [`docs/recipes/cloud-native-logging.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/recipes/cloud-native-logging.md). |
| **cli**, **library** | All-applicable. LG-06's *rotating-file* branch typically applies. |

---

## 1. Mental model

Logging exists to answer four questions, in order of how often they get asked:

1. **What is happening right now?** — operator watching a tail.
2. **Why did this fail?** — debugger reading after the fact.
3. **What was the user's full session?** — correlation across services.
4. **Are we within budget?** — metrics derived from logs.

A logging stack that cannot answer all four is incomplete. The standard prescribes:

- **Per-module loggers** so output can be filtered by source.
- **Structured records** so machines can parse them.
- **Correlation IDs** (OpenTelemetry trace_id / span_id) so a single user action can be reconstructed end-to-end.
- **Multiple sinks** — console for the human, rotating file for the operator, OTLP/syslog for the aggregator.
- **Redaction** so secrets don't leak.
- **A GUI viewer** so non-CLI users can see what's happening.

The standard library `logging` module is **sufficient** for all of this, and it is the **canonical reference path** every drop-in template in this doc is written against. The binding requirement is the *output contract*, not the library identity: records satisfy the per-module-logger, single-setup-point, JSON-when-not-a-TTY, redaction, OTel-context, and `extra={}`-promotion rules (**LG-01..LG-04**, **LG-08..LG-09**), reach a searchable aggregator (**LG-06**), and — where the JSON sink feeds the federation — match the OB-09/OB-10 canonical `LogRecord` wire byte-for-byte (§5.3). `structlog` / `loguru` are therefore **not adopted by default** — stdlib meets the contract with no extra dependency or learning curve — but a consumer that wires `structlog`/`loguru` (or any other framework) so it demonstrably produces the same output contract (and passes the OB-10 byte-exact pin where the wire is in scope) is conformant; record the choice per [`principles/non-adoption-is-valid.md`](principles/non-adoption-is-valid.md). Bind the output, not the import.

---

## 2. Logger acquisition

### 2.1 Naming

Every logger MUST be acquired with the **canonical module path** as its name. The conventional shape is `<app>.<subpackage>.<module>`. For Python, this is `__name__`:

```python
# RIGHT
import logging
LOG = logging.getLogger(__name__)
```

If the canonical `__name__` is awkward (very deep path, or test code), an explicit hierarchical string is acceptable:

```python
# RIGHT (still hierarchical)
LOG = logging.getLogger("<app>.gui.dashboard")
```

**MUST NOTs:**
- `logging.info("...")` (uses the root logger; can't be filtered).
- `logging.getLogger("dashboard")` (flat name; can't be category-filtered).
- A function-level `logging.getLogger(...)` call (allocation per call; create once at module scope).

### 2.2 The `LOG` convention

The conventional binding is `LOG = logging.getLogger(__name__)` at module scope. Use uppercase `LOG` (some shops use lowercase `log` or `logger` — pick one per project and stick to it).

**Both shapes are acceptable** per this standard (explicit
string literals OR `__name__`), but **`__name__` is preferred
for new code** because it never drifts when modules are
renamed. Existing string-literal codebases MAY remain unchanged.

---

## 3. Levels and what they mean

| Level | Meaning | Example |
|---|---|---|
| **DEBUG** | Internal state useful when chasing a bug. Off in production by default. | `LOG.debug("snapshot tree built: %d nodes", len(nodes))` |
| **INFO** | Normal lifecycle events that an operator wants to see. | `LOG.info("started VM %s", vm.name)` |
| **WARNING** | Something unexpected that the application recovered from. | `LOG.warning("notify-send missing; falling back to log notifier")` |
| **ERROR** | An operation failed. The user (or operator) should know. MUST include `exc_info`. | `LOG.exception("failed to attach disk %s to %s", disk, vm.name)` |
| **CRITICAL** | A condition that may take the process down or makes further work impossible. | `LOG.critical("metrics DB is corrupt; shutting down poller")` |

**MUSTs:**
- INFO is the default level a user sees. The default verbosity SHOULD print INFO + WARN + ERROR.
- DEBUG MUST NOT contain user-facing instructions; it's for engineers.
- ERROR MUST include `exc_info` — see rule **LG-05**. The wrong shape (`LOG.error(f"failed: {e}")`) loses the traceback.
- CRITICAL MUST trigger the crash-reporter pipeline if reached during a top-level handler — see [`ERROR_HANDLING.md`](ERROR_HANDLING.md) §6.

---

## 4. Message style and structured fields

### 4.1 Lazy %-formatting

```python
# RIGHT — lazy: format only happens if level passes
LOG.info("processing %s for %s", item, user)

# WRONG — eager: format runs even if INFO is filtered
LOG.info(f"processing {item} for {user}")
```

The cost difference is small per record but adds up to milliseconds in tight loops. More importantly, the lazy form keeps the message template stable across records, which makes log aggregators happy (they group by template).

### 4.2 Structured `extra={}` fields

```python
# RIGHT — fields promoted to JSON top-level keys
LOG.info(
    "vm started",
    extra={"vm_name": vm.name, "vm_uuid": vm.uuid, "duration_ms": elapsed_ms},
)

# WRONG — fields buried in the message
LOG.info("vm %s (uuid=%s) started in %d ms", vm.name, vm.uuid, elapsed_ms)
```

The wrong shape is fine for human eyes; the right shape lets the aggregator filter on `vm_name="prod-1"` without regex.

### 4.3 The `operation` field convention (rule LG-11)

Every operation that crosses a meaningful boundary (≥100 ms wall,
OR a process / network / storage hop) MUST log a start AND end
record with a stable `operation` name. This is the single most
useful pattern for reconstructing flows from logs alone — and the
reason it's a MUST, not a SHOULD, is that an OTel backend is not
always available (consumer in dev, production with OTel disabled,
incident response after the collector goes down). Logs MUST stand
on their own.

Short pure-Python helpers (sub-millisecond, no I/O) are exempt;
adding start/end records would dominate the runtime and add noise
without diagnostic value.

```python
LOG.info("operation start", extra={"operation": "vm.create", "vm_name": spec.name})
try:
    handle = build_and_run(spec, provider)
except AppError:
    LOG.exception("operation failed", extra={"operation": "vm.create", "vm_name": spec.name})
    raise
else:
    LOG.info(
        "operation end",
        extra={"operation": "vm.create", "vm_name": spec.name, "duration_ms": ...},
    )
```

This lets a single log query (`operation:vm.create`) list every
create attempt across users / time. Pair with the `event` field
(rule **LG-15**, §4.5) when the operation crosses a security or
lifecycle boundary worth alerting on independently.

### 4.5 The `event` field convention (rule LG-15)

The `event` field is the **alert-target identifier**. An operator
who wants to be paged when `bootstrap.replace` fires twice in
ten seconds writes an alert against `event:bootstrap.replace` —
not a regex on the message text (which rots the moment a
contributor reformats the string).

```python
# Vault read — security-boundary event (audit=True per SC-17).
LOG.info(
    "vault key read",
    extra={
        "audit": True,
        "event": "vault.read",
        "key": key_name,
        "backend": self.backend,
    },
)

# Bootstrap re-entry — lifecycle event.
LOG.warning(
    "bootstrap replacing existing AppContext",
    extra={
        "audit": True,
        "event": "bootstrap.replace",
        "prior_app": prior.app_name,
        "new_app": new_ctx.app_name,
    },
)

# Failure with a specific recovery path — boundary event.
LOG.warning(
    "runtime snapshot failed; bootstrap continues without it",
    extra={
        "audit": True,
        "event": "bootstrap.runtime_snapshot_failed",
        "error_type": type(exc).__name__,
        "error_message": str(exc),
    },
)
```

**The naming contract** — `<subject>.<action>` form, all
lower-case, dot-delimited, no verbs in `<subject>` (the subject
is a noun: `vault`, `bootstrap`, `ssh`, `process`, `crash`).
The action MAY itself be dotted when the granularity matters
(`ssh.connect.failed` vs `ssh.connect.timeout` vs `ssh.connect`
for success). Once shipped, the name is **public-API-stable**:
consumers' alerts, dashboards, and SIEM rules grep it. Rename
only with a deprecation cycle (emit BOTH names for one MINOR,
then drop the old).

**What deserves an `event` field** (the MUST surface):

| Category | Example events |
|---|---|
| **Security boundaries** (SC-17 audit set) | `vault.read`, `vault.write`, `auth.fail`, `priv.escalate`, `ssh.connect`, `exec.user` |
| **Lifecycle transitions** | `bootstrap.replace`, `bootstrap.shutdown`, `reload.unrecoverable` |
| **External-boundary failures with a recovery path** | `ssh.connect.timeout`, `db.connect.refused`, `http.5xx.retry` |
| **Defense-in-depth events** | `crash.redaction_failed`, `process.reap_timeout`, `process.streamer_read_failure` |
| **Quota / capacity boundary hits** | `rate_limit.exceeded`, `pool.saturated`, `sse.cap.reached` |

**What does NOT deserve an `event` field**:

- Routine DEBUG / TRACE-level state dumps.
- Per-record progress noise (use the `operation` field per LG-11
  + duration_ms instead).
- Free-form INFO messages without a corresponding alert /
  dashboard / runbook entry. If no one would alert on it, the
  event name has no operational meaning.

The discipline: an event name **without** a `docs/reference/EVENTS.md`
entry is a defect. The registry IS the contract — see
[`docs/reference/EVENTS.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/reference/EVENTS.md) for the
project's actual event catalogue and §19 for the registry shape.

### 4.4 Forbidden in messages

- **No newlines.** A record with `\n` confuses aggregators that delimit on newline. Use `extra={...}` for multi-field info.
- **No trailing periods.** Log messages are statements, not sentences.
- **No emoji** — they break terminal renderers and confuse some aggregators.

### 4.6 Message content — what / why / how (rule LG-16)

A log line is the glass operators (and your future self) see
through. The plumbing rules (lazy %-formatting, `extra={}`, OTel
keys) make the line **machine-readable**. **LG-16 makes it
human-actionable.**

Every WARNING and ERROR record MUST answer three questions:

**WHAT** — the message itself. Past-tense, declarative, the
event named in five words or fewer:

```python
# RIGHT — past-tense declarative; the message is what happened
LOG.warning("SSH authentication failed for user %s", username)
LOG.error("vault decrypt failed: bad passphrase")

# WRONG — present continuous; what's the state right now?
LOG.warning("authenticating user %s over SSH", username)

# WRONG — question form; logs are statements
LOG.error("could not decrypt the vault?")
```

**WHY** — the structured context. Every fact the next reader
needs to triage the failure goes as a field in `extra={}` —
never concatenated into the message:

```python
# RIGHT — fields are queryable
LOG.warning(
    "SSH authentication failed",
    extra={
        "event": "ssh.auth.failed",        # LG-15
        "username": username,
        "host": host,
        "port": port,
        "auth_method": auth_method,        # password / key / agent
        "attempt": attempt_count,
        "retry_after_seconds": backoff,
    },
)

# WRONG — fields buried in message; the aggregator can't query them
LOG.warning("ssh auth failed for %s@%s:%d (method=%s, attempt=%d)",
            username, host, port, auth_method, attempt_count)
```

**HOW** — when remediation is known, include an actionable hint.
Two equivalent shapes; pick the one that fits:

```python
# Shape A — hint appended to message (the EH-09 shape; works
# well for short, single-action remediations).
LOG.error(
    "vault decrypt failed: bad passphrase — "
    "rerun with --passphrase-file=<path> or check ~/.config/<app>/vault.key",
    extra={"event": "vault.decrypt.failed", "vault_path": str(path)},
)

# Shape B — hint as a structured field (better for multi-step
# remediation or runbook references; renders cleanly in JSON).
LOG.error(
    "vault decrypt failed: bad passphrase",
    extra={
        "event": "vault.decrypt.failed",
        "vault_path": str(path),
        "hint": "rerun with --passphrase-file=<path>",
        "runbook": "RB-vault-recovery",     # docs/runbooks/vault-recovery.md
    },
)
```

When remediation is genuinely unknown (an unexpected bug — the
catch-all `except Exception` branch in a translation seam), say
so explicitly rather than leaving the field absent:

```python
LOG.error(
    "unexpected error in monitor poll",
    exc_info=True,
    extra={
        "event": "monitor.poll.unexpected",
        "hint": "report at github.com/<org>/<repo>/issues with the crash-report path below",
    },
)
```

**The discipline test** (apply at code review):

> *Read the log line. Can someone who has never seen the code
> (a) name what failed, (b) name why, (c) take a useful next
> step — without opening the source?*

If yes to all three: ship it. If no to any: rewrite. This is
why LG-16 is a MUST at WARNING+ specifically — informational
INFO records get a lighter contract; warnings and errors are
where operators look for action.

**Cross-references:**

- **EH-09** — the same what/why/how discipline for user-facing
  *exception messages*. LG-16 generalises it to log records;
  the two rules share the action-suffix style and the
  pinned-substring testing pattern.
- **LG-15** §4.5 — the `event` field is the alert-target
  identifier; LG-16 is the human-readable companion.
- **LG-09** §4.2 — the structured-fields rule (`extra={}` not
  concatenation) is what makes the "WHY" half of LG-16
  mechanically queryable.

### 4.7 Error fingerprinting (rule LG-17)

When ERROR records flow into Sentry / Datadog / OTLP-attached
APM, the platform groups occurrences for paging and trend
analysis. **The default grouping algorithm is poor for typed
exceptions with parameterised messages:**

```python
# Without LG-17, these three messages produce three "different"
# Sentry issues — even though the root cause is identical:
LOG.exception("vault read failed: key=db_password not found")
LOG.exception("vault read failed: key=api_token not found")
LOG.exception("vault read failed: key=ssh_passphrase not found")

# A flooded alerting pipeline → real signal lost in three-per-day
# duplicate noise.
```

**The fix is a stable `fingerprint` field.** Two patterns:

**Default (automatic) fingerprint** — when an ERROR record
carries `exc_info`, the standard formatter computes
`fingerprint = f"{type(exc).__module__}.{type(exc).__qualname__}"`
automatically. Three different `VaultKeyNotFoundError` raises
above all fingerprint to
`mgf.common.vault.VaultKeyNotFoundError` — one Sentry issue,
trend over time, one page.

```python
# Automatic: no override needed; the type IS the identity.
raise VaultKeyNotFoundError(key=key_name)
# Caught + logged at the translation seam:
LOG.exception("vault read failed", extra={"event": "vault.read.failed"})
# Formatter adds: fingerprint = "mgf.common.vault.VaultKeyNotFoundError"
```

**Explicit override** — when args ARE part of identity
(rate-limit-per-endpoint, per-tenant resource exhaustion,
per-host timeout patterns), override via the `fingerprint`
extra field:

```python
# An HTTP 429 on /api/v1/orders is operationally distinct from a
# 429 on /api/v1/users — different upstream, different runbook,
# different alert routing.
LOG.error(
    "rate limit exceeded",
    extra={
        "event": "rate_limit.exceeded",
        "endpoint": endpoint,
        "tenant_id": tenant_id,
        "fingerprint": f"rate_limit.{endpoint}",  # endpoint-grouped
    },
)

# A timeout from postgres is operationally identical regardless
# of which row was being read; default fingerprint (exception
# type) is correct — don't override.
```

**The override discipline:**

- The fingerprint MUST be stable across runs (no timestamps, no
  random IDs, no UUIDs).
- The fingerprint MUST be coarse enough that an operator
  actually wants to be paged once per fingerprint, not per
  occurrence. `rate_limit.exceeded.tenant_42.endpoint_orders.user_alice_at_2026-05-14`
  is a unique string, not a fingerprint.
- The fingerprint MUST contain ONLY data that's safe in an
  alert payload (no user input, no PII — those are scrubbed
  by the redactor before egress, but the fingerprint string
  arrives at the alert backend before the redactor runs for
  some sinks). When in doubt, hash the variable component:
  `f"rate_limit.{hashlib.sha256(endpoint.encode()).hexdigest()[:8]}"`.

**Coverage carve-out:** LG-17 is a SHOULD (not a MUST) because
many ERROR sites don't ship to a grouping aggregator at all
(local-file-only consumers, dev environments). The cost of
adding a fingerprint to every ERROR is small; the gain when a
consumer DOES add Sentry / Datadog is large. Default to setting
it; carve out only when you've verified no grouping backend is
in scope.

**Cross-references:**

- **LG-15** §4.5 — the `event` field clusters records by event
  type for alerting; `fingerprint` clusters error occurrences
  for grouping. They are complementary, not redundant.
- **EH-09** + **SC-11** — actionable error messages and
  traceback scrubbing; the fingerprint flows through the same
  egress path and is subject to the same redaction
  invariants.

---

## 5. Configuration (rule LG-02)

### 5.1 Single setup point

Each entry point calls one setup function:

```python
# src/<app>/cli/__main__.py
from <app>.observability.logging_setup import setup_logging

@click.group(...)
def main(ctx, log_level, log_format):
    setup_logging(level=log_level, fmt=log_format)
    ...
```

### 5.2 Drop-in template — `<app>/observability/logging_setup.py`

```python
"""Single-point logging configuration.

Call ``setup_logging(...)`` once from each entry point (CLI main,
GUI app.run, scheduler service entry, mobile background task entry).
Calling it twice is harmless (idempotent — `force=True`).

Sinks installed by default:
  - Console (stderr) — text on TTY, JSON when not a TTY (LG-02).
  - Rotating file under <state>/logs/<app>.log.

Optional sinks (opt-in via env / config):
  - Journald (Linux) when running under systemd.
  - OTLP exporter when OTLP_ENDPOINT is set.
  - Syslog when SYSLOG_ADDR is set.
  - HTTP webhook when LOG_HTTP_SINK is set.

Every record carries OpenTelemetry trace_id + span_id when a span is
active (LG-03). Sensitive fields are redacted (LG-04).
"""

from __future__ import annotations

import json
import logging
import os
import sys
from logging.handlers import RotatingFileHandler
from pathlib import Path
from typing import Any, Literal

from <app>.observability.json_formatter import JsonFormatter
from <app>.observability.text_formatter import TextFormatter
from <app>.observability.redaction import Redactor

LogFormat = Literal["text", "json", "auto"]

def _state_dir() -> Path:
    """Lazy resolve so tests can monkeypatch HOME (rule CF-04 generalised)."""
    xdg = os.environ.get("XDG_STATE_HOME")
    base = Path(xdg) if xdg else Path.home() / ".local" / "state"
    return base / "<app>"

def setup_logging(
    *,
    level: str = "INFO",
    fmt: LogFormat = "auto",
    log_file: Path | None = None,
    redactor: Redactor | None = None,
    enable_otlp: bool | None = None,
) -> None:
    """Configure the root logger and standard handlers.

    Parameters
    ----------
    level : str
        Root level (DEBUG/INFO/WARNING/ERROR/CRITICAL).
    fmt : LogFormat
        "text" for human-readable, "json" for structured, "auto" to
        pick based on ``stdout.isatty()``.
    log_file : Path or None
        Override the rotating-file destination. Default:
        ``<state>/logs/<app>.log``.
    redactor : Redactor or None
        Custom redactor. Default: ``Redactor.default()``.
    enable_otlp : bool or None
        Enable the OTLP log exporter. Default: ``None`` (env-driven).
    """
    redactor = redactor or Redactor.default()
    chosen_fmt = _resolve_format(fmt)

    handlers: list[logging.Handler] = []

    # 1) Console handler — always present.
    console = logging.StreamHandler(stream=sys.stderr)
    console.setFormatter(_make_formatter(chosen_fmt, redactor))
    handlers.append(console)

    # 2) Rotating file handler — always present.
    file_path = log_file or (_state_dir() / "logs" / "<app>.log")
    file_path.parent.mkdir(parents=True, exist_ok=True)
    file_handler = RotatingFileHandler(
        filename=str(file_path),
        maxBytes=10 * 1024 * 1024,        # 10 MiB
        backupCount=5,                    # keep 5 rotations
        encoding="utf-8",
    )
    # File handler ALWAYS uses JSON — no colours, no TTY heuristics.
    file_handler.setFormatter(JsonFormatter(redactor=redactor))
    handlers.append(file_handler)

    # 3) Optional sinks (opt-in via env).
    _maybe_add_journald(handlers, redactor)
    _maybe_add_syslog(handlers, redactor)
    _maybe_add_otlp(handlers, redactor, enable_otlp=enable_otlp)
    _maybe_add_http(handlers, redactor)

    logging.basicConfig(
        level=getattr(logging, level.upper()),
        handlers=handlers,
        force=True,  # override any handlers from earlier setup_logging calls
    )

    # Lower the noise from chatty third-party libraries by default.
    for noisy in ("urllib3", "asyncio", "PIL", "matplotlib"):
        logging.getLogger(noisy).setLevel(logging.WARNING)

def _resolve_format(fmt: LogFormat) -> Literal["text", "json"]:
    if fmt == "auto":
        return "text" if sys.stderr.isatty() else "json"
    return fmt

def _make_formatter(fmt: Literal["text", "json"], redactor: Redactor) -> logging.Formatter:
    if fmt == "json":
        return JsonFormatter(redactor=redactor)
    return TextFormatter(redactor=redactor)

def _maybe_add_journald(handlers: list[logging.Handler], redactor: Redactor) -> None:
    """Add JournalHandler when running under systemd."""
    if "JOURNAL_STREAM" not in os.environ and "INVOCATION_ID" not in os.environ:
        return  # not under systemd
    try:
        from systemd.journal import JournalHandler
    except ImportError:
        return
    h = JournalHandler(SYSLOG_IDENTIFIER="<app>")
    h.setFormatter(JsonFormatter(redactor=redactor))
    handlers.append(h)

def _maybe_add_syslog(handlers: list[logging.Handler], redactor: Redactor) -> None:
    addr = os.environ.get("<APP>_SYSLOG_ADDR")
    if not addr:
        return
    from logging.handlers import SysLogHandler
    host, _, port = addr.partition(":")
    h = SysLogHandler(address=(host, int(port) if port else 514))
    h.setFormatter(JsonFormatter(redactor=redactor))
    handlers.append(h)

def _maybe_add_otlp(
    handlers: list[logging.Handler],
    redactor: Redactor,
    *,
    enable_otlp: bool | None,
) -> None:
    """Add the OTLP log exporter when OTLP is enabled.

    Lazy import — opentelemetry deps are optional.
    """
    if enable_otlp is None:
        enable_otlp = bool(os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT"))
    if not enable_otlp:
        return
    try:
        from <app>.observability.otel_setup import build_otlp_log_handler
    except ImportError:
        return
    h = build_otlp_log_handler()
    h.setFormatter(JsonFormatter(redactor=redactor))
    handlers.append(h)

def _maybe_add_http(handlers: list[logging.Handler], redactor: Redactor) -> None:
    url = os.environ.get("<APP>_LOG_HTTP_SINK")
    if not url:
        return
    from logging.handlers import HTTPHandler
    from urllib.parse import urlparse
    parsed = urlparse(url)
    h = HTTPHandler(host=parsed.netloc, url=parsed.path or "/", method="POST", secure=parsed.scheme == "https")
    h.setFormatter(JsonFormatter(redactor=redactor))
    handlers.append(h)
```

### 5.3 Drop-in template — `<app>/observability/json_formatter.py`

> **Canonical-wire binding (OB-09 / OB-10).** This formatter's output
> IS the federation's canonical `LogRecord` wire. **OBSERVABILITY.md
> owns that schema** (OB-09 the record model, OB-10 the byte-exact
> wire + the generated golden `log_record_fixture.jsonl` /
> `log_record_schema.json`); this `JsonFormatter` is its **Python
> reference emitter**, not an independent shape. The mapping onto the
> OB-09 model: the flat top-level keys below (`ts`, `level`, `logger`,
> `msg`, …) plus every promoted `extra={…}` key are the Python
> projection of OB-09's typed top-level keys, `fields`, and `labels`;
> unknown keys from a newer producer round-trip via `extra`; the
> manifest's `emit_order` fixes key order (do not rely on dict-insertion
> order across languages). A consumer that ships a *different* shape
> here — renamed keys, reordered output, `ensure_ascii=True` — passes
> its own eyes but **fails the OB-10 byte-exact conformance test**, the
> only CI-enforced rule on this wire. An equivalent emitter that
> reproduces the golden byte-for-byte is fully conformant; deviate only
> with a recorded rationale per `non-adoption-is-valid`.

```python
"""JSON formatter that adds OpenTelemetry trace context and runs
the redactor before emit. The output is one JSON object per line —
the format every mainstream aggregator (Loki, ELK, Splunk, Datadog,
Cloudwatch) understands out of the box.

This is the Python reference emitter of the canonical `LogRecord`
wire owned by OBSERVABILITY.md (OB-09 / OB-10). The emitted bytes are
pinned against the shared golden (`log_record_fixture.jsonl` /
`log_record_schema.json`, generated by `tools/gen_log_record_schema.py`);
key order follows the manifest's `emit_order` and the output is raw
UTF-8 (`ensure_ascii=False`) so non-Python emitters match natively.
Adding/renaming a key here MUST go through the OB-09 schema-evolution
rules + a `LOG_SCHEMA_VERSION` bump or the conformance pin fails CI.
"""

from __future__ import annotations

import json
import logging
import time
from typing import Any

from <app>.observability.redaction import Redactor

# Standard LogRecord attributes — anything else is treated as a custom
# field (came from `extra={...}`) and promoted to a top-level JSON key.
_RESERVED = frozenset({
    "name", "msg", "args", "levelname", "levelno", "pathname",
    "filename", "module", "exc_info", "exc_text", "stack_info",
    "lineno", "funcName", "created", "msecs", "relativeCreated",
    "thread", "threadName", "processName", "process",
    "message", "asctime", "taskName",
})

class JsonFormatter(logging.Formatter):
    """One JSON object per record. Always single-line, always UTC."""

    def __init__(self, *, redactor: Redactor) -> None:
        super().__init__()
        self._redactor = redactor

    def format(self, record: logging.LogRecord) -> str:
        # Compose the base record.
        payload: dict[str, Any] = {
            "ts": _iso_utc(record.created),
            "level": record.levelname,
            "logger": record.name,
            "msg": record.getMessage(),
            "module": record.module,
            "func": record.funcName,
            "line": record.lineno,
            "thread": record.threadName,
            "process": record.process,
        }

        # Custom extras.
        for key, value in record.__dict__.items():
            if key in _RESERVED or key.startswith("_"):
                continue
            payload[key] = value

        # OpenTelemetry trace context (LG-03).
        _add_otel_context(payload)

        # Exception info.
        if record.exc_info:
            payload["exception"] = self.formatException(record.exc_info)
        if record.stack_info:
            payload["stack"] = self.formatStack(record.stack_info)

        # Redact (LG-04). The redactor walks the dict in place.
        self._redactor.redact(payload)

        return json.dumps(payload, default=_json_default, ensure_ascii=False)

def _iso_utc(epoch: float) -> str:
    return time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(epoch)) + f".{int((epoch % 1) * 1000):03d}Z"

def _json_default(obj: Any) -> Any:
    """Best-effort JSON encoder for non-JSON-native types."""
    if isinstance(obj, bytes):
        return obj.decode("utf-8", errors="replace")
    return repr(obj)

def _add_otel_context(payload: dict[str, Any]) -> None:
    """Attach trace_id / span_id when an OTel span is active."""
    try:
        from opentelemetry import trace
    except ImportError:
        return
    span = trace.get_current_span()
    ctx = span.get_span_context() if span else None
    if ctx and ctx.is_valid:
        payload["trace_id"] = format(ctx.trace_id, "032x")
        payload["span_id"] = format(ctx.span_id, "016x")
```

### 5.4 Drop-in template — `<app>/observability/text_formatter.py`

```python
"""Human-readable text formatter for TTY console output.

ANSI-coloured by level. Falls back to plain text when stderr is not
a TTY (the JSON formatter takes over via the auto-format heuristic).
"""

from __future__ import annotations

import logging
import sys
import time

from <app>.observability.json_formatter import _RESERVED  # canonical reserved-attr set (§5.3)
from <app>.observability.redaction import Redactor

_LEVEL_COLORS = {
    "DEBUG":    "\033[90m",  # bright black
    "INFO":     "\033[37m",  # light grey
    "WARNING":  "\033[33m",  # yellow
    "ERROR":    "\033[31m",  # red
    "CRITICAL": "\033[1;31m",  # bold red
}
_RESET = "\033[0m"

class TextFormatter(logging.Formatter):
    def __init__(self, *, redactor: Redactor) -> None:
        super().__init__()
        self._redactor = redactor
        self._supports_color = sys.stderr.isatty()

    def format(self, record: logging.LogRecord) -> str:
        ts = time.strftime("%H:%M:%S", time.localtime(record.created))
        msg = record.getMessage()

        # Promote extras to a `key=value` tail so humans see them too.
        # Reserved-attribute deny-list is the SINGLE canonical `_RESERVED`
        # defined in §5.3 (`json_formatter.py`) — imported, never re-typed,
        # so the two formatters can never drift (DOC-18).
        extras: dict = {}
        for key, value in record.__dict__.items():
            if key.startswith("_") or key in _RESERVED:
                continue
            extras[key] = value

        self._redactor.redact_text_extras(extras)

        tail = " " + " ".join(f"{k}={v!r}" for k, v in extras.items()) if extras else ""

        head = f"{ts} {record.levelname:<8} {record.name}: {msg}{tail}"

        if record.exc_info:
            head = head + "\n" + self.formatException(record.exc_info)

        if self._supports_color:
            color = _LEVEL_COLORS.get(record.levelname, "")
            return f"{color}{head}{_RESET}"
        return head
```

---

## 6. Sensitive-field redaction (rule LG-04)

### 6.1 What gets redacted

Default redaction applies to:

- **Exact key names** (case-insensitive): `password`, `passwd`, `secret`, `token`, `api_key`, `apikey`, `private_key`, `cookie`, `set-cookie`, `authorization`, `auth`, `session`, `client_secret`, `access_token`, `refresh_token`.
- **Pattern matches** in string values: `Bearer <...>`, `Basic <...>`, JWT-shaped strings (3 base64 segments separated by `.`), PEM blocks (`-----BEGIN ... PRIVATE KEY-----`).
- **Optionally** (per-project): IP addresses, email addresses, full file paths under `$HOME` (replaced with `~/...`).

### 6.2 Drop-in template — `<app>/observability/redaction.py`

```python
"""Redact sensitive fields from log records before any sink sees them.

The redactor walks the record's structured payload (the dict produced
by JsonFormatter, or the extras dict produced by TextFormatter) in
place. The match set is configurable via constructor; defaults are
the well-known secret-bearing keys.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import Any

_DEFAULT_KEY_PATTERNS = frozenset({
    "password", "passwd", "secret", "token", "api_key", "apikey",
    "private_key", "privatekey", "cookie", "set-cookie",
    "authorization", "auth", "session", "client_secret",
    "access_token", "refresh_token", "x-api-key",
})

_BEARER_RE = re.compile(r"\b(Bearer|Basic)\s+[A-Za-z0-9._\-+/=]{8,}\b")
_JWT_RE = re.compile(r"\beyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}\b")
_PEM_RE = re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----", re.S)

_REDACTED = "[REDACTED]"

@dataclass
class Redactor:
    """Walks log payloads and redacts sensitive values.

    Configurable: pass extra key names or value-pattern regexes via
    the constructor. The default configuration covers the common
    secret-bearing keys and Bearer/JWT/PEM value shapes.
    """

    extra_keys: frozenset[str] = field(default_factory=frozenset)
    extra_value_patterns: tuple[re.Pattern[str], ...] = ()

    @classmethod
    def default(cls) -> Redactor:
        return cls()

    def redact(self, payload: dict[str, Any]) -> None:
        """Walk the dict in place, replacing sensitive values."""
        keys_to_redact = _DEFAULT_KEY_PATTERNS | self.extra_keys
        self._walk(payload, keys_to_redact)

    def redact_text_extras(self, extras: dict[str, Any]) -> None:
        """Convenience for the TextFormatter — same as redact()."""
        self.redact(extras)

    # Hard bounds on the walk — see "Worst-case bounds" below.
    _MAX_DEPTH = 64
    _DEPTH_EXCEEDED = "[REDACTION-DEPTH-EXCEEDED]"

    def _walk(
        self,
        node: Any,
        keys: frozenset[str],
        *,
        depth: int = 0,
        seen: set[int] | None = None,
    ) -> Any:
        if depth >= self._MAX_DEPTH:
            return self._DEPTH_EXCEEDED
        if seen is None:
            seen = set()
        if isinstance(node, (dict, list)):
            node_id = id(node)
            if node_id in seen:
                return self._DEPTH_EXCEEDED  # cycle
            seen.add(node_id)
            try:
                if isinstance(node, dict):
                    for k in list(node.keys()):
                        if k.lower() in keys:
                            node[k] = _REDACTED
                        else:
                            node[k] = self._walk(
                                node[k], keys, depth=depth + 1, seen=seen,
                            )
                    return node
                # list
                return [
                    self._walk(v, keys, depth=depth + 1, seen=seen)
                    for v in node
                ]
            finally:
                seen.discard(node_id)  # sibling paths must walk through
        if isinstance(node, str):
            return self._scrub_string(node)
        return node

    def _scrub_string(self, s: str) -> str:
        s = _BEARER_RE.sub(_REDACTED, s)
        s = _JWT_RE.sub(_REDACTED, s)
        s = _PEM_RE.sub(_REDACTED, s)
        for pat in self.extra_value_patterns:
            s = pat.sub(_REDACTED, s)
        return s
```

**MUSTs:**
- The redactor MUST run before the formatter writes to any sink.
- The default key set MUST include the canonical secret-bearing names (above).
- A unit test MUST assert that a record with `extra={"password": "x"}` produces `"password": "[REDACTED]"` in the JSON output.
- Per-project additions go in the `extra_keys` constructor arg, not by editing the default set.
- The walk MUST be depth-capped (`_MAX_DEPTH ≥ 32`) AND cycle-guarded
  (an `id()`-keyed `seen` set with `try / finally` discard so that
  sibling paths — legitimately shared substructure — are walked
  both times; only true cycles are caught). Without these bounds,
  a pathologically-nested payload raises `RecursionError` *inside
  the redactor itself*, mid-scrub, which propagates out to the
  `LOG.info(...)` call site — a debug-hostile failure mode.

**Worst-case bounds (rationale).** A redactor that crashes on
adversarial input is worse than one that doesn't catch every
secret: the crash propagates to the calling logger and ultimately
to the caller's exception handler. The depth cap inserts a
sentinel; the cycle guard short-circuits the recursion. Both
return promptly; the caller's `LOG.info(...)` always returns.

---

## 7. Correlation — OpenTelemetry (rule LG-03)

Every L2 project uses OpenTelemetry as the **strong, reliable tracing path**.

### 7.1 What you get

- **Trace ID** + **span ID** automatically attached to every log record (rule LG-03, via `JsonFormatter._add_otel_context`).
- **Spans** for every operation, with start/end times, attributes, and parent/child relationships.
- **OTLP export** to any OTel-compatible backend (Tempo, Jaeger, Datadog, Honeycomb, AWS X-Ray, etc.).
- **Auto-instrumentation** for `httpx`, `paramiko`, `subprocess` (via wrapper), DB drivers.

### 7.2 Drop-in template — `<app>/observability/otel_setup.py`

```python
"""OpenTelemetry initialization — Tier C observability.

Lazy imports — opentelemetry deps are optional. Calling
``setup_otel(enabled=False)`` short-circuits and is safe even when
the OTel SDK is not installed.

Endpoints + service identity come from standard OTel env vars:
  - OTEL_EXPORTER_OTLP_ENDPOINT
  - OTEL_EXPORTER_OTLP_HEADERS
  - OTEL_SERVICE_NAME      (default: "<app>")
  - OTEL_RESOURCE_ATTRIBUTES
"""

from __future__ import annotations

import contextlib
import logging
import os
from typing import Iterator

def setup_otel(*, enabled: bool | None = None, service_name: str = "<app>") -> None:
    """Initialize the OpenTelemetry SDK + OTLP exporter.

    Idempotent — call once per entry point. When disabled (default if
    OTEL_EXPORTER_OTLP_ENDPOINT is unset and `enabled` is None), the
    no-op tracer remains active and `trace.get_current_span()` keeps
    returning the no-op span; `trace_id` is NOT added to log records.
    """
    if enabled is None:
        enabled = bool(os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT"))
    if not enabled:
        return

    try:
        from opentelemetry import trace
        from opentelemetry.sdk.resources import Resource
        from opentelemetry.sdk.trace import TracerProvider
        from opentelemetry.sdk.trace.export import BatchSpanProcessor
        from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
    except ImportError:
        logging.getLogger(__name__).warning(
            "OTEL_EXPORTER_OTLP_ENDPOINT is set but opentelemetry SDK is not installed; "
            "install with `pip install <app>[observability]`"
        )
        return

    resource = Resource.create({"service.name": os.environ.get("OTEL_SERVICE_NAME", service_name)})
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)

    # Auto-instrument common libraries.
    _maybe_instrument_httpx()
    _maybe_instrument_subprocess()

def _maybe_instrument_httpx() -> None:
    try:
        from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
        HTTPXClientInstrumentor().instrument()
    except ImportError:
        pass

def _maybe_instrument_subprocess() -> None:
    """Wrap our own shell wrapper instead of monkey-patching subprocess.

    See <app>/common/shell.py — when otel is active, run_command opens
    a span around each invocation and tags argv[0], returncode, and
    duration.
    """

@contextlib.contextmanager
def operation_span(name: str, **attributes) -> Iterator[None]:
    """Open a span around an operation. Attributes auto-attach.

    Usage:
        with operation_span("vm.create", vm_name=spec.name):
            handle = build_and_run(spec, provider)

    When OTel is not initialised, this is a no-op (returns the
    no-op span; ``with`` block runs normally).
    """
    try:
        from opentelemetry import trace
        tracer = trace.get_tracer("<app>")
    except ImportError:
        yield
        return
    with tracer.start_as_current_span(name) as span:
        for k, v in attributes.items():
            span.set_attribute(k, v)
        yield

def build_otlp_log_handler() -> logging.Handler:
    """Build an OTLP log exporter handler (used by setup_logging)."""
    from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
    from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
    from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter

    provider = LoggerProvider()
    provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
    return LoggingHandler(level=logging.NOTSET, logger_provider=provider)
```

### 7.3 Wrapping operations

```python
# src/<app>/operations/clone.py
from <app>.observability.otel_setup import operation_span

def full_clone(provider, source_handle, *, name) -> EntityHandle:
    with operation_span("clone.full", source=source_handle.name, target=name):
        ...
```

**MUSTs:**
- Every L2 use case (the function the CLI / GUI invokes) MUST be wrapped in an `operation_span`.
- Span names MUST follow `<subject>.<verb>` (e.g., `vm.create`, `clone.full`, `backup.export`).
- Attributes attached to the span MUST use snake_case keys.

### 7.4 What kinds of failures actually appear in traces

A subtle and important consequence of the wrapping shape: **traces only show *execution* failures, not *validation* failures.**

Reason: parameter validation, name normalisation, and entity-handle resolution all happen at the entry-point boundary (CLI / GUI) — *upstream* of the L2 use case. By the time `operation_span` opens, the inputs are already valid and the entity exists. So a span fires only if the validated, well-formed operation reaches some real-world failure later (libvirt rejected the start, the disk file is missing, `qemu-img convert` exited non-zero, the snapshot name already exists, etc.).

Concretely, you will **not** see:
- "VM name has illegal characters" — rejected at the CLI parser.
- "no VM named X" — caught by `_resolve_handle` in the CLI before the operation runs.
- "spec file is missing required field" — caught by the spec loader.

You **will** see:
- "libvirt rejected `start_domain`" — provider call inside the span.
- "qemu-img convert exited 2" — subprocess child span with non-zero `returncode`.
- "host port already forwarded" — caught inside `port_forward.add` after the validation pass.

This is the right scope. Validation errors are the user's problem; their feedback channel is the immediate CLI output / GUI dialog. Execution errors are operational; their feedback channel is the trace. Mixing them would inflate trace volume with low-signal "user typed it wrong" entries while diluting the high-signal "the platform broke" entries.

**Practical implication when demoing:** a "let's see the failure span!" demo via the CLI is surprisingly hard to construct because most CLI commands cleanly reject bad input *before* opening any span. The reliable demo path is a Python one-liner that explicitly raises inside `operation_span`:

```python
from <app>.observability import setup_otel, operation_span
from opentelemetry import trace
setup_otel()
try:
    with operation_span("demo.failing", purpose="failure-demo"):
        raise RuntimeError("synthetic failure for demo")
except RuntimeError:
    pass
trace.get_tracer_provider().shutdown()
```

---

## 8. Sinks summary

| Sink | When active | Format | Notes |
|---|---|---|---|
| Console (stderr) | Always | text on TTY, JSON otherwise | Default level INFO in CLI; configurable via `--log-level` |
| Rotating file | Always | JSON | 10 MiB × 5 rotations under `<state>/logs/<app>.log` |
| Journald | Only when running under systemd | JSON | Auto-detected via `JOURNAL_STREAM` / `INVOCATION_ID` env |
| Syslog | When `<APP>_SYSLOG_ADDR=host:port` set | JSON | TCP/UDP per OS |
| OTLP logs | When `OTEL_EXPORTER_OTLP_ENDPOINT` set | OTel proto | Requires `opentelemetry` extra installed |
| HTTP webhook | When `<APP>_LOG_HTTP_SINK=http://...` set | JSON POST | One record per request (no batching by default) |

**Web-shape note (LG-06).** Container-orchestrated services
(Kubernetes, Docker, Heroku, Cloud Run, Fly) typically deploy with
**stdout-only** logging — the orchestrator captures stdout/stderr
and forwards to the platform aggregator (Loki / CloudWatch /
Stackdriver) per 12-factor §XI. The rotating-file sink is omitted
(filesystem is ephemeral; rotation produces noise on container
restart). See [`docs/recipes/cloud-native-logging.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/recipes/cloud-native-logging.md)
for the canonical wiring + the wire-shape guidance, and
[`web/README.md`](web/README.md) for the cross-domain narrative.

---

## 9. The GUI log-viewer dialog (rule LG-07)

Every desktop app MUST ship a log-viewer dialog accessible from the Help menu. It reads the rotating file sink and renders records with level-based colouring, filter controls (by level, by logger), and a Copy button.

### 9.1 Drop-in template — `<app>/gui/dialogs/log_viewer_dialog.py`

```python
"""Log-viewer dialog — reads the rotating file sink and renders it
with level filtering and search.

Designer file: gui/ui/log_viewer.ui (a QDialog with a QPlainTextEdit
for the body, a QComboBox for level filter, a QLineEdit for search,
and Copy/Clear/Open-folder buttons).

Tail mode: reads new lines as the file grows (5-second poll) so the
user can watch a long operation without re-opening the dialog. Off
by default; toggled by a checkbox.
"""

from __future__ import annotations

import json
from pathlib import Path

from PySide6.QtCore import QTimer, Qt
from PySide6.QtWidgets import QDialog, QPlainTextEdit

from <app>.gui.ui_loader import load_ui, require_child

class LogViewerDialog:
    def __init__(self, log_file: Path, parent=None) -> None:
        self.dialog = load_ui("log_viewer.ui")
        self._body: QPlainTextEdit = require_child(self.dialog, "logBody", QPlainTextEdit)
        self._log_file = log_file
        self._tail_timer = QTimer(self.dialog)
        self._tail_timer.setInterval(5000)
        self._tail_timer.timeout.connect(self._refresh)
        self._refresh()

    def _refresh(self) -> None:
        if not self._log_file.exists():
            self._body.setPlainText(f"(no log file at {self._log_file})")
            return
        text_lines: list[str] = []
        with self._log_file.open(encoding="utf-8", errors="replace") as fh:
            for line in fh.readlines()[-2000:]:  # cap at last 2000 lines
                text_lines.append(self._render_line(line))
        self._body.setPlainText("\n".join(text_lines))

    def _render_line(self, raw: str) -> str:
        try:
            rec = json.loads(raw)
        except json.JSONDecodeError:
            return raw.rstrip()
        ts = rec.get("ts", "?")
        lvl = rec.get("level", "?")
        logger = rec.get("logger", "?")
        msg = rec.get("msg", "")
        extras = " ".join(
            f"{k}={v!r}" for k, v in rec.items()
            if k not in {"ts", "level", "logger", "msg", "module", "func", "line", "thread", "process"}
        )
        return f"{ts} {lvl:<8} {logger}: {msg} {extras}".rstrip()
```

**MUSTs:**
- The dialog MUST tail the file — refresh on a timer so a watching user sees new records.
- A "Open log folder" button MUST open the log directory in the OS file manager (`QDesktopServices.openUrl(QUrl.fromLocalFile(str(folder)))`).
- The dialog MUST gracefully handle a missing log file (first run, never written) and a corrupt JSON line (display raw).

**Common confusion:** an in-app "log panel" widget that
streams worker-progress messages is NOT the same thing as the
GUI log viewer LG-07 requires. LG-07 is specifically about
**the rotating file logs**, presented in a dialog reachable
from Help → View Logs, with level filter / search / copy /
tail. A worker-progress panel is a separate UX surface.

---

## 10. Headless services (rule LG-10)

systemd timer / service entry points MUST configure logging too — they cannot inherit the CLI's setup because they run as separate processes.

### 10.1 Pattern

```python
# scheduler/cli_entry.py — invoked by systemd
def main() -> int:
    from <app>.observability.logging_setup import setup_logging
    from <app>.observability.otel_setup import setup_otel

    setup_logging(level="INFO", fmt="json")  # always JSON for headless
    setup_otel()
    ...
```

systemd auto-detects journald (`JOURNAL_STREAM` is set), so the journald handler attaches automatically. systemd captures stdout/stderr too, but JSON-on-stdout to journald is awkward; the journald handler is the right path.

**MUSTs:**
- Headless entry points MUST call `setup_logging(fmt="json")` (not `"auto"`) — there is no TTY to detect.
- `journalctl --user -u <unit>` MUST show structured records.
- Failure inside the headless tick MUST NOT prevent subsequent ticks (rule **EH-10** never-raises).

**Web-shape note (LG-10).** Container-orchestrated web services
(Kubernetes Deployments, Docker containers, Cloud Run revisions)
already run as headless processes BUT the orchestrator captures
stdout/stderr — there is no `journald` to write to from inside a
container. LG-10 is therefore **N/A under cloud-native deployment**;
the cloud-native consumer reaches LG-06's stdout/stderr branch
instead. See [`docs/recipes/cloud-native-logging.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/recipes/cloud-native-logging.md)
+ [`web/README.md`](web/README.md). LG-10 still applies for systemd
services running on a host with journald (single-server VMs,
on-prem desktop services).

---

## 11. Testing pitfalls

When you adopt the standard logging stack and your `pyproject.toml`
has `filterwarnings = ["error"]` (recommended for any project that
takes warnings seriously), `RotatingFileHandler` will trip the test
suite with `PytestUnraisableExceptionWarning` on every test that
calls `setup_logging` — the file object is closed by Python's GC
during pytest teardown, and on Python 3.12+ that GC fires a
`ResourceWarning` which the filter escalates to an error.

**Fix:** in the test fixture that resets logging between tests,
explicitly close every handler before removing it from the root
logger:

```python
@pytest.fixture(autouse=True)
def _isolate_logging():
    yield
    for handler in logging.root.handlers[:]:
        with contextlib.suppress(Exception):
            handler.close()
        logging.root.removeHandler(handler)
```

Also: **mypy override for the optional observability deps.** When
`pyproject.toml` declares `[observability]` as an extra (which the
default install does NOT pull in), `mypy --strict` fails on the
lazy `import opentelemetry` / `import sentry_sdk` / `import systemd`
unless they are listed as `ignore_missing_imports = true`:

```toml
[[tool.mypy.overrides]]
module = [
    "sentry_sdk",
    "opentelemetry",
    "opentelemetry.*",
    "systemd",
    "systemd.*",
]
ignore_missing_imports = true
```

Also: **the `JOURNAL_STREAM` env var is set when running under
systemd**, including some IDE / CI / dev sessions. Tests that assert
"no journald handler attached" MUST `monkeypatch.delenv("JOURNAL_STREAM", raising=False)` and `monkeypatch.delenv("INVOCATION_ID", raising=False)` first.

Also: **OpenTelemetry tests MUST share a session-scoped
TracerProvider via `conftest.py`.** `set_tracer_provider` is a
one-shot per-process operation in OTel — calling it again
silently does nothing and leaves the no-op proxy in place. So if
two test modules each install their own module-scoped
`TracerProvider`, only the first one wins; every span assertion
in the second module fails because `trace.get_tracer().start_span()`
returns the no-op span. The fix is to put the provider + the
in-memory exporter in `tests/<dir>/conftest.py` at session scope,
and let each test pull a per-test fixture that just clears the
exporter:

```python
# tests/unit/observability/conftest.py
@pytest.fixture(scope="session")
def _session_tracer_provider() -> Iterator[InMemorySpanExporter]:
    exporter = InMemorySpanExporter()
    provider = TracerProvider(resource=Resource.create({"service.name": "test"}))
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
    yield exporter

@pytest.fixture
def in_memory_spans(_session_tracer_provider) -> Iterator[InMemorySpanExporter]:
    _session_tracer_provider.clear()
    yield _session_tracer_provider
```

Test modules then just declare `in_memory_spans` as a parameter and
get a clean exporter per test. **Do NOT install a per-module
provider in test files** — only one wins and the others silently
break.

### 11.1 Pytest stderr-capture distorts full-stack logging benchmarks

When you write a benchmark for rule LG-12 (≤ 100 µs per log record)
that uses the default `setup_logging()` output, the number you
measure will be dominated by **pytest's stderr-capture machinery**
rather than by the logging stack itself. The default setup attaches
a console `StreamHandler` on `sys.stderr`; under pytest, every
record that handler emits goes through capture, which adds real
overhead per line. A first cut at the LG-12 benchmark with
the default setup can easily come in 25–50 % over the budget
even though production emission is well under 60 µs/record —
because the pytest-capture overhead is being measured, not the
logging stack.

**What LG-12 actually constrains.** The rule says "heavy formatting
/ IO MUST happen in the handler, not the call site". The handler
the rule is worried about is the **rotating file sink** (synchronous
I/O, per-record `write()`). That's what a benchmark should measure.
Pytest's stderr-capture is not part of any production path.

**Fix.** After `setup_logging(...)` completes, drop the console
`StreamHandler` so only the rotating file handler participates in
the timed loop:

```python
from logging.handlers import RotatingFileHandler

setup_logging(level="DEBUG", fmt="json", log_file=tmp_path / "perf.log")

# Drop the console StreamHandler so we measure the rotating-file
# sink path (the surface LG-12 really constrains) without also
# paying for pytest's stderr-capture machinery per record.
for handler in logging.root.handlers[:]:
    if not isinstance(handler, RotatingFileHandler):
        logging.root.removeHandler(handler)

# ... warm-up + timed loop here
```

This is a reasonable proxy for production: in production stderr
either lands in journald (a pipe — effectively free) or on a
terminal (similar cost to pytest capture, but not a latency
concern at interactive typing rates). The file sink is the
one-per-record synchronous surface that matters.

### 11.2 Consumer-side guards when the implementation lives upstream

Once a consumer project depends on `mgf.common.observability`, it
does not own the formatter, the redactor, the handler wiring, or
the OTel setup. Upstream (here, `mgf-common`) owns the fine-grained
benchmarks and contract tests. A consumer might then be tempted to
drop its own logging tests entirely — the upstream covers it.

**Don't.** Keep a thin *consumer-side guard* for every rule the
upstream enforces. Three reasons:

1. **Shim regressions.** A consumer's `src/<app>/observability/`
   package is typically a re-export shim over `mgf.common.observability`.
   A future PR that reaches for a sibling-module import reintroduces
   coupling the shim was supposed to prevent. An import-linter
   contract catches that statically; a consumer-side test catches
   it dynamically.
2. **Consumer-specific extensions.** When a project grows its own
   formatter, redactor, or handler (e.g., scrub a domain-specific
   secret name pattern), the upstream benchmarks no longer cover
   the full path. A consumer-side test is the only place that sees
   the composed shape.
3. **Pin regressions.** A future `mgf-common` release that
   accidentally violates its own rule (imagine LG-12 silently
   regressing in a refactor) lights up on the consumer's CI
   before it reaches a production caller.

A single `tests/unit/test_logging_perf.py` test — one test,
one assertion, runs in ~0.6 s — catches all three regression
classes above. Every consumer SHOULD ship one.

---

## 12. Anti-patterns (forbidden)

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| `LOG.error(f"failed: {e}")` | Loses traceback | `LOG.exception("failed")` or `LOG.error("failed", exc_info=True)` |
| `print("starting...")` | Bypasses sinks, levels, redaction | `LOG.info("starting...")` |
| `LOG.info(f"x={x}")` | Eager format; no structured field | `LOG.info("x", extra={"x": x})` |
| `logging.getLogger("x")` at module top with hardcoded short name | Flat name, can't be category-filtered | `logging.getLogger(__name__)` |
| Multiple `setup_logging` calls in one process from different modules | Handler duplication | One call per entry point only |
| Log a password / token in any field | Leaks secrets to file/aggregator | Redactor handles common cases; per-call MUST use `extra={"redacted_field": "..."}` if explicit |
| Synchronous network call in a custom handler | Blocks the calling thread | Use a `QueueHandler` + `QueueListener` for any remote sink that may block |
| INFO-level "successfully did X" after every method | Log noise | INFO for user-visible lifecycle events only; DEBUG for detail |
| `LOG.info("...", extra={"name": x, "msg": y, "args": z, "level": q})` | `KeyError("Attempt to overwrite 'name' in LogRecord")` at runtime — these are reserved `LogRecord` attributes (see §12.1 below) | Use distinct field names: `subject` / `kind` / `payload_args` / `severity` |

---

### 12.1 Reserved `LogRecord` attribute names

Python's stdlib `logging.LogRecord` reserves a fixed set of
attribute names. Passing any of them via `extra={...}` raises
`KeyError("Attempt to overwrite '<name>' in LogRecord")` at the
`LOG.info(...)` call site — runtime, not lint-time, so the bug
ships unnoticed until the offending log line fires.

The reserved set is the canonical `_RESERVED` frozenset defined once
in the §5.3 `json_formatter.py` drop-in (and re-used by the §5.4
text formatter via import) — that single definition is the source of
truth; it is **not** re-listed here, to avoid the three-copy drift
DOC-18 warns against. It mirrors the attributes Python's stdlib
`logging.LogRecord.__init__` sets (Python 3.12 adds `taskName`).

**MUSTs:**

- `extra={...}` keys MUST NOT collide with the reserved set.
- A common collision: passing the user / resource / domain entity
  by `name` (`extra={"name": vm_name}`) — this is the trap that
  bites first. Use `subject`, `subject_name`, `entity_name`,
  `vm_name`, `widget_name`, etc.
- A custom JsonFormatter that writes structured fields MUST treat
  the reserved set as a deny-list when promoting `__dict__` entries
  to JSON keys (the standard formatter at
  `mgf.common.observability.json_formatter` already does this via
  the `_RESERVED` frozenset).

The standard `JsonFormatter` ALSO promotes `extra={...}` keys to
top-level JSON keys, so a renamed key like `subject` lands as
`"subject": "..."` in the output — operators filter on it the
same way they would have filtered on `name`.

---

## 13. Performance

**MUSTs:**
- Per-record cost MUST stay ≤100 µs at INFO (rule **LG-12**).
  Profile with `pytest --benchmark` if a logger sits in a hot
  loop.
- Remote sinks (HTTP, OTLP) MUST be batched — use `QueueHandler`
  + `QueueListener` to push emission off the calling thread:

```python
import logging
import queue
from logging.handlers import QueueHandler, QueueListener

# In setup_logging:
q: queue.Queue = queue.Queue(-1)
queue_handler = QueueHandler(q)
listener = QueueListener(q, http_handler, otlp_handler, respect_handler_level=True)
listener.start()
# atexit.register(listener.stop)
```

This way the call site only enqueues; emission happens on a background thread.

**Beyond LG-12: per-call perf pins.** A library that ships logging
infrastructure to every consumer SHOULD pin its other hot paths
too — not just per-record cost. `mgf-common` ships four
additional pins under `pytest -m perf` (the nightly sweep), each
with a generous budget over the measured value to catch
order-of-magnitude regressions and silent creep without
false-positiving on CI noise:

| Surface | Budget | Measured | What a regression means |
|---|---|---|---|
| `current_context()` | 500 ns | ~170 ns | The contextvar lookup chained from every log call's identity resolution slowed down |
| `Redactor.redact_string` | 5 µs | ~1.5 µs | A new BUILTIN regex with bad worst-case backtracking, or `_scrub_string` doing per-call allocations |
| `JsonFormatter.format` | 30 µs | ~10 µs | Per-record allocation pressure or a new OTel-context lookup landed |
| `bootstrap()` cold | 5 ms | ~1.2 ms | Wiring sequence regression — usually a heavy module import or an extra sync I/O |

Reference implementation:
[`tests/unit/test_perf_pins.py`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/tests/unit/test_perf_pins.py).
A consumer that ships its own observability stack SHOULD set up
the same shape of pins for its hot paths.

**Beyond per-call: capacity gates.** Per-call budgets protect
against regressions in *one* record's cost. They do NOT protect
against sustained-throughput regressions (a 10 % allocation-pressure
increase that slides past LG-12 but shows up as queue backlog under
load) or memory-footprint regressions (a 5 MB lookup table added to
a redactor pattern that compounds across every consumer process).
A mature observability stack SHOULD additionally pin:

- **Sustained throughput** — emit N records/sec for K seconds
  against an in-process memory sink; assert the async-handler
  `dropped` counter stays at zero, the final-record-to-sink
  latency stays under a bound, and RSS growth across the window
  is bounded.
- **Memory footprint baseline** — spawn a fresh subprocess,
  `bootstrap()`, emit ~10 k records, fire a few crash reports,
  assert peak RSS is under a budget. Catches the slow-leak
  failure mode that hits OOM days after deployment.

See mgf-common's `PERFORMANCE_AND_CAPACITY.md` for the audit
shape that produces these.

---

## 14. Cross-references

- Crash reporter integration (ERROR / unhandled records): [`ERROR_HANDLING.md`](ERROR_HANDLING.md) §6.
- Performance budgets per record: [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md) §6.
- The `extra={...}` field pattern is the same one pydantic-settings uses for env-var aliases — naming consistency matters: [`CONFIGURATION.md`](CONFIGURATION.md) §3.
- Sensitive-field redaction policy + per-project additions: [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) §9.
- Audit log records (rule SC-17): [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) §9.3.
- Test patterns for log-record content + perf benchmarks: [`TESTING.md`](TESTING.md) §8 + §10.3.

---

## 15. Log volume control (rule LG-13)

A logger that fires ten thousand records per second silently turns a
log aggregator into a denial-of-service victim, and the signal you
needed disappears in noise. High-frequency sites MUST limit their
output.

### 15.1 The four shapes

| Shape | When to use | Cost |
|---|---|---|
| **Rate-limit per logger** | A loop that may fire often but where each record is individually meaningful | Bounded ops/s |
| **Sample 1-in-N** | Per-record telemetry where the aggregate matters but individuals don't | 1/N records |
| **Dedup window** | Repeated identical messages from the same site | Each unique message ≤1× per window |
| **Aggregate** | Counters / gauges that should be a metric, not a log | One record per flush interval |

### 15.2 Rate-limit per logger

The drop-in pattern:

```python
# src/<app>/observability/rate_limit.py
import logging
import time
from collections import defaultdict
from threading import Lock

class RateLimitFilter(logging.Filter):
    """Allow at most `max_per_second` records per (logger, level) pair.

    Excess records are dropped. A WARNING fires once per window
    indicating how many records were dropped.

    Use this for high-frequency sites where each record is meaningful
    but the aggregate volume may overwhelm sinks.
    """

    def __init__(self, max_per_second: float = 100.0) -> None:
        super().__init__()
        self._max = max_per_second
        self._counts: dict[tuple[str, int], int] = defaultdict(int)
        self._dropped: dict[tuple[str, int], int] = defaultdict(int)
        self._window_start = time.monotonic()
        self._lock = Lock()

    def filter(self, record: logging.LogRecord) -> bool:
        key = (record.name, record.levelno)
        with self._lock:
            now = time.monotonic()
            if now - self._window_start >= 1.0:
                self._flush(now)
            if self._counts[key] >= self._max:
                self._dropped[key] += 1
                return False
            self._counts[key] += 1
            return True

    def _flush(self, now: float) -> None:
        for key, dropped in self._dropped.items():
            if dropped:
                logger_name, _ = key
                logging.getLogger(logger_name).warning(
                    "rate-limited: dropped %d records in the last second",
                    dropped,
                )
        self._counts.clear()
        self._dropped.clear()
        self._window_start = now
```

Wire from `setup_logging`:

```python
rate_limit = RateLimitFilter(max_per_second=200.0)
for handler in handlers:
    handler.addFilter(rate_limit)
```

### 15.3 Sample 1-in-N

For per-event telemetry where individuals don't matter but the
distribution does:

```python
import random

class SampleFilter(logging.Filter):
    """Keep 1 in N records per logger; the rest dropped silently."""

    def __init__(self, *, sample_rate: int = 100) -> None:
        super().__init__()
        self._sample_rate = sample_rate

    def filter(self, record: logging.LogRecord) -> bool:
        return random.randint(1, self._sample_rate) == 1
```

`random.randint` is OK here — the choice is not security-relevant.

### 15.4 Dedup window

Repeated identical messages collapse to one record + a count:

```python
class DedupFilter(logging.Filter):
    """Within a window, suppress identical (logger, level, msg) tuples."""

    def __init__(self, window_s: float = 5.0) -> None:
        super().__init__()
        self._window_s = window_s
        self._last: dict[tuple[str, int, str], float] = {}
        self._lock = Lock()

    def filter(self, record: logging.LogRecord) -> bool:
        key = (record.name, record.levelno, record.getMessage())
        now = time.monotonic()
        with self._lock:
            last_seen = self._last.get(key)
            if last_seen is not None and (now - last_seen) < self._window_s:
                return False
            self._last[key] = now
            return True
```

### 15.5 Aggregate (logs as metrics anti-pattern)

Counters, gauges, and histograms are NOT log records. A loop that
emits one INFO record per processed item to "count throughput" is a
metrics emitter wearing a logging costume. The right shape is to
emit a metric (OpenTelemetry meter, Prometheus counter) and log
only at the operation boundary:

```python
# WRONG — one log per item, ten thousand records per second
for item in stream:
    process(item)
    LOG.info("processed item")

# RIGHT — metric per item, log at the boundary
for item in stream:
    process(item)
    items_processed.add(1)

LOG.info(
    "stream processed",
    extra={"operation": "stream.process", "count": items_processed.get()},
)
```

### 15.6 Choosing a shape

| Symptom | Use |
|---|---|
| "Some records here are critical; we just can't have all of them" | Rate-limit |
| "We need to know the typical record but not every one" | Sample |
| "The same message keeps repeating" | Dedup |
| "We're tracking a number, not an event" | Metric (not a log) |

A site that fires >100 records/s in steady state at INFO is almost
certainly using the wrong shape. Re-classify before adding more
handlers.

---

## 16. Async-context propagation (rule LG-14)

A long-running operation typically spans many `await` points across
many tasks. Correlation context (request ID, user ID, lab name,
trace ID) MUST follow the operation across those points or a single
user action's logs scatter across unrelated entries.

### 16.1 Why contextvars

Python's `contextvars.ContextVar` was introduced in 3.7 specifically
to solve this for asyncio. asyncio creates a **copy** of the context
for each task, so a `ContextVar` set inside a task does not leak to
its parent — but it DOES propagate to the task's awaited callees.

Threads have their own context; thread-local storage works for
synchronous code but not for async (a single thread runs many
tasks). `contextvars` works for both.

### 16.2 Drop-in template

```python
# src/<app>/observability/correlation.py
"""Correlation context that propagates across thread + async boundaries.

Register a ContextVar here, set it at the entry point that owns the
correlation, and the JSON formatter will promote it to a top-level
field on every log record fired inside the context.
"""

from __future__ import annotations

import contextvars
from typing import Any

# Each ContextVar registered here gets promoted to a log field with
# the same name. Keep the set small — every record pays the lookup
# cost.
_REGISTERED: dict[str, contextvars.ContextVar[Any]] = {}

def register(name: str, default: Any = None) -> contextvars.ContextVar[Any]:
    """Register a correlation field. Returns the ContextVar to set / read."""
    if name in _REGISTERED:
        return _REGISTERED[name]
    cv: contextvars.ContextVar[Any] = contextvars.ContextVar(name, default=default)
    _REGISTERED[name] = cv
    return cv

def snapshot() -> dict[str, Any]:
    """Read all registered correlation values for the current context."""
    out: dict[str, Any] = {}
    for name, cv in _REGISTERED.items():
        try:
            value = cv.get()
        except LookupError:
            continue
        if value is not None:
            out[name] = value
    return out
```

### 16.3 Wiring into the formatter

```python
# In <app>/observability/json_formatter.py — extend _add_otel_context:

def _add_correlation(payload: dict[str, Any]) -> None:
    """Promote registered correlation contextvars to log fields."""
    from <app>.observability.correlation import snapshot
    payload.update(snapshot())
```

### 16.4 Setting and clearing

At the boundary that owns the correlation (CLI command, HTTP
handler, scheduler tick, GUI worker), set + reset via a token:

```python
from <app>.observability.correlation import register

request_id = register("request_id")
user_id = register("user_id")

async def handle_request(req_id: str, user: str) -> None:
    tokens = [request_id.set(req_id), user_id.set(user)]
    try:
        await do_work()  # every log record inside carries request_id + user_id
    finally:
        for token in reversed(tokens):
            request_id.reset(token) if token.var is request_id else user_id.reset(token)
```

The `try / finally` pattern is critical — without `reset`, the
context value persists into whatever ran after the boundary.
`contextvars.copy_context().run(...)` is an alternative for code
that prefers a snapshot/restore shape.

### 16.5 Cross-thread propagation

When dispatching to a thread (`run_in_executor`,
`asyncio.to_thread`, plain `Thread`), the asyncio runtime
**already** propagates the context to the thread for its lifetime
of the dispatch — `to_thread` and `run_in_executor` both copy
context. For raw `threading.Thread`, copy explicitly:

```python
import contextvars
import threading

def _run_in_context(ctx: contextvars.Context, fn, *args, **kwargs):
    return ctx.run(fn, *args, **kwargs)

ctx = contextvars.copy_context()
thread = threading.Thread(target=_run_in_context, args=(ctx, do_work))
thread.start()
```

Without the copy, the new thread starts with an empty context and
the correlation fields disappear from every log record it emits.

### 16.6 Cost

The lookup is `O(registered fields)` per log record. Keep the
registered set small — five fields is fine; fifty is not. Use
`extra={...}` for one-off fields that don't need cross-await
propagation.

---

*End of `LOGGING.md`.*
