# Performance & Capacity

> **Status:** v0.1 — first pass. Last updated 2026-05-14.
> A leaf-library performance audit: walked the per-call hot paths
> (logging, redaction, context lookup, bootstrap), the per-process
> startup cost (`import mgf.common`, settings load, OTel wiring),
> and the per-operation maintenance paths (crash reporter, atomic
> write, codemod, vault KDF). Every finding is pinned to `file:line`
> and verified with a local measurement.
>
> **Scope:** `mgf-common`'s own performance footprint — the cost
> every consumer pays per log record, per subprocess line, per
> bootstrap, per import. This is NOT a capacity audit of any
> deployed consumer (those live in the consumer's own tracker —
> e.g. PlasmaMapper's PC-* items cover ingest throughput, SSE
> fan-out, map render, DB growth). It is also NOT an audit of
> sibling packages (mgf-http, mgf-fastapi, mgf-sqlalchemy,
> mgf-django, mgf-alembic) — each gets its own tracker in its own
> repo. The mgf-common surface is: every call site that runs on
> every consumer's hot path, plus the few one-shot operations
> whose latency budget consumers can observe.
>
> **Audience:** Bassam-tomorrow, AI coding agents, future hires
> doing performance work on the federation core.
> **Cross-references:** [`RELIABILITY_AND_ACCURACY.md`](RELIABILITY_AND_ACCURACY.md)
> (sibling tracker — RA-11 already closed the per-call perf pins
> the rest of this tracker builds on; see §3),
> [`SECURITY_HARDENING.md`](SECURITY_HARDENING.md)
> (sibling tracker — SH-03 added the redactor depth cap + cycle
> guard that bounds redactor worst-case cost),
> [`docs/ops/benchmarks.md`](../ops/benchmarks.md) (the
> release-over-release bench-numbers log; PC-06 wires its
> regression detection into CI),
> `mgf-fed standards LOGGING` §13 (rule LG-12 — the per-record
> logging budget).

This document is the **living priority tracker** for performance +
capacity work in `mgf-common`. The shape mirrors
[`RELIABILITY_AND_ACCURACY.md`](RELIABILITY_AND_ACCURACY.md) and
[`SECURITY_HARDENING.md`](SECURITY_HARDENING.md): priority table at
the top, work-unit detail below, status flipped as items close. The
discipline that makes it work is keeping the items **narrow** (each
one a single concrete commit) and **falsifiable** (each one has a
written "concrete output" that can be checked off).

---

## §1 You are here

`mgf-common` is a **leaf library** — no network listener, no
database, no ingest pipeline, no fleet UI. That means the typical
"performance and capacity" concerns of a deployed app (request
throughput, queue depth, connection pool headroom, render cliffs)
have no analogue here: that surface lives in the deployed
consumers (PlasmaMapper, vm-vmanager-app, magogi-foundation) and (for
async I/O) in the `mgf-http` / `mgf-fastapi` siblings.

What `mgf-common` *does* have is three concentrated cost surfaces.
The audit found one load-bearing fact and three patterns:

**The headline:** the per-call hot paths are **already well
gated**. LG-12 + RA-11 closed five per-call budgets — per-record
logging, `current_context()`, `Redactor.redact_string`,
`JsonFormatter.format`, and `bootstrap()` cold-start. The
`mgf-fed bench` subcommand exists; `docs/ops/benchmarks.md`
records release-over-release numbers. Per-call regression catches
are in place. This tracker is **not** a re-audit of those — see §3
for the closed items it leans on.

What this tracker exposes is the **second-order** stuff: the
costs that compound across patterns the per-call gates can't see.

**1 — `import mgf.common` is ~100 ms on a cold interpreter.**
Measured locally with `python -X importtime` (median of 3 runs):
`mgf.common` loads in 100–103 ms, of which
`mgf.common._lifecycle` is **101 ms** cumulative (1.7 ms own —
the rest is what it imports), and `mgf.common.settings` is
**90 ms** cumulative (6.3 ms own — the rest is pydantic +
pydantic-settings). The single load-bearing import site:
**`src/mgf/common/_lifecycle.py:60`** does
`from mgf.common.settings import MgfSettings` at module top,
which pulls the whole `pydantic` + `pydantic_settings`
(provider chain ~22 ms — env / aws / azure / cli / dotenv / gcp
/ json / toml / pyproject / secrets / yaml are all imported
eagerly even though most consumers use one or two) +
`mgf.common.config._settings` (27 ms) chain. The wall-clock
falls out: `mgf-fed explain --help` is **148 ms wall**
locally (148 ms = ~100 ms import + ~48 ms argparse + Python
startup), and every CLI tool in every consumer that imports
`mgf.common` at module top pays the same. For long-running
daemons it is one-shot and ignorable; for the expanding set of
one-shot CLI utilities (the `mgf-common` subcommands,
consumer-side `vmanager doctor`, codemod helpers, git hooks
that wrap a CLI) it is **user-visible startup latency** — this
is **PC-01**, the single P0.

**2 — There is no capacity gate.** LG-12 measures one record;
RA-11 measures one redaction pass; the bench measures one call.
**Nothing** asserts a sustained-throughput floor (e.g. 50 k
records/sec sustained for 10 s without backlogging the async
queue, OOM, or GC pause), and **nothing** asserts the scaling
shape on large inputs (a 10 k-key extras dict, a deeply-nested
crash-report cause chain). Locally, `Redactor.redact({1000 keys
each Bearer-bearing})` is 1.35 ms — linear in dict size, fine —
but the bound is *measured ad hoc*, not pinned. A future regression
that flips the scaling to O(n²) would slide through every
per-call gate.

**3 — A few maintenance paths have measured latency but no
formal budget.** `report_now()` (crash reporter end-to-end:
build → redact → atomic-write → ship) measures **median 3.1 ms,
p99 4.8 ms** locally — but EH-05 only pins "never raises"; nothing
pins "fast enough to land before signal handlers cascade".
`atomic_write(4 KB)` is **3.2 ms** locally (315 ops/sec,
fsync-bound — fine outside a hot loop), but the `mgf-common
catch-up --apply` codemod and the vault rotation path both
fan-out atomic writes; an O(N×3 ms) ceiling there is
non-obvious. The vault KDF cost (PBKDF2 ~600 k iterations per
SC-05) is deliberately slow — but a regression to 60 k would
land silently because nothing gates it.

**Counts (2026-05-14, first pass):**

| Priority | Count | Effort | Status |
|---|---|---|---|
| P0 — v1.0 surface-lock blockers | 1 | ~M | ✅ closed (PC-01) |
| P1 — should land before/with v1.0 | 4 | ~1.5 session-weeks (4xS–M) | ✅ all closed (PC-02..PC-05) |
| P2 — defer to post-v1.0 | 5 | — | 3 closed (PC-07 / PC-09 / PC-10, 2026-05-17); 2 deferred to v1.1+ (PC-06 + PC-08) |

**Effort calibration:** S ≤ 1 day, M ≈ 2–4 days, L ≈ 1
session-week. Anchored against the RA tracker's actual closure
pace (S items closed in a few hours; M items in 1–2 sessions).

**Key things to do first:** **PC-01** (import time) is the
single P0 — it is the only item whose cost is user-visible
*outside* the library, paid by every CLI invocation on every
consumer. **PC-05** (memory baseline) is the cheapest P1 and
should run early so the rest of the perf work has a regression
floor to land against. The other three P1 items
(PC-02 / PC-03 / PC-04) can land in any order; none blocks the
others.

The honest framing: **this is a thin tracker for a healthy
library.** mgf-common has good perf hygiene (LG-12, RA-11, bench
subcommand, benchmarks.md). The 10 items here close residual
gaps and add capacity gates that don't exist yet — they don't
fix a broken state, they raise the bar on a working one.

---

## §2 Priority table

### P0 — v1.0 surface-lock blockers (1 item)

| ID | Theme | One-liner | Effort | Status |
|---|---|---|---|---|
| [**PC-01**](#pc-01-import-mgf-common-is-100-ms-on-cold-start) | Cold-start | `import mgf.common` is ~100 ms on a cold interpreter — `mgf-fed explain --help` is 148 ms wall. Driver: `_lifecycle.py:60` eager-imports `MgfSettings`, which transitively loads the full pydantic + pydantic-settings provider chain. | M | ✅ closed |

### P1 — should land before/with v1.0 (4 items)

| ID | Theme | One-liner | Effort | Status |
|---|---|---|---|---|
| [**PC-02**](#pc-02-no-sustained-throughput-gate-for-logging) | Capacity | LG-12 measures *one* record; nothing asserts a sustained floor (records/sec for 10 s without async-queue backlog / OOM / GC pause) | M | ✅ closed |
| [**PC-03**](#pc-03-redactor-large-payload-scaling-is-unpinned) | Capacity / scaling | `Redactor.redact({1000-key dict})` is 1.35 ms locally — linear in dict size, but measured ad hoc; an O(n²) regression slides through every per-call gate | S | ✅ closed |
| [**PC-04**](#pc-04-crash-reporter-end-to-end-latency-budget) | Latency | `report_now()` measures median 3.1 ms / p99 4.8 ms locally; EH-05 pins "never raises", nothing pins "fast" — a regression in build/redact/write/ship would land silently | S | ✅ closed |
| [**PC-05**](#pc-05-memory-footprint-baseline) | Memory | 37 MB RSS after `bootstrap()` — bare Python+imports is ~12 MB, `bootstrap()` adds ~25 MB on top (pydantic-settings, handlers, redactor patterns, drain thread). No baseline test, no regression detection — quantify and pin. | S | ✅ closed |

### P2 — 3 closed for v1.0, 2 deferred to v1.1+ (5 items)

| ID | One-liner | Effort | Status |
|---|---|---|---|
| PC-06 | `docs/ops/benchmarks.md` is updated manually each release — no CI diff against a checked-in baseline JSON; a 2× drift release-over-release lands silently | S | ⏳ v1.1+ — CI infrastructure work |
| PC-07 | `operation_span()` is 3.8 µs with OTel configured; the *no-op* cost (OTel disabled) is unmeasured — disambiguate so consumers know what they pay when they don't use spans | S | ✅ closed 2026-05-17 |
| PC-08 | `mgf-fed migrate` / `catch-up --apply` codemod runtime on a large consumer (1 k+ Python files) is never measured; could be seconds or minutes — a developer-experience signal that should at least have a number | M | ⏳ v1.1+ — needs synthetic 1k-file tree |
| PC-09 | `EncryptedFileVault` PBKDF2 KDF is intentionally slow (≥600 k iters per SC-05) — but nothing gates the iteration count; a silent downgrade to e.g. 60 k would weaken the security posture without any test catching it | S | ✅ closed 2026-05-17 |
| PC-10 | `AsyncJsonHandler.emit` (queue put) is measured by `mgf-fed bench` (~6 µs; the async-vs-sync gap is 2.6×) but has no `test_perf_pins.py` gate; formalise it so the gap (the whole reason async logging exists) is regression-protected | S | ✅ closed 2026-05-17 |

---

## §3 Invariants we hold

These are load-bearing performance properties already in place —
verified during the audit. Performance work either preserves them or
escalates. Several were closed by the sibling reliability tracker;
the rest the audit confirmed sound.

**Closed in [`RELIABILITY_AND_ACCURACY.md`](RELIABILITY_AND_ACCURACY.md) — do not re-open:**

- **LG-12** — per-record logging budget (≤ 100 µs at INFO under the
  rotating-file sink). Gated by
  [`tests/unit/test_logging_perf.py`](../../tests/unit/test_logging_perf.py).
- **RA-11** — four per-call perf pins (under
  [`tests/unit/test_perf_pins.py`](../../tests/unit/test_perf_pins.py),
  pytest marker `perf`, nightly sweep):
  - `current_context()` — budget 500 ns (measured ~170 ns)
  - `Redactor.redact_string` — budget 5 µs (measured ~1.5 µs)
  - `JsonFormatter.format` — budget 30 µs (measured ~10 µs)
  - `bootstrap()` cold — budget 5 ms (measured ~1.2 ms)
- **SH-03** — Redactor `_walk` is depth-bounded
  (`_MAX_REDACT_DEPTH = 64`) and cycle-guarded (`id()` set with
  `try / finally`). Bounds redactor worst-case cost on
  adversarial-shape inputs.

**Confirmed sound by the audit:**

- **Redactor BUILTIN value patterns are character-class-bounded**
  — every pattern in `_BUILTIN_VALUE_PATTERNS` (`Bearer\s+[A-Za-z0-9._\-+/=]{8,}`,
  `eyJ[...]{8,}\.[...]{8,}\.[...]{8,}`, etc.) has no nested
  quantifier, no backreference, no lookbehind. ReDoS via
  catastrophic backtracking is not on the menu. The PEM pattern
  uses `.*?` (lazy) but is bounded by the literal
  `-----END...PRIVATE KEY-----` close, so the worst case is a
  single linear scan
  ([`redaction.py:90-104`](../../src/mgf/common/observability/redaction.py#L90)).
- **Redactor scales linearly with payload size** — measured locally:
  1000-key dict (each value Bearer-bearing) = 1.35 ms (≈1.35 µs/key);
  60 KB string = 2 µs; 50-level deep nested dict = 47 µs.
  Linear in keys, linear in string length, linear in depth
  (capped at 64 by SH-03). No quadratic blow-up anywhere.
  PC-03 pins this so a regression to O(n²) would fail loud.
- **`mgf-fed bench` subcommand exists** — seven microbench rows
  (`current_context`, `bootstrap` cold + warm, `operation_span`,
  sync + async log call, redaction-pass-on-10-key-dict) with
  `--quick` and `--json` flags. Source:
  [`src/mgf/common/cli/_bench.py`](../../src/mgf/common/cli/_bench.py);
  release-over-release tracking in
  [`docs/ops/benchmarks.md`](../ops/benchmarks.md).
- **`AsyncJsonHandler` keeps the formatter + write off-CPU** — the
  calling-thread cost is the `queue.put_nowait` (measured
  **~6 µs** — `mgf-fed bench --quick` 2026-05-14 reports
  5 834 ns/call); JSON encode + redaction + sink-write run on
  the drain thread. The 2.6× speedup over the sync path
  (15.4 µs → 5.8 µs) is what makes the async surface worth
  shipping. Source:
  [`src/mgf/common/observability/async_handler.py`](../../src/mgf/common/observability/async_handler.py).
- **`atomic_write` is fsync-bounded, not throughput-bounded** —
  measured locally: 4 KB = 3.2 ms (315 ops/sec), 1 MB = 3.7 ms
  (268 MB/sec). The fsync is the floor; for one-shot operations
  (config save, vault write, crash report) this is fine and
  not on a hot path.
- **The crash reporter's redact-then-write-then-ship sequence is
  bounded** — measured locally: median 3.1 ms, p99 4.8 ms over
  100 calls. The fail-safe path (SH-01) replaces the payload
  with a minimal dict if redaction raises, so the failure mode
  is "cheaper", not "unbounded". PC-04 formalises the budget.
- **Memory footprint is bounded and steady-state** — measured
  locally: bare Python interpreter + the `mgf.common` import
  tree is **~12 MB** (pydantic + pydantic-settings + yaml +
  stdlib); `bootstrap()` adds **~25 MB** (pydantic-settings
  provider instantiation, root-logger handlers, redactor
  patterns compiled to `re.Pattern` objects, async-handler
  drain-thread setup), for a total of **~37 MB RSS**. Emitting
  10 000 log records on top adds **0 MB** of measurable RSS
  growth (verified locally — the per-record allocations are
  reclaimed). PC-05 pins the post-bootstrap baseline against
  silent regressions.
- **Per-call hot-path allocations are bounded** — `JsonFormatter.format`
  allocates one dict per record (the JSON output shape) plus a
  short JSON string from `json.dumps`; no surprise per-record
  I/O, no per-record subprocess, no per-record HTTP. Confirmed
  by reading
  [`src/mgf/common/observability/json_formatter.py`](../../src/mgf/common/observability/json_formatter.py)
  end-to-end. The redactor walks the payload dict in place —
  no copy.

---

## §4 Work units

> One section per row. Each entry has: **Why** (with `file:line`
> evidence and a local measurement where applicable), **Concrete
> output**, **Dependencies**, **Status**.

### PC-01 `import mgf.common` is 100 ms on cold start

**Why:** `python -X importtime -c "import mgf.common"` (median
of 3 runs on the maintainer's laptop, Linux 6.17, Python 3.12.3,
v0.33.0 commit):

```
mgf.common                                  ~100 ms  (cumulative)
├── mgf.common._errors                          7 ms
├── mgf.common._identity                      0.2 ms
├── mgf.common.exceptions (._base, ._well_known) 0.7 ms
├── mgf.common._lifecycle                    101 ms  ← driver
│   └── mgf.common.settings                  ~90 ms
│       ├── pydantic (BaseModel/Field/...)   (transitive, heavy)
│       └── mgf.common.config._settings        27 ms
│           └── pydantic_settings              22 ms
│               ├── pydantic_settings.sources.utils  3.3 ms
│               ├── pydantic_settings.sources.providers (cli/aws/azure/...)  4.6 ms
│               └── pydantic_settings.main      0.8 ms
└── mgf.common._test_helpers                 0.13 ms
```

The single load-bearing import site is
[`src/mgf/common/_lifecycle.py:60`](../../src/mgf/common/_lifecycle.py#L60):

```python
from mgf.common.settings import MgfSettings
```

That one line at `_lifecycle.py` module-top is what makes
`import mgf.common` (which re-exports
`bootstrap` / `current_context` / `AppContext` from `_lifecycle`)
pull the entire pydantic + pydantic-settings stack. Pydantic-settings
eagerly imports its full provider chain (env / aws / azure / cli
/ dotenv / gcp / json / toml / pyproject / secrets / yaml — see
[the breakdown above]) even though a typical consumer uses one
or two providers. Wall-clock fallout: `mgf-fed explain --help`
is **148 ms** locally (≈100 ms import + ≈48 ms argparse + Python
startup). For a long-running daemon this is one-shot and
ignorable; for the expanding set of one-shot CLI utilities
(every `mgf-common` subcommand, consumer-side `vmanager doctor`,
codemod helpers, any git hook that wraps a CLI) **every
invocation pays the same 100 ms before any useful work starts**.
This is a federation-wide tax — multiply by every consumer's
every CLI hook.

**Concrete output:**
- An `import mgf.common` budget test (`tests/unit/test_import_perf.py`,
  `pytest -m perf` marker like the other perf pins) that spawns
  a fresh subprocess
  (`subprocess.run([sys.executable, "-c", "import mgf.common"], ...)`)
  and asserts wall-clock under a generous budget — e.g. 60 ms
  median over 5 runs (current 100 ms × 0.6, room for CI noise).
- The fix in
  [`_lifecycle.py`](../../src/mgf/common/_lifecycle.py): the
  module already uses `from __future__ import annotations`
  (line 47), so all type hints are strings at runtime. Move the
  `from mgf.common.settings import MgfSettings` line under
  `if TYPE_CHECKING:`, and inside `bootstrap()` (the one function
  that actually needs to construct or default an `MgfSettings`
  instance) do a deferred `from mgf.common.settings import MgfSettings`.
  `current_context()` returns an existing `AppContext` instance
  whose `settings` field is already populated — it never
  constructs `MgfSettings` and so doesn't need the import at
  function-call time either.
- Verify the public re-exports in
  [`mgf.common.__init__.py`](../../src/mgf/common/__init__.py)
  still resolve (`bootstrap`, `current_context`, `AppContext`
  are imported from `_lifecycle`; none of them need
  `MgfSettings` at module-load time).
- Confirm: re-running `python -X importtime -c "import
  mgf.common"` shows the `mgf.common.settings` cumulative cost
  drop to <10 ms (its eager-import path gone). Re-running
  `time mgf-common explain --help` shows wall-clock under
  ~80 ms (the 100 ms import gone; Python startup + argparse
  remain).

**Dependencies:** none. This is the single P0 because it is the
only item whose cost is user-visible *outside* the library —
every other item in this tracker is paid inside a process that's
already running.

**Status:** ⏳ open. Effort: M (the move itself is a few lines,
but `mgf.common.settings` is a public re-export surface — every
caller that does `from mgf.common.settings import MgfSettings`
must still work. The fix changes the *implementation* of
`_lifecycle.py`, not the public API; the work is in proving
nothing else in the lifecycle chain depends on the eager import).

---

### PC-02 No sustained-throughput gate for logging

**Why:** LG-12
([`tests/unit/test_logging_perf.py::test_log_per_record_under_budget`](../../tests/unit/test_logging_perf.py))
asserts that **one** record costs less than 100 µs. RA-11
([`tests/unit/test_perf_pins.py`](../../tests/unit/test_perf_pins.py))
asserts that **one** formatter call, **one** redaction, **one**
context lookup each stay under budget. **Nothing** asserts that
e.g. 50 000 records/sec sustained for 10 seconds completes
without (a) the `AsyncJsonHandler` queue filling up and starting
to drop or block — the queue is bounded
(`maxsize=10_000` default, `on_overflow="drop"` default at
[`async_handler.py:127-128`](../../src/mgf/common/observability/async_handler.py#L127));
the bound means a producer faster than the drain thread *will*
silently drop, and nothing currently exercises that boundary;
(b) the GC kicking in and stalling the drain thread; (c) RSS
growing through the test window. The per-call budget protects
against a *regression* in the per-call shape; it does not protect
against a regression in the *steady-state*. A change that adds
10 % allocation pressure per record, or 10 % drain-thread CPU,
would slide past LG-12 and show up as a `dropped` counter
climbing in production — silent data loss the per-call test
cannot catch.

**Concrete output:**
- A new perf test (`tests/unit/test_sustained_throughput.py`,
  `pytest -m perf` marker) that:
  - Builds an `AsyncJsonHandler` with a `MemoryHandler` sink (no
    real I/O; the gate measures the in-process path).
  - Emits 50 000 records over ~5 seconds at a steady rate
    (`time.perf_counter()` paced loop, not a `for i in range`
    burst — bursts mask backlog).
  - Asserts the handler's `dropped` counter stays at zero AND
    the final-record-to-sink latency (measured via
    `MemoryHandler.flush()` + the last record's timestamp) stays
    under M ms (e.g. 50 ms) AND RSS growth across the window is
    bounded (e.g. < 5 MB).
- Numbers picked from the *first* CI green run, then × 1.5 for
  noise (same generous-budget approach as RA-11).
- A short note in [`docs/ops/benchmarks.md`](../ops/benchmarks.md)
  on "what sustained means" — this is the capacity floor every
  consumer can lean on, and the explicit point that **the
  10 000-record default bound is sufficient for the floor**
  (a documented invariant, not just a measured number).

**Dependencies:** PC-05 (memory baseline) makes the RSS-growth
clause meaningful; do PC-05 first.

**Status:** ✅ closed (2026-05-17). See §5.

---

### PC-03 Redactor large-payload scaling is unpinned

**Why:** Measured locally (`Redactor.default().redact(...)`):
- 1 000-key dict, each value Bearer-bearing: **1.35 ms** (≈1.35 µs/key)
- 50-level deeply-nested dict: **47 µs** (depth-capped at 64 by SH-03)
- 60 KB single string: **2 µs**

All linear scaling — fine. But the bound is **measured ad hoc**,
not pinned. The 1.35 µs/key is what's *currently* achieved;
nothing fails CI if a refactor lands a quadratic shape (e.g. a
nested-pattern-list scan added inside `_walk` that goes O(keys ×
patterns × ...)). For a consumer logging a record with a 10 k-key
extras dict (LOG.info(..., extra={lots_of_telemetry})), 1.35 µs/key
is 13 ms inside one log call — fine; 13.5 µs/key would be 135 ms,
which a per-call gate measuring a 10-key dict wouldn't see.

**Concrete output:**
- A new perf test in
  [`tests/unit/test_perf_pins.py`](../../tests/unit/test_perf_pins.py)
  (same `perf` marker, same nightly sweep):
  - `test_redactor_scales_linearly_in_dict_size` — measures
    `redact({100 keys})` and `redact({1000 keys})`; asserts the
    1000-key cost is < 12× the 100-key cost (linear + slack;
    quadratic would be ~100×).
  - `test_redactor_scales_linearly_in_string_length` — measures
    `redact_string(10KB)` and `redact_string(100KB)`; asserts
    the 100K cost is < 15× the 10K cost.
  - `test_redactor_large_dict_under_budget` — `redact({1000
    keys, mixed-pattern values})` under a generous bound (e.g.
    5 ms, vs. measured 1.35 ms).
- Both ratios + the absolute bound carry the same generous-budget
  philosophy as RA-11 — catch order-of-magnitude regressions and
  silent creep; don't false-positive on CI noise.

**Dependencies:** none.

**Status:** ✅ closed (2026-05-17). See §5.

---

### PC-04 Crash reporter end-to-end latency budget

**Why:** Measured locally over 100 calls to `report_now()`
(build report + redact via Redactor + atomic_write the local
file + dispatch through registered sinks):

```
median   3.08 ms
p99      4.82 ms
max      4.82 ms
```

Within the redaction pass (SH-01) + the atomic-write (CF-04)
budget, this is fine. But **EH-05** pins "`report_now` never
raises"; it pins nothing about latency. A crash reporter is
called at the worst possible time — inside an except branch,
often with the process about to exit, sometimes from a signal
handler context where a 50 ms stall could let a SIGKILL race
the final write. A regression to e.g. 100 ms median (a new
synchronous remote sink, an unbounded loop in the cause-chain
walker, a regex-disaster in a custom redactor pattern) would
land silently and show up as "tail-end crashes don't produce
reports" in production — a debug-hostile failure mode.

**Concrete output:**
- A new perf test in
  [`tests/unit/observability/test_crash_reporter.py`](../../tests/unit/observability/test_crash_reporter.py)
  (`pytest -m perf` marker so it runs nightly, not on every PR):
  - 100 calls to `report_now()` with a representative
    exception (a 3-level cause chain, args of mixed primitive
    + custom-class types after SH-06).
  - Asserts median < 10 ms AND p99 < 25 ms (3× / 5× the
    measured numbers).
  - All sinks set to no-op (or `local` only) so the test measures
    the library cost, not the speed of a SaaS endpoint.
- A short note in
  [`docs/reference/crash_reporting.md`](../reference/crash_reporting.md)
  stating the budget — operators integrating bespoke sinks know
  the budget they MUST stay under to avoid causing tail-end
  drops.

**Dependencies:** none.

**Status:** ✅ closed (2026-05-17). See §5.

---

### PC-05 Memory footprint baseline

**Why:** Measured locally on a fresh interpreter (via
`resource.getrusage(RUSAGE_SELF).ru_maxrss`, with `gc.collect()`
between checkpoints to settle short-lived allocations):

```
after `import mgf.common` only:                  ~12 MB
after `bootstrap("m", "0")`:                     ~37 MB   (Δ ~25 MB)
after 10 000 LOG.info(...) calls:                ~37 MB   (Δ  0 MB)
```

The split is roughly half-and-half: ~12 MB is the Python
interpreter + the `mgf.common` import tree (pydantic +
pydantic-settings + yaml + stdlib — PC-01's cost in memory),
and the other ~25 MB is what `bootstrap()` adds: pydantic-settings
provider instantiation, the root-logger handler chain (stream
sink, rotating-file sink, async drain thread), the redactor's
compiled `re.Pattern` objects, the OTel-disabled (stub) provider,
and the AppContext itself. The per-record allocations are
reclaimed cleanly (10 000 records add zero measurable RSS).

But: **nothing pins this.** A change that adds e.g. a 5 MB
lookup table to a redactor pattern (an embedded yara-style ruleset,
say), or that reads `/proc/<pid>/limits` into a cache, or that
keeps every log record's `extra` dict in a sliding window for
"recent records" diagnostics, would land silently — and compound
across every consumer process. A long-running daemon hits OOM
many days after deployment, with no obvious code change to point
at.

**Concrete output:**
- A new perf test (`tests/unit/test_memory_footprint.py`,
  `pytest -m perf` marker) that:
  - Spawns a fresh subprocess (so the parent test runner's RSS
    doesn't pollute the measurement).
  - In the subprocess: `bootstrap()`, emit 10 000 log records,
    fire 10 `report_now()` calls, then `print(RUSAGE_SELF.ru_maxrss)`.
  - The parent test asserts subprocess-reported peak RSS <
    budget (e.g. 80 MB — 2× the measured 37 MB for CI-hardware
    noise + Python-version drift).
- A new row in
  [`docs/ops/benchmarks.md`](../ops/benchmarks.md): "RSS after
  bootstrap + 10K records + 10 crash reports" with the measured
  number, so the regression-detection story is also documented
  for operators.

**Dependencies:** none — this is the cheapest P1 and should land
first so PC-02's RSS-growth clause has a baseline to lean against.

**Status:** ✅ closed (2026-05-17). See §5.

---

### PC-06 + PC-08 (deferred to v1.1+)

- **PC-06** — [`docs/ops/benchmarks.md`](../ops/benchmarks.md)
  is updated by hand each release; nothing diffs the new numbers
  against the prior baseline. A 2× regression release-over-release
  is invisible until a consumer notices. Wire `mgf-fed bench
  --json` into CI: on `main`, compare against a checked-in
  `docs/ops/baselines/<latest>.json`; fail if any row drifts >
  50 %. On a release commit, update the baseline as part of the
  bump (REL-06 already pins the bump order). **Deferred to v1.1+:**
  CI-infrastructure work; not blocking for v1.0 — the per-call
  perf pins (PC-02..05 + PC-07 + PC-09 + PC-10) catch the regressions
  that matter for v1.0 surface lock.
- **PC-08** — `mgf-fed migrate` and `mgf-fed catch-up
  --apply` codemod runtimes on a large consumer (1 k+ Python
  files) are never measured. Locally on the mgf-common repo's
  own 69-file `src/` tree, dispatching the CLI is 150 ms wall
  (mostly import cost — PC-01); the actual rewrite scaling is
  unknown. For a federation-wide cutover (`v0.x → v0.x+1` codemod
  applied to all consumers), this is developer-experience
  latency. Generate a synthetic 1 000-file Python tree, run
  `migrate --dry-run` against it, record the wall-clock; budget
  on the first measurement. **Deferred to v1.1+:** synthetic-tree
  generation + threshold-on-first-measurement is throwaway
  scaffolding for a one-shot data point. Not a v1.0 blocker —
  PC-01 already pins the per-invocation import cost.

---

## §5 Closed items

- **PC-01** — `import mgf.common` cold-start cost: **100 ms →
  18 ms** (median of 3 runs, 2026-05-15, post-fix). The
  surgical fix the audit identified worked as designed.
  Changes:
  - `src/mgf/common/_lifecycle.py`: moved
    `from mgf.common.settings import MgfSettings` (line 60)
    under `if TYPE_CHECKING:`; added a new
    `_default_mgf_settings()` factory at module scope that
    does the lazy import for `AppContext.settings`'s
    `default_factory`; added a lazy in-function import inside
    `bootstrap()` for the `MgfSettings(preset="auto")`
    fallback path.
  - `src/mgf/common/_test_helpers.py`: same shape — the
    second eager-import path (line 52) was the reason the
    first attempt didn't fully drop the cost.
    `_test_helpers` is re-exported from `mgf.common.__init__`,
    so its eager `from mgf.common.settings import MgfSettings`
    was pulling the chain too. Moved under `TYPE_CHECKING` +
    lazy import inside the function that calls
    `MgfSettings.testing()`.
  - `tests/unit/test_import_perf.py`: new perf-marker test
    spawning a fresh subprocess and asserting median of 5
    runs < 60 ms (3× the measured 18 ms). The error message
    points back at `_lifecycle.py` + `_test_helpers.py` so a
    future regressor knows exactly where to look.
  - 1411 tests pass; mypy --strict clean.

  **Wall-clock impact:** `mgf-fed explain --help` is still
  ~148 ms — the saved 80 ms applies to `import mgf.common`
  but the CLI's own subcommand-chain pulls
  `mgf.common.observability` separately (10 ms) + has Python
  startup + argparse overhead. PC-01's core target (the cost
  paid by EVERY consumer importing mgf.common) is achieved.
  Further CLI-side wall-clock reduction is a separate scope
  if needed. (commit pending, 2026-05-15).

- **PC-05** — Memory footprint baseline. New
  `tests/unit/test_memory_footprint.py` (perf marker) spawns a
  fresh subprocess that runs the canonical consumer shape
  (`bootstrap` + 10 000 log records + 10 `report_now()` calls)
  and prints `RUSAGE_SELF.ru_maxrss`. Budget 80 MB (~2.1x the
  local ~38 MB measurement). Catches the new-compile-time-
  resident-object regression class — embedded yara-style
  redactor rule tables, settings caches, sliding-window
  "recent records" buffers — at the moment they land.
  (2026-05-17, commit pending.)

- **PC-03** — Redactor large-payload scaling. Three new tests
  in `tests/unit/test_perf_pins.py` under the existing
  `TestRedactorPerf` class:
  * `test_redactor_scales_linearly_in_dict_size`: 1000-key
    cost < 12x the 100-key cost (linear + 20 % slack).
  * `test_redactor_scales_linearly_in_string_length`: 100K cost
    < 15x the 10K cost.
  * `test_redactor_large_dict_under_budget`: 1000-key absolute
    < 5 ms (measured ~0.7 ms locally).
  An O(n²) refactor inside `_walk` would breach the ratio
  tests; a constant-factor blowup at linear scaling would
  breach the absolute test. (2026-05-17, commit pending.)

- **PC-04** — Crash reporter end-to-end latency pin. New
  `test_report_now_latency_under_budget` in
  `tests/unit/observability/test_crash_reporter.py` (perf
  marker). 100 `report_now()` calls with a representative
  3-level cause chain + mixed primitive/custom-class args
  (the SH-06 shape). Asserts median < 10 ms (measured ~2.8 ms
  locally), p99 < 25 ms (measured ~3.6 ms). The crash reporter
  is on the SIGKILL-race path — a regression to 100 ms median
  means tail-end crashes silently lose their reports.
  (2026-05-17, commit pending.)

- **PC-02** — Sustained-throughput gate for logging. New
  `tests/unit/test_sustained_throughput.py` (perf marker).
  Drives ~50 000 paced records over ~5 s through an
  `AsyncJsonHandler` with an in-memory recording sink (no
  real I/O). Asserts: `dropped_count == 0`; drain-completion
  latency after the last emit < 1 s; peak-RSS growth across
  the window < 5 MB. Catches the silent-data-loss regression
  class (a 10 % drain-thread CPU bump that lets the queue
  saturate under steady load) that LG-12 + RA-11's per-call
  budgets can't see. (2026-05-17, commit pending.)

- **PC-07** — `operation_span` no-op cost pin. New
  `TestOperationSpanNoOpPerf` class in `tests/unit/test_perf_pins.py`.
  Bootstraps with `setup_otel` NOT called (the worst case for
  the no-op branch — OTel is installed via `[observability]`
  extra but never configured), invokes `with operation_span(...)`
  1000 times, asserts the per-call cost stays under 10 µs.
  Catches the regression class where a "harmless" change drags
  real work into the disabled path. Local measurement: ~1-2 µs
  per call. (2026-05-17, P2 sweep.)

- **PC-09** — PBKDF2 iteration floor + slow-derive gate. New
  `TestPBKDF2IterationFloor` class in
  `tests/unit/test_phase5_vault.py` with two tests:
  * `test_default_iterations_meet_owasp_2023_baseline` —
    asserts `_DEFAULT_KDF_ITERATIONS >= 600_000` via direct
    attribute read. Catches the silent downgrade case (someone
    edits the constant thinking it's a perf win).
  * `test_first_op_under_default_iterations_is_slow` —
    constructs an `EncryptedFileVault` with DEFAULT iterations
    (not the test override) and asserts the first `set()`
    call takes ≥ 50 ms. A fast KDF is the suspicious signal
    — it means iterations dropped or the KDF got replaced.
  Both run as regular unit tests (NOT perf-marker) because the
  security posture is too important to gate behind a nightly
  sweep. The slow-derive test adds ~250-400 ms to every test
  run that runs it; acceptable for the floor it provides.
  (2026-05-17, P2 sweep.)

- **PC-10** — `AsyncJsonHandler.emit` perf pin + ratio gate.
  New `TestAsyncHandlerEmitPerf` class in
  `tests/unit/test_perf_pins.py`. Builds two parallel paths:
  a sync `_FormattingSink` that runs the JsonFormatter on the
  calling thread, and the production `AsyncJsonHandler` that
  pushes formatting to the drain thread. Iterates 5000 emits
  through each, then asserts:
  * Absolute: async per-emit cost < 18 µs (bench shows ~6 µs;
    ~3× slack matches the generous-budget philosophy).
  * Ratio: `async_per_emit / sync_per_emit < 0.5` — async must
    be meaningfully faster than the equivalent sync chain, OR
    the design is broken. The ratio assertion catches the case
    where both regress together (absolute budget masks this).
  (2026-05-17, P2 sweep.)

*Items close here with a one-line summary and a commit ref,
newest last. Don't let items rot in "in progress" — keep each
one small enough to close in one sitting, or split it.*

---

## §6 Notes on methodology

This tracker was seeded by a **breadth audit** across four
surfaces: per-call hot paths (logging, redaction, context),
per-process startup (`import mgf.common`, settings, OTel wiring),
per-operation maintenance paths (crash reporter, atomic write,
codemod, vault KDF), and per-handler concurrency (async drain,
subprocess streamer). Each finding was verified against
`file:line` evidence AND with a local measurement (numbers cited
inline are from the maintainer's laptop, Linux 6.17, Python
3.12.3, on the v0.33 commit).

The shape is deliberately stolen from
[`RELIABILITY_AND_ACCURACY.md`](RELIABILITY_AND_ACCURACY.md) and
[`SECURITY_HARDENING.md`](SECURITY_HARDENING.md): priority table
→ work units → invariants → closed items; narrow, falsifiable
items; status flipped as items close.

**The tracker is intentionally thin.** mgf-common has good perf
hygiene already (LG-12, four RA-11 pins, the `mgf-fed bench`
subcommand, `docs/ops/benchmarks.md`). This audit found one
genuine P0 (`import mgf.common` cost is user-visible across the
federation), four residual P1 capacity gaps (sustained
throughput, large-payload scaling, crash reporter latency budget,
memory baseline), and five post-v1.0 polish items. **It is not
a 40-item tracker because mgf-common doesn't have a 40-item
problem.** The honest call for a leaf library: gate what is
load-bearing, measure what's worth measuring, don't manufacture
work.

**What a v0.2 depth pass should do** (when P0/P1 clears):

- **Measure the actual sustained-throughput ceiling** —
  PC-02's gate asserts a floor; the depth pass should record the
  *ceiling* (the records/sec at which the async queue starts
  backing up, the records/sec at which RSS grows monotonically,
  the records/sec at which p99 latency leaves the budget). The
  number goes in `docs/ops/benchmarks.md` as a capacity
  statement consumers can plan against.
- **Profile `import mgf.common` after the PC-01 fix** — a
  second-pass `python -X importtime` measurement will surface
  the next-biggest cost. Candidates: `pyyaml` at module load
  (it ships in core deps for YAML config loading), or `pydantic`
  itself (the core, not pydantic-settings — pydantic is heavy on
  its own). Decide whether further lazy-loading is worth the
  maintenance cost; the threshold should be "any remaining single
  import > 20 ms cumulative".
- **Walk the load-bearing modules under `cProfile`** —
  `redaction.py`, `json_formatter.py`, `_lifecycle.py`,
  `crash_reporter.py`. The breadth pass measured per-call cost;
  a profile-based pass shows where inside the call the time goes.
- **Resolve the open uncertainties this pass flagged:** whether
  PBKDF2's `derive()` actually takes ≥ 100 ms at the configured
  iteration count on contemporary hardware (PC-09 assumes; not
  measured here — the vault is in the `[vault]` extra and wasn't
  installed in the audit environment). Verified-absent in v0.1:
  OTel does NOT appear in the eager `mgf.common` import chain
  (the v0.1 first pass initially hypothesised it might;
  re-checked the `importtime` output and it isn't there — the
  OTel import is correctly deferred to inside `_wire_otel()`).
- **Cross-check against the sibling trackers** — PC-02
  (sustained throughput) interacts with mgf-http's HTTP-request
  throughput when consumers route logs over HTTP sinks; PC-04
  (crash-reporter latency) interacts with the bootstrap timeline
  RA-15 closed.

When an item closes, move it to §5. When the depth pass sharpens
an item, rewrite it in place and note the change at the top of
§1.
