# Performance

> **Conformance:** L1 (every Magogi-foundation project that
> ships perf-sensitive code paths)
> **Audience:** Engineers writing performance-sensitive code or
> benchmarks; reviewers triaging "this is slower than it used
> to be" PRs; AI agents deciding whether a perf test failure is
> a real regression or hardware noise.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **Scope.** The federation's perf-test discipline. Closes the
> class of issues that PAPER-50 surfaced — perf tests with
> absolute-time budgets that fire on slower-than-modern-laptop
> hardware regardless of code changes. The rules below force
> budgets to be *relative* (% of a recorded baseline), force
> baselines to live in-tree (auditable + portable), and
> separate "should be fast" benchmarks from "must be fast"
> regression gates.

---

## Rules box (read this first)

| ID | Rule |
|---|---|
| <a id="PF-01"></a>**PF-01** | **Performance budgets MUST be expressed as a regression threshold against a recorded baseline**, NEVER as an absolute wall-clock value. Right shape: "no more than 20% slower than `tests/perf/baselines.json::log_emit_p50_ns`." Wrong shape: "must complete in <100 µs/record." Absolute budgets are brittle across hardware tiers (PAPER-50). |
| <a id="PF-02"></a>**PF-02** | **Perf tests MUST be marked with `@pytest.mark.perf`** and excluded from the default four-gate baseline (`addopts = "-m 'not perf'"` in `pyproject.toml`). They run in a separate CI job, on a tagged hardware tier, or are skipped on local dev. Mixing perf tests into the unit-test baseline makes the unit-test baseline non-deterministic. |
| <a id="PF-03"></a>**PF-03** | **Every active perf budget MUST have a baseline recorded in `tests/perf/baselines.json`** with: the measurement (median, p95, p99 — pick one and stick with it), the recording date, the recording hardware tier, and the recording commit SHA. A perf test without a baseline file is a failing test. |
| <a id="PF-04"></a>**PF-04** | **Re-baselining MUST require human approval** — a separate PR whose only change is the `baselines.json` value, with a body that explains *why* the baseline moved (faster algorithm, more work per call, library upgrade, etc.). Automated baseline updates ("oh, this is the new normal") hide regressions. |
| <a id="PF-05"></a>**PF-05** | **Perf tests MUST run on a harness that enforces the warm-up phase, the minimum sample count, the statistical-summary shape, and the `baselines.json` result-format that PF-01/PF-03/PF-07 require.** The canonical reference harness is **`mgf-fed bench`** + its `tests/perf/` conftest (the `bench` + `baselines` fixtures, `assert_no_regression`, and `bench rebaseline` for PF-04) — both ship in mgf-common today and are the proven default. A consumer MAY copy that conftest, or wire pytest-benchmark / any equivalent runner directly, as long as the result shape and failure-message numerics match PF-01/PF-03/PF-07; a recorded equivalent is conformant (per `principles/non-adoption-is-valid.md`). Reinventing the *result shape* is the defect — not declining the named runner. |
| <a id="PF-06"></a>**PF-06** | **Public API performance characteristics SHOULD be documented in `PUBLIC_API.md`** for any name whose performance class is non-obvious. Annotate the line with the algorithmic complexity (`O(n)`, `O(log n)`) and the memory class (`O(1)`, `O(n)` allocations per call). A consumer choosing between two API shapes deserves this. |
| <a id="PF-07"></a>**PF-07** | **Perf-test failures MUST surface the comparison numerically** — "120 ns measured vs 100 ns baseline = +20% regression" — never as a binary pass/fail. The pytest failure message MUST contain: the metric name, the baseline value with units, the measured value with units, the regression percentage, and a pointer to `tests/perf/baselines.json` for re-baselining. |
| <a id="PF-08"></a>**PF-08** | **Hardware-tier-sensitive results MUST record the tier** (e.g., `"hw_tier": "modern-laptop-2024"`, `"hw_tier": "ci-amd64-small"`). Baselines from one tier MUST NOT be compared against measurements from another. Tier definitions live in `tests/perf/hw_tiers.md` (or similar — a short doc that describes the reference hardware behind each tag). |

---

## 1. Why PF-01 (relative budgets) matters

PAPER-50 documented the exact failure mode the rules prevent:
LG-12 in LOGGING.md said "≤ 100 µs per record" as an absolute
number. A consumer's pin-test baked the threshold in as a
hardcoded 1.0 s budget for 10,000 records. The test's failure
message asserted the only causes were "synchronous remote ship,
per-record I/O, or a formatter regression." In reality, the
test ALSO fires on slower-than-"modern-laptop" hardware
regardless of code changes.

The maintainer spent half an hour reverting + re-running before
realising the failure was the dev hardware, not the code.

Two compounding problems:

1. **Absolute budgets bake in one hardware tier.** A laptop in
   2024 isn't a laptop in 2030; a `c6i.xlarge` isn't a
   Raspberry Pi.
2. **The failure message points away from the real cause.**
   "Synchronous remote ship" sounds expensive; "your hardware
   is slow" doesn't.

PF-01 fixes (1) by forcing relative budgets; PF-07 fixes (2)
by forcing the failure message to lead with the actual numbers.

---

## 2. The baseline file shape (PF-03)

`tests/perf/baselines.json`:

```json
{
  "$schema_version": 1,
  "$updated_at": "2026-05-24T17:30:00+00:00",
  "$hw_tier": "modern-laptop-2024",
  "metrics": {
    "log_emit_p50_ns": {
      "value": 8200,
      "units": "ns",
      "recorded_at": "2026-05-24",
      "recorded_commit": "5fe78d4",
      "regression_threshold_pct": 20,
      "description": "JsonFormatter emit() median per record, 10k iterations"
    },
    "config_load_p95_ms": {
      "value": 14,
      "units": "ms",
      "recorded_at": "2026-05-24",
      "recorded_commit": "5fe78d4",
      "regression_threshold_pct": 30,
      "description": "BaseAppSettings load from env + file, p95 over 100 cold loads"
    }
  }
}
```

Per-metric `regression_threshold_pct` lets noisy metrics (cold
load times) tolerate more drift than tight inner loops.

---

## 3. The pytest-marker pattern (PF-02)

`pyproject.toml`:

```toml
[tool.pytest.ini_options]
markers = [
    "perf: performance / regression tests (run via `pytest -m perf`, excluded from default).",
]
addopts = "--strict-markers --strict-config -m 'not perf'"
```

`tests/perf/test_log_emit.py`:

```python
import pytest

@pytest.mark.perf
def test_log_emit_p50_ns_no_regression(bench, baselines):
    metric = bench.measure(
        lambda: emit_one(),
        iterations=10_000,
        warmup=1_000,
    )
    baselines.assert_no_regression("log_emit_p50_ns", metric.median_ns)
```

The default `pytest` (the four-gate baseline) skips perf tests
because of the `-m 'not perf'` filter. CI runs `pytest -m perf`
in a separate job on tagged hardware.

---

## 4. The failure-message shape (PF-07)

```
RegressionError: log_emit_p50_ns regressed by +24.4%
  baseline:    8200 ns   (recorded 2026-05-24, commit 5fe78d4, hw_tier=modern-laptop-2024)
  measured:   10200 ns
  threshold:    +20%
  description: JsonFormatter emit() median per record, 10k iterations

If the regression is real, find and fix it.
If the baseline is genuinely outdated, re-baseline with:
  $ uv run python -m mgf.common.bench rebaseline log_emit_p50_ns

A re-baselining PR MUST explain WHY the baseline moved.
```

This message:
- Leads with numbers (baseline + measured + threshold).
- Cites the recording context (hardware tier, commit).
- Gives the next-step command verbatim.
- Closes with the policy reminder (PF-04).

A reader who has never seen a perf test before can act on this
message.

---

## 5. Hardware tiers (PF-08)

`tests/perf/hw_tiers.md`:

```markdown
# Hardware tiers for perf baselines

## modern-laptop-2024
Reference: Lenovo X1 Carbon Gen 12 (Intel Ultra 7 155H, 32 GB RAM,
NVMe SSD), Ubuntu 24.04, Python 3.13.x. CPU governor: `performance`.

Baselines recorded here: log_emit_*, config_load_*, vault_unlock_*.

## ci-amd64-small
Reference: Woodpecker runner on a 4-vCPU / 8 GB shared host. CPU
governor: unknown. Highly variable; baselines stored here use
wider thresholds.

Baselines recorded here: integration_smoke_*.
```

Two tiers is enough to start. Adding a tier is a PR with a
short hardware description; tiers DON'T need to be machine-
verified (the baseline file's `$hw_tier` tag is operator-
discipline, not auto-detected).

---

## 6. `mgf-fed bench` — the canonical reference harness (PF-05)

The federation's canonical perf-test reference is `mgf-common
bench` + the `tests/perf/` conftest that ships with mgf-common.
The runner enforces:

- Minimum warm-up samples (drops cold-start noise).
- Minimum measurement samples (ensures the statistic is
  meaningful — median over 5 samples is not).
- Statistical summary (median, p95, p99, std-dev).
- Result format (JSON-shaped, compatible with `baselines.json`).
- Re-baselining flow (`mgf-fed bench rebaseline <metric>`).

What ships in mgf-common: the `bench` CLI subcommand (its own
fixed microbench set), and — load-bearing for consumers — the
`tests/perf/conftest.py` harness exposing the `bench` and
`baselines` fixtures, `assert_no_regression` (emitting the
PF-07 numeric message), `baselines.json`, and `hw_tiers.md`.

Consumers add their own perf tests in their own `tests/perf/`
directory (marked `@pytest.mark.perf`). The conformant path is
to copy the reference conftest, OR wire pytest-benchmark / any
equivalent runner directly — what binds is the result shape, not
the runner: the JSON output MUST match `baselines.json` and the
failure message MUST carry the PF-07 numerics, so cross-project
benchmark output stays ingestible. (A generic `mgf-bench`
plug-in entry-point group for third-party runners is a possible
future; it is not required for conformance today.)

---

## 7. Self-review checklist

- [ ] Every perf test is marked `@pytest.mark.perf` (PF-02).
- [ ] Default `pytest` invocation excludes `perf` marker (`addopts = "... -m 'not perf'"`) (PF-02).
- [ ] Every perf test compares against `tests/perf/baselines.json` (PF-03).
- [ ] Every metric in `baselines.json` carries `value`, `units`, `recorded_at`, `recorded_commit`, `regression_threshold_pct`, `description` (PF-03).
- [ ] No absolute-time threshold appears in any perf-test assertion (PF-01).
- [ ] Failure messages cite baseline + measured + threshold + commit + units (PF-07).
- [ ] Hardware tier is recorded in `baselines.json::$hw_tier` (PF-08).
- [ ] Re-baselining is a separate PR with rationale, never bundled with the change that moved the baseline (PF-04).

---

## 8. Mechanical enforcement

| Rule | Enforcement |
|---|---|
| PF-01 | Review-only. A custom check could grep perf-test code for absolute-time literals; not implemented. |
| PF-02 | `--strict-markers` (pytest config) rejects unknown markers — surfaces typos. The `addopts = "-m 'not perf'"` line is the actual enforcement. |
| PF-03 | `mgf-fed bench` + the `tests/perf/` conftest reject a perf test with no baseline entry (`assert_no_regression` raises PF-03). Shipped in mgf-common; also review-enforced for consumers not on that harness. |
| PF-04 | Review-only. Reviewer prompt: a PR that changes `baselines.json` MUST have rationale in the body. |
| PF-05 | Review-only. |
| PF-06 | Review-only. |
| PF-07 | The `mgf-common` `tests/perf/` conftest (`assert_no_regression`) formats the numeric message; it ships today. A consumer on an equivalent runner MUST reproduce the same numerics — review-enforced. |
| PF-08 | Review-only. A `$hw_tier` field missing from `baselines.json` is a review block. |

Several rules (PF-03, PF-05, PF-07) are enforced by the
`mgf-fed bench` + `tests/perf/` conftest harness, which ships
today. The rules are L1-binding regardless of which runner a
consumer chooses; the canonical harness is the proven default,
and any equivalent that produces the same result shape +
failure-message numerics is conformant.
