# Hardware tiers for perf baselines (PF-08)

> **PF-08:** hardware-tier-sensitive results MUST record the tier, and
> **baselines from one tier MUST NOT be compared against measurements from
> another.** The tier is operator discipline — the harness does **not**
> auto-detect it.

Two halves, and as of 2026-08-06 both are implemented:

- **Recording.** `baselines.json` carries `$hw_tier`, stamped by
  `Baselines.save()` from the tier the numbers were actually measured on.
- **Enforcing.** The running tier comes from `MGF_PERF_HW_TIER`, defaulting to
  the `HW_TIER` constant in [`conftest.py`](conftest.py). When it disagrees
  with `$hw_tier`, `assert_no_regression` **refuses**: it skips, naming both
  tiers, rather than rendering a verdict PF-08 does not entitle it to.

Until 2026-08-06 only the first half existed. This file described the rule and
the nightly went on breaking it — the gap being that nothing in the running
system knew which tier it was on. (Reported by TelePlasma as `FEEDBACK.md`
PAPER-154; mechanism adopted from TelePlasma `3753a00`.)

## Tiers

| Tag | Reference hardware | Used for |
|---|---|---|
| `dev-local-2026` | The maintainer's development workstation — Linux x86_64, Python 3.13.x, NVMe SSD. CPU governor not pinned, so thresholds are generous: these pins catch *gross* regressions and creeping drift, not micro-noise. **The tier the current baselines were recorded on** (all of them: `current_context_*`, `redact_string_*`, `json_formatter_*`, `redactor_*`, `async_emit_*`, `operation_span_*`, `bootstrap_*`, `log_file_record_*`). | Local `pytest -m perf`. The default when `MGF_PERF_HW_TIER` is unset, so this box needs no declaration. |
| `ci-amd64-small` | The Woodpecker runner executing `.woodpecker.yml`'s `perf-nightly` step — shared vCPUs, variable, on `python:3.12-slim`. Highly noisy; baselines stored under this tier would want a wider `regression_threshold_pct`. | Declared by the step's `MGF_PERF_HW_TIER` env. **No baselines recorded on it yet**, so every comparison there is refused — see below. |

## What the refusal looks like

Run the perf tier while declaring a tier the baselines were not recorded on:

```console
$ MGF_PERF_HW_TIER=ci-amd64-small pytest -m perf -rs --override-ini "addopts="
=========================== short test summary info ============================
SKIPPED [1] tests/perf/conftest.py:168: current_context_p50_ns:
  refusing a cross-tier perf comparison (PF-08).
    running tier:   ci-amd64-small (from $MGF_PERF_HW_TIER, default dev-local-2026)
    baselines tier: dev-local-2026 (from baselines.json $hw_tier)
Baselines recorded on one hardware tier MUST NOT be compared against
measurements taken on another, so this budget is neither passed nor failed
here. The benchmark itself still ran — only the comparison is withheld.
To get a verdict on this tier, record a baseline for 'ci-amd64-small';
see tests/perf/hw_tiers.md.
6 passed, 10 skipped
```

**Skip, not fail** — deliberately. A red that actually means *different
hardware* is how a gate teaches people to ignore its reds; and a green would
be the vacuity PF-08 exists to prevent. An outlawed comparison must render no
verdict *and say so*, which is the third outcome the harness now has.

Note the `passed`: the benchmarks still execute, and the relative
scaling/ratio invariants (`test_redactor_scales_linearly_in_*`,
async-vs-sync) are tier-independent by construction — they compare two
measurements from the *same* run — so they keep rendering a verdict on any
tier. On CI the step therefore remains a real smoke test; it simply no longer
claims a speed verdict it cannot support.

This is enforced, not merely documented:
[`test_baselines_selftest.py`](test_baselines_selftest.py) seeds a cross-tier
comparison in **both** directions — a 40x regression and a healthy
measurement — and asserts each one skips. "Cross-tier ⇒ green" would satisfy
a single arm and be exactly the failure mode the rule targets.

### The mute is pinned too

A skip renders no verdict, so the silence needs its own guard. If `HW_TIER`
ever drifts away from the shipped `$hw_tier`, every local `pytest -m perf` run
would go green-by-skip and every perf budget in the repo would stop being
enforced anywhere, silently and permanently.
`test_the_shipped_baselines_are_comparable_on_the_default_tier` is the test
that notices — and it runs in the **default** suite, not the perf tier, so the
notice does not depend on the step that would be muted.

The same trap applies to any test that seeds its own baselines file: it MUST
pin `MGF_PERF_HW_TIER` to the tier it seeded, or a runner's declaration is
inherited and the test passes by skip. The self-test's autouse fixture does
exactly that; without it, three of its arms go green under a CI-declared tier.

## Re-baselining (PF-04)

```sh
MGF_PERF_REBASELINE=1 pytest -m perf --override-ini "addopts="   # every metric
MGF_PERF_REBASELINE=1 pytest -m perf --override-ini "addopts=" -k <metric>
```

`save()` stamps `$hw_tier` from the **running** tier, so a re-baseline on a
non-reference host cannot file its numbers under the reference box's name.
And one file carries one tier: if re-tagging would leave rows behind that this
process did not re-record, `save()` raises rather than mint a file that is
half true. That is what stops a `-k`-filtered partial re-baseline from
defeating the whole mechanism.

`regression_threshold_pct` is *preserved* across a re-baseline, not
regenerated — deliberately, so a routine value refresh cannot quietly widen
the gate. Changing a threshold is a two-step act: drop that one field from the
row, then regenerate, so value and threshold are re-recorded together.

A re-baseline is a **deliberate, separate commit** whose body explains *why*
the baseline moved (PF-04) — never folded into a feature change.

## Recording the `ci-amd64-small` baseline

A **named operator step**, never something the pipeline does to itself: a gate
that re-baselines itself ratchets to whatever the hardware did that week and
can therefore never fail.

**Before you start, know the trade.** One baselines file carries one tier, so
landing a `ci-amd64-small` baseline moves authority to CI and makes every
*local* `pytest -m perf` skip instead. Today that is a swap, not an addition.
Having both at once needs per-tier baseline *sets* (`load()` selecting by
tier), which is unwritten.

The runner's workspace is discarded, so the file has to come out through the
log. On a branch:

1. Add a temporary step beside `perf-nightly` in `.woodpecker.yml`, same
   image, and give it an event the branch actually fires:

   ```yaml
     # TEMPORARY — delete in the same PR that lands the baseline.
     - name: perf-record-ci-baseline
       image: python:3.12-slim
       environment:
         MGF_PERF_HW_TIER: ci-amd64-small
         MGF_PERF_REBASELINE: "1"
       commands:
         - pip install --quiet -e ".[dev,observability,vault]" --index-url "$DEVPI_URL"
         - pytest -m perf --override-ini "addopts="
         - echo '--- BEGIN baselines.json ---'
         - cat tests/perf/baselines.json
         - echo '--- END baselines.json ---'
   ```

2. Push and let it run. **Re-run the pipeline several times** and record each
   metric's spread before trusting any of them — the runner is a shared,
   burstable host and its spread will be wider than the workstation's. Size
   `regression_threshold_pct` from *those* samples; the current values are
   `dev-local-2026` numbers and do not travel.
3. Copy the final block out of the log into `tests/perf/baselines.json`,
   delete the temporary step, and open it as its own PF-04 PR quoting the
   measurements.
4. `test_the_shipped_baselines_are_comparable_on_the_default_tier` will then
   fail on `dev-local-2026` — correctly, and that is the swap made visible.
   Either update `HW_TIER` to match, or land per-tier baseline sets first.

---

Adding a tier is a short PR: describe the reference hardware in the table
above, then record baselines tagged with the new `$hw_tier`. Two tiers are
enough to start.
