# Testing

> **Conformance:** L0 / L1 / L2 (see `README.md`)
> **Audience:** Human engineers AND AI coding agents.
> **Propagation (DOC-18):** when a TS-* rule changes, also update → README *Index of MUSTs* + the `mgf-fed standards` CLI + (for supervisor rules TS-22..TS-28) `mgf-test-supervisor` + the two-sided `TestRecord` alignment tests; new public testing surface ripples to `PUBLIC_API.{md,json}` + `CHANGELOG`. See DOCUMENTATION.md §10.4.

> **Scope.** Cross-project testing standard. Consolidates patterns
> previously scattered across `DESIGN_PRINCIPLES.md` (rule DP-05),
> `LOGGING.md` (§11 pitfalls), `ERROR_HANDLING.md` (§5.1 GUI worker
> tests, §9 testing error paths), and `CONFIGURATION.md` (§11). All
> those references remain valid; this document is the single home for
> cross-cutting testing rules and the place where new patterns land.

---

## Rules box (read this first)

| ID | Rule |
|---|---|
| <a id="TS-01"></a>**TS-01** | **`tests/` MUST mirror `src/` 1:1.** A new module at `src/<app>/foo/bar.py` MUST land with a corresponding `tests/unit/foo/test_bar.py`. CI MUST lint that no source file lacks a mirror beyond a documented allow-list. |
| <a id="TS-02"></a>**TS-02** | **The unit suite MUST run with no production credentials, no network, and no real third-party daemon.** External I/O is reached only through provider ABCs (rule **DP-02**), and tests use `MockProvider`. |
| <a id="TS-03"></a>**TS-03** | **Every test MUST run in isolation.** No shared module-level state, no order dependencies, no leakage of `HOME`/`XDG_*`/working directory between tests. Tests that mutate process-global state (logging handlers, OTel TracerProvider, sys.excepthook) MUST save + restore via fixtures. |
| <a id="TS-04"></a>**TS-04** | **`filterwarnings = ["error"]` MUST be set in `pyproject.toml`.** Warnings escalate to test failures so deprecations and resource leaks surface immediately. Per-test relaxation uses `pytest.warns(...)` or `@pytest.mark.filterwarnings(...)`. |
| <a id="TS-05"></a>**TS-05** | **`--strict-markers` and `--strict-config` MUST be set.** A typo in a marker or option fails the test run instead of being silently ignored. |
| <a id="TS-06"></a>**TS-06** | **Coverage gate ≥80% on every CI job, ≥90% on changed lines in a PR.** Lazy-imported sink code (Sentry, OTLP, journald, syslog, HTTP webhook) is excluded from coverage by `pragma: no cover` only when it is genuinely unreachable without the real backend. |
| <a id="TS-07"></a>**TS-07** | **Every typed exception MUST have a unit test** asserting (a) the message format, (b) field values are preserved, (c) field values are copied (not aliased) where applicable. (See `ERROR_HANDLING.md` §9.) |
| <a id="TS-08"></a>**TS-08** | **Every translation seam (rule EH-02) MUST have tests** for: success, the typed translation of each known third-party error, the catch-all "unknown vendor error" path. |
| <a id="TS-09"></a>**TS-09** | **Every "best-effort observer" (rule EH-10) MUST have a test** that injects a raising observer and asserts the loop continues. |
| <a id="TS-10"></a>**TS-10** | **Every security-relevant rule (SC-01..SC-18) MUST be pinned by a test** where mechanically possible. The pin test asserts the negative case (a bad input is rejected, a forbidden pattern absent in `src/`). See `SECURITY.md` §15. |
| <a id="TS-11"></a>**TS-11** | **GUI workers MUST have tests** that exercise both the success path AND the untyped-exception path (rule EH-06). The untyped path asserts `error` signal fires AND a crash report is written. |
| <a id="TS-12"></a>**TS-12** | **Async code MUST be tested with `pytest-asyncio` (or `anyio` plugin).** A coroutine that has no test is a defect. The event loop is per-test by default; cross-test loop reuse requires a documented justification. |
| <a id="TS-13"></a>**TS-13** | **Property tests (Hypothesis) MUST cover any code that round-trips data** — parse/serialise, save/load, encode/decode. Strategies live next to the round-trip code in a `_strategies.py` module so they can be reused across multiple property tests. |
| <a id="TS-14"></a>**TS-14** | **Performance budgets MUST be pinned by a benchmark that measures the production code path.** The budget's *shape* is owned by `PERFORMANCE.md`: it MUST be a regression threshold against a recorded baseline, never an absolute wall-clock value (**PF-01**, closing PAPER-50), and the benchmark MUST be marked `@pytest.mark.perf` and excluded from the default unit baseline (**PF-02**) — perf tests run in the nightly/tiered perf sweep (`pytest -m perf`, TS-19), not the per-PR four-gate run, so CI-hardware noise never fails the unit suite. The benchmark MUST measure the real production path, not pytest's stderr-capture (see `LOGGING.md` §11.1). |
| <a id="TS-15"></a>**TS-15** | **Smoke tests against real backends MUST be opt-in** (separate marker, separate CI job) and MUST be read-only by default. Mutating smoke tests run only against scratch environments and clean up after themselves. |
| <a id="TS-16"></a>**TS-16** | **Every test failure message MUST be self-explanatory.** A reader who has never seen the test before MUST understand from the failure output what assertion failed and why. Use `assert x == y, f"context: {ctx}"` when the bare assert is ambiguous. |
| <a id="TS-17"></a>**TS-17** | **Test fixtures MUST be discoverable via the conftest hierarchy**, never imported across test packages. A fixture used by one test file lives in that file; used by a directory's tests lives in `tests/<dir>/conftest.py`; used everywhere lives in `tests/conftest.py`. |
| <a id="TS-18"></a>**TS-18** | **Slow tests (>1 s individually) MUST be marked `@pytest.mark.slow`** and excluded from the default `pytest` invocation. Default CI runs only the fast suite; a separate "slow" job runs the marked tests. |
| <a id="TS-19"></a>**TS-19** | **Test layer markers MUST be registered.** A project's `pyproject.toml` MUST declare every pytest marker its tests use under `[tool.pytest.ini_options].markers`. Combined with `--strict-markers` (TS-05), this makes the layer structure explicit and prevents silent typos. **Two marker axes:** (a) **layer markers** tied to the test directories — `smoke` / `e2e` / `integration` (with `unit/` being the unmarked default); (b) **orthogonal markers** for cross-cutting concerns — `perf` (RA-11 nightly performance-pin sweep, excluded from per-PR default) and `slow` (TS-18 — tests >1 s wall-clock, excluded from per-PR default). A project SHOULD register `slow` even if no test currently uses it — defensive registration prevents `--strict-markers` from rejecting a future addition. |
| <a id="TS-20"></a>**TS-20** | **Tests MUST be partitioned across the four canonical directories.** `tests/unit/` (default; mirrors `src/` per TS-01), `tests/integration/` (cross-module), `tests/e2e/` (full bootstrap-to-output chains), `tests/smoke/` (post-install sanity). Each non-unit directory's tests SHOULD apply the matching marker via a top-level `pytestmark` so a layer can be isolated with `pytest -m <layer>`. |
| <a id="TS-21"></a>**TS-21** | **Every published library MUST have a smoke layer.** The smoke layer (`tests/smoke/`) runs in <100 ms total and verifies (a) the package imports, (b) each name in `__all__` is reachable, (c) the shortest happy path of every public entry function works. It exists so a downstream CI job can pre-flight a wheel install before kicking off heavier suites. CLI / desktop applications without a wheel surface MAY skip this. |
| <a id="TS-22"></a>**TS-22** | **Tiered runs MUST be supported.** Every test in the federation MUST be assignable to one of four tiers via marker overlay (tier-0 / tier-1 / tier-2 / tier-3, ordered by per-test runtime budget — see §13.11 for the marker → tier mapping). The test runner MUST accept `--tier N` (single tier), `--tier N-M` (range), and `--budget T` (time budget; greedy knapsack selection with tier-0 always included). The four-tier model is the federation's coordination contract: a sibling that says "run my tier-0" runs a sub-second smoke check; "run my tier-0-2" runs the daily contract suite; "run my tier-0-3" runs everything. Reference implementation: `mgf.test_supervisor.tiers` (in the published [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor) sibling). |
| <a id="TS-23"></a>**TS-23** | **Every CI run MUST survive and structure a pytest crash, hang, or signal-death.** The run harness MUST achieve all four outcomes: (1) a pytest crash, hang, or signal-driven death is recorded as a **structured per-test outcome** rather than killing the run driver itself; (2) a hang is **bounded** (a wall-clock cap fires) and captured **with a thread dump**, not left to the CI wallclock; (3) failure context for each failed / hung / crashed test is assembled into a **single file** (TS-24/TS-27); (4) the harness introduces **no bootstrap circularity** — the tool that runs the tests does not have to import the project being tested. The **canonical reference path** — a proven default — runs pytest as a **subprocess** (not as a pytest plugin) under a small **pure-stdlib** supervisor that outlives the test process: `mgf.test_supervisor.supervisor` (in the published [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor) sibling), driven via `mgf-test-supervisor` / `mtest` or the `mgf-fed test` shim. An **equivalent mechanism that demonstrably meets all four outcomes is conformant** (the consumer records the choice per [`non-adoption-is-valid`](principles/non-adoption-is-valid.md) — e.g. a leaf that would otherwise close a dependency cycle by adopting the supervisor). What is **never** compliant: plain `pytest` invoked directly from CI **with no crash / hang / signal capture at all** (exit-code-only CI that surfaces "exit 139" or a wallclock-timeout kill with no per-test attribution). |
| <a id="TS-24"></a>**TS-24** | **Per-test records MUST be structured.** Every test outcome MUST be written as **one JSON object per line** to `.gates/last-run/run.jsonl`. The schema MUST mirror `mgf.common.observability.DoctorFinding` (`code` / `severity` / `summary` / `detail` / `why` / `remediation`) plus the testing-specific fields (`test_id`, `outcome`, `duration_ms`, `tier`, `markers`, `captured_stdout`, `captured_stderr`, `run_id`, `ts_iso`). The `schema_version` field is mandatory. The importable schema type lives at `mgf.common.testing.TestRecord`; downstream tooling (Magogi dashboard, Slack relay, status badges) MUST use that type rather than parsing the JSONL by hand. Reference: `mgf.test_supervisor.records` (in the published [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor) sibling). |
| <a id="TS-25"></a>**TS-25** | **Flaky tests MUST be auto-quarantined.** *(L2 — every L2 project; L1 projects MAY defer.)* A test that fails 3 or more times in the last 5 outcomes MUST be auto-quarantined (skipped on subsequent default runs). The supervisor MUST accept `--quarantine-attempt` to include quarantined tests AND track passes toward eviction; 3 consecutive passes under `--quarantine-attempt` MUST evict the test from quarantine. Quarantine state lives in `.gates/quarantine.json` with atomic-write persistence (temp + rename). The runner MUST also accept `--no-skip-quarantined` for one-off "show me everything" runs that don't touch the eviction counter. Reference: `mgf.test_supervisor.quarantine` (in the published [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor) sibling). |
| <a id="TS-26"></a>**TS-26** | **Built-binary dependencies MUST have a freshness check.** A project shipping (or depending on) a compiled artifact — Zig broker, Go agent, Rust extension, C library, Swift bundle — MUST verify the binary's mtime is ≥ the source-tree's mtime before the suite runs. **Stale** (binary older than source) → FAIL (NOT skip) with the rebuild command in the error message. **Missing** → SKIP with the same command. A dedicated smoke test at `tests/smoke/test_<binary>_freshness.py` MUST fire the check early in the run so a 30-minute test session doesn't end with "oh, your binary was from yesterday." Projects with no compiled artifacts (pure-Python libraries, mgf-common itself) MAY declare TS-26 N/A in their conformance tracker. Reference: `tests/_broker.py` (mgf-hrb's exemplar; mgf-common's pyproject extras / native deps are the analogue for libraries that DO ship compiled bits). |
| <a id="TS-27"></a>**TS-27** | **Failures MUST be explorable from a single file.** Every failed / hung / crashed test MUST produce a per-failure detail bundle at `.gates/last-run/failures/<safe-test-id>.txt` carrying ALL of: traceback (deduplicated to the failing test); captured stdout + stderr (last 4 KB each); live process state of the project's daemon (PID + `/proc/<pid>/wchan` + thread count from `/proc/<pid>/task/`); dmesg tail (best-effort, last 30 lines); environment fingerprint (Python version, project install path, language-toolchain version, git branch + HEAD); the last 5 outcomes of this test; the chunk's wall-time + exit code + signal name; the **perf attribution** for the test (TS-28) when captured; a pointer to the project's `DIAGNOSING_HANGS.md` for failure-mode recipes. The bundle is **one file per failure** so a triager can open it directly — no scrolling pytest output to assemble the picture. Reference: `mgf.test_supervisor.supervisor::_write_failure_bundle` (in the published [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor) sibling). |
| <a id="TS-28"></a>**TS-28** | **Per-test perf attribution MUST be opt-in, surfaced through the supervisor, and fail-open.** *(L2 — every L2 project; L1 projects MAY defer.)* When the supervisor is invoked with `--perf` (env `MGF_PERF=1`), every test MUST be capturable at two granularities: **function-level** via a per-test `cProfile` (top-N frames by cumulative time — covering the test body AND the code under test), and **section/log-level** via a `perf_span("label")` context manager any test or production code path MAY wrap around a meaningful unit (a DB call, a render, the block around a log line). Capture writes **one JSON artifact per test** carrying `{test_id, total_ms, functions[], spans[]}`; the supervisor reads them out-of-process (TS-23 — it never imports the writer) and surfaces them in three places: the `TestRecord.perf` field of `run.jsonl` (schema ≥ `1.1.0`), the TS-27 failure bundle's `perf attribution` section, and a run-level `perf-summary.md` (slowest tests + hottest functions by self-time). Capture MUST be **fail-open** (a profiler or artifact-IO error never changes a test's outcome), **bounded** (top-N functions, capped span count), and MUST NOT pollute the duration baseline the budget selector + auto-tier rely on (perf runs are diagnostic, not timing samples). The `perf_span` cost when capture is inactive MUST be negligible (a single context-var read) so it is safe to leave in production code paths. Reference: writer `mgf.common.testing.perf_span` + the `mgf_common` pytest plugin's `--mgf-perf`; reader `mgf.test_supervisor.perf` (in the published [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor) sibling). |

---

## 1. Mental model — the confidence pyramid

Tests answer one question: **how much would I bet that this code is correct?**

The test pyramid maps types of tests to the confidence they buy:

```
                    ▲                      Real-environment
                    │                      smoke (rare)
                    │   ┌─────────────┐
                    │   │ End-to-end  │   Read-only against real backends.
                    │   │   (1–10)    │   Slow, expensive. Marker-gated.
                    │   └─────────────┘
                    │ ┌─────────────────┐
                    │ │  Integration    │  Multiple modules wired together,
                    │ │   (10–50)       │  but external I/O through Mock.
                    │ └─────────────────┘
                    │┌──────────────────────┐
                    ││       Unit           │  Single module + its direct
                    ││    (100s–1000s)      │  collaborators. Pure-Python.
                    ││                      │  Fast (<10ms per test).
                    │└──────────────────────┘
                    │┌────────────────────────────┐
                    ││  Property (Hypothesis)     │  Round-trip + invariant tests.
                    ││  (10–50, generated 100s)   │  Catch what unit tests miss.
                    │└────────────────────────────┘
                    │┌────────────────────────────────┐
                    ││  Static analysis: mypy + ruff  │  Free, runs on every save.
                    ││  + import-linter               │
                    │└────────────────────────────────┘
                                                              ▶ Speed / volume
```

The bottom of the pyramid is wide and fast; the top is narrow and
slow. A test suite without a wide unit base is not a test suite — it
is theatre.

**Coverage % is a FLOOR, not the bar.** The real question the
pyramid answers is *behavioral adequacy* — does every meaningful
path have a test that asserts the right BEHAVIOUR? A line-coverage
number is the canonical proxy for that outcome, not the outcome
itself: 90% reached by a tautological assertion on a re-export shim
is theatre that happens to be green. Hit the gate (TS-06) AND meet
the bar. See §9.5 for the adequacy criterion + the
design-for-testability finding.

---

## 2. Test layout (rule TS-01)

`tests/` mirrors `src/` 1:1. The mapping rule:

| Source | Test |
|---|---|
| `src/<app>/<sub>/foo.py` | `tests/unit/<sub>/test_foo.py` |
| `src/<app>/<sub>/bar/baz.py` | `tests/unit/<sub>/bar/test_baz.py` |
| Cross-cutting concern (e.g., schema versioning) | `tests/unit/test_<concern>.py` at the unit root |

Multiple tests per source file is fine (`test_foo_happy_path.py`,
`test_foo_edge_cases.py`) when one file would exceed ~500 lines.
Splitting by concern beats one mega-file.

The mirror rule has two pragmatic exemptions:

- **Internal dunder helpers** that are exercised only through their
  public caller — no separate test needed.
- **Trivial re-export shims** (a `<sub>/__init__.py` that does
  `from <sub>._impl import X`) — covered by tests of the impl.

CI MUST surface "source file with no mirror" findings; the
allow-list lives in a small file under `tests/` so additions require
review.

---

## 3. Fixture patterns (rule TS-17)

### 3.1 The conftest hierarchy

Pytest discovers `conftest.py` walking from the test file up to
`rootdir`. Fixtures defined higher are visible lower. The rule:

| Scope of use | Lives in |
|---|---|
| One test file | The test file itself |
| Tests in one directory | `tests/<dir>/conftest.py` |
| Tests in one package | `tests/unit/<package>/conftest.py` |
| Every test in the project | `tests/conftest.py` |

**Never** import a fixture across test packages — it breaks pytest's
discovery contract and produces fixture-resolution-by-luck. If two
distant test files need the same fixture, promote it to a higher
conftest.

### 3.2 Scope choice

| Scope | When to use |
|---|---|
| `function` (default) | Almost always. Cheap setup; clean isolation. |
| `module` | The fixture's setup is expensive and shared by a whole module of tests that don't mutate it (e.g., parsed YAML schema). |
| `class` | A test class clusters around shared state (rare in pytest-style). |
| `session` | The setup is genuinely process-global and one-shot — OTel `TracerProvider`, a real-DB schema migration, a slow image build. |

**Session-scoped fixtures need extreme care** — they leak state
across tests by definition. Two patterns that work well:

```python
@pytest.fixture(scope="session")
def _session_tracer_provider() -> Iterator[InMemorySpanExporter]:
    """Process-global setup of the OTel TracerProvider.

    `set_tracer_provider` is one-shot per process — calling twice
    is a silent no-op. Session scope is the only correct choice.
    """
    exporter = InMemorySpanExporter()
    provider = TracerProvider(...)
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
    yield exporter

@pytest.fixture
def in_memory_spans(_session_tracer_provider) -> Iterator[InMemorySpanExporter]:
    """Per-test view onto the session exporter — clears between tests."""
    _session_tracer_provider.clear()
    yield _session_tracer_provider
```

### 3.3 autouse vs explicit

Autouse fixtures fire on every test in their scope. Use them for:

- **Process-global state restoration** (sys.excepthook, root logger
  handlers, environment variables that may leak).
- **Identity / context that every test needs** (e.g., `mgf.common`'s
  observability tests autouse-set a known app name; see
  `tests/unit/observability/conftest.py`).

Avoid autouse for:

- Fixtures that some tests don't need — pay the setup cost
  needlessly.
- Fixtures that change behaviour in surprising ways — explicit is
  better.

### 3.4 Cleanup discipline

Fixtures that allocate resources MUST yield + clean up:

```python
@pytest.fixture
def isolated_logging() -> Iterator[None]:
    yield
    for handler in logging.root.handlers[:]:
        with contextlib.suppress(Exception):
            handler.close()
        logging.root.removeHandler(handler)
```

Closing handlers explicitly is critical when `filterwarnings =
["error"]` is set — Python's GC fires `ResourceWarning` on a
RotatingFileHandler that wasn't closed, which the filter escalates
to a hard test failure. (See `LOGGING.md` §11.)

---

## 4. The mock provider pattern (rule DP-02 + TS-02)

Every third-party I/O dependency sits behind a provider ABC. Tests
exercise the **mock** implementation:

```python
# src/<app>/providers/mock_provider.py
from dataclasses import dataclass, field
from typing import Callable

from <app>.exceptions import ProviderError

@dataclass
class FailurePlan:
    """Register targeted failures for a MockProvider.

    Each entry maps (method_name, match_predicate) → exception to raise.
    Used by chaos / degradation tests without touching real backends.
    """
    _plan: list[tuple[str, Callable[..., bool], Exception]] = field(default_factory=list)

    def inject(
        self,
        *,
        method: str,
        exc: Exception,
        match: Callable[..., bool] = lambda *a, **k: True,
    ) -> None:
        self._plan.append((method, match, exc))

    def consume(self, method: str, *args, **kwargs) -> None:
        for m, pred, exc in self._plan:
            if m == method and pred(*args, **kwargs):
                raise exc

class MockProvider(ResourceProvider):
    def __init__(self) -> None:
        self.failure_plan = FailurePlan()
        self._store: dict[str, EntityHandle] = {}

    def get(self, name: str) -> EntityHandle:
        self.failure_plan.consume("get", name)
        if name not in self._store:
            raise ResourceNotFoundError(name)
        return self._store[name]

    # ... etc
```

### 4.1 The lock-in test

The set of methods on the ABC is **frozen** by a test:

```python
# tests/unit/test_provider_base.py
EXPECTED_METHODS = frozenset({
    "get", "create", "delete", "list", "state",
    # ... full set
})

def test_provider_methods_locked_in() -> None:
    actual = {
        name for name in dir(ResourceProvider)
        if not name.startswith("_") and callable(getattr(ResourceProvider, name))
    }
    assert actual == EXPECTED_METHODS, (
        "ResourceProvider abstract surface changed. Update both "
        "MockProvider AND every production implementation in the "
        "same commit, then update EXPECTED_METHODS."
    )
```

This test catches:

- A new abstract method added without a corresponding mock impl.
- A method removed from the ABC but still called by core code.
- A method renamed without updating the mock or production impl.

Without the lock-in, a refactor that adds a method silently breaks
in production but passes tests because the mock didn't get the new
method.

---

## 5. Property tests (rule TS-13)

Property tests use [Hypothesis](https://hypothesis.readthedocs.io/)
to generate hundreds of inputs and check an invariant.

### 5.1 When to use

- **Round-trip code** — parse / serialise, encode / decode, save /
  load, compress / decompress. The invariant: `f(g(x)) == x`.
- **Order independence** — the result of an operation MUST not
  depend on input order. The invariant: `op(sorted(xs)) ==
  op(reversed(xs))`.
- **Idempotence** — the result of running an operation twice equals
  running it once. The invariant: `op(op(x)) == op(x)`.
- **Bounds** — `0 <= f(x) <= MAX` for all `x`.

### 5.2 Strategies live next to the code

```python
# tests/unit/config/_strategies.py
from hypothesis import strategies as st

config_dict_strategy = st.fixed_dictionaries({
    "version": st.just(1),
    "backend_uri": st.text(min_size=1, max_size=200),
    "ssh_timeout_seconds": st.integers(min_value=1, max_value=600),
})

@st.composite
def settings_subclass_strategy(draw):
    cfg = draw(config_dict_strategy)
    return Settings(**cfg)
```

```python
# tests/unit/config/test_loader_property.py
from hypothesis import given
from .{_strategies} import settings_subclass_strategy

@given(settings=settings_subclass_strategy())
def test_save_load_roundtrip(settings, tmp_path):
    path = tmp_path / "x.yaml"
    save_settings(settings, path)
    loaded = load_settings(type(settings), path=path)
    assert loaded == settings
```

### 5.3 Failing examples are saved

Hypothesis stores failing examples in `.hypothesis/examples/` —
this directory MUST be `.gitignore`d (it's machine-local) but the
`@example(...)` decorator can pin a specific reproducer for a
known regression:

```python
@example(value="\x00") # null byte — historically broke YAML roundtrip
@given(value=st.text())
def test_yaml_handles_arbitrary_unicode(value, tmp_path):
    ...
```

### 5.4 Performance

Property tests run hundreds of examples per test. Keep the body
fast (<10 ms per example) so the suite stays under the 5 s budget.
For tests that are inherently slow per example, set
`@settings(max_examples=20)` and document why.

### 5.5 Pattern-coverage property tests (the redactor recipe)

A code path that scans for **a named class of inputs** (a redactor
matching token shapes, a parser accepting protocol versions, a
validator rejecting malformed identifiers) deserves a property
test of the form **"for every valid sample of the named class,
the scanner catches it"**. Unit tests with hand-picked examples
prove only that the scanner catches *those* examples; the
property test proves it catches the *class*.

Reference shape — `mgf-common`'s `Redactor` value-pattern
coverage (`RA-12`,
[`tests/unit/observability/test_redaction_property.py`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/tests/unit/observability/test_redaction_property.py)):

```python
# tests/unit/observability/_strategies.py
from hypothesis import strategies as st

# Strategy: generate strings that look like AWS access keys.
# Real keys are AKIA + 16 base32 chars; the strategy generates
# the entire valid space.
aws_access_key_strategy = st.builds(
    lambda body: f"AKIA{body}",
    st.text(alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ234567", min_size=16, max_size=16),
)

# tests/unit/observability/test_redaction_property.py
from hypothesis import given
from mgf.common.observability.redaction import Redactor
from ._strategies import aws_access_key_strategy

@given(key=aws_access_key_strategy)
def test_redactor_catches_every_valid_aws_access_key(key: str) -> None:
    """RA-12: for every string matching the AWS access-key shape,
    Redactor.redact_string MUST scrub it. Hand-picked unit tests
    catch the canonical examples; this property test catches the
    long-tail."""
    r = Redactor.default()
    scrubbed = r.redact_string(f"prefix {key} suffix")
    assert key not in scrubbed
```

One strategy + one property test per named class of input. The
strategy is the contract — keep it in a `_strategies.py` module
alongside the code (rule TS-13). When a real-world leak surfaces
a shape the strategy doesn't generate, **extend the strategy
first**, then prove the regression with `@example(...)`.

---

## 6. GUI testing (rule TS-11)

PySide6 + `pytest-qt` is the reference stack.

### 6.1 The qtbot pitfall

`qtbot.addWidget(...)` accepts only `QWidget` subclasses. A
`CoreWorker(QThread)` is NOT a widget — it cannot be registered.
The right shape:

```python
def test_worker_completes(qtbot):
    worker = MyWorker(arg="x")
    with qtbot.waitSignal(worker.finished_with_result, timeout=5_000) as block:
        worker.start()
    assert block.args[0] == "expected result"
    worker.wait(1_000)  # ensure clean thread join
```

The `worker.wait(1_000)` is critical — without it, pytest-qt's
teardown can run before the worker's `do_work()` returns, leading
to flaky signal counts.

### 6.2 The untyped-exception path

Rule **TS-11** requires every worker to have a test for the
untyped-exception path:

```python
def test_worker_routes_untyped_exception_through_crash_reporter(
    qtbot, monkeypatch, tmp_path,
):
    monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path))

    class BadWorker(CoreWorker):
        def do_work(self) -> Any:
            raise KeyError("synthetic untyped failure")

    worker = BadWorker()
    with qtbot.waitSignal(worker.error, timeout=5_000) as block:
        worker.start()

    assert "synthetic untyped failure" in block.args[0]

    crashes = list((tmp_path / "<app>" / "crashes").glob("*.json"))
    assert len(crashes) == 1, "untyped exception did not produce a crash report"

    worker.wait(1_000)
```

### 6.3 Designer-loaded UI tests

Tests that load a `.ui` file via `QUiLoader` need the Qt application
running:

```python
@pytest.fixture(scope="session")
def qapp() -> Iterator[QApplication]:
    app = QApplication.instance() or QApplication([])
    yield app
    # don't close — pytest-qt manages lifecycle

def test_my_dialog_loads(qapp):
    dlg = MyDialog()
    assert dlg.dialog is not None
    assert dlg.ok_button.text() == "OK"
```

### 6.4 Headless CI

CI MUST run the GUI test job under `xvfb-run` (Linux) or with a
virtual display. The Woodpecker / GitHub-Actions step:

```yaml
- name: gui-tests
  image: python:3.12
  commands:
    - apt-get update && apt-get install -y xvfb libegl1 libxkbcommon0
    - xvfb-run -a uv run pytest tests/unit/gui/ -q
```

---

## 7. Async testing (rule TS-12)

Use `pytest-asyncio` (`@pytest.mark.asyncio`) or `anyio` plugin
(works for both asyncio and trio).

### 7.1 Per-test event loop

The default `pytest-asyncio` mode is `function`-scoped event loops:

```python
# pyproject.toml
[tool.pytest.ini_options]
asyncio_mode = "auto" # mark async test functions automatically
asyncio_default_fixture_loop_scope = "function"
```

Per-test loops mean cancellation, exception, and pending-task state
do not bleed across tests. Cross-test loop reuse (session scope)
is occasionally needed for tests that share an expensive async
client; document why and clean up explicitly.

### 7.2 Cancellation tests

Every long-running async operation MUST have a cancellation test:

```python
@pytest.mark.asyncio
async def test_long_op_respects_cancellation():
    task = asyncio.create_task(long_op())
    await asyncio.sleep(0.1)
    task.cancel()
    with pytest.raises(asyncio.CancelledError):
        await task
    assert long_op.cleanup_ran
```

### 7.3 No `asyncio.sleep` for synchronisation

`asyncio.sleep(0.5)` to "let the worker get to the right state" is
flaky. Use explicit synchronisation:

```python
# WRONG
async def test_x():
    task = asyncio.create_task(producer(queue))
    await asyncio.sleep(0.5)
    assert queue.qsize() == 3

# RIGHT
async def test_x():
    task = asyncio.create_task(producer(queue))
    for _ in range(3):
        await queue.get()
    assert queue.empty()
```

---

## 8. Performance benchmarks (rule TS-14)

A standard with a budget (LG-12: ≤100 µs per log record) is unowned
without a benchmark that asserts the budget.

### 8.1 Shape

```python
# tests/unit/observability/test_logging_perf.py
import logging
import time
from logging.handlers import RotatingFileHandler

import pytest

from mgf.common.observability import setup_logging

@pytest.mark.perf
def test_log_record_under_100us(tmp_path):
    """LG-12: per-record cost MUST stay ≤100 µs at INFO."""
    setup_logging(level="DEBUG", fmt="json", log_file=tmp_path / "perf.log")

    # Drop the console StreamHandler so we measure the rotating-file
    # sink path (the surface LG-12 actually constrains) without also
    # paying for pytest's stderr-capture machinery per record.
    for handler in logging.root.handlers[:]:
        if not isinstance(handler, RotatingFileHandler):
            logging.root.removeHandler(handler)

    LOG = logging.getLogger("perf-test")

    # Warm-up
    for _ in range(100):
        LOG.info("warm-up", extra={"i": _})

    # Timed loop
    n = 10_000
    start = time.perf_counter()
    for i in range(n):
        LOG.info("benchmark message", extra={"iteration": i, "user": "bob"})
    elapsed = time.perf_counter() - start

    per_record_us = (elapsed / n) * 1_000_000
    assert per_record_us < 100, f"LG-12 budget exceeded: {per_record_us:.1f} µs/record"
```

### 8.2 The pytest-capture trap

`pytest` captures stderr by default; under capture, every record
through a `StreamHandler(sys.stderr)` pays real overhead per line.
The benchmark MUST measure the production path. The shape above
removes the console handler before the timed loop. (See
[`LOGGING.md`](LOGGING.md) §11.1 for the full story.)

### 8.3 Budget tracking

A benchmark that drifts toward its budget is a regression in
progress. Optional: write the measured value to a small file
(`docs/benchmarks/<rule>.csv`) on every CI run; a regression PR
that pushes the value up by >10% triggers a code-review flag.

### 8.4 The `perf` marker — opt-in by default (the nightly sweep)

Performance budgets fail on noise more than they fail on real
regressions: CI hardware throttles, GC kicks in, a neighbour
container saturates a core. A perf test that *fails CI* on every
noise blip is worse than no perf test — engineers learn to ignore
red, and a real regression slides through.

The discipline: **register a `perf` pytest marker, default-exclude
it from the per-PR suite, run it nightly with green-streak
tracking**. A regression that's real surfaces over multiple nights;
a single noise blip is visible but ignorable.

```toml
# pyproject.toml
[tool.pytest.ini_options]
markers = [
    "perf: performance budget assertion (run via `pytest -m perf`)",
    # ... other markers
]
addopts = "-m 'not perf and not slow'"
```

```python
# tests/unit/test_perf_pins.py
import pytest
pytestmark = pytest.mark.perf

class TestRedactorPerf:
    def test_redact_string_under_5us(self) -> None:
        ...
```

CI: a nightly Woodpecker / GitHub Actions job runs `pytest -m
perf` and surfaces failures to a dashboard, not as merge blocks.
The per-PR suite stays fast and reliable; the perf sweep stays
honest.

**Reference shape (Woodpecker):**

```yaml
- name: perf-nightly
  image: python:3.12-slim
  when:
    event: cron
  commands:
    - pip install --quiet --upgrade pip
    - pip install --quiet -e ".[dev,observability]"
    - pytest -m perf -v --override-ini "addopts="
```

Schedule on the platform side (Codeberg / Forgejo:
`Settings → Crons → New Cron`; GitHub Actions:
`schedule: - cron: '0 3 * * *'` at the workflow top). The
`--override-ini "addopts="` is needed because the per-PR
default suite excludes perf via `-m "not perf and not slow"`;
the nightly step has to opt back IN to what it's there to run.

### 8.5 Generous budgets — catch the order-of-magnitude, not the noise

A budget set to "current measurement × 1.05" will fail on the
first warm-cache miss. A budget set to "current measurement
× 2-3" catches the regressions that actually matter
(order-of-magnitude drift, silent creep across releases) without
false-positiving on CI noise.

The reference table from `mgf-common`'s RA-11 closure
([`tests/unit/test_perf_pins.py`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/tests/unit/test_perf_pins.py)):

| Surface | Measured | Budget | Slack |
|---|---|---|---|
| `current_context()` | ~170 ns | 500 ns | 3× |
| `Redactor.redact_string` | ~1.5 µs | 5 µs | 3× |
| `JsonFormatter.format` | ~10 µs | 30 µs | 3× |
| `bootstrap()` cold | ~1.2 ms | 5 ms | 4× |

Tighten when measurements stabilise across multiple nightly runs;
loosen if a CI hardware change makes noise spike. The point is
to catch GROSS regressions, not to be a precise speedometer.

---

## 9. Coverage (rule TS-06)

### 9.1 The gate

`pyproject.toml`:

```toml
[tool.coverage.run]
source = ["src/<app>"]
branch = true

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "raise NotImplementedError",
    "if TYPE_CHECKING:",
    "if __name__ == .__main__.:",
]
show_missing = true
skip_covered = false
```

CI:

```bash
uv run pytest --cov --cov-fail-under=80
```

### 9.2 What's NOT excluded

- Error paths. A test for the success path AND a test for each
  documented error path is the standard.
- Conditional branches. Branch coverage > line coverage.
- Logging calls. `LOG.info(...)` lines count; if you can't reach
  them in test, ask why the code path exists.

### 9.3 What IS excluded

- Lazy-import branches (`try: import opentelemetry; except ImportError:
  pass`) — the negative branch is exercised in the default install
  CI job; the positive branch in the `[observability]`-extra job.
  Both jobs together hit the line.
- `if TYPE_CHECKING:` blocks (no runtime path).
- Defensive impossibilities (`assert never_reached`) explicitly
  marked `pragma: no cover`.

### 9.4 Per-module floors

The 80% project-wide gate is a floor. New modules (added in the
last 30 days) MUST be at ≥90%. The convention:

```bash
# CI helper script:
uv run pytest --cov --cov-report=json
python tools/coverage_check_new_modules.py --min 90 --window 30
```

### 9.5 The bar is behavioral adequacy; the % is the floor

The coverage gate (TS-06) binds the **outcome**: every meaningful
path is either exercised by a test that asserts the right
BEHAVIOUR, or recorded as deliberately untested. The line/branch
percentage is the canonical *proxy* for that outcome — a proven
default, cheap to measure in CI — not the outcome itself. Clearing
the gate is necessary, not sufficient.

The two failure modes a green number hides:

- **Tautological coverage.** A test that imports a re-export shim
  and asserts the import succeeded touches the line without
  asserting any behaviour. It moves the %; it buys no confidence.
  Coverage of a line is evidence only when an assertion on that
  line's *effect* would fail if the line broke.
- **Mocking the SUT to hit the gate.** If the only way to make a
  unit reach the gate is to mock the function under test (the §14
  anti-pattern), that is a **design finding, not a coverage
  problem** — the code is not testable by behaviour. Report it as a
  design defect against [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md)
  §1.4 ("external side effects MUST be injectable"); fix the seam,
  don't bolt a tautological test onto untestable code to colour the
  line green.

**"No additional test needed" is a first-class outcome — with a
recorded rationale.** A path may be legitimately exempt: a trivial
re-export shim covered by the impl's tests (§2), an internal dunder
helper exercised only through its public caller, a defensive
impossibility marked `pragma: no cover` (§9.3). Naming the path and
the rationale is conformant; it is the recorded equivalent of a
test, the same shape `non-adoption-is-valid` blesses for a declined
rule. A path that is *neither* behaviorally tested *nor* recorded
as exempt is the gap the gate cannot see — that, not a dip in the
percentage, is the real coverage defect.

---

## 10. Security tests (rule TS-10)

Every security MUST in [`SECURITY.md`](SECURITY_STANDARD.md) §1 has a pin
test. The shapes:

### 10.1 Negative-pattern grep

```python
# tests/unit/security/test_no_shell_true.py
import subprocess

def test_no_shell_true_in_src():
    """SC-02: subprocess MUST use argv lists with shell=False."""
    out = subprocess.run(
        ["grep", "-rn", "shell=True", "src/"],
        capture_output=True, text=True,
    )
    # Allow-list: if any legitimate use exists (none today), enumerate here.
    allow_list = set()
    findings = {line for line in out.stdout.splitlines() if line not in allow_list}
    assert not findings, f"shell=True found in src/:\n{chr(10).join(sorted(findings))}"
```

### 10.2 Positive-behaviour assertion

```python
# tests/unit/security/test_yaml_safe.py
import pytest
import yaml

from <app>.config._loader import _read_yaml

def test_loader_rejects_malicious_yaml(tmp_path):
    """SC-03: loader MUST use safe_load — refuse !!python tags."""
    f = tmp_path / "evil.yaml"
    f.write_text("!!python/object/apply:os.system [echo pwned]\n")
    with pytest.raises(yaml.YAMLError):
        _read_yaml(f)
```

### 10.3 Redaction pin

```python
# tests/unit/security/test_redaction_in_log_output.py
def test_password_in_extra_is_redacted(caplog, tmp_path):
    """SC-09 / LG-04: sensitive keys MUST be redacted before any sink."""
    setup_logging(level="DEBUG", fmt="json")
    LOG = logging.getLogger("test")
    LOG.info("connecting", extra={"password": "hunter2", "username": "alice"})
    # caplog captures the formatted record; assert the secret never appears.
    formatted = caplog.records[-1].getMessage() + str(caplog.records[-1].__dict__)
    assert "hunter2" not in formatted
    assert "[REDACTED]" in formatted or formatted.count("password") == 1
```

### 10.4 Audit-event pin

```python
# tests/unit/security/test_audit_events.py
def test_vault_read_emits_audit_record(caplog):
    """SC-17: vault read MUST emit audit=true at INFO."""
    vault = MyVault()
    with caplog.at_level("INFO"):
        vault.get("some-key")
    audit_records = [r for r in caplog.records if getattr(r, "audit", False)]
    assert any(r.event == "vault.read" for r in audit_records)
```

The full set of pins is the [`SECURITY.md`](SECURITY_STANDARD.md) §15
mechanical-enforcement matrix.

---

## 11. Test isolation (rule TS-03)

### 11.1 Process-global state

Several Python facilities are process-global. Tests that touch them
MUST save + restore:

| State | Save / restore pattern |
|---|---|
| `sys.excepthook` | `prior = sys.excepthook; ...; sys.excepthook = prior` |
| `threading.excepthook` | same |
| `logging.root.handlers` | iterate, close, remove; restore on teardown |
| `os.environ` | use `monkeypatch.setenv` / `monkeypatch.delenv` |
| `os.chdir` | use `monkeypatch.chdir(path)` |
| Pydantic settings cache | `Settings.model_rebuild()` if mutating model_config |
| OTel `TracerProvider` | session-scope fixture; per-test fixture clears the in-memory exporter |

### 11.2 HOME monkeypatching

Tests that touch any `~/`-derived path MUST monkeypatch HOME:

```python
@pytest.fixture
def home_isolated(tmp_path, monkeypatch):
    fake_home = tmp_path / "home"
    fake_home.mkdir()
    monkeypatch.setenv("HOME", str(fake_home))
    monkeypatch.delenv("XDG_CONFIG_HOME", raising=False)
    monkeypatch.delenv("XDG_STATE_HOME", raising=False)
    monkeypatch.delenv("XDG_DATA_HOME", raising=False)
    yield fake_home
```

This is why every `_default_X()` helper in `mgf.common` is a
function (rule **CF-04**) — module-load expansion captures the
parent process's `$HOME` once and breaks the per-test isolation.

### 11.3 Pin file for lazy-default helpers (CF §10.1)

Every `_default_X()` helper MUST be pinned in a single consolidated
test file:

```python
# tests/unit/test_default_path_resolution.py
class TestLazyHomeResolution:
    def test_log_path_follows_home_monkeypatch(self, tmp_path, monkeypatch):
        fake_home = tmp_path / "alt-home"
        monkeypatch.setenv("HOME", str(fake_home))
        from <app>.observability.logging_setup import _default_log_path
        resolved = _default_log_path()
        assert str(resolved).startswith(str(fake_home))

    def test_state_dir_follows_xdg_state_home(self, tmp_path, monkeypatch):
        # ... etc, one per helper
```

---

## 11.5 Failure-injection tests (the rollback recipe)

A defensive-handler that catches an exception, performs cleanup,
and re-raises (or swallows) MUST have a test where the **cleanup
step itself raises**. The "happy path under failure" test
(EH-10 / TS-09 "raising observer") proves the loop continues; the
"failure under failure" test proves the cleanup chain doesn't
silently fall apart when the cleanup is the thing that broke.

Reference shape — `mgf-common`'s bootstrap rollback (`RA-02`,
[`tests/unit/test_lifecycle.py::TestRollbackChainFailureInjection`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/tests/unit/test_lifecycle.py)):

```python
def test_rollback_continues_when_a_rollback_step_raises(
    monkeypatch: pytest.MonkeyPatch,
) -> None:
    """RA-02: bootstrap()'s LIFO rollback chain MUST continue
    even when a rollback step itself raises. Otherwise a
    half-rolled-back process is left in an inconsistent state
    no caller can recover from."""

    # Patch the OTel-unwire step (one specific rollback action)
    # to raise. The earlier rollback steps must still run.
    def boom() -> None:
        raise RuntimeError("synthetic-rollback-failure")

    monkeypatch.setattr(_lifecycle, "_unwire_otel", boom)

    # Force the bootstrap to fail at a step AFTER OTel is wired,
    # so the rollback walks back through OTel.
    monkeypatch.setattr(
        _lifecycle, "_install_excepthook", lambda *_: 1/0,
    )

    with pytest.raises(BootstrapError):
        bootstrap(app_name="t", app_version="0.0")

    # The defining assertion: the EARLIER rollback steps still
    # ran, even though OTel-unwire raised mid-rollback.
    assert logging.root.handlers == _original_handlers
    assert _GLOBAL_CONTEXT is None
```

Three patterns this catches:

| Failure mode | Without the test | With the test |
|---|---|---|
| Cleanup raises → outer raises something else | Caller sees the cleanup exception, not the original | Cleanup exception is suppressed / logged; original propagates |
| Rollback step N raises → steps N+1..M don't run | Process left with steps 1..N undone, N+1..M wired | All steps still run; the rollback step's failure is logged |
| Two nested defers, inner raises | `except` chain decides which fires; usually wrong one | Explicit test pins the contract |

The pattern: **inject a raise at the cleanup site, assert the
outer invariants still hold**. Use `monkeypatch.setattr` to swap
the specific function for one that raises; don't try to mock the
whole rollback machinery.

---

## 12. Smoke tests against real backends (rule TS-15)

A smoke test against the real backend (libvirt, real SSH host, real
HTTP API) is the reality check that mocks alone cannot provide. The
rules:

1. **Marker-gated.** Pin behind `@pytest.mark.smoke` or
   `@pytest.mark.real_backend`. Default `pytest` invocation does
   not run them.
2. **Read-only by default.** A smoke test that observes the live
   system is safe to run regularly. A smoke test that mutates
   needs separate gating + a scratch environment.
3. **Self-contained cleanup.** A mutating smoke test MUST clean up
   on success AND on failure. Use a `try / finally` or a fixture
   with teardown.
4. **Idempotent.** Running the same smoke test twice MUST produce
   the same result. The first run leaves no residue that affects
   the second.
5. **Documented.** The test docstring states what real surface it
   touches and what permissions the runner needs.

### 12.1 Example shape

```python
@pytest.mark.smoke
@pytest.mark.skipif(
    not os.environ.get("<APP>_REAL_BACKEND_URI"),
    reason="set <APP>_REAL_BACKEND_URI to run real-backend smoke",
)
def test_real_backend_lists_resources():
    """Read-only smoke against the real backend.

    Required perms: read-only on resources.list().
    Run via: <APP>_REAL_BACKEND_URI=qemu:///system pytest -m smoke
    """
    provider = LibvirtProvider(uri=os.environ["<APP>_REAL_BACKEND_URI"])
    handles = provider.list()
    assert isinstance(handles, list)
```

### 12.2 CI

A separate CI job invokes the smoke suite against a sacrificial
environment. It MUST run after the main suite passes (no point
spending real-backend time on a PR that fails unit tests).

---

## 13. Layer markers — the canonical set (rules TS-19, TS-20, TS-21)

The mental-model pyramid in §1 is conceptual; this section is
**operational**. It codifies the four canonical test layers that
every project SHOULD adopt, the directory shape, the marker
registration, and the runtime budgets that keep each layer fast
enough to be useful.

### 13.1 The four layers

| Layer | Directory | Marker | Runtime budget | What it catches |
|---|---|---|---|---|
| **smoke** | `tests/smoke/` | `@pytest.mark.smoke` | < 100 ms total | Packaging breaks: missing `__init__.py` import, typo in `__all__`, missing optional-extra dependency, broken wheel install |
| **unit** | `tests/unit/` (mirrors `src/`) | (none — default) | < 10 ms per test | Per-component logic regressions; isolated, no I/O, no network |
| **integration** | `tests/integration/` | `@pytest.mark.integration` | < 5 s per test | Cross-module wiring; multiple subsystems composed; in-process only (no real third-party daemons — TS-02 still applies) |
| **e2e** | `tests/e2e/` | `@pytest.mark.e2e` | < 30 s per test | Full bootstrap-to-output chains; verifies the contract that consumers actually exercise |

Two related layers from older rules:

- **slow** (`@pytest.mark.slow`, TS-18) — orthogonal to the four
  above; applies to any test (unit / integration / e2e) that
  individually exceeds 1 second. Default `pytest` runs exclude
  it; a separate "slow" job picks it up.
- **real-backend smoke** (TS-15) — the §12 "smoke against real
  backends" pattern is distinct from this section's `smoke`
  marker. The §12 pattern is read-only against live external
  systems (Slack workspace, Sentry org, etc.) and uses a
  separate marker (commonly `live` / `realbackend`) — NOT the
  TS-19/20/21 `smoke` marker which is "post-install sanity, no
  external dependencies."

### 13.2 Directory shape (rule TS-20)

```
tests/
├── conftest.py              # repo-wide fixtures
├── unit/                    # mirrors src/ 1:1 (TS-01)
│   ├── conftest.py
│   ├── observability/
│   │   ├── test_redaction.py
│   │   └── test_logging_setup.py
│   ├── process/
│   │   └── test_run.py
│   └── …
├── integration/
│   ├── conftest.py          # cross-module fixtures
│   └── process/
│       ├── test_multi_service.py
│       └── test_concurrent_run.py
├── e2e/
│   ├── conftest.py          # bootstrap fixtures
│   ├── test_process_redaction_e2e.py
│   └── test_process_correlation_e2e.py
└── smoke/
    ├── conftest.py          # rare; smoke needs no fixtures
    └── test_<package>_smoke.py
```

The mirror-`src/` rule (TS-01) applies only inside `tests/unit/`.
Integration / e2e / smoke organise by **what's being verified**,
not by source path — a single e2e file may exercise five source
modules at once.

### 13.3 Marker registration (rule TS-19)

Every marker MUST be declared in `pyproject.toml`:

```toml
[tool.pytest.ini_options]
addopts = ["--strict-markers", "--strict-config"]
markers = [
    "smoke: minimal post-install sanity (fast; no fixtures)",
    "e2e: end-to-end test exercising bootstrap + multiple subsystems",
    "integration: cross-module integration test (slower; may launch real subprocesses)",
    "slow: individually-slow test (>1s); excluded from default fast suite",
    # Add project-specific markers here (live/realbackend per §12, etc.)
]
```

With `--strict-markers` (TS-05) on, an unregistered marker fails
the test run — typos can't silently disable a layer.

### 13.4 Per-test marker (the canonical pattern)

For a whole file in a layer, declare once at module top:

```python
# tests/e2e/test_process_redaction_e2e.py
import pytest

pytestmark = pytest.mark.e2e

class TestRedactionChain:
    def test_bootstrap_run_redacts_secret(self): ...
```

For one-off cross-layer tests, decorate the function/class:

```python
class TestSomething:
    @pytest.mark.slow
    def test_warm_cache_eventually_converges(self): ...
```

### 13.5 Running a layer

```bash
pytest -m smoke              # 7 tests in mgf-common; <100ms
pytest -m e2e                # 8 tests; <2s
pytest -m integration        # 5 tests; <5s
pytest                       # everything (excluding marked-slow if so configured)
pytest -m "not slow"         # explicit fast-only
```

The `-m smoke` invocation is the canonical "did the wheel install
correctly?" CI pre-flight (rule TS-21). A failed smoke run aborts
before spending budget on the heavier layers.

### 13.6 The smoke layer (rule TS-21)

A library's smoke layer answers one question: **if I install the
wheel from PyPI, can I import it and call its happy-path
functions?** It is the cheapest possible insurance against
packaging breaks.

The minimum viable smoke test for a library:

```python
# tests/smoke/test_<lib>_smoke.py
import pytest

pytestmark = pytest.mark.smoke


def test_top_level_imports() -> None:
    """Every name in __all__ is reachable."""
    import mylib
    for name in mylib.__all__:
        assert hasattr(mylib, name), f"{name!r} declared in __all__ but unreachable"


def test_happy_path() -> None:
    """The shortest path through the library's main entry function works."""
    import mylib
    result = mylib.do_thing("trivial input")
    assert result.ok
```

That's it. **No fixtures, no bootstrap, no extras, no network.**
A smoke test that takes more than 100 ms is doing too much.

CLI / desktop applications without a wheel surface MAY skip the
smoke layer (TS-21) — there's nothing to install to break.

### 13.7 The e2e layer

E2E tests verify the contract that consumers exercise: the full
chain from bootstrap → wired-subsystem → external observable.
They use real `mgf.common.bootstrap` (or the project equivalent)
to ensure the integration is wired correctly:

```python
# tests/e2e/test_redaction_e2e.py
import pytest

pytestmark = pytest.mark.e2e


@pytest.fixture
def bootstrapped_test_app(tmp_path, monkeypatch):
    monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path))
    monkeypatch.setenv("HOME", str(tmp_path))
    from mgf.common import bootstrap_for_test
    ctx = bootstrap_for_test()
    try:
        yield ctx
    finally:
        ctx.shutdown()


def test_full_redaction_chain(bootstrapped_test_app, caplog):
    """bootstrap → run() → child secret → assert scrubbed in BOTH capture + log."""
    from mgf.common.process import run
    result = run([sys.executable, "-c", "print('Bearer abc123…')"])
    assert "[REDACTED]" in result.stdout         # captured was scrubbed
    assert all("abc123" not in r.message for r in caplog.records)  # logs too
```

E2E tests are slower (real bootstrap, real I/O paths) but catch
integration breaks that no individual unit test can — for
example, "the streamer's `redact()` call passes the wrong
argument type to the live `Redactor` wired by bootstrap."

### 13.8 The integration layer

Integration tests sit between unit and e2e. Multiple modules
compose, but external I/O still goes through mocks (TS-02):

```python
# tests/integration/process/test_multi_service.py
import pytest

pytestmark = pytest.mark.integration


def test_two_services_lifo_teardown_via_exitstack():
    """ExitStack semantics + Service.stop() composition."""
    from contextlib import ExitStack
    from mgf.common.process import Service, PortReady
    with ExitStack() as stack:
        a = stack.enter_context(Service([...], ready=PortReady(...)))
        b = stack.enter_context(Service([...], ready=PortReady(...)))
        # both alive
    # both stopped, b before a
```

Integration is the right home for tests that span 2-3 modules
and verify a pattern (multi-service, concurrent calls, error
propagation across module boundaries).

### 13.9 Decision tree — which layer for a new test?

```
  ↓ Does the test verify a single function / class / module?
  └─ Yes → unit (tests/unit/, mirroring src/)

  ↓ Does the test verify two-or-more modules composed?
  └─ Yes (no real bootstrap) → integration (tests/integration/)

  ↓ Does the test require mgf.common.bootstrap() (or equivalent)?
  └─ Yes → e2e (tests/e2e/)

  ↓ Does the test verify "this wheel imports + happy path"?
  └─ Yes → smoke (tests/smoke/)

  ↓ Does the test take > 1 s?
  └─ Add @pytest.mark.slow on top of its layer marker.

  ↓ Does the test hit a real external system (Slack, Sentry, …)?
  └─ Use the §12 pattern with a separate marker (live/realbackend).
```

### 13.10 Layer counts (illustrative, non-binding)

A per-layer count + runtime table is a useful project artifact —
smoke (<100 ms), a wide unit base (the bulk), a thin integration
layer, a handful of e2e. Concrete counts are point-in-time and
live in the code, not in this standard. `mgf-common`'s live
breakdown — and a template consumers can copy for their own
coverage map — is
[`docs/internal/TEST_COVERAGE.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/internal/TEST_COVERAGE.md).

### 13.11 Extended marker taxonomy (the federation supervisor superset, TS-22..TS-28 era)

The four layers in §13.1 are the **organisational** axis (test
directories, where things live). The federation test-supervisor
work (TS-22..TS-28, §19) adds a **characteristics** axis — markers
that describe what a test *is doing*, orthogonal to where it
lives. The supervisor's tier model (TS-22) consumes this extended
set.

**The thirteen-layer canonical taxonomy** (originated in `mgf-hrb/docs/testing/TAXONOMY.md`,
federation-adopted via this section as of mgf-common 0.40.0):

| Marker | Family | Tier | One-line definition |
|---|---|---|---|
| (unmarked) / `unit` | correctness@unit | tier-1 | Does this one function do what it claims? Sub-ms. No I/O. |
| `functional` | correctness@unit | tier-1 | Does this one feature produce the right user-visible outcome? Sub-second. In-process. |
| `integration` | correctness@unit | tier-1 | Do these N components together behave correctly? Sub-second; may use threads. |
| `contract` | correctness@surface | tier-1 | Does the public surface match what consumers depend on? Pins exact wire shape. |
| `smoke` | correctness@surface | tier-0 | Is the freshly-installed wheel minimally alive? Post-install sanity only — imports + `__all__` reachable + shortest happy path; no fixtures, no network, no real backend. <100 ms total (rule TS-21). |
| `e2e` | correctness@surface | tier-3 | Does a real consumer over a real transport produce the right outcome? Seconds per run. |
| `use_case` | correctness@surface | tier-2 | Does the real consumer scenario (named in a FEEDBACK paper) work end-to-end? |
| `property` | correctness@adversity | tier-1 | Does this invariant hold for **all** valid inputs? (Hypothesis-style; rule TS-13.) |
| `fuzz` | correctness@adversity | tier-1 | Can hostile / random / malformed input crash the system? Asserts safety only. |
| `concurrency` | correctness@adversity | tier-2 | Do the locks / atomics / serialization invariants hold under real parallel execution? |
| `stress` | correctness@adversity | tier-3 | Does the system stay correct + bounded under sustained high load? Long-running. |
| `perf` | correctness@time | tier-3 | Does this op meet its latency / throughput budget? (Rule TS-14; PERFORMANCE.md PF-* governs.) |
| `conformance` | correctness@time | tier-2 | Do the design's named guarantees hold? Release-gated. |
| `regression` | correctness@time | tier-1 | Pin for a specific past bug. Docstring cites the fixing commit. |

**Marker → tier mapping** (the canonical default; consumers MAY override per-project):

```python
# from mgf.common.testing import MARKER_TIER
MARKER_TIER = {
    "smoke": 0,
    "contract": 1, "property": 1, "regression": 1, "fuzz": 1,
    "conformance": 2, "use_case": 2, "concurrency": 2,
    "perf": 3, "stress": 3, "e2e": 3,
}
```

A test inherits its tier from its **strongest** (highest) marker;
an explicit `@pytest.mark.tier(N)` overrides; an unmarked test is
implicitly tier-1 (unit). The supervisor's auto-tier heuristic
bumps a test up one tier if its last-5 median runtime is ≥5× the
tier's typical budget (so a slow unit migrates to integration's
tier automatically).

**Relationship to §13.1's four layers:** the four layers are
DIRECTORIES (where the test file lives). The thirteen markers are
TAGS (what the test does). A test in `tests/integration/` typically
carries the `integration` marker (or stays unmarked and is
implicitly tier-1); the same file may ALSO carry `contract` or
`property` markers if it's pinning a wire-format contract or
asserting an invariant. Markers compose; directories don't.

**Backward compatibility with TS-18's `slow` marker:** `slow` stays
registered + accepted for one minor cycle as a deprecation alias
that maps to tier-2 (the equivalent budget). New projects SHOULD
use the explicit marker (`use_case` / `conformance` / `concurrency`)
that describes WHY the test is slow rather than just THAT it is.
Removed in v2.0.

---

## 14. Anti-patterns (forbidden)

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| `time.sleep(0.5)` to wait for a thread/worker to make progress | Flaky | Explicit sync (`worker.wait()`, `qtbot.waitSignal`, `queue.get()`, `Event.wait`) |
| Tests that depend on collection order | Brittle; pytest may reorder | Each test independent; use fixtures for shared setup |
| `assert result` (without context) | Failure message says nothing | `assert result, f"expected non-empty, got {result!r}"` |
| Pickling a fixture across tests for "speed" | Cross-test state leakage | Re-build per test; use session scope only when justified |
| `print(...)` for debugging in committed test code | Pollutes test output | Use `caplog` to assert log content; remove debug prints |
| `try: do_thing(); except: pass` in a test | Hides real errors | Let exceptions propagate; pytest reports them |
| Catching `AssertionError` in a test helper | Defeats the test | Helpers raise; test functions assert |
| Mocking the SUT (system under test) | Tests the mock, not the code — and signals the code is testable only by mocking itself (a **design** finding, not a coverage gap; see §9.5 + DESIGN_PRINCIPLES §1.4) | Mock collaborators; never mock the function you're testing. If you can't, fix the seam so the side effect is injectable |
| `monkeypatch` of an internal that the test asserts on | Tautological | Patch at the boundary (network, time, random); assert behaviour, not internals |
| One test asserting many unrelated things | Failure tells you nothing about which thing broke | One concept per test; multiple tests if needed |
| Skipping a test because it's "broken" without a tracking issue | Bit-rot | `@pytest.mark.skip(reason="...", strict=False)` with link to issue |
| Testing private helpers directly when the public surface is reachable | Couples tests to implementation | Test through the public API; private tests only when impossible otherwise |
| `rm -rf tmp_path/...` in a test | Leakage if test fails before cleanup | `tmp_path` is auto-cleaned; trust pytest |

---

## 15. Self-review checklist (every PR)

Before requesting review, the author MUST be able to answer YES:

1. Does every new code path have a test that exercises it?
2. Does every new typed exception have a unit test?
3. Does every new boundary (HTTP, SSH, subprocess) have a translation
   test (success + each typed error + unknown vendor error)?
4. Does every new config field have a test for default + override
   precedence + validation rejection?
5. If touching a `_default_X()` helper: did I add a row to
   `tests/unit/test_default_path_resolution.py`?
6. If touching async code: is there a cancellation test?
7. If touching GUI: did I exercise both the success path AND the
   untyped-exception path?
8. If touching crypto / auth / vault: did I add a security pin test?
9. If adding a performance-sensitive code path: is there a benchmark
   asserting the budget?
10. Did I run the full unit suite locally? Does coverage stay ≥80%?

---

## 16. Mechanical enforcement matrix

| Rule | Mechanical check |
|---|---|
| TS-01 mirror | A small `tests/test_mirror.py` script walking `src/` and asserting every file has a mirror or appears in a documented allow-list |
| TS-02 no real I/O | `import-linter` contract forbidding `socket`, `subprocess` (outside the wrapper), real-backend SDKs from inside `tests/unit/`; CI runs unit suite without the real backend installed |
| TS-03 isolation | Pytest plugin `pytest-randomly` shuffles test order on every run; persistent failures expose order dependencies |
| TS-04 filterwarnings | `pyproject.toml` `[tool.pytest.ini_options]` `filterwarnings = ["error"]` |
| TS-05 strict markers/config | same |
| TS-06 coverage | `--cov-fail-under=80` on every CI job |
| TS-07 typed exception tests | code review + dedicated `test_exceptions_base.py` ensuring every `__all__` entry has a test |
| TS-08 translation tests | code review at provider PR |
| TS-09 best-effort observer tests | code review |
| TS-10 security pins | full set in `tests/unit/security/` per [`SECURITY.md`](SECURITY_STANDARD.md) §15 |
| TS-11 GUI worker tests | code review at worker PR; the `core_worker` test file is the template to copy |
| TS-12 async tests | `pytest-asyncio` plugin + code review |
| TS-13 property tests | code review at parser/serialiser PRs |
| TS-14 perf benchmarks | dedicated `tests/unit/perf/` directory + CI runs them in a separate fast job |
| TS-15 smoke gating | `@pytest.mark.smoke` + CI runs them in a separate slow job |
| TS-16 self-explanatory failures | code review |
| TS-17 conftest hierarchy | code review; lint via `ruff` rule disallowing cross-package fixture imports |
| TS-18 slow marker | `pytest -m "not slow"` is the default; separate CI step runs `-m slow` |
| TS-22 tiered runs | `mgf-test-supervisor --tier N` (or `--tier N-M`); marker → tier mapping in `mgf.common.testing.MARKER_TIER` |
| TS-23 out-of-process supervisor | `mgf.test_supervisor.supervisor` (published [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor) sibling) runs pytest as subprocess; `mgf-fed catch-up` audit verifies presence |
| TS-24 structured records | `mgf.common.testing.TestRecord` (importable schema, stays in mgf-common); `mgf.test_supervisor.records` (writer, in published sibling); JSON-schema-validate hook on `.gates/last-run/run.jsonl` is post-v1.x work |
| TS-25 quarantine | `mgf.test_supervisor.quarantine` (published sibling) + `.gates/quarantine.json`; review-only (no automatic enforcement that the policy is followed; if disabled the gate becomes noisier — operator pain is the enforcement) |
| TS-26 binary freshness | `tests/smoke/test_<binary>_freshness.py`; project-specific. N/A for mgf-common (pure-Python). Review-enforced on consumers with compiled artifacts. |
| TS-27 failure bundles | `mgf.test_supervisor.supervisor::_write_failure_bundle` (published sibling); verifiable post-hoc on a failed run (file exists at `.gates/last-run/failures/<id>.txt`). |

---

## 17. AS-IS in `mgf-common`

- **Test count** (illustrative, non-binding — the live count lives
  in the repo): a wide unit base under `tests/unit/`, full suite
  comfortably sub-second on a warm machine.
- **Mirror discipline** — every source module under `src/mgf/common/`
  has a corresponding test under `tests/unit/`.
- **`filterwarnings = ["error"]`** — set in `pyproject.toml`,
  drives the RotatingFileHandler-close discipline (rule TS-03).
- **Strict markers / config** — set.
- **Coverage** — CI gate at 80% (`--cov-fail-under=80`); the actual
  measured % (illustrative, non-binding) sits above the gate. The
  gate is deliberately at 80% rather than higher because
  lazy-imported sink code inflates the unreachable surface.
- **Provider mocks** — N/A in `mgf.common` itself (no third-party
  I/O dependencies); the canonical mock-provider pattern lives in
  vm-vmanager-app (`src/vmanager/providers/mock_provider.py` +
  `tests/unit/test_provider_base.py::EXPECTED_METHODS`).
- **Property tests** — none yet in `mgf.common`. **Gap (TS-13)** —
  the YAML round-trip in `config/_loader.py` and the redaction in
  `observability/redaction.py` are the obvious round-trip
  candidates.
- **Identity-isolation autouse fixture** —
  `tests/unit/observability/conftest.py` saves + restores the app
  identity around every test in that directory; `tests/unit/test_identity.py`
  lives outside the fixture's scope to verify placeholder defaults.
  This is the load-bearing test-isolation pattern; documented in
  [`docs/CLAUDE.md`](../claude/CLAUDE.md) Cross-cutting pattern §6.
- **Session-scoped TracerProvider** —
  `tests/unit/observability/conftest.py::_session_tracer_provider`
  installs the OTel TracerProvider once per session; per-test
  `in_memory_spans` fixture clears the exporter. (Rule TS-03 +
  `LOGGING.md` §11.)
- **Excepthook tests** — replace `sys.excepthook` with a no-op
  before installing the standard one, so pytest's prior hook
  doesn't capture the synthetic exception. Documented in
  `tests/unit/observability/test_excepthooks.py`.
- **Performance benchmarks** — none in `mgf.common`. **Gap (TS-14)**
  — LG-12 (≤100 µs per record) is asserted in vm-vmanager-app only
  (`vm-vmanager-app/tests/unit/test_logging_perf.py`); a thin upstream
  pin would catch regressions before they reach the consumer.
- **Security pins** — basic shape exists; full SC-* matrix not yet
  ported. **Gap (TS-10)** — add `tests/unit/security/` with the
  pins from [`SECURITY.md`](SECURITY_STANDARD.md) §15.

---

## 18. Cross-references

- The provider ABC pattern (rule DP-02) underlying the mock-provider
  test pattern: [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md) §1.2.
- Test patterns for error paths (typed exceptions, translation seams,
  best-effort observers): [`ERROR_HANDLING.md`](ERROR_HANDLING.md) §9.
- Test isolation for logging (RotatingFileHandler close,
  filterwarnings, JOURNAL_STREAM): [`LOGGING.md`](LOGGING.md) §11.
- Per-module pin tests for lazy-default helpers (CF-04 generalised):
  [`CONFIGURATION.md`](CONFIGURATION.md) §10.1.
- Security pin tests (SC-* mechanical-enforcement matrix):
  [`SECURITY.md`](SECURITY_STANDARD.md) §15.
- The federation test supervisor (TS-22..TS-28): §19 below.
- Test-mode marker taxonomy (the 13-layer superset): §13.11.
- Performance-budget discipline (perf marker; PF-01..PF-08):
  [`PERFORMANCE.md`](PERFORMANCE.md).
- Diagnosing test hangs / crashes: [`DIAGNOSING_HANGS.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/testing/DIAGNOSING_HANGS.md)
  (consumer-shippable playbook; sections 0-2, 4-7 are
  project-agnostic; §3 is broker-specific).

---

## 19. The federation test supervisor (TS-22..TS-28)

> **The four-line summary:** every Magogi CI run drives pytest as
> a **child process** of a small stdlib-only supervisor that
> records each test outcome as one JSON line, quarantines flakes
> automatically, and produces a single-file failure bundle per
> failed test. The bar is binding across the federation
> (TS-22..TS-28, L1 + L2; TS-28 is L2); the implementation lives in the
> published sibling library
> [`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor)
> (installable via dev extras — see §19.8). The canonical console
> script is `mgf-test-supervisor` (with `mtest` as an alias); the
> importable invocation form is `python -m mgf.test_supervisor`.
> `mgf-fed test` remains as a thin shim that delegates to
> `mgf.test_supervisor.__main__.main`.

### 19.1 Why a supervisor — what plain `pytest` cannot do

Plain `pytest`-from-CI has four blind spots:

1. **A pytest crash kills the reporter.** SIGSEGV in a C
   extension, an `os._exit(139)` from a broken native lib, a
   stack-overflow in a recursive fixture — pytest dies and CI
   shows "exit code 139" with no per-test attribution.
2. **A hang looks the same as "still running."** A deadlocked
   asyncio loop, a missed `Event.set()` in a thread fixture, a
   socket that never closes — pytest reports nothing until the CI
   wallclock timeout kicks in, often half an hour later.
3. **Failure context is scattered.** A failed test surfaces a
   traceback in one place, stdout in another, env in `pytest
   --collect-only` output, broker state in `dmesg`, history in
   nowhere — triage is a scrolling exercise.
4. **Flakes erode the gate.** A test that fails 1-in-10 forces
   developers into "re-run CI until green" — a habit that masks
   real regressions.

The supervisor closes all four — it is TS-23's canonical reference path, not its sole conformant one: any harness that demonstrably delivers the same four outcomes (structured crash/hang/signal capture, bounded hang + thread dump, single-file context, no bootstrap circularity) is conformant, recorded per [`non-adoption-is-valid`](principles/non-adoption-is-valid.md). The mechanism per blind spot:

| Blind spot | Mechanism (TS-#) |
|---|---|
| pytest crash | Run pytest in a subprocess + parse exit code + signal name; record `crashed` outcome (TS-23) |
| Hang | Per-chunk wall-clock timeout (default 5 min) + per-test pytest-timeout (default 30s) + faulthandler kernel safety net (60s) → record `hung` outcome with thread dump (TS-23 + TS-27) |
| Scattered context | Single `.gates/last-run/failures/<id>.txt` bundle per failure carrying traceback + I/O + process state + dmesg + env + history + chunk diagnostics (TS-27) |
| Flakes | Sliding-window quarantine: 3 fails in last 5 → quarantine; 3 passes under `--quarantine-attempt` → evict (TS-25) |

### 19.2 The four-tier budget model (TS-22)

```
tier-0 — Build-broken signal      ≤  0.10 s / test    suite budget ≤   1 s
tier-1 — Daily contract           ≤  0.30 s / test    suite budget ≤  30 s
tier-2 — Promise pins             ≤  2.00 s / test    suite budget ≤ 120 s
tier-3 — Long signal              ≤ 10.00 s / test    suite budget ≤ 1800 s
                                    (stress, perf, e2e)
```

Tests inherit their tier from their strongest marker (see §13.11);
explicit `@pytest.mark.tier(N)` overrides. The supervisor's
`--tier N` / `--tier N-M` / `--budget T` flags select tiers; with
`--budget T`, tier-0 is **always included** and the remaining
budget is filled via greedy knapsack (value = recency + 1; cost
= rolling-median duration).

**The CI cadence the four tiers buy you:**

| CI job | Invocation | What it catches |
|---|---|---|
| pre-commit | `mgf-test-supervisor --tier 0` (or `mgf-fed test -- --tier 0`) | Build-broken, import errors, smoke regressions. ≤5s. |
| per-PR | `mgf-test-supervisor --tier 0-2 --budget 5m` (or `mgf-fed test -- --tier 0-2 --budget 5m`) | Daily contract + promise pins; budget-bounded. |
| nightly | `mgf-test-supervisor --tier 0-3` (or `mgf-fed test -- --tier 0-3`) | Everything, including stress / perf / e2e. |

The canonical console script is `mgf-test-supervisor` (`mtest` is
an alias). When invoking through the `mgf-fed test` shim, the
`--` separator is required so the shim does not consume the
supervisor's own flags. The importable form `python -m
mgf.test_supervisor` is equivalent.

### 19.3 Per-test record schema (TS-24)

One JSON line per test outcome to `.gates/last-run/run.jsonl`.
Schema mirrors `mgf.common.observability.DoctorFinding` so
downstream tooling (a future Magogi dashboard, Slack relay,
status badges) uses one parser across every federation project:

```json
{
  "test_id": "tests/unit/foo.py::TestBar::test_baz",
  "outcome": "passed|failed|error|skipped|hung|crashed|quarantined",
  "severity": "OK|WARN|FAIL|SKIP",
  "duration_ms": 123,
  "tier": 1,
  "markers": ["contract", "regression"],
  "code": "AssertionError|WALL_CLOCK_HUNG|PROCESS_CRASHED|...",
  "summary": "one-line headline (nullable)",
  "detail": "multi-line traceback (nullable)",
  "why": "captured-context dump (nullable)",
  "remediation": "known fix or pointer (nullable)",
  "captured_stdout": "last 4 KB",
  "captured_stderr": "last 4 KB",
  "run_id": "run-20260526T011104278Z",
  "ts_iso": "2026-05-26T01:11:37.906Z",
  "schema_version": "1.0.0"
}
```

The importable type is `mgf.common.testing.TestRecord`. Consumers
that parse `run.jsonl` MUST go through that dataclass — DO NOT
hand-roll a parser. Schema-version evolution follows AP-04 (one
MINOR cycle of deprecation if a field is renamed).

### 19.4 The per-failure detail bundle (TS-27)

Every failed / hung / crashed test produces
`.gates/last-run/failures/<safe-test-id>.txt`. The single-file
discipline matters: a triager opens ONE file and has the whole
diagnostic picture in front of them.

**Mandatory sections, in order:**

```
=== TEST ===
tests/unit/foo.py::TestBar::test_baz

=== OUTCOME ===
failed (AssertionError)
chunk wall-time: 0.18s
chunk exit code: 1
chunk signal: -

=== TRACEBACK ===
(deduplicated to the failing test)

=== STDOUT (last 4 KB) ===
...

=== STDERR (last 4 KB) ===
...

=== PROCESS STATE ===
broker pid 12345
wchan: futex_wait_queue_me
threads: 3 (main, worker-0, worker-1)

=== DMESG TAIL ===
(last 30 lines; best-effort; "no dmesg access" if not root)

=== ENV ===
python 3.13.12
mgf-common 0.40.0 at /.../site-packages/mgf
git: main @ a1b2c3d
toolchain: (project-specific; ziglang 0.16 for mgf-hrb; none for mgf-common)

=== HISTORY (last 5 outcomes) ===
2026-05-26T01:11:37Z  failed   AssertionError
2026-05-26T00:42:12Z  passed
...

=== NEXT STEPS ===
See docs/testing/DIAGNOSING_HANGS.md
```

### 19.5 The auto-quarantine layer (TS-25 — L2)

A test that fails 3 times in its last 5 outcomes is automatically
added to `.gates/quarantine.json` and skipped on subsequent
default runs. Quarantined tests appear in `mgf-test-supervisor
--quarantine` listings + are flagged in the per-run summary
("3 quarantined tests skipped").

To attempt eviction: `mgf-test-supervisor --quarantine-attempt`.
The flag includes quarantined tests in the run AND tracks
consecutive passes. 3 consecutive passes evict the test from
quarantine.

State persistence uses atomic write (temp + rename) so a
crash mid-write doesn't corrupt the file. The state is
git-ignored (it's CI-environment local).

### 19.6 Reference implementation

The supervisor is a published sibling library:
[`mgf-test-supervisor`](https://codeberg.org/magogi-admin/mgf-test-supervisor)
(stdlib-only, importable as `mgf.test_supervisor`). The canonical
console script is `mgf-test-supervisor`; `mtest` is shipped as an
alias; `python -m mgf.test_supervisor` is the importable
invocation form. Repo-root discovery walks up from CWD looking
for `pyproject.toml`; override via the
`MGF_TEST_SUPERVISOR_REPO_ROOT` env var.

| Module | Responsibility |
|---|---|
| `mgf.test_supervisor.supervisor` | Chunked pytest runs, hang/crash detection, per-failure bundle writer |
| `mgf.test_supervisor.records` | `TestRecord` writer (schema imported from `mgf.common.testing.TestRecord`) |
| `mgf.test_supervisor.quarantine` | Sliding-window flake state machine |
| `mgf.test_supervisor.tiers` | Tier definitions + auto-tier heuristic |
| `mgf.test_supervisor.selection` | Budget knapsack |
| `mgf.test_supervisor.durations` | Per-test rolling 5-run history |
| `mgf.test_supervisor.diagnostics` | Failure-context capture (process state, dmesg, env) |
| `mgf.test_supervisor.self_check` | Supervisor verifies its own pass/fail/hang/crash detection |

The schema types (`TestRecord`, `MARKER_TIER`, `SCHEMA_VERSION`,
…) deliberately stay in `mgf.common.testing` — they're the wire
contract every downstream parser depends on, not the runtime
supervisor. Splitting writer (in `mgf-test-supervisor`) from
schema (in `mgf-common`) keeps the dependency direction clean:
the supervisor depends on the schema, not the other way around.

#### Distribution

The supervisor ships as the published sibling `mgf-test-supervisor`;
the federation consumes it uniformly (no sibling carries an in-tree
copy). PyPI publication is deferred — consumers install via git+ssh:

```
uv add --dev "mgf-test-supervisor @ git+ssh://git@codeberg.org/magogi-admin/mgf-test-supervisor.git"
```

For up-to-date adoption instructions, see the published library's
README.

### 19.7 Per-test perf attribution (TS-28 — L2)

Tier budgets (§19.2) and `duration_ms` (§19.3) tell you *which* test
is slow. **TS-28 tells you why** — at two granularities, captured
in-process and surfaced out-of-process so the stdlib-only supervisor
never has to import the code it's profiling (TS-23 holds).

**Two granularities.**

- **Function-level** — a per-test `cProfile`, reported as the top-N
  frames by cumulative time. It covers both the test body and the
  code under test, so "this test spends 38 ms in `select.poll`"
  falls straight out without any code change.
- **Section / log-level** — wrap a unit of work in `perf_span`:

  ```python
  from mgf.common.testing import perf_span

  def test_checkout_flow(client):
      with perf_span("seed fixtures"):
          seed(client)
      with perf_span("POST /checkout"):     # nestable
          resp = client.post("/checkout")
      assert resp.status_code == 200
  ```

  `perf_span` is **nestable** (records a `depth`), **fail-open**, and
  costs a single context-var read when capture is off — cheap enough
  to leave wrapped around hot paths in *production* code, not just
  tests. Wrap the block that emits a log line and you get the
  wall-time attributed to that log point.

**How it flows (the writer/reader split).** When the supervisor runs
with `--perf` it sets `MGF_PERF=1` + `MGF_PERF_DIR=<run>/perf` in the
pytest subprocess env. mgf-common's pytest plugin (the **writer**)
profiles each test's call phase, collects its `perf_span` samples,
and writes one JSON artifact per test (`{test_id, total_ms,
functions[], spans[]}`) — the `test_id` is carried *inside* the file
so indexing is by content, not filename. `mgf.test_supervisor.perf`
(the **reader**) ingests them and surfaces perf in three places:

1. **`run.jsonl`** — the `TestRecord.perf` field (schema ≥ `1.1.0`).
2. **The TS-27 failure bundle** — a `perf attribution` section, so a
   slow/hung failure shows its own call breakdown inline.
3. **`.gates/last-run/perf-summary.md`** — a run-level rollup:
   slowest tests by wall-time + hottest functions by **self-time**
   summed across the whole run (self-time, not cumulative, so a
   function isn't double-counted via its callers).

**Invariants (all enforced):** capture is **opt-in** (zero cost off);
**fail-open** (a profiler/IO error never changes an outcome);
**bounded** (top-N functions via `--mgf-perf-top-n`, capped span
count); and it **does not record durations into the baseline** the
budget selector + auto-tier read — a profiled run is inflated by
cProfile overhead, so it's treated as diagnostic, never as a timing
sample.

**Invocation.**

```sh
mtest --perf                 # whole selection, with perf capture
mtest --perf --tier 0-1      # compose with tiering / budget as usual
pytest --mgf-perf --mgf-perf-dir .gates/last-run/perf   # writer alone, no supervisor
```

Reference: writer `mgf.common.testing.perf_span` + the `mgf_common`
pytest plugin (`--mgf-perf`); reader `mgf.test_supervisor.perf`.

### 19.8 Implementing TS-22..TS-28 in a new project

For a new federation sibling or consumer:

```
[ ] Add the supervisor to your dev dependencies:
      uv add --dev "mgf-test-supervisor @ git+ssh://git@codeberg.org/magogi-admin/mgf-test-supervisor.git"
[ ] Add to pyproject.toml [tool.pytest.ini_options]:
      timeout = 30
      timeout_method = "thread"
      faulthandler_timeout = 60
      markers = [ ... per §13.11 ... ]
    (See mgf-common's pyproject.toml for the canonical block;
    `pytest-timeout` is a transitive dependency of
    mgf-test-supervisor, no separate add needed.)
[ ] Add .gates/ to .gitignore.
[ ] Run: mgf-test-supervisor --self-check   (must exit 0)
[ ] Run: mgf-test-supervisor --tier 0       (smoke must pass)
[ ] Document conformance in MGF_STANDARDS_CONFORMANCE.md:
    TS-22..TS-28 = ✅ adopted (or 🚫 with rationale; TS-28 is L2 —
    L1 projects MAY mark it deferred).
```

The published library's README is the authoritative adoption
guide; this checklist is the standards-level summary. The
`mgf-fed test` shim and the `python -m mgf.test_supervisor`
invocation form are equivalent to the canonical
`mgf-test-supervisor` console script.

For projects that prefer NOT to adopt (small CLIs, scripts):
TS-22..TS-28 carry an explicit decline-with-rationale path —
declare "🚫 TS-2X declined — single-file CLI, no CI sophistication
warranted" in your conformance tracker. The federation's "small
consumer" carve-out applies.

---

*End of `TESTING.md`.*
