# Supervisor design — how `mtest` works

> The internals doc `mtest --help` points at. Distilled from the
> module docstrings (which stay the source of truth for per-function
> detail); this page is the 5-minute architecture tour.

## Why outside pytest

A pytest plugin lives *inside* pytest. If pytest crashes (interpreter
deadlock, segfault in a C extension, a hook that infinite-loops), the
plugin dies with it. The supervisor is a separate process that **runs
pytest as a subprocess**, so a pytest crash is just one more outcome
to record. Per TS-23 the supervisor is **stdlib-only at runtime** —
it must run even when the project's venv is half-broken, so it must
not depend on anything that might also be broken (schema constants
shared with `mgf.common.testing` are *vendored* and byte-pinned by
the two-sided alignment suites, never imported).

## The run pipeline

1. **Collect — one pytest spawn.** `collect_tests` runs
   `pytest --collect-only -q --no-header` with a tiny marker-dump
   plugin injected via a temp dir on `PYTHONPATH` (`-p
   _mgf_ts_collect_plugin`). The plugin writes `[[nodeid,
   [markers]], ...]` JSON at `pytest_collection_finish`, so ONE spawn
   yields node-ids *and* real markers, in pytest's own collection
   order. (Pre-Wave-2: 1 + 13 per-marker collect-only spawns;
   measured ~14x slower, and the stdout-regex parse under-collected
   parametrized ids.) If the dump is missing, collection falls back
   to parsing the `-q` listing with empty markers — degraded tiers,
   never a blocked run.
2. **Select.** `selection.select_tests` applies the tier filter
   (`--tier`), the quarantine skip, and the greedy budget knapsack
   (`--budget`; tier-0 always included). Every excluded test is
   accounted for in the printed `SelectionReport`. An **empty
   selection on an explicit `--tier` run exits 2 by default**
   (`--require-nonempty` semantics; `--allow-empty` opts out) — a
   gate that ran nothing must never look green.
3. **Run in chunks.** One pytest invocation per chunk of 50 tests —
   amortises pytest startup without losing crash isolation (a crash
   takes down at most one chunk; subsequent chunks still run). Each
   chunk gets `--junit-xml=<tmp>` for structured per-test outcomes.
4. **Record.** One `TestRecord` JSON line per test, appended to
   `.gates/last-run/run.jsonl` (schema `mgf.common.testing.
   TestRecord`, `SCHEMA_VERSION 1.1.0`). Failures also get a
   per-test failure bundle (below).

## Tiers and markers

Markers say what KIND of test something is (taxonomy, twelve
canonical markers); tiers say how much TIME it may cost (budget,
0..3). `tier_for_markers` is highest-wins with an explicit
`override` arg matching `mgf.common.testing.effective_tier`
semantics exactly. Unmarked tests are tier-1. **Auto-tier** promotes
a test one tier when its observed mean duration exceeds 5x its
tier's `per_test_seconds` (5+ observations), starting from the
test's real marker-derived tier — an `integration` test (tier-2)
can be promoted to tier-3. Never demotes below the marker floor.

## Outcome classification

Per test, from the chunk's JUnit XML when present
(`passed` / `failed` / `error` / `skipped`). Tests **missing from
the XML** are classified from chunk evidence:

- Chunk hit the outer wall-clock budget (per-test timeout x chunk
  size + grace) → `hung` (`WALL_CLOCK_HUNG`). The kill ladder is
  SIGTERM → grace → SIGKILL, signalled to the whole process group;
  every wait is bounded, so a leaked grandchild holding the pipes
  can never wedge the run.
- The **internal pytest-timeout heuristic** → `hung`
  (`PYTEST_TIMEOUT_FIRED`). pytest-timeout's thread method dumps
  stacks then calls `os._exit(1)`, skipping XML writing. All three
  legs must hold: wall-clock >= **0.95x** the per-test timeout, exit
  code != 0 with no fatal signal, and **no output in the final
  `HANG_SILENCE_S` (5s) window** — output activity is tracked by a
  streaming drain of the child's pipes, and the verdict is
  burst-tolerant (a stack-dump burst within `HANG_BURST_GRACE_S`
  (2s) of death, after a >= 5s silence, still counts as silent). A
  busy multi-test chunk that is legitimately slow near the timeout
  keeps trickling pytest progress output, so it is NOT mislabeled
  hung — it falls through with its real exit metadata.
- Fatal signal (SIGSEGV / SIGABRT / SIGBUS / SIGILL / SIGFPE) →
  `crashed`.
- Non-zero exit, no silence evidence → `crashed`
  (`CHUNK_EXIT_<rc>`); clean exit but missing → `error`
  (`MISSING_FROM_XML`).

## Quarantine (TS-25)

Sliding window of the last 5 outcomes; 3+ failures → quarantined;
3 consecutive passes under `--quarantine-attempt` → evicted.
Quarantined tests are skipped by default — and **every skip is
visible**: the run emits a `quarantined` record (severity **WARN**,
same `outcome -> severity` mapping as
`mgf.common.testing.DEFAULT_SEVERITY`) into `run.jsonl`, the
terminal summary counts them separately, and `--quarantine-report`
prints the table after the run. A gate that silently drops flaky
tests is a false-green vector; this one doesn't.

## Failure bundles (TS-27)

One file per failure at `.gates/last-run/failures/<safe-id>.txt`,
emitted through the canonical `SECTION_ORDER` — every section, in
order, headed `=== <NAME> ===` (empty sections keep their header):
`TEST, OUTCOME, TRACEBACK, STDOUT, STDERR, PROCESS STATE, DMESG
TAIL, ENV, HISTORY, NEXT STEPS`, with perf attribution (TS-28)
appended as an extra section when `--perf` captured data. The
constants are vendored in `records.py` and byte-pinned against
`mgf.common.testing.SECTION_ORDER` by the alignment tests.

## State under `.gates/`

```
last-run/run.jsonl      one TestRecord per outcome (append-only;
                        preserved + appended-to across --resume)
last-run/summary.md     human summary, written at end
last-run/summary.json   machine rollup: counts + per-outcome id lists
last-run/junit.xml      consolidated JUnit built from run.jsonl
last-run/failures/*.txt per-failure bundles
last-run/junit-*.xml    raw per-chunk XML (debugging aid)
state.json              resume checkpoint (atomic write; --resume)
durations.json          duration history + recency (auto-tier input)
quarantine.json         quarantine state
.lock                   advisory lock — one run per checkout
```

Writes are atomic (temp + rename) or append-only; a killed
supervisor leaves at worst a partial trailing JSONL line, which the
reader tolerates. `--self-check` proves the pass/fail/hang/crash
paths against canned tests in a temp tree before you trust any
green.

## Exit codes

`0` clean · `1` at least one failed/errored/hung/crashed · `2`
supervisor failed before completing (including empty tiered
selection) · `3` usage error · `4` self-check failed.
