# mgf-test-supervisor

Budget-aware, crash-survivable, resumable test supervisor that drives `pytest` from the **outside**.

A sibling of `mgf-common` under the `mgf.*` namespace. Federation-standard test infrastructure (TS-22..TS-27).

> **Conformance: L2** per [`mgf-standard/docs/standards/`](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/). Tracker: [`docs/inprogress/MGF_STANDARDS_CONFORMANCE.md`](docs/inprogress/MGF_STANDARDS_CONFORMANCE.md). Runtime LG / CF / OB rules declined per the TS-23 stdlib-only constraint — full rationale in §3 of the tracker.

## Why a supervisor (not a pytest plugin)

A pytest plugin lives **inside** pytest. If pytest crashes — interpreter deadlock, segfault in a C extension, a hook that infinite-loops — the plugin dies with it. The supervisor runs pytest as a subprocess. A pytest crash becomes just one more outcome to record.

What it gives you:

- **Hang detection.** A per-test `pytest-timeout` (in-process) backed by an outer wall-clock SIGTERM → SIGKILL (out-of-process). Tests that genuinely hang are reported `hung`, not "still running."
- **Crash isolation.** One chunk's segfault doesn't take the run with it. Subsequent chunks run. The crashed chunk's in-flight test is recorded `crashed`.
- **Budget-aware selection.** Pass `--budget 5m` and the supervisor packs a greedy knapsack of tests under that wall-clock, biased to tier-0 + recently-failing tests.
- **Resumability.** A run interrupted by SIGINT (or by your laptop sleeping) can be resumed with `--resume`. Already-completed tests aren't repeated.
- **Quarantine.** Flaky tests auto-quarantine after 3-of-5 fails. They evict after 3 consecutive passes. No more "this test is flaky, ignore it" Slack pings.
- **Per-failure diagnostic bundles.** Each failure writes one self-contained `.txt` under `.gates/last-run/failures/` — pytest stdout/stderr, traceback, durations, environment fingerprint, daemon state (if configured). One file = one diagnostic.
- **JSONL outcomes.** `.gates/last-run/run.jsonl` is append-only and parseable. Read via `mgf.common.testing.TestRecord.from_jsonl_line(...)` or any JSON-aware tool.

## Install

Published to the federation devpi index (`magogi/prod`) — pin a version range like any sibling:

```bash
uv add --dev "mgf-test-supervisor>=0.1,<0.2"
```

(`[tool.uv] index-url` in federation projects already points at the devpi index; the old `git+ssh://` install path is retired.)

You also need `pytest-timeout` for the per-test timeout that backs the supervisor's outer wall-clock:

```bash
uv add --dev "pytest-timeout>=2.3,<3"
```

## Adopting in your project

Three steps:

### 1. Install the dep (above) + pytest-timeout.

### 2. Add the pytest config to your `pyproject.toml`.

```toml
[tool.pytest.ini_options]
timeout = 30
timeout_method = "thread"
faulthandler_timeout = 60
markers = [
    "smoke: minimal post-install sanity (fast; no fixtures); tier-0",
    "e2e: end-to-end test exercising bootstrap + multiple subsystems; tier-3",
    "integration: cross-module integration test (may launch real subprocesses); tier-2",
    "contract: wire-format + SDK-surface pins; tier-1",
    "property: invariant-under-all-inputs test (Hypothesis); tier-1",
    "regression: pin for a specific past bug; docstring cites the fixing commit; tier-1",
    "fuzz: random / generated / hostile input — asserts safety properties; tier-1",
    "concurrency: real-threads contention test; tier-2",
    "conformance: design-promise tests (G/S guarantees); release-gated; tier-2",
    "use_case: reproduces a consumer scenario from FEEDBACK paper or recipe doc; tier-2",
    "perf: performance pin — pins a hot path's per-call cost (PF-* + TS-14). tier-3",
    "stress: sustained high-load test; tier-3; nightly only",
]
```

### 3. Add `.gates/` to `.gitignore` (the supervisor's run state lives there).

```gitignore
.gates/
```

### 4. Verify.

```bash
mgf-test-supervisor --self-check   # ~35s — canned pass/fail/hang/crash
mgf-test-supervisor --tier 0       # run your tier-0 smoke tests
```

If `--self-check` reports `PASS`, the supervisor + your project's config are wired correctly.

## CLI

```bash
mgf-test-supervisor                       # run all tier 0-3 tests
mgf-test-supervisor --tier 0              # smoke only
mgf-test-supervisor --tier 0-2            # tiers 0, 1, 2
mgf-test-supervisor --budget 5m           # greedy knapsack under 5min
mgf-test-supervisor --resume              # resume the last interrupted run
mgf-test-supervisor --report              # show last run's summary.md
mgf-test-supervisor --quarantine          # show currently-quarantined tests
mgf-test-supervisor --quarantine-add ID   # manually quarantine a known-flaky test (--reason "...")
mgf-test-supervisor --quarantine-remove ID  # manually evict a just-fixed test
mgf-test-supervisor --self-check          # canned scenarios — verifies the supervisor itself
mgf-test-supervisor --fail-fast           # stop after first failing chunk
mgf-test-supervisor --print-failures      # force the failure detail onto stdout
mgf-test-supervisor --no-print-failures   # suppress it (bundles still written)
```

`mtest` is a short alias for the same binary.

### Failure detail on stdout (CI)

A gate whose failures you cannot read is a gate you cannot act on. On a
red run, after the one-line summary, `mtest` echoes **every** failing
node-id and the tail of its captured detail (assertion / traceback) to
stdout, so a CI log — which never uploads `.gates/last-run/failures/*.txt`
— is self-sufficient.

Default is **automatic**: on when stdout is not a TTY or `CI=` is set
(i.e. in a pipeline); off for an interactive human, who has the bundles
one `cat` away. Force with `--print-failures`, suppress with
`--no-print-failures`.

Bounded so a 500-failure run can't flood the log: the detail is expanded
for the first 20 failures, tail-truncated to the last 40 lines each; the
rest print their node-id plus a pointer to their bundle. **Node-ids are
never withheld** — you always learn WHICH tests failed.

## State files

### Retention (OP-08) — bounded by design

`.gates/` growth is bounded; there is no unbounded log:

- `.gates/last-run/` is **overwritten every fresh run** (run.jsonl, failures/*.txt,
  junit-*.xml) — one run's worth, not a history. A `--resume` run *appends* to the
  interrupted run's record instead of wiping it (PAPER-05), so an interrupt+resume
  still totals one run's worth.
- `.gates/durations.json` keeps a **rolling last-5** timing history per test.
- `.gates/quarantine.json` is a **sliding 5-outcome window** per *flaky* test;
  healthy tests are dropped from it. Evicted tests (3 consecutive passes) are removed.
- `.gates/state.json` is a single resume checkpoint, overwritten per run.

No size/age cap is needed because nothing accumulates across runs. Safe to delete
`.gates/` wholesale at any time — the supervisor recreates it.


Everything under `.gates/`:

```
.gates/
  last-run/
    run.jsonl              — one TestRecord per test outcome (append-only, JSONL;
                             preserved + appended-to across --resume)
    summary.md             — human-readable summary
    summary.json           — machine rollup: counts + per-outcome test-id lists (CI-consumable)
    junit.xml              — consolidated JUnit built from run.jsonl (includes the
                             supervisor-inferred hung/crashed/quarantined outcomes)
    failures/
      <test_id>.txt        — per-failure diagnostic bundle (one file = one failure)
    junit-NNNN.xml         — raw pytest XML per chunk (debugging aid)
  state.json               — resume checkpoint (atomic write)
  durations.json           — per-test duration history + recency score
  quarantine.json          — quarantine state (3-of-5-in / 3-consec-out)
```

## Tier system

The federation twelve-marker taxonomy maps onto 4 tiers (see TS-22 in mgf-standard/docs/standards/TESTING.md §13.11; the legacy `slow` marker completed its deprecation cycle and was removed in M2 Wave-2). The `--tier 0` selection runs only `smoke`-marked tests — the canonical "is the build broken?" signal.

| Tier | Wall-clock guideline | Markers |
|---|---|---|
| 0 | <30s total | smoke |
| 1 | <2min total | contract / property / regression / fuzz |
| 2 | <5min total | concurrency / conformance / use_case / integration |
| 3 | nightly / release | perf / stress / e2e |

## Federation alignment

mgf-test-supervisor is **stdlib-only**. It does not depend on `mgf-common` at runtime — per TS-23, the supervisor must run even when the project under test is broken.

The JSONL schema (`TestRecord` shape, `MARKER_TIER` mapping, `SCHEMA_VERSION`) is mirrored in `mgf.common.testing` for consumer-side parsing. Both surfaces are alignment-pinned by tests in mgf-common's `tests/unit/testing/test_supervisor_alignment.py` AND in mgf-test-supervisor's `tests/unit/test_alignment_with_mgf_common.py`. A field rename in one without the other fails CI on both.

## License

MIT. See LICENSE.

## Related federation libraries

- [mgf-common](https://codeberg.org/magogi-admin/mgf-common) — cornerstone (`mgf.common.testing` carries the consumer-readable schema)
- [mgf-fastapi](https://codeberg.org/magogi-admin/mgf-fastapi), [mgf-django](https://codeberg.org/magogi-admin/mgf-django), [mgf-sqlalchemy](https://codeberg.org/magogi-admin/mgf-sqlalchemy), [mgf-alembic](https://codeberg.org/magogi-admin/mgf-alembic), [mgf-http](https://codeberg.org/magogi-admin/mgf-http), [mgf-apiprobe](https://codeberg.org/magogi-admin/mgf-apiprobe) — the 6 framework-shaped siblings, all adopters
