mgf-test-supervisor

Budget-aware, crash-survivable, resumable test supervisor that drives pytest from the outside.

A sibling of mgf-common under the mgf.* namespace. Federation-standard test infrastructure (TS-22..TS-27).

Conformance: L2 per mgf-standard/docs/standards/. Tracker: docs/inprogress/MGF_STANDARDS_CONFORMANCE.md. Runtime LG / CF / OB rules declined per the TS-23 stdlib-only constraint — full rationale in §3 of the tracker.

Why a supervisor (not a pytest plugin)

A pytest plugin lives inside pytest. If pytest crashes — interpreter deadlock, segfault in a C extension, a hook that infinite-loops — the plugin dies with it. The supervisor runs pytest as a subprocess. A pytest crash becomes just one more outcome to record.

What it gives you:

• Hang detection that names the test — per test, from outside, even when pytest-timeout cannot fire. A per-test pytest-timeout (in-process) backed by an outer wall clock (out-of-process) that is itself per test: the supervisor watches which test is in flight and kills the chunk 10s past the budget your project declared for that test, so a hang that pytest-timeout can't catch (a C extension holding the GIL, the plugin not installed) costs ~40s instead of the whole chunk budget (1510s at the defaults). @pytest.mark.timeout(0) opts a test out. Tests that genuinely hang are reported hung, not "still running" — and the report names the one test that was running, with its age against its own budget ("in-flight for 30.4s when chunk 3 died (budget 30s)").
• A pytest that won't exit is named, not waited out. A test that passes but leaks a non-daemon thread or a child process leaves pytest blocked in interpreter shutdown after its session finished. Instead of silently paying the rest of the chunk budget, the supervisor kills the group 5s later and warns, naming the chunk. The run's outcomes and exit code are unchanged — every test already has its real result.
• Crash isolation, down to one test. One chunk's segfault doesn't take the run with it: subsequent chunks run. And it doesn't take the chunk with it either — a per-chunk progress log, written from inside pytest with an append-per-line that survives os._exit and SIGKILL, means the tests that had already passed keep their real outcomes, exactly the test that crashed is recorded crashed, and the tests that never started are re-queued and actually run.
• Budget-aware selection. Pass --budget 5m and the supervisor packs a greedy knapsack of tests under that wall-clock, biased to tier-0 + recently-failing tests.
• Resumability. A run interrupted by SIGINT (or by your laptop sleeping) can be resumed with --resume. Already-completed tests aren't repeated.
• Quarantine. Flaky tests auto-quarantine after 3-of-5 fails. They evict after 3 consecutive passes. No more "this test is flaky, ignore it" Slack pings.
• Per-failure diagnostic bundles. Each failure writes one self-contained .txt under .gates/last-run/failures/ — pytest stdout/stderr, traceback, durations, environment fingerprint, daemon state (if configured). One file = one diagnostic.
• JSONL outcomes. .gates/last-run/run.jsonl is append-only and parseable. Read via mgf.common.testing.TestRecord.from_jsonl_line(...) or any JSON-aware tool.

Install

Published to the federation devpi index (magogi/prod) — pin a version range like any sibling:

uv add --dev "mgf-test-supervisor>=0.1,<0.2"

([tool.uv] index-url in federation projects already points at the devpi index; the old git+ssh:// install path is retired.)

You also need pytest-timeout for the per-test timeout that backs the supervisor's outer wall-clock:

uv add --dev "pytest-timeout>=2.3,<3"

Adopting in your project

Three steps:

1. Install the dep (above) + pytest-timeout.

2. Add the pytest config to your pyproject.toml.

[tool.pytest.ini_options]
timeout = 30
timeout_method = "thread"
faulthandler_timeout = 60
markers = [
    "smoke: minimal post-install sanity (fast; no fixtures); tier-0",
    "e2e: end-to-end test exercising bootstrap + multiple subsystems; tier-3",
    "integration: cross-module integration test (may launch real subprocesses); tier-2",
    "contract: wire-format + SDK-surface pins; tier-1",
    "property: invariant-under-all-inputs test (Hypothesis); tier-1",
    "regression: pin for a specific past bug; docstring cites the fixing commit; tier-1",
    "fuzz: random / generated / hostile input — asserts safety properties; tier-1",
    "concurrency: real-threads contention test; tier-2",
    "conformance: design-promise tests (G/S guarantees); release-gated; tier-2",
    "use_case: reproduces a consumer scenario from FEEDBACK paper or recipe doc; tier-2",
    "perf: performance pin — pins a hot path's per-call cost (PF-* + TS-14). tier-3",
    "stress: sustained high-load test; tier-3; nightly only",
]

3. Add .gates/ to .gitignore (the supervisor's run state lives there).

.gates/

4. Verify.

mgf-test-supervisor --self-check   # ~20s — canned pass/fail/crash + a 3-test chunk with one hang
mgf-test-supervisor --tier 0       # run your tier-0 smoke tests

If --self-check reports PASS, the supervisor + your project's config are wired correctly.

CLI

mgf-test-supervisor                       # run all tier 0-3 tests
mgf-test-supervisor --tier 0              # smoke only
mgf-test-supervisor --tier 0-2            # tiers 0, 1, 2
mgf-test-supervisor --budget 5m           # greedy knapsack under 5min
mgf-test-supervisor --resume              # resume the last interrupted run
mgf-test-supervisor --report              # show last run's summary.md
mgf-test-supervisor --quarantine          # show currently-quarantined tests
mgf-test-supervisor --quarantine-add ID   # manually quarantine a known-flaky test (--reason "...")
mgf-test-supervisor --quarantine-remove ID  # manually evict a just-fixed test
mgf-test-supervisor --self-check          # canned scenarios — verifies the supervisor itself
mgf-test-supervisor --fail-fast           # stop after first failing chunk
mgf-test-supervisor --print-failures      # force the failure detail onto stdout
mgf-test-supervisor --no-print-failures   # suppress it (bundles still written)

mtest is a short alias for the same binary.

Failure detail on stdout (CI)

A gate whose failures you cannot read is a gate you cannot act on. On a red run, after the one-line summary, mtest echoes every failing node-id and the tail of its captured detail (assertion / traceback) to stdout, so a CI log — which never uploads .gates/last-run/failures/*.txt — is self-sufficient.

Default is automatic: on when stdout is not a TTY or CI= is set (i.e. in a pipeline); off for an interactive human, who has the bundles one cat away. Force with --print-failures, suppress with --no-print-failures.

Bounded so a 500-failure run can't flood the log: the detail is expanded for the first 20 failures, tail-truncated to the last 40 lines each; the rest print their node-id plus a pointer to their bundle. Node-ids are never withheld — you always learn WHICH tests failed.

State files

Retention (OP-08) — bounded by design

.gates/ growth is bounded; there is no unbounded log:

• .gates/last-run/ is overwritten every fresh run (run.jsonl, failures/.txt, junit-.xml, progress-*.jsonl) — one run's worth, not a history. A --resume run appends to the interrupted run's record instead of wiping it (PAPER-05), so an interrupt+resume still totals one run's worth. The per-chunk junit-*.xml / progress-*.jsonl debugging aids are wiped by every run including --resume (they are not the record, and a leftover one from the interrupted run would be read as this run's answer).
• .gates/durations.json keeps a rolling last-5 timing history per test.
• .gates/quarantine.json is a sliding 5-outcome window per flaky test; healthy tests are dropped from it. Evicted tests (3 consecutive passes) are removed.
• .gates/state.json is a single resume checkpoint, overwritten per run.

No size/age cap is needed because nothing accumulates across runs. Safe to delete .gates/ wholesale at any time — the supervisor recreates it.

Everything under .gates/:

.gates/
  last-run/
    run.jsonl              — one TestRecord per test outcome (append-only, JSONL;
                             preserved + appended-to across --resume)
    summary.md             — human-readable summary
    summary.json           — machine rollup: counts + per-outcome test-id lists (CI-consumable)
                             `warnings`: attention-worthy things that are nobody's
                             failure (today: a chunk whose pytest would not exit
                             after its session finished). ALWAYS present — an empty
                             list on a clean run — so consumers need no version check
    junit.xml              — consolidated JUnit built from run.jsonl (includes the
                             supervisor-inferred hung/crashed/quarantined outcomes)
    failures/
      <test_id>.txt        — per-failure diagnostic bundle (one file = one failure)
    junit-NNNN.xml         — raw pytest XML per chunk (debugging aid)
    progress-NNNN.jsonl    — per-chunk event log: start / report / finish /
                             sessionfinish, one JSON line each (debugging aid;
                             the evidence behind per-test attribution when a
                             chunk dies before writing its XML)
  state.json               — resume checkpoint (atomic write)
  durations.json           — per-test duration history + recency score
  quarantine.json          — quarantine state (3-of-5-in / 3-consec-out)

Tier system

The federation twelve-marker taxonomy maps onto 4 tiers (see TS-22 in mgf-standard/docs/standards/TESTING.md §13.11; the legacy slow marker completed its deprecation cycle and was removed in M2 Wave-2). The --tier 0 selection runs only smoke-marked tests — the canonical "is the build broken?" signal.

Federation alignment

mgf-test-supervisor is stdlib-only. It does not depend on mgf-common at runtime — per TS-23, the supervisor must run even when the project under test is broken.

The JSONL schema (TestRecord shape, MARKER_TIER mapping, SCHEMA_VERSION) is mirrored in mgf.common.testing for consumer-side parsing. Both surfaces are alignment-pinned by tests in mgf-common's tests/unit/testing/test_supervisor_alignment.py AND in mgf-test-supervisor's tests/unit/test_alignment_with_mgf_common.py. A field rename in one without the other fails CI on both.

License

Apache-2.0. See LICENSE.

Related federation libraries

• mgf-common — cornerstone (mgf.common.testing carries the consumer-readable schema)
• mgf-fastapi, mgf-django, mgf-sqlalchemy, mgf-alembic, mgf-http, mgf-apiprobe — the 6 framework-shaped siblings, all adopters