mgf-test-supervisor

Budget-aware, crash-survivable, resumable test supervisor that drives pytest from the outside.

A sibling of mgf-common under the mgf.* namespace. Federation-standard test infrastructure (TS-22..TS-27).

Conformance: L2 per mgf-standard/docs/standards/. Tracker: docs/inprogress/MGF_STANDARDS_CONFORMANCE.md. Runtime LG / CF / OB rules declined per the TS-23 stdlib-only constraint — full rationale in §3 of the tracker.

Why a supervisor (not a pytest plugin)

A pytest plugin lives inside pytest. If pytest crashes — interpreter deadlock, segfault in a C extension, a hook that infinite-loops — the plugin dies with it. The supervisor runs pytest as a subprocess. A pytest crash becomes just one more outcome to record.

What it gives you:

• Hang detection. A per-test pytest-timeout (in-process) backed by an outer wall-clock SIGTERM → SIGKILL (out-of-process). Tests that genuinely hang are reported hung, not "still running."
• Crash isolation. One chunk's segfault doesn't take the run with it. Subsequent chunks run. The crashed chunk's in-flight test is recorded crashed.
• Budget-aware selection. Pass --budget 5m and the supervisor packs a greedy knapsack of tests under that wall-clock, biased to tier-0 + recently-failing tests.
• Resumability. A run interrupted by SIGINT (or by your laptop sleeping) can be resumed with --resume. Already-completed tests aren't repeated.
• Quarantine. Flaky tests auto-quarantine after 3-of-5 fails. They evict after 3 consecutive passes. No more "this test is flaky, ignore it" Slack pings.
• Per-failure diagnostic bundles. Each failure writes one self-contained .txt under .gates/last-run/failures/ — pytest stdout/stderr, traceback, durations, environment fingerprint, daemon state (if configured). One file = one diagnostic.
• JSONL outcomes. .gates/last-run/run.jsonl is append-only and parseable. Read via mgf.common.testing.TestRecord.from_jsonl_line(...) or any JSON-aware tool.

Install

Published to the federation devpi index (magogi/prod) — pin a version range like any sibling:

uv add --dev "mgf-test-supervisor>=0.1,<0.2"

([tool.uv] index-url in federation projects already points at the devpi index; the old git+ssh:// install path is retired.)

You also need pytest-timeout for the per-test timeout that backs the supervisor's outer wall-clock:

uv add --dev "pytest-timeout>=2.3,<3"

Adopting in your project

Three steps:

1. Install the dep (above) + pytest-timeout.

2. Add the pytest config to your pyproject.toml.

[tool.pytest.ini_options]
timeout = 30
timeout_method = "thread"
faulthandler_timeout = 60
markers = [
    "smoke: minimal post-install sanity (fast; no fixtures); tier-0",
    "e2e: end-to-end test exercising bootstrap + multiple subsystems; tier-3",
    "integration: cross-module integration test (may launch real subprocesses); tier-2",
    "contract: wire-format + SDK-surface pins; tier-1",
    "property: invariant-under-all-inputs test (Hypothesis); tier-1",
    "regression: pin for a specific past bug; docstring cites the fixing commit; tier-1",
    "fuzz: random / generated / hostile input — asserts safety properties; tier-1",
    "concurrency: real-threads contention test; tier-2",
    "conformance: design-promise tests (G/S guarantees); release-gated; tier-2",
    "use_case: reproduces a consumer scenario from FEEDBACK paper or recipe doc; tier-2",
    "perf: performance pin — pins a hot path's per-call cost (PF-* + TS-14). tier-3",
    "stress: sustained high-load test; tier-3; nightly only",
]

3. Add .gates/ to .gitignore (the supervisor's run state lives there).

.gates/

4. Verify.

mgf-test-supervisor --self-check   # ~35s — canned pass/fail/hang/crash
mgf-test-supervisor --tier 0       # run your tier-0 smoke tests

If --self-check reports PASS, the supervisor + your project's config are wired correctly.

CLI

mgf-test-supervisor                       # run all tier 0-3 tests
mgf-test-supervisor --tier 0              # smoke only
mgf-test-supervisor --tier 0-2            # tiers 0, 1, 2
mgf-test-supervisor --budget 5m           # greedy knapsack under 5min
mgf-test-supervisor --resume              # resume the last interrupted run
mgf-test-supervisor --report              # show last run's summary.md
mgf-test-supervisor --quarantine          # show currently-quarantined tests
mgf-test-supervisor --quarantine-add ID   # manually quarantine a known-flaky test (--reason "...")
mgf-test-supervisor --quarantine-remove ID  # manually evict a just-fixed test
mgf-test-supervisor --self-check          # canned scenarios — verifies the supervisor itself
mgf-test-supervisor --fail-fast           # stop after first failing chunk
mgf-test-supervisor --print-failures      # force the failure detail onto stdout
mgf-test-supervisor --no-print-failures   # suppress it (bundles still written)

mtest is a short alias for the same binary.

Failure detail on stdout (CI)

A gate whose failures you cannot read is a gate you cannot act on. On a red run, after the one-line summary, mtest echoes every failing node-id and the tail of its captured detail (assertion / traceback) to stdout, so a CI log — which never uploads .gates/last-run/failures/*.txt — is self-sufficient.

Default is automatic: on when stdout is not a TTY or CI= is set (i.e. in a pipeline); off for an interactive human, who has the bundles one cat away. Force with --print-failures, suppress with --no-print-failures.

Bounded so a 500-failure run can't flood the log: the detail is expanded for the first 20 failures, tail-truncated to the last 40 lines each; the rest print their node-id plus a pointer to their bundle. Node-ids are never withheld — you always learn WHICH tests failed.

State files

Retention (OP-08) — bounded by design

.gates/ growth is bounded; there is no unbounded log:

• .gates/last-run/ is overwritten every fresh run (run.jsonl, failures/.txt, junit-.xml) — one run's worth, not a history. A --resume run appends to the interrupted run's record instead of wiping it (PAPER-05), so an interrupt+resume still totals one run's worth.
• .gates/durations.json keeps a rolling last-5 timing history per test.
• .gates/quarantine.json is a sliding 5-outcome window per flaky test; healthy tests are dropped from it. Evicted tests (3 consecutive passes) are removed.
• .gates/state.json is a single resume checkpoint, overwritten per run.

No size/age cap is needed because nothing accumulates across runs. Safe to delete .gates/ wholesale at any time — the supervisor recreates it.

Everything under .gates/:

.gates/
  last-run/
    run.jsonl              — one TestRecord per test outcome (append-only, JSONL;
                             preserved + appended-to across --resume)
    summary.md             — human-readable summary
    summary.json           — machine rollup: counts + per-outcome test-id lists (CI-consumable)
    junit.xml              — consolidated JUnit built from run.jsonl (includes the
                             supervisor-inferred hung/crashed/quarantined outcomes)
    failures/
      <test_id>.txt        — per-failure diagnostic bundle (one file = one failure)
    junit-NNNN.xml         — raw pytest XML per chunk (debugging aid)
  state.json               — resume checkpoint (atomic write)
  durations.json           — per-test duration history + recency score
  quarantine.json          — quarantine state (3-of-5-in / 3-consec-out)

Tier system

The federation twelve-marker taxonomy maps onto 4 tiers (see TS-22 in mgf-standard/docs/standards/TESTING.md §13.11; the legacy slow marker completed its deprecation cycle and was removed in M2 Wave-2). The --tier 0 selection runs only smoke-marked tests — the canonical "is the build broken?" signal.

Federation alignment

mgf-test-supervisor is stdlib-only. It does not depend on mgf-common at runtime — per TS-23, the supervisor must run even when the project under test is broken.

The JSONL schema (TestRecord shape, MARKER_TIER mapping, SCHEMA_VERSION) is mirrored in mgf.common.testing for consumer-side parsing. Both surfaces are alignment-pinned by tests in mgf-common's tests/unit/testing/test_supervisor_alignment.py AND in mgf-test-supervisor's tests/unit/test_alignment_with_mgf_common.py. A field rename in one without the other fails CI on both.

License

MIT. See LICENSE.

Related federation libraries

• mgf-common — cornerstone (mgf.common.testing carries the consumer-readable schema)
• mgf-fastapi, mgf-django, mgf-sqlalchemy, mgf-alembic, mgf-http, mgf-apiprobe — the 6 framework-shaped siblings, all adopters