Metadata-Version: 2.5
Name: mgf-test-supervisor
Version: 0.1.21
Summary: Budget-aware, crash-survivable, resumable test supervisor that drives pytest from the outside. Federation-wide sibling of mgf-common under the mgf.* namespace.
Project-URL: Homepage, https://codeberg.org/magogi-admin/mgf-test-supervisor
Project-URL: Issues, https://codeberg.org/magogi-admin/mgf-test-supervisor/issues
Project-URL: Changelog, https://codeberg.org/magogi-admin/mgf-test-supervisor/src/branch/main/CHANGELOG.md
Author: Bassam Alsanie, mgf-test-supervisor contributors
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: budget,ci,crash-detection,hang-detection,pytest,quarantine,supervisor,test-runner
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: import-linter>=2.0; extra == 'dev'
Requires-Dist: mgf-common<1.0,>=0.41; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest-timeout<3,>=2.3; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff<0.16,>=0.4; extra == 'dev'
Provides-Extra: standards
Requires-Dist: mgf-common[standards]<1.0,>=0.46; extra == 'standards'
Description-Content-Type: text/markdown

# mgf-test-supervisor

Budget-aware, crash-survivable, resumable test supervisor that drives `pytest` from the **outside**.

A sibling of `mgf-common` under the `mgf.*` namespace. Federation-standard test infrastructure (TS-22..TS-27).

> **Conformance: L2** per [`mgf-standard/docs/standards/`](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/). Tracker: [`docs/inprogress/MGF_STANDARDS_CONFORMANCE.md`](docs/inprogress/MGF_STANDARDS_CONFORMANCE.md). Runtime LG / CF / OB rules declined per the TS-23 stdlib-only constraint — full rationale in §3 of the tracker.

## Why a supervisor (not a pytest plugin)

A pytest plugin lives **inside** pytest. If pytest crashes — interpreter deadlock, segfault in a C extension, a hook that infinite-loops — the plugin dies with it. The supervisor runs pytest as a subprocess. A pytest crash becomes just one more outcome to record.

What it gives you:

- **Hang detection that names the test — per test, from outside, even when `pytest-timeout` cannot fire.** A per-test `pytest-timeout` (in-process) backed by an outer wall clock (out-of-process) that is itself **per test**: the supervisor watches which test is in flight and kills the chunk 10s past the budget *your project* declared for that test, so a hang that pytest-timeout can't catch (a C extension holding the GIL, the plugin not installed) costs ~40s instead of the whole chunk budget (1510s at the defaults). `@pytest.mark.timeout(0)` opts a test out. Tests that genuinely hang are reported `hung`, not "still running" — and the report names **the one test that was running**, with its age against its own budget ("in-flight for 30.4s when chunk 3 died (budget 30s)").
- **A pytest that won't exit is named, not waited out.** A test that passes but leaks a non-daemon thread or a child process leaves pytest blocked in interpreter shutdown after its session finished. Instead of silently paying the rest of the chunk budget, the supervisor kills the group 5s later and warns, naming the chunk. The run's outcomes and exit code are unchanged — every test already has its real result.
- **Crash isolation, down to one test.** One chunk's segfault doesn't take the run with it: subsequent chunks run. And it doesn't take the chunk with it either — a per-chunk progress log, written from inside pytest with an append-per-line that survives `os._exit` and SIGKILL, means the tests that had already *passed* keep their real outcomes, exactly the test that crashed is recorded `crashed`, and the tests that never started are re-queued and actually run.
- **Budget-aware selection.** Pass `--budget 5m` and the supervisor packs a greedy knapsack of tests under that wall-clock, biased to tier-0 + recently-failing tests.
- **Resumability.** A run interrupted by SIGINT (or by your laptop sleeping) can be resumed with `--resume`. Already-completed tests aren't repeated.
- **Quarantine.** Flaky tests auto-quarantine after 3-of-5 fails. They evict after 3 consecutive passes. No more "this test is flaky, ignore it" Slack pings.
- **Per-failure diagnostic bundles.** Each failure writes one self-contained `.txt` under `.gates/last-run/failures/` — pytest stdout/stderr, traceback, durations, environment fingerprint, daemon state (if configured). One file = one diagnostic.
- **JSONL outcomes.** `.gates/last-run/run.jsonl` is append-only and parseable. Read via `mgf.common.testing.TestRecord.from_jsonl_line(...)` or any JSON-aware tool.

## Install

Published to the federation devpi index (`magogi/prod`) — pin a version range like any sibling:

```bash
uv add --dev "mgf-test-supervisor>=0.1,<0.2"
```

(`[tool.uv] index-url` in federation projects already points at the devpi index; the old `git+ssh://` install path is retired.)

You also need `pytest-timeout` for the per-test timeout that backs the supervisor's outer wall-clock:

```bash
uv add --dev "pytest-timeout>=2.3,<3"
```

## Adopting in your project

Three steps:

### 1. Install the dep (above) + pytest-timeout.

### 2. Add the pytest config to your `pyproject.toml`.

```toml
[tool.pytest.ini_options]
timeout = 30
timeout_method = "thread"
faulthandler_timeout = 60
markers = [
    "smoke: minimal post-install sanity (fast; no fixtures); tier-0",
    "e2e: end-to-end test exercising bootstrap + multiple subsystems; tier-3",
    "integration: cross-module integration test (may launch real subprocesses); tier-2",
    "contract: wire-format + SDK-surface pins; tier-1",
    "property: invariant-under-all-inputs test (Hypothesis); tier-1",
    "regression: pin for a specific past bug; docstring cites the fixing commit; tier-1",
    "fuzz: random / generated / hostile input — asserts safety properties; tier-1",
    "concurrency: real-threads contention test; tier-2",
    "conformance: design-promise tests (G/S guarantees); release-gated; tier-2",
    "use_case: reproduces a consumer scenario from FEEDBACK paper or recipe doc; tier-2",
    "perf: performance pin — pins a hot path's per-call cost (PF-* + TS-14). tier-3",
    "stress: sustained high-load test; tier-3; nightly only",
]
```

### 3. Add `.gates/` to `.gitignore` (the supervisor's run state lives there).

```gitignore
.gates/
```

### 4. Verify.

```bash
mgf-test-supervisor --self-check   # ~20s — canned pass/fail/crash + a 3-test chunk with one hang
mgf-test-supervisor --tier 0       # run your tier-0 smoke tests
```

If `--self-check` reports `PASS`, the supervisor + your project's config are wired correctly.

## CLI

```bash
mgf-test-supervisor                       # run all tier 0-3 tests
mgf-test-supervisor --tier 0              # smoke only
mgf-test-supervisor --tier 0-2            # tiers 0, 1, 2
mgf-test-supervisor --budget 5m           # greedy knapsack under 5min
mgf-test-supervisor --resume              # resume the last interrupted run
mgf-test-supervisor --report              # show last run's summary.md
mgf-test-supervisor --quarantine          # show currently-quarantined tests
mgf-test-supervisor --quarantine-add ID   # manually quarantine a known-flaky test (--reason "...")
mgf-test-supervisor --quarantine-remove ID  # manually evict a just-fixed test
mgf-test-supervisor --self-check          # canned scenarios — verifies the supervisor itself
mgf-test-supervisor --fail-fast           # stop after first failing chunk
mgf-test-supervisor --print-failures      # force the failure detail onto stdout
mgf-test-supervisor --no-print-failures   # suppress it (bundles still written)
```

`mtest` is a short alias for the same binary.

### Failure detail on stdout (CI)

A gate whose failures you cannot read is a gate you cannot act on. On a
red run, after the one-line summary, `mtest` echoes **every** failing
node-id and the tail of its captured detail (assertion / traceback) to
stdout, so a CI log — which never uploads `.gates/last-run/failures/*.txt`
— is self-sufficient.

Default is **automatic**: on when stdout is not a TTY or `CI=` is set
(i.e. in a pipeline); off for an interactive human, who has the bundles
one `cat` away. Force with `--print-failures`, suppress with
`--no-print-failures`.

Bounded so a 500-failure run can't flood the log: the detail is expanded
for the first 20 failures, tail-truncated to the last 40 lines each; the
rest print their node-id plus a pointer to their bundle. **Node-ids are
never withheld** — you always learn WHICH tests failed.

## State files

### Retention (OP-08) — bounded by design

`.gates/` growth is bounded; there is no unbounded log:

- `.gates/last-run/` is **overwritten every fresh run** (run.jsonl, failures/*.txt,
  junit-*.xml, progress-*.jsonl) — one run's worth, not a history. A `--resume` run
  *appends* to the interrupted run's record instead of wiping it (PAPER-05), so an
  interrupt+resume still totals one run's worth. The per-chunk `junit-*.xml` /
  `progress-*.jsonl` debugging aids are wiped by **every** run including `--resume`
  (they are not the record, and a leftover one from the interrupted run would be
  read as this run's answer).
- `.gates/durations.json` keeps a **rolling last-5** timing history per test.
- `.gates/quarantine.json` is a **sliding 5-outcome window** per *flaky* test;
  healthy tests are dropped from it. Evicted tests (3 consecutive passes) are removed.
- `.gates/state.json` is a single resume checkpoint, overwritten per run.

No size/age cap is needed because nothing accumulates across runs. Safe to delete
`.gates/` wholesale at any time — the supervisor recreates it.


Everything under `.gates/`:

```
.gates/
  last-run/
    run.jsonl              — one TestRecord per test outcome (append-only, JSONL;
                             preserved + appended-to across --resume)
    summary.md             — human-readable summary
    summary.json           — machine rollup: counts + per-outcome test-id lists (CI-consumable)
                             `warnings`: attention-worthy things that are nobody's
                             failure (today: a chunk whose pytest would not exit
                             after its session finished). ALWAYS present — an empty
                             list on a clean run — so consumers need no version check
    junit.xml              — consolidated JUnit built from run.jsonl (includes the
                             supervisor-inferred hung/crashed/quarantined outcomes)
    failures/
      <test_id>.txt        — per-failure diagnostic bundle (one file = one failure)
    junit-NNNN.xml         — raw pytest XML per chunk (debugging aid)
    progress-NNNN.jsonl    — per-chunk event log: start / report / finish /
                             sessionfinish, one JSON line each (debugging aid;
                             the evidence behind per-test attribution when a
                             chunk dies before writing its XML)
  state.json               — resume checkpoint (atomic write)
  durations.json           — per-test duration history + recency score
  quarantine.json          — quarantine state (3-of-5-in / 3-consec-out)
```

## Tier system

The federation twelve-marker taxonomy maps onto 4 tiers (see TS-22 in mgf-standard/docs/standards/TESTING.md §13.11; the legacy `slow` marker completed its deprecation cycle and was removed in M2 Wave-2). The `--tier 0` selection runs only `smoke`-marked tests — the canonical "is the build broken?" signal.

| Tier | Wall-clock guideline | Markers |
|---|---|---|
| 0 | <30s total | smoke |
| 1 | <2min total | contract / property / regression / fuzz |
| 2 | <5min total | concurrency / conformance / use_case / integration |
| 3 | nightly / release | perf / stress / e2e |

## Federation alignment

mgf-test-supervisor is **stdlib-only**. It does not depend on `mgf-common` at runtime — per TS-23, the supervisor must run even when the project under test is broken.

The JSONL schema (`TestRecord` shape, `MARKER_TIER` mapping, `SCHEMA_VERSION`) is mirrored in `mgf.common.testing` for consumer-side parsing. Both surfaces are alignment-pinned by tests in mgf-common's `tests/unit/testing/test_supervisor_alignment.py` AND in mgf-test-supervisor's `tests/unit/test_alignment_with_mgf_common.py`. A field rename in one without the other fails CI on both.

## License

Apache-2.0. See LICENSE.

## Related federation libraries

- [mgf-common](https://codeberg.org/magogi-admin/mgf-common) — cornerstone (`mgf.common.testing` carries the consumer-readable schema)
- [mgf-fastapi](https://codeberg.org/magogi-admin/mgf-fastapi), [mgf-django](https://codeberg.org/magogi-admin/mgf-django), [mgf-sqlalchemy](https://codeberg.org/magogi-admin/mgf-sqlalchemy), [mgf-alembic](https://codeberg.org/magogi-admin/mgf-alembic), [mgf-http](https://codeberg.org/magogi-admin/mgf-http), [mgf-apiprobe](https://codeberg.org/magogi-admin/mgf-apiprobe) — the 6 framework-shaped siblings, all adopters
