# FEEDBACK — mgf-test-supervisor

> Paper-tracking file for issues, friction, and requests against
> this project. Consumer projects (and any future adopter) file
> PAPER-NN entries here.
>
> **Where to file what:**
>
> - **mgf-test-supervisor-specific** → file HERE.
> - **Federation-wide** (cross-sibling patterns, standards
>   proposals) → file in
>   [mgf-common/FEEDBACK.md](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md).
>
> **Numbering convention:** paper numbers (`PAPER-NN`) are local
> to this repo (FB-01). When a paper cross-references a
> federation-wide paper, the federation number appears in the
> metadata table as a cross-reference row.
>
> **Discipline:** see
> [mgf-standard/docs/standards/DOCUMENTATION.md](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/DOCUMENTATION.md)
> §FEEDBACK (rules FB-01..FB-04, DOC-13).

---

## §1 Process

1. **File freely.** Any consumer hitting friction MUST file a
   PAPER-NN under §2. Don't wait for permission; the maintainer
   triages.
2. **Status flips** through `OPEN` → `SHIPPED` / `DEFERRED` /
   `WONTFIX` / `MOVED` as the paper resolves. A `SHIPPED` row
   carries a commit-SHA reference + release-version reference.
3. **Renumbering** never happens within this repo — papers keep
   their PAPER-NN forever (FB-01).

---

## §2 Open feedback (active)

#### `PAPER-01` — `mtest` reports a green pytest run as thousands of failures (junit-XML correlation breaks under chunking + deselection)

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟡 High |
| Submitter | `mgf-vm` (commit [`58f7f65`](https://codeberg.org/magogi-admin/mgf-vm/commit/58f7f65)) |
| Submitted | 2026-05-29 |
| Federation paper | — |

**Concern.** A full `mtest` run over `mgf-vm`'s suite reported
**"2228 failed"** while the underlying pytest run was **fully
green**. Every single reported "failure" carried the marker
`chunk exited with 0` and the supervisor-internal code
`MISSING_FROM_XML` — i.e. the chunk's pytest process exited
**successfully (rc=0)**, but the supervisor could not find those
tests' results when it parsed the chunk's junit XML back, so it
defaulted them to `failed`. Running `pytest` directly on the same
tree, in the same venv, at the same commit is green:
`2388 passed, 1 skipped, 17 deselected` (the 1 skip is the
documented `yara not installed`).

**Why it matters.** This is a **false-negative-inverted-to-
false-positive** on the federation's primary test runner — the
exact failure mode a test runner must never have. Concretely:

- It makes `mtest` **unusable as a CI gate** for any suite that
  trips the condition: a maintainer cannot distinguish "2228 real
  regressions" from "0 regressions, parser desync" without falling
  back to raw `pytest`, which defeats the point of the supervisor.
- It **erodes trust** — once a runner cries wolf at this scale,
  green/red from it stops being believed federation-wide.
- It is **silent and data-dependent**: it only appears at a
  certain suite size / chunk count / deselection combination, so
  it will surface intermittently as suites grow, not reproducibly
  at first contact.

**Concrete evidence.**
- Repo / commit: `mgf-vm` @ `58f7f65` (branch `federation-sync-0.42`).
- Suite shape that triggers it: ~2400 collected tests, run with
  the project default that **deselects the `perf` marker** (17
  deselected), executed in **chunked** mode by the supervisor.
- Supervisor verdict: `2228 failed`; **every** failing row shows
  `chunk exited with 0` and `code=MISSING_FROM_XML`. Zero rows
  carry an actual assertion/error traceback.
- Ground truth (same venv, same commit):
  `pytest -q` → `2388 passed, 1 skipped, 17 deselected`, rc=0.
- So: chunk subprocesses **passed** (rc=0) and emitted XML, but
  the result-merge step failed to correlate the emitted node-IDs
  back to the expected set and scored the unmatched expected IDs
  as failures.

**Likely root cause (hypothesis, for triage).** A mismatch
between the **expected** test-ID set the supervisor computes for a
chunk and the **actual** `testcase` node-IDs present in that
chunk's junit XML. Prime suspects, in order:
1. **Deselection accounting** — tests removed by `-m "not perf"`
   (or `deselect`/`--deselect`) are still in the supervisor's
   expected set, so they come back `MISSING_FROM_XML`. The
   supervisor must reconcile its expected set against pytest's
   *deselected* report, not just *collected*.
2. **Chunk ↔ XML pairing** — node-ID normalization drift between
   how the supervisor keys tests (e.g. file path vs `classname`
   + `name`, parametrization brackets, path separators) and how
   pytest writes them into junit `testcase@classname/name`.
3. **Per-chunk XML path collision / overwrite** — chunks writing
   to the same junit path, so a later chunk's XML clobbers an
   earlier one and those tests read as missing.

**Suggested shape.**
- **Fail-loud invariant:** if a chunk process exits `0` **and**
  the merge yields `MISSING_FROM_XML` rows, that is a *supervisor
  internal inconsistency*, not a test failure. Treat it as an
  `INFRA`/`ERROR` outcome with a distinct, un-ignorable diagnostic
  ("N tests expected, M found in XML, P unaccounted — runner bug,
  not your tests"), never silently scored as `failed`.
- **Reconcile against deselection:** subtract pytest-reported
  deselected/skipped IDs from the expected set before diffing
  against the XML.
- **Round-trip self-test:** a regression test that runs a known
  suite which deselects a marker under chunking and asserts
  `mtest` agrees with raw `pytest` (passed/failed/skipped/
  deselected counts match exactly).
- **Diagnostic dump:** on any `MISSING_FROM_XML`, emit the first
  few expected-but-missing IDs and the junit path read, so the
  desync is debuggable from the log alone.

*(Filed per AG-13 / DOC-13: a sibling-specific defect goes in the
target library's FEEDBACK.md. `mgf-vm`'s own suite is unaffected —
it passes under raw `pytest`; this is a supervisor↔consumer
integration defect. No code in this repo was edited.)*

---

#### `PAPER-02` — Concurrency control is a fail-fast per-checkout `flock` only — no shared single-flight authority

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟡 High |
| Submitter | maintainer review (executive library overview, 2026-07-01) |
| Submitted | 2026-07-01 |
| Federation paper | — |

**Concern.** All cross-process coordination is exactly one non-blocking
`fcntl.flock(LOCK_EX | LOCK_NB)` on `.gates/.lock`, scoped to a single checkout
(`__main__.py:124-152`). There is **no daemon, lease, heartbeat, coalescing, or
queue**. A second run on the same checkout (editor-on-save + pre-commit + a CI shard)
does not wait or merge — it raises `SystemExit` immediately. Runs on different
checkouts / CI shards / hosts do not coordinate at all; `MGF_GATES_DIR` deliberately
gives each its own `.gates/` to *avoid* the lock rather than share authority.
(`diagnostics.daemon_state()` probes the *project-under-test's* daemon for failure
bundles — it is **not** a supervisor coordination daemon; none exists.)

**Why it matters.** "A given test can never run twice at once" is enforced only
*within one checkout*, and only by killing the loser. Concurrent launchers can't
coalesce onto one run and cross-shard results can't be merged, so the supervisor can't
be a single source of truth about "is this test running / what was its last verdict"
across the federation.

**Suggested shape.** A lightweight single-flight authority — a short-lived lease +
heartbeat with **request coalescing** (same test → join the in-flight run) and a
**queue** (different test), instead of fail-fast. **Cross-reference:** this is exactly
the role of the planned **`mgf-test-web` / ProofWarden warden**, designed to sit
*above* `mtest` as the sole executor + atomic lease registry. If the warden owns
single-flight, `mtest` itself may stay as-is by design — this paper records the gap so
the decision (warden-owns-it → WONTFIX-in-mtest, vs a lightweight in-`mtest` lease) is
explicit either way.

---

#### `PAPER-03` — Chunks run strictly sequentially — no internal parallelism

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟡 High |
| Submitter | maintainer review (executive library overview, 2026-07-01) |
| Submitted | 2026-07-01 |
| Federation paper | — |

**Concern.** `Supervisor.run()` iterates chunks in a plain for-loop, one pytest
subprocess at a time (`supervisor.py:186-221`). Chunking exists purely for
crash-isolation + startup amortization, not speed; there is no worker pool /
`pytest-xdist` path.

**Why it matters.** Wall-time is the sum of all chunks. On the federation's biggest
suites (`mgf-common` is 700+ tests) that is the dominant cost of the primary gate —
yet the suite is *already* sharded into independent chunks each emitting its own JUnit
XML, so the hard part (isolation) is done.

**Suggested shape.** Run a **bounded pool** of chunk-subprocesses concurrently
(`min(cpu-1, N)`), merging their JUnit + records into one `run.jsonl` — cutting
wall-time roughly linearly with cores while **preserving** per-chunk crash isolation.
Likely the single biggest throughput win; composes naturally with the existing chunk
design. (Pairs well with collapsing the current O(markers) `--collect-only` spawns
into a single-pass collection to cut fixed pre-run overhead.)

---

#### `PAPER-04` — Flaky handling is reactive-only — no in-run retry

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟡 High |
| Submitter | maintainer review (executive library overview, 2026-07-01) |
| Submitted | 2026-07-01 |
| Federation paper | — |

**Concern.** Quarantine is the sole flake mechanism and it's cross-run: a test must
fail **3 of its last 5** recorded runs before it's quarantined
(`quarantine.py:141-207`). A test that flakes *once* still reds the current run, and it
takes ≥3 runs of accumulated history to react. There is no automatic rerun-on-failure
within a single invocation (à la `pytest-rerunfailures`).

**Why it matters.** A single flaky test reds CI *today*, and the system only adapts
after three red runs — slow and noisy for the federation's primary CI gate, and it
trains maintainers to re-run blindly.

**Suggested shape.** An opt-in `--reruns N`: retry a just-failed test up to N times
before recording its **final** outcome. De-flakes CI immediately, and feeds cleaner
pass/fail signal into the quarantine window (a rerun that passes isn't counted as a
fail) rather than making users eat three red runs before the system adapts.

---

#### `PAPER-05` — No CI-consumable result rollup; `--resume` wipes the run record

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟡 High |
| Submitter | maintainer review (executive library overview, 2026-07-01) |
| Submitted | 2026-07-01 |
| Federation paper | — |

**Concern.** Two coupled gaps: (1) the supervisor *consumes* per-chunk JUnit XML but
*emits* only a human `summary.md` + per-test `run.jsonl` (the per-chunk
`junit-NNNN.xml` are explicitly labeled debugging aids) — there is **no consolidated
JUnit or `summary.json`** for a CI system/dashboard to ingest. (2) `run()` unlinks
`run.jsonl` at the start of every run (`supervisor.py:160-167`) and `--resume` filters
already-completed tests out of the selection — so after a resume, `run.jsonl` +
`summary.md` reflect **only the resumed subset**, not the full suite (records for
tests completed before the interruption are gone).

**Why it matters.** Results aren't first-class for CI/dashboards — a system can't
ingest a machine verdict without re-parsing prose — and a resumed run silently loses
the record of everything that ran before the interruption, defeating the point of
resumability.

**Suggested shape.** Emit a merged JUnit + a machine `summary.json` (counts +
per-outcome ID lists) alongside `summary.md`; make `--resume` **append** to
`run.jsonl` rather than wipe it, so the record stays complete across interruptions.

---

#### `PAPER-06` — Manual-quarantine CLI is promised but unwired (dead API)

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟢 Nice-to-have (quick win) |
| Submitter | maintainer review (executive library overview, 2026-07-01) |
| Submitted | 2026-07-01 |
| Federation paper | — |

**Concern.** `QuarantineStore.force_quarantine()` / `force_evict()` exist and their
docstrings explicitly cite `--quarantine-add` / `--quarantine-remove`
(`quarantine.py:209-226`), but the argparse parser wires **neither**
(`__main__.py:219-305` has only `--quarantine` to *view*). Dead API + doc/behavior
drift.

**Why it matters.** Maintainers currently cannot pre-quarantine a known-flaky test
ahead of auto-detection, nor evict a just-fixed one, without hand-editing
`quarantine.json`.

**Suggested shape.** Wire `--quarantine-add <test-id>` / `--quarantine-remove
<test-id>` to the existing `force_quarantine()` / `force_evict()`. Low-effort (the
store already supports it fully) and it removes the doc↔behavior drift.

---

#### `PAPER-07` — `PAPER-01` reads `OPEN` but its fixes appear shipped — close it with a commit SHA, or pin the deselection round-trip test its own suspect #1 asks for

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟡 High |
| Submitter | sooth family — cross-family reuse review |
| Submitted | 2026-07-01 |
| Federation paper | — |

**Concern.** `PAPER-01` is still marked `OPEN`, yet the two mechanisms it
names as prime suspects both appear to be addressed in the current
`supervisor.py`, and only one of them is pinned by a regression test:

- **Suspect #2 (node-id ↔ junit `classname`/`name` keying drift)** — fixed
  and locked by `tests/unit/test_junit_reconciliation.py`, which keys both
  sides by `(classname, name)`.
- **Suspect #1 (deselection accounting)** — the project default
  `-m 'not perf'` leaving deselected IDs in the expected set — appears
  mitigated by blanking the project `addopts` in the chunk runner
  (`supervisor.py:245-246`, and the collect-only paths at `:773-774` /
  `:844-845`), **but there is no regression test exercising the deselection
  round-trip the paper itself calls for.**

So a reader of `FEEDBACK.md` cannot tell whether `PAPER-01` is a live bug or
a shipped fix whose paper was never flipped. Either the paper should close
with the commit SHA that fixed it, or the still-missing deselection
round-trip test should be named as the reason it stays `OPEN`.

**Why it matters.** Trust-in-tool. `mtest`'s whole value is a runner you
can believe, and `PAPER-01` documents the one failure mode that most erodes
that — a fully green suite reported as thousands of failures. A high-severity
paper that is actually (at least partly) fixed but still reads `OPEN` tells an
adopter "this catastrophic desync is unresolved," which is now false for the
class-based half and merely *unclear* for the deselection half. Doc drift on
the very paper about false signals is itself a false signal. And because the
deselection fix is unpinned, a future refactor could silently regress
suspect #1 with no test going red — the exact "silent, data-dependent"
property `PAPER-01` warns about.

**Concrete evidence.**
- `FEEDBACK.md` — PAPER-01 `| Status | OPEN |` (still open).
- PAPER-01 suspect #1: *"Deselection accounting — tests removed
  by `-m "not perf"` … are still in the supervisor's expected set, so they
  come back `MISSING_FROM_XML`. The supervisor must reconcile its expected
  set against pytest's* deselected *report, not just* collected*."*
- PAPER-01's own **Round-trip self-test** ask: "a
  regression test that runs a known suite which deselects a marker under
  chunking and asserts `mtest` agrees with raw `pytest` (passed/failed/
  skipped/deselected counts match exactly)."
- `src/mgf/test_supervisor/supervisor.py:245-246` — the chunk runner blanks
  the project addopts: `"-o", "addopts=",  # blank the project default
  -m '...' exclusion` (same blank at `:773-774` and `:844-845`). Suspect #1
  is thus handled **in code**.
- `tests/unit/test_junit_reconciliation.py` — regression test for **suspect #2
  only**: it pins `_node_id_to_classkey` and `Supervisor._parse_junit` keying
  by `(classname, name)` for the class-based, module-level, parametrised and
  nested-class cases, using a hand-built JUnit document. It never runs pytest
  and never exercises deselection. A grep of `tests/` for `deselect` /
  `not perf` / `addopts` returns **no** hits — the deselection round-trip has
  no regression test anywhere in the suite.

**Suggested shape.** (Offered respectfully — the supervision design itself is
excellent; the sooth family is reusing its budget-knapsack + quarantine-window
+ resume model as a *pattern*, reimplemented in Zig. This is purely a
doc-coherence ask on one paper's lifecycle.)
- **Close it** — flip `PAPER-01`'s `Status` to `SHIPPED` with the commit
  SHA(s) that fixed it (the `(classname, name)` re-keying and the
  `addopts=` blank) plus a release-version reference, per the §1.2 rule that a
  `SHIPPED` row carries a commit-SHA + release-version reference; **or**
- **Keep it `OPEN` but name the blocker** — add the deselection round-trip
  regression test its own suspect #1 / "Round-trip self-test" calls for (a
  suite that deselects a marker under chunking, asserting `mtest`'s
  passed/failed/skipped/deselected counts equal raw `pytest`'s), and cite that
  missing test as the reason the paper stays open.
- Optionally, add a one-line "partially shipped" note to `PAPER-01`
  distinguishing the class-based half (fixed + pinned) from the deselection
  half (mitigated, unpinned), so the status stays legible while it remains
  open.

*(Filed by the sooth family during a cross-family reuse review.
`mgf-test-supervisor`'s supervision design — budget knapsack + quarantine
window + resume — is being adopted as a reusable pattern and is excellent;
this paper is meta-feedback on `FEEDBACK.md`'s own coherence, not a defect in
the runner. No code or docs in this repo were edited.)*

---

## §3 Reviewer's meta-feedback

(none yet — placeholder for cross-cutting observations about
the FEEDBACK process itself.)

---

## §4 How consumers submit feedback

1. **Open a Codeberg issue** at
   <https://codeberg.org/magogi-admin/mgf-test-supervisor/issues> if you have a
   quick question. Maintainer triages.
2. **For a structured pain point**: file a PAPER-NN here as a PR
   that adds the entry to §2.
3. **For AI-agent-driven consumer sessions**: see your project's
   `docs/CLAUDE.md` auto-block (managed by `mgf-common catch-up`)
   — the `feedback-discipline` section spells out where to file
   what.

---

## §5 Cross-references

- [mgf-common/FEEDBACK.md](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md)
  — federation-wide PAPERs.
- [mgf-standard/docs/standards/DOCUMENTATION.md](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/DOCUMENTATION.md)
  — FEEDBACK discipline rules (FB-01..FB-04, DOC-13).
