# FEEDBACK — mgf-test-supervisor

> Paper-tracking file for issues, friction, and requests against
> this project. Consumer projects (and any future adopter) file
> PAPER-NN entries here.
>
> **Where to file what:**
>
> - **mgf-test-supervisor-specific** → file HERE.
> - **Federation-wide** (cross-sibling patterns, standards
>   proposals) → file in
>   [mgf-common/FEEDBACK.md](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md).
>
> **Numbering convention:** paper numbers (`PAPER-NN`) are local
> to this repo (FB-01). When a paper cross-references a
> federation-wide paper, the federation number appears in the
> metadata table as a cross-reference row.
>
> **Discipline:** see
> [mgf-standard/docs/standards/DOCUMENTATION.md](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/DOCUMENTATION.md)
> §FEEDBACK (rules FB-01..FB-04, DOC-13).

---

## §1 Process

1. **File freely.** Any consumer hitting friction MUST file a
   PAPER-NN under §2. Don't wait for permission; the maintainer
   triages.
2. **Status flips** through `OPEN` → `SHIPPED` / `DEFERRED` /
   `WONTFIX` / `MOVED` as the paper resolves. A `SHIPPED` row
   carries a commit-SHA reference + release-version reference.
3. **Renumbering** never happens within this repo — papers keep
   their PAPER-NN forever (FB-01).

---

## §2 Open feedback (active)

#### `PAPER-01` — `mtest` reports a green pytest run as thousands of failures (junit-XML correlation breaks under chunking + deselection)

| Field | Value |
|---|---|
| Status | OPEN |
| Severity | 🟡 High |
| Submitter | `mgf-vm` (commit [`58f7f65`](https://codeberg.org/magogi-admin/mgf-vm/commit/58f7f65)) |
| Submitted | 2026-05-29 |
| Federation paper | — |

**Concern.** A full `mtest` run over `mgf-vm`'s suite reported
**"2228 failed"** while the underlying pytest run was **fully
green**. Every single reported "failure" carried the marker
`chunk exited with 0` and the supervisor-internal code
`MISSING_FROM_XML` — i.e. the chunk's pytest process exited
**successfully (rc=0)**, but the supervisor could not find those
tests' results when it parsed the chunk's junit XML back, so it
defaulted them to `failed`. Running `pytest` directly on the same
tree, in the same venv, at the same commit is green:
`2388 passed, 1 skipped, 17 deselected` (the 1 skip is the
documented `yara not installed`).

**Why it matters.** This is a **false-negative-inverted-to-
false-positive** on the federation's primary test runner — the
exact failure mode a test runner must never have. Concretely:

- It makes `mtest` **unusable as a CI gate** for any suite that
  trips the condition: a maintainer cannot distinguish "2228 real
  regressions" from "0 regressions, parser desync" without falling
  back to raw `pytest`, which defeats the point of the supervisor.
- It **erodes trust** — once a runner cries wolf at this scale,
  green/red from it stops being believed federation-wide.
- It is **silent and data-dependent**: it only appears at a
  certain suite size / chunk count / deselection combination, so
  it will surface intermittently as suites grow, not reproducibly
  at first contact.

**Concrete evidence.**
- Repo / commit: `mgf-vm` @ `58f7f65` (branch `federation-sync-0.42`).
- Suite shape that triggers it: ~2400 collected tests, run with
  the project default that **deselects the `perf` marker** (17
  deselected), executed in **chunked** mode by the supervisor.
- Supervisor verdict: `2228 failed`; **every** failing row shows
  `chunk exited with 0` and `code=MISSING_FROM_XML`. Zero rows
  carry an actual assertion/error traceback.
- Ground truth (same venv, same commit):
  `pytest -q` → `2388 passed, 1 skipped, 17 deselected`, rc=0.
- So: chunk subprocesses **passed** (rc=0) and emitted XML, but
  the result-merge step failed to correlate the emitted node-IDs
  back to the expected set and scored the unmatched expected IDs
  as failures.

**Likely root cause (hypothesis, for triage).** A mismatch
between the **expected** test-ID set the supervisor computes for a
chunk and the **actual** `testcase` node-IDs present in that
chunk's junit XML. Prime suspects, in order:
1. **Deselection accounting** — tests removed by `-m "not perf"`
   (or `deselect`/`--deselect`) are still in the supervisor's
   expected set, so they come back `MISSING_FROM_XML`. The
   supervisor must reconcile its expected set against pytest's
   *deselected* report, not just *collected*.
2. **Chunk ↔ XML pairing** — node-ID normalization drift between
   how the supervisor keys tests (e.g. file path vs `classname`
   + `name`, parametrization brackets, path separators) and how
   pytest writes them into junit `testcase@classname/name`.
3. **Per-chunk XML path collision / overwrite** — chunks writing
   to the same junit path, so a later chunk's XML clobbers an
   earlier one and those tests read as missing.

**Suggested shape.**
- **Fail-loud invariant:** if a chunk process exits `0` **and**
  the merge yields `MISSING_FROM_XML` rows, that is a *supervisor
  internal inconsistency*, not a test failure. Treat it as an
  `INFRA`/`ERROR` outcome with a distinct, un-ignorable diagnostic
  ("N tests expected, M found in XML, P unaccounted — runner bug,
  not your tests"), never silently scored as `failed`.
- **Reconcile against deselection:** subtract pytest-reported
  deselected/skipped IDs from the expected set before diffing
  against the XML.
- **Round-trip self-test:** a regression test that runs a known
  suite which deselects a marker under chunking and asserts
  `mtest` agrees with raw `pytest` (passed/failed/skipped/
  deselected counts match exactly).
- **Diagnostic dump:** on any `MISSING_FROM_XML`, emit the first
  few expected-but-missing IDs and the junit path read, so the
  desync is debuggable from the log alone.

*(Filed per AG-13 / DOC-13: a sibling-specific defect goes in the
target library's FEEDBACK.md. `mgf-vm`'s own suite is unaffected —
it passes under raw `pytest`; this is a supervisor↔consumer
integration defect. No code in this repo was edited.)*

---

## §3 Reviewer's meta-feedback

(none yet — placeholder for cross-cutting observations about
the FEEDBACK process itself.)

---

## §4 How consumers submit feedback

1. **Open a Codeberg issue** at
   <https://codeberg.org/magogi-admin/mgf-test-supervisor/issues> if you have a
   quick question. Maintainer triages.
2. **For a structured pain point**: file a PAPER-NN here as a PR
   that adds the entry to §2.
3. **For AI-agent-driven consumer sessions**: see your project's
   `docs/CLAUDE.md` auto-block (managed by `mgf-common catch-up`)
   — the `feedback-discipline` section spells out where to file
   what.

---

## §5 Cross-references

- [mgf-common/FEEDBACK.md](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md)
  — federation-wide PAPERs.
- [mgf-standard/docs/standards/DOCUMENTATION.md](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/DOCUMENTATION.md)
  — FEEDBACK discipline rules (FB-01..FB-04, DOC-13).
