# Diagnosing hangs, crashes, and "what is pytest doing?"

> **When to read this.** A pytest run sits silent for more than
> a minute, a test "always hangs locally but never in CI," or a
> process is alive but doing nothing visible. The recipes here
> are the field-tested first moves; nothing requires root.
>
> **Provenance.** This playbook originated in `mgf-hrb` (the
> federation's first project-with-daemon adopter) and is
> federation-shipped from `mgf-common` per `docs/standards/TESTING.md`
> §19 (TS-22..TS-27). The recipes here are
> **project-agnostic**; the `mgf-hrb`-specific §3 covering the
> live Zig broker process was dropped during the federation
> transfer — projects with their own daemon-shaped processes
> add a §3-equivalent in their own fork of this file.

This doc is the on-call playbook. Each section is a self-contained
recipe: symptom → diagnosis → fix. The first three account for
~80% of the "my pytest seems hung" reports.

## 0. Is pytest actually hung, or is your output buffered?

**Symptom.** Your terminal hasn't updated for a while. You're
running pytest through a pipe: `pytest ... | tail -30` or
`pytest ... | head -50` or similar.

**Diagnosis.** `tail` and `head` buffer their input. They do NOT
forward incoming lines until either the upstream process exits OR
their internal buffer (4 KiB on Linux) fills. A test that's
*running normally* through `| tail` will look frozen for minutes.

**Fix.** Either drop the pipe (`pytest ... 2>&1 | less +F` works
without buffering), or wrap pytest in `stdbuf -oL -eL`:

```bash
stdbuf -oL -eL .venv/bin/pytest -v
```

**Don't conclude "pytest is hung" before checking this.** Three
out of three "stress test hangs" reports during the 2026-05-25
session were `| tail -30` buffering. The fourth was a genuine hang
and the recipes below caught it in 90 seconds.

## 1. pytest-timeout will tell you if a test is hung

Every test has a 30 s default timeout (per `[tool.pytest.ini_options]`
in `pyproject.toml`). Slow markers (`stress`, `perf`, `e2e`,
`use_case`, `conformance`) override to 60–120 s. If a test exceeds
its budget, pytest fails it with **a full Python stack dump of
every thread** — read the dump first; it usually pinpoints the
blocked call.

If pytest hangs PAST the timeout, the timeout mechanism itself
is wedged — proceed to the kernel-level recipes below.

## 2. Inspect a running Python process without root

`ptrace_scope=1` is the default on most Ubuntu/Debian/Fedora
installs. It blocks `strace -p`, `gdb -p`, and similar
attach-to-running-PID workflows unless the caller is root. The
following recipes work without root:

### What is this process waiting on right now?

```bash
PID=$(pgrep -f "pytest")
cat /proc/$PID/wchan       # one-word kernel wait point (e.g. unix_stream_data_wait)
echo                        # wchan is unterminated; add newline
```

`unix_stream_data_wait` → waiting on a Unix socket read/accept.
`futex_wait_queue` → blocked on a mutex/condvar.
`pipe_read` → waiting for a pipe.
`epoll_wait` → in a poll loop.
`do_wait` → a parent waiting for a child.

The wchan tells you *which kernel object*; combine with the next
recipes to learn *which Python frame*.

### How many threads, and what is each doing?

```bash
ls /proc/$PID/task
for tid in $(ls /proc/$PID/task); do
    printf "TID %s: " "$tid"
    cat /proc/$PID/task/$tid/wchan
    echo
done
```

50+ threads is a sign of a ThreadPoolExecutor or asyncio executor.
A single thread in `unix_stream_data_wait` is the classic
"blocked on broker hello" signature.

### Pull a Python-level stack dump (no root)

If `py-spy` is installed (it is in the `[dev]` extras),
`py-spy dump --pid $PID` prints every Python thread's frame
without attaching as ptrace. Bypasses ptrace_scope entirely:

```bash
.venv/bin/python -m pip install py-spy   # one-time
py-spy dump --pid $PID
```

`py-spy` reads from `/proc/<pid>/maps` + `/proc/<pid>/mem`, both
of which a normal user can read for their own processes.

### Force a Python `faulthandler` dump from any process

`pyproject.toml` enables faulthandler with a 60 s timeout. After
60 s of no progress the broker process dumps every thread's frame
to stderr. You can also trigger it manually:

```bash
kill -SIGABRT $PID   # if you set faulthandler.enable() in the code
```

For pytest, `pytest --tb=auto -v` automatically wires this up.

## 4. "Tests pass against unexpected code"

**Symptom.** A test asserts behaviour you just removed, or fails
on behaviour you just added. Adding a print to `_client.py`
doesn't show up.

**Diagnosis.** The editable install is pointing somewhere
unexpected:

```bash
.venv/bin/python -c "import mgf.hrb; print(mgf.hrb.__file__)"
```

If the printed path isn't under `<repo>/src/mgf/hrb/`, you've
been running tests against a stale tree (often a
`.claude/worktrees/<name>/` directory left by a prior Claude
session).

**Fix.**

```bash
uv pip install -e . --python .venv/bin/python
```

`tests/smoke/test_install_points_to_main.py` enforces this on
every default-suite run. If you're hitting the symptom, that
smoke test should also be failing — check.

## 5. A test passes locally but fails in CI (or vice versa)

**Symptoms + first-line checks.**

- *Network differences.* The broker binds `/tmp/mgf-hrb.sock` — if
  CI uses a non-tmpfs `/tmp` mount it may behave differently under
  load (rare).
- *Time-based flakiness.* `tests/perf/` budgets are 5–20× over
  typical observed numbers, but a starved CI runner can still trip
  them. Check whether the test is `perf`-marker-gated (CI doesn't
  run those by default).
- *Order dependency.* Today there is no enforced randomization
  (`pytest-randomly` is a Phase-6 follow-up). If you suspect
  order-of-execution sensitivity, run the failing test in
  isolation: `pytest <path>::<name> -v`.
- *Stale CI cache.* The `.forgejo/workflows/gates.yml` caches the
  venv keyed on `pyproject.toml` hash and the Zig cache keyed on
  every `*.zig` hash. If a regression survived through unrelated
  caching, force a fresh CI run by re-pushing or invalidating the
  cache key.

## 6. Coverage looks wrong

**Symptom.** `pytest --cov` reports much lower than expected
total coverage, or reports 0% on every file in `src/mgf/hrb/`.

**Diagnosis.** The `[tool.coverage.run].source = ["mgf.hrb"]`
pin in `pyproject.toml` is missing or being overridden by an
explicit `--cov=<other>` argument.

**Fix.** Check `pyproject.toml` has the `[tool.coverage.run]`
block intact; run `pytest --cov` (no source arg) — this honours
the pin.

The historical trap was running `pytest --cov` against a
test-tree where `[tool.coverage.run].source` wasn't pinned;
coverage measured EVERYTHING including the test files themselves,
inflating the denominator and tanking the percentage. The pin
makes the result deterministic.

## 7. Reading `mtest` output

When `scripts/mtest` is the runner (CI, pre-push hook, `mtest`
locally), every test outcome lands in `.gates/last-run/`:

```
.gates/
├── last-run/
│   ├── run.jsonl            ← one JSON record per test outcome
│   ├── summary.md           ← human-readable end-of-run summary
│   ├── junit-NNNN.xml       ← raw pytest XML per chunk (debug aid)
│   └── failures/
│       └── <test-id>.txt    ← one file per failed/hung/crashed test
├── durations.json           ← rolling per-test duration + history
├── quarantine.json          ← currently-quarantined tests
└── state.json               ← resume checkpoint (atomic)
```

**The first thing to read on a failure** is the failure bundle.
Each `failures/<test-id>.txt` carries:

* The traceback (already deduplicated to the failing test).
* Captured stdout + stderr (last 4 KB each).
* Live `mgf-hrb-server` process state at failure (pid + wchan +
  thread count — diagnostic gold for deadlocks).
* `dmesg` tail (last 30 lines), if readable on the host.
* Environment fingerprint (Python version, mgf.hrb install path
  — catches the worktree-shadowed install trap, ziglang version,
  current git HEAD + branch).
* Last 5 outcomes of THIS test (passed/failed history).
* The chunk's wall-time + exit code + signal name.

One file open. No piecing seven sources together.

**JSONL record schema** (the `run.jsonl` format):
```jsonc
{
  "test_id": "tests/foo.py::test_bar",
  "outcome": "passed",          // passed|failed|error|skipped|hung|crashed|quarantined
  "severity": "OK",             // OK|WARN|FAIL|SKIP — mirrors DoctorFinding
  "duration_ms": 123,
  "tier": 0,                    // 0..3
  "markers": ["smoke"],
  "code": null,                 // short failure tag — same string across runs
  "summary": null,              // one-line headline
  "detail": null,               // multi-line traceback
  "captured_stdout": "",
  "captured_stderr": "",
  "run_id": "run-20260525T...Z",
  "ts_iso": "...",
  "schema_version": "1.0.0"
}
```

Schema versioning means a parser can refuse to read newer formats
rather than silently misinterpret a renamed field. `outcome` and
`severity` are closed enums — unknown values are rejected.

**Common mtest invocations for triage:**

```bash
./scripts/mtest --report          # show summary.md
./scripts/mtest --quarantine      # which tests are flaky right now?
./scripts/mtest --self-check      # is the supervisor itself OK?
./scripts/mtest --resume          # continue an interrupted run
./scripts/mtest --tier 0 -v       # smallest signal, max verbosity
```

To diff two runs: keep `.gates/last-run/run.jsonl` around (it's
gitignored but you can copy it manually) and `jq` the result.
E.g., "tests that failed in run A but passed in run B":
```bash
jq -r 'select(.outcome=="failed") | .test_id' run-a.jsonl > a.fails
jq -r 'select(.outcome=="passed") | .test_id' run-b.jsonl > b.pass
comm -12 <(sort a.fails) <(sort b.pass)
```

## 8. "Where is the recipe for X?"

If a hang/crash you hit isn't documented here, please add it after
you debug it. The convention:

- **One short H2 per symptom.**
- **Show the failing user-visible state first**, then the
  diagnostic command(s), then the fix.
- **Cite the cost.** If this trap took an hour to debug, note that
  — future-you reads cost first.
- **Push.** The doc lives in the repo precisely so it accumulates.

The 2026-05-25 reliability pass added sections 0, 2, 3, 4 from
direct session experience. Every one of those was 15–30 minutes
of confused debugging before the right recipe surfaced. They're
captured here so the next contributor's debug is 60 seconds.

Section 7 (Reading mtest output) was added in the supervisor pass
on the same day. With `scripts/mtest` driving CI + the pre-push
hook, most failures are now read from `.gates/last-run/failures/`
first, not from a pytest scrollback.
