# Test Taxonomy

> **What this is.** The canonical vocabulary for talking about tests
> in `mgf-hrb` and (by reuse) sibling federation projects. Thirteen
> layers, each with a one-paragraph definition, a "what makes a good
> X" checklist, and pointers to existing examples in this repo.
>
> **What this is NOT.** A how-to-write-the-test guide — that's
> [`RECIPES.md`](RECIPES.md). And not a runbook — that's
> [`EXECUTION.md`](EXECUTION.md).

## The shape of the taxonomy

Every test answers ONE QUESTION about the system. Layers differ in
**what question** and **at what granularity** — not in importance.
The 13 layers cluster into four families:

| Family | Layers | Common trait |
|---|---|---|
| **Correctness at the unit** | unit, functional, integration | Asserts one logical fact, fast, no external resources |
| **Correctness at the surface** | contract, smoke, end-to-end, use-case | Asserts what consumers see across the wire / process boundary |
| **Correctness under adversity** | property, fuzz, stress, concurrency | Asserts the system survives inputs / loads it didn't expect |
| **Correctness over time** | performance, conformance, regression | Asserts a budget / promise / past-bug stays held |

Within a family, layers escalate: faster→slower, narrower→broader,
controlled→adversarial. A change should add tests at the **lowest
layer that catches the regression**; tests at higher layers exist for
properties the lower layers can't see (concurrency, latency budgets,
wire-format pins).

---

## 1. Unit tests

**Question.** Does this one function do what it claims?

**Scope.** One function or one method in isolation. No I/O, no
sleeping, no threads (except where the function under test IS the
thread primitive). Sub-millisecond runtime per test.

**What makes a good unit test:**

* Asserts ONE behaviour. Multiple asserts are fine if they pin one
  property; one test asserting "x AND y AND z" is three tests.
* Names the property it pins, not the function it calls. `test_socket_missing_returns_fail` > `test_check_socket`.
* The test failure message tells the next reader what regressed
  without them opening the test file.
* Uses fixtures only for setup the caller has no reason to care
  about (an allocator, an in-memory backend).
* DOES NOT mock. If the test needs a mock, it's testing the wrong
  layer — there's an integration concern hiding.

**Examples in this repo:**

* `hrb-code/src/core_test.zig` — every claim-mode interaction.
* `tests/unit/mgf/hrb/test_domain.py` — `from_wire` defensive
  coercion across all three typed records.
* `tests/unit/mgf/hrb/test_identity.py` — comparator weight math.

---

## 2. Functional tests

**Question.** Does this one feature produce the right user-visible
outcome?

**Scope.** One feature end-to-end at the in-process layer. May span
a couple of components (e.g. `Client.claim` exercises Lease + Transport
+ exceptions module) but stays inside one process. Sub-second runtime.

**What makes a good functional test:**

* Walks the happy path AND at least one edge case (error, empty,
  boundary).
* Uses a fake / in-memory peer instead of real I/O when the wire is
  not the property under test.
* Asserts the user-visible outcome, not the implementation steps.

**Examples:**

* `tests/unit/mgf/hrb/test_client.py::test_claim_returns_lease_and_release_round_trip`
* `tests/unit/mgf/hrb/test_cli_doctor.py::test_doctor_command_runs_end_to_end`

---

## 3. Integration tests

**Question.** Do these N components together behave correctly?

**Scope.** Multiple in-process components composed. Still no
external processes. Sub-second; may use threads.

**What makes a good integration test:**

* Names the interaction. `test_facade_routes_claim_to_owning_worker`
  > `test_facade`.
* Catches contract-mismatch between components (a return type one
  thought was X, another thought was Y).
* Lives in the test file of the OUTER component (the one composing
  the inner ones), not the inner one.

**Examples:**

* `hrb-code/src/facade_test.zig` — facade + 2 workers + caps + liveness
  all wired up.
* `tests/unit/mgf/hrb/test_cli_doctor.py` — CLI renderer + diagnostics
  engine.

---

## 4. Contract tests

**Question.** Does the public surface match what consumers depend
on?

**Scope.** The "API + wire format" pin. Catches a renamed field, a
removed exception class, a changed dict shape. Lives in dedicated
files so failures point unambiguously at a contract change.

**What makes a good contract test:**

* Pins the EXACT shape: field names spelled out, every key asserted,
  every type asserted.
* When a contract intentionally changes, the test changes in the
  SAME COMMIT — review then catches that the change is intentional.
* Doesn't assert behaviour (that's functional / unit) — only shape.

**Examples in this repo:**

* `tests/unit/mgf/hrb/test_client.py::test_capabilities_returns_wire_payload`
  pins the §6.2.1 wire fields.
* `tests/unit/mgf/hrb/test_client.py::test_capabilities_degraded_mode_carries_reason`
  pins `mode_reason` not getting dropped.

**Convention:** future contract tests live under `tests/contract/`
once that directory has more than 5 files — graduates from a
sub-section of `tests/unit/`.

---

## 5. Property tests

**Question.** Does this invariant hold for **all** valid inputs?

**Scope.** Generates inputs systematically (random, hypothesis-style,
or exhaustive) and asserts a property holds. Fast (milliseconds per
input, hundreds to thousands of inputs per test).

**What makes a good property test:**

* The property is a TRUE INVARIANT, not a happy-path assertion.
  "Every URN with a `**` glob matches its prefix" is a property;
  "list returns 36 PCI devices" is not.
* Failure produces a minimum-reproducer the human can read.
* Has a fixed seed so re-runs find the same failures.

**Examples:**

* `hrb-code/src/cap_test.zig::test "cap U3: minting 100 caps yields 100 distinct non-zero ids"`
* `hrb-code/src/ring_test.zig::tier0_5: fuzzer` — 1 000 rounds of
  random garbage in producer fields, consumer never panics.

---

## 6. Smoke tests

**Question.** Is the system minimally alive?

**Scope.** Start the broker, do one round-trip, succeed. Designed to
catch "the build is broken" / "the broker won't even start" — the
fastest test that exercises real processes. Single-digit-second
runtime.

**What makes a good smoke test:**

* Asserts as little as possible — "broker accepts a connection and
  responds to one op." Anything more is functional / e2e.
* Runs as the FIRST test in CI so a broken build fails in 5 seconds,
  not 5 minutes.
* Cleans up unconditionally (broker subprocess gets killed even on
  test failure).

**Recommended addition** (gap — see [`COVERAGE_MAP.md`](COVERAGE_MAP.md)):
`tests/smoke/test_broker_boots.py` — broker spawn, hello handshake,
single `list()` call. Today's e2e test does this AND much more; a
dedicated smoke pin would let CI fail-fast on build-only issues.

---

## 7. Concurrency tests

**Question.** Do the locks / atomics / serialization invariants
hold under real parallel execution?

**Scope.** Spawns real threads (or processes) and exercises the
component under contention. Asserts no race-driven corruption, no
deadlock, no double-mutation. Runtime: hundreds of milliseconds.

**What makes a good concurrency test:**

* Uses ENOUGH threads to make a race likely if one exists (8+, not
  2).
* Has a stress-loop count that can be tuned upward in CI nightly.
* Asserts on the END STATE, not on intermediate observations.

**Examples:**

* `core_test.zig::concurrency: N threads claiming the SAME URN`
* `test_client.py::test_transport_call_holds_lock_across_send_recv`

---

## 8. End-to-end tests

**Question.** Does a real consumer talking to a real broker over a
real socket produce the right outcomes?

**Scope.** Spawns the actual broker binary, opens a real Unix
socket connection, exercises every major op. Slow (seconds per run);
opt-in via the `e2e` marker.

**What makes a good e2e test:**

* Hits every layer end-to-end — broker process, kernel-attested
  peer cred, hello handshake, wire serialization, SDK projection.
* Asserts the user-visible outcome AT EACH layer (not just the
  final return value).
* Has explicit cleanup of broker subprocess + socket file.
* Runs in CI on every PR but is gated so contributors without the
  broker binary built can still run the unit suite.

**Examples:**

* `tests/e2e/test_e2e.py::test_e2e` — 36 checks against a live
  broker, covering every shipped SDK verb.

---

## 9. Use-case tests

**Question.** Does the real consumer scenario (the one a paper
named) actually work?

**Scope.** Reproduces a consumer's exact usage pattern end-to-end.
Distinct from e2e: e2e is breadth (every verb once), use-case is
depth (one scenario, full sequence with realistic timing /
ordering / interleaving).

**What makes a good use-case test:**

* Cites a FEEDBACK paper or a recipe doc by name in the docstring.
* Walks the scenario in the same order a human would.
* Asserts at points where the consumer would be observing
  outcomes — not at every internal step.

**Recommended addition** (gap):
* `tests/use_cases/test_gui_device_pane.py` — the hrb-app Phase-2
  device-tree + own-leases scenario (FEEDBACK SDK-03 motivation).
* `tests/use_cases/test_doctor_at_install_time.py` — TOOLS-02
  scenario: operator runs `Client.doctor()`, finds IOMMU off,
  follows the remediation.
* `tests/use_cases/test_usb_passthrough_to_vm.py` —
  `recipes/claim-a-device-for-vfio-passthrough.md` walked
  programmatically.

---

## 10. Fuzz tests

**Question.** Can hostile / random / malformed input crash the
broker?

**Scope.** Bombards a parser / decoder / state machine with
randomised input, asserting only that it never panics, never
escapes to an unbounded error, never produces undefined behaviour.
The CONTRACT being tested is "never crashes" — not "returns the
right answer."

**What makes a good fuzz test:**

* Tests the input-validation layer (the trust gate), not deeper.
* Has multiple SEEDS so a flaky failure on one seed gets caught
  by another.
* Asserts ONLY safety properties — never functional correctness on
  random input.

**Examples:**

* `request_test.zig::S1 fuzz: 4_000 random byte sequences never
  panic`
* `ring_test.zig::tier0_5: fuzzer — random garbage in producer
  fields`

---

## 11. Stress tests

**Question.** Does the system stay correct + bounded under sustained
high load?

**Scope.** Runs an op (or mix of ops) at high frequency / parallelism
for a meaningful duration. Asserts on STATE INVARIANTS (lease count
bounded, memory bounded, no resource leaks) — not throughput
(that's performance).

**What makes a good stress test:**

* Has both a "tight load" mode (CI minutes-budget) and an "endurance"
  mode (nightly hours-budget) gated by env var.
* Asserts the END STATE matches the START STATE — the system
  returned to equilibrium.
* Distinguishable from a benchmark — no timing assertion.

**Examples:**

* `stub_usb_test.zig::stub-usb stress: contention storm`
* `core_test.zig::concurrency: stress — interleaved claim+release
  across threads`

---

## 12. Performance tests

**Question.** Does this op meet its latency / throughput budget?

**Scope.** Measures wall-clock time or ops/sec for a target
operation; asserts it falls within a documented budget. Nightly-only;
not part of the four-gate PR contract because hardware variance
would make it flaky.

**What makes a good performance test:**

* Has a NUMERIC budget in the docstring with rationale (where the
  budget came from — design Part 5.x, mgf-usb v0.2.5 measurement,
  etc.).
* Reports the actual number alongside pass/fail.
* Uses a warmup, then a measurement window — no cold-cache lies.

**Examples:**

* `ring_test.zig::BENCHMARK: single-thread throughput` — declares
  "millions of items/sec" as the budget.

**Recommended additions** (gap):
* `tests/perf/test_claim_latency.py` — p50 / p99 latency for a
  hot-path claim against the fake backend.
* `tests/perf/test_drain_events_throughput.py` — events / sec
  delivered through the polling poll.

---

## 13. Conformance tests

**Question.** Do the design's named guarantees (G1–G4 / S1–S5) hold?

**Scope.** Asserts the architectural properties spelled out in the
design — not specific behaviours, but classes of behaviour
("ghosting impossible," "deadline always honoured," "fault contained
to one worker"). Slow (full scenarios per assertion); release-gated.

**What makes a good conformance test:**

* Cites the design section by number in the docstring.
* Pin one design promise per test — `test_G3_holder_death_reclaims_all_leases`,
  not `test_supervisor`.
* Documents WHAT failure mode the test would have caught — so a
  future reader knows whether to delete it after a refactor.

**Examples:**

* `conformance.zig` — the S1–S5 suite.
* `core_test.zig::core G3: client death reclaims ALL its leases`

---

## When in doubt: which layer?

A decision tree:

1. **Does the test need an external process / socket / device?**
   → e2e (if broker) or use-case (if scenario)
2. **Is the property "doesn't crash under any input"?**
   → fuzz
3. **Is the property "stays correct under high load"?**
   → stress (no timing) or performance (with timing)
4. **Is the property "this exact wire field exists with this exact
   spelling"?** → contract
5. **Is the property a true invariant ("for all X, P(X) holds")?**
   → property
6. **Does the test spawn real threads to expose a race?**
   → concurrency
7. **Does the test reproduce a specific bug?** → regression (sub-type
   of unit / functional — see RECIPES.md)
8. **Otherwise** → unit (one function) / functional (one feature) /
   integration (multiple components) — pick the smallest that
   exercises the regression.

The taxonomy is a tool, not a religion: a test that fits two
categories is fine. Tag it with the marker for the MORE-EXPENSIVE
of the two (it'll get gated correctly that way).
