# Part 12 — Survivability & Adversarial-Resistance Review

A staged review establishing that reliability is *natively enforced and verified*, not merely intended — across both adversarial boundaries (hostile clients, hostile/broken hardware) and the broker's own faults, under heavy concurrent multi-tenant load. This part defines the survivability guarantees S1–S5 (summarized in Part 1.6), the threat model, and the hardening roadmap. Peak performance is preserved throughout because all hardening lands on the cold control path, never the hot use path (G4).

> **Hardening status (as of 2026-05-24).** This document was written as a
> forward-looking review; several Tier-0 items it called 🔴 CRITICAL have
> since landed in code. Read the section text as the *original analysis*
> and consult this banner for the *current state*.
>
> - **Tier-0 #1 — Trust gate** — ✅ **SHIPPED** (commit `45efd04`,
>   `hrb-code/src/request.zig`). The skeleton's `.string`/`.integer`
>   access pattern flagged in §3.2 is gone; every byte now passes through
>   typed extractors and a 7 000-iteration fuzzer asserts only
>   `BadRequest` or a valid `Request` ever comes out.
> - **Tier-0 #3 — Deadlines on the seam** (D-029) — ✅ **SHIPPED**
>   (commit `0aa9077`). Every `KernelBackend` op now carries a
>   `kabi.Deadline`; `Err.Timeout` is a typed outcome with atomicity
>   preserved through the timeout path. The §2.2 "no concept of time"
>   finding is closed *at the contract level*; the watchdog/killable-
>   context half — kill a worker wedged on a syscall the deadline
>   can't abandon — remains a Tier-0 hardening pass.
> - **MODEL-01 identity comparator** — ✅ **SHIPPED** (commit `14d3d5e`,
>   `hrb-code/src/identity.zig` + Python mirror at `src/mgf/hrb/identity.py`).
>   Not a survivability-review finding directly, but referenced by §3.4's
>   revocation discussion as a prerequisite for "same physical unit
>   across a mode change."
>
> **2026-05-26 update — Tier-0 closures.**
> - **Tier-0 #2** (unforgeable + connection-bound capabilities, §3.3) ✅ **SHIPPED**.
>   SO_PEERCRED on every accepted client + CSPRNG-sourced cap ids (no more
>   "guess id=1"); cap binds to first-seen peer and refuses every other
>   identity (commits `47e2cf3` / `5003d5c` / `06a0709`). Helper privilege-drop
>   seam at `dd69abc`.
> - **Tier-0 #3 watchdog half** ✅ **SHIPPED** (commit `dc7d481`). SO_RCVTIMEO=5s
>   on the helper socket; on deadline the supervisor SIGKILLs the helper, waits
>   with waitpid, respawns on the next op. A wedged ioctl costs one bus worker,
>   never the broker.
> - **Tier-0 #4** (revoke below the fd, §3.4) ✅ **SHIPPED** (commit `61de2b4`).
>   Broker holds the USBDEVFS fd; revoke calls USBDEVFS_RELEASEINTERFACE + close.
>   New `steal` right gates the op; `Err.Revoked` is distinct from
>   `Err.InvalidToken`. PCI/VFIO branch plugs into the same wire surface when
>   `vAcquire` lands on a real host.
>
> **Still open** — only **Tier-0 #5** (asymmetric shared-memory trust + per-tenant
> private regions, §3.5).


---


A staged review against one bar: **as the standard way userspace touches hardware, the broker must never crash, hang, leak, or misbehave — no matter how hostile or broken the world above and below it is.** Reliability is the top priority (design §1.1); this review tests whether reliability is *natively* enforced or merely intended, and whether peak performance survives the hardening.

Severity scale used throughout:
- **🔴 CRITICAL** — can cause a crash, hang, security breach, or cross-tenant effect. Must be closed before this is anyone's standard.
- **🟠 MAJOR** — degrades reliability/isolation under stress; closeable without architectural change.
- **🟡 MINOR** — hardening or defense-in-depth; good hygiene.
- **🟢 STRENGTH** — already designed correctly; preserve it.

---

# Stage 1 — The Threat Model (the frame everything hangs on)

Before auditing failure surfaces, we fix *who can do what*, *what crosses each boundary*, and *what we are allowed to assume*. A survivability claim is meaningless without this.

## 1.1 The actors

| Actor | Trust | Can be | Wants |
|---|---|---|---|
| **Client / tenant app** | **Untrusted** | buggy, hostile, compromised, greedy | resources; possibly to crash the broker, escape its capability, read another tenant's data, or starve rivals |
| **Multiple concurrent tenants** | **Mutually untrusted** | competing for the same hardware | their own resources; possibly to deny service to each other |
| **Hardware device** | **Untrusted** | dead, frozen, malicious (rogue DMA, bad firmware), slow, lying | (if malicious) to corrupt the host, hang the broker, exfiltrate via DMA |
| **Kernel / driver** | **Semi-trusted** | buggy, slow, can return errors, can block indefinitely | — (not adversarial, but unreliable) |
| **The broker itself** | trusted core | can have bugs, can exhaust memory, can panic | to survive its own faults |
| **Privileged helper** | most-trusted, smallest | the only component with raw privilege | to do the minimum and nothing else |

The defining feature of this threat model: **both sides of the broker are untrusted.** The client above and the device below can each be actively hostile. The broker is a *membrane between two adversarial worlds*, plus mutually-adversarial tenants on the upper side. This is a stricter model than most middleware assumes.

## 1.2 The two boundaries (and a third, internal one)

```
     ┌─────────────────────────────────────────────┐
     │  TENANT A   TENANT B   TENANT C  (untrusted,  │
     │                                  mutually so) │
     └───────┬─────────┬─────────┬──────────────────┘
             │  BOUNDARY 1 (UP): the client boundary
   ══════════╪═════════╪═════════╪═══════════════════  ← every byte crossing is hostile-until-validated
             ▼         ▼         ▼
     ┌─────────────────────────────────────────────┐
     │              THE BROKER                       │
     │  facade → workers → core → cap → backend      │
     │   · BOUNDARY 3 (INTERNAL): worker↔helper,     │
     │     worker↔worker, facade↔worker — a fault in │
     │     one must not take down the others         │
     └───────────────────┬─────────────────────────┘
             │  BOUNDARY 2 (DOWN): the hardware/kernel boundary
   ══════════╪═══════════════════════════════════════  ← the device may be dead, frozen, or malicious
             ▼
     ┌─────────────────────────────────────────────┐
     │   KERNEL / DRIVER / DEVICE  (unreliable;      │
     │                              device hostile)  │
     └─────────────────────────────────────────────┘
```

- **Boundary 1 (UP) — the client boundary.** Every request, every byte in shared memory, every fd, every capability token presented is hostile until proven safe. The defining rule: *nothing a client does may cause the broker to crash, hang, corrupt state, breach isolation, or behave outside the design.*
- **Boundary 2 (DOWN) — the hardware/kernel boundary.** Every syscall may block forever, return an error, or succeed-then-lie. The device may be frozen, may raise spurious interrupts, may attempt rogue DMA. The defining rule: *no hardware or kernel pathology may propagate upward into a broker crash/hang or into a false success reported to a client.*
- **Boundary 3 (INTERNAL) — the fault-isolation boundary.** Workers, the facade, and the privileged helper are separate processes (design §7, §11). The defining rule: *a fault in any one component may not crash, corrupt, or hang any other — blast radius is contained to a single bus worker at most.*

## 1.3 What we are explicitly allowed to assume

A survivability design must state its assumptions or it is dishonest. We assume:
- **The kernel is not actively malicious.** It can be buggy, slow, or blocking, but we are not defending against a hostile kernel (that is the kernel's own threat model; on the future OS, the broker *is* more privileged). We *do* defend against a buggy/blocking/erroring kernel.
- **The hardware can be malicious only within what the IOMMU contains.** We assume an IOMMU exists and is configured; rogue DMA is contained by it. A platform with no IOMMU is explicitly a degraded-security deployment (must be detected and declared — see Stage 2).
- **The privileged helper's small TCB is correct.** We minimize it precisely so this assumption is affordable to audit. We do *not* assume the large Python policy layer or any client is correct.
- **Side channels (Spectre-class, power, EM) are out of scope** for this review except where the broker's own design widens them (e.g. shared-memory timing across tenants — flagged in Stage 5). Full microarchitectural isolation is a platform concern.

## 1.4 The survivability guarantees we will hold the design to

G1–G4 (design §1.4) cover *truthful representation*. They do **not** cover survivability under attack. This review adds five guarantees and audits the design against them. Proposing these as first-class, named, testable properties — peers of G1–G4 — is itself the first recommendation of this review.

- **S1 — Upward crash-safety.** No sequence of client actions (malformed requests, bad fds, poisoned shared memory, capability forgery attempts, request floods) can crash, hang, or corrupt the broker. The worst a client can do to *the broker* is get its own connection dropped.
- **S2 — Downward fault-containment.** No device or kernel behavior (timeout, freeze, privilege error, spurious interrupt, hotplug-during-op, lie) can crash or indefinitely block the broker. Every downward call is bounded in time and total in error handling.
- **S3 — Tenant isolation.** No tenant can read, corrupt, observe (beyond policy), or deny service to another tenant's resources or data, except through explicitly modeled, policy-permitted sharing.
- **S4 — Self-fault containment.** A bug, panic, or resource exhaustion in any broker component is contained to the smallest blast radius (one bus worker), with automatic recovery, and never escalates to total broker failure or to a reliability lie (a ghosted lease, a corrupted ownership table).
- **S5 — Bounded everything.** Every resource the broker consumes on behalf of clients (memory, fds, table entries, CPU time, lease count) is bounded per-tenant and in aggregate, so no actor can exhaust the broker or the host.

The rest of the review walks the boundaries and rates the current design against S1–S5, finding by finding.

---

## Stage 1 findings (the framing itself)

**🟠 MAJOR — survivability is intended but not named.** The design's priority order puts reliability first, and G1–G4 are excellent, but there is no explicit, testable statement of "no input crashes us / no device hangs us / no tenant touches another." Reliability-under-adversity is currently *emergent from good taste*, not *enforced by named guarantees with a conformance suite*. **Recommendation:** adopt S1–S5 as first-class guarantees alongside G1–G4, each with its own contract tests (mirroring how the kabi conformance suite makes LSP-under-failure testable rather than hoped). Everything downstream in this review becomes a test against one of these.

**🟢 STRENGTH — the architecture already chose the hard, correct primitives.** Separate processes (§7), a minimized privileged helper (§11), capabilities not identities (§4), leases bound to liveness (§3), the IOMMU as the DMA-containment assumption — these are exactly the load-bearing choices a survivability design *needs*, and they were made for reliability reasons already. The review's job is therefore mostly to find the *gaps between these good bones and the stated bar*, not to redesign. That is a good place to start from.

**Next stages:** Stage 2 audits Boundary 2 (downward) against S2; Stage 3 audits Boundary 1 (upward) against S1+S3; Stage 4 audits Boundary 3 (internal) against S4; Stage 5 audits concurrency/fairness against S3+S5; Stage 6 synthesizes the roadmap.

---


**S2 — Downward fault-containment:** no device or kernel behavior (timeout, freeze, privilege error, spurious interrupt, hotplug-during-op, lie) may crash or indefinitely block the broker. Every downward call is bounded in time and total in error handling.

This is your first stated example — "hard errors or crash shouldn't happen because hardware or kernel issues (timeout, frozen, privilege error, anything)." The downward boundary has a property the upward one does not: **a failure here is often not an error code but the absence of one** — the call simply never returns. That reframes the whole analysis.

## 2.1 The taxonomy of downward pathologies

Every way the world below can hurt the broker, and whether the current design addresses it:

| # | Pathology | Example | Current design | Verdict |
|---|---|---|---|---|
| D1 | **Error return** | ioctl returns `-EBUSY`, `-EINVAL` | mapped to `Err`, errdefer unwinds | 🟢 handled |
| D2 | **Honest unsupported** | no VFIO substrate | returns `Unsupported`, never lies | 🟢 handled (G2) |
| D3 | **Indefinite block** | frozen device, wedged driver, ioctl never returns | **nothing bounds the call in time** | 🔴 **CRITICAL gap** |
| D4 | **Slow-but-eventual** | device takes 8s to reset | no distinction from D3; no deadline policy | 🔴 part of the same gap |
| D5 | **Privilege error** | `EPERM`/`EACCES` mid-operation | mapped to `Err` if returned synchronously | 🟠 handled if it returns; see D3 if it blocks |
| D6 | **Success-then-lie** | reset reports OK, device still dirty | G2 says "never lie," but a *backend* could be fooled by the kernel | 🟠 needs post-condition verification |
| D7 | **Spurious/flood interrupts** | malicious device raises IRQs continuously | not addressed (interrupts are a use-path concern) | 🟠 needs an interrupt-rate policy |
| D8 | **Hotplug during operation** | device unplugged mid-claim | partially: presence events exist; race with in-flight op not specified | 🟠 race window unspecified |
| D9 | **Rogue DMA** | device DMAs into host memory | IOMMU assumption (Stage 1); only safe *if* IOMMU present+configured | 🟠 must be enforced + declared, not assumed |
| D10 | **Partial/torn state** | crash between bind and configure | errdefer-unwind on the acquire path | 🟢 handled (Part 3.4) |
| D11 | **Firmware/microcode hang** | FLR (function-level reset) wedges the bus | bus-level reset exists as a level; escalation policy unspecified | 🟠 needs reset-escalation ladder |

The headline: **error-shaped failures are well handled; time-shaped and lie-shaped failures are not.** The design implicitly assumes downward calls return. Against a frozen or malicious device, that assumption is false, and it is the single most important gap in the whole review.

## 2.2 ⚙️ PARTIALLY CLOSED (was 🔴 CRITICAL) — the seam has no concept of time (D3/D4)

> **Status: contract half SHIPPED (commit `0aa9077` / D-029).** Every
> `KernelBackend` op now carries a `kabi.Deadline`; `Err.Timeout` is a
> typed outcome with atomicity preserved through the timeout path (a
> claim whose `acquire` step times out commits no lease, leaves the
> resource safe and claimable). `conformance.zig` pins the contract
> against any backend; `FakeBackend` declares per-op work durations
> via `slow_*_ms` toggles for deadline coverage without a frozen
> device. **Still open** — the watchdog half (recommendation point 2/4
> below): the killable-context process model so a worker wedged on a
> syscall the deadline alone cannot abandon is killed + respawned
> rather than hanging. That is the next Tier-0 hardening pass.

**Original finding.** The `KernelBackend` contract (kabi.zig) specifies *what* each operation does and *what errors* it may return, but never *how long it may take* or *what happens if it never returns*. `acquire`, `reset`, `map_region` all call into the kernel/device. A frozen device makes `reset` block forever. Today, that blocks the calling worker thread indefinitely — a permanent hang of one bus worker, and any client waiting on it hangs too. This directly violates S2 and your explicit "frozen … timeout" requirement.

**Why it's critical.** "The broker must never hang" cannot be satisfied by a contract that permits unbounded calls. No amount of careful coding in one backend fixes this; it must be a *contract-level* guarantee so every backend (including `mgf-hrb-youros`) inherits it and is *tested* for it.

**Recommendation — make time a first-class part of the seam contract:**

1. **Every downward operation carries a deadline.** The seam signature gains a deadline/budget parameter (or the worker enforces one around every vtable call). The contract states: *a backend operation must either complete, error, or be abandonable within its deadline.*

2. **Operations that can block run in an isolated, killable context.** The blocking ioctl/syscall executes where it can be abandoned without corrupting the worker — either (a) on a dedicated I/O thread the worker can declare dead and replace, or (b) in the per-bus worker *process* such that a watchdog can kill and respawn the worker without touching the facade or other buses. Option (b) aligns with the existing process model (§7) and is preferred: **a frozen device costs you one bus worker, which the supervisor restarts — never the broker.**

3. **A timeout is a typed outcome, not an exception.** Add `Err.Timeout` (or a `DeadlineExceeded` status). A timed-out operation transitions the resource to a defined state — `Faulted{reason=timeout}` — never an ambiguous one. The lease, if any, is reclaimed and the resource quarantined (see 2.4).

4. **The watchdog is the enforcement mechanism.** A supervisor (the facade or a dedicated monitor) tracks each worker's liveness *and progress*. A worker that hasn't made progress within the deadline of its current operation is presumed wedged-on-hardware and is killed + respawned; its leases are reclaimed (the device is quarantined, not safe-reset, because reset is exactly what may have hung). This reuses the liveness machinery already built for the *upward* boundary — symmetry worth noting.

> Performance note: deadlines cost nothing on the success path (a deadline that isn't hit is free). This hardening does **not** tax the hot path — and the hot path is the *use* path (direct fd), which doesn't cross the seam at all (G4). So S2 hardening is entirely on the cold control path. Reliability bought at zero performance cost — the priority order's ideal case.

## 2.3 🟠 MAJOR — "never lie" needs post-condition verification (D6)

**Finding.** G2 says the model never lies, and the contract says `reset` must "achieve safe state or fail honestly." But a backend can be *honestly fooled*: the kernel returns success from `VFIO_DEVICE_RESET`, yet the device is still dirty (buggy firmware). The backend reported the truth as it knew it, but the truth was wrong. Under an adversarial-device model, "the device lied to the kernel which lied to us" is in scope.

**Recommendation.** For state-critical operations (reset above all), the contract should *prefer* a verifiable post-condition where the hardware exposes one: read back a register, check a status bit, confirm the device returned to a known-safe config. Where verification is impossible, the resource is marked `fidelity=unknown` for the affected fields (the design already has this vocabulary — use it here) and, for reset specifically, quarantined rather than returned to the `Free` pool until proven clean. This converts "trust the kernel's success" into "verify or quarantine," which is the survivability-grade posture.

## 2.4 🟠 MAJOR — the missing state: `Quarantined`

**Finding.** The state model (Part 2.7) has `Free / Claimed / Reserved / Absent / Faulted`. There is no **`Quarantined`** — a resource that is *not safe to hand out* because an operation on it timed out, its reset is unverified, or it faulted in a way that may have left it dirty. Without it, the only options after a timeout are "return to Free" (dangerous — may be dirty) or "Faulted forever" (wasteful — may be fine after recovery).

**Recommendation.** Add `Quarantined{reason, since}` as a first-class state. A quarantined resource:
- is never granted to any client (isolation: a dirty device can't leak prior tenant state to the next);
- is subject to a recovery policy (escalating reset, cool-down, operator alert);
- returns to `Free` only after a *verified* clean reset (ties to 2.3);
- is reported truthfully to clients as unavailable-with-reason (G2).

This single addition closes the "what do we do after a timeout/lie" question for the whole downward boundary. It is the downward-boundary analog of how the lease engine handles the upward boundary.

## 2.5 🟠 MAJOR — reset-escalation ladder (D11)

**Finding.** Two reset levels exist (`function_level`, `bus_level`) but no policy for *when to escalate*. A device that ignores FLR needs bus-level reset; one that ignores that needs power-cycle or operator intervention. Undefined escalation means a stuck device either hangs (if reset blocks — see 2.2) or silently stays dirty.

**Recommendation.** Define the ladder explicitly: `FLR → (verify) → bus reset → (verify) → quarantine + alert`. Each rung is deadline-bounded (2.2) and verified (2.3). The ladder is policy (Python, cold path), driven by mechanism (Zig, the seam). The device never blocks the broker because each rung is abandonable; it ends in `Quarantined`, never in a hang.

## 2.6 🟠 MAJOR — IOMMU must be enforced and declared, not assumed (D9)

**Finding.** Stage 1 assumes an IOMMU contains rogue DMA. But "assume" is not "enforce." If the broker binds a device for DMA-capable passthrough on a host with no IOMMU (or with it disabled), the device can DMA anywhere in host memory — a total compromise, silently.

**Recommendation.** The backend must, at startup and before any DMA-capable grant, *verify* the IOMMU is present and active, and refuse DMA-capable claims otherwise (returning a typed `Unsupported{reason=no_iommu}`). A no-IOMMU host is a **declared degraded mode**: the broker reports it loudly, and DMA-capable passthrough is disabled, not silently unsafe. Observe/inspect and non-DMA paths can still function. This makes the Stage-1 assumption a *checked precondition*, which is the difference between a safe design and a hopeful one.

> **🟡 Partially closed (2026-05-29) — the checked precondition.** `linux_iommu` resolves a PCI device's IOMMU group from sysfs and yields a DMA-safety `Posture`; `linux_backend.vAcquire` now **refuses** a shared/exclusive (DMA-capable) PCI claim on a device with no IOMMU group — `Err.NotPermitted`, logged — while `observe` (no DMA) is exempt. So the broker no longer *silently* grants DMA-unsafe access; it declares the degraded posture by saying no (and the SDK `doctor` already WARNs on no-IOMMU hosts, `IOMMU_PRESENT` / FEEDBACK LINUX-01). **VFIO bind orchestration shipped (2026-05-30, R5 slice 1).** `vfio.zig` is the enforcement spine: open the container, prove the IOMMU group VIABLE, `SET_CONTAINER` / `SET_IOMMU`, `VFIO_GROUP_GET_DEVICE_FD` — behind an injectable seam, with **atomic errdefer-unwind** so a partial bind leaks no descriptor (Part 3.4), `unbind` teardown, and an honest errno→`Err` map. Wired into `vAcquire` (DMA-capable claims) / `vRelease`; the orchestration + unwind + VIABLE gate are pinned with an injected fake (8 Zig tests, unwind negative-control verified). **Host-driver detach shipped (slice 2):** `detachHostDriver`/`reattachHostDriver` (a `DriverOps` seam) unbind the host driver → `driver_override=vfio-pci` → `drivers_probe`, with **atomic reattach** so a failed detach never orphans the device; `vAcquire` does detach→bind as one atomic operation. **Revoke-below-the-fd + fd delivery shipped (slice 3):** the two previously-`null` PCI vtable slots are filled — `fd_for_token` hands the client the bound VFIO device fd (G4 delegate-usage), and `revoke` is the forced reclaim (§3.4): unbind the VFIO device (closing the container fd drops the IOMMU mappings → the client's dup'd fd is inert) + reattach the host driver, idempotent when nothing is bound. The `vfio`/`driver` seams are now injected on the backend, so the whole acquire→deliver→revoke lifecycle is pinned host-independently with fakes (revoke teardown + idempotency negative-control verified). **Remaining (hardware-gated):** the raw VFIO ioctls + sysfs writes in `realOps`/`realDriverOps` are written to the UAPI but UNVERIFIED without root + a passthrough device — on this sandbox the privileged open fails and the caller returns an honest `Err`, never a faked success. What's left is purely *running* the full lifecycle on a real passthrough device.

## 2.7 🟡 MINOR — interrupt-rate containment (D7)

**Finding.** A malicious device raising continuous interrupts is a use-path DoS against the host. The broker hands off the fd and steps out (G4), so it isn't in the interrupt path — good — but it *granted* the device, so it owns the policy of *whether* to keep it granted under abuse.

**Recommendation.** Where the kernel exposes interrupt counters, the broker (control path, cold) can monitor for runaway interrupt rates on granted devices and revoke + quarantine an abusive one. This is defense-in-depth; the IOMMU and kernel IRQ handling are the first line. Worth a hook, not worth slowing anything.

> **✅ Closed (2026-05-29) — the hook + the real reader.** `irq_monitor.IrqMonitor` (per-device rate policy) + `Core.containByFlooding` (revoke-below-the-fd + quarantine) + `Facade.scanInterruptRates`, reachable via the steal-gated `irq_scan` op / `Client.irq_scan()`. Operator/supervisor-triggered (no background poller, per "worth a hook, not worth slowing anything"). The interrupt-counter source is a seam; the production reader (`linux_irq`) attributes a PCI device's count by summing its `/sys/.../msi_irqs` IRQ rows in `/proc/interrupts` (USB shares the HCD IRQ → null). Policy + containment tested via injection (Zig facade test: flood → revoke+quarantine); the real reader's parsers are unit-tested and verified against live host `/proc/interrupts`.

## 2.8 Stage 2 summary

| Finding | Severity | Closes |
|---|---|---|
| 2.2 deadlines + killable context + watchdog on the seam | 🔴 CRITICAL | D3, D4, the core "never hang" requirement |
| 2.3 post-condition verification for reset | 🟠 MAJOR | D6 (the device-lie case) |
| 2.4 the `Quarantined` state | 🟠 MAJOR | the "after timeout/lie, then what" gap |
| 2.5 reset-escalation ladder | 🟠 MAJOR | D11 |
| 2.6 IOMMU enforced + declared | 🟠 MAJOR | D9 (silent total compromise) |
| 2.7 interrupt-rate containment | 🟡 MINOR | D7 |

**The through-line:** the downward boundary is well-defended against failures that *return* and undefended against failures that *don't return* or that *lie*. The fix is three coupled additions — **deadlines** (time becomes part of the contract), the **`Quarantined` state** (a safe place to put the uncertain), and **verification** (don't trust success) — all on the cold control path, none touching the hot use path. The process model already in the design (§7) is the natural enforcement substrate: a frozen device kills one worker, which respawns, and the device is quarantined. Survivability at zero hot-path cost.

---


**S1 — Upward crash-safety:** no client action can crash, hang, or corrupt the broker. The worst a client can do *to the broker* is lose its own connection.
**S3 — Tenant isolation:** no tenant can read, corrupt, observe beyond policy, or deny service to another tenant, except via explicitly modeled sharing.

This is your second stated example — "the user/client should never be able to cause a crash or security issue." The upward boundary differs from the downward one: here failures are *active and intelligent*, not just random. A hostile client probes for the one unvalidated field, the one unchecked length, the one forgeable token. The standard is therefore not "handle the common bad input" but "every byte is hostile until validated."

## 3.1 The taxonomy of upward attacks

| # | Attack | Example | Current design | Verdict |
|---|---|---|---|---|
| U1 | **Malformed request** | non-JSON, truncated, wrong types (`{"op":123}`) | `parseFromSlice ... catch` handles bad JSON | 🟠 JSON-shape ok; *type* confusion not |
| U2 | **Type confusion** | `op` is a number, `holder` is a string | code does `.string` / `.integer` directly → **panic** | 🔴 **CRITICAL** in skeleton |
| U3 | **Capability forgery** | client sends `cap: 999` it was never granted | caps are bare `u64` ids — guessable/forgeable | 🔴 **CRITICAL** (skeleton); design intends unforgeable |
| U4 | **Capability confusion** | client uses tenant A's cap id from a leaked log | no binding of cap to connection identity | 🔴 **CRITICAL** without authenticator |
| U5 | **Oversized input** | 4GB request line, 10^9 enumerate results | no input-size or result bounds | 🟠 MAJOR (S5) |
| U6 | **Poisoned shared memory** | client mutates ring indices/lengths the broker reads | broker trusts shm fields it reads back | 🔴 **CRITICAL** if broker reads client-writable control fields |
| U7 | **Malicious fd** | client sends a crafted fd via SCM_RIGHTS | broker may receive fds; type/rights unverified | 🟠 MAJOR |
| U8 | **Request flood / DoS** | 10^6 claims/sec, connection storm | no per-tenant rate/quantity limits | 🟠 MAJOR (S5) |
| U9 | **Slowloris** | open connection, send 1 byte/min, hold resources | no per-connection progress deadline | 🟠 MAJOR |
| U10 | **Use-after-release** | release a lease, then use the (cached) fd | fd-passing means client *keeps* the fd; revocation? | 🔴 **CRITICAL** — the revocation gap |
| U11 | **Cross-tenant enumerate** | tenant A lists tenant B's devices/state | enumerate not capability-filtered by default | 🟠 MAJOR (S3) |
| U12 | **TOCTOU on claim** | race the check-then-act in a grant | grant path atomicity (Part 3.4) | 🟢 handled (lease engine) |
| U13 | **Reentrancy / state confusion** | interleave operations to corrupt the table | single-writer core, typed states | 🟢 mostly handled |

The headline: **the lease engine's *logic* is sound (U12/U13 handled), but the *boundary* — parsing, capability authenticity, shared-memory trust, and revocation — is where the survivability gaps cluster.** A clean core behind an unvalidated boundary is not survivable.

## 3.2 ✅ CLOSED (was 🔴 CRITICAL) — every input field must be validated, not accessed (U1/U2)

> **Status: SHIPPED (commit `45efd04`).** The trust gate now lives at
> `hrb-code/src/request.zig`; every byte from the client passes through
> typed extractors before becoming a `Request`. The dispatcher in
> `server.zig` never reads `std.json.Value`. Pinned by
> `request_test.zig` — 29 cases covering type confusion, missing fields,
> oversized input, out-of-range integers, plus a 7 000-iteration
> deterministic-seed fuzzer asserting random bytes only ever yield
> `BadRequest` or a valid `Request`, never a panic. The original
> finding and recommendation are preserved below as the rationale.

**Original finding.** The skeleton server does `obj.get("op").?.string` and `.integer`. If a client sends a value of the wrong JSON type, Zig's tagged-union access panics — a client-triggered crash. This is the most basic S1 violation and, in the skeleton, a live one.

**Recommendation — a hardened request-decode layer that is the *only* way bytes become typed requests:**
- Parse into a strict schema with **typed extractors** that return `Err.BadRequest` on any mismatch — never a direct `.string`/`.integer` access anywhere in the request path. (`getString(obj, "op") orelse return BadRequest`.)
- Validate **before** dispatch: op is a known verb, every required field present and correctly typed, every value within bounds (urn length, mode is a valid enum, ints in range).
- Treat decode as a **trust-establishing gate**: nothing past the decoder ever sees raw client bytes; everything past it is a validated typed request. This is the single highest-leverage S1 fix.
- This is contract-testable: a fuzzer throwing random/malformed bytes at the decoder must *never* produce anything but `BadRequest` or a clean drop. Add it to the S1 conformance suite.

## 3.3 🔴 CRITICAL — capabilities must be unforgeable and connection-bound (U3/U4)

**Finding.** In the skeleton, a capability is a plain `u64` id; a client can present any number. The *design* (Part 4) intends unforgeable capabilities, but the mechanism must be pinned down, because this is the entire security model. Two distinct problems:
- **Forgery (U3):** a guessable/sequential id can be presented by a client that was never granted it.
- **Confusion (U4):** even an unguessable id, if leaked (logs, error messages, a compromised tenant), can be replayed by another tenant — unless the capability is *bound to the connection/identity* that holds it.

**Recommendation — two layers, both required:**
1. **Unforgeable handles.** A capability presented over the socket must be either (a) a kernel-backed reference the client cannot fabricate — an **fd to the capability object**, passed via SCM_RIGHTS, so possession is proof and the kernel guarantees authenticity — or (b) a cryptographically authenticated token (HMAC/MAC over the capability contents with a broker-held key) that the broker verifies and that cannot be forged or widened. The fd approach is preferred: it makes "holding the capability" and "holding the fd" the same fact, and aligns with the future kernel making capabilities native.
2. **Connection binding via `SO_PEERCRED`.** On a local Unix socket, the broker reads the connecting process's credentials (uid/pid) from the kernel — unforgeable. A capability is bound to the identity that holds it; presenting another tenant's leaked id fails because the connection credential doesn't match. (The design's near-settled authenticator default — `SO_PEERCRED` local / mTLS remote — is exactly right; this review elevates it from "default" to "load-bearing security control, must be present.")

This closes the gap between "the design intends unforgeable capabilities" and "the mechanism guarantees it."

## 3.4 🔴 CRITICAL — the revocation gap: use-after-release with a held fd (U10)

**Finding.** The design's great performance win is "mediate ownership, delegate usage": the broker hands the client an fd and steps out (G4). The fd-passing review (Phase 7) *proved* the dividend — the fd keeps working after the broker closes its copy. **That same property is a security gap on release/revocation:** if a client's lease is revoked (it died, was preempted, violated policy, or its device was reclaimed for a higher-priority tenant), the client *still holds a working fd*. It can keep touching the hardware after losing the right to.

This is the deepest tension in the whole design — the performance model (broker not in the use path) directly conflicts with the security need (revoke access instantly). It deserves a first-class answer, not a patch.

**Recommendation — revocation must be enforced *below* the fd, where the broker still has authority:**
- **IOMMU/mapping revocation.** The durable handle is the IOMMU mapping, not the fd. On revocation, the broker (via the privileged helper) tears down the device's IOMMU mappings and DMA authority. The client's fd becomes inert — reads/writes fault — because the *kernel-level binding* behind it is gone. The fd is a straw; pulling the mapping pulls what the straw drinks from.
- **This is exactly a place the future kernel does better** (kernel-design-brief): native capability revocation invalidates the handle itself, no IOMMU teardown dance. On Linux, IOMMU/VFIO teardown is the mechanism; document it as the Linux approximation of native revocation.
- **Define the revocation contract** at the seam: `revoke(token)` must render all delegated access inert within a bounded time, and is itself deadline-bounded (Stage 2) and verified. Revocation that "asks nicely" is not revocation; revocation that "pulls the mapping" is.
- **Graceful vs. forced.** Graceful revocation signals the client (return the device, you have N ms); forced revocation tears down the mapping. The design's revocation model (Family A) gains a forced tier here. Family B (Part 10) already has graduated reclaim — the symmetry is good.

This single finding may be the most consequential in the review: it reconciles the performance model with the security model by putting revocation *below* the delegated handle, not in front of it.

## 3.5 🔴 CRITICAL — shared memory: the broker must never trust client-writable fields (U6)

**Finding.** The data plane uses shared-memory rings (Part 5.3, the 822M-items/sec ring). If the broker and client share a ring and the *broker reads control fields (head/tail/length) that the client can write*, a malicious client can write out-of-range indices and walk the broker out of bounds — a classic shared-memory escape leading to broker memory corruption.

**Recommendation — asymmetric trust in shared memory:**
- **The broker reads only fields it owns.** In an SPSC ring where the broker is consumer, the client (producer) writes `head`; the broker writes `tail`. The broker must treat the client-written `head` as *untrusted*: every index is masked to the ring size and bounds-checked before use, so a poisoned `head` can at worst cause the broker to read a stale/garbage *slot* (harmless — it's validated payload), never an out-of-bounds access.
- **Payloads are validated, not trusted.** Anything the broker reads from a client-written slot is a request to be validated by the decode gate (3.2), not a structure to be dereferenced.
- **Separate mappings for separate trust.** Control fields the broker must trust live in broker-only memory; only the data payload area is client-writable. Never share a struct where the broker dereferences a client-written pointer/length without validation.
- This is contract-testable: a test that scribbles garbage into every client-writable shm field and asserts the broker never faults or reads OOB.

## 3.6 🟠 MAJOR — capability-filtered enumeration (U11, S3)

**Finding.** `enumerate` returns resources matching a scope. If it returns *all* resources regardless of the caller's capability, tenant A learns about tenant B's devices and their ownership state — an isolation leak (S3), even if A can't claim them.

**Recommendation.** Enumeration is filtered by the caller's capability scope: a tenant sees only resources its capability could name. This makes the namespace itself capability-scoped — A cannot even *observe the existence* of resources outside its grant. (Observe-mode within scope is fine; cross-scope visibility is the leak.) Cheap, on the cold path, closes the enumeration side channel.

## 3.7 🟠 MAJOR — input, result, and connection bounds (U5/U8/U9, S5)

**Finding.** No stated limits on request size, result count, request rate, connection count, or per-connection progress. Each is a DoS vector: a giant request exhausts memory; a 10^9-result enumerate exhausts memory/time; a flood saturates a worker; a slowloris holds resources forever.

**Recommendation — bound everything, per-tenant and in aggregate (this is S5 made concrete):**
- max request size (reject oversized lines before buffering);
- max results per enumerate (paginate);
- per-tenant rate limit on control-path ops (token bucket; cold path, cheap);
- max concurrent connections per tenant and in aggregate;
- per-connection progress deadline (a connection that doesn't make progress in N seconds is dropped — kills slowloris);
- per-tenant caps on live leases, fds, and table entries (no tenant can fill the ownership table).
These belong in policy (Python, cold path) with mechanism hooks in the worker. None touches the hot path.

## 3.8 🟠 MAJOR — fd validation on receipt (U7)

**Finding.** Where clients send fds (if any inbound fd-passing exists), a crafted fd (wrong type, unexpected rights) must not be trusted.

**Recommendation.** Validate received fds: confirm type (fstat), confirm it's the expected kind, never assume an inbound fd is benign. Prefer a design where clients *receive* fds (devices) and rarely *send* them, minimizing this surface.

## 3.9 Stage 3 summary

| Finding | Severity | Closes |
|---|---|---|
| 3.2 typed-extractor decode gate | 🔴 CRITICAL | U1, U2 (input crashes) |
| 3.3 unforgeable + connection-bound capabilities | 🔴 CRITICAL | U3, U4 (forgery/confusion) |
| 3.4 revocation below the fd (IOMMU teardown) | 🔴 CRITICAL | U10 (use-after-release) — *the deepest finding* |
| 3.5 asymmetric shm trust | 🔴 CRITICAL | U6 (shm escape) |
| 3.6 capability-filtered enumerate | 🟠 MAJOR | U11 (isolation leak) |
| 3.7 bound input/result/rate/connection | 🟠 MAJOR | U5, U8, U9 (DoS) |
| 3.8 fd validation on receipt | 🟠 MAJOR | U7 |

**The through-line:** the lease engine's *logic* is already survivable (atomic grant, typed states, single-writer). The gaps are all at the *membrane*: bytes must be validated before they're trusted (3.2), capabilities must be unforgeable and bound (3.3), shared memory must be read with asymmetric trust (3.5), and — the deepest one — **revocation must reach below the delegated fd** (3.4), because the very performance trick that makes the broker fast (stepping out of the use path) is what makes instant revocation hard. The answer is to revoke the *mapping*, not the *fd* — which is also precisely where the future kernel will do better natively.

---


**S4 — Self-fault containment:** a bug, panic, or resource exhaustion in any broker component is contained to the smallest blast radius (one bus worker), recovers automatically, and never escalates to total failure or a reliability lie.

The boundaries (Stages 2–3) defend against the outside world. This stage asks: **when the broker itself fails — and it will — does the failure stay small and never corrupt the truth?**

## 4.1 The taxonomy of self-faults

| # | Self-fault | Current design | Verdict |
|---|---|---|---|
| I1 | **One bus worker panics/crashes** | per-bus worker processes (§7); supervisor respawns | 🟢 strong bones; recovery contract needs pinning |
| I2 | **The facade crashes** | facade holds connections, no ownership table (§7.3) | 🟠 client connections drop; reconnect/state-rebuild unspecified |
| I3 | **The privileged helper crashes** | smallest TCB (§11) | 🟠 most dangerous to lose; restart + re-attest unspecified |
| I4 | **Ownership table corrupted** | single-writer core, typed states | 🟠 no integrity check / no crash-consistent persistence |
| I5 | **Memory exhaustion in a worker** | not bounded | 🟠 unbounded growth → OOM-kill → see I1 |
| I6 | **Panic mid-grant** | errdefer-unwind (Part 3.4) | 🟢 atomicity handles the half-grant |
| I7 | **Lease table lost on worker restart** | leases bound to liveness; clients reconnect | 🟠 must rebuild from authoritative source, not guess |
| I8 | **Deadlock between components** | facade↔worker↔helper messaging | 🟠 no stated lock-ordering / deadlock-freedom argument |

## 4.2 🟢 STRENGTH — the process model is the right blast-radius container

Per-bus worker *processes* mean a panic in the PCI worker cannot corrupt the USB worker — separate address spaces. This is the correct, hard choice, already made. Combined with Stage 2's watchdog, a frozen-or-crashed worker is killed and respawned without touching siblings or the facade. **Preserve this; it is the foundation of S4.**

## 4.3 🟠 MAJOR — define the worker-restart recovery contract (I1/I7)

**Finding.** "Supervisor respawns the worker" is stated, but *what state the respawned worker comes up with* is not. Leases are bound to liveness — but the lease *table* lived in the dead worker. On restart, the worker must reconstruct the truth, and the truth must come from an authoritative source, not be guessed.

**Recommendation.** Define recovery precisely:
- The respawned worker **re-enumerates hardware** (the kernel/IOMMU is the ground truth for what's bound) and **re-derives ownership** from surviving client connections (each client re-presents its capability and lease; the facade holds the connection set). The reconstructed table is the *intersection* of "what the hardware says is bound" and "what live clients can prove they hold."
- Anything bound in hardware but unclaimed by any live client after restart is **quarantined** (Stage 2.4), then safe-reset and freed — never assumed free, never assumed owned.
- This makes worker restart *crash-consistent*: the post-restart truth is provably correct, derived from two authoritative sources, never from lost in-memory state. (This is the upward-G3 and downward-quarantine machinery composing — a good sign the model is coherent.)

> **Progress (2026-05-29) — the "what the hardware says is bound" half is now real for PCI.** `R-T1-8-RESTART-WIRING` wired `reconcileAfterRestart` (the present-and-unclaimed → quarantine sweep, the second bullet); this pass replaces the present-set proxy with the genuine bound signal: `linux_bound` classifies a PCI device as an orphan candidate iff it is bound to `vfio-pci` (the broker's passthrough binding, which survives a crash), sparing host-driver-bound and unbound devices. So reconcile no longer over-quarantines idle / host-used hardware. And broker self-restart **detection** now closes the loop: `restart_marker` (a one-byte marker beside the socket — a host reboot clears tmpfs → cold start, a crash leaves it → restart) makes the broker AUTO-reconcile on a detected restart with a STRICT bound source (confident `vfio-pci` orphans only), so re-derivation is now unattended rather than operator-triggered. Still future: the USB bound signal (a detached interface is merely driverless — ambiguous), exercising a REAL vfio-pci orphan end-to-end (a passthrough device — the #7 VFIO gate), and a `SIGTERM` handler so a clean stop clears the marker (today it reads as a restart next boot — harmless, since the auto-reconcile is strict).

## 4.4 🟠 MAJOR — the privileged helper must restart and re-attest (I3)

**Finding.** The privileged helper (§11) is the smallest, most-trusted component. If it crashes, the worker loses its only path to privileged operations. Restarting it is more delicate than a worker because it holds the raw privilege.

**Recommendation.** The helper is stateless-by-design where possible (it performs operations on request and holds no durable authority of its own beyond the live mappings). On restart: the worker re-establishes the helper channel, the helper re-attests (confirms it is the genuine helper, e.g. via a kernel-verified credential), and any in-flight privileged op that was interrupted is treated as failed → the affected resource is quarantined. The helper never comes back assuming prior state; it rebuilds from the worker's authoritative request stream.

> **🟡 Partially closed (2026-05-29) — the in-flight-op quarantine.** The "any in-flight privileged op that was interrupted → quarantine the affected resource" half is now wired for the client-facing helper-touching ops: `core.quarantineOnHelperFault` quarantines on an `Err.Timeout` (the Tier-0 #3 supervisor killed the helper mid-op → state unknown), already done for `acquire`, now extended to `raw` (`core.raw`) and `control` (the new `core.control`, which `facade.control` routes through); `reset` is covered by `recoverByReset`. Pinned with a fault-injecting test backend (Timeout → quarantine; clean error → not). The helper RE-ATTEST half (kernel-verified credential on respawn) remains future. Residual sliver: `release()` surfaces no error (`void`) + the U7 bad-fd `HardwareFault` path.

## 4.5 🟠 MAJOR — ownership-table integrity and crash-consistency (I4)

**Finding.** The ownership table is the one structure that must never be silently wrong (a corrupted entry is a reliability lie — the worst outcome per §1.1). There's no integrity check or crash-consistent representation specified.

**Recommendation.**
- **In-memory integrity:** the table is single-writer (good); add cheap invariant checks at transitions (a resource is in exactly one state; a lease references a live client; no two exclusive leases on one resource). A failed invariant is a *controlled* fault → quarantine the affected resource + alert, never a silent bad grant.
- **Crash-consistency:** the table need not be persisted (it's reconstructable per 4.3), but the *reconstruction* path must be the only source of post-crash truth. Persisting would risk persisting *corruption*; reconstruction from hardware + live clients is safer. This is a deliberate "don't persist, re-derive" choice — record it.

## 4.6 🟠 MAJOR — bound worker memory; OOM is a containment event, not a crash (I5)

**Finding.** Unbounded per-worker memory (lease entries, pending requests, event buffers) lets a flood (Stage 3.7) grow a worker until the OOM killer takes it — an *uncontrolled* version of I1.

**Recommendation.** Every per-worker allocation that scales with client activity is bounded (S5): max leases, max pending requests, bounded event ring (already bounded — the ring drops-and-counts). When a bound is hit, the worker rejects new work with a typed `Overloaded` error (back-pressure), never grows unboundedly. OOM becomes impossible-by-construction rather than a containment-of-last-resort.

## 4.7 🟡 MINOR — deadlock-freedom argument (I8)

**Finding.** Three communicating process types (facade, worker, helper) with request/response messaging can deadlock if a cycle of waits forms. No explicit ordering/freedom argument exists.

**Recommendation.** State a simple invariant: messaging is strictly hierarchical (facade → worker → helper, never upward-blocking), all waits are deadline-bounded (Stage 2), and a deadline expiry breaks any incipient deadlock by abandoning the wait. Deadline-bounded hierarchical messaging is deadlock-free by construction. Document it.

> **✅ Closed (2026-05-29).** Invariant written out in `usb_backend_via_ipc.zig` (`setRecvTimeout` doc-comment): the helper is a pure request/reply responder — it never initiates a call back into the broker, so the wait graph facade → worker → helper is acyclic by construction; the only residual risk (a wedged-but-alive helper) is removed by the deadline-bounded recv (`SO_RCVTIMEO` → `recvWithSupervisor` kill+respawn, Tier-0 #3). Pinned by a regression test asserting `SO_RCVTIMEO` is armed at a non-zero timeout.

## 4.8 Stage 4 summary

| Finding | Severity | Closes |
|---|---|---|
| 4.2 process model as blast container | 🟢 STRENGTH | preserve |
| 4.3 worker-restart recovery contract | 🟠 MAJOR | I1, I7 |
| 4.4 helper restart + re-attest | 🟠 MAJOR | I3 |
| 4.5 table integrity + re-derive-don't-persist | 🟠 MAJOR | I4 |
| 4.6 bound worker memory + back-pressure | 🟠 MAJOR | I5 |
| 4.7 deadlock-freedom invariant | 🟡 MINOR | I8 |

**Through-line:** the process model gives the right blast radius for free. The work is making *recovery* as rigorous as the design's *steady state* — every component must come back from a crash deriving truth from authoritative sources (hardware + live clients), never from lost memory, with anything uncertain quarantined. Recovery composes the upward (liveness) and downward (quarantine) machinery, which is evidence the model is internally coherent.

---

# Stage 5 — Concurrency & Multi-Tenant Fairness, audited against S3 + S5

**S3 — tenant isolation; S5 — bounded everything.** Your requirement: many customers run apps on top, *competing and accessing resources concurrently*. This stage audits the design as a contended, multi-tenant system, not a single-user one.

## 5.1 The taxonomy of contention pathologies

| # | Pathology | Current design | Verdict |
|---|---|---|---|
| C1 | **Race on claim** (two tenants, same device) | atomic grant, claim lattice (Part 3.4) | 🟢 handled |
| C2 | **Starvation** (tenant A never gets a contended device) | no fairness/queueing policy | 🟠 MAJOR |
| C3 | **Priority inversion** (low-pri holds what high-pri needs) | no priority/preemption model | 🟠 MAJOR |
| C4 | **DoS by contention** (tenant spams claims to lock others out) | rate limits (Stage 3.7) help; no fairness | 🟠 MAJOR |
| C5 | **Noisy-neighbor on the control path** (one tenant floods the facade) | per-tenant rate limit (3.7) | 🟠 closeable |
| C6 | **Shared-memory timing side channel** (tenant A infers B's activity) | shared rings per-tenant? unspecified | 🟠 MAJOR (S3) |
| C7 | **Lock contention in the core** (single-writer becomes a bottleneck) | single-writer core | 🟡 perf, not safety |
| C8 | **Fairness across buses** (one worker starved by another) | per-bus workers are independent | 🟢 handled |
| C9 | **Concurrent revocation vs. use** (revoke races a client's use) | ties to revocation (3.4) | 🟠 must be atomic |

## 5.2 🟢 STRENGTH — claim atomicity and per-bus independence

C1 (the fundamental race) is already correct: the lease engine's atomic grant means two tenants racing for one exclusive device produce exactly one winner, no torn state. And per-bus workers mean contention on PCI never affects USB latency (C8). The *mechanical* concurrency safety is sound.

## 5.3 🟠 MAJOR — fairness and anti-starvation policy (C2/C4)

**Finding.** Atomicity guarantees *a* winner, not a *fair* winner. Under contention, the current model is first-come-first-served with no anti-starvation: a tenant repeatedly claiming/releasing a device can lock a rival out indefinitely. For "many customers competing," fairness is a reliability property (a tenant that never makes progress is, to that tenant, an outage).

**Recommendation.** Introduce a **contention policy** at the (cold) grant path:
- a **fair queue** per contended resource (FIFO or weighted-fair), so a waiting tenant eventually wins;
- **hold-time limits** on contended resources (a lease on a contended device has a max duration before it must be renewed, giving the queue a turn);
- the policy is pluggable (DIP — the core calls a `ContentionPolicy` abstraction), so FCFS / fair-queue / priority are swappable without core changes.
This is cold-path (only engaged under contention) and turns "atomicity" into "atomicity + fairness."

> **🟡 Partially closed (2026-05-29) — the anti-starvation core.** `fairness.zig` (`Fairness`) is the contention policy the core consults at the claim + release seams: a per-resource FIFO wait queue keyed on the SO_PEERCRED uid, plus a **reservation-on-release** — a freed contended resource is reserved for its longest-waiting tenant for a grace window, so a spammer can't lock out a waiter (the C2/C4 closure). Wired into `claimImpl`/`release`; pinned end-to-end (a spammer racing every release still loses to the fair waiter). The §5.4 priority-ordered admission half then shipped 2026-05-30 (see §5.4). **Gated remainder:** weighted-fair queues, ACTIVE preemption of a live holder (= forced revocation, §5.4), hold-time limits, and tuning of the grace window — all need real multi-tenant traffic or a mechanism not yet built. `Fairness` is the swap point (the core calls it through one surface), so those slot in without touching the claim path.

## 5.4 🟠 MAJOR — priority and preemption (C3/C9)

**Finding.** No priority model means a low-priority tenant can hold a device a high-priority tenant urgently needs (classic priority inversion). And preemption, if added, races with the holder's in-use fd (C9) — which is the revocation problem (3.4) again.

**Recommendation.**
- Add an optional **priority/preemption tier** in the contention policy: a higher-priority claim can preempt a lower one, subject to policy.
- Preemption *is* forced revocation (3.4): graceful first (signal the holder, grace period), then forced (tear down the IOMMU mapping). Because revocation is defined to render access inert atomically (3.4) and is deadline-bounded (Stage 2), preemption is safe — the preempted tenant's fd goes inert before the new tenant's claim completes. **C9 is closed by the same mechanism as U10** — a sign the design is converging on the right primitive.
- Family B (Part 10) already has graduated reclaim; this brings Family A to parity.

> **🟡 Partially closed (2026-05-30) — priority-ordered admission.** `fairness.zig`
> gained a `PrioritySource` seam (uid → priority, injected so the priority VALUES
> stay policy, not mechanism) and `onRelease` now reserves a freed contended
> resource for the HIGHEST-priority waiter (longest-waiting within a priority)
> instead of strictly the FIFO front. This closes the priority-inversion-among-
> **waiters** half of C3: a high-priority tenant no longer queues behind
> lower-priority ones. With no source configured every waiter is priority 0, so
> the selection collapses to the original pure FIFO — exact back-compat (the C2/C4
> pins are unchanged), and the core leaves the source unset today (no priority
> policy is wired yet). Pinned: priority jumps the queue, equal-priority falls
> back to FIFO, highest-of-several wins, and no-source == FIFO. **Gated remainder:**
> ACTIVE preemption of a live lower-priority HOLDER (C3's other half / C9) — per
> the recommendation above this *is* forced revocation (graceful signal + grace
> period, then forced IOMMU/VFIO teardown), so it waits on the holder-signal
> channel + #7's VFIO teardown; plus weighted-fair queues, hold-time limits, and
> the actual uid→priority policy values. `Fairness` stays the single swap point.

## 5.5 🟠 MAJOR — shared-memory tenant isolation (C6, S3)

**Finding.** If tenants share any memory region (a common ring, a shared event buffer), one can infer another's activity via timing, or worse, read residual data. Per-tenant memory is implied but not stated.

**Recommendation.** **Each tenant gets its own private shared-memory region with the broker** — no ring is shared *between tenants*, only between a tenant and the broker. Regions are zeroed before reuse (no residual data across tenants — ties to the `Quarantined`/safe-reset discipline). This makes cross-tenant shm side channels structurally absent rather than mitigated. Cost is per-tenant memory (bounded by S5), which is acceptable for isolation.

## 5.6 🟡 MINOR — core write-path scalability (C7)

**Finding.** A single-writer core is correct but could bottleneck under extreme control-path concurrency (many tenants claiming simultaneously). This is a *performance* concern, not safety (priority #2, not #1).

**Recommendation.** Per-bus workers already shard the write path by bus (the natural partition). If a single bus's grant rate ever bottlenecks, the §9.7 measurement questions (RCU vs. seqlock for the read path) apply — but only on profiling evidence. Not a survivability issue; noted for completeness.

## 5.7 Stage 5 summary

| Finding | Severity | Closes |
|---|---|---|
| 5.2 atomic grant + per-bus independence | 🟢 STRENGTH | C1, C8 |
| 5.3 fairness / anti-starvation policy | 🟠 MAJOR | C2, C4 |
| 5.4 priority + preemption (= forced revocation) | 🟠 MAJOR | C3, C9 |
| 5.5 per-tenant private shm | 🟠 MAJOR | C6 |
| 5.6 write-path scalability | 🟡 MINOR | C7 (perf) |

**Through-line:** mechanical concurrency safety is already correct; what's missing is *policy* — fairness, anti-starvation, priority — which turns "exactly one winner" into "the right winner, and everyone eventually progresses." And preemption reduces to forced revocation (3.4), so the multi-tenant fairness story and the security revocation story are *the same mechanism* viewed from two sides. Per-tenant private shared memory makes cross-tenant side channels structurally absent. All of it is cold-path policy behind abstractions (DIP) — no hot-path cost, swappable, OCP-clean.

---


Five stages produced ~30 findings. This stage does the most valuable thing a review can: show that they are not 30 separate patches but a **small number of load-bearing primitives**, each of which closes many findings at once. A design hardens best when a few right mechanisms dissolve many problems — and that is what the findings reveal.

## 6.1 The convergence — four primitives close most of the review

```
   FINDINGS (across all stages)              CONVERGE ONTO              PRIMITIVE
   ─────────────────────────────             ─────────────              ─────────
   D3 frozen device hangs worker  ┐
   D4 slow-but-eventual           ├──────────────────────────────►  ① DEADLINES
   D11 reset-escalation timing    │                                   (time in the
   I8 deadlock-freedom            │                                    seam contract)
   C9 preemption timing           ┘

   D6 device-lie / unverified reset ┐
   D4 timed-out reset (dirty?)      ├────────────────────────────►  ② QUARANTINE
   I3 helper crash mid-op           │                                   (a safe place for
   I4 table integrity violation     │                                    the uncertain)
   4.3 unclaimed-but-bound on restart┘

   U10 use-after-release (held fd)  ┐
   C3 priority inversion            ├────────────────────────────►  ③ REVOCATION
   C9 preemption vs. in-use fd      │                                  BELOW THE FD
   5.4 forced preemption            ┘                                  (pull the mapping,
                                                                        not the fd)

   U1/U2 input crashes              ┐
   U3/U4 capability forgery         ├────────────────────────────►  ④ THE TRUST GATE
   U6 shm escape                    │                                  (nothing is trusted
   U7 malicious fd                  │                                   until validated at
   U11 cross-tenant enumerate       ┘                                   a single membrane)
```

Plus two cross-cutting disciplines that aren't single primitives but apply everywhere:
- **⑤ Bounded everything (S5):** every client-driven resource is capped per-tenant and in aggregate. Closes U5, U8, U9, I5, C4. The mechanism is back-pressure (`Overloaded`), not unbounded growth.
- **⑥ Re-derive, don't persist:** every component recovers by reconstructing truth from authoritative sources (hardware + live clients), never from lost memory. Closes I1, I2, I7, and makes recovery crash-consistent.

**The deep insight:** primitives ① and ③ also show up as the bridge between *survivability* and *the future kernel*. Deadlines and revocation-below-the-fd are exactly what Linux makes *hard* (timeouts bolted on, IOMMU teardown to invalidate a delegated fd) and what a capability-native kernel makes *easy* (native time-bounded operations, native handle revocation). The hardening this review demands on Linux is, simultaneously, the specification of two more features for `mgf-hrb-youros`. **The survivability review feeds the kernel design.**

## 6.2 The three new state/contract additions (the concrete design deltas)

The review's findings reduce to a small set of additions to the *existing* design — not a redesign:

1. **`Quarantined{reason, since}` state** added to the resource state model (Part 2.7). The safe home for anything uncertain — timed-out, unverified-reset, faulted-maybe-dirty, bound-but-unclaimed-after-restart. Never granted; recovered via verified reset; reported truthfully (G2).

2. **Deadline on every seam operation** (Part 6 / kabi). Time becomes part of the LSP behavioral contract: every backend op completes, errors, or is abandonable within its deadline. `Err.Timeout` is a typed outcome. Enforced by the per-bus-worker watchdog (reusing liveness machinery).

3. **`revoke(token)` contract that renders delegated access inert** (Part 3 / kabi), implemented on Linux as IOMMU/VFIO mapping teardown via the privileged helper, deadline-bounded and verified. This is the single most consequential addition: it reconciles the performance model (broker out of the use path) with the security model (instant revocation) and simultaneously enables preemption/fairness.

Plus the membrane: **a single hardened decode/trust gate** (Part 8 client surface) through which all client bytes pass and beyond which everything is validated and typed — capabilities unforgeable (fd-backed + `SO_PEERCRED`-bound), shm read with asymmetric trust, inputs bounded.

## 6.3 The two new guarantees to promote to first-class (alongside G1–G4)

The review recommends elevating, with their own conformance suites:
- **S-suite (S1–S5):** survivability guarantees, tested by adversarial conformance tests — a fuzzer at the decode gate (S1), a frozen-device fake at the seam (S2), a cross-tenant probe (S3), fault-injection per component (S4), a flood/exhaustion harness (S5). Just as the kabi conformance suite made *LSP-under-failure* testable, the S-suite makes *survivability* testable rather than hoped-for. **This is the meta-recommendation of the whole review:** make survivability a property you *verify*, not one you *intend*.

## 6.4 The prioritized roadmap

Ordered by (severity × centrality), with the convergence in mind so each item closes many findings:

### Tier 0 — must exist before this is anyone's standard (the 🔴s)
| # | Action | Closes | Primitive |
|---|---|---|---|
| 1 | **Decode/trust gate**: typed extractors, no raw field access, validate-before-dispatch, fuzz-tested | U1, U2 | ④ |
| 2 | **Unforgeable + bound capabilities**: fd-backed handles + `SO_PEERCRED` connection binding | U3, U4 | ④ |
| 3 | **Deadlines on the seam** + per-worker watchdog + `Err.Timeout` | D3, D4 | ① |
| 4 | **`revoke()` below the fd**: IOMMU teardown, deadline-bounded, verified | U10, C3, C9 | ③ |
| 5 | **Asymmetric shm trust**: bounds-check every client-written field; per-tenant private regions | U6, C6 | ④ |

### Tier 1 — required for the survivability bar (the 🟠s that close core gaps)
| # | Action | Closes | Primitive |
|---|---|---|---|
| 6 | **`Quarantined` state** + verified-reset recovery | D6, I4, 4.3 | ② |
| 7 | **IOMMU enforced + no-IOMMU declared degraded** | D9 | ④/safety |
| 8 | **Worker/helper restart recovery contract** (re-derive, don't persist) | I1, I3, I7 | ⑥ |
| 9 | **Bound everything + back-pressure** (`Overloaded`) | U5, U8, U9, I5, C4 | ⑤ |
| 10 | **Fairness/anti-starvation + priority/preemption policy** (pluggable) | C2, C3, C4 | ③ + policy |
| 11 | **Capability-filtered enumeration** | U11 | ④ |
| 12 | **Reset-escalation ladder** (FLR→bus→quarantine, each verified) | D11 | ①+② |

### Tier 2 — defense-in-depth (the 🟡s)
| # | Action | Closes |
|---|---|---|
| 13 | Interrupt-rate containment hook | D7 |
| 14 | Deadlock-freedom invariant (hierarchical + deadline-bounded messaging) | I8 |
| 15 | fd validation on receipt | U7 |
| 16 | Table invariant checks at transitions | I4 |

### Throughout — the meta-action
| # | Action |
|---|---|
| ★ | **Promote S1–S5 to first-class guarantees with an adversarial S-conformance-suite.** Every Tier-0/1 item gets a test in it. This is what converts "we intend to be survivable" into "we verify we are." |

## 6.5 What the review did *not* find — the design's validated strengths

It is as important to say what holds as what doesn't. These are correct and should not be churned:
- **The priority order** (reliability > performance > …) is the right backbone and made every trade-off in this review decidable.
- **The lease engine's logic** — atomic grant, leases-bound-to-liveness (G3), typed states — is survivable as designed. The gaps were at the membrane, not the core.
- **The process model** (per-bus workers, minimized privileged helper) is *exactly* the right blast-radius container, and made S4 mostly a matter of pinning recovery contracts rather than restructuring.
- **G1–G4** are sound; the review *adds* S1–S5 rather than correcting G1–G4.
- **"Mediate ownership, delegate usage"** is the right performance model — the review doesn't weaken it; it completes it, by defining how revocation reaches below the delegated handle.
- **The kabi conformance-suite pattern** is the model for how to make the new S-guarantees testable. The design already knew how to make a property verifiable; we apply that same move to survivability.

## 6.6 The performance verdict — does hardening cost peak performance?

**No, and this is structural, not lucky.** Every Tier-0/1 hardening lands on the **cold control path** (claim, release, revoke, enumerate, restart). The **hot path is the use path** — direct fd, broker absent (G4) — and *nothing in this review touches it*. Deadlines are free when not hit; quarantine is a state, not a check; revocation is a rare event; the trust gate runs once per request, not per hardware access; bounds are counters. The priority order's ideal — reliability gained at zero performance cost — holds because the architecture already separated the slow ownership path from the fast use path. **The very design decision that makes the broker fast (G4) is what makes it cheap to harden.** Survivability and peak performance are not in tension here; they are both consequences of the same control/use split.

---

## One-paragraph executive summary

The broker's *core* (lease engine, process model, priority order, fidelity guarantees) is soundly designed for survivability; the gaps are at its two adversarial membranes and in its recovery contracts. Thirty findings converge onto four primitives — **deadlines** (time in the seam contract, so a frozen device can never hang the broker), **quarantine** (a safe state for anything uncertain, so a dirty device never leaks to the next tenant), **revocation below the fd** (tear down the IOMMU mapping, not the descriptor — reconciling the performance model with instant revocation, and doubling as preemption for fairness), and **a single trust gate** (every client byte validated, capabilities unforgeable and connection-bound, shared memory read with asymmetric trust) — plus two disciplines, **bound everything** and **re-derive don't persist**. All hardening sits on the cold path; the hot use path is untouched, so peak performance survives intact. The meta-recommendation is to promote survivability to five first-class, conformance-tested guarantees (S1–S5) peer to the existing G1–G4 — converting survivability from an intention into a verified property. And notably, the two hardest Linux hardenings (deadlines, revocation-below-the-fd) are precisely what the future capability-native kernel makes easy — so this review also specifies two more features of `mgf-hrb-youros`.
