# Self-Aware Hardware Introspection — Current-State Analysis (HISTORICAL)

> **⚠ STATUS: HISTORICAL — superseded by the Part 13 shipped status (2026-05-26).**
>
> This file was the **pre-implementation** gap inventory written on
> 2026-05-26 BEFORE Part 13 §9's six phases shipped. The verdict
> table (§2 below) was correct at write-time; almost every row has
> since been closed. Do NOT use this file as a current-state
> snapshot — read it as the "what we were missing before we built
> it" record.
>
> **What actually shipped (per Part 13 §9 tracker):**
> - Scenario 1 (multi-client concurrent ops): always ✅; subscribe with
>   per-peer predicate filters now adds the missing event surface.
> - Scenario 2 (driver introspection): ✅ — `inspect` carries
>   `driver_name` + `driver_module` (`5f9c4c6`); `driver_info(driver)`
>   wire op (`2348f00`).
> - Scenario 3 (driver lifecycle events): ✅ — `driver_bound` /
>   `driver_unbound` event kinds from netlink uevents (`00a9120`).
> - Scenario 4 (module load/unload): 🔴 **STILL MISSING** — needs the
>   `mgf-hrb-modules-priv` sibling repo (Phase 6 of Part 13);
>   explicitly out of scope this quarter.
> - Scenario 5 (condition-driven callbacks): ✅ — `subscribe` with
>   predicate AST (`418a248`).
> - Scenario 6 (bulk dump + tree walk): ✅ — `inspect_full`
>   (`b0640be`), `dump_chain` (`9c9b347`), `dump_all` (`f0671b9`).
> - Meta requirement (broker self-awareness): ✅ — `audit_tail` +
>   `state_snapshot` shipped together (`d906bce`), gated by the
>   `audit_read` cap right (`c6116e8`).
>
> The verdict table below (§2) is preserved for historical context.
> For the live status of each Part 13 sub-feature with its commit
> SHA, see [`mgf-hrb-self-aware-introspection.md`](mgf-hrb-self-aware-introspection.md) §9.

> **What this WAS.** A code-verified honest assessment of where
> `mgf-hrb` stood on 2026-05-26 against six concrete user-driven
> scenarios for hardware self-awareness (multi-client concurrency,
> driver introspection, module lifecycle, condition callbacks,
> bulk dump) plus the meta requirement that the broker itself be
> "aware of what is going on near real-time."
>
> **Why filed.** The scenarios were filed 2026-05-26 as a "does
> the design natively handle this?" request. Before designing
> extensions ([`mgf-hrb-self-aware-introspection.md`](mgf-hrb-self-aware-introspection.md)),
> the gap inventory had to be honest — false confidence here would
> have designed extensions that closed the wrong gaps.
>
> **Method (preserved for re-runnability of historical commands).**
> Every claim about behaviour at write-time was verified by reading
> the source — file + line citations are embedded so a future
> reader can re-run the check against the historical tree. Note:
> line numbers may have drifted with the implementation work.

---

## §1. The user's prompt, captured

The session that produced this analysis asked the broker to be
checked against the following:

> The system itself `mgf-hrb` is aware of its self. It knows what
> is going on near real-time, it's an aware system that can track
> and follow any actions from start to end, it knows what
> happened, why it happened and how to handle that.
>
> 1. More than one client/user request a hardware resource and
>    they are using it at the same time.
> 2. More than one client request a device write operation or
>    some attribute changes, etc...
> 3. Does the current mgf-hrb able to detect the actual driver
>    own a device and report back the driver details and report
>    back the interface to the device created by the driver?
>    Such as ioctl and sysfs attribute and any other interface
>    available for user mode. This is important; ensure we have
>    a reliable and consistency way to do it.
> 4. Show device driver dependency on other drivers including
>    the bus drivers. Load and unload driver, etc...
> 5. User/client can request and register a callback on some
>    condition, such as: driver loaded, device attached, device
>    initializing, device ready, etc...
> 6. The ability to dump and view all info, data, anything across
>    all the hardware chain. Max data gathering.

The scenarios share a common thread: **the broker must observe
and report; arbitration alone is not enough**. The current
design's framing is "lease arbitrator" (Part 3) plus "typed ops +
raw channel" (Part 2.6) — both correct, both narrower than this
prompt.

This document maps each scenario onto today's surface and names
the gap. The companion design paper
([`mgf-hrb-self-aware-introspection.md`](mgf-hrb-self-aware-introspection.md))
proposes the additive extensions.

---

## §2. Meta — "the system is aware of itself"

### §2.1 What the broker observes today

The broker already publishes structured self-observation at four
levels. None of them was designed under the "self-aware" banner;
they each grew from a different driver:

| Surface | Source | Where it lives |
|---|---|---|
| **Lease ledger** — every active claim across all clients | Part 3 (lease engine) | `list_leases(scope)` wire op; `Client.list_leases()` SDK |
| **Event stream** — `added` / `removed` / `state_changed` / `gap` | Part 2.9 (event model) | `drain_events(scope)` op; `Client.monitor` / `amonitor` / `monitor_fd` |
| **Capabilities introspection** — per-worker per-op support + mode + degradation reason | Part 4 + commit `0efba63` | `capabilities()` wire op |
| **Diagnostics** — 7 structured `DoctorFinding`s (socket / PCI sysfs / USB sysfs / VFIO / IOMMU / udev / backend capabilities) | `src/mgf/hrb/diagnostics.py` | `Client.doctor()` SDK |
| **Per-test telemetry** — JSONL records + per-failure detail bundles | `mgf-test-supervisor` (TS-22..27) | `.gates/last-run/run.jsonl` |

Each of these answers a *different* "what's the broker doing?"
question. They are not assembled into a single self-aware surface.

### §2.2 What it does NOT observe

Three categories of self-awareness are absent:

- **Causal traces.** "Client X's `claim` failed because client Y's
  lease (granted 12 ms earlier in response to wire op Z) holds
  the URN." No correlation IDs, no per-op spans, no caller-effect
  chain in the response.
- **Internal-state inspection.** Active connection count, wire
  buffer fill, per-worker thread state, allocator high-water mark,
  per-client cap-table size — none of these is reachable via the
  wire. A `doctor` check could be added but is not.
- **Operation audit log.** "Every wire op the broker processed in
  the last N seconds, with caller / latency / outcome." Today
  the broker prints sparse stderr messages
  (`server.zig:143` "broker listening on ..." and similar);
  there is no structured per-op log.

These are the three "near real-time, knows why" capabilities the
prompt asks for that the current surface does not deliver.

---

## §3. The scenarios — one-line verdict each

| # | Scenario | Verdict | Verified at |
|---|---|---|---|
| 1 | Multi-client concurrent use of a hardware resource | 🟢 Native (lease layer) | `hrb-code/src/core.zig:compatible()` |
| 2 | Multi-client writes / control-verb changes | 🟡 Partial (lease ops serialised; control ops not) | `domain.py:ControlVerb` + `linux_usb.zig:vControl` |
| 3 | Detect driver owning a device + report interfaces | 🟡 Partial (bind/unbind works; `inspect` omits driver name; ioctl table not enumerable) | `linux_usb.zig:vInspect` + the broader Linux UAPI |
| 4 | Driver dependency graph + module load/unload | 🔴 Missing (interface-level bind/unbind only) | wire-op list in `request.zig` |
| 5 | Callbacks on conditions (driver loaded, device ready) | 🟡 Partial (polling + iter works; no predicate filter; narrow event taxonomy) | `model.zig:Event` + `linux_usb.zig:pumpUevent` |
| 6 | Max data gathering — dump entire hardware chain | 🟡 Partial (per-URN inspect with fixed attr set; no full-tree dump) | `linux_usb.zig:vInspect` |
| Meta | Near-real-time self-awareness | 🟡 Partial (4 observation surfaces; 3 gaps named in [§2.2](#22-what-it-does-not-observe)) | see [§2](#2-meta--the-system-is-aware-of-itself) |

The next sections walk each row with verified detail.

---

## §4. Scenario-by-scenario

### §4.1 Scenario 1 — multi-client concurrent use 🟢

**Native.** The `ClaimMode` lattice (`OBSERVE` / `SHARED_RW` /
`EXCLUSIVE`) is implemented end-to-end with the compatibility
matrix verified in `hrb-code/src/core.zig`:

```zig
// from hrb-code/src/core.zig
// "Shared(rw) is compatible with observe and other shared, not exclusive."
.shared_rw => held == .observe or held == .shared_rw,
// "Exclusive is compatible only with nothing-or-observers."
.exclusive => held == .observe,
```

Translated to operator semantics:

| Existing hold ↓ / New request → | `OBSERVE` | `SHARED_RW` | `EXCLUSIVE` |
|---|---|---|---|
| (nothing) | ✓ | ✓ | ✓ |
| `OBSERVE` (any number) | ✓ | ✓ | ✓ |
| `SHARED_RW` (any number) | ✓ | ✓ | ✗ Conflict |
| `EXCLUSIVE` (must be 1) | ✓ | ✗ Conflict | ✗ Conflict |

Pinned end-to-end by `tests/use_cases/test_vfio_passthrough.py::test_second_agent_claim_during_vfio_hold_fails` + `tests/conformance/test_g3_no_ghost_leases.py`.

**Where the broker stops.** It hands back the raw channel (fd)
for the granted lease and steps out of the data-path. Two
`SHARED_RW` clients writing to the same USB endpoint at the same
time — the kernel + the device firmware decide what happens.
The broker does not multiplex or serialise data writes; that
would put it back on the hot path, which Part 5 rules out.

**Native, with one design honesty:** if a consumer wants
broker-mediated write serialisation, that is a NEW concept (a
write-queue-per-URN) and a NEW design discussion. It is not
covered by the lease lattice. See [Design paper §5.1](mgf-hrb-self-aware-introspection.md#51).

### §4.2 Scenario 2 — multi-client writes / attribute changes 🟡

**Two paths, only one serialised:**

- **Lease-gated raw I/O** (`SHARED_RW` / `EXCLUSIVE`): see [§4.1](#41-scenario-1--multi-client-concurrent-use-).
- **Cap-gated control verbs** (`set_authorized` / `set_autosuspend` /
  `set_port_power` / `bind_driver` / `unbind_driver` — verified
  in `src/mgf/hrb/domain.py:ControlVerb`): currently **NOT
  serialised across clients**. Two clients calling
  `set_authorized(true)` and `set_authorized(false)` simultaneously
  race at the sysfs layer; last-writer-wins; neither client knows.

The design comment in `domain.py:63-68` says these verbs "mutate
hardware state without acquiring it, so they are cap-gated
rather than lease-gated." That's the intentional design — control
verbs are operator actions, not exclusive resource holds. But
"cap-gated without ordering" is silent on the multi-client case.

**Manifest gap.** A consumer reading the wire today sees:

```python
client.control("hw:/usb/.../if0", ControlVerb.SET_AUTHORIZED, True)
# returns {"ok": True}  ← but is the device actually authorized now?
# Another client may have written `False` 5ms after this write.
```

There is no `applied_value`, no `superseded_by`, no
`generation_counter` in the response. Today this is masked because
in practice operator UIs assume a single operator; in a
multi-agent federation (vm-vmanager-app + a CLI + a script) the race is
real.

See [Design paper §5.1](mgf-hrb-self-aware-introspection.md#51)
for proposed shapes.

### §4.3 Scenario 3 — driver-owning detection + interfaces 🟡

**Three sub-asks at different difficulty:**

#### §4.3.1 Report the driver currently bound to the device — easy

Verified absent. `linux_usb.zig:vInspect` reads
`bInterfaceClass`, `bInterfaceSubClass`, `bInterfaceProtocol`,
`vid`, `pid`, `serial`, `busnum`, `devnum` from sysfs. It does
NOT read the `driver` symlink (`/sys/bus/usb/devices/<iface>/driver`
→ `/sys/bus/usb/drivers/<name>`) or the `driver/module` symlink.

```bash
# what's in sysfs but NOT in inspect()'s response:
ls -l /sys/bus/usb/devices/1-1.4:1.0/driver
# lrwxrwxrwx ... driver -> ../../../../../bus/usb/drivers/usbhid
```

Adding `driver_name` and `driver_module` to the attrs is ~20
lines of Zig (read the symlink, basename it). [Design paper §5.2](mgf-hrb-self-aware-introspection.md#52)
makes it part of the next-up batch.

#### §4.3.2 Enumerate sysfs attributes — easy

The directory walk that gets you "every attribute file under this
device's sysfs node" is straightforward (the kernel exposes them
as ordinary files; `readdir` + `read`). Today's `vInspect` reads
a fixed list. A `--full` variant (or a `inspect_full(urn)` verb)
would walk the dir and emit every attr. [Design paper §5.5](mgf-hrb-self-aware-introspection.md#55).

#### §4.3.3 Enumerate ioctls a driver exposes — structurally hard

**Linux has no discoverable ioctl table per driver.** ioctls are
`#define`d in C headers (uapi); the driver registers handlers via
its `file_operations.unlocked_ioctl`; the kernel does not retain
the encoding metadata. There is no `/sys/.../ioctls/` directory.
There is no "ask the driver what numbers it answers to" syscall.

The only **reliable, consistent way** to know what ioctls a
driver exposes is a hand-maintained registry that maps driver
names to their known ioctl numbers + arg-shapes:

```yaml
# example shape — proposed for src/mgf/hrb/data/driver_interfaces/usbdevfs.yml
driver: usbdevfs
header: <linux/usbdevice_fs.h>
ioctls:
  - name: USBDEVFS_CONTROL
    number: 0xc0185500
    arg: usbdevfs_ctrltransfer
  - name: USBDEVFS_BULK
    number: 0xc0185502
    arg: usbdevfs_bulktransfer
  # ... ~25 USBDEVFS ioctls total
```

This is the unavoidable cost of the "report driver interfaces"
ask. No clever code makes it go away. [Design paper §5.2.3](mgf-hrb-self-aware-introspection.md#523)
discusses the curation policy.

### §4.4 Scenario 4 — driver dependency + load/unload 🔴

**Three sub-asks; today supports only one:**

| Sub-ask | Today | Cost |
|---|---|---|
| Bind/unbind a *loaded* driver to a device | ✅ `control` verbs `bind_driver` / `unbind_driver` (USB only) | shipped |
| Load/unload a kernel module (`modprobe` / `rmmod`) | ❌ not exposed | new privileged-helper sibling |
| Driver dependency graph (which modules depend on which) | ❌ not exposed | parser + new wire op |

**Why module load/unload is hard.** Loading a kernel module
needs `CAP_SYS_MODULE`. The mgf-hrb broker runs unprivileged
(deliberately — Part 12 §2). To support module ops the federation
would need:

- A new privileged helper subprocess (sibling to the
  USB-worker's planned `mgf-hrb-usb-priv` per §U3b), running with
  `CAP_SYS_MODULE`, accepting only typed kabi verbs over a typed
  socketpair.
- A new wire-op family (`module_load`, `module_unload`,
  `module_info`, `module_deps`) routed through the existing
  capability check.
- A udev policy decision: who is allowed to call these? A
  developer on their own machine? A vm-vmanager-app admin? A locked-down
  appliance with a fixed module set?

**This is a substantial new scope** — comparable in size to the
USB worker itself. Not in any current phase. [Design paper §5.3](mgf-hrb-self-aware-introspection.md#53)
sketches the helper.

**Dependency graph is the easier half.** `/lib/modules/$(uname
-r)/modules.dep` is a plain-text file mapping each module to its
transitive dependencies. Parsing it into a structured response is
a new wire op (`module_deps(name)`) that needs no privilege.
That can ship independently of the load/unload work.

### §4.5 Scenario 5 — condition-filtered callbacks 🟡

**Polling and iter work; predicate filtering does not.**

#### §4.5.1 What works

```python
async for event in client.amonitor("hw:/usb/**"):
    if event["kind"] == "added" and event["urn"].endswith(":1.0"):
        handle(event)
```

`Client.monitor(scope)` (sync iter), `Client.amonitor(scope)`
(async), `Client.monitor_fd(scope)` (pipe fd) all ship.
Scope filter is a URN glob (`"hw:/usb/**"`, `"hw:/test/dev0"`).
Verified in `src/mgf/hrb/_client.py:356-810`.

#### §4.5.2 What doesn't

**Event taxonomy is narrow.** `model.zig:Event` defines four
kinds:

```zig
pub const Event = union(enum) {
    added: struct { urn: Urn },
    removed: struct { urn: Urn },
    state_changed: struct { urn: Urn, old: State, new: State, previous_urn: ?Urn },
    gap: struct { source: GapSource, since_ms: u64 },
};
```

`state_changed`'s State enum is broker-internal (claimable /
claimed / quarantined / degraded — not "driver bound" / "device
ready" / "initializing"). The kernel uevent stream actually
contains driver-bind events (`ACTION=bind`, `SUBSYSTEM=drivers`);
the USB worker filters them out today (`linux_usb.zig:pumpUevent`
only routes device-level add/remove).

**No server-side predicate.** The "register a callback that fires
when driver `usbhid` binds to ANY device with `vid=0x1234`" shape
requires either:

- Client-side filtering of a coarse stream (works; wasteful).
- A server-side filter language attached to the subscription.

[Design paper §5.4](mgf-hrb-self-aware-introspection.md#54)
proposes both halves.

#### §4.5.3 "Device ready" — not a first-class kernel concept

The kernel emits `ACTION=add` when the device node appears and
`ACTION=bind` when a driver binds. After bind, "ready" is
driver-specific:

- **USB Mass Storage**: ready when `/dev/sdX` appears (block subsystem `ACTION=add`).
- **USB HID**: ready when `/dev/input/event*` appears.
- **DRM**: ready when `/dev/dri/card*` + `/dev/dri/renderD*` appear.
- **NVMe**: ready when `/dev/nvme*n*` appears AND the namespace controller is responsive.

Each driver class needs an adapter that maps "the driver is now
useful" to one of the above. No federation-wide kernel signal
exists. [Design paper §5.4.3](mgf-hrb-self-aware-introspection.md#543).

### §4.6 Scenario 6 — max data gathering 🟡

**Per-URN inspect works; full-tree dump does not.**

`Client.inspect("hw:/usb/.../if0")` returns a `ResourceDetail`
with a *fixed attr set* (see [§4.3.2](#432-enumerate-sysfs-attributes--easy)).
It does NOT walk the sysfs subtree, does NOT walk the parent chain
(interface → device → hub → root-hub → bus), and does NOT assemble
a topology view.

A consumer wanting "everything sysfs knows about this device" must
read sysfs themselves — which defeats the design's "broker is the
only one with hardware authority" stance (Part 12 §2).

Three additive ops close this:

- **`inspect_full(urn)`** — walks the device's sysfs dir + all
  subdirs; returns the full attr tree.
- **`dump_chain(urn)`** — walks parent links upward + returns
  the assembled chain (interface → device → hub → root-hub).
- **`dump_all(scope)`** — walks every URN matching scope, returns
  `{urn: ResourceDetail}` assembled.

All three are read-only, additive, fit the existing typed-projection
pattern (SDK-02). [Design paper §5.5](mgf-hrb-self-aware-introspection.md#55).

---

## §5. The wire-op inventory (today)

Verified in `hrb-code/src/request.zig`:

```zig
if (std.mem.eql(u8, s, "list"))          return .list;
if (std.mem.eql(u8, s, "claim"))         return .claim;
if (std.mem.eql(u8, s, "release"))       return .release;
if (std.mem.eql(u8, s, "raw"))           return .raw;
if (std.mem.eql(u8, s, "inspect"))       return .inspect;
if (std.mem.eql(u8, s, "capabilities")) return .capabilities;
if (std.mem.eql(u8, s, "drain_events"))  return .drain_events;
if (std.mem.eql(u8, s, "control"))       return .control;
if (std.mem.eql(u8, s, "list_leases"))   return .list_leases;
```

**Nine ops total.** The design paper proposes adding:

| New op | Purpose | Scenario | Privileged? |
|---|---|---|---|
| `inspect_full` | Walk full sysfs subtree | 6 | No |
| `dump_chain` | Walk parent chain (interface → bus) | 6 | No |
| `dump_all` | Walk every URN matching scope | 6 | No |
| `driver_info` | Driver name + module + loaded ioctls (from registry) | 3 | No |
| `module_deps` | Parse `modules.dep`; return dep graph for module | 4 | No |
| `module_load` | `modprobe <name>` | 4 | YES (via helper) |
| `module_unload` | `rmmod <name>` | 4 | YES (via helper) |
| `subscribe` | Subscribe with predicate; server-side filter | 5 | No |
| `unsubscribe` | Drop a subscription | 5 | No |
| `audit_tail` | Last N broker ops; ring buffer | meta | No |
| `state_snapshot` | Internal-state self-report | meta | No |

That takes the broker from 9 to 20 wire ops. The design paper
discusses which subset is single-PR-shipping vs phased.

---

## §6. Synthesis

**The design is *consistent* with all six scenarios but does not
*natively* deliver 2 / 3 / 4 / 5 / 6 at their full scope today.**

The gaps fall into four categories, ranked by additive cost:

| Category | Cost | Scenarios | Architecturally new? |
|---|---|---|---|
| Cheap additive ops on existing surface | 1 PR each | 3a (driver name) · 6 (dump variants) · 5a (driver-bound event kind) | No |
| Medium additive ops requiring small wire extensions | 1-2 PRs each | 2 (control-verb serialisation) · 5b (server-side predicate) · meta (audit log) | No |
| New scope — new helper sibling | A new design paper + a new sibling repo | 4 (module load/unload) | Yes |
| Structurally hard, requires hand-curated registry | Ongoing data-curation effort | 3b (ioctl table) | No, but no clever code makes it go away |

The cheap + medium categories are the next phase's work; the
new-helper category is a separate design conversation; the
registry-curation work is a federation maintenance task.

**Honest reading.** None of the gaps invalidates the existing
design. They are all extensions of the typed-op / event /
capabilities model already in place. The risk is *underestimating*
the new-helper category — module ops with `CAP_SYS_MODULE` are
in a different security tier than the unprivileged broker, and
the threat model deserves its own design.

---

## §7. Verification commands

Re-run the analysis end-to-end from the repo root:

```bash
# §2.1 — five observation surfaces (greps that find each)
grep -n "list_leases\|drain_events\|capabilities\|doctor" src/mgf/hrb/_client.py
grep -n "DoctorFinding" src/mgf/hrb/diagnostics.py
grep -n "TestRecord" /home/bassam/PycharmProjects/mgf-test-supervisor/src/mgf/test_supervisor/records.py

# §4.1 — claim-mode compatibility matrix
grep -B1 -A6 "shared_rw \|exclusive =>" hrb-code/src/core.zig

# §4.2 — control-verb list (no serialisation check)
grep -A12 "class ControlVerb" src/mgf/hrb/domain.py
grep -n "vControl" hrb-code/src/linux_usb.zig

# §4.3.1 — vInspect attr set (no driver_name)
grep -A 80 "fn vInspect" hrb-code/src/linux_usb.zig | head -80
grep -nE "\"driver\"|\"module\"" hrb-code/src/linux_usb.zig  # → no matches

# §4.4 — module-op absence in wire vocabulary
grep -nE "modprobe|rmmod|init_module|modules\\.dep" hrb-code/src/
grep -E '"(list|claim|release|raw|inspect|capabilities|drain_events|control|list_leases)"' hrb-code/src/request.zig

# §4.5.2 — event-kind taxonomy
grep -B1 -A10 "pub const Event = union" hrb-code/src/model.zig
grep -n "ACTION=bind\|ACTION=unbind\|SUBSYSTEM=drivers" hrb-code/src/linux_usb.zig  # → no matches today

# §4.6 — vInspect omits sysfs subtree walk
grep -n "rglob\|walkSubtree\|readdir" hrb-code/src/linux_usb.zig | head
```

Each command's output is consistent with the verdicts in [§3](#3-the-scenarios--one-line-verdict-each).
If any output diverges from this document's claims, this document
is stale and needs updating against the new code state.

---

## §8. Cross-references

- [`mgf-hrb-complete-design.md`](mgf-hrb-complete-design.md) —
  master design. Part 2 (model), Part 3 (lease engine), Part 4
  (capabilities), Part 7 (workers), Part 12 (survivability).
- [`mgf-hrb-self-aware-introspection.md`](mgf-hrb-self-aware-introspection.md)
  — companion forward-design paper that proposes the extensions
  named here.
- [`mgf-hrb-usb-worker.md`](mgf-hrb-usb-worker.md) — the USB
  worker design this analysis tested against (especially the
  privileged-helper section §U3b that the module-ops sibling
  would mirror).
- [`mgf-hrb-survivability-review.md`](mgf-hrb-survivability-review.md)
  — Tier-0 hardening; the audit log + state-snapshot ops touch
  the same surfaces.
