# Performance & Capacity

> **Status:** v0.1 — first pass, finalized 2026-05-15
> (review-pass verified all file:line citations against the
> live source). Last updated 2026-05-15.
> A federation-sibling-scoped performance audit. The largest
> sibling by surface (~5000 LOC); PC concerns split into
> **suite-execution throughput**, **per-probe latency
> budget**, **check-chain iteration cost**, **cold-import
> time**, and **memory baseline** for the harness runtime.
>
> **Scope:** mgf-apiprobe's OWN performance footprint as
> a Python library + as a verification harness. NOT a re-
> audit of mgf-common / httpx / pydantic.
>
> **Audience:** the maintainer + future AI agents.
> **Cross-references:**
> [`RELIABILITY_AND_ACCURACY.md`](RELIABILITY_AND_ACCURACY.md),
> [`SECURITY_HARDENING.md`](SECURITY_HARDENING.md),
> [`mgf-common/docs/inprogress/PERFORMANCE_AND_CAPACITY.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/inprogress/PERFORMANCE_AND_CAPACITY.md)
> (cornerstone),
> [`mgf-http/docs/inprogress/PERFORMANCE_AND_CAPACITY.md`](https://codeberg.org/magogi-admin/mgf_http/src/branch/main/docs/inprogress/PERFORMANCE_AND_CAPACITY.md)
> (sister-sibling — the transport-layer concerns).

mgf-apiprobe is the verification harness; consumers run
suites with hundreds of probes against production APIs. PC
concerns: **suite throughput** (how many probes/sec the
runner sustains), **per-probe latency** (auth + request +
check chain), **memory growth** under long suite runs, and
**cold-import** (the runner is invoked from CLI; startup
counts).

---

## §1 You are here

mgf-apiprobe ships a large surface; per-probe execution
flow is: auth.prepare() → transport.send() (httpx wrapper) →
N × check.run(). For a 100-probe suite with 5 checks each,
the runner executes 100 sends + 500 check.runs. Concurrency-
capped via `asyncio.Semaphore(max_concurrency=4)` (default).

What this tracker exposes is the **harness-side
performance shape** — the wrapper tax over httpx + the
check-chain iteration + memory growth under long runs.

**Counts (2026-05-17):**

| Priority | Open | Closed | Total |
|---|---|---|---|
| P0 — pre-v1.0 blockers | 0 | 0 | 0 |
| P1 — should land before/with v1.0 | 0 | 4 | 4 |
| P2 — post-v1.0 hardening | 2 | 2 | 4 |
| **Total** | **2** | **6** | **8** |

**Do first:** PC-01..04 closed (P1). PC-06 (matcher DSL pin,
11 matchers x 10k evals) + PC-08 (report serialisation pin,
5000-finding worst-case) closed 2026-05-17. PC-05 (memory
baseline; M-effort scaffolding) + PC-07 (Finding evidence
profile; no clear gate) deferred to v1.1+.

---

## §2 Priority table

| ID | Theme | One-liner | Effort | Status |
|---|---|---|---|---|
| [**PC-01**](#pc-01-suite-throughput-at-max-concurrency) | Capacity | `Suite.run` with `max_concurrency=4` against a localhost mock should sustain ~50-100 probes/sec. No measurement; no budget. | M | ✅ closed |
| [**PC-02**](#pc-02-per-probe-wrapper-tax-over-httpx) | Latency | `_run_one_probe` adds work over httpx's send: AuthFlow.prepare() (env-var resolution, header build), URL construction, check-chain iteration, severity-override apply. Quantify the per-probe wrapper time. | S | ✅ closed |
| [**PC-03**](#pc-03-check-chain-iteration-cost) | Latency | Each probe runs N checks (default ~5). Each check.run is sub-µs to µs for simple checks; a regex-based schema check could be ms. Pin per-check budget + total chain budget. | S | ✅ closed |
| [**PC-04**](#pc-04-cold-import-time) | Startup | `import mgf.apiprobe` transitively loads httpx + pydantic + the full submodule tree (~45 modules). On CLI invocation, startup dominates short suite runs. Measure + budget. | S | ✅ closed |
| [**PC-05**](#pc-05-memory-baseline-after-suite-run) | Memory | A long suite run (1000 probes) builds 1000 Response objects + 1000s of Finding objects + the httpx connection pool. Memory growth profile is unmeasured. | M | ⏳ v1.1+ — subprocess RSS scaffolding is M-effort; not a v1.0 blocker |
| [**PC-06**](#pc-06-matcher-dsl-evaluation-cost) | Latency | The matcher DSL (`core/matchers/*.py` — primitives, format, semantic) is the per-finding hot path. A schema check evaluates many matchers per response. Sub-µs per matcher; pin to catch regressions. | S | ✅ closed |
| [**PC-07**](#pc-07-finding-evidence-allocation-profile) | Memory | Each `Finding` allocates: a code str, severity enum, root_cause enum, message str, evidence dict, threat_refs tuple, tags tuple. For a 1000-probe suite with 5 findings per probe, 5000 Finding allocations. Profile + budget. | S | ⏳ v1.1+ — allocation profile has no clear regression-trigger gate |
| [**PC-08**](#pc-08-report-serialisation-throughput) | Capacity | `reporting/json_writer.py` serialises the Report to JSON. For a 5000-finding report, the serialisation cost adds up. Pin. | XS | ✅ closed |

Effort scale: XS (≤ 15 min); S (≤ 1 h); M (≤ 1 session); L
(multiple sessions).

---

## §3 Invariants we hold

The list of "closed elsewhere; do not re-open":

- **PC-INV-1** — Probe execution is **concurrency-capped**
  via `asyncio.Semaphore(self.settings.max_concurrency)`
  ([`suite.py:109`](../../src/mgf/apiprobe/core/suite.py#L109)).
  Default 4; configurable per-suite.
- **PC-INV-2** — Suite-level fields are **stored as tuples**
  (`self.probes`, `self.checks`)
  ([`suite.py:78-79`](../../src/mgf/apiprobe/core/suite.py#L78)).
  Immutable; no per-call reallocation.
- **PC-INV-3** — `Finding` is a **frozen dataclass** —
  `dataclasses.replace` for mutation; immutability prevents
  accidental shared-state regressions.
- **PC-INV-4** — `HttpxTransport` holds **one httpx client**
  per transport; connection pool reused across probes.
- **PC-INV-5** — Transport errors **yield synthetic
  Findings** (don't propagate); the check chain skips for
  the failed probe. No wasted check.run calls on a probe
  with no response.
- **PC-INV-6** — Severity-overrides use
  `dataclasses.replace`, not in-place mutation — one extra
  allocation per overridden finding, but the immutability
  invariant holds.
- **PC-INV-7** — `_CIDR_CACHE` (LRU) is shared across
  IpAllowlist instances. Repeated suite runs reuse parsed
  CIDRs.

---

## §4 Work units

### PC-01 Suite throughput at max_concurrency

**Why:** `Suite.run` with default
`max_concurrency=4` is the baseline shape. Throughput
depends on:

- Network latency (provider-side; uncontrolled).
- Per-probe wrapper time (PC-02).
- Check-chain time per probe (PC-03).

For a localhost-mock test: zero network latency → wrapper
tax + check time dominate. Pin a baseline budget.

For real-provider testing: ~50-100 probes/sec at
concurrency=4 against a sub-100 ms-latency API.

**Concrete output:**

- Add `tests/perf/test_suite_throughput.py` (under
  `pytest.mark.slow`):
  - Build a 100-probe suite + a localhost mock transport.
  - Run with max_concurrency=4.
  - Measure total wall time.
  - Assert ≥50 probes/sec sustained.
- Document baseline.

**Locked in by:** the slow test.

**Cross-references:** mgf-http PC-06 (sustained-throughput
twin at a different layer).

**Status:** ✅ closed (2026-05-16). See §5.

---

### PC-02 Per-probe wrapper tax over httpx

**Why:** `_run_one_probe`
([`suite.py:133-171`](../../src/mgf/apiprobe/core/suite.py#L133))
+ `HttpxTransport.send`
([`httpx_transport.py:80-133`](../../src/mgf/apiprobe/transport/httpx_transport.py#L80))
add work over `httpx.AsyncClient.request`:

1. Probe URL resolution.
2. `probe.auth.prepare()` — env var resolution + creds
   build.
3. Header / cookie / param assembly.
4. Timeout select.
5. The httpx send (THE COST — uncontrolled).
6. `_to_apiprobe_response()` — Response dataclass alloc.
7. Check chain iteration.

Estimated wrapper tax: ~30-100 µs per probe.

**Concrete output:**

- Add `tests/perf/test_probe_wrapper_overhead.py`:
  - Mock httpx via respx; return zero-cost responses.
  - Drive 10 000 `_run_one_probe` calls.
  - Measure per-call Python time excluding httpx.
  - Assert ≤ 100 µs per probe.

**Locked in by:** the perf test.

**Status:** ✅ closed (2026-05-16). See §5.

---

### PC-03 Check chain iteration cost

**Why:** `_run_one_probe`'s check loop
([`suite.py:167-171`](../../src/mgf/apiprobe/core/suite.py#L167)):

```python
for check in self.checks:
    for finding in check.run(probe, response, ctx):
        findings.append(self._apply_severity_override(finding))
```

Per probe: N check.runs (typically 5). Per check.run: depends
on the check.

- Simple checks (header presence): ~1-5 µs.
- Regex-based checks (schema validation): ~10-100 µs.
- BodyEchoesSecret with substring search: O(body_size).

For a 100-probe suite × 5 checks: 500 check.runs. At
~10 µs average: 5 ms total check chain work. At ~100 µs
average: 50 ms total.

**Concrete output:**

- Add `tests/perf/test_check_chain_cost.py`:
  - Build a Response with a 4 KB body.
  - Run the default check set against it.
  - Measure per-check time + total chain time.
  - Assert per-check ≤ 50 µs; total ≤ 250 µs.
- Document per-check baseline in benchmarks.md.

**Locked in by:** the perf test.

**Status:** ✅ closed (2026-05-16). See §5.

---

### PC-04 Cold-import time

**Why:** `import mgf.apiprobe` transitively loads:

- `mgf.common` (`~100 ms` per mgf-common PC-01 closed).
- `httpx` + h11 + h2 (~50 ms).
- `pydantic` (cached if already loaded by anything else).
- The full `mgf.apiprobe` submodule tree (~45 modules,
  ~50 ms).

Estimated cold-import: ~200-300 ms.

For CLI invocations (`mgf-apiprobe run suite.yml`), startup
time matters. A 200 ms startup on a 30 s suite is 0.7%;
on a 1 s smoke run, it's 20%.

**Concrete output:**

- Add `tests/perf/test_cold_import.py`:
  - Fork a fresh Python interpreter.
  - `python -c 'import mgf.apiprobe'`.
  - Measure wall time.
  - Assert ≤ 500 ms.
- Document baseline.

**Locked in by:** the perf test.

**Cross-references:** mgf-common PC-01 (closed; the
cornerstone baseline).

**Status:** ✅ closed (2026-05-16). See §5.

---

### PC-05 Memory baseline after suite run

**Why:** A 1000-probe suite allocates:

- 1000 Response objects (URL + status + headers dict +
  body bytes + elapsed).
- 1000+ Finding objects (each: code str + enums + message
  + evidence dict).
- 1000 sets of request_headers + auth creds + cookies.
- The httpx connection pool (one persistent connection per
  host).

If the suite probes 50 distinct hosts, the pool holds
~50 keepalive connections.

After the suite finishes, are these GC'd cleanly? Or does
something hold a reference?

**Concrete output:**

- Add `tests/perf/test_memory_baseline.py` under
  `pytest.mark.slow`:
  - Measure RSS before suite.
  - Run a 1000-probe suite against localhost mocks.
  - Force `gc.collect()`.
  - Measure RSS after.
  - Assert delta ≤ 20 MB.

**Locked in by:** the slow test catches memory leaks.

---

### PC-06 Matcher DSL evaluation cost

**Why:** The matcher DSL
(`core/matchers/{primitives,format,semantic}.py`) is the
per-finding hot path. A schema check might compose:

```python
expects_json_shape({
    "id": is_uuid,
    "email": is_email,
    "created_at": is_iso_datetime,
    "items": [is_dict_with({...})],
})
```

Each value gets a matcher; each matcher is a callable.
For a moderate-shape response: 20-50 matcher evaluations
per check.

Per-matcher cost: ~0.5-5 µs depending on the matcher
(equality is sub-µs; regex-based is µs).

**Concrete output:**

- Add `tests/perf/test_matcher_evaluation.py`:
  - Construct each matcher type from the public DSL.
  - Evaluate × 10 000.
  - Assert per-evaluation ≤ 5 µs (generous; equality should
    be sub-µs).

**Locked in by:** the perf test.

---

### PC-07 Finding evidence allocation profile

**Why:** Each `Finding` allocates:

- A `code` string (mostly literal; interned).
- `Severity` + `RootCause` enum members (cached).
- `message` string (formatted per finding).
- `evidence` dict (varies per check).
- `threat_refs` tuple.
- `tags` tuple.

For a 1000-probe suite × 5 findings per probe = 5000
Finding allocations. Each ~500 bytes-ish. Total ~2.5 MB.

For a long-running operator session: many suite runs
accumulate.

**Concrete output:**

- Add `tests/perf/test_finding_allocation.py`:
  - Build 1000 Finding objects.
  - Measure total allocation + per-instance avg.
  - Assert per-finding ≤ 1 KB.

**Locked in by:** the perf test.

---

### PC-08 Report serialisation throughput

**Why:** `reporting/json_writer.py` serialises the Report
to JSON. For a 5000-finding report, the JSON file may be
several MB.

Serialisation cost: ~10-50 ms for a 5 MB JSON file.

For consumers piping the report to a CI artifact, this is
once per suite — amortised.

**Concrete output:**

- Add `tests/perf/test_report_serialisation.py`:
  - Build a Report with 5000 findings.
  - Serialise to JSON × 10.
  - Assert ≤ 200 ms per serialisation.

**Locked in by:** the perf test.

---

## §5 Closed items

- **PC-01** — Suite throughput floor pinned at ≥ 50 probes/sec
  for a 100-probe DEFAULT_CHECKS suite at default `max_concurrency=4`
  (measured ~94k probes/sec locally with a zero-latency stub
  transport; floor is ~1880x cushion for regression-detection,
  not creep-detection). Added `tests/perf/test_suite_throughput.py`
  with a `slow`-marker test that exercises the full `Suite.run`
  path (asyncio.gather + Semaphore + per-probe orchestration +
  full DEFAULT_CHECKS chain). Catches deep harness regressions:
  accidentally-sequential execution despite the semaphore,
  per-probe sleep, synchronous I/O in the hot path. (2026-05-16,
  commit pending.)
- **PC-02** — Per-probe wrapper tax (Suite._run_one_probe +
  HttpxTransport.send) pinned at ≤ 100 µs/probe (measured
  ~8-15 µs locally; ~10x cushion). Added
  `tests/perf/test_probe_wrapper_overhead.py` with two perf-marker
  tests: no-auth GET baseline + realistic shape (headers + query +
  json body POST). httpx is stubbed at the `AsyncClient.request`
  boundary via `AsyncMock` so the measurement excludes httpx's
  own protocol code — what's measured is the cost OWNED by
  apiprobe (URL resolution, header / query assembly, auth-fragment
  merge, timeout select, Response alloc). Tracker originally
  suggested respx, but respx intercepts at the underlying
  transport so httpx's protocol code still runs — that would
  measure "wrapper + httpx" not "wrapper over httpx"; the choice
  is documented in the test docstring. (2026-05-16, commit pending.)
- **PC-03** — `DEFAULT_CHECKS` per-check + total-chain iteration
  cost pinned at ≤ 50 µs per `check.run` and ≤ 250 µs for the full
  18-check chain (against a 4 KB JSON body with standard contract +
  observability headers). Added `tests/perf/test_check_chain_cost.py`
  with two perf-marker tests: one walks every check in
  `DEFAULT_CHECKS` and asserts each stays under the per-call budget
  (error message names the offending class for fast triage); the
  other measures the cumulative chain time per probe. Catches
  silent regressions where a single check grows expensive AND
  death-by-thousand-paper-cuts where the whole chain creeps up.
  (2026-05-16, commit pending.)
- **PC-04** — `import mgf.apiprobe` cold-import wall-clock pinned
  at ≤ 500 ms (measured ~136 ms median locally over 5 fresh-
  subprocess runs; ~3.5x cushion for CI noise + slower hardware).
  Added `tests/perf/test_cold_import.py` which spawns 5 fresh
  Python subprocesses, imports `mgf.apiprobe`, and asserts the
  median wall-clock under budget. New `perf` + `slow` markers
  registered in `pyproject.toml`; default suite deselects them
  via `-m "not perf and not slow and not live"`. Run via
  `pytest -m perf` for the nightly sweep. (2026-05-16, commit
  pending.)
- **PC-06** — Matcher DSL evaluation pin. New
  `tests/perf/test_matcher_evaluation.py` runs each public matcher
  (`equals`, `gt`, `in_`, `matches`, `is_uuid`, `is_iso8601_z`,
  `is_url`, `is_email`, `is_iban`, `is_imei`, `is_e164`) 10 000
  times and asserts the per-call median stays under 5 µs.
  Parametrised so a regression in any single matcher fails with
  that matcher's name in the assertion message. Local: every
  matcher comes in sub-µs to ~1 µs. (2026-05-17, P2 sweep.)
- **PC-08** — Report serialisation throughput. New
  `tests/perf/test_report_serialization.py` builds a synthetic
  5000-finding Report (covering every severity, every root cause,
  with realistic evidence dicts + remediation strings) and
  asserts `JsonReportWriter.write()` completes under 100 ms.
  Local: ~3 ms — 30x headroom catches the regression class where
  the per-finding render path grows expensive without showing up
  in a 5-finding test. (2026-05-17, P2 sweep.)

---

## §6 Notes on methodology

This first pass sampled the load-bearing files (~1500 LOC
of ~5000 total) against:

- mgf-common's PC tracker methodology.
- Each subsystem's perceived performance profile (auth
  flows = sub-ms; transport = network-dominated; checks =
  µs; matchers = sub-µs; reporting = serialisation-cost).

The audit method:

1. Trace per-probe execution end-to-end identifying each
   step's cost contribution.
2. Pin separate budgets for each layer.
3. Add sustained-throughput + memory-baseline tests for
   patterns single-call budgets can't catch.

Eight items + seven invariants. The harness has the
largest PC surface among siblings. Future passes should
look for:

- **Profile-guided optimisation.** Once budgets are pinned,
  run cProfile on a representative suite to find the
  actual hot paths. Optimise where the budgets are tightest.
- **Memory pooling.** Finding objects are frozen dataclasses;
  if allocation cost dominates, consider an object-pool
  pattern. Defer until measurement shows it matters.
- **Parallel suite execution.** Currently suites run with
  asyncio.Semaphore-capped concurrency. For very large
  suites, multi-process (forking) may scale further.
- **Per-check profiling helpers.** Add tooling for
  consumers to profile their own custom checks.

When an item closes, move it to §5 with the closing-commit
SHA + measured baseline.
