# mgf-apiprobe — Design Document

> **Status:** v0.4 — pre-implementation; sibling-package decision locked. Last updated 2026-04-29.
> **Authors:** Bassam Alsanie, Claude.
> **Stage:** design + alignment. Code starts when `HANDOFF.md` Step 0 is acknowledged.
> **Audience:** Bassam, future contributors, AI coding agents.

> **⚠ READ THIS BEFORE THE REST OF THE DOCUMENT — v0.4 ADDENDUM (2026-04-29):**
>
> This project's packaging shape and naming have changed twice since
> v0.3. The body of this document still references the **older** plans
> in many places. The current state is:
>
> 1. **Renamed: `mgf_apiverify` → `mgf-apiprobe`** (PyPI distribution
>    name; import path `mgf.apiprobe`). Wherever the body of this
>    document says `mgf_apiverify` / `apiverify`, mentally substitute
>    `mgf_apiprobe` / `apiprobe`. (The body is intentionally NOT
>    sweep-renamed — the rename happens at integration time, file by
>    file, so the v0.3 spec stays auditable as it was authored.)
>
> 2. **Sibling, NOT subpackage of `mgf-common`.** The 2026-04-28 plan
>    to merge into `mgf-common` as `mgf.common.apiverify` was reviewed
>    and reversed the same day. The library lives as a sibling at
>    `/home/bassam/PycharmProjects/mgf-apiprobe`, distributed as
>    `mgf-apiprobe` on PyPI, depending on `mgf-common` at runtime.
>    Reasons: surface explosion, closed-box leak, versioning collision,
>    federation precedent. Full rationale in `HANDOFF.md` §2.
>
> 3. **Runtime dependency stance INVERTED from §15.** §15 says "does
>    NOT depend on `mgf_common` at runtime." That is now WRONG.
>    apiprobe DEPENDS on `mgf-common` and uses `BaseAppSettings`,
>    `AppError`, `mgf.common.observability` (logging + OTel + redaction
>    + crash reporter), and `mgf.common.vault` directly. See
>    `HANDOFF.md` §3.2 for the per-surface inversion table.
>
> 4. **Scope reduced for first publishable release.** §16's full
>    Phases 1 → 4.5 to v1.0 is a multi-quarter project. v0.1 ships a
>    tighter cut (core primitives + Pydantic schema + ~18 checks +
>    httpbin self-test + 3 CLI subcommands). Everything past that is
>    tracked in `DEFERRED.md`. See `HANDOFF.md` §6 for the v0.1 cut
>    and `DEFERRED.md` for the full deferral list with ship-when
>    markers.
>
> 5. **`HANDOFF_OLD.md`** captures the 2026-04-28 (a.m.) merge-into-
>    mgf-common plan that was reversed in 2026-04-28 (p.m.). It is
>    kept as historical record. **Do not act on it.**
>
> When this document and `HANDOFF.md` / `DEFERRED.md` disagree,
> **`HANDOFF.md` and `DEFERRED.md` win**. This addendum is the only
> change to the v0.3 body of this document; the rest is preserved
> as-authored to keep the spec auditable.

This document is the **single source of truth for WHAT we are building** —
the primitives, the check catalogue, the replay subsystem, the drift /
mutation / plugin / observability designs. `HANDOFF.md` covers HOW it
integrates as a sibling of `mgf-common`. `DEFERRED.md` covers WHICH
parts of WHAT actually ship in the first release vs. later.

Update this file when the design changes; do not let the code drift from
it silently.

**Version history:**

| Version | Date | Summary |
|---|---|---|
| v0.1 | 2026-04-28 | Initial draft — Probe / Check / Suite / Replay / Pydantic schema. |
| v0.2 | 2026-04-28 | Scenarios + Context + variables; AuthFlow as primitive; Differential testing & drift detection (5 shapes); Mutation testing (16 mutations); Plugin architecture (8 extension points); Profiles & environments; Observability of the verifier itself; expanded check catalogue. |
| v0.3 | 2026-04-28 | This round. Multipart / streaming bodies + downloads; long-running-operation polling pattern; richer response matchers DSL; failure attribution on findings; factory functions + Faker integration in Context; suite composition / templates; CI integration patterns (PR comments / Slack / PagerDuty); industry-standard conformance checks (RFC 7807/9457, JSON:API, semantic field validators); rate-limit verification + concurrency-safety probes; snapshot testing; cassette signing; semantic body validation library; determinism check; expanded security-headers baseline; tightened existing sections. |
| v0.4 | 2026-04-29 | **Addendum only — body unchanged.** Renamed `mgf_apiverify` → `mgf-apiprobe`. Sibling package decision locked (NOT subpackage of `mgf-common`). Runtime dependency on `mgf-common` confirmed (inverts §15). v0.1 scope reduced; deferrals tracked in `DEFERRED.md`. See addendum block at top of this document and `HANDOFF.md`. |

---

## Table of contents

1.  [Goal](#1-goal)
2.  [Non-goals](#2-non-goals)
3.  [Primitives — the shape of the library](#3-primitives--the-shape-of-the-library)
4.  [Consumer surfaces — the four ways to use the library](#4-consumer-surfaces)
5.  [Schema source priority](#5-schema-source-priority)
6.  [Async + sync layering](#6-async--sync-layering)
7.  [The check catalogue](#7-the-check-catalogue)
8.  [Replay subsystem — cassettes](#8-replay-subsystem--cassettes)
9.  [Differential testing & drift detection](#9-differential-testing--drift-detection)
10. [Mutation testing around a baseline](#10-mutation-testing-around-a-baseline)
11. [Plugin architecture](#11-plugin-architecture)
12. [Profiles & environments](#12-profiles--environments)
13. [Observability of the verifier itself](#13-observability-of-the-verifier-itself)
14. [Project layout](#14-project-layout)
15. [Standards conformance](#15-standards-conformance)
16. [Phased roadmap](#16-phased-roadmap)
17. [Open questions](#17-open-questions-to-lock-before-phase-1)
18. [What this document is NOT](#18-what-this-document-is-not)
19. [Sign-off](#19-sign-off)

---

## 1. Goal

Build a **Python library + CLI + pytest plugin** for verifying REST APIs —
both **third-party APIs we depend on** (Teltonika, MapTiler, Stripe, Twilio,
…) and **our own APIs** (PlasmaMapper, future products). The library is
designed so a small team can:

- **Manually probe** a provider's API end-to-end and get an actionable list of
  contract / quality / security / drift issues in one run.
- **Automate** the same probes inside CI (per PR or scheduled), so a regression
  in either us or the provider surfaces immediately.
- **Replay** a recorded suite offline so CI doesn't depend on third-party
  uptime, rate limits, or credentials.
- **Detect drift** between an API's documented contract (OpenAPI), its
  observed behaviour, and its behaviour yesterday — so silent breaking
  changes have nowhere to hide.
- **Probe robustness** by mutating valid requests in disciplined, named
  ways and observing how the API copes — without paying the full cost of
  property-based fuzzing.
- **Flow CI signals** to the right place — PR comments, Slack, PagerDuty —
  so a finding becomes a conversation, not a buried JSON file.

The unifying property: **every check is structured, every finding is
actionable, every report machine-readable, every failure attributable to a
named root cause.** "It worked" / "it didn't" is not the answer this
library gives — the answer is a list of findings, each with a stable code,
severity, evidence, remediation hint, and root-cause class.

**Success criteria for v1.0:**

1. A new engineer can write a complete verification suite for a third-party
   API in **≤ 1 hour** with no prior exposure.
2. Running the same suite against the same provider in two environments
   (dev / prod) produces a meaningful diff of findings.
3. The library catches **at least 10** classes of issue that hand-written
   `pytest + httpx` tests typically miss: inconsistent error envelopes,
   pagination cursor instability, idempotency violations, silent
   acceptance of unknown fields, contract drift vs spec, header-case
   inconsistency, secret leakage in error bodies, latency-percentile
   regressions, rate-limit not actually enforced, snapshot drift in
   response shape.
4. CI usage is **hermetic by default** — replay cassettes commit to git;
   live runs are opt-in via a flag.
5. The library is **independently consumable** — works in a project with no
   `mgf_common` dependency.
6. The plugin architecture is **stable** — third parties can ship custom
   checks, auth flows, schema sources, cassette stores, and report
   writers without forking.
7. **CI loops close in one click**: a finding produces a PR comment,
   a Slack message, or a PagerDuty page, depending on profile.

---

## 2. Non-goals

To stay focused and not boil the ocean, the following are explicitly OUT of
scope at least through v1.0. Each lists the ecosystem alternative, in the
spirit of `mgf_common/docs/standards/SCOPE.md`:

| Out of scope | Why | Use this instead |
|---|---|---|
| **Load / stress / soak testing** | Different shape; benchmarking ≠ verification | `locust`, `k6`, `wrk` |
| **Full property-based fuzzing** | Schemathesis already covers schema-driven generation; we ship lighter, named mutations instead (§10) | `schemathesis` |
| **GUI client** | Phase 5 (post-v1.0); CLI + pytest cover the immediate need | `Bruno`, `Postman`, `Insomnia` |
| **Mock server / fake provider** | Different concern (server-side mocking) | `respx`, `wiremock`, `mockserver` |
| **gRPC / GraphQL / SOAP support** | REST-first; other protocols join when a consumer asks | `grpc-py`, `gql`, dedicated tools |
| **WebSocket / Server-Sent Events** | Streaming protocols need a different probe model | `websocat`, `wscat`, custom |
| **Webhook callback verification** | Async server-receiving; needs a public ingress | v2.0+ if a consumer asks |
| **Database / message-queue state assertions** | The library probes HTTP. Side-effect verification on other layers is separate. | Project-specific test fixtures |
| **Browser / E2E automation** | Out of HTTP scope | `playwright`, `cypress` |
| **Auth provider implementation** | We CONSUME auth; we don't BE auth | bring your own (OAuth client, JWT lib) |
| **OpenAPI spec generation from your code** | We CONSUME specs and (optionally) DERIVE specs from observed responses (§9.4) | FastAPI / `apispec` / `drf-spectacular` |
| **Run scheduling / cron** | The library runs on demand; cron is your CI / `systemd-timer` / `cron` | GitHub Actions cron, Codeberg Woodpecker, `cron` |
| **Time-of-day / locale-of-tester sensitivity** | Niche; if a provider is broken on Sundays, that's an incident, not a verification feature | use scheduled runs with calendar metadata |

A future PR proposing one of these is rejected with a pointer to the cited
alternative, per the SCOPE policy this project inherits.

---

## 3. Primitives — the shape of the library

Eight primitives, composed from the bottom up:

```
   Suite                 ← runs many things, produces a Report
     ├── Probe           ← one request + its expectations
     ├── Scenario        ← ordered probes sharing a Context
     │     └── Probe(s)
     ├── Mutation        ← derives variant probes from a baseline (§10)
     └── (Suite)         ← suites compose (§3.5.2)
   Check                 ← verifies one property of a (probe, response) pair
   Context               ← run-scoped variable bag, profiles, factories
   AuthFlow              ← multi-step authentication
   CassetteStore         ← persists / loads recorded interactions (§8)
   SchemaValidator       ← Pydantic / OpenAPI / JSON Schema (§5)
   Matcher               ← typed, composable response assertions (§3.1.5)
```

The next subsections describe each. Read in order — each builds on the previous.

### 3.1 `Probe` — one request

A `Probe` describes a single request the library will send (or that has been
recorded). It is a **frozen** dataclass (rule DP-03 in `mgf_common`).
Builders compose probes fluently:

```python
from mgf_apiverify import Probe, Auth, Tag

probe = (
    Probe.get("/users")
    .id("list-users")                            # stable id for cassette matching + reports
    .base("https://api.example.com/v1")
    .auth(Auth.bearer(env="EXAMPLE_TOKEN"))
    .header("Accept", "application/json")
    .query("limit", 50)
    .expect_status(200)
    .expect_latency_ms(p99=500, p50=200)         # both percentile and absolute budgets supported
    .expect_schema(UserListResponse)              # Pydantic model
    .expect_header("X-Request-Id")
    .with_idempotency_key("req-001")
    .tags(Tag.READ, Tag.PUBLIC)                   # for filtering (§3.5)
    .description("List the first page of users")
)
```

Probes carry their **expectations** alongside the request. This is the key
ergonomic move: a probe is "what I'm sending AND what I expect back", not
two separate objects. Checks read both halves.

A `Probe` builder is a strict-typed fluent API; mypy catches typos.

#### 3.1.1 Variable interpolation

Probes accept **template variables** in their string-valued fields, resolved
at run time against the active `Context` (§3.4):

```python
Probe.get("/users/{{user_id}}")
    .header("X-Tenant", "{{tenant_slug}}")
    .body({"comment": "{{message}}"})
```

The default templating engine is a tiny, deliberately-restricted subset of
Jinja: only `{{var}}` and `{{var | filter}}` are supported. No expressions,
no loops, no `{% if %}`. The constraint is intentional — testing is stateful
enough without templating becoming Turing-complete.

Filters: `urlencode`, `lower`, `upper`, `default(value)`, `quote`, `sha256`,
`base64`, `factory(name)` (calls a factory; see §3.4.2). Filter set is small
and pluggable (§11).

#### 3.1.2 Polling — `wait_for` and long-running operations (LROs)

Real APIs have two common async patterns the library supports as first
class:

**Pattern A — eventual consistency / poll until stable.** A simple poll
on the same URL until expectations are met:

```python
(Probe.get("/jobs/{{job_id}}")
     .expect_status(200)
     .expect_body_jsonpath("$.status", equals="completed")
     .wait_for(
         interval_ms=500,
         timeout_ms=30_000,
         retries_for_status=[404, 425, 503],
     ))
```

**Pattern B — long-running operation (LRO).** The provider returns
`202 Accepted` + a `Location` header pointing to a status resource; the
client polls that. Common in cloud APIs (AWS, Stripe, large file
processing, FOTA WEB workflows in PlasmaMapper):

```python
Probe.lro_post("/imports")                       # convenience for the LRO pattern
    .body({"url": "..."})
    .expect_initial_status(202)
    .extract_status_url(header="Location")        # follow Location header
    .poll_status(
        interval_ms=1_000,
        timeout_ms=300_000,
        terminal_states=["succeeded", "failed", "cancelled"],
        success_states=["succeeded"],
        status_jsonpath="$.state",
    )
    .expect_final_status(200)
    .expect_schema(ImportResult)
```

The LRO machinery records the **initial response and every poll** in the
cassette so replay reproduces the dance faithfully. A finding
`stability.lro.timeout` fires if the operation never reaches a terminal
state; `stability.lro.unexpected-state` fires if the final state isn't in
`success_states`.

Both patterns are deterministic-on-replay: cassettes record every poll
exactly. The replay-time interval is collapsed to zero (no real waiting).

#### 3.1.3 Extraction

A probe extracts values from its response into the `Context` for later
probes:

```python
Probe.post("/users")
    .body({"email": "{{email}}"})
    .expect_status(201)
    .extract("user_id", "$.id")                   # JSONPath on JSON body
    .extract("etag", header="ETag")               # response header
    .extract("location", header="Location", parse=re.compile(r"/users/(?P<id>\w+)"))
    .extract("first_email", "$.users[*].email | first")  # JMESPath also supported
    .extract("blob", body=True, optional=True)    # raw bytes, optional
```

Extraction failures (path doesn't match, regex doesn't capture) produce a
`scenario.extract.failed` finding and abort the surrounding scenario unless
marked `extract(..., optional=True)`.

The extraction engine accepts JSONPath, JMESPath, regex, raw header, and
raw body. Plugin-extensible (§11) for XML / CBOR / MessagePack.

#### 3.1.4 Bodies — JSON, multipart, streaming, downloads

Beyond JSON, Probes support every body shape REST APIs use:

```python
# JSON (default)
Probe.post("/users").body({"email": "a@b.c"})

# Form-encoded (application/x-www-form-urlencoded)
Probe.post("/auth/token").form({"grant_type": "password", "username": "...", "password": "..."})

# Raw bytes (application/octet-stream)
Probe.post("/upload").body_bytes(b"...binary...", content_type="application/octet-stream")

# Multipart with files + fields (multipart/form-data)
Probe.post("/uploads")
    .multipart()
    .field("metadata", json.dumps({"name": "x"}))
    .file("photo", path="fixture.jpg", content_type="image/jpeg")
    .file("attachment", bytes_=open("doc.pdf", "rb").read(), filename="doc.pdf",
          content_type="application/pdf")

# Streamed upload (chunked transfer-encoding) — useful for large files
Probe.post("/streams")
    .body_streamed(chunks=iter_chunks(path), content_type="application/octet-stream")
    .expect_status(201)

# Streamed download — verify large response without loading into memory
Probe.get("/exports/{{export_id}}")
    .expect_status(200)
    .expect_content_type("text/csv")
    .expect_body_streamed(
        chunks_validator=line_count_in_range(min=1, max=1_000_000),
        max_bytes=5 * 1024 * 1024 * 1024,        # 5 GB cap
    )
    .expect_checksum(sha256="{{expected_sha}}")  # optional integrity check
```

**Cassette handling for binary / multipart / streamed bodies:** large
bodies are persisted out-of-line as separate files alongside the cassette
(`cassettes/<name>.yml` + `cassettes/<name>.assets/<sha256>.bin`); the
cassette references them by SHA-256. Configurable `max_inline_bytes`
threshold (default 16 KiB).

Compression is auto-handled (gzip, br, zstd via httpx) — the
verification reads decompressed bodies but cassettes can store raw bytes
when `preserve_compression=True` is set on the probe.

#### 3.1.5 Matchers DSL — typed, composable response assertions

Beyond `expect_schema` (whole-body validation), probes assert on specific
parts of responses via a composable matcher DSL:

```python
from mgf_apiverify import expect, like, gt, lt, in_, is_uuid, is_iso8601_z, matches

probe.expect(
    # Status
    status=in_(200, 304),

    # Headers — case-insensitive keys, value matchers
    headers={
        "content-type": matches(r"^application/json"),
        "x-rate-limit-remaining": gt(0),
        "x-request-id": is_uuid(),
    },

    # Body field-by-field assertions (JSONPath keys → matchers)
    body={
        "$.id": is_uuid(),
        "$.email": matches(r"^[^@]+@[^@]+$"),
        "$.created_at": is_iso8601_z(),
        "$.role": in_("owner", "operator", "manager", "driver"),
        "$.balance.amountMinor": gt(0),
        "$.balance.currency": like("USD"),
        # Cross-field semantic assertion
        "$.total": equals_count_of("$.items"),
        # Conditional: only check when condition holds
        "$.archived_at": is_iso8601_z().when_present(),
    },

    # Latency budget per percentile
    latency=p99(lt(500)),
)
```

Matchers compose with `&` and `|`:

```python
probe.expect(body={"$.score": (gt(0) & lt(100)) | equals(None)})
```

Built-in matchers (extensible via §11):

| Category | Matchers |
|---|---|
| Equality / range | `equals`, `not_equals`, `gt`, `gte`, `lt`, `lte`, `in_`, `not_in` |
| String | `matches` (regex), `starts_with`, `ends_with`, `contains`, `like` (case-insensitive) |
| Type / format | `is_uuid`, `is_iso8601_z`, `is_email`, `is_url`, `is_iban`, `is_e164` (phone), `is_iso_country`, `is_currency_iso4217`, `is_imei`, `is_ipv4`, `is_ipv6` |
| Collection | `is_empty`, `len_eq`, `len_in_range`, `all_of`, `any_of`, `unique_values` |
| Cross-field | `equals_count_of`, `monotonic_increasing` (for arrays), `references_existing` |
| Conditional | `when_present`, `when_status`, `optional`, `forbidden_field` |

The library ships a **semantic field validator** library: things like
`is_iban` validate the IBAN check digits, `is_imei` validates the
Luhn check — not just regex shape. This catches API bugs that
schema validation misses. Domain-specific validators
(`is_teltonika_imei`, `is_mapTiler_tile_url`) live in plugins.

### 3.2 `Check` — one verification rule

A `Check` is a callable that takes a `(probe, response, ctx)` tuple and
returns zero or more `Finding`s:

```python
from mgf_apiverify import Check, Finding, Severity, RootCause

class HasRequestIdHeader(Check):
    """Provider should echo a request-id we sent, or generate one."""
    code = "header.request-id.missing"
    severity_default = Severity.WARN
    root_cause_default = RootCause.PROVIDER_OBSERVABILITY

    def run(self, probe, response, ctx) -> Iterable[Finding]:
        rid = response.headers.get("X-Request-Id")
        if not rid:
            yield Finding(
                code=self.code,
                severity=self.severity_default,
                root_cause=self.root_cause_default,
                message="Response missing X-Request-Id header.",
                evidence={"headers": dict(response.headers)},
                remediation=(
                    "Provider should add X-Request-Id to every response so "
                    "client logs and provider logs can be correlated end-to-end."
                ),
                threat_refs=["OWASP-API-9"],
            )
```

Findings carry:

| Field | Meaning |
|---|---|
| `code` | Stable identifier, dotted, lowercase. **Part of the public contract.** |
| `severity` | `info` / `warn` / `critical`. CI fails on `critical`; warns on `warn`. |
| `root_cause` | One of a small enum (§3.2.1) — for failure attribution. |
| `message` | Human-readable, one sentence, ends with a period (rule EH-09). |
| `evidence` | Machine-readable JSON-able dict (status, headers, body diff, …). |
| `remediation` | Actionable next step. |
| `probe` | Reference to the probe that produced it. |
| `threat_refs` | Optional list of mappings (OWASP API Top 10, CWE IDs, …). |
| `tags` | Optional tags inherited from the probe + check. |
| `since` | First seen in cassette (when known) — turns a finding into a regression marker. |

#### 3.2.1 Failure attribution — `RootCause`

Every finding carries a `RootCause` enum so dashboards can aggregate
meaningfully ("70% of our failures last week were AUTH"). The categories:

| `RootCause` | Meaning |
|---|---|
| `NETWORK` | Connection refused, DNS, TLS handshake, timeout. The probe never got a real response. |
| `AUTH` | 401/403 in a flow that should be authenticated; token expiry, scope mismatch. |
| `RATE_LIMIT` | 429 / `Retry-After`. Suite was throttled. |
| `PROVIDER_5XX` | Provider returned 5xx. Their fault until proven otherwise. |
| `PROVIDER_CONTRACT` | 4xx/2xx with shape violations — provider's schema lies or has drifted. |
| `PROVIDER_QUALITY` | Latency, headers-case, missing trace id — non-functional. |
| `PROVIDER_SECURITY` | Secret echo, weak headers, error leaks. |
| `PROVIDER_OBSERVABILITY` | Missing request id, missing rate-limit headers. |
| `CLIENT_REQUEST_SHAPE` | Our probe sent the wrong shape (mutation tests find these on purpose). |
| `CASSETTE_DRIFT` | Replay differs from cassette. |
| `SPEC_DRIFT` | Spec disagrees with reality. |
| `SUITE_LOGIC` | Suite itself has a bug — extraction failed, scenario state corrupt. |
| `INFRASTRUCTURE` | Harness/runtime issue (plugin load failure, transport error not attributable to provider). |
| `UNKNOWN` | Use sparingly; warns at runtime so we can refine the taxonomy. |

A check declares its **default** root cause; an individual finding can
override (e.g., a generic latency check picks `PROVIDER_QUALITY` by
default but escalates to `NETWORK` if the latency was zero / negative).

#### 3.2.2 Severity policy — overridable per suite

A check declares a default severity. A suite may override:

```python
suite.severity_overrides = {
    "quality.latency.over-budget": Severity.WARN,
    "contract.error.no-stacktrace": Severity.CRITICAL,
}
```

Demotion (critical → warn → info) is always allowed. Promotion (info → warn
→ critical) is also allowed but logged at INFO with `audit=true`, since
"making a thing fail more often" deserves a paper trail.

### 3.3 `Scenario` — a chained sequence of probes

A `Scenario` is an **ordered sequence of probes that share a `Context`**.
Used for any flow more complex than a single request — CRUD, paginate,
auth-then-call, create-then-poll, etc. A scenario has setup, steps, and
teardown; teardown runs even if a step fails.

```python
from mgf_apiverify import Scenario, Probe

users_crud = Scenario(
    name="users CRUD round-trip",
    description="Create a user, read it back, update it, delete it.",
    setup=[
        Probe.post("/auth/token")
            .body({"client_id": "{{client_id}}", "secret": "{{client_secret}}"})
            .extract("access_token", "$.access_token"),
    ],
    steps=[
        Probe.post("/users")
            .auth(Auth.bearer_from("access_token"))
            .body({"email": "{{factory:unique_email}}"})       # see §3.4.2
            .expect_status(201)
            .expect_schema(UserResponse)
            .extract("user_id", "$.id"),

        Probe.get("/users/{{user_id}}")
            .auth(Auth.bearer_from("access_token"))
            .expect_status(200)
            .expect(body={"$.email": equals_template("{{previous.email}}")}),

        Probe.patch("/users/{{user_id}}")
            .auth(Auth.bearer_from("access_token"))
            .body({"display_name": "Updated"})
            .expect_status(200)
            .expect(body={"$.display_name": equals("Updated")}),
    ],
    teardown=[
        Probe.delete("/users/{{user_id}}")
            .auth(Auth.bearer_from("access_token"))
            .expect_status_in(204, 404)
            .always(),
    ],
    on_step_failure="abort",                        # 'abort' | 'continue' | 'skip-rest'
    tags=[Tag.AUTHED, Tag.MUTATING],
    idempotent=True,                                # opt into stability.scenario.idempotent
)
```

What the scenario primitive enables that bare probes cannot:

- **State extraction** — outputs of step N are inputs to step N+1.
- **Idempotency verification** — re-run the same scenario with the same
  context and assert byte-identical responses.
- **Cleanup discipline** — teardown always runs.
- **Cross-step checks** — `equals_template("{{previous.email}}")` and
  `references_existing("$.user_id", lookup="user_id")` reach back into
  prior responses.

A `Suite` accepts a free mix of bare `Probe`s and `Scenario`s.

### 3.4 `Context` — run-scoped variable bag

Every run carries a `Context`:

```python
ctx = Context(profile="staging")
ctx.set("client_id", os.environ["EXAMPLE_CLIENT_ID"])
ctx.set("run_id", uuid7())
```

The context is layered:

1. **Profile defaults** — loaded from the active profile (§12).
2. **Environment variables** — `Context.from_env(prefix="APIVERIFY_")`.
3. **Suite-level constants** — `suite.context_defaults = {...}`.
4. **Scenario-level setup** — `scenario.setup` probes' extractions.
5. **Per-step extractions** — accumulate into the active scenario's context.

Resolution is leftmost-wins. Scenario context is **scoped to that scenario**
(extractions don't leak to siblings). Profile defaults DO span scenarios.

Variables can be marked `secret=True` at definition time. Secret variables:
- Are redacted from cassettes (§8).
- Are redacted from finding evidence and the human report.
- Cannot be templated into URLs or logs (only headers and bodies — defends
  against accidental leaks via access logs).

#### 3.4.1 Built-in variables

Every Context auto-populates a small set of read-only variables:

| Variable | Value |
|---|---|
| `{{run_id}}` | UUID v7 generated at suite start; identical across all probes in the run; useful for unique test data |
| `{{run_ts}}` | ISO 8601 Z timestamp at suite start |
| `{{profile}}` | Active profile name |
| `{{suite_name}}` | The suite's name |
| `{{previous.X}}` | Inside a scenario, the previous step's response — `previous.body.*`, `previous.headers.*`, `previous.status` |

#### 3.4.2 Factories — eliminate hand-typed test data

Hand-typed test data is brittle (duplicate emails fail on re-run) and
pollutes provider state. Factories solve both:

```python
ctx.factories.register("unique_email", lambda: f"verifier-{uuid4().hex[:8]}@example.com")
ctx.factories.register("future_date", lambda: (date.today() + timedelta(days=30)).isoformat())

# In probes:
Probe.post("/users").body({
    "email": "{{factory:unique_email}}",
    "trial_expires": "{{factory:future_date}}",
})
```

Built-in factories ship with the library:

| Factory | Generates |
|---|---|
| `unique_email` | `verifier-<runid>-<n>@example.com` (auto-incremented) |
| `unique_username` | `verifier-<n>` |
| `unique_slug` | `verifier-<runid-prefix>-<n>` |
| `uuid4`, `uuid7` | UUIDs |
| `iso_now`, `iso_in(seconds=…)`, `iso_ago(seconds=…)` | ISO 8601 Z timestamps |
| `random_int(min, max)`, `random_float(min, max)` | (seeded by `run_id` for replay determinism) |
| `random_string(length, alphabet=…)` | Random strings, deterministic per run |
| `lorem_words(n)`, `lorem_sentence`, `lorem_paragraph` | Lorem ipsum |
| `iban`, `e164_phone`, `iso_country`, `currency_iso4217` | Valid synthetic instances |

Factories are deterministic per `run_id` so replay reproduces the same
data. A probe that depends on uniqueness across runs uses
`{{factory:unique_email}}` (which uses `run_id`); a probe that wants
fresh-per-call uses `{{factory:unique_email_per_call}}` (NOT replay-safe;
flagged as `non_deterministic=True`).

**Faker integration** — a `faker` plugin (Phase 1.5) ships factories
backed by `Faker` for richer generation (full names, addresses, company
names, etc.) when locale-aware data matters.

### 3.5 `Suite` — the run orchestrator

A `Suite` runs N units (probes + scenarios) × M checks against a target
and produces a `Report`:

```python
suite = Suite(
    name="example.com v1 contract",
    base_url="https://api.example.com/v1",
    auth_flow=OAuth2ClientCredentials(
        token_url="https://auth.example.com/oauth2/token",
        client_id_var="client_id",
        client_secret_var="client_secret",
    ),
    profile="staging",
    units=[probe1, probe2, scenario1, scenario2],
    checks=[*DEFAULT_CHECKS, HasRequestIdHeader()],
    mode=Suite.Mode.LIVE,
    cassette=FileCassette("cassettes/example.yml"),
    concurrency=8,
    rate_limit=RateLimit(per_second=10, burst=20),
    fail_fast=False,
    severity_overrides={"quality.latency.over-budget": Severity.WARN},
    include_tags=[],
    exclude_tags=[Tag.PAID_TIER],
)
report = await suite.run()
```

Suite-level features:

- **Tag filtering** (`include_tags`, `exclude_tags`) — at unit and check level.
- **Concurrency** — bounded fan-out, with per-host limits.
- **Client-side rate limit** — global token bucket; helps avoid 429 storms
  while testing rate-limited APIs.
- **Fail-fast vs. gather-all** — default gather-all so one critical
  finding doesn't mask twenty others.
- **Severity overrides + audit on promotion** — see §3.2.2.
- **Pluggable report writers** (§3.7).

#### 3.5.1 Probe templates — DRY for medium suites

When dozens of probes share most of their shape:

```python
authed = ProbeTemplate(
    base="https://api.example.com/v1",
    headers={"Accept": "application/json", "User-Agent": "apiverify/0.1"},
    auth=Auth.bearer_from("access_token"),
    expect_header("X-Request-Id"),
    expect_latency_ms(p99=1000),
    tags=[Tag.AUTHED],
)

# Inherit from template; override + add per-probe.
list_users = authed.get("/users").id("list-users").expect_schema(UserListResponse)
get_user = authed.get("/users/{{user_id}}").id("get-user").expect_schema(UserResponse)
```

A template is a *factory* — every `template.get(path)` returns a fresh
probe builder with the template's defaults applied. Templates can extend
templates (`paid_authed = authed.extend(tags=[Tag.PAID_TIER])`).

#### 3.5.2 Suite composition

Large APIs break naturally into multiple suites (auth, users, billing,
admin). Suites compose:

```python
master = Suite.compose(
    name="full contract",
    suites=[auth_suite, users_suite, billing_suite, admin_suite],
    profile="staging",                              # propagates
    severity_overrides={...},                       # merged with each child's
    only_tags=[Tag.SMOKE],                          # filter applied to all children
)
```

Composed suites share a single `Context` and report. A unit's findings
are tagged with its source-suite name so the report is groupable.

### 3.6 `AuthFlow` — multi-step authentication

Bearer tokens are easy. Real APIs aren't always bearer tokens. `AuthFlow`
is the ABC for any sequence of steps that produces credentials a probe
can attach. Built-in flows:

| Flow | Use |
|---|---|
| `Auth.bearer(env=…)` | Static token (env var or literal). |
| `Auth.basic(user, pass)` | HTTP Basic. |
| `Auth.api_key(header, value)` | Header-based API key. |
| `Auth.api_key_query(name, value)` | Query-parameter API key (legacy). |
| `Auth.cookie(name, value)` | Cookie-based session. The cookie persists across the suite. |
| `OAuth2ClientCredentials` | RFC 6749 §4.4 — server-to-server; tokens cached + refreshed. |
| `OAuth2AuthorizationCode` | RFC 6749 §4.1 with refresh — for tests against user-context endpoints. |
| `OAuth2DeviceCode` | RFC 8628 — device flow; useful for record-time auth on machines without browsers. |
| `MfaTotp` | TOTP-based MFA layered on top of another flow. Codes from a seed env var; rotated per run. |
| `Custom(...)` | A user callable: `(context) -> Credentials`. |

Flow lifecycle:

- Flow invoked once at suite start (or once per scenario, if scoped) via
  the **preflight** step.
- Resulting credentials cached in `Context` under a known key.
- A probe declares `.auth(flow)` or `.auth(Auth.bearer_from("access_token"))`
  to consume the cached credentials.
- The flow declares its credentials' **expiry**; on expiry, the suite
  re-runs the flow transparently. Expiry events emit a structured log.

`AuthFlow`s are pluggable (§11). v1.0+ planned: AWS SigV4, Google
service-account JWT, Twilio API-key.

### 3.7 `Report` + writers

A `Report` is the structured result of a suite run:

```python
@dataclass(frozen=True)
class Report:
    suite: str
    profile: str
    started_at: datetime
    duration_ms: int
    target: TargetSummary           # base_url, mode, cassette path, …
    units: list[UnitResult]         # one per probe or scenario
    findings: list[Finding]
    metrics: RunMetrics             # counts, durations, latency percentiles
    fingerprint: str                # stable hash for diffing (§9.1)
    root_cause_breakdown: dict[RootCause, int]   # for §3.2.1 attribution
```

Built-in writers:

- `human` — colourised text summary + grouped detail.
- `json` — full structured dump (the canonical machine format).
- `junit` — for CI dashboards.
- `markdown` — long-form, suitable for GitHub PR comments.
- `sarif` — for security-finding ingestion (GitHub Advanced Security, Sonar).
- `slack`, `discord`, `teams` — webhook posts of summary + critical findings.
- `pagerduty` — incidents on critical findings (with deduplication keys).
- `gitlab_mr_note`, `github_pr_comment` — direct PR/MR comments via the
  forge's API.
- `csv`, `prometheus` — minor formats for trend tracking.

Writers are pluggable (§11). Each writer documents the env vars / secrets
it expects (`SLACK_WEBHOOK_URL`, `PAGERDUTY_ROUTING_KEY`, etc.).

---

## 4. Consumer surfaces

One engine. Four ergonomic shells.

### 4.1 CLI — manual verification

```bash
# Run a suite defined in YAML (declarative, version-controlled)
apiverify run suites/maptiler.yml

# Probe a single endpoint with default checks (one-off exploration)
apiverify probe https://api.example.com/v1/users \
    --bearer "$EXAMPLE_TOKEN" \
    --schema UserListResponse \
    --check schema,contract,quality,security \
    --latency-budget p99=500ms,p50=200ms

# Compare two reports (drift between two runs)
apiverify diff before.json after.json --by code,severity,root_cause

# Compare actual response shape with documented OpenAPI (drift vs spec)
apiverify drift --spec https://api.example.com/openapi.json --record cassettes/example.yml

# Snapshot — first run records, subsequent runs flag any shape change
apiverify run suites/example.yml --snapshot snapshots/example/

# Record a live run as a cassette for replay
apiverify run suites/maptiler.yml --record cassettes/maptiler-2026-04-28.yml

# Replay a cassette (no network)
apiverify run suites/maptiler.yml --replay cassettes/maptiler-2026-04-28.yml

# Refresh a cassette (re-run live, rewrite cassette in place)
apiverify cassette refresh cassettes/maptiler.yml

# Generate a starter suite from an OpenAPI spec (one probe per operation)
apiverify scaffold --spec https://api.example.com/openapi.json --out suites/example.yml

# Run a mutation pass around an existing suite
apiverify mutate run suites/maptiler.yml --kinds drop_required,wrong_type,oversize

# List / describe checks
apiverify checks list
apiverify checks describe contract.error.envelope-shape

# Self-check
apiverify doctor
```

CLI output is human-readable by default. `--json` flips to machine output;
`--format markdown|junit|sarif|slack` selects writer.

The CLI honours a project-local `.apiverify.toml` for default flags
(profile, concurrency, default checks, tag filters). Environment
variables prefixed `APIVERIFY_` override the file; CLI flags override env.

### 4.2 pytest plugin — automated testing

```python
import pytest
from mgf_apiverify.pytest import apiverify

@pytest.fixture
def example_api(apiverify):
    return apiverify(
        suite="suites/example.yml",
        profile="ci",
        replay="cassettes/example.yml",
    )

async def test_users_endpoint(example_api):
    await example_api.run_unit("list-users")
```

The plugin can also generate **one pytest test per probe / scenario step**:

```python
# tests/test_example_api.py
from mgf_apiverify.pytest import generate_tests
generate_tests(suite="suites/example.yml", profile="ci")
```

`pytest --collect-only` shows one test per probe / scenario step. This is
the difference between "1 test failed (with a giant traceback)" and "12
tests failed: 3 contract, 5 quality, 4 security" on the dashboard.

When a finding fires:
- pytest fails the test with the finding's message + evidence diff.
- The `code` becomes part of the test ID.
- `severity=info` is a `pytest.warns`, not a fail.
- `severity=warn` honours `--strict-warn` (off by default).

### 4.3 Programmatic / library

```python
from mgf_apiverify import Probe, Suite, DEFAULT_CHECKS

report = await Suite(
    name="my smoke",
    units=[Probe.get("/healthz").expect_status(200)],
    checks=DEFAULT_CHECKS,
).run()

if report.has_critical:
    raise SystemExit(1)
```

### 4.4 CI integration patterns — close the loop

A finding that lands in a JSON file nobody reads is no finding at all.
The library ships first-class **CI integration patterns** so a finding
becomes a conversation:

#### 4.4.1 Per-PR scheduled live run

```yaml
# .woodpecker.yml fragment
steps:
  - name: apiverify-pr
    image: ghcr.io/.../uv:py312
    secrets: [example_bearer_token, slack_webhook_url]
    when: { event: pull_request }
    commands:
      - uv run apiverify run suites/external.yml \
          --profile ci-pr \
          --replay cassettes/external.yml \
          --format markdown \
          --output report.md
      - uv run apiverify report comment-pr \
          --report report.json \
          --forge codeberg \
          --pr ${CI_PR_NUMBER}
```

The `comment-pr` subcommand posts (or updates an existing) sticky comment
on the PR with the finding summary and a diff against `main`'s last
recorded report.

#### 4.4.2 Scheduled live run against production

```yaml
# Codeberg cron / Woodpecker cron
when:
  event: cron
  cron: ["live-prod"]
steps:
  - name: live-prod-verification
    secrets: [prod_readonly_bearer, slack_webhook_url, pagerduty_key]
    commands:
      - uv run apiverify run suites/external.yml \
          --profile prod-readonly \
          --live \
          --severity-override quality.latency.regression=critical \
          --writer slack \
          --writer pagerduty
```

Critical findings page on-call; warns post to Slack.

#### 4.4.3 Scheduled drift watchdog

```yaml
when:
  event: cron
  cron: ["drift-watch"]
steps:
  - name: spec-drift
    commands:
      - uv run apiverify drift \
          --spec https://api.example.com/openapi.json \
          --replay cassettes/external.yml \
          --writer slack \
          --severity-floor warn
```

Catches cases where the provider's published OpenAPI spec evolves and the
real API has already drifted from it (or the other way around).

#### 4.4.4 Bisect — when did the regression appear?

```bash
apiverify bisect \
    --suite suites/example.yml \
    --cassettes cassettes/external/ \
    --until-finding contract.error.envelope-shape
```

Walks a directory of dated cassettes (e.g., one per day from a scheduled
record job), runs the suite against each, finds the cassette where a
specific finding first appeared. Useful for "this broke sometime last
month — when?" investigations.

---

## 5. Schema source priority

A probe's `expect_schema(...)` accepts:

1. **Pydantic v2 models** (preferred) — most ergonomic; validates with rich
   typing; round-trips with FastAPI / our own services.

2. **OpenAPI specs** — load a spec, reference an operationId, the library
   resolves the response schema. Suite-level `auto_schema_from_spec=spec`
   auto-attaches schemas to probes whose URL+method+status match a spec
   operation.

3. **Raw JSON Schema** — for providers that publish neither but maintain
   a JSON Schema fragment.

Internally all three normalise to a single validation surface
(`SchemaValidator` ABC). v0 ships Pydantic; OpenAPI lands in Phase 1.5;
JSON Schema in Phase 2. `SchemaValidator` is pluggable (§11).

---

## 6. Async + sync layering

`mgf_apiverify` is **async-first** — built on `httpx.AsyncClient`. A
`SyncSuite` wrapper provides a sync entry point for casual scripts.

```python
report = await suite.run()                          # async (default)

from mgf_apiverify import SyncSuite
report = SyncSuite(...).run()                       # sync wrapper
```

The CLI is sync at the command boundary (uses the wrapper); pytest tests
are typically async (rule TS-12).

The HTTP transport is pluggable. `httpx.AsyncClient` is the default;
others (raw-socket, offline-only, etc.) drop in via the `HttpTransport`
ABC.

---

## 7. The check catalogue

Built-in checks organised by category. Each ships with a stable `code`
(the dotted path); each is independently togglable. **The catalogue is
the project's primary value proposition** — a curated list of the things
APIs commonly get wrong.

### 7.1 Schema (the contract layer)

| Code | What it verifies |
|---|---|
| `schema.status.allowed` | Response status is in the probe's declared set. |
| `schema.body.matches` | Response body validates against the declared schema. |
| `schema.body.extra-fields` | Response carries no unexpected top-level fields. |
| `schema.body.required-fields` | All schema-required fields present. |
| `schema.headers.required` | All required headers present on response. |
| `schema.content-type.honored` | `Content-Type` matches `Accept`'s preference. |
| `schema.charset.utf8` | Response declares + uses UTF-8. |
| `schema.length.matches-content-length` | `Content-Length` matches actual bytes. |

### 7.2 Contract (cross-endpoint consistency)

| Code | What it verifies |
|---|---|
| `contract.error.envelope-shape` | All 4xx/5xx use the same documented error shape. |
| `contract.error.has-trace-id` | Errors include a stable trace identifier. |
| `contract.error.no-stacktrace` | 5xx body does NOT include a stack trace (rule SC-11). |
| `contract.error.actionable-message` | Error message is human-readable. |
| `contract.error.problem-details` | Errors conform to RFC 7807 / 9457 Problem Details when claimed. |
| `contract.method.not-allowed` | `POST` on a `GET`-only endpoint returns 405. |
| `contract.unknown-fields.rejected` | Unknown request body fields produce 422. |
| `contract.auth.boundary` | Unauthenticated → 401; insufficient scope → 403. |
| `contract.cross-tenant.404-not-403` | Cross-tenant access returns 404. |
| `contract.options.preflight-handled` | `OPTIONS` per CORS spec. |
| `contract.head-equals-get` | `HEAD` returns same headers + zero body as `GET`. |

### 7.3 Stability (deterministic + concurrency-safe)

| Code | What it verifies |
|---|---|
| `stability.idempotency.same-key-same-response` | Two requests with same `Idempotency-Key` → same response. |
| `stability.idempotency.no-key-no-promise` | Repeating a non-idempotent request without a key MAY produce different responses (informational). |
| `stability.determinism.repeated-read-stable` | A pure-read endpoint returns identical responses across N consecutive calls. |
| `stability.pagination.cursor-stable` | A cursor returned in page N still works on a later request. |
| `stability.pagination.no-duplicates` | Sequential pages don't overlap. |
| `stability.pagination.complete` | Walking pages yields the expected total. |
| `stability.pagination.link-header` | `Link: rel=next` (RFC 5988) present and parseable when used. |
| `stability.list.ordering-deterministic` | A list endpoint with same query returns rows in the same order. |
| `stability.timezone.utc-z` | All datetime fields end with `Z` or `+00:00`. |
| `stability.eventual-consistency.timeout` | A `wait_for` probe didn't reach its expected state. |
| `stability.lro.timeout` | A long-running operation never reached a terminal state. |
| `stability.lro.unexpected-state` | LRO ended in a state outside the declared success states. |
| `stability.scenario.idempotent` | Re-running an idempotent scenario produces the same final state. |
| `stability.concurrency.race-safe` | N parallel identical requests produce the same response, no 5xx, no duplicates. |

### 7.4 Quality (operational discipline)

| Code | What it verifies |
|---|---|
| `quality.latency.over-budget` | A single response time exceeds the per-probe budget. |
| `quality.latency.percentile.over-budget` | Aggregate p50/p95/p99 across N runs exceeds budget. |
| `quality.latency.regression` | Median latency for this probe regressed > X% from the last cassette. |
| `quality.id-format.uuid` | Fields named `id`, `*_id` are valid UUIDs (configurable per field). |
| `quality.json.well-formed` | Body parses as JSON when content-type says so. |
| `quality.json.no-trailing-comma` | Server JSON is RFC-clean. |
| `quality.headers.canonical-case` | Custom headers use a single canonical case across responses. |
| `quality.compression.honored` | `Accept-Encoding: gzip` returns gzipped responses for non-trivial bodies. |
| `quality.compression.size-actually-smaller` | Compressed body is actually smaller than uncompressed (bug check). |
| `quality.cache.headers-coherent` | `Cache-Control` + `ETag` + `Last-Modified` are mutually consistent. |
| `quality.cache.etag-stable-on-pure-read` | Two pure-read requests return the same `ETag`. |

### 7.5 Security signals

| Code | What it verifies |
|---|---|
| `security.no-secret-echo` | Request bearer / api-key not in any response field. |
| `security.cors.headers-sane` | `Access-Control-Allow-Origin` not `*` on authenticated endpoints. |
| `security.rate-limit.headers-present` | A 429 response carries `Retry-After`. |
| `security.rate-limit.actually-enforced` | Burst of N requests in window M actually triggers 429. |
| `security.rate-limit.headers-truthful` | `X-Rate-Limit-Remaining` decrements consistently. |
| `security.https.enforced` | An `http://` request is redirected to `https://`. |
| `security.auth.no-leak-in-error` | A 401 doesn't enumerate (rule SC-14 mirror). |
| `security.auth.timing-stable` | Response time is consistent for valid/invalid auth (no timing oracle). |
| `security.tls.version-modern` | TLS handshake uses ≥ TLS 1.2. |
| `security.headers.hsts` | `Strict-Transport-Security` present, `max-age` ≥ 31536000. |
| `security.headers.x-content-type-options` | `X-Content-Type-Options: nosniff`. |
| `security.headers.x-frame-options` | `X-Frame-Options` or CSP `frame-ancestors`. |
| `security.headers.referrer-policy` | `Referrer-Policy` set. |
| `security.headers.permissions-policy` | Permissions-Policy present where relevant. |
| `security.headers.content-security-policy` | CSP present on HTML responses. |
| `security.error.no-internal-paths` | 5xx body does NOT include filesystem paths, hostnames, or library versions. |
| `security.cookies.secure-and-httponly` | Auth cookies have `Secure` + `HttpOnly` + sane `SameSite`. |

### 7.6 Hygiene

| Code | What it verifies |
|---|---|
| `hygiene.path.canonical` | Trailing-slash policy consistent across the API. |
| `hygiene.versioning.in-url-or-header` | API has a versioning strategy (URL prefix or `Accept` header). |
| `hygiene.deprecation.signaled` | Deprecated endpoints carry `Deprecation` / `Sunset` headers. |
| `hygiene.userid.opaque-not-incrementing` | IDs aren't auto-incrementing integers (information leak). |

### 7.7 Drift (from §9 — cross-reference)

| Code | What it verifies |
|---|---|
| `drift.spec.shape-mismatch` | Observed response shape differs from declared schema. |
| `drift.spec.field-removed` | Field documented in spec is missing from a real response. |
| `drift.spec.field-added` | Field present in real responses is undocumented. |
| `drift.spec.type-changed` | A field's runtime type doesn't match its declared type. |
| `drift.spec.status-undocumented` | Observed status code not in spec's response set. |
| `drift.spec.required-field-missing` | Spec marks field required; reality returns null/missing. |
| `drift.cassette.field-removed` | Field present in previous cassette is missing now. |
| `drift.cassette.field-added` | New field appearing since last cassette. |
| `drift.cassette.status-changed` | Same probe returns a different status code than recorded. |
| `drift.cassette.error-shape-changed` | Error envelope shape differs from previous recording. |
| `drift.cassette.latency-shifted` | Median latency shifted ≥ N% from previous cassette. |
| `drift.snapshot.changed` | Snapshot test detected response shape change (§9.6). |
| `drift.dual-target.body-mismatch` | A/B comparison: two targets returned different bodies. |
| `drift.dual-target.status-mismatch` | A/B: two targets returned different status codes. |
| `drift.dual-target.latency-skew` | A/B: two targets' latencies differ ≥ N%. |

### 7.8 Mutation (from §10)

| Code | What it verifies |
|---|---|
| `mutation.input.silent-acceptance` | Mutated invalid input was accepted with 2xx. |
| `mutation.input.500-on-malformed` | Malformed input produced 5xx. |
| `mutation.input.reflected-payload` | Mutated payload appeared verbatim in response. |
| `mutation.input.timing-leak` | Response time differs significantly between valid/invalid. |
| `mutation.input.error-message-leaks-internal` | Validation error reveals internal field names / SQL / paths. |

### 7.9 Scenario (cross-step semantics)

| Code | What it verifies |
|---|---|
| `scenario.extract.failed` | An `extract(...)` call failed. |
| `scenario.teardown.failed` | A teardown probe didn't return its expected status. |
| `scenario.create-then-read.mismatch` | Object read back doesn't match what was created. |
| `scenario.create-then-list.missing` | A newly-created entity is not in the subsequent list response. |
| `scenario.context.unbound-variable` | A probe templated a `{{var}}` that wasn't in context. |
| `scenario.factory.collision` | A factory generated a value that collided with an existing record. |

### 7.10 Industry-standard conformance

When a provider claims to follow a standard, we verify mechanically.

| Code | What it verifies |
|---|---|
| `standards.problem-details.shape` | RFC 7807 / 9457: error body has `type`, `title`, `status`, `detail`, `instance`. |
| `standards.problem-details.content-type` | `Content-Type: application/problem+json`. |
| `standards.json-api.resource-shape` | JSON:API: `data`, `data.type`, `data.id`, `data.attributes`. |
| `standards.json-api.error-shape` | JSON:API: errors are arrays of objects with `id`, `status`, `title`, `detail`. |
| `standards.json-api.content-type` | `Content-Type: application/vnd.api+json`. |
| `standards.hal.links` | HAL: `_links.self.href` present. |
| `standards.openapi.discovery.well-known` | `/.well-known/openapi.json` (or similar) present and valid. |
| `standards.health.shape-rfc-health-check` | `/healthz` follows the IETF Health Check Response Format. |
| `standards.cors.preflight-completeness` | OPTIONS responses include all required CORS headers per spec. |
| `standards.well-known.security-txt` | `/.well-known/security.txt` exists when claimed. |

Each standards check is opt-in (fires only when the suite declares the
standard via `suite.standards = [Standards.PROBLEM_DETAILS]` or per-probe
`probe.expect_standard(Standards.JSON_API)`). The list grows; each entry
cites its RFC / spec URL.

The catalogue grows. Adding a new check is a single class + a registration.

---

## 8. Replay subsystem — cassettes

A **cassette** is a deterministic, human-readable record of a suite run.
Replay mode reads the cassette instead of touching the network.

### 8.1 Why both live and replay

| | Live | Replay |
|---|---|---|
| **Detects new provider regressions** | ✅ | ❌ |
| **Hermetic CI** | ❌ | ✅ |
| **Reproduces a finding from yesterday** | ❌ | ✅ |
| **Protects credentials in CI** | needs secrets | ✅ |
| **Portable — runs offline / on a laptop** | ❌ | ✅ |
| **Catches drift vs cassette** | ❌ | ✅ |
| **Catches drift vs spec** | ✅ | ✅ |

Both have value. Typical flow: developer runs **live** locally with real
credentials, records a cassette, commits it; **CI runs replay** + a
scheduled live job catches provider drift.

### 8.2 Cassette format

YAML with optional out-of-line binary assets:

```yaml
version: 1
suite: example.com v1 contract
recorded_at: 2026-04-28T10:42:00Z
recorded_by: bassam@magogi.org
profile: staging
target:
  base_url: https://api.example.com/v1
match_policy: strict
redaction:
  headers: [authorization, cookie, x-api-key]
  body_jsonpaths: [$.client_secret]
signing:
  algorithm: ed25519
  signature: <base64>                     # optional, see §8.7
interactions:
  - probe_id: list-users
    request:
      method: GET
      path: /users
      headers:
        accept: application/json
        authorization: <REDACTED:bearer>
      query: {limit: "50"}
      body: null
      body_hash: null
    response:
      status: 200
      headers:
        content-type: application/json; charset=utf-8
      body_kind: json
      body: {items: [{id: "01...", email: "a@b.c"}], next_cursor: null}
      latency_ms: 124
  - probe_id: upload-photo                # binary out-of-line
    request:
      method: POST
      path: /uploads
      headers: {content-type: multipart/form-data; boundary=...}
      body_kind: multipart
      body_assets:
        - field: photo
          asset: assets/9c1f4d3a...e7.bin
          sha256: 9c1f4d3a...e7
          content_type: image/jpeg
    response: { ... }
```

Sensitive headers + JSONPaths redacted at record time. Bodies above
`max_inline_bytes` go to the assets directory.

### 8.3 Matching strategy

Three modes:

- **`exact`** — `(method, path, query, body_hash)` all match.
- **`strict`** (default) — `(method, path, query)` match; body shape hashed.
- **`lenient`** — `(method, path)` only.

Mismatches:

- **Probe with no cassette match** → `replay.miss`, suite continues.
- **Cassette entry never used** → `replay.unused`, informational.
- **Multiple probes match same entry** → `replay.ambiguous`, critical.
- **Probe matches but response shape differs** → `drift.cassette.*` per §7.7.

### 8.4 Lifecycle

- **Record**: `--record cassettes/<name>.yml`. Atomic write.
- **Replay**: `--replay` or auto-discover by suite name.
- **Refresh**: `apiverify cassette refresh <file>` re-runs live and
  rewrites preserving probe IDs.
- **Diff**: `apiverify cassette diff <a> <b>` shows what changed.
- **Migrate**: `apiverify cassette migrate <file>` updates format version.
- **Verify**: `apiverify cassette verify <file>` checks signature + integrity.

### 8.5 Storage backends

`FileCassette` (YAML/JSON, default) ships v0. `CassetteStore` ABC enables:
`PostgresCassette`, `S3Cassette`, `OciRegistryCassette`. Pluggable (§11).

### 8.6 Determinism guarantees

A replay run is **bit-identical** to its recording when:
- Suite definition unchanged.
- Active profile unchanged.
- All `Context` variables that influence requests are recorded.
- No probe uses `non_deterministic=True` templating.

Factory-generated values are deterministic per `run_id` (§3.4.2), so
factories don't break determinism.

### 8.7 Cassette signing & integrity

For security-sensitive use cases — cassettes shared in a registry,
cassettes used as the SOURCE OF TRUTH for compliance evidence — the
library supports cryptographic signatures.

Workflow:

```bash
# Generate a signing key (once per environment/role)
apiverify cassette keygen --out signing-key.pem

# Sign a cassette at record time
apiverify run suites/example.yml --record cassettes/example.yml \
    --sign-with signing-key.pem

# Verify signature on replay (refuses if tampered)
apiverify run suites/example.yml --replay cassettes/example.yml \
    --verify-with signing-pub.pem
```

Algorithm: Ed25519 (default; small, fast, modern). Signature over the
canonical JSON form of the cassette (excluding the signature field
itself). Public-key fingerprints listed in `.apiverify.toml` per profile;
unverified cassettes refuse to load when a profile requires verification.

Why this matters: in regulated contexts (compliance audits, security
attestations) cassettes serve as evidence. Without signing, anyone can
hand-edit a cassette to make a verification "pass."

---

## 9. Differential testing & drift detection

This is one of the library's main value propositions. Six shapes of
differential analysis ship in v1.0:

### 9.1 Report-vs-report diff

```bash
apiverify diff before.json after.json --by code,severity,root_cause
```

Output: new findings, resolved findings, severity changes, evidence
drift. Reports carry a `fingerprint`; the diff command refuses to
compare reports with different fingerprints unless `--force`.

### 9.2 Cassette-vs-cassette diff

```bash
apiverify cassette diff cassettes/maptiler-{2026-04-01,2026-04-28}.yml
```

Output: status changes, field-set changes, type changes, header changes,
latency shifts. Diff presented as findings; in CI a scheduled live job
records and diffs against the committed cassette.

### 9.3 Spec-vs-actual drift

```bash
apiverify drift --spec https://api.example.com/openapi.json \
    --replay cassettes/example.yml
```

Findings: `drift.spec.shape-mismatch`, `field-removed`, `field-added`,
`type-changed`, `status-undocumented`, `required-field-missing`.

**The highest-value feature for teams consuming third-party APIs.**
Provider docs lie or rot constantly; spec-vs-actual is the only
mechanical defence.

### 9.4 Spec-vs-spec diff (semantic versioning helper)

```bash
apiverify spec-diff old-openapi.json new-openapi.json
```

Categorises diffs as breaking / non-breaking / patch. A consumer team
predicts whether a provider's announced "v2.3.0" actually obeys semver,
before flipping the integration over.

### 9.5 Differential testing at runtime — A/B probes

```python
Probe.dual_target(
    primary="https://api.staging.example.com/v1/users",
    shadow="https://api.canary.example.com/v1/users",
).expect_status(200).expect_bodies_equal(
    ignore_jsonpaths=["$.timestamp", "$.request_id"],
)
```

Use cases: refactor verification, third-party-vs-self-hosted comparison,
CDN-edge-vs-origin verification.

### 9.6 Snapshot testing

Between "schema validates" and "drift vs cassette" sits a third option:
**snapshot the response shape** (field names, types, structure — not
values), and flag any change.

```python
probe.expect_snapshot("snapshots/list-users.json")
```

On first run with `--snapshot-update`: writes the snapshot. On
subsequent runs: compares. Output:

- `drift.snapshot.field-added` — a new field appeared.
- `drift.snapshot.field-removed` — a field disappeared.
- `drift.snapshot.type-changed` — a field's type changed (e.g.,
  string → number).
- `drift.snapshot.cardinality-changed` — array became scalar or vice
  versa.

Snapshots are stored as **JSON Schema drafts** (auto-derived from sample
responses), which means they're human-readable and editable. Snapshots
ignore values; they care about shape only. A snapshot is dead-simple
contract documentation — when no formal spec exists, your snapshots
are the spec.

Per-probe option `snapshot_strictness=` controls:
- `strict` — any added or removed field flags.
- `additive_only` — added fields are info; removed fields are warn.
- `value_aware` — also flag value-set changes for enum-typed fields.

Workflow:

```bash
# First run — record snapshots (just like cassettes)
apiverify run suites/example.yml --snapshot snapshots/example/ --update-snapshots

# Subsequent runs — compare
apiverify run suites/example.yml --snapshot snapshots/example/

# Refresh
apiverify snapshot refresh suites/example.yml
```

Snapshots vs cassettes:
- **Cassettes** preserve every byte: replay verifies *behaviour* hasn't
  changed.
- **Snapshots** preserve shape only: verify *structure* hasn't changed
  even when values do (timestamps, IDs, prices).

Most projects use both: cassettes for replay, snapshots for shape.

---

## 10. Mutation testing around a baseline

Property-based fuzzing (Schemathesis, Hypothesis) is a separate space and
deliberately out of scope (§2). What we DO ship is **named mutations
around a known-valid request** — much smaller surface, much higher
signal.

### 10.1 Concept

Given a probe with a known-valid request body, generate variants that
mutate one aspect at a time and observe how the API responds:

```python
Probe.post("/users")
    .body({"email": "valid@example.com", "name": "Alice"})
    .expect_status(201)
    .with_mutations([
        Mutation.drop_required("email"),               # expect: 422
        Mutation.wrong_type("name", value=12345),      # expect: 422
        Mutation.empty_string("email"),                # expect: 422
        Mutation.oversize_string("name", length=10000),# expect: 413 or 422
        Mutation.unicode_zalgo("name"),                # expect: 422 OR sanitized
        Mutation.sql_injection_canary("email"),        # expect: 422; no echo back
        Mutation.null_value("name"),                   # expect: 422
        Mutation.unexpected_field("admin", value=True),# expect: 422 (PM-WA-15 mirror)
    ])
```

### 10.2 Built-in mutations

| Mutation kind | Targeted bug | Default expectation |
|---|---|---|
| `drop_required` | Server forgot a required-field check | 4xx |
| `wrong_type` | Server doesn't validate types | 4xx |
| `empty_string` | Server treats `""` as missing | 4xx |
| `null_value` | Server panics on `null` | 4xx |
| `oversize_string` | Server has no size limit | 4xx (413/422) |
| `oversize_array` | Same, for lists | 4xx |
| `negative_number` | Server allows negative where positive expected | 4xx |
| `boundary_int` | Int overflow / negative pagination | 4xx |
| `unicode_zalgo` | Server doesn't normalise unicode | 4xx OR sanitised |
| `unexpected_field` | Server silently accepts unknown keys | 4xx |
| `sql_injection_canary` | Server reflects SQL fragments | no echo, no 5xx |
| `xss_canary` | Server reflects HTML/script payload | no echo, no 5xx |
| `path_traversal` | Server accepts `../` in path/query | 4xx |
| `header_case_flip` | HTTP/1.1-non-compliant on case | identical to baseline |
| `duplicate_header` | Server bug on duplicates | identical or 4xx |
| `wrong_content_type` | Server doesn't honor `Content-Type` | 415 |
| `crlf_injection` | Server reflects CRLF into headers | no header injection |
| `trailing_whitespace` | Server fails to strip when expected | identical to baseline |

Each mutation has a stable ID; users disable, add, or override. Power
users add domain-specific mutations.

### 10.3 Output

Findings: `mutation.input.silent-acceptance`,
`mutation.input.500-on-malformed`, `mutation.input.reflected-payload`,
`mutation.input.timing-leak`, `mutation.input.error-message-leaks-internal`.

### 10.4 Coexistence with Schemathesis

Mutation tests are deterministic, ~N requests per probe, finite-and-
auditable. Schemathesis runs in a separate CI job (slower, broader). Both
produce findings the report aggregator can ingest.

---

## 11. Plugin architecture

Eight extension points; each is an ABC + a Python entry-point group:

| Extension point | ABC | Entry-point group |
|---|---|---|
| **Check** | `Check` | `mgf_apiverify.checks` |
| **AuthFlow** | `AuthFlow` | `mgf_apiverify.auth_flows` |
| **SchemaValidator** | `SchemaValidator` | `mgf_apiverify.schema_sources` |
| **CassetteStore** | `CassetteStore` | `mgf_apiverify.cassette_stores` |
| **HttpTransport** | `HttpTransport` | `mgf_apiverify.transports` |
| **ReportWriter** | `ReportWriter` | `mgf_apiverify.report_writers` |
| **Mutation** | `Mutation` | `mgf_apiverify.mutations` |
| **TemplateFilter** | `TemplateFilter` | `mgf_apiverify.template_filters` |
| **Matcher** | `Matcher` | `mgf_apiverify.matchers` |
| **Factory** | `Factory` | `mgf_apiverify.factories` |

Plugin discovery via `importlib.metadata.entry_points`. Plugin packages
declare:

```toml
[project.entry-points."mgf_apiverify.checks"]
"plasma_specific.fleet_geofence" = "plasma_apiverify_plugin:GeofenceCheck"

[project.entry-points."mgf_apiverify.auth_flows"]
"aws_sigv4" = "mgf_apiverify_aws:AwsSigV4"

[project.entry-points."mgf_apiverify.matchers"]
"is_teltonika_imei" = "plasma_apiverify_plugin:IsTeltonikaImei"
```

Stability tiers (mirrors `mgf_common/docs/standards/API_DESIGN.md`):
`stable`, `experimental` (warns on import), `deprecated`. Documented in
`PUBLIC_API.md` (lands v1.0).

**The default check set, default auth flows, default cassette store,
default matchers, and default factories are all registered through the
SAME entry-point mechanism** — they are first-party plugins. This
validates the architecture and prevents "works for first-party, breaks
for third-party" drift.

ABC compatibility versioning: each ABC declares a `MAJOR_VERSION`;
plugins declare what they support; mismatches refuse to load with a
clear error.

---

## 12. Profiles & environments

Real teams run the same suite against multiple environments. Profiles let
one suite definition target many.

### 12.1 Profile shape

```toml
# .apiverify.toml
[profiles.local]
base_url = "http://localhost:8000"
credentials.client_id = "dev-client"
credentials.client_secret_env = "DEV_CLIENT_SECRET"
concurrency = 4
mode = "live"

[profiles.staging]
base_url = "https://api.staging.example.com/v1"
credentials.client_id_env = "STAGING_CLIENT_ID"
credentials.client_secret_env = "STAGING_CLIENT_SECRET"
mode = "live"
exclude_tags = ["paid-tier"]

[profiles.ci]
extends = "staging"
mode = "replay"
cassette_dir = "cassettes/"
exclude_tags = ["paid-tier", "destructive"]

[profiles.prod-readonly]
base_url = "https://api.example.com/v1"
mode = "live"
include_tags = ["readonly"]
severity_overrides."quality.latency.over-budget" = "warn"
require_signed_cassettes = true

[profiles.ci.report_writers]
slack.webhook_env = "SLACK_WEBHOOK_URL"
slack.severity_floor = "warn"
github_pr_comment.token_env = "FORGE_TOKEN"
```

### 12.2 Resolution order

1. CLI flag (`--base-url`, `--bearer`).
2. Environment variable (`APIVERIFY_BASE_URL`).
3. Active profile section.
4. `[profiles.default]` section if present.
5. Built-in default.

Mirrors `mgf_common`'s CF-02.

### 12.3 Secrets

Profiles never store secrets verbatim:
- `*_env` — env var.
- `*_file` — file path.
- `*_vault` — vault provider (mgf_common.vault when available; pluggable).

Per `mgf_common/docs/standards/SECURITY.md` SC-01, SC-13.

### 12.4 Profile inheritance

Profiles `extends = "<base>"` to inherit and override. Encourages DRY config.

---

## 13. Observability of the verifier itself

The verifier needs to be observable too — both because it dogfoods the
discipline it preaches AND because operators investigating "why did this
suite fail in CI?" need decent telemetry.

### 13.1 Logging

- Stdlib `logging` with JSON formatter; one logger per package.
- Per `mgf_common` LG-* rules: lazy formatting, structured `extra={}`,
  redaction of secrets, OpenTelemetry trace correlation when active.
- Default level INFO; suite-run lines include suite, profile, mode,
  finding counts.
- DEBUG level enables per-probe request/response dumps (after redaction).

### 13.2 OpenTelemetry

When `OTEL_EXPORTER_OTLP_ENDPOINT` is set:
- One root span per `Suite.run()`.
- Child span per `Scenario`, then per `Probe`, then per `Check`.
- Span attributes: `apiverify.probe_id`, `apiverify.check_code`,
  `apiverify.severity_max`, `apiverify.root_cause`, `http.method`,
  `http.status_code`, `apiverify.profile`, `apiverify.mode`.
- Findings emit OTel events on their probe's span.

Console fallback exporter activates when no OTLP endpoint is configured.

### 13.3 Metrics

Optional Prometheus-format metrics endpoint when running long-lived
(e.g., `apiverify watch suites/`):

- `apiverify_runs_total{suite,profile,mode}`
- `apiverify_findings_total{suite,code,severity,root_cause}`
- `apiverify_run_duration_seconds{suite,profile}` histogram
- `apiverify_probe_latency_seconds{suite,probe_id}` histogram

For one-shot CLI invocations, metrics emit as a final structured log
record.

### 13.4 Self-checks

`apiverify doctor` runs:
- Plugins discovered + loadable.
- Cassette store reachable.
- HTTP transport works (against a known-good endpoint).
- Schema parsers work for each ingested format.
- Signing keys valid + accessible per profile.

### 13.5 Observability budget

The verifier MUST not contaminate its own measurements. Constraints:
- OTel instrumentation overhead per probe: ≤ 1 ms p99.
- Logging per probe: ≤ 100 µs (rule LG-12).
- Latency reported in findings excludes the verifier's own time —
  measured at the `httpx`-transport boundary, not the user-code boundary.

---

## 14. Project layout

```
mgf_apiverify/
├── DESIGN.md                       ← you are here
├── README.md                       ← user-facing introduction
├── PUBLIC_API.md                   ← v1.0+; stability tier per public name
├── pyproject.toml
├── uv.lock
├── src/mgf/apiverify/              ← namespace package, sibling to mgf.common
│   ├── __init__.py                 ← public surface, __all__
│   ├── py.typed
│   ├── _version.py
│   ├── core/
│   │   ├── probe.py                ← Probe + ProbeBuilder + LRO
│   │   ├── scenario.py             ← Scenario + setup/teardown
│   │   ├── context.py              ← Context, Variable, templating
│   │   ├── factories.py            ← Built-in factories + Factory ABC
│   │   ├── auth.py                 ← AuthFlow ABC + builtins
│   │   ├── tag.py
│   │   ├── matcher.py              ← Matcher ABC + DSL primitives
│   │   ├── matchers/
│   │   │   ├── primitives.py       ← equals, gt, in_, matches, …
│   │   │   ├── format.py           ← is_uuid, is_iso8601_z, …
│   │   │   └── semantic.py         ← is_iban, is_imei, is_e164, …
│   │   ├── response.py             ← typed Response wrapper
│   │   ├── finding.py              ← Finding, Severity, RootCause
│   │   ├── report.py               ← Report aggregation
│   │   ├── suite.py                ← Suite + execution engine (async)
│   │   ├── template.py             ← ProbeTemplate
│   │   └── ratelimit.py            ← client-side rate limiter
│   ├── checks/
│   │   ├── base.py
│   │   ├── schema.py               ← §7.1
│   │   ├── contract.py             ← §7.2
│   │   ├── stability.py            ← §7.3 (incl. concurrency, LRO)
│   │   ├── quality.py              ← §7.4
│   │   ├── security.py             ← §7.5 (incl. rate-limit verify)
│   │   ├── hygiene.py              ← §7.6
│   │   ├── drift.py                ← §7.7
│   │   ├── mutation.py             ← §7.8
│   │   ├── scenario.py             ← §7.9
│   │   ├── standards.py            ← §7.10 (RFC 7807/9457, JSON:API, …)
│   │   └── default.py              ← DEFAULT_CHECKS aggregate
│   ├── schema/
│   │   ├── base.py
│   │   ├── pydantic.py             ← v0
│   │   ├── openapi.py              ← Phase 1.5
│   │   └── jsonschema.py           ← Phase 2
│   ├── replay/
│   │   ├── cassette.py
│   │   ├── store.py
│   │   ├── file_store.py
│   │   ├── matcher.py
│   │   ├── redactor.py
│   │   ├── migrate.py
│   │   ├── signing.py              ← §8.7 Ed25519 sign + verify
│   │   └── assets.py               ← out-of-line binary asset handling
│   ├── snapshot/
│   │   ├── snapshot.py             ← §9.6 Snapshot + diff
│   │   └── derivation.py           ← infer JSON Schema from samples
│   ├── drift/
│   │   ├── report_diff.py          ← §9.1
│   │   ├── cassette_diff.py        ← §9.2
│   │   ├── spec_drift.py           ← §9.3
│   │   ├── spec_diff.py            ← §9.4
│   │   └── dual_target.py          ← §9.5
│   ├── mutation/
│   │   ├── base.py
│   │   ├── builtins.py             ← §10.2
│   │   └── runner.py
│   ├── plugins/
│   │   ├── registry.py
│   │   └── stability.py
│   ├── profile/
│   │   ├── loader.py
│   │   └── secrets.py
│   ├── observability/
│   │   ├── logging_setup.py
│   │   ├── otel_setup.py
│   │   └── metrics.py
│   ├── transport/
│   │   ├── base.py
│   │   ├── httpx_transport.py
│   │   └── streamed.py             ← chunked / streaming bodies
│   ├── cli/
│   │   ├── __main__.py
│   │   ├── run.py
│   │   ├── probe.py
│   │   ├── diff.py
│   │   ├── drift.py
│   │   ├── cassette.py
│   │   ├── snapshot.py
│   │   ├── mutate.py
│   │   ├── scaffold.py             ← OpenAPI → starter suite
│   │   ├── checks.py
│   │   ├── doctor.py
│   │   ├── bisect.py               ← §4.4.4
│   │   └── report_subcmd.py        ← report comment-pr, report post-slack, …
│   ├── pytest_plugin/
│   │   ├── __init__.py
│   │   ├── fixtures.py
│   │   ├── generation.py
│   │   └── reporting.py
│   ├── reporting/
│   │   ├── base.py                 ← ReportWriter ABC
│   │   ├── human.py
│   │   ├── json_writer.py
│   │   ├── junit.py
│   │   ├── markdown.py
│   │   ├── sarif.py
│   │   ├── slack.py
│   │   ├── discord.py
│   │   ├── teams.py
│   │   ├── pagerduty.py
│   │   ├── github_pr.py
│   │   ├── gitlab_mr.py
│   │   ├── codeberg_pr.py
│   │   ├── csv_writer.py
│   │   └── prometheus.py
│   ├── exceptions.py               ← ApiVerifyError(Exception) hierarchy
│   └── _sync.py                    ← SyncSuite wrapper
├── tests/
│   ├── unit/                       ← mirror src/ 1:1 (rule TS-01)
│   ├── integration/                ← real httpbin / OpenAPI / recorded targets
│   └── conftest.py
├── cassettes/                      ← committed example cassettes for self-tests
│   └── httpbin/
├── snapshots/                      ← committed example snapshots
└── examples/
    ├── maptiler/
    ├── plasmamapper/
    ├── teltonika/
    ├── openapi/
    ├── auth_flows/
    ├── lro/                        ← long-running operations example
    ├── multipart/                  ← file-upload example
    ├── mutations/
    └── ci/                         ← .woodpecker.yml + .github/ examples
```

---

## 15. Standards conformance

Plan: **L1 — Production-Ready** per `mgf_common/docs/standards/`. Bumps
to **L2 — Observability-Native** when v1.0 ships.

What that means concretely:

- **DP-** (design): strict layering, frozen domain types, mypy strict,
  every external dep behind an ABC, no silent fallbacks.
- **EH-** (errors): typed `ApiVerifyError(Exception)` hierarchy; retries
  with exponential backoff + jitter for live HTTP; circuit breakers per
  host so a flaky provider doesn't tank the suite.
- **LG-** (logging): JSON formatter, correlation IDs, redaction.
- **CF-** (config): pydantic-settings; layered profile + env precedence.
- **TS-** (testing): mirror src 1:1; ≥ 85% coverage (one notch above the
  standard 80% — this library IS testing infrastructure); property tests
  on the matcher, redactor, signing, and diff engines.
- **SC-** (security): no secrets in logs / cassettes / errors; redaction
  on by default; TLS verified; subprocess never used; cassette signing.
- **AP-** (API design): `__all__` discipline; semver; `PUBLIC_API.md`
  from v1.0; deprecation cycle; experimental tier marked.
- **SP-** (scope): the §2 cuts ARE the scope policy.

`mgf_apiverify` adopts the standards but does NOT depend on `mgf_common`
at runtime. The library exists partly to test APIs in projects that don't
use `mgf_common`. If `mgf_common` is installed, optional integrations
light up (vault provider, OTel auto-config); otherwise the library uses
its own minimal equivalents.

---

## 16. Phased roadmap

| Phase | Scope | Exit criterion |
|---|---|---|
| **0** | THIS DOCUMENT | Sign-off; design locked enough to start coding. |
| **1** | Core engine: `Probe`, `Check`, `Suite`, `Context`, `AuthFlow` (bearer + basic + api-key + cookie + custom), Pydantic schema source, matchers DSL primitives + format matchers, ~18 checks (schema + contract + initial security/quality), built-in factories. | Hand-written suite against `httpbin.org` runs live, produces structured report with multiple findings, exits non-zero on planted findings. Coverage ≥ 85%. |
| **1.5** | OpenAPI schema source. `Scenario` + setup/teardown. `wait_for` + LRO patterns. Plugin registry wired (entry-points). Auto-suite scaffolding from OpenAPI. Probe templates. Suite composition. Faker factory plugin. Multipart + streaming bodies. Semantic matchers (IBAN, IMEI, etc.). | A multi-step CRUD scenario verifies a real API end-to-end. A suite auto-scaffolded from PlasmaMapper's `/openapi.json` runs. Plugin discovery works for first- and third-party plugins. |
| **2** | Replay (record + replay + matcher + redactor + asset handling). Cassette store ABC + file backend. Cassette diff. Cassette signing. | A live run records cassettes (incl. binary assets); replay produces an identical report; cassette diff catches a planted change; signature verification refuses tampered cassettes. |
| **2.5** | Drift detection (report diff, spec drift, spec-vs-spec, dual-target). Snapshot testing. | A scheduled job catches a deliberately-introduced provider response field-removal via spec drift. Snapshot test catches a shape change. |
| **3** | CLI (`run`, `probe`, `diff`, `cassette`, `snapshot`, `drift`, `scaffold`, `bisect`, `checks`, `doctor`) + YAML suite format. Profile system (`.apiverify.toml`). Industry-standard conformance checks (§7.10). | Manual operator runs a YAML suite end-to-end without writing Python. RFC 7807/9457 conformance verified. |
| **3.5** | Mutation testing (`Mutation` ABC, ~18 built-in mutations, `apiverify mutate run`). Rate-limit verification. Concurrency-safety probes. Determinism check. | A mutation pass against a real API surfaces at least one silent-acceptance or 500-on-malformed finding. Rate-limit-actually-enforced check verified against a known rate-limited endpoint. |
| **4** | pytest plugin + JUnit + SARIF + markdown writers. CI integration patterns: PR-comment writers (Codeberg / GitHub / GitLab), Slack / Discord / Teams / PagerDuty writers. OAuth2 flows (client-credentials, auth-code, device-code). | A PlasmaMapper PR uses `mgf_apiverify` for its API contract tests; CI shows individual findings as test cases; SARIF feeds GitHub Advanced Security; PR comments post automatically. |
| **4.5** | Observability: OTel spans + metrics; `apiverify watch` mode for long-lived runs. Failure attribution dashboards. | Every Suite.run() produces a complete OTel trace; metrics endpoint emits Prometheus on demand; `root_cause_breakdown` available in every report. |
| **5** | GUI (likely Tauri + SvelteKit, mirroring PlasmaMapper). | Operator workflow: pick a suite, run, browse findings interactively, schedule recurring runs. **Out of v1.0 scope.** |
| **6** | Postgres/S3/OCI cassette stores; OpenAPI 3.1 full coverage; AWS SigV4 + Google JWT auth flows; rich domain-specific matchers (Teltonika, MapTiler-tile-URL). | Team-shared cassettes; arbitrary OpenAPI spec ingestion; cloud-provider auth without third-party SDKs. **Post-v1.0.** |

**v1.0 = end of Phase 4.5.** Library + CLI + pytest + replay + drift +
mutation + observability + CI integration. GUI is post-v1.0.

Each phase has its own retrospective doc when it ships.

---

## 17. Open questions to lock before Phase 1

1. **Exact name + repo location.** Default: `mgf_apiverify` under
   `magogi-admin/` (sibling to `mgf_common`).

2. **Public eventually?** Default: private until v1.0; re-evaluate when
   stable. PR-friendly licensing (Apache-2.0) when public.

3. **OpenAPI versions.** Default: 3.0 + 3.1 in Phase 1.5; Swagger 2.0
   only on demand.

4. **YAML suite format DSL.** Default: YAML for declarative simple suites;
   Python files allowed for advanced ones (CLI accepts either via
   extension).

5. **Cassette redaction defaults.** Default: aggressive (every
   common-secret header + every JSONPath matching `*token*`,
   `*secret*`, `*password*`); opt-in conservative via profile.

6. **Coverage of the library itself.** Plan: a CI suite hits
   `httpbin.org` live AND replays its own committed cassettes. Both
   must be green.

7. **Findings deduplication.** Default: group in human report, keep
   individual records in JSON. Configurable threshold.

8. **Plugin compatibility versioning.** Default: each ABC declares
   `MAJOR_VERSION`; plugins declare what they support; mismatches
   refuse to load.

9. **Default mutation set.** Default: only the "safe and high-signal"
   set runs by default (drop_required, wrong_type, empty_string,
   unexpected_field); the rest opt-in to avoid burning provider
   quotas in CI.

10. **Snapshot strictness default.** Default: `additive_only` (added
    fields are info, removed fields are warn, type changes are
    critical). Strict mode opt-in per probe.

11. **Cassette signing default.** Default: signing OFF; profiles can
    require signed cassettes via `require_signed_cassettes = true`.

These unblock Phase 1 once they have committed answers.

---

## 18. What this document is NOT

- **Not the API reference.** Generates from the code in v0.5+ (Sphinx /
  mkdocs).
- **Not a contract with consumers.** v0.x is allowed to break itself.
  v1.0+ honors AP-* deprecation cycle.
- **Not a marketing document.** No claims about industry leadership;
  just a design.
- **Not finished.** Update this file when a section gets answered or a
  new question emerges. Date the change in the version-history table.

---

## 19. Sign-off

| Date | Reviewer | Decision |
|---|---|---|
| 2026-04-28 | Claude (drafted v0.1) | initial draft for review |
| 2026-04-28 | Claude (drafted v0.2) | scenarios, drift, mutation, plugins, profiles, observability |
| 2026-04-28 | Claude (drafted v0.3) | LRO, multipart/streaming, matchers DSL, factories, attribution, snapshots, signing, standards conformance, rate-limit verify, concurrency, CI integration writers, suite composition |
| | Bassam | (pending) |

When Bassam approves the doc (or proposes edits we converge on), the
next session starts with `git init` + scaffolding Phase 1.

---

*End of `DESIGN.md`. Cross-references: `mgf_common/docs/standards/` for
the engineering bar this library inherits; `PlasmaMapper/docs/standards/WEB_API.md`
for the API conventions our own services follow.*
