# Handoff — `mgf-apiprobe` (sibling package, depends on `mgf-common`)

> **Audience:** the AI agent (and human contributors) maintaining
> `mgf-apiprobe` at `/home/bassam/PycharmProjects/mgf-apiprobe`.
> **Author:** Bassam (decision); Claude (drafted).
> **Status:** active handoff — the mgf-apiprobe agent OWNS this from receipt.

---

## TL;DR

`mgf-apiprobe` is a generic REST API verification library. It lives as a
**sibling package** to `mgf-common`, in the `mgf.*` namespace, and depends
on `mgf-common` at runtime.

The package's primary primitive is the `Probe` (see `DESIGN.md`).

**The spec is `DESIGN.md`. This handoff is the *integration plan*.** When
the two disagree, this document overrides `DESIGN.md`'s assumptions
about packaging.

---

## 1. Layout

| File | Purpose |
|---|---|
| `DESIGN.md` (≈ 1950 lines) | The design specification: primitives, ~100 checks across 10 categories, replay/cassette subsystem, differential testing, mutation testing, plugin architecture. **This is the spec; read it before writing code.** |
| `HANDOFF.md` | This file — the integration plan. |

---

## 2. Why sibling, not subpackage of `mgf-common`

apiprobe could in principle live inside `mgf-common` as
`mgf.common.apiverify`, but five concerns make a sibling package the
right shape:

1. **Surface explosion.** DESIGN.md ships 8 primitives, ~100 public
   check codes (each a public contract), 10 entry-point groups, the
   cassette format, the pytest plugin, and a second console-script.
   Absorbing this would roughly double `mgf-common`'s public surface.

2. **Closed-box invariant.** `mgf-common`'s load-bearing principle is
   that consumers don't reach past it. apiprobe's plugin architecture
   explicitly *opens* the surface (10 entry-point groups). That
   tension is incompatible with what `mgf-common` is for.

3. **Versioning.** Coupling apiprobe's release cadence to `mgf-common`
   would drag every consumer pinning `mgf-common` into apiprobe's churn.

4. **Standards conformance.** Landing apiprobe's surface (~1500 lines
   of design + ~100 checks + plugin architecture) into `mgf-common`'s
   L2 gates day-one (mypy --strict, ruff, lint-imports, ≥85% coverage,
   AP-* doc sync, AST-pinned public surface) is multi-quarter work. As
   a sibling with its own release cadence, apiprobe can land L1 first
   and graduate to L2 on its own schedule.

5. **SCOPE.md collision.** `mgf-common`'s SCOPE.md §2 lists "Testing
   framework" as OUT and "Plugin loading / discovery framework" as OUT.
   The sibling shape sidesteps both — apiprobe is the federation member
   that carries this concern; `mgf-common` does not.

6. **Federation precedent.** `mgf-common` ships `LIFT_CHECKLIST.md`,
   the `mgf-common-is-the-federation-leaf` import-linter contract, and
   the `mgf.*` namespace specifically so siblings (`mgf-qt`, `mgf-vmanager`,
   …) can grow with their own velocity while sharing `mgf-common`.
   **`mgf-apiprobe` is the first test case for that shape.** Absorbing
   it back would invalidate the federation design.

The "single dependency" rationale is satisfied without absorption: pip
resolves transitive deps, and `mgf-common explain federation` (future)
will list siblings for discoverability.

---

## 3. Architecture decisions — locked

### 3.1 Distribution name + import path

| Surface | Value |
|---|---|
| PyPI distribution | `mgf-apiprobe` |
| Import path | `mgf.apiprobe` |
| Console script | `mgf-apiprobe` (canonical), `apiprobe` (short alias — register both, mirroring the `mgf-common` / `register_cli_subcommand` + `subcommand` short-alias pattern) |
| Entry-point groups | `mgf_apiprobe.<group>` (underscore form is mandatory; entry-point group names cannot contain hyphens) |
| Namespace shape | PEP 420 namespace package under `mgf.*`, matching `mgf-common`'s layout |

### 3.2 Runtime dependency on `mgf-common`

apiprobe **depends on** `mgf-common` for these surfaces:

| Surface | Approach |
|---|---|
| Exception hierarchy | Subclass `mgf.common.exceptions.AppError`. The root is `mgf.apiprobe.exceptions.ApiprobeError`. Categorise concretes per `mgf-standard/docs/standards/ERROR_HANDLING.md`. |
| Build own logging setup | Use `mgf.common.observability.logging_setup` directly. Obey LG-* rules. |
| Build own OTel wiring | Use `mgf.common.observability.otel_setup`. apiprobe adds its own span attributes on top. |
| Build own redaction | Use `mgf.common.observability.redaction`. Add domain-specific `extra_keys` (auth headers, cassette body fields). |
| Build own settings layering | Subclass `mgf.common.config.BaseAppSettings` with `env_prefix="APIPROBE_"`. |
| Build own vault adapter | Use `mgf.common.vault` directly. Cassette signing keys live there per SC-13. |
| Build own crash reporter | Use `mgf.common.observability.crash_reporter`. Set `app_name="mgf-apiprobe"`. |
| Build own CLI top-level | Use `mgf.common.cli.run_and_translate` / `main_entry` for exit-code translation. apiprobe's CLI subcommands register via `register_cli_subcommand` if we want them available under `mgf-common doctor` too — TBD; default standalone. |
| Define own `RootCause` enum (DESIGN.md §3.2.1) | KEEP — domain-specific to apiprobe. |
| Define own `Finding` shape | KEEP — domain-specific. |

Net effect: ~30% smaller library, no parallel implementation of
infrastructure `mgf-common` already ships.

### 3.3 Standards conformance — L1 day one, L2 graduation later

`mgf-common` is L2. apiprobe targets L1 from day one (every MUST in
`mgf-standard/docs/standards/` that applies to a sibling), and graduates
to L2 as the sibling matures. Concretely:

- L1 day-one: `BaseAppSettings` subclass, `AppError` hierarchy +
  translation seams, structured-logging + redaction, `__all__`, py.typed,
  semver, mirrored test layout, ≥80% coverage (TS-* baseline), no
  secrets in cassettes / logs / errors (SC-*).
- L2 graduation: ≥85% coverage, property tests on matchers / redactors /
  signing / diff, `PUBLIC_API.md` with AP-10 sync test, OTel correlation
  on every probe + suite + check span, AST-pinned public surface (the
  `tests/public_surface/` shape `mgf-common` uses).

The CI gates apiprobe runs: `ruff`, `mypy --strict`, `import-linter`,
`coverage run -m pytest && coverage report --fail-under=80`, then bump
the coverage threshold when L2 graduation is ready. **Use `coverage
run -m pytest` not `pytest --cov`** — `mgf-common` proved via incident
that pytest-cov under-reports because pytest's collection imports run
before pytest-cov starts collection.

### 3.4 Plugin discovery — stdlib `importlib.metadata.entry_points`

Plugin discovery uses stdlib entry points; no custom loader. Each
entry-point group is namespaced `mgf_apiprobe.<group>` (underscore
form). The plugin discovery code lives at `src/mgf/apiprobe/plugins/`
— apiprobe-local, not a federation plugin host.

### 3.5 Federation contract

apiprobe MUST register itself as a federation sibling:

- Top-level `LIFT_CHECKLIST.md` (mirrors `mgf-common`'s) — confirms the
  sibling has read the federation rules and committed to them.
- `pyproject.toml` declares `mgf-common ~= 0.5` (or whatever the current
  minor is at first publish) under `[project.dependencies]`.
- An import-linter contract `mgf-apiprobe-only-imports-from-mgf-common`
  forbids imports from any other federation sibling. apiprobe is a
  federation member, NOT a federation hub.

---

## 4. Architecture decisions — open for the agent

These are reasonable to lock during integration; defaults given.

| Decision | Default | Why this default |
|---|---|---|
| Public eventually? | Private until v1.0. | Surface is still maturing; private avoids breaking-change cycles in public APIs. |
| License when public | Apache-2.0 | Matches `mgf-common`. |
| Pre-1.0 stability tier of public symbols | `experimental` | Standard pre-1.0 posture. After v1.0, follow the AP-* deprecation cycle. |
| Default profile shipped | `httpbin` | Useful for `apiprobe doctor` self-test + first-run smoke. Cheap to ship. |
| pytest async mode | Honour the consumer's config; default `auto`. | Same posture as `mgf-common`'s pytest plugin work. |
| Whether `mgf-common` itself uses apiprobe in its CI | Yes, eventually — once apiprobe is stable. Adds a CI job that probes httpbin live + replay every merge. | Eat the dog food; catches contract regressions in the live HTTP shape we depend on. |

When the agent locks any of these during integration, append the
decision IN this file (§4 → "Decided during integration") with date.

---

## 5. SCOPE policy

**No SCOPE.md edit needed in `mgf-common`.** The previous handoff
required updating `mgf-common`'s SCOPE.md to add API verification
infrastructure to §1 IN scope. Sibling shape removes that need — the
"Testing framework: pytest is the ecosystem default" entry stays clean,
"Plugin loading / discovery framework" stays OUT, and the federation
roadmap absorbs the work cleanly.

apiprobe MAY ship its own `docs/standards/SCOPE.md` for its own
sub-surface (which 100 checks ARE in scope, which checks are OUT) — but
that's apiprobe's policy, not `mgf-common`'s.

---

## 6. Scope

The shipped surface covers the load-bearing core:

| Shipped | Why |
|---|---|
| Core primitives: `Probe`, `Check`, `Suite`, `Context`, `AuthFlow` | The 5 primitives required for any non-trivial probe |
| Pydantic schema source | The first schema-source backend; OpenAPI is a future add |
| ~18 checks (schema, contract, initial security, initial quality) | A demonstrably useful catalogue without the long tail |
| Built-in factories (UUID, ISO timestamps, IBAN, IMEI, E.164) | Needed for any realistic probe |
| Bearer + basic + api-key + cookie + custom AuthFlow | The five most common auth flows |
| Console + JSON report writers | Two writers cover 80% of consumer needs |
| Async core; sync wrapper via `_sync.py` | Per DESIGN.md §6 |
| `apiprobe run`, `apiprobe checks`, `apiprobe doctor` | Three subcommands cover the canonical workflow |
| `httpbin.org` self-test | Live integration smoke |

Everything else — replay/cassette subsystem, signing, OpenAPI ingestion,
Scenario, drift, mutation, full plugin architecture, full check
catalogue, all the report writers (markdown, JUnit, SARIF, Slack,
Discord, Teams, PagerDuty, GitHub PR, GitLab MR, Codeberg PR, Prometheus),
GUI, Postgres/S3/OCI cassette stores — **deferred** (see DESIGN.md
for design intent).

---

## 7. Concrete integration tasks (do these in order)

### Step 0 — Receipt + acknowledgment

1. Read `DESIGN.md` end to end (≈ 1950 lines).
2. Read this `HANDOFF.md`.

### Step 1 — Project skeleton

```
mgf_apiprobe/
├── DESIGN.md                       ← spec
├── HANDOFF.md                      ← this file
├── LIFT_CHECKLIST.md               ← federation conformance ack
├── README.md                       ← consumer-facing
├── pyproject.toml                  ← hatchling, pytest, ruff, mypy strict, import-linter
├── .gitignore
├── .python-version                 ← 3.11+
├── docs/
│   ├── standards/
│   │   └── SCOPE.md                ← apiprobe's own scope policy
│   └── public/
│       └── (per-component docs, mirroring mgf-common's docs/public/ shape)
├── src/mgf/apiprobe/
│   ├── __init__.py
│   ├── py.typed
│   ├── _version.py
│   ├── exceptions.py               ← ApiprobeError(AppError) + concretes
│   ├── config.py                   ← BaseAppSettings subclass
│   ├── core/
│   │   ├── probe.py
│   │   ├── check.py
│   │   ├── suite.py
│   │   ├── context.py
│   │   ├── auth.py
│   │   ├── matcher.py
│   │   ├── finding.py
│   │   └── response.py
│   ├── checks/
│   │   ├── schema.py
│   │   ├── contract.py
│   │   ├── security.py             ← initial subset
│   │   └── quality.py              ← initial subset
│   ├── schema/
│   │   └── pydantic.py
│   ├── matchers/
│   │   ├── primitives.py
│   │   ├── format.py
│   │   └── semantic.py             ← IBAN, IMEI, E.164
│   ├── reporting/
│   │   ├── base.py
│   │   ├── human.py
│   │   └── json_writer.py
│   ├── transport/
│   │   ├── base.py
│   │   └── httpx_transport.py
│   ├── cli/
│   │   ├── __main__.py
│   │   ├── run.py
│   │   ├── checks.py
│   │   └── doctor.py
│   └── _sync.py
└── tests/
    ├── unit/
    ├── integration/                ← httpbin
    └── conftest.py
```

### Step 2 — `pyproject.toml`

```toml
[project]
name = "mgf-apiprobe"
requires-python = ">=3.11"
dependencies = [
    "mgf-common >= 0.12, < 0.13",
    "httpx >= 0.27",
    "pydantic >= 2.7",
    "jsonpath-ng >= 1.6",
    "jmespath >= 1.0",
]

[project.optional-dependencies]
faker = ["faker >= 25"]                 # declared early so consumers can preview
openapi = ["openapi-core >= 0.19"]
signing = ["cryptography >= 42"]
dev = ["pytest", "pytest-asyncio", "ruff", "mypy", "import-linter", "coverage"]

[project.scripts]
mgf-apiprobe = "mgf.apiprobe.cli.__main__:main"
apiprobe = "mgf.apiprobe.cli.__main__:main"     # short alias
```

### Step 3 — Standards conformance scaffolding

- `LIFT_CHECKLIST.md` — copy `mgf-common`'s shape and tick every box
  apiprobe commits to.
- `docs/standards/SCOPE.md` — apiprobe's own scope policy.
- `import-linter` contracts (in `pyproject.toml`):
  - `mgf-apiprobe-only-imports-from-mgf-common` — apiprobe MUST NOT
    import from any other `mgf.*` sibling.
  - `mgf-apiprobe-layering` — `core/` does not import from `cli/`,
    `reporting/`, `transport/` (mirrors `mgf-common`'s layering posture).

### Step 4 — Implementation

Follow `DESIGN.md` for the design intent. Concrete order:

1. Skeleton: `__init__.py`, `py.typed`, `_version.py`,
   `exceptions.py` (subclass AppError), `config.py` (subclass
   BaseAppSettings), `core/finding.py`, `core/response.py`.
2. Probe + Tag + Matcher DSL primitives.
3. Pydantic schema validator + Pydantic-based response shape checks.
4. Default checks subset (~18 checks).
5. AuthFlow primitives (bearer, basic, api-key, cookie, custom).
6. Suite + execution engine (async).
7. Built-in factories (UUID, ISO timestamps, IBAN, IMEI, E.164).
8. Console + JSON report writers.
9. CLI core: `apiprobe run`, `apiprobe checks`, `apiprobe doctor`.
10. Tests against `httpbin.org` (live).

Exit criterion: a hand-written suite against httpbin runs live,
exits non-zero on planted findings, coverage ≥ 80%.

### Step 5 — Release

- `coverage run -m pytest && coverage report --fail-under=80` green
- `ruff check . && mypy --strict src/`
- `lint-imports` clean
- `pyproject.toml` validates; build + twine check pass
- Manual `pip install dist/mgf_apiprobe-*-*.whl` smoke test
- Tag, push to PyPI manually (Codeberg has no OIDC).

---

## 8. What the agent decides vs. what is locked

### Locked by `DESIGN.md` (do not re-litigate)

- The 8 primitives.
- The check catalogue codes (each `code` is a public contract; any
  ship-time check enters that contract).
- The cassette format v1 (when replay lands).
- The findings shape (code, severity, root_cause, message, evidence,
  remediation, threat_refs, tags).

### Locked by THIS handoff (do not re-decide)

- Distribution name `mgf-apiprobe`, import `mgf.apiprobe`.
- Sibling, depends on `mgf-common`.
- Use `mgf.common.*` primitives directly per §3.2.
- L1 day-one, L2 graduation later.
- Plugin discovery uses stdlib `entry_points` namespaced `mgf_apiprobe.<group>`.

### Open for the agent (defaults given in §4)

See §4 above.

---

## 9. Foot-guns

1. **Cassette signing key management** — when replay/signing lands,
   keys live under `mgf.common.vault` per SC-13. Don't invent a
   parallel key store.
2. **Redaction defaults** — confirm aggressive default aligns with
   `mgf-common`'s redaction defaults at `mgf.common.observability.redaction`.
   Override if needed and document why.
3. **Pydantic v2 baseline** — `mgf-common` requires pydantic ≥ 2.6;
   apiprobe requires ≥ 2.7. Either bump `mgf-common`'s pin or pick a
   compatible 2.6 version. Do NOT introduce conflicting pins.
4. **YAML cassette format and `pyyaml` safety** — `yaml.safe_load`
   only (rule SC-03). No tag-based deserialization.
5. **Plugin entry-point group names** — `mgf_apiprobe.<group>`
   (underscore). Hyphens illegal in entry-point group names.
6. **Async test fixtures** — pytest plugin must work under `strict`
   and `auto` asyncio modes; default to honoring the consumer's
   config.
7. **Coverage tooling** — `coverage run -m pytest` then
   `coverage report --fail-under=80`. NOT `pytest --cov` (under-reports
   per the `mgf-common` incident).

---

*End of `HANDOFF.md`. The spec is `DESIGN.md`. When this handoff
disagrees with DESIGN.md on packaging, this handoff overrides.*
