# Performance & Capacity

> **Status:** v0.1 — first pass, finalized 2026-05-15
> (review-pass verified all file:line citations against the
> live source). Last updated 2026-05-15.
> A federation-sibling-scoped performance audit of the single
> source file (`_env.py`). The library is invoked from
> `alembic env.py` at migration time — performance is mostly
> about **migration-run latency** + **startup cost when run
> in CI** + **autogenerate cost on large schemas**.
>
> **Scope:** mgf-alembic's OWN performance footprint. NOT a
> re-audit of alembic / SQLAlchemy / asyncpg.
>
> **Audience:** the maintainer + future AI agents.
> **Cross-references:**
> [`RELIABILITY_AND_ACCURACY.md`](RELIABILITY_AND_ACCURACY.md),
> [`SECURITY_HARDENING.md`](SECURITY_HARDENING.md),
> [`mgf-common/docs/inprogress/PERFORMANCE_AND_CAPACITY.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/inprogress/PERFORMANCE_AND_CAPACITY.md)
> (cornerstone),
> [`mgf-sqlalchemy/docs/inprogress/PERFORMANCE_AND_CAPACITY.md`](https://codeberg.org/magogi-admin/mgf_sqlalchemy/src/branch/main/docs/inprogress/PERFORMANCE_AND_CAPACITY.md)
> (sister-sibling — pool_pre_ping concerns parallel).

Migrations are mostly **one-shot**: a developer or CI runs
`alembic upgrade head`, waits for completion, moves on.
Performance items here are about:

1. **Configure-env latency** — the wrapper's Python overhead
   at invocation.
2. **Migration-run wall time** — alembic's work, dominated
   by the actual DDL.
3. **Autogenerate cost on large schemas** — `--autogenerate`
   compares model metadata against the live DB; scales
   linearly in column / index / FK count.

---

## §1 You are here

mgf-alembic ships one public name (`configure_env`).
The wrapper's Python overhead is ~10-50 µs at invocation;
the rest is alembic + SQLAlchemy + DDL execution.

The relevant PC question for consumers: **how long does a
typical migration cycle take?** For:

- A 2-column schema change: ~1-3 s (driven by network +
  DDL).
- A schema with 100 tables + autogenerate: ~5-30 s (the
  metadata-diff dominates).
- A migration that adds an index on a 10M-row table: 30s-
  several minutes (the actual `CREATE INDEX` blocks).

The library doesn't control most of this — but does
contribute the wrapper-tax + the offline-mode SQL
generation cost.

**Counts (2026-05-17):**

| Priority | Open | Closed | Total |
|---|---|---|---|
| P0 — pre-v1.0 blockers | 0 | 0 | 0 |
| P1 — should land before/with v1.0 | 0 | 2 | 2 |
| P2 — post-v1.0 hardening | 0 | 3 | 3 |
| **Total** | **0** | **5** | **5** |

**Do first:** ALL items closed. PC-01 + PC-02 (2026-05-16);
PC-03 + PC-04 + PC-05 (2026-05-17, P2 sweep — docstring
sections covering autogenerate scaling, offline-mode SQL
throughput, and engine-construction NullPool choice).

---

## §2 Priority table

| ID | Theme | One-liner | Effort | Status |
|---|---|---|---|---|
| [**PC-01**](#pc-01-configure-env-python-overhead) | Startup latency | `configure_env(...)` does URL-prefix detection + naming-convention apply + alembic-context configure dispatch. Python overhead ~10-50 µs. Pin a budget. | XS | ✅ closed |
| [**PC-02**](#pc-02-asyncio-run-startup-cost) | Startup latency | The async path calls `asyncio.run(_run_online_async(...))`. `asyncio.run` creates a fresh event loop — ~5-20 ms cost. For test fixtures running many short migrations, the cost compounds. | XS | ✅ closed |
| [**PC-03**](#pc-03-autogenerate-cost-on-large-schemas) | Capacity / scaling | `alembic revision --autogenerate` is alembic-owned, NOT our wrapper. But the `include_object` callback we pass is consumer-controlled and runs per object. A slow callback scales linearly with schema size. Document the budget. | S | ✅ closed (docs) |
| [**PC-04**](#pc-04-offline-mode-sql-generation-throughput) | Capacity | `alembic upgrade --sql ...` emits SQL for ALL pending migrations. For a project with 100s of migrations, the offline-mode generation iterates them in series. Document; quantify if it becomes a hot path. | XS | ✅ closed (docs) |
| [**PC-05**](#pc-05-engine-construction-overhead) | Startup | `create_engine(url, poolclass=NullPool)` happens once per migration run. NullPool means every operation gets a fresh connection — but for a one-shot migration that's correct. Document the choice. | XS | ✅ closed (docs) |

Effort scale: XS (≤ 15 min); S (≤ 1 h); M (≤ 1 session).

---

## §3 Invariants we hold

The list of "closed elsewhere; do not re-open":

- **PC-INV-1** — `configure_env` does **no per-object
  work**; everything heavy is delegated to alembic.
- **PC-INV-2** — `NullPool` is used for both sync + async
  paths
  ([`_env.py:233`, `_env.py:260`](../../src/mgf/alembic/_env.py#L233)).
  No pool warmup, no checkout overhead — one connection,
  one migration, dispose.
- **PC-INV-3** — Engine **disposed in `finally`** on both
  paths. No leaked connections from migration runs.
- **PC-INV-4** — `compare_type=True` + `compare_server_default=True`
  defaults are **stricter than alembic's stock**. They
  detect more changes; cost is some extra autogenerate
  work. Documented choice.
- **PC-INV-5** — `naming_convention` is applied to
  `target_metadata` **once** before `context.configure`
  runs. No per-object naming work.

---

## §4 Work units

### PC-01 configure_env Python overhead

**Why:** `configure_env`
([`_env.py:48-155`](../../src/mgf/alembic/_env.py#L48))
runs:

1. Import alembic context (cached).
2. Set `target_metadata.naming_convention` (if provided).
3. `context.is_offline_mode()` check.
4. URL-prefix detection.
5. Dispatch to one of three paths.

Python overhead before alembic actually starts working:
~10-50 µs.

This isn't a per-request hot path — it's once per migration
run. But pinning catches regressions where the dispatch
logic somehow grows expensive.

**Concrete output:**

- Add `tests/perf/test_configure_env_overhead.py`:
  - Mock `_run_offline` / `_run_online_sync` / `_run_online_async`
    as no-op functions.
  - Call `configure_env(...)` × 1000.
  - Measure per-call Python time.
  - Assert ≤ 100 µs per call.

**Locked in by:** the perf test.

**Cross-references:** mgf-common PC-01 (cold-import; affects
the first `configure_env` call).

**Status:** ✅ closed (2026-05-16). See §5.

---

### PC-02 asyncio.run startup cost

**Why:** The async path
([`_env.py:134-145`](../../src/mgf/alembic/_env.py#L134))
calls:

```python
asyncio.run(_run_online_async(...))
```

`asyncio.run` creates a fresh event loop:

1. Allocate loop.
2. Run the coroutine to completion.
3. Tear down the loop.

Startup cost: ~5-20 ms on a modern CPython.

For a single migration run: amortised over the migration's
DDL execution (which is seconds), the overhead is invisible.

For test fixtures running many short migrations (a project's
test suite might run hundreds of mini-migrations), the
cost compounds: 100 migrations × 10 ms = 1 s of just
loop-creation time.

RA-01 surfaces this from the correctness angle (asyncio.run
fails from a running loop); this surfaces it from the
performance angle.

**Concrete output:**

- Add a note to the docstring:
  > **Performance:** Each async-path invocation creates a
  > fresh event loop (~5-20 ms). For test fixtures running
  > many short migrations, consider invoking the async
  > primitives directly via `await _run_online_async(...)`
  > from your own event loop.
- Pair with RA-01's fix that exports `_run_online_async`
  as a public name so consumers CAN call it directly.

**Locked in by:** the docstring + the public-name export
(per RA-01).

**Cross-references:** RA-01 (correctness twin); CPython
docs on asyncio.run cost.

**Status:** ✅ closed (2026-05-16). The performance-impact
docstring note shipped as part of PC-02; the public-name
export it suggests (currently `_run_online_async`) waits on
RA-01's closure. See §5.

---

### PC-03 Autogenerate cost on large schemas

**Why:** `alembic revision --autogenerate` is alembic-owned;
the wrapper just passes through. But the consumer-supplied
`include_object` callback runs per object during
autogenerate. For:

- 50 tables × 20 columns × 5 indexes ≈ 5000 objects.
- Each invocation of `include_object` runs Python.
- Total: 5000 × callback-time.

A slow callback (e.g., one that queries a registry for each
object) becomes the bottleneck.

The wrapper can't enforce callback speed but CAN document
the budget.

**Concrete output:**

- Add a note to `include_object`'s docstring:
  > **Performance:** The callback runs once per object
  > during autogenerate. For a schema with N objects and
  > callback time T, total autogenerate time is ~N×T plus
  > alembic's metadata-diff work (~100 µs per object). Keep
  > callbacks fast (≤10 µs); avoid I/O / database queries
  > from inside the callback.

**Locked in by:** the docstring.

**Cross-references:** PAPER-22 (the include_object
extension reference).

---

### PC-04 Offline-mode SQL generation throughput

**Why:** `alembic upgrade --sql` emits SQL for all pending
migrations. The wrapper takes the offline-mode path
([`_env.py:123-132`](../../src/mgf/alembic/_env.py#L123))
which calls `_run_offline(...)`.

For a project with 100 migrations: alembic iterates them in
series, emitting the migration SQL for each. Total time:
~1-5 s for moderate-size projects.

The wrapper's overhead here is negligible compared to
alembic's own iteration cost.

Document + leave as a known concern.

**Concrete output:**

- Document in `configure_env`'s docstring + the offline-mode
  guide (recipes):
  > **Offline-mode performance:** Generates SQL for ALL
  > pending migrations in series. For projects with 100+
  > migrations, the generation takes seconds. This is
  > alembic's behaviour; the wrapper doesn't add overhead.

**Locked in by:** the docstring; no test (operational
concern, not a regression case).

---

### PC-05 Engine construction overhead

**Why:** Both online paths call:

```python
connectable = create_engine(url, poolclass=pool.NullPool)
# OR
connectable = create_async_engine(url, poolclass=pool.NullPool)
```

Per-migration: ~50-200 µs for SQLAlchemy's engine
construction. Per-process: once. Amortised over the
migration's DDL.

The NullPool choice is correct: migrations are one-shot;
no pool reuse needed.

**Concrete output:**

- Document the NullPool choice in `configure_env`'s
  docstring:
  > **NullPool by design.** Migrations are one-shot; the
  > engine connects once + dispose at run end. Connection
  > pooling adds startup overhead with no benefit for
  > one-shot use cases.

**Locked in by:** docstring (no test — the choice is
documentation-shaped).

**Cross-references:** mgf-sqlalchemy PC-02 (pool-pre-ping —
different choice for long-running apps).

---

## §5 Closed items

- **PC-01** — `configure_env` per-call Python overhead pinned at
  ≤ 100 µs/call (measured ~5 µs/call locally; 20x cushion for CI
  noise + slower hardware). Added `tests/perf/test_configure_env_overhead.py`
  with three pinned dispatch shapes: offline, online-sync, and
  the naming-convention apply path. New `perf` marker registered
  in `pyproject.toml`; default suite deselects it via
  `-m "not perf"` (matches mgf-common's convention). Run via
  `pytest -m perf` for the nightly sweep. (2026-05-16, commit `09c906c`.)
- **PC-02** — `asyncio.run` startup cost (~5-20 ms per async-path
  invocation) documented in `configure_env`'s docstring under a
  new "Notes" section. For test fixtures running many short
  migrations, the cost compounds (100 invocations ≈ 1 s of pure
  loop-creation time); the docstring states the budget so
  consumers know what they pay. A public async entry-point is
  the pairing fix in RA-01; until it lands, the docstring
  forward-references RA-01 + suggests the alembic-native
  `async_engine_from_config` / `run_sync` pattern as the
  workaround. (2026-05-16, commit pending.)
- **PC-03** — Autogenerate-cost expectations documented in the
  `include_object` docstring ("Performance (PC-03)" section).
  Callback runs once per schema object; total autogenerate time
  is ~N*T (N objects, T per-call cost) plus alembic's own
  metadata-diff work (~100 µs per object). Documented budget:
  callbacks SHOULD be ≤10 µs; avoid I/O / database queries.
  No code change — the library can't enforce callback cost,
  but consumers now know the contract. (2026-05-17, P2 sweep
  — documentation-only close.)
- **PC-04** — Offline-mode SQL generation throughput documented
  in the `configure_env` docstring ("Performance — offline-mode
  SQL throughput (PC-04)" section). `alembic upgrade --sql`
  iterates ALL pending migrations in series; for a project with
  100+ migrations this is ~1-5 s of pure alembic time (the
  wrapper adds no measurable overhead). Documented for operator
  awareness; no mitigation needed at the library layer.
  (2026-05-17, P2 sweep — documentation-only close.)
- **PC-05** — Engine-construction NullPool choice documented in
  the `configure_env` docstring ("Performance — engine
  construction (PC-05)" section). Both online paths construct a
  fresh engine with `poolclass=NullPool` (~50-200 µs). Correct
  for the migration-as-one-shot shape: every operation gets a
  fresh connection, no pool overhead, no leaked connections
  between migrations. Consumers wanting long-lived connections
  should not reach for this helper. (2026-05-17, P2 sweep —
  documentation-only close.)

---

## §6 Notes on methodology

This first pass walked the single source file (~230 LOC)
against:

- mgf-common's PC tracker methodology.
- alembic's documented per-object autogenerate cost.

The audit method:

1. Identify wrapper's per-call work vs alembic's per-call
   work.
2. Pin the wrapper Python overhead.
3. Document budgets for consumer-controlled extension
   points (`include_object`).

Five items + five invariants. The library's PC shape is
thin — it's a one-shot helper. Future passes should look for:

- **Migration-script optimisation patterns.** Consumers
  writing slow migrations (`ALTER TABLE` on huge tables)
  benefit from runbook content + recipe.
- **Concurrent-migration coordination.** If a CI runs the
  same migrations against multiple test environments
  concurrently, the lock-on-the-version-table matters.

When an item closes, move it to §5 with the closing-commit
SHA + measured baseline.
