# Configuration

> **Conformance:** L1 / L2 / L3 (see `README.md`)
> **Audience:** Human engineers AND AI coding agents.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **📦 Note.** The reusable base lives in **`mgf.common.config`**: `BaseAppSettings` (pydantic-settings base with layered sources, YAML override, version-check validator), `settings_config(env_prefix=...)` (the canonical helper for setting `model_config` on a subclass — DO NOT use `**BaseAppSettings.model_config` spread; it doesn't work, see `src/mgf/common/config/_settings.py` docstring), `load_settings` / `save_settings` (typed loader + atomic writer with optional migration callback), `platform_dir` / `default_config_path` (cross-OS dirs), and `expand_path` (path-validator helper). Consumers subclass `BaseAppSettings` with their own typed fields.

---

## Rules box (read this first)

CF-* rules apply to consumers that **own their config schema**
(the canonical mgf-common shape: pydantic-settings + typed
nested fields + atomic save). For consumers using
**framework-native config** (Django's `from django.conf import
settings`, Rails-equivalent), a subset of rules is N/A — see
[§0a Framework-native config](#0a-framework-native-config-django-rails-equivalent)
below. The `Applies to` column documents the scope.

| ID | Rule | Applies to |
|---|---|---|
| <a id="CF-01"></a>**CF-01** | Application configuration MUST live in a single typed object — a `pydantic_settings.BaseSettings` subclass. **No module reads the config file or env vars directly.** | consumer-owned-config |
| <a id="CF-02"></a>**CF-02** | Precedence MUST be, top-to-bottom: **CLI flags → environment variables → `.env` file → on-disk config file → typed defaults in code.** Higher entries override lower. | all |
| <a id="CF-03"></a>**CF-03** | The config schema MUST carry a `version` field. Loading a higher version than the code knows MUST fail loud. Loading a lower version MUST run a migration chain that brings the dict up to current. | consumer-owned-config |
| <a id="CF-04"></a>**CF-04** | Writes to the on-disk config file MUST be atomic (temp file + rename). Mode MUST be `0o600`. Path expansion (`~`, `$VARS`) MUST happen lazily (at call time, not module import). | consumer-owned-config |
| <a id="CF-05"></a>**CF-05** | Secrets MUST NOT live in the config file. They live in a vault (system keyring / Keychain / Credential Manager / encrypted file fallback) accessed through a separate `Vault` interface. | all |
| <a id="CF-06"></a>**CF-06** | Loading an **unknown key** from the config file MUST raise (typo protection). Missing keys MUST fall back to typed defaults silently. | consumer-owned-config |
| <a id="CF-07"></a>**CF-07** | Sub-config files (hooks, schedules, alerts, profiles, lab specs, etc.) MUST follow the same shape: typed model + atomic write + version field + unknown-key rejection. | consumer-owned-config |
| <a id="CF-08"></a>**CF-08** | The `Settings` object MUST be loaded **once** at the entry point and passed down by constructor injection. Module-global singletons are forbidden. | consumer-owned-config |
| <a id="CF-09"></a>**CF-09** | A `--config <path>` CLI flag MUST always exist. The default path MUST honour XDG / OS conventions (`$XDG_CONFIG_HOME` / `~/.config/<app>/config.yaml` on Linux; `%APPDATA%\<app>\config.yaml` on Windows; `~/Library/Application Support/<app>/config.yaml` on macOS; sandboxed app dir on iOS/Android). | consumer-owned-config |
| <a id="CF-10"></a>**CF-10** | Tests MUST construct `Settings` with explicit values (no env-var leakage). Use `Settings(**overrides)` or `Settings.model_construct` for unit tests. | all |
| <a id="CF-11"></a>**CF-11** | Renaming a config field MUST NOT break existing on-disk configs. Use pydantic `Field(alias=...)` + `validation_alias` to accept the old name; emit a one-shot DEPRECATION WARNING; rewrite to the new name on next save. After two MINOR releases, the alias MAY be removed. | consumer-owned-config |
| <a id="CF-12"></a>**CF-12** | When a config field has multiple mutually-exclusive backend variants (`vault.backend = "keyring" \| "file" \| "kms"`), the variants MUST be modelled as a **discriminated union** with a `kind` discriminator field. Each variant carries only its own backend's settings. See §14. | all |

---

## 0a. Framework-native config (Django, Rails-equivalent)

Closes FEEDBACK CONFIG-01. Web frameworks like Django own the
config schema themselves (settings as Python module + env-var
overlays). Several CF-* rules conflict with the framework's
canonical pattern — applying them literally would mean
abandoning the framework's intended use.

### Which CF-* rules are N/A for framework-native config

The four rules that encode the *consumer-owned, on-disk YAML,
CLI-loadable* shape are explicitly N/A under framework-native
config:

| Rule | Why it's N/A under framework-native config |
|---|---|
| **CF-01** | Django's `config/settings/*.py` IS where env-var reads happen. The settings module IS the consumer-of-env-vars by design. |
| **CF-04** | Django settings are version-controlled Python code, deployed via Ansible/Terraform/Docker. There is no runtime write of "the config file." |
| **CF-08** | `from django.conf import settings; settings.X` IS a module-global singleton lookup. It's the framework's documented access pattern; rewriting it would touch every call site for no behavioural improvement. |
| **CF-09** | Django uses `DJANGO_SETTINGS_MODULE` env var to point at a Python module, not a config file path. ASGI/WSGI processes don't take CLI args. |

### Which CF-* rules still apply

Most of the standard's intent translates cleanly:

- **CF-02** (precedence) — Django reads env vars in
  `settings.py` via `os.environ.get(...)` or
  `decouple.config(...)`. The precedence chain becomes
  *env > .env > settings.py defaults*. Same idea, framework-native
  mechanics.
- **CF-05** (secrets in vault) — Always applies. Django
  `settings.py` MUST NOT contain literal secrets; use
  Ansible Vault / .env / Cloud secrets manager.
- **CF-10** (tests with explicit values) — Pytest fixtures use
  Django's `@override_settings` decorator + per-test env-var
  overrides. Same intent as the canonical pattern.
- **CF-12** (discriminated unions) — Applies wherever the
  consumer has mutually-exclusive backend variants in their
  Django config (cache backend, vault backend, etc.).

CF-03 (schema version), CF-06 (unknown-key reject), CF-07
(sub-config shape), CF-11 (rename via alias) are also N/A for
framework-native — Django settings are Python code with no
schema-version marker, attribute access on a missing key
already raises `AttributeError`, sub-config files don't apply
the same way, and renaming a setting is a code refactor, not
a data migration.

### Framework-native isn't an exemption from discipline

The carve-out narrows the rule scope; it doesn't reduce the
discipline. A framework-native consumer is still expected to:

- **Lint-enforce** that env-var reads happen ONLY inside the
  framework's settings module(s). A `TestEnvVarsOnlyInSettings`
  test that bans `os.environ.get(`, `os.environ[`,
  `decouple.config(` outside `config/settings/` is the
  canonical shape.
- **Document** which CF-* rules apply and which are N/A in the
  project's adoption tracker (e.g., `docs/MGF_COMMON_ADOPTION.md`).
  The worked example below is the recommended shape.
- **Adopt CF-05** (secrets in vault) without exception. Vault
  discipline is framework-agnostic.

### Worked example — a Django consumer's CF-* disposition

```
CF-01 — N/A (Django-owned). Env-var reads happen ONLY in
        config/settings/*.py; lint-enforced via
        TestEnvVarsOnlyInSettings.
CF-02 — APPLIES. Order is: env > .env > settings.py defaults.
CF-03 — N/A (Django settings have no schema_version field;
        settings ARE Python code).
CF-04 — N/A (settings are deployed code, not runtime data).
CF-05 — APPLIES. Secrets in Ansible Vault (prod) / .env (dev),
        never in settings.py.
CF-06 — N/A (Django attribute access on missing key already
        raises AttributeError).
CF-07 — Project chooses; project-local config helpers (e.g.,
        tools/lib/test_config.py) stay on the typed-config
        pattern (CF-* compliant) for the test harness.
CF-08 — N/A (Django mandates module-global access).
CF-09 — N/A (DJANGO_SETTINGS_MODULE env var replaces it).
CF-10 — APPLIES. pytest fixtures construct Settings with
        explicit overrides; env leaks banned by conftest.
CF-11 — N/A (settings rename is a code refactor, not data
        migration).
CF-12 — APPLIES where relevant (e.g., CACHE_BACKEND alternatives).
```

### Future framework support

Today the carve-out is documented for Django specifically.
Rails (via Ruby), Flask, and FastAPI have similar but distinct
shapes; consumers in those domains MAY follow the same
pattern (declare which CF-* apply, which are N/A) and file
`FRAMEWORK-NN` papers if the carve-out needs framework-specific
clarification.

---

## 0b. Auto-provisions you inherit from `BaseAppSettings`

Closes [PAPER-33](../../FEEDBACK.md). A consumer that subclasses
`mgf.common.config.BaseAppSettings` (per the canonical CF-01 shape)
inherits **four CF-* rules conformant-for-free**. Audits often flag
these as `⏳ no-surface-yet` or `❓ unclear` when the consumer's own
`Settings` class doesn't visibly carry the field; in fact the base
class provides it. This section lists what's covered without
further work so audit rows can land as `✅ conformant` directly.

| Rule | What `BaseAppSettings` provides | Where to look |
|---|---|---|
| **CF-02** (precedence) | Layered source ordering — CLI overrides → env vars → `.env` → on-disk YAML → typed defaults — is set up by `BaseAppSettings.settings_customise_sources` + `load_settings(overrides=...)`. Subclasses don't reimplement. | `src/mgf/common/config/_settings.py:settings_customise_sources` + `_loader.py::load_settings` |
| **CF-03** (schema version) | `version: int = 1` is declared on the base class. Subclasses get the field automatically; override the default + add a migration callback when the schema evolves past v1. | `src/mgf/common/config/_settings.py:175` |
| **CF-06** (unknown-key reject) | `model_config = settings_config()` sets `extra="forbid"`. Typos in the YAML file surface as a single `ValidationError` listing every bad key. Subclasses inherit the strict mode unless they override `extra=` explicitly. | `src/mgf/common/config/_settings.py:settings_config` defaults block (`"extra": "forbid"`) |
| **CF-08** (loaded once at entry point) | Honoured at the wiring layer, not the model: `mgf.common.bootstrap()` and `mgf.common.config.load_settings()` together implement the "load once, pass via constructor injection" pattern. Subclasses don't reimplement this either. | `src/mgf/common/_lifecycle.py::bootstrap` + `src/mgf/common/config/_loader.py::load_settings` |

**Override discipline.** A subclass MAY override `version: int = N`
or `model_config = settings_config(extra="allow")` if its domain
genuinely requires a different default; in that case the override
is the explicit decision and the auto-provision no longer applies.
The audit should then flip from "inherited" to "explicit decision
documented in subclass."

**What's NOT auto-provisioned.** CF-01 (the typed-model rule
itself — you still write the subclass), CF-04 (atomic write — you
call `save_settings(...)`), CF-05 (vault — you wire a
`Vault` instance), CF-07 (sub-config files — you write a satellite
loader), CF-09 (CLI `--config` flag — you wire the click option),
CF-10 (tests — you write per-test fixtures), CF-11 (rename via
alias — you add the `Field(alias=)` when you rename), CF-12
(discriminated unions — you model the backend variants).

**Implicit footprint (for partial adopters).** The four
auto-provisions above are also four defaults you inherit whether or
not you think about them. A consumer that depends on `mgf-common`
gets `extra="forbid"` and `version: int = 1` as standing behaviour:
an unknown YAML key is a hard failure, and a config stamped by a
newer build refuses to load on an older one. That is the intended
contract — but a *partial* adopter (one lifting only part of the
shape, or wrapping `BaseAppSettings` in a looser parent) MUST decide
each of CF-02 / CF-03 / CF-06 / CF-08 explicitly rather than assume
the default still holds. Document the disposition in the adoption
tracker (`MGF_COMMON_ADOPTION.md`) exactly as the framework-native
carve-out does in §0a: which rules you inherit, which you re-own,
and why.

---

## 0. Flat vs nested Settings — explicit choice

The drop-in template in §3.1 below shows a **nested** schema
(`LoggingSettings`, `StorageSettings`, ...). That is the *recommended*
shape for projects starting from scratch. It is **not** required by
any CF-* MUST.

For projects migrating from an existing flat config (e.g.,
when porting from a stdlib `@dataclass`), a **flat** `BaseSettings`
with the same field names is equally compliant — every CF MUST
(typed model, layered precedence, schema versioning, atomic write,
secrets→vault, unknown-key reject) is satisfied either way. The flat
shape avoids touching every `cfg.X` consumer in the project.

New projects SHOULD pick nested upfront because it groups
related fields and reads better in a deep configuration;
existing projects MAY stay flat unless they have a concrete
reason to migrate.

The rest of this document describes the nested template. Where the
flat shape differs (almost nowhere — the model_config, validators,
loader, save, and migration patterns are identical), the differences
are noted in line.

## 1. Mental model

Configuration is a **typed, validated, versioned object** built once at process startup from layered sources, then passed by reference to anything that needs it. Everything else is a code smell:

- "Reading the env var here because it's convenient" → CF-01 violation.
- "Loading config.yaml in this module to check one field" → CF-08 violation.
- "Adding a string field to the YAML and parsing it ad-hoc" → CF-01 + CF-06 violation.

The standard prescribes:

1. **`pydantic-settings.BaseSettings`** as the schema.
2. **Layered loading** with documented precedence (CF-02).
3. **Schema versioning** with migration (CF-03) so you can refactor the schema without breaking users.
4. **Atomic writes** (CF-04) so a crash never corrupts state.
5. **Vault separation** (CF-05) so secrets never leak into config dumps, version control, or logs.
6. **Same shape for satellite files** (CF-07) — hooks.yaml, schedules.yaml, etc. each get their own typed model with the same load/save/migrate/validate machinery.

---

## 2. Why pydantic-settings (not stdlib dataclass)

A stdlib `@dataclass` works for L1 but does not scale to:

- **Env-var merging.** `pydantic-settings` reads env vars natively (`SECRET_KEY=...` → `Settings.secret_key`) with prefix support (`<APP>_SECRET_KEY`).
- **`.env` file support.** Built in. Critical for local development and 12-factor-style deployment.
- **Validation messages.** Pydantic's errors are precise and aggregated — every bad field reported, not just the first one.
- **Nested config.** Sub-models compose cleanly. A `LoggingSettings`, `DatabaseSettings`, `VaultSettings` group themselves.
- **JSON / YAML round-trip.** `model_dump(mode="json")` produces a serializable dict; `Settings.model_validate(dict)` parses it back.
- **Discriminated unions.** Per-backend variants (e.g., `LibvirtBackend | MockBackend`) are first-class.
- **Aliases.** Field renames without breaking older configs (`Field(alias="old_name")`).

Cost: one extra dependency (`pydantic`, `pydantic-settings`). Both are mainstream, well-maintained, and add ~3 MB. Worth it.

---

## 3. The `Settings` schema

### 3.1 Drop-in template — `<app>/settings.py`

```python
"""Application settings — single typed source of truth.

Loaded once at startup by the CLI / GUI / service entry point and
passed down by constructor injection. No other module reads env
vars, the config file, or .env directly (rule CF-01).

Precedence (rule CF-02): CLI flags > env vars > .env > config file > defaults.
Implementation: settings_customise_sources() defines the order.

Sensitive fields: NOT stored here. Use the Vault (rule CF-05).

See ``docs/standards/CONFIGURATION.md`` for the full standard.
"""

from __future__ import annotations

import os
from pathlib import Path
from typing import Any, Literal

import yaml
from pydantic import Field, ValidationError, field_validator
from pydantic_settings import (
    BaseSettings,
    PydanticBaseSettingsSource,
    SettingsConfigDict,
    YamlConfigSettingsSource,
)

from <app>.exceptions import AppConfigError, SchemaValidationError

CURRENT_SCHEMA_VERSION = 1

# ---------------------------------------------------------------------------
# Sub-models — group related fields into nested schemas
# ---------------------------------------------------------------------------

class LoggingSettings(BaseSettings):
    model_config = SettingsConfigDict(extra="forbid")

    level: Literal["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"] = "INFO"
    format: Literal["text", "json", "auto"] = "auto"
    file_path: Path | None = None  # None → default <state>/logs/<app>.log
    enable_otlp: bool = False
    redact_extra_keys: list[str] = Field(default_factory=list)

class StorageSettings(BaseSettings):
    model_config = SettingsConfigDict(extra="forbid")

    image_store: Path = Path("~/.local/share/<app>/images")
    state_dir: Path = Path("~/.local/state/<app>")

class GuiSettings(BaseSettings):
    model_config = SettingsConfigDict(extra="forbid")

    theme: Literal["dark", "light", "system"] = "dark"
    refresh_interval_ms: int = Field(default=2000, ge=100)

# ---------------------------------------------------------------------------
# Top-level Settings
# ---------------------------------------------------------------------------

class Settings(BaseSettings):
    """Single typed source of truth for the application config.

    Construct via ``load_settings()``, NOT by calling ``Settings()``
    directly — load_settings wires the file path and the migration
    chain.
    """

    model_config = SettingsConfigDict(
        env_prefix="<APP>_",                        # SECRET → <APP>_SECRET
        env_nested_delimiter="__",                  # GUI__THEME → gui.theme
        env_file=".env",                            # local dev
        env_file_encoding="utf-8",
        extra="forbid",                             # rule CF-06
        case_sensitive=False,
        validate_default=True,
        # The yaml_file is set dynamically via _yaml_file_path; see
        # settings_customise_sources below.
    )

    # ── Schema version (rule CF-03) ────────────────────────────────
    version: int = CURRENT_SCHEMA_VERSION

    # ── Sub-models ─────────────────────────────────────────────────
    logging: LoggingSettings = Field(default_factory=LoggingSettings)
    storage: StorageSettings = Field(default_factory=StorageSettings)
    gui: GuiSettings = Field(default_factory=GuiSettings)

    # ── Top-level fields ───────────────────────────────────────────
    backend_uri: str = "qemu:///system"
    ssh_timeout_seconds: int = Field(default=30, ge=1, le=600)
    health_check_timeout_seconds: int = Field(default=120, ge=1)

    # ── Validators ─────────────────────────────────────────────────

    @field_validator("version")
    @classmethod
    def _check_version(cls, v: int) -> int:
        if v > CURRENT_SCHEMA_VERSION:
            raise ValueError(
                f"config version {v} is newer than supported "
                f"version {CURRENT_SCHEMA_VERSION}; upgrade <app>"
            )
        return v

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        """Define precedence (rule CF-02).

        Order: init (CLI flag overrides) > env > .env > YAML file > defaults.
        Pydantic resolves left-to-right (left wins).
        """
        yaml_path = _yaml_file_path()
        yaml_source = YamlConfigSettingsSource(settings_cls, yaml_file=yaml_path) if yaml_path.exists() else None
        sources: list[PydanticBaseSettingsSource] = [
            init_settings,
            env_settings,
            dotenv_settings,
        ]
        if yaml_source is not None:
            sources.append(yaml_source)
        sources.append(file_secret_settings)
        return tuple(sources)

    # ── Path expansion happens lazily on demand (rule CF-04) ───────

    def expanded_image_store(self) -> Path:
        return _expand(self.storage.image_store)

    def expanded_state_dir(self) -> Path:
        return _expand(self.storage.state_dir)

# ---------------------------------------------------------------------------
# Load / save / migrate
# ---------------------------------------------------------------------------

def default_config_path() -> Path:
    """OS-aware default path (rule CF-09).

    Linux  : $XDG_CONFIG_HOME/<app>/config.yaml or ~/.config/<app>/config.yaml
    macOS  : ~/Library/Application Support/<app>/config.yaml
    Windows: %APPDATA%/<app>/config.yaml
    """
    if os.name == "nt":
        base = Path(os.environ.get("APPDATA", Path.home() / "AppData" / "Roaming"))
    elif (xdg := os.environ.get("XDG_CONFIG_HOME")):
        base = Path(xdg)
    else:
        base = Path.home() / ".config"
    return base / "<app>" / "config.yaml"

def _yaml_file_path() -> Path:
    """Used by settings_customise_sources to know which YAML to read.

    Honours the <APP>_CONFIG_PATH env var override, falls back to the
    OS default. Tests override via monkeypatch on this function.
    """
    override = os.environ.get("<APP>_CONFIG_PATH")
    return Path(override) if override else default_config_path()

def load_settings(
    *,
    config_path: Path | None = None,
    overrides: dict[str, Any] | None = None,
) -> Settings:
    """Build the Settings object from layered sources.

    Parameters
    ----------
    config_path : Path or None
        Override the YAML file path. When provided, sets the
        <APP>_CONFIG_PATH env var so settings_customise_sources sees
        it. (We do not pass config_path through Settings(...) because
        pydantic-settings does not support per-call yaml_file overrides
        on YamlConfigSettingsSource.)
    overrides : dict or None
        CLI-level field overrides. These win over env vars (rule CF-02).
    """
    if config_path is not None:
        os.environ["<APP>_CONFIG_PATH"] = str(config_path)

    yaml_path = _yaml_file_path()
    if yaml_path.exists():
        try:
            raw = yaml.safe_load(yaml_path.read_text(encoding="utf-8")) or {}
        except yaml.YAMLError as e:
            raise AppConfigError(f"could not parse {yaml_path}: {e}") from e
        if not isinstance(raw, dict):
            raise AppConfigError(
                f"{yaml_path} must contain a YAML mapping at the top level, got {type(raw).__name__}"
            )
        # Migration chain (rule CF-03).
        version = raw.get("version", CURRENT_SCHEMA_VERSION)
        if isinstance(version, int) and version < CURRENT_SCHEMA_VERSION:
            raw = _migrate(raw, from_version=version)
            yaml_path.write_text(yaml.safe_dump(raw, sort_keys=False), encoding="utf-8")  # rewrite migrated

    # pydantic-settings reads everything; we just pass overrides for the highest-precedence layer.
    try:
        return Settings(**(overrides or {}))
    except ValidationError as e:
        raise AppConfigError(f"invalid configuration: {e}") from e

def save_settings(settings: Settings, *, path: Path | None = None) -> None:
    """Atomic write (rule CF-04).

    Writes to a temp file in the same directory then renames, so a
    crash mid-write never leaves a truncated YAML file. File mode is
    0o600 to keep the config user-private.
    """
    target = path if path is not None else _yaml_file_path()
    target.parent.mkdir(parents=True, exist_ok=True)

    payload = settings.model_dump(mode="json", exclude_unset=False)
    payload["version"] = CURRENT_SCHEMA_VERSION  # always written

    tmp = target.with_suffix(target.suffix + ".tmp")
    try:
        tmp.write_text(
            yaml.safe_dump(payload, sort_keys=False, default_flow_style=False, allow_unicode=True),
            encoding="utf-8",
        )
        tmp.chmod(0o600)
        tmp.replace(target)
    except Exception:
        tmp.unlink(missing_ok=True)
        raise

def _migrate(raw: dict[str, Any], *, from_version: int) -> dict[str, Any]:
    """Walk the migration chain from `from_version` to CURRENT_SCHEMA_VERSION.

    Each step: read the current shape, write the next shape. Steps
    are idempotent and ordered. Add a new step here when you bump
    CURRENT_SCHEMA_VERSION.
    """
    if from_version == 1 == CURRENT_SCHEMA_VERSION:
        return raw
    # Example placeholder for future bumps:
    # if from_version == 1:
    #     raw["new_field"] = "default"
    #     raw["version"] = 2
    #     from_version = 2
    return raw

def _expand(p: Path) -> Path:
    """Lazy ~ / $VAR expansion (rule CF-04 — never at module import time)."""
    return Path(os.path.expandvars(str(p))).expanduser()

__all__ = [
    "CURRENT_SCHEMA_VERSION",
    "GuiSettings",
    "LoggingSettings",
    "Settings",
    "StorageSettings",
    "default_config_path",
    "load_settings",
    "save_settings",
]
```

### 3.1.1 Per-call YAML override via `ClassVar`

Pydantic interprets any annotated class-body assignment as a model
field or `ModelPrivateAttr` (per-instance). A normal `_x: Path | None = None`
declaration becomes a private attribute, NOT the class-level
attribute that `settings_customise_sources` needs.

The right shape: declare it as `ClassVar`:

```python
from typing import ClassVar

class Settings(BaseSettings):
    # NOT a field, NOT a private attr — a real class attribute that
    # ``settings_customise_sources`` (a classmethod) can read + write
    # without per-instance state.
    _yaml_file_override: ClassVar[Path | None] = None
```

Then `load_settings(...)` sets it before constructing the model:

```python
def load_settings(*, config_path: Path | None = None) -> Settings:
    target = config_path or default_config_path()
    # ... (read + migrate the file shape if needed) ...
    prior = Settings._yaml_file_override
    Settings._yaml_file_override = target if target.exists() else None
    try:
        return Settings()
    finally:
        Settings._yaml_file_override = prior
```

And `settings_customise_sources` reads it:

```python
@classmethod
def settings_customise_sources(cls, settings_cls, init_settings, env_settings,
                                dotenv_settings, file_secret_settings):
    sources = [init_settings, env_settings, dotenv_settings]
    path = cls._yaml_file_override
    if path is not None and path.exists():
        sources.append(YamlConfigSettingsSource(settings_cls, yaml_file=path))
    sources.append(file_secret_settings)
    return tuple(sources)
```

**MUST NOT:** load YAML by passing values as init kwargs to
`Settings(...)`. That makes them top-precedence and inverts CF-02
(env should win over YAML).

### 3.2 Wiring at the entry point

```python
# src/<app>/cli/__main__.py
@click.group(...)
@click.option("--config", "config_path", type=click.Path(path_type=Path), default=None)
@click.option("--log-level", default=None)
@click.pass_context
def main(ctx, config_path: Path | None, log_level: str | None) -> None:
    overrides: dict = {}
    if log_level is not None:
        overrides["logging"] = {"level": log_level.upper()}

    settings = load_settings(config_path=config_path, overrides=overrides)

    setup_logging(
        level=settings.logging.level,
        fmt=settings.logging.format,
        log_file=settings.logging.file_path,
        enable_otlp=settings.logging.enable_otlp,
    )

    ctx.ensure_object(dict)
    ctx.obj["settings"] = settings
    # provider, history, vault factories etc. — all take `settings`.
```

---

## 4. Sub-config files (rule CF-07)

Sub-config files are *not* the same as the main `Settings` — they have their own loaders because they are typically **content** (rules, scripts, schedules) rather than **settings**. But they MUST follow the same shape.

### 4.1 The shape every satellite file follows

| Aspect | Requirement |
|---|---|
| Schema | A typed pydantic model (or frozen dataclass for very simple cases) |
| Version field | Yes, top-level `version: int` |
| Loader | `<Type>.from_yaml_file(path) -> <Type>` |
| Saver | `<Type>.save_to_file(path)` (atomic write, mode 0o600) |
| Missing file | Returns an empty default silently if appropriate (e.g., "no hooks defined") |
| Bad YAML | Raises `<App>SpecError` with file path + cause |
| Unknown key | Raises `<App>SpecError` with the offending key name |
| Bad version | Raises `<App>SpecError` (forward-incompatible) or runs migration (backward-compatible) |
| Validation errors | Aggregated — all errors reported in one exception (`SchemaValidationError(errors)`) |

### 4.2 Drop-in template — sub-config loader

```python
# src/<app>/scheduler/store.py — example shape every satellite uses
from __future__ import annotations
from pathlib import Path
from typing import Any
import yaml
from pydantic import BaseModel, ConfigDict, Field, ValidationError

from <app>.exceptions import SchemaValidationError

CURRENT_SCHEDULES_VERSION = 1

class ScheduleEntry(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)

    name: str
    cron: str
    action: str
    target: str
    enabled: bool = True

class ScheduleStore(BaseModel):
    model_config = ConfigDict(extra="forbid")

    version: int = CURRENT_SCHEDULES_VERSION
    entries: list[ScheduleEntry] = Field(default_factory=list)

    @classmethod
    def from_yaml_file(cls, path: Path) -> "ScheduleStore":
        if not path.exists():
            return cls()  # empty default (rule CF-07 missing-file)
        try:
            raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
        except yaml.YAMLError as e:
            raise SchemaValidationError([f"{path}: invalid YAML: {e}"]) from e
        if not isinstance(raw, dict):
            raise SchemaValidationError([f"{path}: top level must be a mapping"])

        version = raw.get("version", CURRENT_SCHEDULES_VERSION)
        if isinstance(version, int) and version > CURRENT_SCHEDULES_VERSION:
            raise SchemaValidationError([
                f"{path}: schedules version {version} is newer than supported "
                f"version {CURRENT_SCHEDULES_VERSION}; upgrade <app>"
            ])

        try:
            return cls.model_validate(raw)
        except ValidationError as e:
            errors = [f"{path}: {err['loc']}: {err['msg']}" for err in e.errors()]
            raise SchemaValidationError(errors) from e

    def save_to_file(self, path: Path) -> None:
        """Atomic write (rule CF-04)."""
        path.parent.mkdir(parents=True, exist_ok=True)
        payload = self.model_dump(mode="json")
        payload["version"] = CURRENT_SCHEDULES_VERSION

        tmp = path.with_suffix(path.suffix + ".tmp")
        try:
            tmp.write_text(yaml.safe_dump(payload, sort_keys=False), encoding="utf-8")
            tmp.chmod(0o600)
            tmp.replace(path)
        except Exception:
            tmp.unlink(missing_ok=True)
            raise
```

**MUSTs:**
- Every satellite loader MUST aggregate errors — never raise on the first one.
- Every saver MUST be atomic (`tmp.replace(path)`).
- The path MUST come from `Settings`, not from a hardcoded constant. (Tests override `Settings.<sub>_file` for isolation.)

**Common consumer-side pitfall.** Satellite-file stores (e.g.,
schedules, alerts, hooks, lab specs, profiles) often start
life as stdlib `dataclass` + manual validation and only later
migrate to pydantic. That's compliant *if and only if* the
atomic-write MUST holds — the most common gap is a satellite
store using plain `write_text` rather than temp+rename. Audit
every satellite save path for atomicity before claiming L1.

---

## 5. The Vault — secrets boundary (rule CF-05)

Secrets MUST NOT live in `config.yaml`. The boundary:

| Storage | What goes there |
|---|---|
| `config.yaml` | Non-secret settings, paths, intervals, feature flags, schema version |
| `<state>/credentials.yaml` (encrypted) or system keyring | Passwords, tokens, private keys, API keys |
| `.env` (local dev only — `.gitignore`d) | Per-developer overrides; SHOULD NOT contain secrets in production deploys |
| `secrets/` files (read by pydantic-settings via `file_secret_settings`) | Container/Kubernetes-style file-mounted secrets |

### 5.1 The Vault interface

```python
# src/<app>/vault/__init__.py
from __future__ import annotations
from abc import ABC, abstractmethod
from dataclasses import dataclass

from <app>.exceptions import VaultError, KeyringUnavailableError

@dataclass(frozen=True)
class Credential:
    """Opaque credential object — never logged, never serialized to YAML."""
    username: str
    secret: str  # password or private key text

    def __repr__(self) -> str:
        return f"Credential(username={self.username!r}, secret=[REDACTED])"

class Vault(ABC):
    """The credential storage interface.

    Implementations:
      - SystemKeyringVault — libsecret / Keychain / Credential Manager
      - EncryptedFileVault — Fernet AES-128 fallback
    """

    @abstractmethod
    def get(self, key: str) -> Credential: ...

    @abstractmethod
    def set(self, key: str, credential: Credential) -> None: ...

    @abstractmethod
    def delete(self, key: str) -> None: ...

    @property
    @abstractmethod
    def backend(self) -> str:  # "keyring" | "file"
        ...
```

**MUSTs:**
- `Credential.__repr__` MUST redact `secret`.
- The `Vault` instance MUST be constructed from `Settings` paths and injected — same rule as the provider (CF-08).
- The active backend MUST be queryable (`vault.backend`) and SHOULD be logged at startup (rule **DP-10** no silent fallbacks).

**AS-IS in `mgf-common`.** `src/mgf/common/vault/` ships the
Vault registry with `keyring` (libsecret / Keychain / Credential
Manager), encrypted-file (`Fernet`), and env-var backends, all
selectable per deployment. The active backend is queryable and
logged at construction time per **DP-10**.

---

## 6. CLI flag overrides

Every project ships a small standard set of flags that override the corresponding `Settings` field. These are the only places the CLI **overrides** config:

| Flag | Field | Purpose |
|---|---|---|
| `--config <path>` | (file path) | Use this file instead of the default |
| `--log-level <level>` | `logging.level` | Bump verbosity for this run |
| `--log-format <text\|json\|auto>` | `logging.format` | Force structured output for piping |
| `--no-color` | (TextFormatter) | Disable ANSI colours (also honours `NO_COLOR` env) |

Wire them via the `overrides` parameter of `load_settings()`:

```python
@click.group(...)
@click.option("--log-level", default=None, type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR"]))
@click.pass_context
def main(ctx, log_level: str | None) -> None:
    overrides: dict = {}
    if log_level is not None:
        overrides.setdefault("logging", {})["level"] = log_level
    ctx.obj["settings"] = load_settings(overrides=overrides)
```

**MUSTs:**
- A CLI flag MUST NOT bypass the `Settings` object. It MUST always feed into `overrides`.
- Adding a new CLI flag without a corresponding `Settings` field is a defect (the field is the source of truth; the flag is just a way to set it).

**Web-shape note (CF-09).** Web services (Django, FastAPI, Flask,
Pyramid) consume config from environment variables + an optional
`.env` file at the project root; they do NOT take a `--config` CLI
flag because ASGI / WSGI processes don't run with consumer-facing
argv. CF-09 is **N/A** under the framework-native carve-out — see
[§0a "Framework-native config"](#0a-framework-native-config-django-rails-equivalent)
above and [`web/README.md`](web/README.md) for the cross-domain
narrative.

---

## 7. Environment variables

`pydantic-settings` reads env vars natively. The convention:

- **Prefix** = `<APP>_` (uppercase). Example: `<APP>_BACKEND_URI`.
- **Nested fields** use the configured `env_nested_delimiter` (we use `__`). Example: `<APP>_LOGGING__LEVEL=DEBUG`.
- **Booleans** accept `1/0`, `true/false`, `yes/no` (case-insensitive).
- **Lists** are comma-separated (`<APP>_LOGGING__REDACT_EXTRA_KEYS=foo,bar,baz`) or JSON (`'["foo","bar"]'`).

**MUSTs:**
- Env vars MUST follow the prefix convention. No bare `LOG_LEVEL` — too prone to collision with other apps.
- Sensitive env vars (`<APP>_DB_PASSWORD`, `<APP>_API_TOKEN`) MUST go through the Vault, not into `Settings`.
- `.env` files MUST be `.gitignore`d. A `.env.example` SHOULD ship with the project showing the available knobs (without real values).

---

## 8. Schema versioning + migration (rule CF-03)

When the schema changes, you have two paths:

### 8.1 Backward-compatible (most common)

Add a new field with a default. Older configs load fine because the field is missing → default applies. No version bump needed.

```python
# Before
class Settings(BaseSettings):
    version: int = 1
    backend_uri: str = "qemu:///system"

# After (no version bump)
class Settings(BaseSettings):
    version: int = 1
    backend_uri: str = "qemu:///system"
    new_optional_field: int = 30  # default → backward-compat
```

### 8.2 Breaking change (rare)

Bump `CURRENT_SCHEMA_VERSION`. Add a migration step. Older configs load → migration runs → file is rewritten on disk.

```python
CURRENT_SCHEMA_VERSION = 2

def _migrate(raw: dict[str, Any], *, from_version: int) -> dict[str, Any]:
    if from_version == 1:
        # v1 had `timeout_ms`; v2 split into ssh_timeout_seconds + http_timeout_seconds
        old = raw.pop("timeout_ms", 30_000)
        raw["ssh_timeout_seconds"] = old // 1000
        raw["http_timeout_seconds"] = old // 1000
        raw["version"] = 2
        from_version = 2
    return raw
```

**MUSTs:**
- Loading a higher version than known MUST raise `AppConfigError` with a message telling the user to upgrade the app.
- Loading a lower version MUST run migration silently and rewrite the file with the migrated content.
- A migration step MUST be idempotent (running it twice produces the same result).
- A unit test MUST exercise every migration step end-to-end (load v(N-1) → assert fields after migration).

### 8.3 Accepting files that predate the `version` field

When you **add** the `version:` field to an existing sub-config
format (i.e., you are tightening CF-07 on a file format users are
already writing), every user-authored file on disk predates the
rule. Three choices:

1. **Fail loud.** Refuse to load any file without `version:`.
   Worst UX — users who upgrade the app see every existing
   `profile.yaml` / `hooks.yaml` / `schedules.yaml` reject on
   first run.
2. **Silent default.** Treat missing `version:` as the current
   version with no signal. Best UX but costs you a diagnostic —
   users who later hit a migration bug have no way to tell
   whether their file was ever versioned.
3. **WARN and default** (recommended). Log a WARNING that names
   the file + the current version, treat the load as
   `CURRENT_VERSION`, then rewrite the file with `version:` at
   the top on the next save.

```python
@classmethod
def from_dict(cls, raw: dict[str, Any]) -> MyConfig:
    # ... unknown-key rejection ...

    raw_version = raw.get("version")
    if raw_version is None:
        LOG.warning(
            "config %r: missing 'version' field — treating as v%d. "
            "Re-save the file to add 'version: %d' at the top.",
            raw.get("name", "<unknown>"),
            CURRENT_VERSION,
            CURRENT_VERSION,
        )
        version = CURRENT_VERSION
    elif isinstance(raw_version, bool) or not isinstance(raw_version, int):
        # bool is a subclass of int in Python — treat it as a
        # type error rather than silently accepting True as 1.
        raise SchemaValidationError(
            [f"version must be an int, got {type(raw_version).__name__}"],
        )
    else:
        version = raw_version

    if version > CURRENT_VERSION:
        raise SchemaValidationError(
            [f"version {version} is not supported; expected <= {CURRENT_VERSION}"],
        )
    if version < CURRENT_VERSION:
        raw = _migrate(raw, from_version=version)

    # ... rest of parsing ...
```

**MUSTs for this path:**
- The WARNING MUST carry the file identifier (path or name) so the
  user can find the right file to fix.
- The next `save_*` call MUST emit `version: <CURRENT_VERSION>` so
  re-saving a legacy file silently upgrades it on disk.
- A unit test MUST pin the "missing-version accepted with WARNING"
  behaviour — otherwise a future PR that switches to "fail loud"
  silently breaks every existing user's install on upgrade.
- `bool` MUST be rejected as a version value even though it is an
  `int` subclass in Python — `True` / `False` in the YAML is
  almost always a typo.

The pattern is idempotent: once a legacy file is re-saved, the
WARNING stops firing and the file is indistinguishable from a
natively-versioned one.

---

### 8.4 Serialising nested sub-models — prefer `model_dump(mode="json")`

`mgf.common.config.BaseAppSettings.to_yaml_dict()` is **shallow**
— it iterates `model_fields` and converts only `Path` instances
explicitly, returning every other value as-is. Nested
`BaseModel` sub-fields (`LoggingSettings`, the discriminated
`VaultConfig` from §14, etc.) emerge as Python objects that
`yaml.safe_dump` cannot represent. Calling `save_settings(cfg)`
on a `BaseAppSettings` subclass with nested sub-models therefore
raises `yaml.representer.RepresenterError`.

This is a known limitation of the helper. Two paths:

1. **Use `model_dump(mode="json")` directly** (the recommended
   shape per `to_yaml_dict`'s own docstring). Then write
   atomically via the canonical temp + rename pattern:

   ```python
   payload = settings.model_dump(mode="json")
   payload["version"] = type(settings).CURRENT_VERSION
   tmp = path.with_suffix(path.suffix + ".tmp")
   tmp.write_text(yaml.safe_dump(payload, sort_keys=False))
   tmp.chmod(0o600)
   tmp.replace(path)
   ```

2. **Stay flat** until `to_yaml_dict` recurses. Flat schemas
   (every field a scalar / Path / `Literal`) work with
   `save_settings` directly. The discriminated-union pattern
   (§14) also depends on nested models, so this only buys time.

**Forward path:** a future `mgf-common` release SHOULD make
`to_yaml_dict` use `model_dump(mode="json")` internally so
`save_settings` works for any pydantic model shape. Tracked in
[`FEEDBACK.md`](../../FEEDBACK.md).

---

## 9. Atomic writes (rule CF-04)

**MUSTs:**
- Any state file the app owns MUST be written atomically: write
  to `<file>.tmp` in the same directory, then `tmp.replace(target)`.
- File mode MUST be `0o600` for any file that may carry private
  info.
- The body MUST be `fsync`'d before the rename, so a crash
  between write and rename leaves either the old file or no
  file — never a half-written one.
- The temp file MUST be cleaned up on exception (try/except
  around the write, `tmp.unlink(missing_ok=True)` in the except).
- For files that MUST survive a power loss (the vault, the
  config file), parent-directory `fsync` MUST also fire so the
  rename itself is durable.

This applies to `config.yaml`, every satellite YAML, the alert
state cursor, the scheduler tick cursor, the port-forward state
file, the encrypted vault file — **everything** the application
writes.

**Canonical helper (`mgf-common` ships it).** `mgf.common._io.atomic_write`
([`src/mgf/common/_io.py`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/src/mgf/common/_io.py)) is the
extracted helper covering this contract end-to-end:

```python
from mgf.common._io import atomic_write

atomic_write(path, payload, mode=0o600, fsync_dir=True)
```

Signature: `atomic_write(path: Path, content: str | bytes, *, mode: int = 0o600, fsync_dir: bool = True) -> None`.
Handles temp-file allocation in the target's directory, body
`fsync`, mode chmod, atomic `os.replace`, optional parent-dir
`fsync`, and cleanup on every error path. **Consumers SHOULD use
this helper rather than re-implementing the recipe**: three
historical implementations diverged on which subsidiary step
they handled (some skipped `fsync`, some used a global-temp
path that broke `os.replace`'s atomicity guarantee, some left
half-written `.tmp` files on exception). The shared helper is
the one verified contract.

Call sites in `mgf-common` core (`atomic_write` is the *only*
write recipe — pinned by `RA-01`):

- Config save: `mgf.common.config._loader::save_settings`
- Vault save: `mgf.common.vault.EncryptedFileVault::save`
- Crash report write: `mgf.common.observability.crash_reporter._write_local`
- Runtime snapshot: `mgf.common.cli._doctor_live::emit_runtime_snapshot`
  (mode `0o600` per `SC-04`)
- Codemod / catch-up consumer-source rewrite:
  `mgf.common.cli._migrate` / `_catch_up` (preserves the target
  file's existing mode rather than forcing `0o600`)

**Consumer guidance.** Most config-save / state-save call
sites in a typical consumer use a temp+rename pattern already.
The common gap is a satellite store using plain `write_text`
that skips the body `fsync`. The recommended remediation is to
call `mgf.common._io.atomic_write` rather than copy-paste the
recipe.

---

## 10. Lazy path expansion (rule CF-04)

```python
# WRONG — captures $HOME at module import time, breaks tests that monkeypatch HOME per test
DEFAULT_PATH = Path("~/.config/<app>/x.yaml").expanduser()

# RIGHT — resolves at call time
def _default_path() -> Path:
    return Path("~/.config/<app>/x.yaml").expanduser()
```

This rule applies *both* to module-level constants *and* to dataclass `default_factory`. Every default that depends on `$HOME` or `$XDG_*` MUST be wrapped in a function called at use time.

**Why:** unit tests monkeypatch `HOME` per test (`monkeypatch.setenv("HOME", str(tmp_path / "home"))`). Module-load expansion captures the parent process's `$HOME` once and breaks per-test isolation — every test ends up writing to the developer's real home directory.

**Cross-platform path composition.** `mgf.common.config._paths.platform_dir`
([`src/mgf/common/config/_paths.py`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/src/mgf/common/config/_paths.py))
is the canonical platform-aware path resolver. It returns:

- **Linux:** `$XDG_*_HOME` (with the appropriate fallback —
  `~/.config` / `~/.local/state` / `~/.cache`).
- **macOS:** `~/Library/Application Support/<app>` for state and
  config; `~/Library/Caches/<app>` for caches.
- **Windows:** `%LOCALAPPDATA%/<app>` or
  `<home>/AppData/Local/<app>` as the fallback.

Helpers that build app-state paths (e.g.,
`mgf.common.observability.paths.default_state_dir`) MUST compose
with `platform_dir` rather than hardcoding `~/.local/state`
(that was the historical pre-`RA-29` shape; the cross-platform
breakage on macOS / Windows was real and is now closed). The
rule is: if a default involves a directory on disk and the
target platform varies, route through `platform_dir`.

**Consumer pattern.** The rule applies to every subsystem
default that depends on `$HOME` or `$XDG_*`: helpers like
`_default_state_root`, `_default_db_path`, `_default_user_dir`,
`_default_vault_file` MUST all resolve lazily. Pin them with a
consolidated test (see §10.1).

### 10.1 Pinning the lazy-default pattern

When a project has many `_default_X()` helpers — one per
subsystem that caches something under `$HOME` — the regression
protection is best concentrated in a **single consolidated pin
test file**, not scattered across each subsystem's own test
module.

The reference shape: `tests/unit/test_default_path_resolution.py`
with one class `TestLazyHomeResolution` containing one test
per `_default_X()` helper, each following this template:

```python
def test_history_db_path_follows_home_monkeypatch(
    self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
    """_default_db_path() resolves HOME lazily."""
    fake_home = tmp_path / "alternate-home"
    monkeypatch.setenv("HOME", str(fake_home))
    resolved = _history._default_db_path()
    assert str(resolved).startswith(str(fake_home)), (
        f"_default_db_path() returned {resolved!r} — expected a path "
        f"under monkeypatched HOME ({fake_home!r})."
    )
```

**Why one file, not one per module:**
- Future readers see the full "lazy HOME default" pattern
  enforced in one visible place. Scattered tests make the rule
  feel like it applies only where the specific test happens to
  cover.
- Adding a new default-path helper in a new subsystem becomes a
  one-line-in-one-file obligation: add a row here. A new test
  file per subsystem invites the "I'll add the test later"
  anti-pattern.
- The failure mode under regression is legible — "the builder's
  default path isn't HOME-isolated" is easier to act on when the
  test is named `test_builder_state_root_follows_home_monkeypatch`
  in a file labelled `test_default_path_resolution.py`.

**Bonus:** add a second test class `TestBackwardsCompatAliases`
that asserts every public `DEFAULT_X` module-level constant is
still importable (for callers that grep for the legacy name).
Those constants still capture `$HOME` at module load, but keeping
them importable preserves source-compat. The pin test is your
safety net against someone removing the backwards-compat alias
without a deprecation cycle.

---

## 11. Testing config

**MUSTs:**
- Tests MUST construct `Settings` with explicit values:

```python
def test_some_thing(monkeypatch, tmp_path):
    # Don't read the developer's actual config
    monkeypatch.delenv("<APP>_CONFIG_PATH", raising=False)
    monkeypatch.setattr("<app>.settings._yaml_file_path", lambda: tmp_path / "x.yaml")
    settings = load_settings(overrides={"backend_uri": "mock://"})
    assert settings.backend_uri == "mock://"
```

- Tests MUST NOT assume the developer's local `.env` or `~/.config/<app>/config.yaml` exists.
- A test that needs a temporary config file MUST use `tmp_path` and pass it via `config_path=`.

**SHOULDs:**
- Property-test the round-trip: `Settings(...)` → `save_settings(s, path)` → `load_settings(config_path=path)` → equal.

---

## 12. Anti-patterns (forbidden)

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| `os.environ.get("X_THING")` in a use-case module | Config bypass; inconsistent | Add field to `Settings`; consume from there |
| `yaml.safe_load(open("config.yaml"))` outside the loader | CF-01 violation | `ctx.obj["settings"]` |
| Plain `path.write_text(yaml.dump(...))` for owned state | Non-atomic; partial write on crash | `tmp = path.with_suffix(...+".tmp"); ...; tmp.replace(path)` |
| `DEFAULT_PATH = Path("~/.x").expanduser()` at module top | Captures HOME at import | `def _default_path(): return Path("~/.x").expanduser()` |
| Storing a password/token in `config.yaml` | Plaintext secret in version control / logs | Vault |
| Swallowing `ValidationError` to "be lenient" | Silent typos; user thinks setting was applied | Re-raise as `SchemaValidationError(errors)` |
| Module-level `Settings()` singleton | Untestable; first-import wins | Constructor injection from entry point |
| Adding a new sub-config file with bespoke parsing | Drift from the standard shape | Use the §4.2 template |

---

## 13. Cross-references

- The `Vault` interface lives next door to `Settings` in `<app>/vault/`. See [`COMMON_COMPONENTS.md`](COMMON_COMPONENTS.md) for the catalogue entry.
- Every sub-config file's parse error MUST raise `SchemaValidationError` from [`ERROR_HANDLING.md`](ERROR_HANDLING.md) §2.3.
- Logging is configured *from* `Settings.logging` — see [`LOGGING.md`](LOGGING.md) §5.
- Secrets boundary, vault backend selection, no silent fallback (rule SC-01..SC-05): [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) §2.
- Test patterns for config (HOME isolation, schema migration, round-trip property tests): [`TESTING.md`](TESTING.md) §11.3 + §5.

---

## 14. Discriminated unions for backend variants (rule CF-12)

When a config field has multiple mutually-exclusive backend variants
— a vault that is keyring / file / KMS, a metrics sink that is
prometheus / OTLP / disabled, a queue backend that is sqlite /
postgres / redis — the wrong shape is a flat structure with every
backend's fields as siblings:

```python
# WRONG — flat shape, ambiguous
class VaultSettings(BaseSettings):
    backend: Literal["keyring", "file", "kms"]
    keyring_service: str = "<app>"
    file_path: Path | None = None
    file_passphrase_env: str | None = None
    kms_endpoint: str | None = None
    kms_key_id: str | None = None
```

Problems:
- Most fields are `None` for any given backend; the type system
  doesn't enforce which.
- Adding a new backend touches the same flat class; the file grows
  without bound.
- A typo (`backend = "kerying"`) silently picks defaults from the
  wrong variant.
- Validation can't say "this field is required for the file
  backend" without an after-validator full of `if/elif`.

The right shape is a **discriminated union** with a `kind` field:

```python
# RIGHT — variants typed independently, discriminator picks
from typing import Literal, Annotated, Union

from pydantic import BaseModel, ConfigDict, Field

class KeyringVaultConfig(BaseModel):
    model_config = ConfigDict(extra="forbid")
    kind: Literal["keyring"] = "keyring"
    service: str = "<app>"

class FileVaultConfig(BaseModel):
    model_config = ConfigDict(extra="forbid")
    kind: Literal["file"] = "file"
    path: Path
    passphrase_env: str = "<APP>_VAULT_PASSPHRASE"

class KMSVaultConfig(BaseModel):
    model_config = ConfigDict(extra="forbid")
    kind: Literal["kms"] = "kms"
    endpoint: str
    key_id: str

VaultConfig = Annotated[
    Union[KeyringVaultConfig, FileVaultConfig, KMSVaultConfig],
    Field(discriminator="kind"),
]

class Settings(BaseAppSettings):
    vault: VaultConfig = KeyringVaultConfig()
```

In YAML:

```yaml
vault:
  kind: file
  path: ~/.local/share/myapp/vault.enc
  passphrase_env: MYAPP_VAULT_PASSPHRASE
```

Pydantic uses `kind` to pick the right model class, validates the
backend's required fields, and rejects fields that belong to a
different backend. `extra="forbid"` makes the rejection loud
(rule **CF-06**).

### 14.1 Adding a new backend

A new backend is a new class plus one entry in the union — the
rest of the codebase is untouched:

```python
class HSMVaultConfig(BaseModel):
    model_config = ConfigDict(extra="forbid")
    kind: Literal["hsm"] = "hsm"
    pkcs11_lib: Path
    slot_id: int

VaultConfig = Annotated[
    Union[KeyringVaultConfig, FileVaultConfig, KMSVaultConfig, HSMVaultConfig],
    Field(discriminator="kind"),
]
```

The factory that builds the runtime `Vault` from `Settings.vault`
matches on `kind` and constructs the right impl:

```python
def build_vault(cfg: VaultConfig) -> Vault:
    match cfg:
        case KeyringVaultConfig(service=s):
            return SystemKeyringVault(service=s)
        case FileVaultConfig(path=p, passphrase_env=env):
            return EncryptedFileVault(path=p, passphrase=os.environ[env])
        case KMSVaultConfig(endpoint=e, key_id=k):
            return CloudKMSVault(endpoint=e, key_id=k)
        case HSMVaultConfig(pkcs11_lib=lib, slot_id=s):
            return HSMVault(pkcs11_lib=lib, slot_id=s)
```

The `match` exhaustiveness is checked by mypy — adding a new
variant without updating the factory fails CI.

### 14.2 When NOT to use

A discriminated union is overkill when:
- There's only one variant (it's just a struct).
- Variants share most of their fields and differ only in one
  scalar (a single `Literal` field is simpler).
- The "variant" is really a future plan, not present-day requirement
  (YAGNI; introduce when the second variant exists).

---

## 15. Environment-tier layering (dev / staging / prod)

Most projects distinguish at least two environments — local
development and production. The standard's layered precedence
(rule **CF-02**) handles per-developer overrides via `.env`, but
team-wide environment tiers need a tier-aware loader.

### 15.1 Tier as a first-class concept

The active tier is a single environment variable (`<APP>_ENV`)
read at the entry point. Acceptable values: `dev`, `staging`,
`prod`, plus any custom tier the project needs (`ci`, `local`).

### 15.2 The tier-aware load order

For each tier, `load_settings` looks for a tier-specific YAML in
addition to the base config:

```
~/.config/<app>/config.yaml ← base
~/.config/<app>/config.<tier>.yaml ← tier overlay
.env, env vars, CLI flags ← per CF-02
```

Layered precedence (highest wins): CLI > env > `.env` > tier YAML
> base YAML > defaults. The tier YAML overrides the base; both are
checked into git for team-wide consistency. Per-developer secrets
go into `.env` (not committed).

### 15.3 Implementation

```python
def load_settings(
    settings_cls: type[T],
    *,
    path: Path | None = None,
    tier: str | None = None,
) -> T:
    base = path or default_config_path()
    tier = tier or os.environ.get("<APP>_ENV", "")
    tier_path = base.with_name(base.stem + f".{tier}" + base.suffix) if tier else None

    raw: dict[str, Any] = {}
    if base.exists():
        raw = _read_yaml(base)
    if tier_path and tier_path.exists():
        overlay = _read_yaml(tier_path)
        raw = _deep_merge(raw, overlay)

    # ... migration, validation, etc.
    return settings_cls.model_validate(raw)
```

`_deep_merge(base, overlay)` recurses into nested dicts; lists are
replaced wholesale (an "append" semantic invites surprises).

### 15.4 Tier-MUST conventions

| Tier | What MUST be true |
|---|---|
| `dev` | No real credentials; mocks where possible; full debug logging |
| `staging` | Production shape with reduced scale; test credentials only |
| `prod` | All credentials from vault; no debug logging by default; OTLP enabled |

Code MUST NOT branch on the tier value:

```python
# WRONG
if settings.tier == "prod":
    enable_security_features()

# RIGHT — security features on by default; tier-specific behaviour
# is expressed through config values, not code branches
enable_security_features(level=settings.security.level)
```

The principle: tiers are config layers, not feature flags. Code
that needs different behaviour reads a config value; the tier
files set the right value for each tier.

### 15.5 CI environments

CI loads `<APP>_ENV=ci` (or `test`) and a `config.ci.yaml` that
points every external endpoint at a mock or in-memory backend.
Tests MUST NOT need a tier file — they construct `Settings`
explicitly per rule **CF-10** and bypass the loader entirely.

---

## 16. Field aliases for renames-without-breaks (rule CF-11)

A field rename (`old_name` → `new_name`) breaks every existing
on-disk config that uses the old name. Pydantic's
`Field(validation_alias=...)` accepts both names while the schema
holds only the new one:

```python
from pydantic import AliasChoices, BaseSettings, Field

class Settings(BaseAppSettings):
    # New canonical name; legacy `connect_timeout_s` accepted for read.
    connection_timeout_seconds: int = Field(
        default=30,
        validation_alias=AliasChoices(
            "connection_timeout_seconds",  # new
            "connect_timeout_s",           # legacy
        ),
    )
```

A YAML with the old key loads cleanly:

```yaml
connect_timeout_s: 60 # OK — accepted via alias
```

### 16.1 The deprecation warning

The first time a legacy alias is used in a process, log a
DEPRECATION WARNING with the file identifier and the new name:

```python
@model_validator(mode="before")
@classmethod
def _warn_on_legacy_aliases(cls, values: dict[str, Any]) -> dict[str, Any]:
    legacy_map = {
        "connect_timeout_s": "connection_timeout_seconds",
        "log_format": "logging.format",
    }
    for legacy, current in legacy_map.items():
        if legacy in values:
            LOG.warning(
                "config field %r is deprecated; use %r. "
                "The legacy name will be removed in v3.0.",
                legacy, current,
            )
    return values
```

### 16.2 The rewrite

`save_settings` writes the canonical form. After the next save, the
on-disk file uses the new name and the warning stops firing.

### 16.3 The deprecation timeline

| Step | When | What |
|---|---|---|
| Add alias + warn | release N | Both names accepted; old name warns |
| Document removal date | release N+1 |notes the removal target |
| Remove alias | release N+2 (≥ 6 months after N) | Old name now an unknown-key error (rule **CF-06**) |

For breaking config changes that cannot be aliased (a field type
narrowing, e.g., `int` → `Literal[1, 2, 3]`), bump the schema
`version` and migrate (rule **CF-03**).

### 16.4 Anti-patterns

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| Rename without alias | Existing configs fail to load on upgrade | `validation_alias=AliasChoices(new, legacy)` |
| `if "old_name" in raw: raw["new_name"] = raw.pop("old_name")` in the loader | Every reader needs to know about the migration | Pydantic alias does it once, declaratively |
| Permanent alias (never removed) | Schema accumulates dead names; cognitive load | Plan removal in the deprecation timeline |
| Aliasing a field that changed semantics | Loads, but means something different now | Use a different name AND migrate via CF-03 |

---

*End of `CONFIGURATION.md`.*
