# Standing up the self-hosted Woodpecker CI host — an illustrated walkthrough

> **What this is.** A step-by-step, screenshot-backed record of how the federation's
> Woodpecker CI host was actually built on Infomaniak Public Cloud — so the next
> engineer can reproduce it with **no surprises**. It complements the concise
> operational runbook in [`README.md`](README.md); this doc is the teaching version.
>
> **Result:** a public, TLS-terminated Woodpecker instance, Codeberg-OAuth login,
> running on one Infomaniak VM — built end-to-end in ~15 minutes (most of it the
> two browser steps). The actual run is captured below, gotchas and all.

Real values from the reference build (yours will differ): host IP `195.15.241.48`,
domain `195.15.241.48.sslip.io`, region `dc3-a`, project `PCP-HOESFJV`.

---

## 0. Prerequisites

- An **Infomaniak Public Cloud** project with a working `clouds.yaml` (see
  `mgf-cloud`'s onboarding recipe). Billing is **on** — this host runs 24/7.
- `mgf-cloud[infomaniak]` installed (the `mgf-fed[cloud]` extra), so
  `mgf-fed provision-ci-host` works.
- A **Codeberg account** that owns (or can admin) the `magogi-admin` org.

---

## 1. Provision the host (one command)

`mgf-fed provision-ci-host` creates a security group (22/80/443), a VM with a
Docker-installing cloud-init that drops this `deploy/woodpecker/` bundle into
`/opt/woodpecker/`, and gives it a public IP. **Secrets are NOT baked into
user-data** (the metadata service can expose it) — the `.env` is added afterward.

First register an SSH keypair so you can reach the host later:

```sh
ssh-keygen -t ed25519 -f my_stuff/infomaniak/woodpecker_id_ed25519 -N ''
# register the public half in OpenStack as "woodpecker-key" (mgf-cloud keypair create, or the API)
```

Then provision (2 vCPU / 4 GB / 40 GB boot volume — a comfortable CI host):

```sh
mgf-fed provision-ci-host \
  --hostname woodpecker --env prod \
  --flavor a2-ram4-disk20-perf1 --volume-size 40 \
  --key-name woodpecker-key \
  --clouds-yaml my_stuff/infomaniak/PCU-HOESFJV-clouds.yaml \
  --cloud PCP-HOESFJV-dc3-a \
  --yes
```

It prints the public IP (here `195.15.241.48`) and the next-steps checklist.
`--dry-run` plans without touching the cloud.

---

## 2. See the host in Infomaniak Horizon

Manager → your Public Cloud project → **Access Horizon** → **Compute → Instances**.

### ⚠️ Gotcha #1 — Horizon defaults to the wrong region

Horizon scopes to **one region at a time**, and it often opens on `dc4-a` (the DR
region) — where you'll see nothing:

![Horizon showing an empty instance list because the region is dc4-a](images/01-horizon-wrong-region.png)

**Fix:** click the region dropdown at the **top-left** (`PCP-… • dc4-a ▾`) and
switch to **`dc3-a`** (where the host was provisioned). Now it appears, and the
overview confirms everything — flavour, the `ext-net1` public IPv4+IPv6, the
`woodpecker-prod-ci` security group (22/80/443), the injected key, and the
`mgf:owner=mgf-fed` tag:

![Horizon instance overview for woodpecker-prod](images/02-horizon-instance-overview.png)

### Optional — the VNC console (emergencies only)

The instance **Console** tab gives a browser screen into the VM. Click **Connect**:

![Horizon noVNC console with a Connect button](images/03-horizon-console-connect.png)

…and you'll see the OS is up. **You can't log in here** — cloud images are
**SSH-key-only**, so the `ubuntu` user has no password and this prompt just rejects
input. It's a viewer for when SSH is broken, not your normal access path:

![VM console showing the Ubuntu login prompt](images/04-horizon-console-login.png)

**Your real access is SSH:**
`ssh -i my_stuff/infomaniak/woodpecker_id_ed25519 ubuntu@195.15.241.48`

---

## 3. Pick a domain

Caddy needs a domain to obtain a Let's Encrypt TLS cert, and Codeberg's OAuth
redirect must point at it.

- **Fastest (no DNS setup):** use **`<IP>.sslip.io`** — it auto-resolves to your IP.
  We used `195.15.241.48.sslip.io`. Let's Encrypt issues for it fine.
- **Proper subdomain:** add a DNS **A record** `woodpecker.<yourdomain>` → the IP.

Lock it into the host's `.env` (the agent secret + admin are pre-filled by the
provisioner):

```sh
ssh -i …/woodpecker_id_ed25519 ubuntu@195.15.241.48
sudo sed -i 's|^WOODPECKER_HOST=.*|WOODPECKER_HOST=https://195.15.241.48.sslip.io|' /opt/woodpecker/.env
sudo sed -i 's|^WOODPECKER_DOMAIN=.*|WOODPECKER_DOMAIN=195.15.241.48.sslip.io|' /opt/woodpecker/.env
```

---

## 4. Create the Codeberg OAuth2 app

Codeberg → **Settings → Applications** (direct:
`https://codeberg.org/user/settings/applications`) → **Create a new OAuth2
Application**:

- **Application Name:** `MGF Woodpecker CI`
- **Redirect URI:** `https://<your-domain>/authorize`
  (here `https://195.15.241.48.sslip.io/authorize` — must match exactly).

### ⚠️ Gotcha #2 — the app MUST be a "Confidential Client"

If the app is registered as a **public** client, login fails with:

> `invalid_request — PKCE is required for public clients`

![Woodpecker login page showing the PKCE invalid_request error](images/05-woodpecker-pkce-error.png)

Woodpecker is a **server-side (confidential) client** — it authenticates with the
secret and doesn't send a PKCE challenge. **Fix:** edit the Codeberg app and **check
the ☑ "Confidential Client" box** (editing it keeps the same Client ID + Secret).
If your Forgejo version only sets it at creation, delete and recreate with it
checked.

Copy the **Client ID** and **Client Secret**, and put them in the host's `.env`:

```sh
sudo sed -i 's|^WOODPECKER_FORGEJO_CLIENT=.*|WOODPECKER_FORGEJO_CLIENT=<client-id>|' /opt/woodpecker/.env
sudo sed -i 's|^WOODPECKER_FORGEJO_SECRET=.*|WOODPECKER_FORGEJO_SECRET=<client-secret>|' /opt/woodpecker/.env
```

---

## 5. Bring the stack up

```sh
cd /opt/woodpecker && sudo docker compose up -d
```

This pulls `woodpecker-server`, `woodpecker-agent`, and `caddy`. Within ~30–60 s
Caddy obtains the Let's Encrypt cert (it needs port 80 reachable for the HTTP-01
challenge — the security group + ufw already allow it). Verify:

```sh
sudo docker compose ps          # all three Up / healthy
curl -I https://<your-domain>/  # HTTP/2 200, valid TLS
```

A quick way to confirm the OAuth wiring end-to-end: `curl -i https://<domain>/authorize`
should `302` to `codeberg.org/login/oauth/authorize?client_id=…&redirect_uri=…`
with **your** client_id and redirect — both matching the app.

### Install the agent self-heal cron (required)

Under load the Docker daemon can time out tearing down an agent's step-containers,
leaking `wp_*` networks/volumes; the next pipeline then dies on `network already
exists` and `worker_count` drifts to 0 (the "wedge", root-caused 2026-06-20).
`self-heal.sh` (shipped in this bundle) reaps the leaked debris every 5 min and
restarts the agent only when it's wedged (stuck `Created` containers + nothing
running — it can never kill a live pipeline). Install it:

```sh
sudo install -m 0750 self-heal.sh /opt/woodpecker/self-heal.sh
# root cron, every 5 minutes (/etc/cron.d format includes the user field)
echo '*/5 * * * * root /opt/woodpecker/self-heal.sh' | sudo tee /etc/cron.d/woodpecker-self-heal
sudo chmod 0644 /etc/cron.d/woodpecker-self-heal
# confirm it runs (writes a heartbeat line)
sudo /opt/woodpecker/self-heal.sh && sudo tail -1 /var/log/woodpecker-self-heal.log
```

---

## 6. Log in + authorize

Open `https://<your-domain>` → **Login with codeberg.org**. Codeberg asks for
consent:

![Codeberg authorization consent for MGF Woodpecker CI](images/06-codeberg-authorize.png)

Click **Authorize Application** (grant the `magogi-admin` org when prompted).
You land in Woodpecker as the admin, on the **Repositories** page — empty until you
enable repos:

![Woodpecker Repositories page after login](images/07-woodpecker-repositories.png)

---

## 7. Enable repositories (finish WF-11)

Click **+ Add repository** and enable the `mgf-*` repos you want CI on (enabling a
repo creates its webhook, so push/PR events start triggering the generated
`.woodpecker.yml`).

**Gate-only caveat:** the generated pipelines have `event: tag` publish steps that
need secrets. Until you add those secrets, set each repo to build on **push +
pull_request only** (repo settings in Woodpecker) so the publish steps don't
run-and-fail. `mgf-fed release-remote` keeps doing the publishing.

Then verify a green run on a pilot (`mgf-common` + `mgf-alembic`), and add Codeberg
branch protection requiring the CI status check (WF-02).

---

## Gotchas, condensed

| # | Symptom | Cause → Fix |
|---|---------|-------------|
| 1 | Horizon instance list empty | Wrong region — switch `dc4-a` → `dc3-a` (top-left dropdown) |
| 2 | VNC console rejects your password | Cloud images are SSH-key-only; use SSH, not the console |
| 3 | `PKCE is required for public clients` | OAuth app is public — check **Confidential Client** on the Codeberg app |
| 4 | TLS cert never issues | Port 80 must be reachable (HTTP-01); confirm the secgroup + ufw allow 80 |
| 5 | `event: tag` steps fail | No publish secrets yet — set repos to push + pull_request only |
| 6 | every build's install fails (`No matching distribution found for mgf-standard`, pip 401/EOF) | CI can't read the **private** Codeberg PyPI registry — add `/opt/woodpecker/netrc` with a read:package token (mounted via the agent's `WOODPECKER_BACKEND_DOCKER_VOLUMES`). See README "Private registry auth". `WOODPECKER_ENVIRONMENT` can't do it (its `KEY:VALUE` parser mangles the URL's `:`). |
| 7 | a timing/bench test times out; vuln-scan flags a CI-tooling CVE; gitleaks flags `.claude/` placeholders | First-run findings, not infra — perf-mark the bench, `[tool.mgf-fed] vuln_ignore`, allowlist `.claude/` in `.gitleaks.toml` (see mgf-common PR #42, mgf-standard #127). |
| 8 | builds stop running; agent `worker_count` = 0; `wp_* network already exists` | Daemon-timeout wedge — the `self-heal.sh` cron (§5) reaps leaked `wp_*` debris + restarts the wedged agent. If absent, install it; to force a recovery now, run `sudo /opt/woodpecker/self-heal.sh`. |

## Upkeep

Pin + update the `woodpecker-server`/`agent`/`caddy` image tags (security), watch
disk (Docker build volumes), and treat the host as **federation-critical infra**.
The **agent self-heal cron** (`self-heal.sh`, installed in §5) reaps leaked `wp_*`
Docker debris every 5 min and un-wedges a stalled agent — keep it installed; its
heartbeat/actions log to `/var/log/woodpecker-self-heal.log`.
Tear down with `mgf-cloud vm delete woodpecker-prod --yes` (then delete its boot
volume — it's `delete_on_termination=False`). The fallback to self-hosting is
applying for Codeberg's shared CI access.
