Drill log — Compose / OSS track · AI Initiative review

Drill log — Compose / OSS track

This is the running log of operational drills against the Compose stack,
per D-007 of PHASE_5_HANDOFF.md and the procedures in runbook.md.

The AWS-track drills (D-007-AWS) live in a separate log when that path
is applied; the runbook procedures are largely shared but the commands
differ (e.g., docker stop vs aws ecs update-service).

Posture

For the internal / research stage, drills fall into three buckets:

Bucket	What it means	Examples here
Validated	drill ran end-to-end, recorded outcome	`data-backup`, `data-restore`
Scripted, runnable	quick scripts checked in under `drills/`; safe to run; outcome recorded once	`peer-collapse`, `liveness`, `readiness`, `charter-refresh`
Doc-only until production trigger	spec'd in `runbook.md` but no automated drill yet because the failure mode requires plumbing not relevant at this stage	`verifier-failures`, `rate-limit`, `ledger-quarantine`, `rollback`, `key-compromise`

Production / AWS deploys flip several "doc-only" entries into "scripted,
runnable" once the target environment exists. Don't wait — close them out
the same week the AWS path goes live.

Validated

data-backup — 2026-05-06

Procedure: trigger a one-shot backup, list snapshots in restic.

Driver: Session 4 (D-005-OSS) live test.

Result: snapshots 577fc3b7, 160e695d, 2f25ecd2, b52c6c71
created via docker exec manthan-backup /opt/backup/backup.sh. Each
captured all three SQLite ledgers via .backup API + Alertmanager state

Grafana state + .env + Charter / LOTS / privacy-policy markdown.

Repo has been re-tested against demo-server SFTP (<backup-host-private-ip>) — repo
materialised at ~/manthan-backups/repo on demo-server with the canonical
restic layout (config / data / index / keys / locks / snapshots).

data-restore — 2026-05-06

Procedure: insert marker into alpha ledger → backup → wipe alpha-data
volume → restore from snapshot → restart alpha → verify marker present

PRAGMA quick_check.

Driver: Session 4 (D-005-OSS) live test.

Result: marker row drill-marker-d005 survived the wipe + restore
cycle. /readyz returned ready=true, db=ok. PRAGMA quick_check
returned ok. Full procedure transcribed in
infra/compose/README.md "Restore drill" section.

Scripted, runnable

The four drills below have executable scripts under drills/. Each:

expects the compose stack to already be up,
prints expected output line-for-line so an operator can compare,
restores normal state at the end (or prints a clear "restart required"),
exits non-zero on assertion failure.

Run any of them ad-hoc; record the outcome at the bottom of this file.

peer-collapse

drills/peer-collapse.sh — docker stop manthan-alpha, wait for the
gossip session count to drop on beta + gamma, restart alpha, verify the
ring reforms.

The ManthanPeerCollapse Prometheus rule fires after 5 minutes; the script
short-circuits the wait by reading the metric directly. If you also want
to verify the alert reaches Teams/PagerDuty, leave alpha down for >5 min.

liveness

drills/liveness.sh — docker stop manthan-alpha, verify
/healthz is unreachable, restart, verify /healthz returns 200 again.

ManthanLivenessLost fires after 1 min in Prometheus; the script doesn't
wait for the alert by default.

readiness

drills/readiness.sh — corrupt alpha's ledger by truncating the
SQLite file, expect /readyz to return 503, restore from the latest
restic snapshot, expect /readyz to return 200 again.

This is the most disruptive of the four; it relies on a recent backup
existing. Don't run it without one.

charter-refresh

drills/charter-refresh.sh — read /metrics for
manthan_charter_age_seconds, restart alpha (which re-signs the Charter
on startup per single-node.ts), confirm the metric drops back near 0.

Doc-only until production trigger

These have failure modes worth specifying but aren't worth automating
at the current stage.

verifier-failures

The runbook.md procedure says "inject malformed envelopes from a test
peer; verify quarantine." Doing that requires a fourth node-shaped client
that produces malformed envelopes deliberately — not a quick script. When
production has external peers, this is worth scripting because adversarial
input is real then.

rate-limit

The procedure says "burst 10× the rate cap; verify drops + alert." Same
shape as verifier-failures — needs a client that can flood a peer. Add
when you have one.

ledger-quarantine

The procedure says "manually quarantine an envelope; verify state machine."
Requires direct SQL surgery on the ledger to flip an envelope to
state='quarantined', then check that the gossip layer respects it. This
is a fine drill to run by hand once; not worth scripting.

rollback

The procedure says "deploy a buggy image; rollback to previous." Compose
rollback is docker-compose down && docker-compose up -d after you've
re-tagged or re-pulled. The non-trivial bit is image-version pinning,
which requires an image registry workflow (ECR or DockerHub) the OSS
track doesn't have. Add when CI publishes images.

key-compromise

The procedure says "rotate operator key in KMS; verify recovery flow."
Doesn't apply to the OSS track at this stage — the trust root is
deterministic per docs/TRUST-ROOT-POSTURE.md. The drill becomes
meaningful when seeds move to sops or KMS. Same trigger as the trust-root
graduation.

Outcomes log

Append one line per drill run. Pattern:

YYYY-MM-DD  HH:MM  drill            outcome  notes

2026-05-06  13:30  data-backup      pass     Session 4 live test
2026-05-06  13:35  data-restore     pass     marker drill-marker-d005 survived; PRAGMA quick_check ok
2026-05-06  14:13  data-backup      pass     SFTP to demo-server; repo a49a623fa5

Drill log — Compose / OSS track