Manthan compose stack — self-hosted observability

Manthan compose stack — self-hosted observability

Open-source alternative to the AWS Fargate + Amazon Managed Prometheus stack
in ../terraform/. Same image, same alerts.yaml, same PagerDuty/Teams/Slack
receivers — just running on docker-compose instead of Fargate, with vanilla
Prometheus + Alertmanager instead of AMP.

Why this exists

The Terraform stack is the production roadmap target: managed services,
HA, auto-scale, paid SLAs. For an internal/research stage where the budget
is "don't spend much", that's overkill. This compose stack runs the entire
testnet on a single Linux host (the existing genesis EC2 is the obvious
target) for ~$0/mo on top of whatever you're already paying for that VM.

What's in the stack

Prometheus is on 19090 (not 9090) and Grafana on 13000 (not 3000) so they

don't collide with anything else already running on the host. Edit the

compose file's ports: if you want different host ports.

First run

# from the repo root
docker build -f infra/docker/Dockerfile -t manthan-testnet-node:0.2.0 .

cp infra/compose/.env.example infra/compose/.env
$EDITOR infra/compose/.env       # paste real PagerDuty / Teams / Slack URLs

docker compose -f infra/compose/docker-compose.yml up -d

Verify

# Each node responds to /healthz
curl -s local-service/healthz | jq
curl -s local-service/healthz | jq
curl -s local-service/healthz | jq

# Each node /canary is signed
curl -s local-service/canary | jq

# Prometheus has discovered all 3 targets
curl -s 'local-service/api/v1/targets' | jq '.data.activeTargets[] | {job, health, instance}'

# Alertmanager has the routing tree
open local-service

# Grafana dashboard ('Manthan — Overview' under General)
open local-service     # admin / admin first login

Receiver test

To prove an alert actually reaches Teams and Slack, kill one node:

docker stop manthan-alpha
# Wait 5 minutes — the ManthanLivenessLost / ManthanPeerCollapse rules fire
# (SEV-1, hits PagerDuty too if PD is enabled).

The same alerts fire on the AWS path; only the transport layer changes.

Cost comparison (3-node testnet, low traffic)

When to switch back to the AWS path

• You need ≥99.9% availability (cross-AZ HA).
• You have paying users and the SLA matters.
• You need separate jurisdictions (per Charter §8.1, Phase 6+ diversity).
• Compliance demands managed-service audit logs out of the box.

The Terraform stack is preserved untouched at ../terraform/ for that
transition. Both paths consume the same manthan-testnet-node:0.2.0 image
and the same alerts.yaml, so moving between them is roughly:

1. Build and push the image to ECR (already documented in ../terraform/README.md).
2. terraform apply — stack comes up.
3. aws secretsmanager update-secret for each receiver URL.
4. Repoint your DNS at the new ALB DNS name.

Backup & restore (D-005-OSS)

The backup sidecar runs restic on a cron schedule (default every 6h) and
captures the full persistent-state inventory:

Snapshots are encrypted with RESTIC_PASSWORD before they leave the host.
Lose that password → backups are unreadable. Store it where you keep other
recovery secrets.

Configure

In infra/compose/.env:

# Local repo — fine for dev/test, but NOT off-host (host loss = backups lost):
RESTIC_REPOSITORY=/backups/manthan

# OR off-host examples (pick one and set the matching creds):
# Demo-server SFTP — second EC2 host in the same AWS account/region. Not
# truly geo-redundant but materially better than single-host. Already
# staged on this genesis host; see "Demo-server SFTP backup" below.
# RESTIC_REPOSITORY=sftp:ec2-user@<backup-host-private-ip>:manthan-backups/repo
# RESTIC_REPOSITORY=s3:s3.amazonaws.com/<bucket>/manthan        # + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
# RESTIC_REPOSITORY=s3:<r2-endpoint>/<bucket>/manthan           # Cloudflare R2 via S3 API
# RESTIC_REPOSITORY=b2:<bucket>:manthan                         # + B2_ACCOUNT_ID / B2_ACCOUNT_KEY

RESTIC_PASSWORD=<long random string — change from the placeholder>

BACKUP_SCHEDULE=0 */6 * * *      # default every 6 hours
BACKUP_RETENTION_HOURLY=24
BACKUP_RETENTION_DAILY=30
BACKUP_RETENTION_WEEKLY=12

Demo-server SFTP backup (the operator's poor-man off-host)

If you have a second EC2 host you can SSH to (the operator's existing
demo-server is the obvious one), restic can use it as an SFTP repo. This
gives two-server resilience — genesis loss no longer loses backups —
without any cloud-storage spend. Caveat: same AWS account / same region,
so partial failure correlation remains. Don't treat it as DR-grade.

What's already wired on this host:

• infra/compose/backup/ssh-key.pem — copy of genesis-dev.pem (mode 600, gitignored).
• infra/compose/backup/known_hosts — ssh-keyscan of the demo-server's public + private IPs (gitignored).
• infra/compose/backup/Dockerfile — installs openssh-client so restic can drive sftp.
• infra/compose/backup/entrypoint.sh — when RESTIC_REPOSITORY starts with sftp:, materialises /root/.ssh/{id_backup,known_hosts,config} from the mounted files.
• infra/compose/docker-compose.yml — bind-mounts the key + known_hosts into the backup container at /etc/manthan/ssh/{id,known_hosts} (read-only).

To enable, in infra/compose/.env:

RESTIC_REPOSITORY=sftp:ec2-user@<backup-host-private-ip>:manthan-backups/repo
RESTIC_PASSWORD=<a long random string — change from the placeholder>

The remote directory was created at first init (~/manthan-backups/repo on demo-server with mode 700). Verify with:

ssh demo-server 'ls ~/manthan-backups/repo'      # should show config / data / index / keys / locks / snapshots

For a fresh clone on a different host: drop your private key at
infra/compose/backup/ssh-key.pem (mode 600), then
ssh-keyscan -H <demo-server-ip> > infra/compose/backup/known_hosts,
and you're set.

Verify backups exist

docker exec manthan-backup restic snapshots --tag manthan

Trigger a one-shot backup (for drills / between cron runs)

docker exec manthan-backup /opt/backup/backup.sh

Restore drill

A full restore drill validates that backups are usable, not just present.
Procedure:

# 1. Note the latest snapshot id
docker exec manthan-backup restic snapshots --tag manthan --latest 1

# 2. Pull a snapshot into the restore-out volume
docker run --rm \
  -v $(pwd)/.env:/opt/.env:ro \
  --env-file ./.env \
  -v manthan-compose_backup-local-repo:/backups \
  -v manthan-compose_backup-cache:/restic-cache \
  -v manthan-compose_restore-out:/restore-target \
  -e BACKUP_MODE=restore \
  manthan-backup:0.2.0 latest

# 3. Tear down the running stack
docker compose -f infra/compose/docker-compose.yml down

# 4. For each node, copy the restored ledger into a fresh volume
for n in alpha beta gamma; do
  docker volume rm manthan-compose_${n}-data 2>/dev/null || true
  docker volume create manthan-compose_${n}-data
  docker run --rm \
    -v manthan-compose_${n}-data:/data \
    -v manthan-compose_restore-out:/restore-out \
    alpine sh -c "cp /restore-out/tmp/backup-staging/ledgers/${n}-ledger.sqlite /data/ledger.sqlite"
done

# 5. Bring the stack back up
docker compose -f infra/compose/docker-compose.yml up -d

# 6. Verify the ledger has the pre-restore envelope counts
for p in 9120 9121 9122; do
  curl -s local-service:$p/sth | jq '.tree_size'
done

Run this drill at least monthly. Document the outcome in
herald-protocol/services/testnet/incident-response/drill-log.md (per D-007).

What's deferred / out of scope

• TLS at the edge: the compose stack listens HTTP on host ports. For a

publicly-reachable deploy, put it behind Caddy / Traefik / nginx with
Let's Encrypt; that's a one-page reverse-proxy config, not a code change.

• Cross-host HA: by definition single-host. If a host migration is

required, it's docker compose down && rsync volumes && up on the new host.

• Backup of operator long-term seeds: for testnet, seeds are deterministic

from node names; for production, set MANTHAN_NODE_SEED_HEX in .env,
which IS captured by the backup sidecar's .env snapshot.

• Backup of Prometheus TSDB: intentionally skipped — TSDB is reconstructible

from scrape data after the next scrape cycle, and including it would balloon
snapshot size for no recovery benefit.

Stop

# Keep volumes (ledgers, TSDB, dashboards):
docker compose -f infra/compose/docker-compose.yml down

# Or full reset:
docker compose -f infra/compose/docker-compose.yml down -v

Production-readiness checklist

The compose stack is the default for internal use. If it accumulates
real users / live agents over time and starts inheriting production
responsibilities (which the planning doc warns is the silent-drift risk),
walk this checklist before treating it as production:

• [ ] TLS at the edge. Reverse proxy (Caddy/Traefik/nginx) with

Let's Encrypt in front of every public host:port.

• [ ] Off-host backups. RESTIC_REPOSITORY is not a local volume —

it's a cloud bucket the host cannot also lose. Restore drill ran
successfully within the last 30 days.

• [ ] RESTIC_PASSWORD lives in a password manager / vault, not just

.env on the host.

• [ ] .env itself is backed up to somewhere other than the host

(the backup sidecar does this if RESTIC_REPOSITORY is off-host;
verify it's covered).

• [ ] Operator seeds are explicit in .env (MANTHAN_NODE_SEED_HEX_*)

rather than implicitly deterministic — so a re-deploy on a new host
keeps node identity.

• [ ] Webhook secrets (PagerDuty / Teams / Slack) are real, not

placeholders. Test alerts have been observed reaching all three.

• [ ] Charter / LOTS / privacy-policy have had the deployment-domain

search-and-replace done (per
herald-protocol/services/testnet/nodes/README.md) and reflect the
actual jurisdiction the operator is in.

• [ ] Monitoring retention — Prometheus TSDB retention (--storage.tsdb.retention.time)

is sized for your investigation window; default 30d may be insufficient.

• [ ] Peer-replacement runbook has been drilled at least once

(D-007 procedure peer-collapse).

• [ ] Host-loss recovery runbook — restore drill from off-host backup

onto a fresh VM, verified end-to-end.

• [ ] Resource limits on each container (mem_limit, cpus:) so a

runaway agent can't OOM the host.

• [ ] docker-compose.yml checked into version control with no

secrets — secrets only in .env, which is gitignored.

• [ ] HSM / key-custody posture documented (D-006-OSS): if you're

still on deterministic / env-supplied seeds, the testnet
"internal use" framing is honest. For production, operator seeds
should move to age/sops-encrypted secret or a real HSM.

When a majority of these aren't ticked, the system is "running in
production" but not "production-ready." That's a real risk; the AWS-managed
path (infra/terraform/) ticks most of them by construction, which is what
the production roadmap is for.