Manthan Testnet Incident Response Runbook (T-338)

Manthan Testnet Incident Response Runbook (T-338)

Per Manthan T-338. Concrete operational procedures for the most common testnet incidents. AWS production deployment uses these procedures verbatim with the operator-specific contact list filled in at the top.

Severity definitions

On-call contacts (AWS production fills in)

• SEV-1 primary: <TBD per deployment — PagerDuty schedule URL>
• SEV-2 primary: <TBD>
• Slack channel: #manthan-ops
• Security incident contact: security@<deployment-domain> (PGP key fingerprint TBD)

Procedures

charter-refresh

Trigger: alert ManthanCharterStaleSoon or ManthanCharterStale.

Symptom: Charter signed >11 months ago; verifiers will reject the Herald's attestations after 12 months.

Procedure:

1. Open the node's filled-in Charter at services/testnet/nodes/<node>/charter.md.
2. Review every section (§1-§9). Change anything that has drifted (witness identities, contact addresses, regulatory relationships, COI declarations).
3. Increment the Charter version in §1.
4. Restart the node — it re-reads the Charter, re-signs at startup, and serves the new signed Charter at /charter.
5. Verify: curl https://<node>/charter | jq .signed_at — should be the current Unix timestamp.
6. Verify: manthan_charter_age_seconds metric drops to <60 within 1 minute.
7. Notify auditors via the contact email in §9.

Time: ~30 minutes including review.

peer-collapse

Trigger: alert ManthanPeerCollapse (peer count <2 for >5min).

Symptom: Node has lost gossip connectivity to most peers.

Procedure:

1. Check the node's /healthz and /readyz endpoints — confirm the node itself is alive.
2. Try connecting from another machine to the node's wss:// MWP port. If the connection itself fails, the issue is network (security groups, ALB health checks).
3. List the current peer set: curl http://<node>/metrics | grep manthan_peer_count.
4. Try connecting outbound to each peer in the bootstrap seed list:
``bash
for peer_endpoint in $(jq -r '.seeds[].endpoint' deploy/bootstrap-seed-list.json); do
wscat -c "$peer_endpoint" -x "ping" || echo "FAILED: $peer_endpoint"
done
`
5. If specific peers are unreachable: contact their operator (per their Charter §9). Track in incident registry.
6. If all peers are unreachable: the network has partitioned. Restore by:
- Verifying DNS for *.testnet.local (testnet) or the production domain
- Verifying TLS certificate validity (production: ACM cert renewal)
- Verifying VPC security groups allow wss://` ingress on the MWP port

Time: 30-60 minutes for connectivity issues; longer if peer-side outage.

verifier-failures

Trigger: alert ManthanRepeatedVerifierFailures (quarantine rate >0.5/sec for 5min).

Symptom: Sustained envelope quarantine rate. Either an adversary is flooding bad signatures or a peer's library has a bug.

Procedure:

1. Sample the quarantined envelopes:
``bash
sqlite3 services/testnet/nodes/<node>/data/ledger.sqlite \
"SELECT id, from_sigil, ts, state_ts FROM envelopes WHERE state='quarantined' ORDER BY state_ts DESC LIMIT 100;"
`
2. Group by from_sigil to find the offending peer.
3. If one peer is responsible:
- Eject the peer from the gossip ring (set its manthan_peer_count for that sigil to 0 by closing the session).
- Notify the peer's operator via their Charter §9 contact.
- File an incident at /incidents.
4. If multiple peers are responsible: investigate whether the local node has a bug (run npm run typecheck` and unit tests).
5. If neither: it's a coordinated attack. Escalate to SEV-1; consider emergency Bestiary listing of the offending sigils.

Time: 1-4 hours.

rate-limit

Trigger: alert ManthanRateLimitSaturation (drop rate >1/sec sustained for 10min).

Symptom: Either legitimate traffic exceeds the rate cap or a peer is flooding.

Procedure:

1. Check manthan_envelopes_received_total rate. If it's high but legitimate: raise rateLimitEnvelopesPerMin in the GossipRuntime config.
2. If it's a flood: identify the source peer and follow the verifier-failures ejection procedure.
3. Document the resolution in /incidents.

Time: 15-30 minutes.

ledger-quarantine

Trigger: alert ManthanLedgerCorruption (any quarantined envelopes present).

Symptom: Some envelope landed in quarantined state.

Procedure:

1. Inspect the quarantined envelopes (see verifier-failures procedure step 1).
2. If the quarantine was correct (verification truly failed), document and close.
3. If the quarantine was a false positive (e.g., a peer's pubkey was rotated and the local resolver was stale), update the resolver and selectively un-quarantine via:
``bash
sqlite3 ledger.sqlite "UPDATE envelopes SET state='verified', state_ts=strftime('%s','now') WHERE id='<id>' AND state='quarantined';"
`
4. Restart the node so its seenIds` rebuild reflects the corrected state.

Time: 30-60 minutes.

readiness

Trigger: alert ManthanReadinessFail (/readyz non-200 for >2min).

Symptom: Node is alive but not ready. Common causes:

• Ledger integrity check failing (PRAGMA quick_check non-OK)
• Operator key not loaded
• Charter signature missing or expired

Procedure:

1. Check curl http://<node>/readyz — response body identifies the failing check.
2. db: fail → ledger corruption. Restore from backup (see data-restore procedure below).
3. key: fail → operator key file missing or unreadable. Restore from key backup; mode 0600.
4. charter: fail → Charter file missing or expired. Run charter-refresh.

Time: 15-60 minutes.

liveness

Trigger: alert ManthanLivenessLost (/healthz non-200 for >1min).

Symptom: Process is dead or hung.

Procedure:

1. Check process status: systemctl status manthan-node (systemd) or ecs:DescribeTasks (AWS).
2. Inspect logs for crash reason:
- OOM → raise instance size (AWS) or set Node --max-old-space-size.
- Uncaught exception → file a bug; restart pod via systemd/ECS task replacement.
- Hung event loop → SIGKILL the process; investigate post-mortem.
3. If repeated: rollback to the previous deployment (see rollback procedure).

Time: 5-30 minutes for restart; longer for root-cause investigation.

rollback (T-338 explicit)

Trigger: new deployment introduces SEV-1/2 issues; need to revert.

Procedure:

1. Stop accepting new envelopes: scale down to 0 instances via aws ecs update-service --desired-count 0 (production) or kill the testnet process (dev).
2. Restore ledger snapshot:
- AWS production: ledger persistence is on EBS-backed EFS. Restore from the latest backup snapshot via aws ec2 describe-snapshots → aws ec2 restore-snapshot-from-recycle-bin.
- Localhost testnet: restore the SQLite file from the manual hourly backup (see data-backup procedure).
3. Redeploy the previous image version:
``bash
aws ecs update-service --cluster manthan-testnet \
--service genesis-alpha --task-definition manthan-node:<previous-revision>
`
4. Verify rollback complete: /healthz returns 200, /readyz ok, ledger integrity check passes.
5. Document: create an incident report at /incidents` with the rollback rationale.

Time: 15-60 minutes for AWS rollback.

key-compromise

Trigger: suspected operator-key compromise (alert from KMS access logs, observed forged signatures, or external notification).

Procedure:

1. Stop signing immediately: scale node to 0; revoke IAM access to the KMS key.
2. Issue herald_compromise envelope: sign with the recovery key declared in the LOTS §2.3.
3. Publish at compromise URL: <node>/compromise per LOTS §9.
4. Issue revoke_attestation envelopes for all attestations issued in the suspected compromise window.
5. Plan emergency molt: schedule a new Ed25519 keypair generation with the 24-hour cool-off rule per [EXT §1].
6. Notify peers and downstream verifiers via the contact list in Charter §9.
7. Forensic investigation: preserve logs, KMS audit trail, ledger state.

This is a SEV-1 incident. Time-to-contain: 4 hours. Time-to-recover: 1-7 days.

Data backup and restore

data-backup

Cadence:

• Hot snapshots every 1 hour (SQLite consistent snapshot via .backup API).
• Cold backups weekly (full directory tar with operator-encrypted key).
• ≥1 off-site copy in a different jurisdiction.

AWS production:

• EFS snapshots every hour, retained 30 days.
• Daily snapshot copied to S3 (cross-region replicated).
• Charter, LOTS, privacy-policy archived in S3 versioned bucket.

data-restore

# Stop the node first
systemctl stop manthan-node-genesis-alpha   # or scale ECS to 0

# Restore from the latest hourly snapshot
cp /backup/genesis-alpha/$(ls -t /backup/genesis-alpha/ | head -1)/ledger.sqlite \
   services/testnet/nodes/genesis-alpha/data/ledger.sqlite

# Decrypt operator key from backup
openssl aes-256-cbc -d -in operator.key.json.enc \
  -out services/testnet/nodes/genesis-alpha/data/operator.key.json
chmod 600 services/testnet/nodes/genesis-alpha/data/operator.key.json

# Restart and verify
systemctl start manthan-node-genesis-alpha
curl local-service/readyz

After-action items (for every SEV-1/2 incident)

1. Post-mortem within 14 days, published at /incidents/<incident-id>.
2. Identify root cause + contributing factors.
3. Identify what monitoring or process failed (the alert that should have fired earlier).
4. File follow-up tickets to prevent recurrence.
5. Update this runbook if the procedure was unclear or wrong.

AWS production deployment checklist

Before this runbook is operational on AWS, verify:

• [ ] PagerDuty service set up with on-call rotation
• [ ] Alertmanager configured with the rules in observability/alerts.yaml
• [ ] ACM certs configured for the production domain
• [ ] EFS backup snapshots configured (hourly + daily + weekly)
• [ ] S3 cross-region replication enabled for backups
• [ ] KMS keys with multi-region replication
• [ ] CloudWatch dashboards mirror observability/dashboard.md
• [ ] Runbook URL reachable from PagerDuty alerts
• [ ] On-call contacts filled in at the top of this file
• [ ] Charter §9 contact addresses use real emails (not <deployment-domain> placeholders)

— Manthan testnet runbook v1, 2026-05-06.