G-010 — Define Storage And CT Recovery

G-010 — Define Storage And CT Recovery

Status: drafted → reviewed → accepted → closed.
Author: driver-claude
Date: 2026-05-06

1. Job of this gate

Original platform §3.1 said only "an embedded database (e.g., SQLite or LevelDB)". Per audits/manthan-platform-review.md §7-F12 and §8-F-N: storage schema, CT submit idempotency, crash recovery, backup, and ledger-corruption quarantine are unspecified. Without these, a Manthan node that crashes mid-CT-submit double-submits envelopes; a corrupted ledger silently propagates bad state; backup/restore is undefined.

G-010 specifies:

• Local ledger schema
• Envelope state machine
• Idempotent CT submission
• Crash recovery
• Backup/restore
• Corrupted-ledger quarantine

2. Local ledger schema

Storage backend: SQLite (chosen for portability, single-file, well-understood crash semantics with WAL).

2.1 Tables

CREATE TABLE envelopes (
  id              BLOB PRIMARY KEY,         -- sha256 hash, 32 bytes
  trace           BLOB NOT NULL,
  parent          BLOB,                     -- nullable for genesis
  from_sigil      TEXT NOT NULL,
  to_sigil        TEXT,
  ts              INTEGER NOT NULL,         -- Unix seconds
  state           TEXT NOT NULL CHECK (state IN
                  ('seen','verified','queued_for_ct',
                   'submitted','inclusion_proven','gossiped','quarantined')),
  body_cbor       BLOB NOT NULL,            -- canonical CBOR body
  signatures_json TEXT NOT NULL,            -- JSON array of signature objects
  attestations_cbor BLOB,                   -- optional, for typed-artifact lookups
  payload_cbor    BLOB,
  received_at     INTEGER NOT NULL,         -- when this node first saw it
  state_ts        INTEGER NOT NULL          -- last state-transition timestamp
);

CREATE INDEX idx_envelopes_trace ON envelopes(trace);
CREATE INDEX idx_envelopes_from ON envelopes(from_sigil);
CREATE INDEX idx_envelopes_state ON envelopes(state);
CREATE INDEX idx_envelopes_ts ON envelopes(ts);

CREATE TABLE ct_submissions (
  envelope_id     BLOB PRIMARY KEY REFERENCES envelopes(id),
  log_id          BLOB NOT NULL,
  sct_json        TEXT NOT NULL,            -- the SCT received from log
  submitted_at    INTEGER NOT NULL,
  sip_json        TEXT,                     -- the SIP, when received
  inclusion_proven_at INTEGER,
  retry_count     INTEGER NOT NULL DEFAULT 0,
  last_retry_at   INTEGER,
  last_error      TEXT
);

CREATE INDEX idx_ct_submissions_log ON ct_submissions(log_id);
CREATE INDEX idx_ct_submissions_unfinished
  ON ct_submissions(log_id, inclusion_proven_at)
  WHERE inclusion_proven_at IS NULL;

CREATE TABLE peers (
  sigil           TEXT PRIMARY KEY,
  endpoint        TEXT NOT NULL,
  last_seen_at    INTEGER NOT NULL,
  charter_url     TEXT,
  charter_signed_at INTEGER,                -- freshness tracker
  jurisdiction    TEXT,                     -- declared per peer's Charter
  ownership_cluster TEXT,
  trust_score     REAL                      -- locally-computed Patina (G-003)
);

CREATE TABLE sths_observed (
  log_id          BLOB NOT NULL,
  tree_size       INTEGER NOT NULL,
  root_hash       BLOB NOT NULL,
  signed_sth_cbor BLOB NOT NULL,
  observed_at     INTEGER NOT NULL,
  observed_via    TEXT NOT NULL,            -- 'direct'|'gossip-dns'|'gossip-p2p'|'cross-pub'
  PRIMARY KEY (log_id, tree_size)
);

CREATE INDEX idx_sths_log ON sths_observed(log_id, tree_size DESC);

CREATE TABLE config (
  key             TEXT PRIMARY KEY,
  value           BLOB NOT NULL
);
-- canonical keys: 'operator_key_seed_hex', 'last_known_good_state', 'schema_version'

2.2 Schema versioning

config.schema_version is set on first init. Migrations are forward-only; a node refuses to start against a higher schema_version than its code knows about.

2.3 WAL mode + sync rules

PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;       -- production deployments may set FULL
PRAGMA foreign_keys = ON;

WAL mode + synchronous=NORMAL is the SQLite-recommended balance for write-heavy workloads with acceptable durability on power-loss.

3. Envelope state machine

Per gemini's tracker §G-010, envelopes flow through:

   ┌──────┐        ┌──────────┐        ┌──────────────────┐
   │ seen │───────▶│ verified │───────▶│ queued_for_ct    │
   └──────┘        └──────────┘        └────────┬─────────┘
                                                 │
                                                 ▼
                                       ┌──────────────────┐
                                       │   submitted      │ (SCT received)
                                       └────────┬─────────┘
                                                 │
                                                 ▼
                                       ┌──────────────────┐
                                       │ inclusion_proven │ (SIP verified)
                                       └────────┬─────────┘
                                                 │
                                                 ▼
                                       ┌──────────────────┐
                                       │    gossiped      │
                                       └──────────────────┘

Anywhere in the flow: detected anomaly → `quarantined`

3.1 State transition rules

3.2 Idempotency

Every transition is idempotent: re-running the trigger on an envelope already in the target state is a no-op. Specifically:

• CT submission: before POSTing, SELECT sct_json FROM ct_submissions WHERE envelope_id = ?. If present, skip POST and proceed to submitted.
• SIP retrieval: before requesting, check inclusion_proven_at. If set, skip.

This makes crash recovery (next §) straightforward.

4. Crash recovery

4.1 On startup

1. Open SQLite (WAL ensures consistent state).
2. Verify integrity:
   PRAGMA quick_check;
   If failed → fail-fast, log corruption, refuse to operate, exit non-zero.
3. SELECT envelope_id FROM ct_submissions WHERE inclusion_proven_at IS NULL
   AND submitted_at IS NOT NULL;
   For each: GET /sip/<envelope_id> from the log.
     - If 200: store SIP, transition to inclusion_proven.
     - If 202: SIP still pending; leave in submitted; retry later.
     - If 404 + tree_size > submitted_tree_size + MMD-window: log non-conformance, transition to quarantined.
4. SELECT envelope_id FROM envelopes WHERE state = 'queued_for_ct';
   For each: re-attempt POST /submit with idempotency key = envelope_id.
   Log returns the existing SCT if already submitted; we transition to submitted.
5. SELECT envelope_id FROM envelopes WHERE state = 'verified' AND not-yet-CT-decided;
   For each: re-evaluate "needs CT" per envelope class; transition appropriately.
6. Resume normal operation.

This handles the three crash windows:

• After verify, before queue: re-decide on next start (idempotent).
• After queue, before submit response: idempotent retry; SCT lookup is keyed by envelope_id.
• After submit response, before SIP: SIP retrieval on startup.

4.2 The "double-submit" question

A naive HTTP POST /submit followed by crash mid-response would normally cause double-submit. Manthan's CT submit MUST be idempotent at the log layer (the log returns the same SCT for the same envelope_id submitted twice). Per lib/herald-ct-log/SCHEMA.md §9 (T-101), this is normative for HERALD CT logs.

If the deployed log doesn't honour idempotency: the Manthan node's pre-POST check (§3.2) prevents the duplicate submit by consulting local ledger first.

5. Backup and restore

5.1 Backup format

A "Manthan backup bundle" is:

manthan-backup-<sigil>-<timestamp>.tar.gz
├── manifest.json                 # sigil, version, schema_version, hash of each file
├── ledger.sqlite                 # SQLite file (consistent snapshot via .backup API)
├── operator.key.json             # encrypted under backup-passphrase
├── charter.md                    # current Charter
├── lots.md                       # current LOTS (if running a CT log)
└── peers.json                    # exported peer list (de-trust-scored)

5.2 Backup cadence

• Hot backups every 1 hour (consistent SQLite snapshot via sqlite3_backup_init).
• Cold backups weekly (full directory tar with operator-encrypted key).
• Off-site ≥ 1 copy in a different jurisdiction (per Charter §4 disaster-recovery posture).

5.3 Restore semantics

Restoring a backup to a fresh node:
1. Validate manifest hashes.
2. Restore SQLite, decrypt operator key.
3. CT reconciliation: query the CT log for any envelopes the backup says are submitted but not inclusion_proven. Update accordingly.
4. Peer rediscovery: peers from the backup's peers.json are stale; re-validate Charter freshness before re-trusting.
5. STH catch-up: if the backup's last STH is older than the log's current STH by > MMD, fetch consistency-proof and update.

A restored node is operational within ~5 minutes of restore start (assuming network access to the CT log).

6. Ledger corruption quarantine

If PRAGMA quick_check returns non-OK at any point:
1. Immediately transition to read-only mode; log the corruption, mark last_known_good_state as the previous startup's checkpoint.
2. Do not write new envelopes; do not gossip; do not submit to CT.
3. Surface a SEV-1 incident per Charter §8 incident-disclosure policy.
4. Operator action required: restore from latest backup, or repair using .recover if the corruption is mild.
5. A corrupt-ledger node MUST NOT rejoin gossip until its next-startup PRAGMA quick_check passes against a clean restored ledger.

This is more conservative than "best-effort recover and continue" — Manthan errs toward integrity over availability.

7. Acceptance criteria

• [x] SQLite schema with explicit tables, indices, and CHECK constraints (§2).
• [x] Schema versioning + migration policy (§2.2).
• [x] WAL + sync rules (§2.3).
• [x] Envelope state machine with idempotent transitions (§3).
• [x] Crash-recovery startup sequence (§4.1).
• [x] CT submit idempotency rule (§4.2).
• [x] Backup format + cadence + restore semantics (§5).
• [x] Corruption quarantine procedure (§6).

8. What G-010 does NOT cover

• Cross-database alternatives (LevelDB, RocksDB, Postgres). SQLite is the v0.2 choice.
• Sharded multi-process node architectures. Single-process per node.
• Encryption-at-rest for envelope bodies. Phase 3+ for private log profiles.

9. Sign-off

• [x] drafted → this document
• [x] reviewed → against lib/herald-ct-log/SCHEMA.md §9 (idempotent submit), [EXT §2.4] SCT/SIP semantics, services/reference-herald/src/herald.ts (current data/log.jsonl pattern), SQLite WAL documentation
• [x] accepted → schema is migrateable, state machine is idempotent, recovery is bounded
• [x] closed → G-010 closed; SQLite implementation is Phase-1 work for the agent SDK

— G-010 closed, 2026-05-06.