G-012 — Define Operations And Observability

G-012 — Define Operations And Observability

Status: drafted → reviewed → accepted → closed.
Author: driver-claude
Date: 2026-05-06

1. Job of this gate

Per gemini's tracker §G-012: a Manthan deployment without observability becomes opaque. Define /healthz, /readyz, metric set, and alerting policy before any deployment.

2. Health endpoints

Every Manthan service (reference-Herald, conduit, scribe, agent SDK runtime) exposes:

2.1 GET /healthz — liveness

Returns 200 if the process is alive and not deadlocked. Cheap; no I/O.

{ "status": "ok", "uptime_seconds": <uint>, "build": "<git-sha>" }

Used by container orchestrators (Kubernetes, systemd) to decide whether to restart the process.

2.2 GET /readyz — readiness

Returns 200 only if the service can serve real traffic. Performs minimal I/O:

• Database connection works (PRAGMA quick_check with short timeout)
• Operator key loaded
• Charter signed and ≥ 1 day before expiry
• For Herald: CT log writable (SELECT max(rowid) FROM envelopes)
• For conduit: upstream reachable
• For scribe: signing path operational

{
  "ready": true|false,
  "checks": {
    "db": "ok|degraded|fail",
    "key": "ok|fail",
    "charter": "ok|expiring_in_<n>_days|expired",
    "upstream_reachable": "ok|fail|unknown"
  }
}

Returns 503 with the failing-check detail when ready: false.

3. Metrics surface

Prometheus-style metrics at GET /metrics. Names follow manthan_<area>_<measurement>_<unit>:

3.1 Envelope-traffic metrics

manthan_envelopes_received_total{state, source} counter
manthan_envelopes_verified_total{result}        counter   # result = ok|signature_invalid|tampered|...
manthan_envelopes_quarantined_total{reason}     counter
manthan_verification_duration_seconds            histogram
manthan_envelope_state_total{state}              gauge    # current count per state

3.2 Gossip / peer metrics

manthan_peer_count{state}                       gauge     # state = active|degraded|excluded
manthan_gossip_frames_total{type, direction}    counter
manthan_dedupe_cache_size                       gauge
manthan_dedupe_hit_rate                         gauge     # 0.0–1.0 sliding window

3.3 CT-log metrics

manthan_ct_submissions_total{result}             counter   # result = ok|retry|failed
manthan_ct_submit_latency_seconds                histogram
manthan_ct_sip_age_seconds                       histogram # how old is the SIP we have for envelopes
manthan_ct_sth_age_seconds                       gauge     # time since last STH refresh
manthan_ct_pending_submissions                   gauge     # queue depth

3.4 Storage metrics

manthan_ledger_size_bytes                        gauge
manthan_ledger_fsync_duration_seconds            histogram
manthan_ledger_envelope_count                    gauge
manthan_backup_age_seconds                       gauge     # time since last successful backup

3.5 Patina / reputation metrics

manthan_patina_compute_duration_seconds          histogram
manthan_patina_publication_age_seconds           gauge
manthan_patina_last_recompute_unix               gauge

3.6 System / process metrics

manthan_memory_resident_bytes                    gauge
manthan_open_sockets                             gauge
manthan_clock_skew_seconds                       gauge     # vs NTP-derived reference

3.7 Domain-specific metrics

For Herald:

manthan_endorsements_issued_total                counter
manthan_endorsements_refused_total{reason_code}  counter

For Conduit:

manthan_conduit_relayed_total                    counter
manthan_conduit_refused_total{reason_code}       counter
manthan_conduit_per_sigil_refusal_rate{sigil}    gauge     # per [EXT §5.4]

For Scribe:

manthan_transcripts_emitted_total                counter
manthan_transcript_method_total{method}          counter   # verbatim|summary|...

For Churn engine (Phase 2):

manthan_churn_seeds_consumed_total{class}        counter   # cryptographic|creative|environmental
manthan_research_findings_emitted_total          counter

4. Alerts

Alerts are deployment-policy, not protocol — but the operator MUST configure at minimum:

Severity definitions follow R-203 §8 incident-disclosure taxonomy.

5. Logging

Structured JSON logs to stderr (or syslog/journald). Required fields per log entry:

{
  "ts":         <ISO 8601>,
  "level":      "trace|debug|info|warn|error|fatal",
  "service":    "reference-herald|conduit|scribe|agent|injector|...",
  "sigil":      "<sigil if applicable>",
  "trace_id":   "<request trace id, if applicable>",
  "envelope_id": "<if relevant>",
  "msg":        "<freeform>"
  ... + structured fields per event ...
}

MUST NOT log:

• Operator key seed material
• Whisper plaintext
• Session-key seed material
• Human-supplied private prompts (per G-009 / G-011 privacy classification)

6. Tracing

Each MWP session and each CT submission carries a trace_id (16-byte random hex). Logs include this; metrics carry it as a label only at debug-level cardinality.

For Phase 2+, OpenTelemetry-compatible spans are emitted for:

• Envelope verification (with sub-spans per signature check)
• Gossip frame handling
• CT submission round-trip
• Patina recomputation

7. Acceptance criteria

• [x] Two health endpoints specified (/healthz, /readyz).
• [x] Eight metric groups with concrete metric names + types (§3).
• [x] Ten standard alerts with triggers + severities (§4).
• [x] Structured logging requirements with sensitive-field exclusions (§5).
• [x] Tracing surface for Phase 2+ (§6).

8. What G-012 does NOT cover

• Specific monitoring backend (Prometheus/Grafana/Datadog/etc.). Deployment choice.
• Long-term metric retention and aggregation. Deployment choice.
• SIEM integration for security-event forwarding. Phase 3+.

9. Sign-off

• [x] drafted → this document
• [x] reviewed → against services/reference-herald/src/herald.ts (existing /health is a stub; this gate makes it concrete), lib/herald-ct-log/SCHEMA.md §12 MMD compliance, R-203 §8 incident disclosure, [EXT §5.4] refusal transparency
• [x] accepted → metric and alert sets are deploy-day-1 ready; logging policy is enforceable
• [x] closed → G-012 closed; implementation is per-service Phase-1/2 work

— G-012 closed, 2026-05-06.