D-315 — Manthan cohort-1 post-mortem

D-315 — Manthan cohort-1 post-mortem

Period covered: 2026-05-05 12:50Z (system bootstrap) → 2026-05-05 14:09Z (D-310 submit, ~80 minutes wall clock).

Cohort scope: First end-to-end run of the manthan coordination system across distinct workers. 17 findings on disk, 16 done in TODO, 9 distinct worker identities (5 dispatched by me as Agent subagents, 4 external agents the user injected from other sessions/runtimes).

This document is a hands-dirty look at what broke, what worked, and what should change before cohort-2.

TL;DR

• Coordination kernel held. Atomic mkdir-claim, perl-i TODO patching, and the activity log all worked across cross-runtime workers. No corrupted findings, no double-claimed tasks, no lost submissions.
• Six bugs found and fixed in-flight: bash-3.2 portability, Write-tool literal interpolation, coordinator stderr suppression, BSD-sed PATH, submit cp src==dest, patch.py CWD math.
• One process gap: I dispatched a subagent (worker-F) onto a task an external agent had just claimed. The atomic-claim subsystem handled it correctly (worker-F lost the race, exited cleanly), but the orchestrator (me) wasted a dispatch.
• Throughput was good: parallel cohort of 4 workers (T-001/T-012/T-013/T-016) all submitted within ~10 minutes of dispatch. Driver tasks (D-302/D-306/D-308/D-310/D-311) ran sequentially in ~5–25 min each.
• External agents joined cleanly via JOIN.md. Two codex- agents and two worker-opus- agents read the onboarding doc and produced submissions indistinguishable from in-process workers.

Cohort-1 inventory

Total: 16 tasks done, ~80 minutes elapsed, single human in the loop.

What broke and what I fixed

B1. mapfile: command not found (macOS bash 3.2)

Site: manthan/bin/list initial draft used mapfile -t lines < <(grep ...).

Symptom: list failed silently on macOS where bash --version reports 3.2.57 (Apple ships an old GPLv2-era bash). mapfile is bash 4+.

Fix: rewrote with the portable while-IFS-read idiom:

while IFS= read -r line; do
  ...
done < <(grep ...)

Lesson: manthan/bin/* runs in whatever shell environment the worker has. Default to POSIX-portable constructs. mapfile, readarray, [[ ... ]] (where avoidable), ${var,,} lowercase, &> file, etc. are out.

B2. Activity log entries containing literal $(date)

Site: the initial coordinator/submit drafts used Write tool to append to manthan/state/log/activity.log with content like "2026-05-05T... $(date) SUBMIT ...".

Symptom: Write does not perform shell expansion. Resulting log lines contained the literal string $(date) instead of a timestamp.

Fix: all activity-log writes go through Bash with echo ... >> activity.log, never through Write.

Lesson: Write is for inert content; Bash is for expansion. Document this in manthan/AGENT_PROTOCOL.md (or a comment at the top of bin scripts) so future contributors don't repeat it.

B3. Coordinator grep -c ... || echo 0 producing multi-line output

Site: manthan/bin/coordinator's STATUS.md regeneration logic.

Symptom: count=$(grep -cE '^\| T-' TODO.md || echo 0) — when grep matched, it returned an integer; when it didn't match (exit code 1), the || echo 0 substituted 0. But grep -c outputs 0 on its own when there are zero matches (and exit code 1), so the chain produced "0\n0" — two lines — and downstream arithmetic broke with value too great for base.

Fix: changed to count=$(grep -cE '^\| T-' TODO.md || true) and accept that grep-c emits 0 on no-match. Pre-clean any stray newlines if needed via tr -d '\n'.

Lesson: Don't double-default. grep -c already gives a numeric on no-match; the || echo 0 was redundant and wrong. When you find yourself defending a tool against its own normal output, re-read the man page first.

B4. BSD-sed silent failure / homebrew PATH inheritance

Site: manthan/bin/submit originally used gsed -i 's/| pending /| done /' TODO.md — homebrew GNU sed's in-place flag.

Symptom: Worker-B (subagent) submitted T-012; the activity log recorded the submit; but TODO.md still showed T-012 | pending. Status was inconsistent across two oracles. After investigation: subagent shells didn't inherit homebrew's /opt/homebrew/bin in PATH, so gsed was missing. gsed printed an error to stderr; submit script didn't fail because set -e wasn't set on that line; activity log got written first, TODO didn't.

Fix: rewrote both submit and release to use perl -i -pe instead of any sed variant. perl ships with macOS, with most Linuxes, and behaves identically. Single rule:

perl -i -pe "if (/^\\| $task_id /) { s/\\| (pending|claimed|blocked)( +)\\|/| done    \$2|/ }" "$TODO"

Lesson: In a multi-runtime environment, "available everywhere" beats "available where I tested it." sed/gsed/awk variants are a swamp; perl is reliably uniform.

Adjacent lesson: set -euo pipefail at the top of every bin script. Submit was robust enough to record the activity log before the sed call failed — but if the sed call had been first, we'd have lost the activity-log entry. Failing loud is better than failing partial.

B5. submit cp source==destination error

Site: manthan/bin/submit <task_id> <worker> <findings_src>.

Symptom: Some workers (driver-claude included) wrote findings directly to the canonical destination (manthan/state/findings/<task>/) rather than a tmp dir. Submit then ran cp -R "$findings_src" "$findings_dest" which errored: "cp: ... and ... are identical (not copied)."

Fix: compute absolute paths, skip the copy when src and dest resolve to the same path:

src_abs="$(cd "$(dirname "$findings_src")" && pwd)/$(basename "$findings_src")"
dest_abs="$(pwd)/$findings_dest"
if [[ "$src_abs" == "$dest_abs" ]]; then
  : # already in place
else
  cp -R "$src_abs" "$dest_abs"
fi

Lesson: Treat both calling conventions as legitimate. Some workers stage to tmp; some write directly. The submit script should be a no-op for the latter, not an error. Document both calling patterns in AGENT_PROTOCOL.md.

B6. patch.py Path(__file__).parents[3] instead of .parents[4]

Site: the D-306 substitution script manthan/state/findings/D-306/patch.py.

Symptom: when the subagent ran the script, its CWD was the project root, but __file__ resolution counted parents from the script's own location. With the script at manthan/state/findings/D-306/patch.py, parents[3] lands on manthan/, not the project root. The script tried to read THREATS.md from the wrong directory.

Fix: parents[4]. Verified by counting: patch.py → D-306 → findings → state → manthan → repo-root. Five hops; index 4.

Lesson: Don't trust the __file__ path for project-root resolution. Either:

• Pass project root as an argv (patch.py <repo_root>), OR
• Use a sentinel file lookup (walk up until finding SPEC.md), OR
• Set an env var (MANTHAN_REPO_ROOT) the spawner provides.

The first option is cleanest for one-shot scripts. The third is cleanest for any multi-script tooling.

P1. Dispatch race vs external agent (worker-F vs worker-opus-T020)

Setup: I was dispatching parallel subagents on T-002, T-019, T-020 (cohort-2). Simultaneously, the user invoked an external opus agent on T-020 from another session. Worker-F's CLAIM raced with worker-opus-2026-05-05-T020's CLAIM.

Outcome: Atomic-mkdir claim worked correctly. worker-opus-2026-05-05-T020 won the race (mkdir succeeded for them, EEXIST for worker-F). Worker-F honoured the AGENT_PROTOCOL.md "no racing" rule and exited cleanly, no submission produced. The external agent submitted T-020 5 minutes later. No double-execution.

The cost: I spent ~30 seconds writing the worker-F prompt; that effort was wasted.

Lesson — for orchestrators (me, future drivers, and any human dispatcher):

1. Read STATUS.md immediately before dispatch, not from a stale view earlier in the session. The status changes constantly when external agents are active.
2. Bias toward self-claiming first, then dispatching. The driver can claim a task atomically before spawning the subagent; the subagent then verifies it owns the claim. This eliminates the race window.
3. Or: dispatch with --check-claim semantics. The subagent's first action is claim, and if it fails, exit cleanly with a clear message. Worker-F did exactly this; the system worked, the orchestrator just wasted a dispatch.

For cohort-2, I'll preview status right before each Agent invocation, and prefer batching sequential dispatches between status reads.

What worked well

W1. Atomic mkdir-claim

This is the cornerstone of the whole system. Across 9 distinct workers, 17 claims, and at least one observed race, the mkdir-EEXIST-on-collision contract held. No double-execution. The activity log gives an audit trail for any reviewer.

The contract is simple enough that external agents (codex-, worker-opus-) implemented it correctly from JOIN.md alone. Simplicity is the feature.

W2. JOIN.md as onboarding

The flat onboarding doc — "if you can read files, run shell scripts, and write files, you can contribute" — produced four external-agent submissions with no human-mediated handholding. This is exactly what we hoped for: the system is self-onboarding for any sufficiently capable agent.

The two codex-20260505- submissions and the two worker-opus- submissions are textually clean, follow the receipt template, and integrate directly into the findings tree. No coordination overhead beyond the system itself.

W3. perl-i over sed

After B4, perl-i became the universal in-place editor for this project. It's worked across:

• manthan/bin/submit (TODO state transitions)
• manthan/bin/release (TODO state rollbacks)
• D-306 patch.py's compound-citation pass (second-pass perl regex over THREATS.md)
• D-308's domain-separation tag fix (s/herald\.([a-z_]+)\.v2/herald.$1.v1/g)

Recommendation: codify "perl -i for in-place edits, never sed" in manthan/AGENT_PROTOCOL.md. New in-place edits should default to perl regardless of whether the worker is on macOS, Linux, or some other Unix-flavoured runtime.

W4. Findings directory as the unit of work

A submission is a directory containing RECEIPT.md plus arbitrary outputs (audits, registries, scripts, regenerated test vectors). This is more flexible than "submission is a single file" because:

• Workers can produce multi-file outputs (T-002 shipped a TS library with 4 test files; T-001 shipped 4 test-vector artefacts).
• Reviewers can read RECEIPT.md as the index and dive into specific outputs.
• Promotion (mv findings/T-NNN/foo.md production-path/foo.md) is straightforward.

W5. launchd LaunchAgent for 24×7 maintenance

com.herald.manthan-coordinator.plist runs the coordinator every 600s. Across the cohort, it:

• Reaped no expired claims (all leases were healthy).
• Refreshed STATUS.md regularly enough that external agents got fresh state when they joined.

This is a low-cost, high-resilience piece. Even if the human and all interactive sessions go away, the coordinator keeps the system tidy.

Concrete improvements for cohort-2

In rough priority order:

I1. Codify portability rules in AGENT_PROTOCOL.md

Add a "Tooling discipline" section with:

• In-place edits: perl -i -pe only. No sed/gsed/awk-i.
• Bash: set -euo pipefail at the top of every bin script. Use POSIX constructs only (no mapfile, no readarray, no associative arrays unless required).
• File writes: Write tool for inert content; Bash echo for any expansion (timestamps, env vars, command substitution).
• Project-root resolution: scripts take repo-root as argv or env var, never __file__-relative parents[N].

Estimated effort: 30 minutes.

I2. Pre-flight status check tool

Add manthan/bin/preflight <task_id> that:

• Reads STATUS.md and the live claim directory.
• Returns 0 if the task is pending and not claimed.
• Returns 1 with a clear message if it's already claimed, done, or blocked.

This is a thin wrapper around what I already check informally. Codifying it lets orchestrators (and external agents) verify before spending effort.

Estimated effort: 20 minutes.

I3. Self-claim-then-dispatch pattern for orchestrators

When the orchestrator (driver) wants to dispatch a subagent on a task:

1. Driver runs claim <task_id> <subagent_name> itself first.
2. Dispatch the subagent with the task ID and instructions to verify it owns the claim (via cat manthan/state/claims/<task_id>/owner).
3. If verification fails, subagent exits without submitting.

This eliminates the race window between dispatch and the subagent's first action.

Document this in manthan/templates/agent-prompt.md as the recommended dispatch pattern.

Estimated effort: 15 minutes (template change only).

I4. manthan/bin/timeline for cohort metrics

A small tool that walks the activity log and emits per-task latency, per-worker throughput, and a histogram of claim-to-submit durations. Useful for retrospective review and for catching workers that are stuck (claim with no submit after 2× median latency).

Estimated effort: 45 minutes.

I5. Submission template per task type

Different task types want different output shapes:

• Audit (D-NNN): audits/<topic>.md + RECEIPT.md. Already de facto standard.
• Registry (T-NNN registry-shaped): registries/<topic>.md + RECEIPT.md. Already de facto.
• Implementation (T-NNN code-shaped): <lang>/src/, <lang>/test/, <lang>/README.md + RECEIPT.md.
• Research (R-NNN): research/<topic>.md + RECEIPT.md.

Document the conventions in AGENT_PROTOCOL.md and provide stub templates in manthan/templates/<task-type>/.

Estimated effort: 1 hour.

I6. External-agent pre-claim hint

JOIN.md should explicitly point external agents at the "claim before doing anything" rule. The cohort-1 external agents got this right by reading AGENT_PROTOCOL.md, but a one-line note in JOIN.md ("Check STATUS.md, claim atomically before working") would reduce the chance of effort waste in cohort-2.

Estimated effort: 5 minutes.

Metrics summary

• Wall-clock cohort duration: ~80 minutes (12:50Z bootstrap to 14:09Z latest submit).
• Submissions: 16 tasks done, 1 task (T-018) submitted by external before being claimed by another.
• Distinct workers: 9 (5 dispatched by driver, 4 external).
• Bugs found and fixed: 6 in tooling (B1–B6), 1 process improvement (P1).
• Lost work: 0. Worker-F's wasted dispatch was ~30 seconds; everyone else's submissions integrated cleanly.
• Race conditions hit: 1 (P1). Resolution: correct (loser exited cleanly, no double-execution).
• Median claim-to-submit duration for parallel cohort: ~8 minutes.

Outstanding observations (not actionable in D-315 scope)

• Cohort-1 had no failed-claim retries. Either every claim was first-try, or the failures were silent. Adding a FAILED_CLAIM activity-log event (when mkdir fails) would make this visible. Currently the script just exits with status 3; no log entry. Consider this for I2 / preflight.

• External agents named themselves with date-prefixed identifiers (worker-opus-2026-05-05-D301, codex-20260505-feedback-intake). This is a self-imposed convention from the agents' side. We could codify: "external agents SHOULD include their runtime and a date stamp in the worker identifier" in JOIN.md. Helpful for audit but not strictly required.

• One submission (T-020) was claimed by an external agent simultaneously with a dispatched subagent (worker-F). This is the only race in cohort-1. As external participation scales, we may see more. The atomic-mkdir contract scales fine; the orchestrator (driver) costs scale linearly with dispatch count.

• No subagent ran longer than its 4-hour lease. All claim leases were honoured; the coordinator's reap path was exercised by the test-worker bootstrap (12:50:22 CLAIM, 12:50:23 RELEASE) but never under a real expiry condition. Lease auto-reap remains theoretically tested but not battle-tested.

Recommendation for next steps

Apply I1, I2, I6 before cohort-2 dispatch. They're cheap and reduce the dominant failure modes.

Defer I3, I4, I5 to opportunistic completion — useful but not blocking.

The cohort-1 system is production-ready for continued use as-is. The improvements above are sharpening, not stabilising.

— D-315, completed by driver-claude, 2026-05-05.