Evaluation harness: separation over sentiment

Evaluation harness — separation, not sentiment

A harness for evaluating things that have no obvious right answer, using the LLM
CLIs already authenticated on this machine.

python3 run_tone.py --set hard                 # the self-test (has ground truth)
python3 run_tone.py --set hard --runtimes claude,codex,copilot,qwen

The one rule

Never report a level. Always report a separation.

An agent asked to rate something returns a generous number almost regardless of
content, so 8.2/10 means nothing. What survives a generous judge is whether it
ranks one condition above another — because bias lifts both sides equally. So
the headline metric is always AUC or a paired difference with a confidence
interval, and 0.5 / zero is the result that kills an idea.

Runtimes

	family	non-interactive	latency
`claude`	anthropic	`claude -p`	~10s
`codex`	openai	`codex exec --skip-git-repo-check`	~64s
`copilot`	openai	`copilot -p`	~65s
`qwen`	qwen	`ollama run qwen2.5-coder:14b`	~12s

gemini is installed but dead — IneligibleTierError, wants migration to
Antigravity. A runtime that errors is recorded as unavailable and excluded from
scoring, never silently substituted.

The self-test: does tone change the answer?

Chosen as the shakedown because it is the only question here with a ground
truth — the task is either done right or it isn't, so no judge is needed and no
judge's agreeableness can contaminate the result.

Same question, four politeness wrappers, a constant format instruction, three
repeats per cell. Repeats are not optional: without them a tone "effect"
cannot be told apart from ordinary run-to-run variance.

Result (2026-09-22, hard set, 336 calls)

	very polite	neutral	blunt	rude
claude	92.9%	92.9%	92.9%	92.9%
qwen 14B	54.8%	50.0%	57.1%	52.4%

Paired polite−rude: claude +0.0 [+0.0, +0.0], qwen +2.4 [−4.8, +9.5].
No detectable tone effect in either.

Noise floor — identical prompt, repeated: claude 0/56 flips, qwen 8/56 (14%).
Claude was perfectly deterministic on these tasks, which points at context
state rather than sampling as the source of day-to-day variability. Short
factual answers are the easiest case for stability, so this does not generalise
to long generative work.

The flaw it caught first

The easy question set put Claude at 100% on all four tones. That reads as a
finding and is actually an instrument with no range. The hard set exists because
a ceiling is not a result.

Layout

harness/runtimes.py   CLI adapters; honest failure, hard timeouts
harness/pool.py       threaded fan-out (subprocesses release the GIL)
harness/stats.py      AUC, bootstrap CIs, paired differences
harness/tone.py       the self-test: questions, tones, exact grader
run_tone.py           entry point
runs/                 raw JSON, one file per sweep

Evaluation harness: separation over sentiment