In the first post of today’s 40-billion-token factory check-in, I covered how Leaf catches 28.3% of feature diffs in adversarial Tech Lead review while still landing 99.6% of its build runs unattended.

Digging into the model breakdown behind those runs surfaced something I didn’t plan to build: a 1,199-room randomized controlled trial across frontier models.

Two weeks ago, I got tired of one provider’s weekly quota pool hitting its buffer on Wednesday while another pool sat half-empty. Inside Leaf’s configuration, I left the pm and architect seats set to model: null. Whenever a new planning room opens, the dispatcher selects uniformly at random from every runnable, unbuffered Class 5 frontier model across my connected harnesses (claude, codex, and antigravity).

I did it purely to smooth out subscription burn. Tonight, when I joined planning_pod_run back to the downstream build rooms that executed each plan between October 29 and November 12, I realized that uniform randomizer had dealt nearly 400 live planning rooms to each of three frontier models, followed by 3,122 downstream engineer pods trying to build what they designed.

The nuance: designing a contract isn’t the same skill as writing the diff

Every time a new model drops, social media argues about whether Model A or Model B is “the best coding model,” usually backed by a single-agent benchmark or a vibe check in a terminal.

Single-agent benchmarks grade a soloist.
Software factories grade the handoff.

Before looking at a single cross-model table in Leaf, one methodology rule from our internal retro skill matters: never compare turns across providers. In our container bridge, the Claude harness increments turns on every assistant message, while the Codex harness counts an entire ask -> result episode as one turn. Comparing raw turn counts across them is apples and lug nuts. Cross-harness comparisons have to stick to wall-clock medians (ms), output tokens, input-equivalent tokens (IET), and Tech Lead send-back rates.

Once you measure the handoff with honest units, “which Class 5 model is best at coding?” splits into two completely different answers depending on which chair the model sits in:

Two-panel data comparison from Oct 29 to Nov 12, 2026: Left panel shows the 1,199-room randomized Architect trial where Claude Opus 5.5 plans yielded 6.0% all-loop downstream rework (20.7% on Dev, 1.47 revisions across 400 rooms) versus Codex GPT-6.1-Sol at 11.0% (26.5% on Dev, 408 rooms) and Codex GPT-5.6-Sol at 13.2% (37.6% on Dev, 391 rooms). Right panel shows Class 5 Engineer performance led by GPT-6.1-Sol at 12.5% rework (75% first-try) and Opus 5.5 at 25.8% rework on 2.35x larger diffs (score 1.37), plus Measured Precedent defaults (12.4% rework) beating downward Architect overrides (29.1% rework).

1. In the Architect chair (1,199 randomized rooms -> 3,122 builder pods)

In a planning room, the Architect reads the repository, checks historical check times and area costs, negotiates scope with the Product Manager (PM), and saves a DAG of node contracts (files, done_when, check, and class). Looking at what happened downstream when engineer pods tried to implement those plans:

  • claude-opus-5-5 (400 rooms, 1,111 downstream engineer pods): Needed an average of 1.47 plan revisions (329.5s wall time, 473K IET) and produced the cleanest handoffs in the factory: 6.0% downstream engineer rework across all loops, and 20.7% downstream rework on human Development builds (87 builds). (On the PM seat across 1,201 randomized rooms, Opus 5.5 was also 45% faster and 3.2x cheaper in IET than GPT-6.1-Sol, finishing in 41.4s and 92.5K IET vs. 75.1s and 299.8K.)
  • codex/gpt-6.1-sol (408 rooms, 987 downstream engineer pods): Averaged 1.63 revisions (389.8s, 468K IET), with downstream builders hitting 11.0% rework overall and 26.5% on Development (78 builds).
  • codex/gpt-5.6-sol (391 rooms, 1,024 downstream engineer pods): Averaged 1.76 revisions (299.7s, 561K IET), with downstream builders hitting 13.2% rework overall and 37.6% on Development (81 builds).

Across more than a thousand rooms, having Opus 5.5 write the specification cut overall downstream builder rejections nearly in half (6.0% vs. 11.0%–13.2%) because it names tighter file boundaries and runnable check commands that leave fewer traps for smaller Class 2 and Class 4 builders.

2. In the Class 5 Engineer chair (63 heavy multi-module Development nodes)

Now move those exact same models out of the planning room and put a hard hat on them as Class 5 Engineers (100% human Development nodes spanning multiple modules):

  • codex/gpt-6.1-sol (n = 16): 75.0% first-try pass rate and 12.5% rework in 1,042s median wall time and 609K median IET (1,206 average changed lines per node).
  • codex/gpt-5.6-sol (n = 16): 25.0% first-try pass rate and 75.0% rework in 966s median wall time and 2,449K median IET (987 average changed lines per node).
  • claude-opus-5-5 (n = 31): 51.6% first-try pass rate and 25.8% rework in 1,362s median wall time and 2,712K median IET (2,839 average changed lines per node, 88,008 lines total).

Two things jump right out of those numbers.

First, look at the generational whiplash inside the exact same Codex harness between gpt-5.6-sol and gpt-6.1-sol. On heavy multi-module work, gpt-6.1-sol slashed Tech Lead rework by 6x (from 75.0% down to 12.5%) and cut median token burn by 4x (2.45M IET down to 609K). Static routing rules age like milk when a point release flips a tier from worst-in-class to cleanest-hands-in-the-room.

Second, notice the crucial practitioner caveat between gpt-6.1-sol and opus-5-5 as builders. Unlike the randomized planning rooms, engineer spawns aren’t uniform across every task shape (and n = 16 vs. n = 31 is a modest two-week sample). Why did Opus 5.5 earn a much better sabermetric node_score (1.37 vs. 4.87, where 1.0 is repository baseline per changed line) despite double the rework rate? Because Opus 5.5 got handed refactors averaging 2,839 changed lines, more than 2.3x the size of gpt-6.1-sol’s nodes. GPT-6.1-Sol was a surgeon on 1,200-line multi-module nodes; Opus 5.5 was a bulldozer on 2,800-line rewrites.

Shaking up Classes 1 through 4

Below Class 5, playing Moneyball with live scorecards forced three roster moves over the last two weeks (3,074 planned Class 1–4 engineer nodes since October 29):

ClassModelNodes (n)Dev NodesFirst-Try Pass %All Rework %Dev Rework %Median Wall TimeMedian IET
Class 1openrouter/deepseek/deepseek-v4.1-flash852195.3%3.5%0.0%87.4s113.2K
Class 1openrouter/z-ai/glm-5.3-flash43293.0%7.0%0.0%112.3s86.2K
Class 1local/factory (retired)25280.0%16.0%50.0%160.7s72.7K
Class 2claude-haiku-5-52,43516694.5%5.1%16.9%27.3s65.6K
Class 3antigravity/gemini-3.8-flash (demoted)1596155.3%44.7%67.2%525.7s1,260.5K
Class 4claude-sonnet-5-531326670.9%22.7%24.4%194.2s292.2K
  • OpenRouter flash models took over Class 1: When every detail is already decided in the brief, deepseek-v4.1-flash landed 95.3% of its nodes on the first try (and went 21-for-21 with 0.0% rework on Development nodes) in 87.4s, cutting rework by more than 4x compared to my local model (16.0% rework, 160.7s).
  • Haiku 5.5 is still Scott Hatteberg: Across 2,435 nodes in 14 days, Class 2 finished in a median of 27.3 seconds at 65.6K IET with a 94.5% first-try pass rate.
  • The Class 3 trap claimed another victim: In late October, Class 3 (antigravity/gemini-3.8-flash) hit 67.2% rework on Development (525.7s median wall time, 1.26M IET) whenever a brief asked it to search a directory on its own. We demoted gemini-3.8-flash down to Class 2 (where briefs name the exact files and pattern) and rotated fresher models into Class 3.

Sabermetrics v2: when the Architect overrides the data, rework doubles

In October, I introduced two mechanisms to automate model routing: a graph.rollups precedent table that the Architect read by prompt instruction, and an automatic 1-in-10 minor-league call-up (node_trial) that ran every tenth Class 3+ node one tier lower.

Both needed tightening once real volume hit them.

First, blind 10% call-up trials put complex nodes onto models too weak for them. Between October 25 and November 1, 39 Class 4 nodes got automatically trialed on Class 3 and racked up 29 Tech Lead rejections (0.81 send-backs per merged node). We patched two guards into src/plan.trials.ts: an Architect now has to opt a node in with trial: true (allowed only when the intent leaves zero open design choices), and if a lower-class trial gets rejected, blocks, or stalls, Factory automatically retires the trial pod and respawns the replacement at its full planned class on the existing branch.

Second, asking an LLM Architect to interpret a rollup table in prose invited hubris. Last week (commit c2a39960), we moved default tier selection out of prompt advice and into deterministic code (src/plan.forecast.ts).

Now, whenever an Architect saves a plan, Factory computes a measured default {class, effort} for every node: the lowest class and effort with at least 5 historical runs in every touched codebase area whose send-back rate sits at or below the repository median. Factory also freezes a forecast (minutes, tokens, sendbacks, and critical path). Across 293 scored nodes so far, portfolio forecast bias is -2.2% (-4.5s) on wall time and +3.4% (+1,604 tokens) (individual nodes still swing by ~52% median absolute error, which is normal task variance, while the batch total lands right on the pin).

If the Architect disagrees with the measured default, code requires an explicit override: {class, effort, why} object. Here’s what happened across 175 nodes with a measured default since November 5:

Routing Choice (since Nov 5)     Nodes   Median / Avg Time   Rework %   Avg Score
------------------------------   -----   -----------------   --------   ---------
Kept measured default               97    2.44m /  6.58m       12.4%        4.88
Architect overrode DOWN a tier      55    6.63m* / 9.35m       29.1%        9.10
Architect overrode UP a tier/eff    23    6.63m* / 24.5m       47.8%        2.15
(*6.63m is the combined median across all 78 overrides; lower score = cheaper per line)

Look at the middle row. In 55 nodes, the Architect looked at a directory whose history demanded Class 4 or Class 5, wrote a confident one-sentence why explaining that this edit was straightforward, and overrode the node down to a cheaper class. Rework more than doubled from 12.4% to 29.1%, average wall time rose 42%, and the class-weighted cost score per changed line nearly doubled from 4.88 to 9.10. (Conversely, the 23 times the Architect overrode up on massive 10k-line files, rework was high because the tasks were gnarly, yet per-line node_score came in at an efficient 2.15.)

LLMs are just like human tech leads when estimating tickets: they’re decent at spotting a monster rewrite that needs extra firepower, and terrible at talking themselves into "this should only take five minutes."

Next up in the final part of today’s trilogy: Checking My Dark Factory’s Receipts After 4,000 Runs, where we test whether cutting 65% of our prompts and adding Dream telemetry survived eight days of full-fleet traffic.

So what?

If you’re routing work across multiple models and harnesses in 2026, build your scorecard around the handoffs:

  • Grade planning models by downstream builder rework. Don’t evaluate an Architect model by how pretty its markdown plan looks. Join its plans to the builders that executed them (Opus 5.5 cut downstream rework to 6.0% overall vs. 11.0%–13.2%), and never assume your cleanest builder is automatically your clearest spec writer.
  • Never compare raw turns across harnesses. Different CLIs and SDKs count turns at completely different granularities. Compare wall-clock medians, output tokens, cache-adjusted IET, and reviewer send-backs.
  • Compute defaults in code, and log every downward override. Don’t let an LLM talk itself into a cheap model tier when historical directory data says otherwise. Set the default mechanically from past send-back rates, require a logged override to depart from it, and escalate failed lower-tier trials automatically.