Moneyball for AI Agents: Stop Buying Models, Buy Runs

Over the last 50 days, Leaf, my local dark software factory, has burned through 15.8 billion tokens (2.84 billion input-equivalent tokens, including 110.3 million output tokens) across 208,649 model turns, 8,853 spawned agent pods, and 2,959 plan runs.

More than half of that volume (9.5 billion tokens across 1,779 plan runs) happened in just the 19 days since I published Part 4 of the Loops series. Over the last two weeks alone, the factory chewed through 7.3 billion tokens.

When you run that kind of volume through a closet of NUCs in your home office, you stop caring about how a model performs in a marketing demo. You care about two things I’ve been hammering on all month: keeping context brutally small (why giant interactive skill repos won’t survive unattended 2027 loops) and putting the right model on the right job instead of throwing every prompt at the biggest frontier model on the menu.

That second idea, using the right model for the job instead of throwing every prompt at the biggest model on the menu, is having a big moment right now. You see it everywhere in discussions around subagent delegation and model routers like LiteLLM.

Most of those conversations still frame model routing as a coupon code for your API bill. Send the easy prompts to a cheap model, send the hard prompts to a frontier model, and save 40% on your monthly invoice.

That framing misses the real lesson. We’re having the exact same argument the Oakland A’s front office had in Moneyball:

In that clip, Peter Brand points out why traditional baseball scouts keep losing to richer teams. They sit around a table talking about how a player looks in a uniform, how fast he runs, or how pretty his swing is. They try to replace a $7.5 million star like Johnny Damon with another star who looks like Johnny Damon. Brand cuts through the noise with one line: “Your goal shouldn’t be to buy players, your goal should be to buy wins. And in order to buy wins, you need to buy runs.”

In agentic engineering, we’re still sitting at the old scouts’ table. We’re buying players instead of buying runs.

The nuance: we’re optimizing for what the human sees

Too often with agentic design, we optimize for what the human sees in the terminal.

When a developer sits in an interactive Level 3 chat session, we judge a model the way an old baseball scout judges a prospect’s jawline. Does the response look smart? Did it write a confident, four-paragraph architectural explanation before touching the file? Where does the model sit on SWE-bench or whatever contrived leaderboard dropped on social media this morning?

Benchmarks test a naked model in a vacuum.
Workflows test a system under load.

The measure of a model isn’t what it does on SWE-bench. It’s what the model does inside your workflow when it has to generate quality output without a human holding the steering wheel.

This is why static model routing (even with a great mechanical tool like LiteLLM) falls short when you only route by prompt length, keyword heuristics, or API price per million tokens. A router that looks at a pricing page and swaps in a “cheap, high-benchmark” model has no idea if that model just spent 45 turns re-reading the same three files, invented an unasked-for abstraction layer, or had its diff rejected three times by the Tech Lead agent downstream.

If you want to play Moneyball in an AI-driven software delivery lifecycle (AI-SDLC), you have to stop grading the model’s swing and start counting how often the workflow gets on base.

What “buying runs” looks like in Leaf

Inside Leaf, I don’t care how eloquent a worker agent sounds in a chat log, because I don’t read the worker chat logs. I care about whether the factory ships working, maintainable software while I’m doing something else.

Through today, Leaf has finished 2,950 plan runs across my onboarded repositories, with 2,934 confirmed merges (99.5%) and 7,562 completed worker nodes. To keep that loop honest without human code review, I measure three workflow outcomes:

1. Tests and classical CI/CD validations passing (the mechanical gate). Moving to an AI-SDLC doesn’t mean throwing out classical engineering gates. Every node in a plan carries a done_when contract and a scoped check command, and as I covered in Loops Part 5, a worker cannot report done until that check passes after its final edit. But the node check is just the entry ticket. Every repository defines its own custom CI/CD pipeline in .factory.json, and those classical validations are enforced on every single run before anything lands or deploys. In brandcast, for example, every plan run has to clear nine deterministic CI jobs (type-gate, changed-scope, deps-overrides, type-check, manifest and shell linters, and script tests), container image build checks, and staging gates like trivy vulnerability scans, Playwright Chromium/WebKit E2E suites, and post-deploy health probes. Over the last seven days, those repo-specific CI pipelines ran 724 times and finished green 718 times (99.2%).
2. Rework and Tech Lead send-backs. When a worker reports done, the Tech Lead (foreman) reviews the diff strictly against the node’s contract. If the worker missed a requirement or touched something it shouldn’t, the Tech Lead runs reject <pod> with the defects. Leaf logs every rejection in a send_back table and charges both the worker’s retry tokens and the Tech Lead’s review tokens against that node’s scorecard.
3. AST simplicity scoring. Passing tests aren’t enough if the agent buried a twenty-line fix under three layers of defensive ceremony. Leaf measures structural simplicity directly from the syntax tree.

Those three numbers are my software delivery lab, but the principle applies anywhere. If your agentic workflow handles customer triage, data extraction, or compliance audits, your “runs” are whatever downstream checks prove the work landed cleanly without a human cleaning up the mess.

The Hatteberg tier and the Damon trap

When you stop routing models by leaderboard rank and start scoring them by rework and landed lines, the lineup changes fast.

In Leaf, the Architect/Planner never picks a specific model string like claude-opus or gemini-flash. It assigns each node a Class from 1 to 6 based on how much open judgement the brief leaves to the worker:

• Class 1 (local via OpenCode): Every detail is decided in the brief.
• Class 2 (Haiku / gpt-oss-120b): The pattern is already in the repo tree, and the brief says where.
• Class 3 (Gemini Flash / Codex light): The brief has decided the work, and the worker searches a small area.
• Class 4 (Sonnet / Codex standard): The worker has to find the pattern across the repository tree (the default).
• Class 5 (Opus / Codex heavy): The worker has to work out how changes fit across several modules.
• Class 6 (Frontier max): Intricate cross-module work where a mistake breaks downstream contracts.

Look at what happened across the 3,898 unattended worker runs over the last two weeks (October 15 to October 29) once those classes hit the field:

Two patterns jump right off the chart.

First, Class 2 is the Scott Hatteberg of my factory. Across 2,169 runs in the last two weeks, Class 2 landed 98.6% of its nodes on the very first try, with a 1.5% rework rate overall (and just 5.0% rework on Development feature and bug nodes). Its median wall time was 31 seconds at 66,000 input-equivalent tokens (IET). Why does a small, cheap tier perform so well? Because when the planning loop spends the judgement upfront to name the exact file paths, the existing pattern in the tree, and the runnable test check, the worker doesn’t need frontier reasoning to get to first base.

Second, look at Class 3. On a provider pricing page or a benchmark blog post, Class 3 models look like the ultimate sweet spot: fast, modern, and dirt cheap per million tokens. If you configured a static router like LiteLLM to send “medium-easy” coding tasks to Class 3, you’d pat yourself on the back for saving money.

In my actual workflow, Class 3 is the Johnny Damon trap.

On Development feature and bug runs over the last two weeks (n = 263), Class 3’s first-try pass rate dropped to 84.0%, with a 17.1% rework rate and a median cost of 558,000 IET. That’s more than double the token cost and nearly triple the rework rate of Class 4 above it (93.9% first-try, 5.9% rework, 227,000 median IET across 574 Development runs). Why? Because one of the Class 3 models spins for a median of 49 turns re-reading files to build confidence, while the other cuts corners on multi-file edits and gets its diff rejected by the Tech Lead.

To be fair to the models, this isn’t a controlled academic benchmark where every tier runs the exact same prompt. Class 5 (90.0% first-try, 11.7% rework, 610K IET) gets handed the nastiest multi-module problems in the factory, while Class 2 gets the cleanest briefs. And since most of my usage runs on flat monthly subscriptions and a local closet of NUCs, my IET math (input + 1.25×cache_w + 0.10×cache_r + 5×output) is a way to measure work and quota pressure, not an itemized AWS invoice. Even so, watching a “cheap” Class 3 node chew through half a million tokens and three Tech Lead rejections will cure you of leaderboard worship pretty quick.

Teaching the factory to run its own sabermetrics

In Part 4 of the Loops series, I noticed early signs of that Class 3 routing quirk by querying SQLite by hand. I didn’t want to keep playing general manager with a spreadsheet every Sunday. I wanted the AI-SDLC to adjust its own lineup from live evidence.

So I built a sabermetric scorecard (src/scorecard.ts) and a minor-league call-up system (src/plan.nodes.ts) directly into Leaf.

1. Scoring every merged line (scorecard.ts)

Whenever a Tech Lead merges a worker’s branch, Leaf computes a permanent node_score for that pod. It doesn’t just count the worker’s tokens. It sums the worker’s tokens plus the tokens of every Tech Lead turn that reviewed, merged, or rejected that worker, multiplies that total by the class number (so Class 4 tokens carry 4x the weight of Class 1), and divides by the number of changed lines:

const ms = Date.parse(at) - Date.parse(start);
const tokenCount = (tokens.get(pod) ?? 0) * cls;
const lines = Math.max(1, merged);
const sendbacks = (db.prepare("SELECT COUNT(*) AS n FROM send_back WHERE room = ? AND pod = ?")
  .get(room, pod) as { n: number }).n;

const rows = db.prepare("SELECT ms, tokens, lines FROM node_score WHERE repo = ?")
  .all(repo) as Array<{ ms: number; tokens: number; lines: number }>;
rows.push({ ms, tokens: tokenCount, lines });
const medianMs = median(rows.map((r) => r.ms / r.lines));
const medianTokens = median(rows.map((r) => r.tokens / r.lines));
const score = (medianMs ? time * (ms / lines / medianMs) : 0)
  + (medianTokens ? tokenWeight * (tokenCount / lines / medianTokens) : 0);

A score of 1.0 means the node cost the repository’s typical median time and class-weighted tokens per changed line. Lower is better.

When the Planner saves a draft plan for a repository, Leaf rolls up those historical scores by codebase directory (area -> parent -> repo) across every class and hands them back in the save response as graph.rollups. The Planner’s prompt rule is explicit:

Read graph.rollups after each save and assign each node’s class from it: the lowest class whose score and send-backs per node in that area are good. Many send-backs mean the class is too low there, and a higher class that scores no better is too high. When the rollup says so, change a class and save again.

Look at what the live graph.rollups table shows right now across factory, brandcast, and calamus:

Repository / Area	Class 2 Score (Send-Backs / Node)	Class 3 Score (Send-Backs / Node)	Class 4 Score (Send-Backs / Node)
factory / src	1.01 (0.07 sb, n=255)	6.25 (0.47 sb, n=15)	0.76 (0.26 sb, n=97)
factory / public/app	1.57 (0.03 sb, n=124)	7.29 (0.86 sb, n=7)	0.91 (0.18 sb, n=45)
brandcast / stripe & repo	1.10 (0.15 sb, n=27)	33.96 (0.75 sb, n=12)	0.84 (0.67 sb, n=9)
calamus / tests	1.32 (0.12 sb, n=57)	5.93 (0.64 sb, n=11)	0.85 (0.29 sb, n=82)

In brandcast, a Class 3 node costs 33.96x the repository baseline per line and averages 0.75 Tech Lead send-backs per node. In the exact same codebase, Class 2 scores 1.10 (0.15 send-backs) and Class 4 scores 0.84. Because the Planner reads graph.rollups on every save, it sees those numbers automatically and routes around the trap without me touching a config file.

2. The 10% minor-league call-up (node_trial)

If a Planner always picks the safe class that worked last week, how does the factory ever learn that a cheaper model class could handle a directory?

It runs automatic call-up trials. In src/plan.nodes.ts, every tenth eligible worker spawn planned at Class 3 or higher automatically runs one class lower:

/** Every tenth eligible first spawn of a repository runs one class lower; the row keeps the choice across retries. */
function trialClass(room: string, repo: string | null, node: Node): { class: number; planned: number; trial: number } | null {
  const planned = node.class ?? DEFAULT_CLASS;
  if (node.model || planned < 3 || !repo) return null;
  const row = db.prepare("SELECT planned_class AS planned, class, trial FROM node_trial WHERE room = ? AND node = ?")
    .get(room, String(node.id)) as { planned: number; class: number; trial: number } | undefined;
  if (row) return row;
  const { n } = db.prepare("SELECT COUNT(*) AS n FROM node_trial WHERE repo = ?").get(repo) as { n: number };
  const trial = +((n + 1) % 10 === 0 && runsOn(planned - 1) === planned - 1);
  const chosen = planned - trial;
  db.prepare("INSERT INTO node_trial (room, node, repo, planned_class, class, trial, at) VALUES (?, ?, ?, ?, ?, ?, ?)")
    .run(room, String(node.id), repo, planned, chosen, trial, now());
  return { class: chosen, planned, trial };
}

If the Planner asks for Class 4 in public/app, every tenth node gets handed to Class 3 or Class 2 instead. If the lower class nails the check with zero send-backs, its node_score feeds into graph.rollups, and future plans shift work down to the cheaper tier in that directory. If it faceplants and gets sent back by the Tech Lead, the rollup score spikes and the Planner keeps that directory at Class 4.

3. Measuring simplicity with tree-sitter instead of vibes (simpler.ts)

When I wrote Loops Part 3 earlier this month, I admitted a gap in Leaf’s autonomous Improvement loop. It was great at deleting dead, unreachable code, but it couldn’t reliably simplify code that was over-engineered yet still in use. I had tried a raw line-count gate earlier, and it was too blunt: an agent can golf five readable lines into one hideous nested ternary just to make a line counter go down.

To fix that, I built a deterministic cognitive complexity scorer (src/complexity.ts) using a vendored Tree-sitter parser across TypeScript, JavaScript, Python, Kotlin, Bash, Svelte, and Astro, applying Sonar’s cognitive complexity rules. Every nested loop, conditional branch, catch block, and mixed boolean operator chain adds to a function’s cognitive score, with deeper nesting penalized more heavily.

Now, when an Improvement examiner agent wakes up on a file, its brief lists the exact cognitive complexity score of every function in that file. To file a reduction issue, the examiner has to make its edit in an isolated worktree and pass two mechanical gates:

1. Remove dead code: what nothing reaches, and an implementation another one duplicates.
2. Simplify what survives: a branch nothing takes, a layer that only forwards, state kept twice,
   a special case no caller or test asks for.
3. Run the focused tests that cover the file. There is no browser.
4. Run `node /opt/factory/src/simpler.ts` in the worktree with your change. It passes only when
   the total score of the scored files you touched fell.

Report a `reduction` only when the tests and `node /opt/factory/src/simpler.ts` both pass.
Otherwise answer `nothing`.

Once five verified reduction issues accumulate in a repository, Leaf bundles them into an Improvement PRD (PRD: Improve src) and hands them to the workers. Because the simplification is already proven against both the test suite and simpler.ts, 77.5% of all Improvement worker nodes (1,911 of 2,466 since October 15) run on Class 2, landing with a 99.0% first-try pass rate in a median of 31 seconds.

Across my onboarded repositories, Leaf has already closed 1,814 verified complexity reductions (1,240 in brandcast, 385 in factory, 139 in binpacker, and 50 in calamus), and merged 1,900 Improvement plan runs alongside 882 Development plan runs. Over two-thirds of the factory’s merged runs exist purely to pay down cognitive complexity that earlier features added.

So what?

If you’re building or running an agentic workflow, stop acting like the old scouts in Moneyball:

• Stop picking models off SWE-bench. A benchmark scores a model’s swing in an empty stadium. Your harness, your context size, and your test gates determine whether a model actually gets on base in your codebase.
• Don’t route by API price alone. Model routers like LiteLLM are great plumbing, but routing by token price without tracking downstream rework is a trap. Always charge Tech Lead rejections, retry turns, and review tokens back to the model tier that caused them.
• Run 10% call-up trials. Don’t hardcode model tiers by gut feel. Automatically run a small slice of your scoped tasks one tier lower, record the send-back rate per codebase directory, and let the data promote or demote models.
• Gate cleanup on AST complexity, not line count or LLM vibes. Never ask an LLM if code “looks cleaner.” Parse the syntax tree with Tree-sitter, score cognitive complexity, and require both a green test suite and a lower complexity score before a simplification can land.

In Moneyball, Peter Brand tells Billy Beane that players get overlooked for all kinds of biased reasons because teams fall in love with the spectacle. Agentic AI has the exact same bias. Stop buying the spectacle, and start buying runs.