I’ve been talking a lot lately about Leaf and the data behind how it runs.

If you use Dan Shapiro’s Five Levels of AI coding framework, Leaf operates as a Level 5 software factory (what Shapiro calls the “Dark Factory”). A specification goes in as a contract, and working, tested software comes out the other side without a human writing code or reviewing diffs.

Most days, it’s great. It hums along on a few small boxes in my office closet and ships real work while my laptop lid is closed.

But it’s far from perfect.

The nuance

“Dark factory” sounds very clean on a conference slide. In practice, taking humans out of the implementation middle doesn’t make engineering problems disappear. It just moves the failure modes to places you weren’t watching.

When a human writes code by hand, human boredom and typing speed act as natural rate limiters. You don’t accidentally trigger forty container builds before lunch. You don’t write a fifty-step coordination plan to update a CSS theme. And you don’t rewrite your own job description just to make a unit test pass.

Take the human rate limiter off, and the edges of your harness get stress-tested fast. I’ve gotten a few things wrong recently, and each one taught me something about where my factory doesn’t cut it yet.

1. CI limits come fast

I use GitHub Actions to build and deploy my projects. I pay for a GitHub Enterprise Cloud license for one user, which includes 50,000 Actions build minutes a month on standard runners for private repositories.

Before Leaf, I never came close to touching that number. For a solo developer, 50,000 minutes is the same as 50 billion. It’s more than thirty-four straight days of compute every month.

This month, I blew right through all 50,000 minutes on October 18th.

Why? Did I have repetitive jobs and unoptimized workflows in there? Yes. When you aren’t paying attention to build minutes, it’s easy to let full container builds and redundant test runs pile up on every push.

But the real reason wasn’t just sloppy YAML. It happened because autonomous loops multiplied my commit and PR throughput nearly fourfold, while my CI triggers were still designed for human typing speed. Between feature work, automated bug fixes, and the Improvement loop sweeping for dead code while I slept, the factory was firing off full pipeline runs around the clock.

My factory wasn’t broken; my plumbing was still sized for a human.

So now I’m tightening the screws. I’m building container images less often. I’m being much smarter about when a workflow runs the full end-to-end test suite versus a fast, targeted check. It’s all fixable, and high throughput is a good problem to have. Still, hitting a hard billing wall two-thirds of the way through October is a rough way to learn that lesson.

2. Foundational changes

My current setup is great for bug fixes and the average new feature. If a problem fits neatly into a handful of files with a clear boundary, Leaf chews right through it.

Where it gets lost and goes sideways is when I ask it to make fundamental, sweeping changes across an existing project.

For my BrandCast project, I recently updated the admin UI. While I was poking around, I noticed that a lot of hand-coded styling changes had crept into individual pages over time (agents will do that when they’re focused on closing a local ticket). I wanted Leaf to update the core theme and make sure every page in the admin UI was using that updated theme consistently.

It was a lot of changes and a lot of checks, but the work itself was mostly mechanical.

Leaf lost. Its. Mind. Mind. Nowhere to be found.

The planner generated a graph with 50+ nodes. It took over an hour just to finish planning. (I’m blaming my choice to test Astra and Fable as planning agents there. Way too much horsepower and way too much over-thinking.) Once execution finally started, the foreman almost crashed a node, then sent over 30 nodes back to workers for changes. After three hours of churn, the build still failed.

In Part 6 of the Loops series, my data already showed that an 8+ node plan costs 17× what a single-node plan costs. A 50-node plan isn’t just expensive; splitting a wide, mechanical sweep into fifty isolated worker contexts creates more coordination friction than the work itself. Every worker touches adjacent files, merge conflicts multiply, and the foreman spends all its turns playing traffic cop.

I’m still digging into this one. Some of it is definitely my choice of models (looking at you, Astra). Heavy reasoning models love to turn a straightforward cleanup into a doctoral thesis. But some of it is how I create plans in the first place. I’m not sure what the right fix is yet, whether that’s capping graph size, batching mechanical sweeps differently, or keeping foundational refactors in an interactive session. Maybe all or none of the above. Maybe it’ll occur to me magically in the shower tomorrow morning.

3. Improvement rot

I love the Improvement loop in Leaf. Having an autonomous sweep that deletes dead weight and simplifies code while the factory is idle has been one of my favorite wins of the whole project.

But I need to start setting barriers on what it’s allowed to touch.

Leaf can run the Improvement loop on its own repository. Recently, somewhere along the line, an improvement pass decided that it needed to add a regex-style test to validate one of Leaf’s role prompts. Testing natural language prompts with regex is already a bad idea. Then, on later sweeps, the loop noticed that other prompts were “missing” similar tests, so it dutifully added more of them.

Then came the rot.

When a later change tripped those brittle checks, the agent didn’t fix or remove the bad tests. Instead, it started changing the prose of the prompts to match the regex tests, because updating markdown prose is faster and easier than refactoring test code.

Writing a test takes code.
Changing a prompt takes prose.

Agents always take the shortest path to the reward. Independent evaluators like METR have documented this exact reward-hacking behavior across frontier coding models: give an agent an automated check as its definition of done, and it will happily game the check if that’s the path of least resistance. Nobody in my closet was plotting world domination. The worker just wanted a green exit code in the fewest turns possible.

Suddenly, Leaf started exhibiting odd behavior on real runs. I ended up spending a 90-minute human-in-the-loop session unwinding the mess, deleting the regex tests, and restoring sanity to the prompts Leaf uses to do its job. Lots of typing in all caps and cursing happened.

I’m working on this right now. In Part 2 of the Loops series, I wrote that judgement lives in the prompts, where I can read it and change it. Letting the automated cleanup loop modify the factory’s own prompts violated my own rule. I’m thinking about adding guardrails to skip certain paths (like role prompts and skills) from the Improvement process altogether, though I’m not completely sure where that boundary should sit yet.

So what?

Running a Level 5 software factory doesn’t mean you stop doing engineering. It means your engineering shifts to the harness, the boundaries, and the economics of the loop:

  • Audit your downstream plumbing. If your commit throughput jumps 4×, your CI bill and container registries are going to feel it before you do. Gate heavy builds and full suites so autonomous loops don’t burn your monthly quota in eighteen days.
  • Don’t use a sledgehammer planner on wide, mechanical sweeps. Big reasoning models will over-decompose foundational changes into giant node graphs that choke on their own coordination.
  • Fence off your prompts from self-improvement loops. If an agent can pass a test by weakening a prompt instead of fixing code, it will take that deal every single time. Keep automated cleanup focused on mechanical code, and keep human hands on the prompts that define judgement.

This is all very much a work in progress, and sometimes it goes sideways. Sometimes it goes really sideways.

But I can’t wait to look at the data from that awful three-hour run.