Inside My Dark Software Factory: How Leaf Runs Locally

When I first introduced Leaf a little over a month ago, I hedged on what to call it. I tried out “multi-agent coding orchestrator,” “fleet manager,” and “meta-harness.”

I’m done hedging. The explicit goal of Leaf is a Dark Software Factory.

In Dan Shapiro’s Five Levels of AI coding framework, Level 5 is the dark factory: a black box that turns specifications into working, tested, deployed software without a human writing code or reading diffs.

Building toward that goal forced me to confront the central question of unattended software engineering: if a human isn’t reviewing the diffs, what stops an agent from hallucinating a mess and shipping it to production?

The nuance: purposeful friction

In the end, the only real tool that isn’t some flavor of human review is adversarial validation.

Give a single coding agent a broad prompt, full repository context, and an open loop, and it will eventually wander afield. It will refactor things you didn’t ask it to touch, invent abstractions it doesn’t need, and introduce large breaking changes because nothing in the room pushed back.

In Part 1 of the Loops series, I wrote about separating judgement from mechanics. The next step in Leaf’s evolution was introducing purposeful friction inside the judgement layers, while keeping the mechanical gates strictly deterministic.

No agent in Leaf works alone or validates its own outcome. Every step pairs a producer with an adversarial counterweight, and every role only sees the exact slice of context it needs to do its job.

The workflow: I’m the product owner now

In the current version of Leaf, I don’t act as a tech lead or a release manager. I am the Product Owner. It’s my vision that is being brought to life.

Every phase of agent collaboration happens inside its own dedicated WebSocket chat room, structured around adversarial checks and strict context boundaries:

1. Human + Skill file the issue. I work with my Leaf skill in an interactive session to pressure-test my own thinking, scope the problem, and file a local issue. That issue goes straight to a Product Manager (PM) agent, and a planning WebSocket chat starts.
2. PM + Architect develop the plan. Once the PM judges the issue ready, an Architect agent joins the WebSocket chat. The PM and the Architect work together to turn the requirement into an implementation plan and a dependency graph. Once the graph is locked, the PM and Architect both drop off. This loops sometimes 3-4 times before a plan that fulfills the goals of both agents gets created. Just like real life.
3. Engineers write code, and the Tech Lead validates all of it. A Tech Lead agent starts in a new WebSocket chat and spawns the Engineer agents. The Engineers do the implementation work per the plan nodes, and the Tech Lead validates every piece of work before it’s allowed to land. Just like the real world, the Tech Lead can send inferior work back.
4. CI is a deterministic task, not an agent. Once the code lands, CI starts. CI steps are tasks, not agents. It’s the first programmatic check, running the exact same linters, type checks, and test suites CI runs today.
5. QA Engineers answer to the Tech Lead too. If CI fails, a QA Engineer agent is spawned to fix it. Crucially, the QA Engineer doesn’t get a free pass to hack the tests until they turn green: QA Engineers join the WebSocket chat with the Tech Lead, who validates their work as well before CI re-runs. That loop repeats until CI passes.
6. CD is a deterministic task, not an agent. Once CI is green, CD starts and deploys the build without an LLM anywhere in the execution path. It’s all mechanical. We’re just deploying bits at this point.
7. Tech Writers summarize what landed, and a Lead Writer runs QA. Once CD is green, a Tech Writer agent starts in its own WebSocket chat to update the repository’s internal wiki.

Why only the PM gets the lore

Two details in that knowledge loop matter in practice.

First, each repository gets its own wiki, stored outside the source code, and only the PM role has it mounted as part of the run. The PM gets the deeper lore: why a service was designed a certain way, how features connect, and what shipped last week. The Engineers don’t get the wiki mounted.

That separation is deliberate. An Engineer agent doesn’t need the philosophical history of the codebase to implement a twenty-minute node. Mounting the deep lore into an Engineer’s context just invites it to over-think local changes and touch files outside its scope. The PM and Architect use the lore to shape the plan; the Engineers only see the contract and the code.

Second, the Tech Writers aren’t synthesizing new work. Left to “write documentation,” an LLM will happily invent architectural manifesto prose. Instead, Tech Writers strictly aggregate and summarize what actually landed from commit logs and changesets. And to keep the wiki itself from drifting, a Lead Writer role runs hourly in what is primarily a QA role, auditing and validating what the Tech Writers wrote.

Even the documentation loop has an adversary checking its work.

Bringing the plumbing home

Last week I wrote about blowing through all 50,000 of my monthly GitHub Actions build minutes by October 18th. My factory’s autonomous commit throughput had quadrupled, and it ran straight into a CI billing wall built for human typing speed.

At first, I thought about trimming my YAML workflows. Instead, I brought the whole stack home.

Version control, issues, CI, CD, and the repository wikis are all local to Leaf now.

Pulling the entire stack inside Leaf solved the 50,000-minute ceiling over a weekend, because every test run and container build now executes on my own local fleet without paying a per-minute cloud runner tax. (To be fair, running your own compute isn’t free either, since my office closet still draws power and wears down SSDs. Still, my power utility hasn’t threatened to shut off my heat on the 18th of the month.) I did invest in a 4th factory node to allow for extra room. My current factory setup:

• 3 older i7 NUCs with 32GB of RAM (approximately 5 years old)
• 1 GMKTec NUC clone with an AMD Ryzen 7 and 32GB of RAM

The Factory UI and other services sit on another node. All four nodes are dedicated wholly to running work through the factory.

The bigger win wasn’t the bill. It was telemetry.

When your issues live in one SaaS tool, your git branches live in another, your agent logs sit on a local disk, and your deployments run in a third-party action runner, stitching together what actually happened on a sideways run is a forensic nightmare. Because Leaf now owns the entire lifecycle locally, I can trace every single touch, model turn, WebSocket chat message, test failure, and deployment log from the moment an issue is opened to the moment CD turns green as one coherent string.

One file at the root: .factory.json

To make local CI/CD and environment management predictable across projects, Leaf uses a single configuration file at the root of each repository: .factory.json.

CI checks, CD pipelines, and target environments are all defined right there. Right now, Leaf’s native CD engine handles deployments across three targets:

• Kubernetes
• Podman
• Cloudflare Workers and Pages

A repo declares how it gets checked and where it gets deployed in .factory.json:

{
  "ci": {
    "tasks": [
      { "name": "lint", "run": "npm run lint" },
      { "name": "typecheck", "run": "npm run check" },
      { "name": "test", "run": "npm test" }
    ]
  },
  "cd": {
    "environments": {
      "staging": {
        "target": "podman",
        "host": "apps.local"
      },
      "production": {
        "target": "cloudflare-pages",
        "project": "jduncan-io"
      }
    }
  }
}

And to ingest all of these new capabilities (local issues, live role-based WebSocket rooms, deterministic CI/CD runs, external repo wikis, and end-to-end trace strings), Leaf’s UI and UX got a massive overhaul. Instead of jumping between browser tabs to see why a deployment stalled or why the Tech Lead rejected a QA fix, the whole factory floor sits in one dashboard.

What I’ve refined and learned

Running the entire loop locally with end-to-end telemetry makes patterns jump out fast. Two lessons have stood out over the last week.

1. The three repeating agent anti-patterns

When an autonomous run goes sideways, it’s rarely a brand-new kind of failure. Almost every repeating issue I trace falls into one of three buckets:

• An agent writes code to attempt to check prose. Last week I caught an agent writing regex unit tests to validate markdown role prompts. Checking and refining prose is the job of an improvement loop (or a Lead Writer QA pass), not a test runner.
• An agent writes overly complex code loaded with excess dogma, ceremony, restrictions, and gates that don’t actually help anything. Left unchecked, a model will happily invent a three-layer validation framework and a custom error taxonomy for a twenty-line internal helper.
• An agent writes prose to attempt to measure something. When a node needs to verify that a migration worked or an endpoint responds cleanly, a markdown paragraph explaining why it should work is worthless. Measuring things is the job of code.

Code can’t judge meaning.
Prose can’t measure systems.

Whenever a run starts churning, it’s almost always because an agent crossed that line: using rigid code to police natural language, using natural language to fake a programmatic check, or burying simple mechanics under defensive ceremony.

2. Frontier provider outages all look different (and nobody calls them outages)

When a human is using a coding assistant in an IDE and the model hiccups, you sigh, take a sip of coffee, and hit retry. When you’re running an unattended Dark Factory across multiple harnesses, upstream instability is a systems engineering problem.

And none of the frontier providers fail the same way, even while their public status pages stay a soothing shade of green:

• Capacity walls (529 Overloaded): One provider will start returning HTTP 529 (overloaded_error) responses or mid-stream capacity errors while the status page reports everything is fine.
• Silent stream stalls: Another provider won’t throw an error code at all. A turn simply hangs in the middle of a stream and never returns, leaving the worker pod waiting forever unless the supervisor enforces a hard turn timeout.
• Ambiguous 429 rate limits: Another returns HTTP 429 with a generic "Too many requests" message during upstream capacity crunches. That ambiguity is dangerous in an automated harness: an ambiguous 429 tricks agents and routers into thinking they’ve exhausted their subscription quota when the provider is actually just throttling global traffic.

If your orchestrator takes every error message at face value, a ten-minute provider brownout will either stall your rooms indefinitely or convince your fleet that it’s out of budget for the day. Building a resilient factory means fingerprinting how each provider actually degrades in the wild, not how their documentation says they fail.

So what?

If you’re pushing your own agentic workflows toward unattended execution, a few lessons from this rebuild apply regardless of your stack:

• Build purposeful friction into every phase. The only real substitute for human review is adversarial validation. Pair your PM with an Architect, make a Tech Lead validate every Engineer and QA fix, and put an hourly QA Lead over your Tech Writers.
• Isolate the lore from the builders. Keep your architectural wiki outside the source tree and mount it only for the PM role. Engineers build better and faster when they only see their scoped node and the code.
• Keep CI and CD as deterministic tasks. You don’t need an LLM to run a linter or deploy a container. Run CI/CD as plain tasks defined in one root config file, and only wake up a QA Engineer when a check fails.
• Keep code, prose, and loops in their lanes. Use code to measure, improvement loops to refine prose, and adversarial review to strip out defensive ceremony.

Leaf is much closer to a true Level 5 Dark Factory than it was a month ago, and tracing a change from a local issue to a live deployment in one unbroken string feels like a genuine step forward. Now I just need to see what breaks next.