Teaching My Dark Factory to Dream: From Agent Telemetry to Self-Healing Code

Tonight at 9:25 PM Eastern (02:25:53 UTC), an unattended simplification run named binpacker765 finished inside Leaf, my local Level 5 dark software factory.

BinPacker is a small inventory management app I built for my wife’s kids’ clothing resale store. It tracks plastic storage bins, seasonal sizing, and pull actions when items move from storage to the floor. Earlier tonight, Leaf’s background Improvement loop bundled five complexity reductions across BinPacker’s React hooks and notification components into a single plan and handed it to an engineer pod.

The engineer pod simplified all five files, passed the repository’s test suite and AST complexity gate, and merged its branch. In almost every agentic coding setup, a green merge is where the story ends.

In Leaf, closing the room set off a chain reaction while I watched from the dashboard:

• 02:25:53 UTC (binpacker765 closes): When the room tore down, Leaf’s history recorder checked the pod’s numbers against the repository’s 30-day baseline. The engineer pod had taken 95 turns over 13 minutes. In binpacker, the 30-day 90th percentile (P90) for an engineer pod is 19 turns. Leaf stamped the closed room with a turns flag and placed it in the telemetry queue.
• 02:26:44 UTC (The Analyst wakes up): Fifty-one seconds later, Leaf opened an analysis room and spawned a Class 5 analyst pod. Its job wasn’t to grade the code. Its job was to read the room log and the pod’s compressed trajectory archive and explain where those 13 minutes went.
• 02:28:01 UTC (The bug is filed): One minute and 17 seconds into its read, the Analyst found the culprit and filed a bug directly against binpacker:

The 95-turn outlier wasn’t an “LLM reasoning failure” or a bad prompt. Because the engineer pod touched five frontend files that all imported into the same page, and scripts/check looped over changed files running vitest related --run one source at a time, the test runner executed src/pages/Find.test.tsx four separate times. At roughly 70 seconds per run (68090ms, 70637ms, 67854ms, and 72548ms), repeating that single test file burned 211 extra seconds, which in turn caused the engineer pod to burn dozens of turns polling background process status and scheduling timers.

• 02:32:29 UTC (Planning and execution start): Because binpacker runs with auto mode enabled, filing that bug immediately woke up a Product Manager (PM) and Architect pod in a planning room. They drafted a single-node plan (Deduplicate change-related frontend test execution) to discover affected tests per source file first, keep the per-file untested: warning, and execute the deduplicated union once.
• 02:50:46 UTC (The fix lands and closes the ticket): Eighteen minutes after the plan started (and 24 minutes and 53 seconds after binpacker765 closed), the engineer pod’s fix cleared the full CI and container build suite, squash-merged onto main, and automatically closed the issue.

From a slow test run in a kids’ clothing inventory app to a root-cause diagnosis, a reproducible bug report, an approved plan, and a merged fix in under 25 minutes. Zero human keystrokes.

The nuance: why counting “moments of friction” failed

To get to that 25-minute loop tonight, I first had to delete 6,067 lines of telemetry code on Thursday morning.

When I first wired up observability for Leaf, I did what most engineers building agent harnesses do. I built a background pipeline that fed every finished agent trajectory to a local model and asked it to extract “moments of friction.” Whenever an agent hit a non-zero exit code, fumbled a tool call, or retried a command, the reader logged a moment and grouped it into a problem list.

On paper, reading every transcript sounds thorough. In practice, it was a firehose of low-grade noise.

A run that finished cleanly in 40 seconds and a run that burned 20 minutes in a retry spiral got the exact same shallow read. Because a small model was grading individual tool steps in isolation, it had no link to what the run actually cost or whether a code change would prevent the hiccup next time. My telemetry tab filled up with hundreds of unranked complaints, and I still had to click through transcripts by hand to figure out what mattered.

Counting how often an agent stumbles tells you nothing if you can’t name the line of code that tripped it.

Two days ago, right after we pruned 65% of our agent prompts, I ripped out the entire moments pipeline and rebuilt Leaf’s telemetry around two ideas: deterministic SQL flags for single rooms, and a daily wake-sleep “dreaming” pass that consolidates patterns across rooms.

Step 1: deterministic flags and a skeptical Analyst

Instead of paying an LLM to scan every transcript looking for vibes, Leaf now evaluates every closed room (build plans, PM/Architect planning rooms, CI/CD runs, Improve examinations, and Security validations) against eight hard facts already recorded in SQLite when a room tears down:

• Six event flags (something went wrong):
  • owner: I stepped into the room with an unprompted message that wasn’t answering an agent’s question.
  • dead time: Two adjacent frames in the room had a silent gap of 10 minutes or more.
  • rejected: The PM or Architect rejected the issue during planning.
  • questions: The planning room had to stop and ask me a round of clarifying questions.
  • send-back: The Tech Lead rejected an engineer’s diff.
  • unfinished: A pod exited without a clean pass or completion report.
• Two relative flags (something cost more than usual):
  • retried: A pod had to be started more than once.
  • turns: A pod’s turn count exceeded the 90th percentile (P90) of its role in that repository over the prior 30 days (with planning seats compared strictly against planning rooms, never build pods).

If none of those eight facts hold true, the room isn’t queued for analysis at all. And if a room’s only flag is turns (meaning nothing broke or got sent back, it just took longer than usual), Leaf samples every fifth room (1-in-5) and skips the other four so the queue never drowns in normal tail variance.

When the supervisor drains the queue, it prioritizes rooms where something clearly went sideways (owner, rejected, or dead time) first, then other planning rooms, and finally build rooms, newest first.

Making “no bug” the default answer

When you hand a transcript to a frontier model and ask it what went wrong, its natural instinct is to invent a problem so it looks helpful. That’s how you end up with the post-mortem prompt bloat we had to clean up earlier this week.

To keep the analyst persona honest, its prompt is tiny and built around a single default: explaining where the room’s time went and closing with no bug is the normal outcome. The Analyst is only allowed to file a bug when three conditions hold at once:

1. It can point to a specific change in a prompt, the factory harness, a brief, the target repository, or the container image that would have prevented the cost.
2. The transcript proves what that cost actually was.
3. The cause is systemic and will happen again on future runs, rather than being a quirk of that one task.

Even then, the Analyst has to check the repository’s existing issues first and link to an existing cause (match) rather than filing a duplicate. If it does file a new bug, the harness requires structured Observed, Expected, Reproduce, and Code facts sections naming the exact files and symbols it verified.

We also caught an amusing cross-repository routing bug this afternoon. While analyzing a BrandCast planning room, an Analyst pod discovered a subtle race condition inside the factory’s own planning state machine and dutifully filed the bug against the BrandCast repo. Two minutes later, the BrandCast PM and Architect pods booted up, searched /work for the factory’s state-machine files, realized they were sitting inside a digital signage codebase, and rejected the ticket as misfiled. We tightened the routing rules right after so harness, prompt, and container causes always route to the factory repository, while application-level causes route to the project repo.

Step 2: teaching the factory to “dream” across runs

A single-room Analyst has a structural blind spot: it only sees one room at a time.

Because the Analyst is told to file a bug only when a cause will repeat, reading one room in isolation creates a catch-22. When an Analyst sees a truncated CI log excerpt or a slow test script in a single run, it often can’t tell whether that hiccup was a one-off fluke or a quiet tax hitting five rooms a day. So it writes a faithful one-line verdict explaining where the time went, closes the room with no bug, and exits.

Even worse, if a bug was filed and fixed last Tuesday, a single-room Analyst reading a room on Friday doesn’t automatically connect the dots across the whole month of closed issues to flag that the regression came back.

To solve that, I added a second telemetry loop built around dreaming.

I didn’t coin the term “dreaming” for offline consolidation. The idea goes all the way back to Francis Crick and Graeme Mitchison’s 1983 Nature paper proposing that brains use REM sleep for “reverse learning” (pruning parasitic noise from overloaded neural networks), and Geoffrey Hinton, Peter Dayan, Brendan Frey, and Radford Neal’s 1995 Science paper on the Wake-Sleep algorithm. Kevin Ellis and Joshua Tenenbaum’s DreamCoder (2021) brought that wake-sleep loop directly into code synthesis, where an agent solves problems while “awake” and refactors its solutions into reusable library abstractions while “asleep.” More recently, Letta (MemGPT) formalized sleep-time compute for stateful LLM agents, and Anthropic shipped Claude Dreams to consolidate raw session transcripts offline.

I borrowed that exact wake-sleep pattern for Leaf’s telemetry.

During the day, single-room Analysts inspect flagged outliers as they finish and write short, one-line verdicts. Then, once every 24 hours per active repository, Leaf spawns a Class 5 dreamer pod.

Unlike the Analyst, the Dreamer doesn’t start by wading through megabytes of raw tool logs. Its brief hands it the last 30 days of analyzed rooms for that repository, every single-room Analyst verdict, and the history of open and closed trajectory issues. Its job is to look across the entire window at once:

• Spot shared causes behind different phrasings. If three separate room verdicts mention different symptoms of the same underlying bottleneck, the Dreamer connects them and files a single bug covering all of those rooms, pulling individual transcripts only when a one-line verdict doesn’t give enough detail.
• Consolidate “cause families.” When two or more rooms link to the same underlying issue, Leaf groups them on the Telemetry board as a cause family and tracks how many rooms hit that cause each week.
• Catch fixes that didn’t stick (Recurred after fix). Whenever an Analyst or a Dreamer matches a room to a closed issue, Leaf compares the timestamps. If the room closed after the issue was marked fixed, the harness automatically records the link as recurred and flips that family’s status on the dashboard from Fixed to Recurred after fix.

Look at what the live 30-day Telemetry scorecard shows across my four onboarded repositories (factory, brandcast, calamus, and binpacker) as of tonight:

Across 953 flagged rooms in the 30-day window, Leaf sampled out 238 turns-only rooms and read 715 rooms all the way through. Out of those 715 analyses and the daily Dream passes:

• Most flagged rooms didn’t need a ticket. In more than 75% of read rooms, the Analyst explained where the clock went, concluded the cost was normal for the size of the task, and closed the analysis without filing noise.
• 69 distinct systemic causes were isolated, and 61 (88.4%) are already fixed. Because telemetry bugs feed directly back into the planning queue, the factory has already planned, tested, and merged fixes for 61 of the 69 causes it discovered on itself.
• 24 multi-room “cause families” emerged. For example, this morning’s Dream pass on the factory repo reviewed 180 Analyst verdicts in a single pass and noticed that four separate rooms had lost real CI failure diagnostics because a test fixture printed oversized marker lines that saturated the log excerpt cap. It consolidated those rooms onto a single issue, and an engineer pod fixed the excerpt slicer 13 minutes later.
• 4 closed fixes were caught recurring (5.8%). One family where a Tech Lead’s completion report was misrecorded as unreported showed up across 16 rooms. When a room closed after the initial fix landed and still hit the bug, the timestamp check caught the regression automatically and flagged the family as Recurred after fix.

A few honest practitioner caveats on these numbers:

• The 715 read rooms include a one-time backfill. When we turned on the new telemetry loop on Thursday, we set a two-week backlog cutoff so the Analyst could chew through recent history right away. That gave the first daily Dream passes across brandcast, factory (180 verdicts), calamus (92 verdicts), and binpacker (19 verdicts) a rich dataset immediately, though steady-state daily volume is much calmer.
• Zero issues on dead time (0 of 19) doesn’t mean rooms never stall. Why did 19 rooms with a 10-minute silent gap yield zero linked issues, while unfinished yielded 38.5% (40 of 104) and turns > P90 yielded 27.8% (62 of 223)? Because most 10-minute gaps between room frames happened while a single long E2E suite or container build was legitimately running inside a pod. When a room actually stalled and failed to finish, it also tripped unfinished or owner, which is where the real stall bugs got filed.
• BinPacker isn’t enterprise production traffic. Using an inventory app for my wife’s kids’ resale shop alongside brandcast, calamus, and factory is my favorite kind of lab test, because the codebase is small enough to trace every turn, but real enough that a 3.5-minute Vitest bottleneck is immediately painful. Still, there’s no revolution in the suburbs just yet.

Measuring orientation and whether the wiki pays its rent

While I was overhauling room telemetry this week, I added two smaller instruments to answer questions I kept guessing at: how long does it take an engineer pod to orient itself before making its first edit, and do agents actually read the documentation other agents write?

1. Engineer orientation

Every time an engineer pod finishes, Leaf unpacks its trajectory once, counts how many tool calls the agent made before its very first file edit, and records which repository files it read during that orientation phase.

Across 7,934 engineer trajectories recorded so far, the median orientation is just 3 tool calls before the first edit, and the 90th percentile is 11 tool calls. Because Leaf also aggregates every non-skill file that engineer pods read across multiple plans, I can see at a glance which files agents keep hunting for on Turn 1 (like test setup helpers and core server entrypoints across 140+ plans) and bake those conventions directly into the Architect’s brief or repository skills.

2. Wiki payoff

Whenever a feature PRD or bug closes and deploys in Leaf, a Tech Writer agent wakes up to update the repository’s product wiki (and an hourly Lead Technical Writer merges duplicates and keeps the index clean). That sounds great in an architecture diagram, until you ask how many tokens the writers are burning and whether the PM and Architect ever read what they wrote.

Yesterday we added a wiki health tracker that measures four signals per repository: writer token cost, page reads by the PM, Architect, and /leaf sessions, explicit wiki citations in planning messages, and PM readiness decisions that cite the wiki.

In the first 31 hours since that counter started:

• brandcast (104 wiki pages): 77 writer jobs and 10 lead passes burned 49.7M input tokens (mostly cache reads) and 205k output tokens. In return, planning seats cited the wiki 80 times, and 58 of the 104 pages have already been read in live planning rooms (46 unread so far).
• factory (148 wiki pages): 24 writer jobs and 11 lead passes burned 19.4M input tokens, producing 17 citations, while 108 of the 148 pages haven’t been read once yet.
• calamus (32 pages) and binpacker (20 pages): 19 and 10 citations respectively, with three-quarters of binpacker’s wiki already read in planning rooms.

Seeing that 73% of factory’s own wiki pages haven’t been read in the last day and a half is the exact kind of uncomfortable feedback loop you want in an unattended harness. If a documentation loop costs 20 million tokens a day and nobody in planning reads two-thirds of the pages, the telemetry should tell you to tighten the Tech Writer’s scope.

So what?

If you’re building multi-agent workflows or running unattended coding loops, stop grading your agents with generic LLM “friction counters”:

• Gate telemetry on deterministic SQL facts, not LLM vibes. Don’t pay a small model to read every transcript looking for “moments of friction.” Trigger analysis only when a run trips a hard mechanical fact: an unprompted human message, a rejected plan, a Tech Lead send-back, a 10-minute silent gap, or a turn count above the role’s 30-day P90 (sampled 1-in-5).
• Make “no bug” the default post-mortem answer. If you ask an LLM to find a problem in every slow transcript, it will invent rules that bloat your prompts. Require your Analyst agent to name the exact file, symbol, and reproducible command that will cause the cost again, or close the reading with no bug.
• Separate single-run analysis from cross-run “dreaming.” An agent reading one transcript can’t tell a fluke from a systemic tax. Run a daily consolidation pass that reads the last 30 days of one-line Analyst verdicts together, groups shared root causes into families, and flags closed issues that recurred after their fix landed.
• Instrument the supporting loops too. Track how many tool calls your engineers spend orienting before their first edit, and measure whether the docs your writer agents generate actually get read and cited by your planners.

By the time I finished writing this draft (02:50:46 UTC), the BinPacker fix had finished its CI suite green, squash-merged onto main, and closed the bug automatically. Next time an Improvement pod simplifies my wife’s resale inventory app, Find.test.tsx will only run once.