Everyone is talking about how AI can write your entire codebase, including your unit tests, in the blink of an eye. They show you slick screen recordings of agents spinning up directories, creating files, and churning out mock assertions. I just finished analyzing a local project, and it changed how I think about this promise.

The truth is simple: a test file that can’t run is just dead weight. It isn’t a testing posture.

The Reality is More Nuanced

The marketing engine tells you that writing tests is solved. The reality is more nuanced.

We look at a folder and see test_scanner.py or PhotoEditor.test.tsx, and we check the mental box. We assume our system is protected. But when you look closer, you find that the developer didn’t install the test runner. The dependencies aren’t even in package.json. The suite is a hollow shell. It looks like a test suite on a screen, but it’s just pixels.

This is the ultimate vibe coding delusion trap. It’s easy to mistake a folder full of files for an active, verifiable testing posture.

This isn’t just an anecdotal problem. GitClear’s extensive study analyzing over 211 million lines of code showed some alarming trends. As Generative AI adoption spiked, they observed a massive surge in “code churn”—code that is reverted or replaced within two weeks of being written—and a steep drop in traditional refactoring. AI assistants are excellent at generating high volumes of code. But they also excel at generating copy-pasted duplicate boilerplate that human developers check in without verifying.

The Good, the Bad, and the ExifTool

Let’s look at photosync, a personal smart photo triage tool I’ve been hacking on. It uses local AI to detect faces, read text on signs, and write metadata directly back to physical images.

The backend test suite is actually a masterclass in scaffolding. It sits on top of pytest and an in-memory SQLite database. It overrides FastAPI’s database dependency. It mocks out the heavy AI pipelines, like BLIP image captioning and EasyOCR. This keeps the test suite fast and offline-friendly.

For instance, look at how the AI engine is mocked in conftest.py to prevent running heavy local models:

@pytest.fixture(autouse=True)
def mock_ai_engine(mocker):
    # Mock methods of the global ai_engine instance
    mocker.patch.object(ai_engine, 'detect_faces', return_value=[])
    mocker.patch.object(ai_engine, 'generate_caption', return_value="A test caption")
    mocker.patch.object(ai_engine, 'extract_text', return_value=["test", "text"])
    return ai_engine

That’s a beautiful harness. It prevents a 10-second cold-start penalty every time you test. It lets you run tests on an airplane without a connection.

But look at the coverage. We only have two test files. The core routers that upload photos and write ExifTool metadata to physical files have zero tests. We have a robust testing engine, but we only used it to test the ignition switch. The critical, side-effect-heavy code is completely exposed.

The Broken Frontend Loop

Then we go to the frontend. There’s a file called PhotoEditor.test.tsx. It uses @testing-library/react and vitest to verify metadata saving and automated analysis.

But when I tried to run it, the system choked.

Why? Because vitest and the testing libraries aren’t listed in package.json. There is no test script configured. There is no configuration setup in vite.config.ts.

The test code is syntactically correct, but the loop is broken. We have the wheels, but we don’t have the car.

Designing the Harness

So what’s the takeaway for practitioners?

For juniors, don’t just accept AI-generated boilerplate. If you don’t run the tests, they don’t exist. Verify your loops.

For seniors, your job is shifting from writing code to building the harness. It’s about orchestration. You need to ensure the boundaries of your systems are wrapped securely.

We need to design the testing harness like a custom wood joint. It must fit snugly, align with the grain of the system, and hold everything together when the pressure is on. Even our brand mascot, the beaver, knows that you can’t build a strong dam out of loose twigs. You need to weave them together into a structured barrier.

Focus on the harness. The code will follow.