A proof that it's done

One of the things everybody hates most right now is the tests agents write. Agents are really good at writing terrible ones.

Terrible tests

Say you ask an agent for a pull request that deletes a config option. It deletes the option, thinks, “Well, I need to do some testing”, and adds a test:

it("removes the legacyExport option", () => {
  expect(defaultConfig).not.toHaveProperty("legacyExport")
})

As a human, you look at this and think: What the fuck is this? Why would you test that we deleted something when we deleted it? But for the task the agent is doing in that second, it makes sense.

The agent needs a proof that it’s done.

Fine, for that one task. Then you merge it, and the next agent sees that we assert this kind of thing and thinks:

Oh, I cleaned up some comments. Let me add a test that verifies this comment no longer exists.

Agents copy patterns much more than humans do. One cuts a corner, and the next sees it: “ah, that’s how you do it”. Each one after does it a little worse, until you have mold growing in your codebase.

Overly specific tests

Agents love overly specific tests. Have you seen your agent write these?

it("registers every rule", () => {
  expect(registry.rules).toHaveLength(81)
})

it("loads the dashboard without excessive queries", async () => {
  const queries = await countQueries(() => loadDashboard(teamId))
  expect(queries).toBe(7)
})

The first causes merge conflicts right away, because anything that changes that number conflicts with anything else that changes it. Add a rule for something unrelated, and you have to update this test for no reason.

The second is meant to catch excessive database queries. So we count them. Agents proliferate these tests: “ah yeah, we do that for every feature now”. From then on, anything that changes a query anywhere turns random tests red.

Tests that test a mock

Then there are tests that do nothing. Some run to 500 lines, and every line tests a mock, so all they verify is that the mocking library accepts anything. I’ve seen huge tests where you replace the mock with the actual system and the test stays green, because it was doing everything right. It was just doing it to a mock.

The whole test was useless, but nobody was reading it, because when you see a 600-line PR, you think:

Ah, fuck this. Yeah, sure, it’s gonna be fine.

The 5,000-line file

Agents make things incrementally worse. A file starts at a hundred lines. Then it’s “oh, this is the perfect spot to put this”, and “the perfect spot to put that”, and at some point the file has 5,000 lines. By then it’s really hard to get it back down, or even to keep it from reaching 10,000.

A human looks at that and says, “At this point I’m just going to split the file.” Not every human will, but agents will not be the ones to say: “Maybe I should actually change this.”

I joined a project where one file was 10,000 lines, and the test file for it was 30,000. When an agent’s PR adds 150 lines to a test file like that, everybody says, “I’m not even reading these tests, why should I bother?”

Small-picture mode

Agents are usually laser-focused on their task and don’t think about long-term consequences. My presumption would be that during training there are none.

They also think in small-picture mode. Context is still somewhat limited, and to be efficient an agent won’t look at everything every time. Working in one part of your software, it doesn’t know about the rest and has to make a lot of assumptions, the same way we humans do. That’s why we like modular designs and interfaces: you say you want to send an email, and dependency injection finds an email service without you knowing the details.

Any bigger project has unintended side effects and emergent behavior. If you’re really good, you know a lot of them. Most agents would have to rediscover them every single time, and nobody can keep all of that documented. What you can do is write tests that encode them at a very high level.

Story tests

So I introduced story testing, or DSL testing. The idea is about twenty years old, but with agents it’s much more valuable now. Agents also make it easy to invent things, so I invented my own version: @shivaedev/test-story.

You build a kit of helpers that uses your app’s defaults for everything, and one call sets up the full actual app, with its actual database and its actual everything. Nothing for agents to mock.

The agent names only what isn’t default, as traits: logged in as Susie, and Susie is an admin. The traits get applied magically, so the test reads like English. Traits, and the verbs a test acts through, are written once per feature, so each test holds only its story:

Imagine you have your app, and there’s Susie, and Susie has three projects, one of which she’s paying for.

She deletes one of the other projects.

Make sure that her paid project is not affected, and that in the end she gets an email about her cancellation.

It has the proper testing structure: the setup, the action, and the assertions. Written with the kit:

app.it("Susie deletes a free project and keeps her paid one", [
  loggedInAs("Susie"),
  isAdmin("Susie"),
  hasProjects("Susie", "blog", "wiki", "shop"),
  paysForPro("shop"),
], function* ({ deletesProject, projectIsActive, outbox }) {
  yield* deletesProject("wiki")

  expect(yield* projectIsActive("shop")).toBe(true)
  expect(outbox.sentTo("Susie")).toContain("cancellation")
})

You assert on outcomes instead of passing in random mocks to check that some random internal did the right thing. So the story accidentally tests your email service, it tests your database, it tests everything. Of course it’s slower to run, but it’s actually useful.

Why it works

If an agent breaks something, chances are some story fails, and the agent realizes:

We deleted the project, but we’re no longer sending the project deletion email.

There’s no super in-the-weeds failure mode, but a really high-level business outcome changed, and that’s what tests should do anyway.

Give an agent a small, clear, short task, and it stays at that level. But when a test written from a bird’s-eye business view fails, the agent thinks:

Wait a moment. This test has nothing to do with my low-level thing. Let me step back and think about where my thing fits in the bigger context.

Guardrails

You’re not giving agents guardrails to stop them from doing bad things because you don’t trust them, but to make it easier for them to do the right thing.

Think of a bowling alley. If you’re really good, you can hit the pins at the end. Pull up the guardrails on the sides, and you can throw as hard as you want and still hit some. So instead of one really good bowler, ten people can throw in parallel, and the guardrails make sure every ball arrives at the end somehow.

Testing the whole app does the same for agents. It reduces the cognitive complexity, so each agent can think locally about the part it’s actually editing, and many can work in parallel while the tests protect them from emergent behavior and side effects.

Learned helplessness

There’s some form of learned helplessness in how people use agents. The AI will not magically do any of this. A lot of people hope they can let the agent do the thinking and the verifying, and then they say:

I asked Claude to do this, and Claude didn’t do it well enough. Claude just writes terrible tests.

It might be that your testing setup is really hard, or that the current training set doesn’t reward good testing.

But maybe you have a way to test in your head, and you never taught Claude.