Your AI Wrote Forty Tests. Do Any of Them Fail?

You ask a model for tests, it produces forty of them, they all pass, and coverage goes from 34% to 91%. Every visible signal says this went well.

The problem is that all of those signals measure the same thing — that the code ran — and none of them measure whether anything was checked. A test that asserts nothing passes. A test that asserts the wrong thing passes. A test that locks in a bug you already have passes, and then defends that bug against anyone who tries to fix it.

This is how to tell the difference, starting with the only question that actually settles it.

The only question that matters

Does the test fail when the code is wrong?

That is the entire definition of a useful test, and it is not what "the tests pass" tells you. Passing tells you the suite is green against code you already believe works. It says nothing about the case you care about, which is the future commit that breaks something.

You can answer it by hand in thirty seconds. Take the function under test, break it deliberately — invert a comparison, return a constant, delete a line — and run the test. If it still passes, you have just proved the test is decoration.

def apply_discount(price, pct):
    if pct > 50:            # break it: change to pct >= 50, or delete the guard
        raise ValueError("discount too large")
    return round(price * (1 - pct / 100), 2)

Do that to three or four functions the model wrote tests for. The result is usually immediately clarifying, and it costs less than reading the tests carefully.

Four tests that pass and prove nothing

These are the patterns that show up repeatedly, and they are worth being able to recognise on sight because a reviewer skimming forty new tests will otherwise approve all of them.

Asserting on the mock. The test replaces every collaborator, calls the function, and then checks that the mocks were called. It verifies that the code is written the way it is written. Change the implementation to something equally correct and the test breaks; introduce a bug that keeps the same call sequence and it does not.

# Proves the function calls the repository. Proves nothing about the result.
def test_get_user(mocker):
    repo = mocker.patch("app.users.repo")
    service.get_user(7)
    repo.find.assert_called_once_with(7)

The truthy assertion. assert result passes for any non-empty string, any non-zero number, any populated dict. It is the assertion equivalent of nodding.

Values copied out of the implementation. The model ran the code, saw it returned 18.7, and wrote assert total == 18.7. If the function has a rounding bug, that bug is now the specification, and the test will fail the day someone fixes it. This is the most damaging pattern on the list because it converts a defect into a requirement.

Snapshots of current output. Same problem, industrialised. An auto-generated snapshot captures whatever the code does today, correct or not, and every future change is reported as a regression against it.

Coverage is measuring the wrong thing

Line coverage records which lines executed. It has no opinion about whether anything was verified. You can reach 100% with a suite that contains no assertions at all — every line runs, nothing is checked, and the badge is green.

Branch coverage is better because it forces both sides of a conditional to execute, which at least surfaces the untested error path. It still cannot tell you that the assertion on the other side was meaningful.

Treat coverage as a floor rather than a target: below about 60% you certainly have untested code, above it the number stops carrying information. Chasing 100% reliably produces tests written to touch lines rather than to check behaviour — which is precisely the kind of test a model is already inclined to write.

A review checklist that fits in a pull request

Six questions, in the order that catches the most for the least effort:

• Does each test have an assertion about a return value or an observable effect? Not about a mock having been called. Search the diff for assert_called and read every hit.
• Are the expected values derived from the requirement, or from running the code? If you cannot say why 18.7 is correct without executing the function, the test is a recording, not a specification.
• Is anything asserted about failure? Bad input, empty collection, None, zero, a negative number, a boundary exactly on the comparison. AI-written suites are overwhelmingly happy-path.
• How much is mocked? If every dependency is replaced, the test exercises a hollow shell. Mock what is slow or external; use the real thing for everything else.
• Does the test name describe a behaviour? test_apply_discount_rejects_over_fifty tells you what broke when it fails. test_apply_discount_2 does not.
• Would it survive a correct refactor? Rename a private method, reorder two independent calls. A test that breaks on that is coupled to implementation detail.

Making the check systematic

Breaking code by hand works and does not scale. The formal version is mutation testing: a tool introduces small changes into your source — flipping a comparison, changing a constant, removing a statement — and runs your suite against each one. A mutant your tests catch is a "killed" mutant. A mutant that survives is a bug your suite would not have noticed.

The mutation score is the measurement that coverage pretends to be, because it answers the actual question rather than a proxy for it. It is also slow — every mutant means another test run — so it belongs on the modules that matter rather than on everything. We have a practical walkthrough with mutmut covering how to scope it so it finishes.

The other tool worth reaching for is property-based testing. Instead of enumerating examples — which is exactly what a model does, and it enumerates the obvious ones — you state a property that should hold for all inputs and let the library hunt for a counterexample.

from hypothesis import given, strategies as st

@given(price=st.floats(min_value=0, max_value=1e6, allow_nan=False),
       pct=st.floats(min_value=0, max_value=50))
def test_discount_never_exceeds_price(price, pct):
    assert apply_discount(price, pct) <= price

Hypothesis will find the floating-point edge, the zero, the value that rounds the wrong way — the inputs nobody writes down because nobody thinks of them. It finds a different class of bug from example tests, which is why it complements rather than replaces them.

Using the model well

None of this is an argument against generating tests. It is very good at the tedious part — the parametrised cases, the fixtures, the setup — and that is real time saved.

Two things change the output materially. Ask for the failure cases explicitly, because the default is the happy path: "write tests for empty input, boundary values, and each raised exception". And give it the requirement rather than the code where you can, because a model shown the implementation writes tests that agree with the implementation — including where the implementation is wrong.

Then run the broken-code check on a sample before you merge. Not on all forty; on four. If all four tests catch the break, the batch is probably sound. If none do, you have forty lines of green that defend nothing.

Scope

Examples are Python with pytest, Hypothesis and mutmut; the reasoning applies unchanged to any language with a test runner and a mutation tool — Stryker for JavaScript and .NET, PIT for the JVM.

We have not measured how often generated tests exhibit each pattern above, and no figures are quoted for that. The patterns are ones we have seen repeatedly in review rather than results from a study, and your mileage will vary by model, prompt and codebase.