Why Hitting Your Coverage Target Is Making Your Tests Worse

I had 87% coverage, and we still broke the billing flow on launch day. Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure. That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome. The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy. Now, when someone asks: how do you know if it is working? The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision. Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered. That is the complete definition. It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically. The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of bugs found after release. Inozemtseva and Holmes found separately that line coverage is the metric that correlates least with real defect detection compared to other available options. You are using the most popular quality metric in software engineering. It is also the least predictive of actual quality. Goodhart's Law comes from economics: when a measure becomes a target, it ceases to be a good measure. The principle describes what happens when you attach consequences to a proxy metric instead of to the underlying goal it was meant to represent. Coverage percentage is a proxy metric. The underlying goal is: tests that catch bugs before production. When you make the proxy metric a gate, teams optimise for the proxy, not the goal. The badge turns green. The risk does not go down. This is not a character flaw. It is a predictable response to an incentive you designed. Under delivery pressure, engineers do the rational thing: they hit the number using the fastest available path. The fastest path to coverage percentage is testing happy paths. User submits form successfully. API returns expected response. Data saves to the database. These scenarios are easy to construct, execute quickly, and each one covers a high line count because they touch the main execution path. What does not get covered: Error handling paths run on exception conditions that are tedious to construct in tests. Retry logic, fallback behaviour, and timeout handling live in branches that require specific setup. These are also, as a direct consequence, the places where the most consequential bugs hide. Boundary conditions require generating test inputs at the edges of valid ranges, null states, empty strings, Unicode characters, and maximum lengths. Each of these covers almost the same lines as a single typical input but requires separate test construction. Under time pressure, one test covers the line. Conditional branches show as "covered" once either branch executes. A function with a condition that checks user role, applied to admin and guest differently, can show as fully covered if only one role ever runs the test. The other branch remains untested with no coverage signal. The outcome is a suite at 85% coverage where the well-tested parts are tested three times over, and the failure-prone parts have one shallow test each. Teams with high line coverage but low integration validation have been observed producing more than double the production incidents of teams with lower but more balanced coverage. Assumption: The claim that coverage-gamed suites produce higher production incident rates than lower but risk-balanced suites is based on observed patterns in industry case studies, not a controlled experiment. The "double production incidents" figure comes from analysis of specific teams and may not generalize uniformly. It should be read as directional evidence of the problem rather than a precise multiplier. In the first article in this series, the framing was: the cost of not investing in testing is invisible until it is catastrophic. Coverage gaming creates a specific variant of that problem. You are paying the full maintenance cost of a test suite that is not delivering the risk reduction you paid for. Every test written to hit a percentage rather than catch a bug still needs to be maintained when the code changes, still runs in every CI cycle, and still occupies space in the mental model of everyone who reads the suite. You have made the right investment decision. A portion of that investment is funding theatre rather than protection. The cost shows up in the budget. The corresponding value does not show up in production stability. The honest version of coverage is mutation score. Mutation testing introduces deliberate small bugs into your code: flipping a comparison operator, removing a conditional check, changing a return value. Your test suite runs against each mutated version. If the tests catch the mutation (they fail), the mutation is killed. If the tests pass despite the code being wrong, the mutation survives. A mutation score of 80% means your tests caught 80% of introduced bugs. That is a direct claim about defect detection, not about line execution. These are different claims. A real case documented on dev.to reported 93% line coverage that turned out to be a 34% mutation score. The team had been reporting a green badge on a suite where two-thirds of the introduced bugs would have survived. Mutation testing is computationally expensive, which is a real constraint. The practical approach: do not run it across the entire codebase. Run it on the paths where failure is most costly, specifically the risk-weighted paths identified in the work from Article 3. Use line coverage as a floor (it tells you about completely untested code) and mutation score as the signal for whether the tests you have are actually doing anything on the critical paths. Branch coverage is a useful intermediate step. It is stricter than statement coverage, it surfaces the conditional gaps that line coverage misses, and most existing coverage tools already support it. Switching from statement to branch coverage as your CI gate raises the bar without requiring new tooling. Assumption: Mutation testing adoption is growing but not mainstream as of mid-2026. Frameworks exist for most major languages (Stryker for JavaScript and TypeScript, PITest for Java, mutmut for Python) but are not a default feature in major CI platforms. The claim that most teams do not use mutation testing is based on tooling adoption patterns, not a formal survey. The tool change matters less than the conversation change. If coverage percentage is a pass/fail gate in your pipeline, you have created the incentive to game it. The alternative is not removing coverage tracking. It is removing the gate behaviour. Report coverage as a trend, not a threshold. A team whose coverage has been at 78% for six months is in a fundamentally different position than one whose coverage dropped from 88% to 68% after a feature push. The trend carries information. The snapshot at a point in time does not. Pair that with the question you actually want answered: are the highest-risk paths in this codebase meaningfully tested? That question does not have a single percentage answer. It requires looking at branch coverage on critical paths and, where the stakes are highest, mutation score. Neither of these is hard to implement. They are just not the defaults. PS: I am Surendranath, leading the AI team at QApilot, and we are solving problems specific to mobile app testing through our AI-native product. Reach out if you are interested to try it out!