Mutation Testing
Mutation testing is a technique for evaluating the effectiveness of a test suite by automatically introducing small, deliberate faults ('mutants') into the source code and checking whether the existing tests detect and fail on each one.
Definition
Mutation testing is a technique for evaluating the effectiveness of a test suite by automatically introducing small, deliberate faults ('mutants') into the source code and checking whether the existing tests detect and fail on each one.
Overview
Standard code coverage metrics tell you which lines of code your tests execute, but not whether the tests would actually catch a bug in that code — a test can execute a line without meaningfully asserting anything about its behavior, giving a false sense of security. Mutation testing addresses that gap directly by asking a more pointed question: if I deliberately broke this line of code, would any test fail? A mutation testing tool works by applying a mutation operator to the source code — for example, flipping a `>` to `>=`, changing a `+` to a `-`, negating a boolean condition, or removing a method call entirely — to produce a 'mutant,' a slightly modified version of the program. It then runs the existing test suite against that mutant. If at least one test fails, the mutant is considered 'killed,' meaning the test suite successfully caught that class of bug. If every test still passes despite the introduced fault, the mutant 'survives,' revealing a gap in the test suite's ability to detect that kind of error. The mutation score — the percentage of mutants killed out of all mutants generated — is a much stronger signal of test suite quality than raw code coverage. Because generating and running tests against many mutants across a codebase is computationally expensive, mutation testing is usually run less frequently than the regular test suite — nightly, on a schedule, or only against changed files — rather than on every commit. Popular mutation testing tools include Stryker (JavaScript/TypeScript, C#), PIT (Java), and mutmut or Cosmic Ray (Python). It's most valuable for critical code paths where a false sense of security from misleadingly high line coverage would be costly, such as financial calculations, security logic, or core business rules.
Key Concepts
- Automatically introduces small deliberate faults ('mutants') into source code
- Runs the existing test suite against each mutant to see if any test fails
- Produces a mutation score, a stronger quality signal than line or branch coverage
- Surviving mutants reveal gaps where tests execute code but don't actually verify behavior
- Computationally expensive, so typically run on a schedule rather than every commit
- Common mutation operators include flipping comparisons, negating conditions, and removing calls
- Popular tools include Stryker, PIT, and mutmut across different language ecosystems