Execute mutation testing to evaluate test suite effectiveness.
ReadWriteEditGrepGlobBash(test:mutation-*)
Mutation Test Runner
Overview
Execute mutation testing to evaluate the effectiveness of a test suite by systematically introducing small code changes (mutants) and checking whether existing tests detect them. A killed mutant means the tests caught the change; a surviving mutant reveals a testing gap.
Prerequisites
- Mutation testing framework installed (Stryker, mutmut, PITest, or go-mutesting)
- Existing test suite with reasonable pass rate (all tests must pass before mutation testing)
- Source code with functions and logic suitable for mutation (conditionals, arithmetic, return values)
- Sufficient CI resources (mutation testing runs the test suite once per mutant -- CPU-intensive)
- Configuration file for the mutation tool specifying target files and test commands
Instructions
- Verify the existing test suite passes completely:
- Run the full test suite and confirm 100% pass rate.
- Fix any failing or skipped tests before proceeding.
- Mutation testing is meaningless if the baseline tests are broken.
- Configure the mutation testing tool:
- Stryker: Create
stryker.config.mjs with mutate patterns, test runner, and thresholds.
- mutmut: Configure
setup.cfg or pyproject.toml with [mutmut] section.
- PITest: Add Maven/Gradle plugin with target classes and test configurations.
- Select target files for mutation:
- Focus on business logic modules (not configuration, constants, or type definitions).
- Exclude auto-generated code, third-party wrappers, and test utilities.
- Start with a small scope (one module) to validate setup before expanding.
- Run the mutation testing suite:
- Execute
npx stryker run, mutmut run, or mvn pitest:mutationCoverage.
- Monitor progress -- expect long execution times (10-100x normal test runtime).
- Use incremental mode if available to skip already-tested mutants.
- Analyze the mutation report:
- Killed mutants: Tests detected the change -- indicates strong test coverage.
- Survived mutants: Tests did not catch the change -- indicates a testing gap.
- Timed out mutants: Mutation caused an infinite loop -- generally acceptable.
- No coverage mutants: The mutated code is not exercised by any test.
- For each surviving mutant, determine the appropriate action:
- Write a new test that specifically catches the mutation.
- Or determine the mutation is equivalent (functionally identical to original) and mark as ignored.
- Set mutation score thresholds (recommended: 80%