Mutation testing is a technique to help identify holes in test suites. It focus not on the quality of code, but on the quality of unit tests. It does this by making small changes in your codebase and running your test suite against it. The assumption is that if your tests don’t fail, then your tests aren’t specific enough and are missing some cases.
This tutorial will walk you through how to get started with mutation testing on python projects. It uses mutmut to generate and analyze mutations. For more information on mutation testing details, check out our blog post.
Getting started with mutmut
mutmut is one of the most used mutation testing frameworks for python. Getting started is quick. In a terminal, type in
pip install mutmut
mutmut run
This will install mutmut locally and run the software.
However, the default configuration is likely to not fit your needs. One thing to note is that you may need to specify both the code directory and the test directory like so
# Example
mutmut run --paths-to-mutate "app/" --tests-dir "tests/"
But if your configuration is a bit more complicated, you can add your settings in a setup.cfg file
[mutmut]
paths_to_mutate=proj1/,proj2/ # comma-separated list of paths to code
runner=python -m unittest proj1 # shell command to run tests
tests_dir=tests/ # comma-separated list of test directories
Seeing mutation results
After you have successfully run mutmut run …, you should see output that looks similar to
2. Checking mutants
⠇ 9/9 🎉 6 ⏰ 0 🤔 0 🙁 3 🔇 0
The first values (9/9) shows that 9 out of 9 mutations have been run.
🎉 6 specifies that 6 of those 9 mutants were successfully killed off
🙁 3 specifies that 3 of those 9 mutants did not get killed off, and the unit tests still pass
To view the full report, type the following in a terminal
mutmut results
This will show you a value like
Survived 🙁 (3)
---- app/calculator.py (3) ----
5-7
It tells you which mutants (5, 6, and 7) survived and the file that had code changed.
To view the individual mutant, you can type in
mutmut show 5
to show you the diff between the original code and the mutation
--- app/calculator.py
+++ app/calculator.py
@@ -10,7 +10,7 @@
return x * y
def divide(x, y):
- if y == 0:
+ if y == 1:
return 'Cannot divide by 0'
return x * 1.0 / y
Making changes based on mutation successes
From this code snippet above, you can see that mutmut has changed the line if y == 0 to if y == 1. Let’s see what I could do to solve this problem.
If this project has test coverage, I could first make sure that the above lines of code have tests. If there are no tests that run those lines of code, then any mutations will not be caught and killed off.
If I know that my code is covered, I should make sure that the branch is fully covered. That is, that the true and false cases of the if statement are being tested for.
If both of those conditions are already met, then I am most likely missing a more comprehensive test case. I should test other cases and see if that will successfully kill the mutant. As you can see, marrying test coverage with mutation testing can give you an extremely resilient test suite. Not only will your code be fully tested, but the tests themselves will be adaptable to code changes in the future.
