Using code coverage helps you find untested parts of your codebase. It does this by tracking which lines of code are executed when running your test suite. At Codecov, we calculate the coverage percentage as the number of lines run divided by the total number of lines.
Since tests are considered running code, they are often included in coverage calculations. When starting with code coverage, it’s not always clear whether or not you should include these files. In this article, we discuss the pros and cons of having test files as a part of code coverage and how to use Codecov to ignore test files.
Why would test files be included in code coverage?
When you run your test suite, each test will run different parts of your codebase. However, they are still written in a language and must be interpreted by a processor. Simply put, tests are still code. As such, many coverage tools automatically collect coverage information for test files and directories.
As we will discuss later, having test files in your coverage can be detrimental. You can configure Codecov to ignore those files altogether. Below is a snippet from our docs on ignoring paths in a codecov.yml file.
ignore:
- "path/to/folder" # ignore folders and all its contents
- "test_*.rb" # wildcards accepted
- "**/*.py" # glob accepted
You can use the configuration to ignore files, directories, or even certain file types.
Pros of including test files in coverage calculations
As mentioned above, tests are still inherently code. If we buy into a technical definition of code coverage, all code run should be considered in the calculation.
But including test files will always increase your coverage percentage if you run all of your tests. This is due to both the numerator and denominator adding the same number of lines. For teams that are just starting out with test coverage, this ego boost can help decrease the anxiety of hitting a high coverage number. Seeing a codebase at 40% covered can be more motivating than 12%.
But including test files can also be useful when developing features. Using code coverage, you can identify when tests aren’t run. This is extremely helpful when working with a new testing framework or implementing test doubles.
Cons of including test files in coverage calculations
Although there are some pros to including test files, considering them in coverage calculations only artificially inflates the number. This can be misleading to developers and management. Worse, it can cause teams to abuse writing tests as a way to increase coverage. This will erode faith in code coverage as a tool in a way that it was not meant to be used.
At the end of the day, code coverage is meant to be a tool to help reduce the number of bugs that occur in production. Covering test cases that should be running won’t help with that.
Test files in coverage verdict
If you are starting out with code coverage or you need to know whether your tests actually run, including test files in your coverage calculation can be extremely useful. But if you are further along in your testing culture, you should not be including test files at all.
If you are struggling with your coverage tools, you can use Codecov’s ignore configuration to remove test files from coverage consideration.
