Creating a testing culture can be a daunting experience for organizations that are not used to writing tests. But as code complexity increases, it becomes more difficult to understand how different parts of a system interact. Ensuring code is well-tested is one of the ways to provide stability when making code changes.
Although code coverage can’t make sure that tests are of high quality, it can point out holes in a testing suite. This is one of the main reasons that organizations will turn to code coverage as a benchmark for testing. It also is a way to build up a testing culture on your team. At Codecov, we deal with tens of thousands of codebases and the many questions that come with starting code coverage. With the learnings from those teams, we present the 5 Levels to Code Coverage Mastery
In the following sections, we go over the traits of each level. We will provide a common key result following the OKR system as well as tips to using Codecov to get ready for the next levels.
Level 0: Recruit
We start our journey at level 0 with those that are have not yet used code coverage. This is where many teams will start whether or not they have begun to unit test their codebases.
If you are reading this article, you might be in this category. And it might not be a bad thing, there are very good reasons to not use code coverage. But as your teams continue to grow, you and your new teammates may not know how bad (or good) things are. Staying in level 0 keeps you unaware from the health of your testing standards.
Key Result: Run a code coverage tool on our codebase to understand what code hasn’t been tested.
Using Codecov: You can upload the coverage reports generated by your language’s coverage tool to Codecov using our uploader. This will help get a snapshot of your coverage, even if you are using multiple languages or have a monorepository.
Level 1: Beginner
Welcome to level 1! This is the point where a team has decided to try out code coverage. They are just dipping their toys in the water and don’t want to rock the boat. The main purpose of being in this level is to get the organization used to seeing and internalizing the current coverage percentage of the codebase.
Key Result Expose code coverage numbers on every code change
Using Codecov Use Codecov’s informational checks to show code coverage numbers on every pull/merge request. These status checks will be non-blocking, meaning they won’t interrupt a developer’s workflow.
You can add the following snippet to the Codecov configuration file (codecov.yml).
coverage:
status:
project:
default:
informational: false
patch:
default:
informational: false
Level 2: Intermediate
At some point, teams that are serious about using code coverage will need to enforce some coverage standards. This level represents those teams that are looking to start increasing the number of lines with tests written against them.
The best way to get started is to enforce that newly written code needs to have tests. This way, the burden of writing tests stems away from legacy code and focuses on a developer’s expertise.
Key Result All new features are 80% covered by tests.
Using Codecov You can use Codecov’s patch check to target a certain percentage for all new code. If, for example, you are aiming for 80% of new code, you can add the following to your codecov.yml file.
coverage:
status:
patch:
default:
target: 80%
Note that this should replace the informational: false line so that the status check will fail if coverage isn’t high enough.
Level 3: Advanced
Now that a team is used to writing tests for their new code, it’s time to start thinking again about the entire codebase. It can still be daunting to target a high code coverage number, so we recommend that teams work on slowly increasing the value. You can do this by enforcing all new code changes should increase the codebase coverage over time.
Key Result All pull/merge requests maintain or increase codebase coverage.
Using Codecov We can focus on the project check to increase coverage over time. Instead of setting a number to target as we did previously, we can set the value to auto. This ensures that each pull/merge request must be maintaining or increasing the entire codebase coverage.
coverage:
status:
project:
default:
target: auto
threshold: 0%
Note that threshold is set to 0%. Threshold represents the “wiggle room” in coverage, meaning that in this case, there is no wiggle room. However, some pull requests might delete a lot of old code which if previously covered, will decrease overall project coverage.
For example, if my codebase has 10 lines, of which 4 are covered with tests, the total code coverage is 40%. If I delete those 4 lines, I will have 0 of 6 lines covered, meaning my coverage decreases to 0%.
It is often in a product’s best interest to remove old code that is unused, but it can be frustrating for a developer to have to write tests somewhere else to make up for the decrease in coverage. For this reason, we recommend setting threshold to some value that gives some leeway when deleting large chunks of unused code.
Level 4: Expert
At this stage, your teams are used to expecting tests on their pull requests. Now is the right time to start aiming for that 80% coverage number.
Key Result Achieve 80% code coverage in the repository
Using Codecov
At this point, you can continue to use the same Codecov configuration. But it may be worth it to start investigating the least covered parts of the codebase. One way to do this is to see which files are the least covered.
Going to the repository dashboard, you can select File list and Missed to see which files have the most room for improvement.
Level 5: Champion
If you are at this point, you are looking to have one of the top testing cultures. 80% is clearly not enough for you, and though we don’t always advocate for 100% coverage, it might be right for you.
You may find that the last 20% is particularly difficult to cover, due to strangeness in a particular language’s coverage collection. But if you decide to prevail, we applaud you.
Key Result Achieve and maintain 100% code coverage in <> repository
Using Codecov You may wish to adjust your yaml configuration to maintain 100% coverage
coverage:
status:
project:
default:
target: 100%
This ensures that the codebase must always be at 100% or the status check will fail. You may also want to include the Codecov status badge on your repository README to show off your incredible work. Congratulations! It’s never easy getting to 100%, but you are now code coverage masters.
If you are looking for another way to improve your coverage, take a look at your branch coverage.
Stories of code coverage success
Although we deal with thousands of organizations, our system is never fool-proof. These steps have helped a significant number of our users, and we feel that it’s the best way to get started using code coverage and Codecov. We are always looking for feedback, and you can email us at devrel@codecov.io if you have any questions or comments.
