In today’s rapidly evolving software development landscape, ensuring the quality and reliability of code is crucial to user trust. Users expect software that just works the way that they expect. One key practice that aids in achieving this goal is code coverage.
Code coverage measures how much of your codebase is exercised by your test suite. It can show you what lines of code have been tested and what haven’t. But many developers use this information at the end of their development cycle, as a final check of whether or not they have written enough tests. What if we could use code coverage during the development process to write high-quality code faster?
This article goes over the benefits of using code coverage while you code and how Codecov is improving that process for developers. Because code coverage is not just a metric —- it’s a tool.
Why use code coverage?
Many developers find value in getting the code coverage percentage of their codebase. This shows the total number of lines that are tested versus the total number of lines in a codebase. This helps them get a gauge of the health of their code. This is a perfectly valid use of code coverage, but other developers consider it a waste of time. Shouldn’t you know whether or not you’ve written enough test cases?
Consider instead how code coverage information can be useful during the development process. Code coverage, at its core, presents the lines of code in your codebase that are hit during a test run, and those that aren’t. Instead of using it as a final check, why not pair it with your coding and testing practices? That way, you can continuously iterate over your code changes to see what edge cases you may not have considered.
We can then use code coverage like spellcheck. Consider the various time pressures on a developer. When you are writing a new feature, it’s better to quickly know what hasn’t been tested while you still have context as opposed to later when you may be rushed to deploy a change. It can also be exhausting and time consuming to sift through hundreds of lines that might need tests written. This leaves more opportunities to miss testing edge cases.
Using code coverage in the development cycle
In an environment where code coverage is treated as a gatekeeper, we see the development cycle as
- Build
- Test
- Repeat, until ready for review
- Check
Code coverage inhabits the final check step. This can be useful for teams that are able to build small features or small fixes. But for larger chunks of code, this can cause the final step to balloon in time.
Let’s evaluate a developer lifecycle that includes code coverage as a spellcheck.
- Build
- Test
- Check
- Repeat, until ready for review
By including check steps like code coverage, linting, and static analysis, we continually iterate on code that is grounded and tested. The tests we write in part 2 let us know if there are any regressions while continuing to write code.
Note that for developers using test-driven development or behavior-driven development, switch the Build and Test steps.
Benefits of code coverage during development
Now that we have introduced the code coverage as a spellcheck paradigm, let’s see how else we can benefit from checking coverage alongside our code.
- Identifying Uncovered Code
Code coverage helps in identifying portions of the codebase that lack test coverage. Uncovered code may include branches, conditions, or exceptional cases that are not tested adequately. Identifying such areas allows developers to direct their efforts toward writing additional tests, ensuring that the entire codebase is thoroughly validated. - Bug Detection and Prevention
Higher code coverage increases the chances of detecting bugs early in the development cycle. A comprehensive suite of tests helps identify code paths that may trigger unexpected behavior or reveal software problems. It also allows them to catch and address edge cases that may not be apparent during initial development stages. - Confidence in Refactoring
Software systems continually evolve, and refactoring plays a vital role in improving code maintainability. However, refactoring without a safety net of tests can be daunting. Code coverage provides developers with the confidence to refactor existing code without introducing regressions. By rerunning the test suite after each refactoring step, developers can ensure that the modified code still passes the same set of tests. This practice mitigates the risk of inadvertently breaking functionality. - Collaboration and Code Reviews
Code coverage analysis fosters effective collaboration among team members. It facilitates code reviews by providing a quantifiable metric that can be used to assess the thoroughness of the test suite. Reviewers can examine the coverage report to identify areas that require additional tests or improvements.
Code Coverage in the Workflow
With the benefits of code coverage in the developer workflow in mind, how does a user actually implement this? Most languages have a testing framework that comes with some code coverage aspect. These tools can run alongside your test runner to collect coverage and display it to the user.
In order to use this information, the developer needs to run their tests after they have finished writing them and investigate what lines might still be missing. Take the example below which shows that a line in app/calculator.py is missing a test.
--> pytest --cov app
============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/thomashu/src/github/codecov/example-python
plugins: asyncio-0.21.0, Faker-8.8.2, mock-3.8.2, sqlalchemy-0.2.1, anyio-3.6.2, cov-3.0.0, celery-0.0.0
asyncio: mode=strict
collected 4 items
app/test_calculator.py .... [100%]
---------- coverage: platform darwin, python 3.10.6-final-0 ----------
Name Stmts Miss Cover
--------------------------------------------
app/__init__.py 0 0 100%
app/calculator.py 11 1 91% # <--- Notice that one line is uncovered
app/test_calculator.py 24 0 100%
--------------------------------------------
TOTAL 35 1 97%
As a developer, I now know that I am missing a line now, and I can investigate to see which line is missing a test. I can open up a report that shows me which line is uncovered.
With this information, I know that I missed testing the divide by 0 case. I can quickly write a test for it and continue on to adding more functionality to the Calculator class.
