Code coverage is a simple metric to measure the effectiveness of your tests; increasing it helps you spot problems. That gives teams confidence in their tests, and increases the quality of your code.
Code coverage isn’t just a number, though. It represents a commitment to quality that can help your entire organization. If different team members understand the role it plays, they can take full advantage of its benefits.
Why Use Code Coverage?
Higher code coverage increases your chances of finding bugs. And while code coverage doesn’t guarantee perfection, you’ll be significantly less effective without it. Put simply, code coverage tells you how much of your code your tests are reaching. 80% code coverage means 80% of your code is executed during test runs.
For test-driven development, you need to aim for 100%. Hitting 100% in a large code base can be a pipe dream, but your emphasis should be on improvement.
When it comes to those large projects, increasing coverage above 70-80% is time consuming, and you need to assess the costs and benefits. For higher risk systems, higher coverage is needed.
Coverage is especially important when it comes to the lesser-known parts of your code base. The rarely executed parts of your code are more likely to contain errors. Edge cases—or code built to handle unusual scenarios—are often hastily written and forgotten about.
Who Needs Code Coverage?
How code coverage relates to your work depends on your role. For some, it will be a key target. For others, code coverage is a means of monitoring progress, or ensuring software meets technical and legal requirements.
Code coverage is easily overlooked, so making sure people are aware of it can help ensure it is used, understood, and supported throughout your team.
In this article, I’ll explore how code coverage impacts people in six different roles. I’ll talk about how they should think about it, and how to integrate it into their work.
The roles are:
- Software Engineer – Designs, develops, and helps maintain the project.
- Software Test Engineer – Designs and develops tests to identify and help solve issues.
- Director of Software Engineering – Assigns roles to software engineers, and defines policies and procedures.
- Functional Safety Engineer – Ensures software is released with the minimal risk to those dependent on it.
- Software Auditor – Makes sure software complies with laws and regulations.
- Product Owner – Runs the show and takes ultimate responsibility for the project.
Software Engineer
A QA-driven culture is based on the assumption that no individual tool or test can guarantee quality. As a software engineer you need to make sure multiple tests are in place, so code can be tested across different conditions and scenarios.
Software engineers need to be proactive about examining their code and informing test engineers about potential omissions. They are the ones with the closest relationship to the code, and will be the most aware of what happens at each specific location. Using this knowledge to find gaps in coverage is invaluable in creating robust code that your tests effectively cover.
Branches, such as if or switch statements, require careful examination to make sure they are tested correctly. An obvious place where code could fail to execute is a try-catch scenario. You need to make sure errors occur in order to catch further issues in the error handling code. Code reviews can also help. Make sure you are talking about tests as a means of helping detect and fix those errors you don’t spot.
By making sure tests cover everything, and reporting your findings to test engineers, you can leverage both your skill sets to improve your code’s reliability. This approach means you spend more time building new features and less fixing errors.
Software Test Engineer
As a test engineer, part of your role is to advocate for the customer and maintain the highest possible quality standards. You should always be looking for opportunities to increase code coverage to help hit your targets.
You also need to evaluate testing completeness and remember that effective coverage is more than just a number. The code covered also has to be covered well; your tests might not be perfect. While a high code coverage figure doesn’t guarantee quality, a low figure does guarantee that code is going untested.
Test engineers can help other team members understand the benefits of testing and explain how code coverage helps them achieve their goals. Decreasing downtime or removing dead code are positive results that have more appeal than simply boosting the code coverage figure for its own sake.
You can also make sure coverage includes recent changes, as these are where bugs are most likely to be found. Making sure builds trigger automatic coverage analysis is a great way to do that.
Software test engineers are key drivers of increased code coverage, and are also responsible for making sure metrics are used wisely to deliver results.
Director of Software Engineering
Leadership roles have many responsibilities, and creating a culture where code quality is at the forefront should be a significant one. As a director you should provide project overviews, allocating tasks to ensure engineers are working to meet code quality standards.
Studies show programmers are only 50% effective at finding bugs, and 35% effective when it comes to tests. With numbers like those, it’s critical that you leverage every tool available to catch as many issues as possible.
Code coverage is a simple way to monitor your testing’s effectiveness, particularly when it comes to improving your QA picture. It offers a quick, easy to understand figure that can indicate how well you are doing.
There are other benefits to increasing coverage too, such as educating developers and making refactoring easier. Directors are in a prime position to understand these benefits and relay them back to the team.
It never pays to rely on a single metric, and you should analyze each project as broadly as possible to best understand your overall code quality. Defect metrics and complexity metrics offer further insight.
Directors can also make sure testing is given the necessary resources. Increasing the amount of time dedicated to testing shows people it’s important, and increases buy-in throughout your company.
Functional Safety Engineer
Functional safety engineers have a higher level of responsibility than other roles. Their work directly focuses on the safety of software users.
Software failure can be catastrophic in some fields, with strict rules governing how software is tested. Mistakes can cost lives in some scenarios, so there’s no excuse for letting standards slip. Insufficient software testing resulted in the Ariane 5 rocket explosion, medical overdosing, and the retraction of hundreds of scientific papers due to calculation errors.
Many industries have regulations that specify minimum standards for code coverage. As a functional safety engineer, it’s essential that you meet—and ideally exceed—these standards.
Minimum code coverage standards vary according to the industry. In aviation, you’ll need to hit 100% coverage for safety critical systems. There are high standards in medicine too; equipment that handles medicine dosage or monitors patients’ health cannot afford to fail. For the automotive industry, you need to provide ‘sufficient coverage’. The ISO 26262 specification requires increased levels of coverage depending on the criticality of the system.
For functional safety engineers, code coverage is both a target and a way to protect their users; it can be the difference between life and death.
Software Auditor
Software audits are common in many industries. They’re especially likely if developing software used for key infrastructure, or in fields with strict standards.
Code coverage is an easy way for auditors to verify software quality—it’s a clear and objective metric. It may not tell the whole story, but code coverage offers a single figure to show that developers are serious about finding errors and measuring improvement.
But auditors can’t just look at numbers. They also need to play detective and determine how much distortion their audit is creating. Developers may perform steps they would omit if not being audited, or tell auditors what they want to hear.
The good news is that code coverage can’t be faked. If code coverage is low, it offers an easy, understandable way for a team to improve. By identifying that, the audit has done its job.
Product Owner
Product owners have their money on the line; they have the most at stake when ensuring the product meets quality standards. Product owners also have to make the big decisions, including getting the balance right between maintaining quality and investing in new features.
Code coverage offers owners an easy way to keep track of the bigger picture when it comes to quality. Even if they aren’t targeting 100% coverage, they can push to get it higher, and shake things up if the number drops too low.
Beyond allocating resources, owners can also drive quality by ensuring those that deliver it are recognized. Acknowledging and rewarding those that hit targets and deliver results keeps the team motivated and shows they’re valued.
Informed product owners know that increasing technical debt can become a burden, and keeping code coverage high is a way to prevent that. Investing in quality from the beginning means less time fixing problems later.
Conclusion
While there are many roles involved in delivering a software project, they all share the same ultimate goal—a successful, quality product.
There are different ways to achieve this, but software is so complicated that providing everyone with an effective overview is challenging. Code coverage provides a metric that everyone can work to improve and edge your projects closer to delivery.
Code coverage isn’t just for the sake of QA. Its advantages can spread through the whole team, improving everybody’s outcomes in the process. Testing benefits engineers, owners, and ultimately your customers; make sure you use it to its full potential.
