Tests are an important component of software development: they help prevent bugs and can act as documentation. One of the best ways to ensure your code quality stays high over time is to monitor your test coverage, through a measure known as code coverage. Monitoring code coverage is essential to maintaining code quality, and there are many tools out there that can assist with this.
OpenCppCoverage is an open-source C++ code coverage tool for Windows that can generate coverage reports locally, and then leverage Codecov GitHub Actions to integrate coverage reports in your CI pipeline.
In this tutorial, you will learn the basics of OpenCppCoverage and how to integrate it into your CI pipeline. If you’d like to skip ahead to the full code for the demo, it is available here.
Running Tests and Coverage Locally
To follow this tutorial, you need Windows Vista or higher and Microsoft Visual Studio 2008 or higher. You can clone the demo repository with the following command:
git clone https://github.com/imaculate/HelloCovDemo.git
Alternatively, you can proceed with the step-by-step instructions below.
1. Creating a Library
First, you will create a static library project from Visual Studio. The library will house computations below that can be performed with bit operations.
// HelloCov.h
bool is_power_of_two(int n);
int multiply_by_two(int n);
int divide_by_two(int n);
//HelloCov.cpp
#include "HelloCov.h"
bool is_power_of_two(int n)
{
if (n < 0)
return false;
return (n & (n - 1)) == 0;
}
int multiply_by_two(int n)
{
return n << 1;
}
int divide_by_two(int n)
{
return n >> 1;
}
Ensure the library compiles successfully, then proceed to add tests.
2. Add Tests
In the same solution, add a Native Unit Test Project
with reference to the HelloCov
library. In a source file, add one test method for each function as follows:
#include "CppUnitTest.h"
#include "../HelloCov/HelloCov.h"
using namespace Microsoft::VisualStudio::CppUnitTestFramework;
namespace HelloCovTest
{
TEST_CLASS(Test1)
{
public:
TEST_METHOD(IsPowerOfTwo)
{
Assert::IsTrue(is_power_of_two(8));
}
TEST_METHOD(MultiplyByTwo)
{
Assert::AreEqual(14, multiply_by_two(7));
}
TEST_METHOD(DivideByTwo)
{
Assert::AreEqual(-4, divide_by_two(-7));
}
};
}
Verify that all the tests pass.
3. Install OpenCppCoverage
There are two ways to install it—using Chocolatey:
choco install opencppcoverage
Or using the interactive installer. Go to releases and download the latest release of the installer.
4. Generate Code Coverage Report
From the root directory of the solution, run the following command:
OpenCppCoverage --sources HelloCov -- "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" debug/HelloTest.dll
The path of vstest.console.exe
depends on your version of Visual Studio, so update it accordingly. The report will be generated in the folder CodeCoverage-<Timestamp>
. You can view it by opening index.html
in a browser. On navigating to the report for HelloTest.dll
> HelloCov.cpp
, you will see the line coverage like below:
Configuring GitHub Actions
Coverage reports are most useful when making changes to your code. In this section, you will integrate Codecov into your workflow so that you are able to view reports on pull requests or pushes to your repo.
1. Repo
Create a public GitHub and push to it the solution above.
2. Sign Up for Codecov
If you don’t already have an account, head over to Codecov and sign up with your GitHub account. Select the repo you will be integrating. This tutorial uses a public repo for simplicity but if your repo is private you will be provided a token. Alternatively, you can add a repo by navigating to https://app.codecov.io/gh/<gh-username>/<repo-name>
.
3. Setup GitHub Actions
GitHub Actions make it easy to automate software workflows. You’ll set up a workflow to essentially run OpenCppCoverage on a Windows machine, then upload the report to CodeCov. These actions are specified in a YAML file in the .github/workflows
directory in your repo. Add the YAML file below to your workflows folder:
name: Upload CodeCov Report
on: [push, pull_request]
jobs:
run:
runs-on: windows-latest
name: Build, Test , Upload Code Coverage Report
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
fetch-depth: ‘2’
id: checkout_code
- name: Setup MSBuild and add to PATH
uses: microsoft/setup-msbuild@v1.0.2
id: setup_msbuild
- name: Run MSBuild
id: run_msbuild
run: msbuild /p:Configuration=Debug /p:Platform=x86 HelloCovDemo.sln
- name: Setup VSTest and add to PATH
uses: darenm/Setup-VSTest@v1
id: setup_vstest
- name: Setup OpenCppCoverage and add to PATh
id: setup_opencppcoverage
run: |
choco install OpenCppCoverage -y
echo "C:\Program Files\OpenCppCoverage" >> $env:GITHUB_PATH
- name: Generate Report
id: generate_test_report
shell: cmd
run: OpenCppCoverage.exe --sources HelloCov --export_type cobertura:HelloCov.xml -- "vstest.console.exe" Debug\HelloTest.dll
- name: Upload Report to Codecov
uses: codecov/codecov-action@v2
with:
files: ./HelloCov.xml
fail_ci_if_error: true
functionalities: fix
The actions are described by their respective name
properties. They mirror what we manually did to generate local reports i.e setup, build, test,k and coverage. Of importance are the last two actions that generate and upload the report. The OpenCppCoverage
command is similar to the one used locally, and the export_type
parameter is added to get the report in xml, one of the Codecov supported formats. The absolute path of vstest is not required since setup_vstest
action added it to global PATH. When you push this change you can monitor the workflow from the Actions tab of the repo on GitHub.
4. View Codecov Report
If the workflow succeeds, at the end of logs for the upload action, there will be a link to the Codecov report like below:
Navigate to the link for a detailed report including the conspicuous coverage percentage. Under the Files tab, you can view line-by-line coverage for each file.
Under the Graphs tab, you can view graphical representations of your coverage, like the sunburst below:
Sunburst graphs enable you to navigate into project folders in order to discover files that lack coverage. The size of each slice is proportional to the number of coverage lines and the color indicates coverage. Click on a block to enter a folder and click the innermost circle to leave it.
5. Change Report
To view coverage reports as comments on pull requests, set up as follows:
- Add empty
codecov.yml
to the root of the repo. - Grant access to the codecov bot to your account.To verify the setup, on a separate branch, write tests to cover the lines that have not been covered:
TEST_METHOD(IsNegativePowerOfTwo)
{
Assert::IsFalse(is_power_of_two(-8));
}
Push and create a pull request from the branch. You should see a report like below on the pull request when the actions complete:
Observe that coverage increases as expected.
Conclusion
Code coverage is vital to ensuring that your software quality stays consistent over time. Now you should have a good grasp of how to integrate C++ code coverage reports in your CI workflows with Codecov in order to ensure you won’t fall behind on your coverage.
Although the tutorial was specifically for Windows, Codecov is platform-agnostic and can be configured not only with other C++ toolchains but also other languages and code-hosting platforms.