Blog Post

How to Set Up Codecov with C and GitHub Actions

May 4, 2021 Aniket Bhattacharyea

Code coverage is a metric used to measure how much of your source code has been covered by your test suite. In other words, it tells you what percentage of your code is executed when you run your tests, and helps you find out which parts are not covered by the test. As a C developer, you can use code coverage to identify gaps in test cases at an early stage and improve the quality of your tests.

Codecov is a reporting tool that can take the coverage generated by your build tools and convert it into a meaningful representation with graphs and charts which can help you better understand your code coverage. Codecov also integrates smoothly with your existing CI/CD systems, like GitHub Actions, and provides coverage reports for every commit and pull request so that you have an understanding of how your tests are performing without any hassle.

In this tutorial, you’ll learn how to set up code coverage using Gcov and how to integrate Codecov with your GitHub repository using a GitHub Action to generate coverage reports every time you push changes or make a pull request.

You can check out the GitHub repo for this tutorial here.

Running Tests Locally

This tutorial assumes you have gcc installed and set up on your computer. The tool Gcov comes bundled with GCC.

In this tutorial, you’ll test a small library with only two functions. First, create a file lib.h with the following declarations:

int f1(int);
int f2();

And then put the definitions in lib.c:

#include "lib.h"

int f1(int a) {
  if(a > 10) {
    return a + 1;
  } else {
    return 2;
  }
}

int f2() {
  int a = 1;
  return a++;
}

And that’s all for the library.

Now, create a test.c file with the following code:

#include<assert.h>
#include "lib.h"

int main() {
  assert(f1(5) == 2);
  assert(f2() == 2);
}

You have two very simple test cases using assert statements that check the return values of f1 and f2.

Now it’s time to compile the code into an object file. In order to enable code coverage, you need to pass the -ftest-coverage and -fprofile-arcs flag to gcc. The flag -O0 is also needed so that the compiler doesn’t optimize away our trivial example.

gcc -ftest-coverage -fprofile-arcs -O0 -o test test.c lib.c

This should create an executable called test in the current directory as well as a lib.gcno and test.gcno files. They contain important information about how many times individual lines have been executed. Note that these .gcno files will be generated for every object file.

Now you need to run the tests:

./test

Since all your tests pass successfully, this should exit without any output. You’ll notice that two more files have been created, namely lib.gcda and test.gcda. Similar to the .gcno files, the gcda files contain information related to the code coverage.

Now that all the necessary files have been generated, you can run gcov to create the actual coverage report. You only want the coverage report of your library (lib.c) and not the unit test file (test.c). So you’ll run gcov on lib.c only:

gcov -abcfu lib.c

Here are the explanations for the flags used:

-a, --all-blocks           Show information for every basic block

-b, --branch-probabilities Include branch probabilities in output

-c, --branch-counts        Given counts of branches taken rather than percentages

-f, --function-summaries   Output summaries for each function

-u, --unconditional-branches    Show unconditional branch counts too

This should give an output like the following:

Function 'f2'
Lines executed:100.00% of 3

Function 'f1'
Lines executed:75.00% of 4

File 'lib.c'
Lines executed:85.71% of 7
Branches executed:100.00% of 2
Taken at least once:50.00% of 2
No calls
Creating 'lib.c.gcov'

Now a file called lib.c.gcov is created in the same directory. This file contains the detailed coverage report:

        -:    0:Source:lib.c
        -:    0:Graph:lib.gcno
        -:    0:Data:lib.gcda
        -:    0:Runs:1
        -:    1:#include "lib.h"
        -:    2:
function f1 called 1 returned 100% blocks executed 75%
        1:    3:int f1(int a) {
        1:    4:  if(a > 10) {
        1:    4-block  0
branch  0 taken 0 (fallthrough)
branch  1 taken 1
    #####:    5:    return a + 1;
    %%%%%:    5-block  0
unconditional  0 never executed
        -:    6:  } else {
        1:    7:    return 2;
        1:    7-block  0
unconditional  0 taken 1
        -:    8:  }
        -:    9:}
        -:   10:
function f2 called 1 returned 100% blocks executed 100%
        1:   11:int f2() {
        1:   12:  int a = 1;
        1:   13:  return a++;
        1:   13-block  0
unconditional  0 taken 1
        -:   14:}

Let’s take a quick minute to understand the report. Here each block is marked by a line with the same number as the last line of the block and the number of branches and calls in the block.

Notice that you’ve only called f1(5) in your test code. Hence the if(a > 10) block was never executed. So, it says never executed in that block and is marked by %%%%. Similarly, line 5 is never executed and is marked by ####. All the other blocks are executed exactly once and therefore show 1 as their execution count.

Each function in the report is preceded with a line stating how many times it was called, how many times it returned, and the percentage of blocks that were executed.

For example, the line function f1 called 1 returned 100% blocks executed 75% tells you that the function f1 was called one time. It returned 100% of the times and 75% of the blocks were executed. This is expected since you already knew that the if(a > 10) block was never executed.

GitHub and Codecov

Now you’ll set up Codecov with GitHub Actions. First, you’ll create a GitHub repository and push your code there. Make sure to add the *.gcno, *.gcda, *.gcov and test executable in your .gitignore. These will be generated when the tests are run.

Now you’ll have to sign up for Codecov. So, navigate to codecov.io and click Sign Up. You can sign up with your GitHub account.

Once you do that, you’ll see a list of your GitHub repositories. Choose the repository you just created to link it with Codecov. You will be presented with an upload token.

Codecov screen with the upload token redacted

You’ll need this token if your repository is private, or becomes private in the future.

Now you’ll create a GitHub Action. If you’ve never worked with GitHub Actions, take a look at the docs first. The actions are just YAML files that need to be put in the .github/workflows directory. Each YAML file will define an action. You can name these files anything you want.

So, create a file .github/workflows/codecov.yml and enter this code:

name: Codecov
on: [push, pull_request]
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - name: Fetch
        uses: actions/checkout@master
      - name: Test
        run: |
          gcc -ftest-coverage -fprofile-arcs -O0 -o test test.c lib.c
          ./test
          gcov -abcfu lib.c
      - name: Upload
        uses: codecov/codecov-action@v1
        with:
          files: ./lib.c.gcov

This configuration tells GitHub that this action runs when you push to a branch or make a pull request. Then it defines a job named run with three steps:

  • Fetch: Clones your repo.
  • Test: Compiles the test code and generates the coverage report, just like you did locally.
  • Upload: This is where the report is uploaded to Codecov. For that we use the codecov-action action provided by Codecov.

And finally, the files option tells Codecov which files to upload.

If you have a private repo, you need to get the upload token from your Codecov dashboard and put it in a GitHub encrypted secret. Then you need to add another option to the with part in the GitHub Actions file:

...
with:
  files: ./lib.c.gcov
  token: ${{ secrets.CODECOV_TOKEN }}

This assumes that you named the secret CODECOV_TOKEN.

Now commit and push the repository to GitHub. As soon as the push completes, the action will start to run. You can check the progress in the Actions tab in the GitHub repo.

Actions tab showing progress

When the action finishes completely, you’ll see check marks by the names of the steps.

Actions tab showing completion of action

You can check that the report was successfully uploaded to Codecov by expanding the Upload step.

Upload step completed successfully

Now you can head over to your Codecov dashboard and navigate to your repository. You can also go to https://codecov.io/gh/USER/REPO directly; just replace USER with your GitHub username, and REPO with your GitHub repo name. Now your coverage report should be visible in the dashboard.

Codecov dashboard showing coverage report

At the bottom of the page, you’ll see a sunburst graph and a file breakdown of your coverage.

Sunburst graph showing coverage details

You can read the docs to learn how to navigate and use the sunburst graph.

Integrating with Pull Requests

Remember, how you had one block that was not tested? Let’s fix that and make a pull request.

First, you’ll create a new branch and make the changes in this new branch:

git checkout -b change

Now, open the test.c file and add another assert statement:

assert(f1(20) == 21);

Adding this new assert statement will cause the previously unexecuted block to execute and should increase your coverage.

So, push the changes to a new branch:

git push -u origin change

Now create a pull request from this new branch to the main branch. When you create the pull request, the action will be run and generate your coverage report.

GitHub Action being run on a pull request

When the checks are successful, the Codecov bot will comment with a coverage report that shows how the coverage will change due to the pull request.

Codecov bot commenting on a pull request

You can check out Codecov’s documentation to know how to read the coverage reach graph. In short, the green block indicates a file where the coverage will increase, whereas a solid red block indicates a file where the coverage will decrease.

In this case, you can see that it says lib.c will have its coverage increased to 100%, as you had guessed.

You can visit the Pulls tab in your Codecov dashboard to check the coverage for all the pull requests ever made.

Codecov pulls tab

Here you can select the pull request, and you’ll get access to the detailed coverage changes, similar to the comment.

Details of individual pull request

You can visit the Commits tab where you can access the coverage for individual commits.

Commits tab showing an individual commit

The 100% in pink indicates the coverage for the total project, and the percentage in green tells you that the coverage increased by 14.29% after this commit.

You can also visit the Branches tab where you can access the coverage for different branches.

Branches tab showing all the branches

Conclusion

In this tutorial, you learned about how code coverage is important for identifying portions of your code that are not covered by the unit tests. Code coverage lets you understand where your unit tests are lacking and helps you detect bugs early.

You also learned how to set up testing for your C code locally using Gcov and how to integrate Codecov with your GitHub repository using GitHub Actions and generate coverage reports for each commit and pull request.

Comments


Leave a Reply

Your email address will not be published. Required fields are marked *