Codecov Components: Breaking Down Coverage By Filters

Code coverage is one of the best tools to identify pieces of your code that aren’t tested. It produces a final percentage that can be used as an indicator of whether or not more tests need to be written. Using code coverage can also help you to see what lines of code are missing tests.

However, not all sections of a codebase are equally important. Some files or directories that touch sensitive data or deal with payments may need to have more stringent guidelines. Codecov Flags was initially meant to help by filtering coverage by test suite or uploads. However, this isn’t the most intuitive approach.

Codecov Components are coverage filters by directory or flags. They give you more power to define what parts of your codebase are important and which need to have different coverage goals.

Why should I break down my code coverage?

You may be wondering, why even break down coverage by directory or flag? Imagine a payments service for a social media app. Although it’s important that the app is well-tested, knowing that your app isn’t erroneously charging your users is critical to maintaining user trust. Having payments reliably come in ensures that your business can pay bills. For this reason, it becomes clear that some parts of your codebase will be more important than others.

Code coverage on a project is useful but can hide issues in your codebase. It is far more important to have high coverage for critical pieces of your system compared to low-priority parts. If the opposite is true, that non-critical pieces are highly tested, it could give your product a false sense of safety.

By breaking down your coverage, you can bubble up important parts of your codebase and their corresponding code coverage. It can help to pinpoint issues in your test suites and where you may need to invest more time and resources to write high-quality tests.

What are Codecov Components?

Codecov Components are a way to see coverage for a given set of directories or flags. They allow you to control how to filter your codebase that isn’t limited to how you write your test suites.

Let’s take a look at an example of Components. The repository listed above is set up with this directory structure

.
├── calculator
│   ├── complex
│   │   ├── complex_calculator.py
│   │   └── test_complex_calculator.py
│   └── simple
│       ├── simple_calculator.py
│       └── test_simple_calculator.py
└── smiles
    ├── smiles.py
    └── test_smiles.py

We have two directories calculator and smiles. The calculator the directory contains two sub-directories complex and simple. The project is set up this way to emulate what might happen if you have multiple sub-directories that may run their tests as part of different suites.

Let’s consider two scenarios to get coverage for the calculator directory.

1. The complex and simple tests are run separately
2. All tests for both calculator and smiles are run as one test suite

In the first case, you could get away with using Codecov Flags by uploading both sets of coverage results together. However, the second scenario would not be possible. But with Components, you can upload all the coverage reports as a single report, and Codecov will filter out the file paths to give you separate coverage.

You can view the result of the second scenario in this pull request. Notice the comment showing the coverage for calculator and smiles, despite having only one coverage report.

Setting up Components

To start using Components, you will only need to make changes to the codecov.yml. As of this writing, Components are only available in the PR comment.

First, update the comment section to include components.

comment:
  layout: "header, diff, flags, components"  # show component info in the PR comment

Then, you can specify individual components in a component_management section.

component_management:
  individual_components:
    - component_id: module_calculator  # this is an identifier that should not be changed
      name: calculator  # this is a display name, and can be changed freely
      paths:
        - calculator/ # accept regex and glob

You can add more individual components in the individual_components section.

To add, status checks to a component, you can add statuses to an individual component

…
  individual_components:
    - component_id: module_calculator  # this is an identifier that should not be changed
      name: calculator  # this is a display name, and can be changed freely
      statuses:
        - type: project
          target: auto

To create default rules for components, you can use the default_rules section of component_management.

component_management:
  default_rules:  # default rules that will be inherited by all components
    statuses:
      - type: project # in this case every component that doens't have a status defined will have a project type one
        target: auto
        branches:
          - "!main"

Well, what about Flags?

Although Flags are extremely powerful, they are limited in their one-to-one relationship with coverage uploads. Components are a better way to understand a particular directory or set of files, regardless of how you are uploading coverage.

To better exemplify the difference in use cases, let’s consider two scenarios again.

1. Unit tests need to cover 80%+ of a codebase that spans multiple projects
2. A billing service needs high coverage, but only integration and end-to-end tests are written

In the first case, we would recommend using Flags. Since the directory structure doesn’t matter, you only need to care about the total unit test coverage. However, in the second case, uploading integration and end-to-end coverage reports will include multiple directories. Using Components will help to filter out that coverage to just a particular service.

This means that Flags are leaving. In fact, if you are currently satisfied with how Flags are working for you, we do not recommend switching to Components. But for new users or users who are looking for a better solution, Components are going to be more useful.

Using Components with Flags

Since Components do not need to be associated with an upload, they only need to be configured in the codecov.yml file, while Flags must be incorporated into the CI/CD workflow.

Take a look at this PR which shows how Components and Flags can be used together. Notice a section for both Flags and Components in the comment. And that the smiles Flag and Component show the same coverage. The calculator component is the combination of both simple and complex coverages.

What’s next for Components?

As you can see, Components are a great way to filter coverage as necessary and set up the proper guard rails. However, the feature is currently limited to the developer workflow. We plan to provide insights in the Codecov UI. If you have feedback or suggestions, please take a moment to create a post on our Feedback board (must be logged in).