New Product: Test only what you change with Carryforward Flags

We are extremely excited to announce our new product, Carryforward Flags.

What are Carryforward Flags and how can I use them?

Carryforward Flags is designed for engineering organizations who do not run their full test suite in their continuous integration (“CI”) build each time they open a pull request or merge code

Codecov is now the first universal code coverage solution to ingest partial test suite runs and still display accurate code coverage.

As a result of Carryforward Flags, we can now get coverage for all our apps and packages in our monorepo (over 70 apps and hundreds of packages) in a way that is easy to use and easy for teams to digest. It also has allowed us to add blocking Github checks for code coverage reductions per app/package. This really helps ensure more thoroughly tested code is getting out the door.

Daniel Straus,
Engineering Manager

Why are some teams not running their full test suite on every commit?

CI tools fundamentally changed development and testing, but CI build processes can get very long.

Many CI/CD providers allow you to build in parallel to decrease the CI build time (Codecov already supports merging reports from parallel builds).

Even with trends in CI run parallelization, organizations are still seeking shorter test run cycles for 1) minimizing developer downtime during the CI build and 2) decreasing compute costs for running excess tests.

Two specific use cases have emerged:

Partial test suite runs by category

In this example, a user may have several logical groups of tests in their test suite. Consider the case of two types of test that can be run on every commit: unit, and end-to-end. Unit tests may be quick to run on every commit; however, end-to-end tests may be time consuming and users may only choose to run these tests as components change that may demonstrably impact the application.

Delta coverage on an entire repo

Some organizations run tests only for code that a developer changes, and that code’s dependencies. Such setups are more complicated than traditional testing setups, but the primary benefit is that you only run the tests you need to run on a particular commit.  For large test suites, this testing approach can dramatically decrease CI run time.

How does Codecov’s Carryforward Flags feature work?

You can read more about Carryforward Flags in our Docs and see them in use in an example repository.

Codecov’s Flags feature already permits users to tag different test coverage outputs (e.g, unit tests, integration tests, etc.) from CI processes such that those outputs can be meaningfully reviewed in Pull Requests and within the Codecov web application.

Carryforward Flags are an enhancement to Flags that allows users to indicate to Codecov that it needs to look at coverage data from earlier commits and apply that coverage information to the commit currently being processed. Using Carryforward Flags, teams that leverage delta coverage and partial test suite runs to lower their CI run time can now have a complete picture of their project’s coverage on any commit. The coverage that is absent on the current commit, is simply “carried forward” from a previous commit where that coverage was uploaded to Codecov.

The key need addressed in Codecov’s Carryforward Flags feature is an understanding that, if parts of the test suite are not run, Codecov needs to go backwards in time and find coverage data from the most recent time that those parts of test suite were run.

Codecov already permits our users to tag different test coverage outputs from CI process via a product called Flags.

Carryforward Flags are specified in the codecov.yml, just like Codecov’s standard Flags. This is best demonstrated with an example codecov.yml, shown below:

This example creates three flags: ui, unit, and enterprise. In each case, two files are specified in the paths block for each flag; but folders, regex’s, and glob patterns are also accepted.

• The ui and unit flags are given the carryforward: true property,
• While enterprise is set to carryforward: false, the default value.

Consider two commits, Commit 0 and Commit 1 that contain the above flag specification. In this example, it is assumed that coverage has been uploaded for all three flags in Commit 0 (providing a coverage baseline). In Commit 1 a coverage report (ui_cov.xml) will be uploaded for the ui flag. No coverage report will be uploaded for the unit and enterprise flags for Commit 1. Given this scenario, the following diagram represents the state of Coverage at Commit 0 and Commit 1:

In this example, the coverage for ui_1.py and ui_2.py was taken directly from the coverage report uploaded for Commit 1. The ui_cov.xml coverage report was uploaded using the following syntax:

./codecov -f ui_cov.xml -F ui

This upload command specifies both the path to the coverage file for files under the ui flag and the appropriate flag name (ui in this case). Once uploaded, coverage was calculated to the ui_1.py and ui_2.py files in the same manner as any report upload.
Coverage remains the same with Carryforward Flags and no coverage reports

Since no report is uploaded for the unit flag, which has a setting of carryforward: true, Codecov reaches back to Commit 0 to carry forward the coverage for all files covered by the unit flag. As a result, those files have coverage information for Commit 1 that is equivalent to Commit 0.
Coverage drops with no coverage reports uploaded and Carryforward set to false

Finally, since the enterprise flag has a setting of carryforward: false no coverage is carried forward for any file covered by the enterprise flag. Since no coverage report was uploaded for the enterprise flag for Commit 1, the ent_1.py and ent_2.py files have 0% coverage for Commit 1.

---

The above example demonstrates the power of Carryforward Flags for those build processes which do not run their full test suites on every commit. Previously, projects that relied on partial test suite runs or delta builds would be unable to have a consistent picture of their project’s total coverage since total coverage would only represent whatever coverage information was uploaded for a particular commit. Now, with Carryforward Flags, Codecov can fully support these more complex testing approaches.