Delay notifications for Flags with after_n_builds

Codecov helps you get actionable code coverage metrics into your workflows. We send comments and status checks to a your pull request which contains a summary of their coverage information.

But sometimes, a project can be large and complex. Maybe you don’t send coverage uploads for every test suite on every commit. Or maybe you don’t need to know the status of every other project in your monorepository. Getting notifications from Codecov where Codecov hasn’t yet received all the necessary information can be annoying and frustrating. As a result, we built out after_n_builds to prevent Codecov from sending notifications until a user-specified n uploads.

For users who are already separating out their coverage based on uploads using Codecov Flags, this annoyance still existed as Flag status checks would still be waiting for the n uploads. As a result, we have released after_n_builds for Flags, so that you can specify how many uploads are in each Flag and will only receive notifications for that Flag after Codecov receives the n uploads for that Flag.

Why you might need after_n_builds for your Flags

If you’re not sure if you should be using Flags, consider how you are uploading your coverage. Do you upload your backend tests and your frontend tests separately? Do you want to know about just your unit test coverage versus your integration test coverage? Or maybe you have a monorepository, where each directory is separate from the others, and coverage for each project is independent.

If you are receiving notifications prematurely for Flags, you may need to use after_n_builds. Let’s take an example. If I have a monorepository with a structure as follows

.
├── projectA
│   ├── __init__.py
│   ├── calculator.py
│   └── test_calculator.py
├── projectB
│   ├── __init__.py
│   ├── emojis.py
│   └── test_emojis.py
└── codecov.yml

and I only work on projectA, I probably only care about the coverage for that project. In that case, I can tag all uploads for projectA with a Flag.

Now, let’s say projectA uploads 3 different coverage reports to Codecov. But projectB uploads only 2. In the past, you might have used after_n_builds over your entire project like

codecov:
  notify:
    after_n_builds: 5

But that would mean that you would have to upload coverage for projectA and projectB. If you only want to upload coverage a changed project, your codecov.yml might look like

codecov:
  notify:
    after_n_builds: 2

Now, however, you would be getting notifications from Codecov after only the 2nd upload. This could give you an incorrect view of your coverage. With after_n_builds for Flags, you could set it up to only send back notifications after a certain number of uploads per Flag.

Getting started with after_n_builds with Flags

In your codecov.yml configuration, you can add after_n_builds directly to any of your flags. For users using automatic Flag management, your yaml might look like this.

flag_management:
  individual_flags:
    - name: projectA
      paths:
        - projectA/
      after_n_builds: 3
    - name: projectB
      paths:
        - projectB/
      after_n_builds: 2

For users using the older Flag system, you can configure after_n_builds like this.

flags:
  projectA:
    paths:
      - projectA/
    after_n_builds: 3
  projectB:
    paths:
      - projectB/
    after_n_builds: 2

Both of the above setups will prevent any notifications for the projectA Flag from being sent until 3 uploads are received by Codecov. projectB will only need to have 2 uploads before sending notifications.

If you have any questions or feedback, please let us know on our feedback page. You can get started with Codecov and try the feature by signing up.