Carryforward Flags
Reference of past coverage for tests that are not run on current commit.
Carryforward Flags are Best Used for Monorepos and Iterative Testing Setups
Carryforward Flags are designed for projects that do not upload total coverage for every commit (e.g., monorepos with multiple applications/languages, iterative/partial/delta testing setups, etc).
Before Getting Started
To benefit from Carryforward Flags, make sure you are set up with Codecov's Flag feature, to tag different sessions/builds.
Basic Flags documentation, including:
- Set Flags in your YAML
- Passing
-F
parameter in your upload
A straightforward example repository using carryforward flags can be seen here:
In order to get the most out of carryforward flags, you will need to upload ALL coverage reports initially so that we have a baseline to carry forward.
How Carryforward Flags Work: A Motivating Example
Before discussing how to use carryforward flags, it's important to discuss how they work. That is best demonstrated with the following diagram:
The following codecov.yml is also supplied for the above example:
flags:
ui:
paths: #note, accepts globs, not regexes
- ui_1.py
- ui_2.py
carryforward: true
unit:
paths:
- unit_1.py
- unit_2.py
carryforward: true
enterprise:
paths:
- ent_1.py
- ent_2.py
carryforward: false
# If no Carryfoward flag specified in YAML, the
# default configuration is false.
In this example, the coverage for ui_1.py
and ui_2.py
was taken directly from the coverage report uploaded for Commit 1. The ui_cov.xml
coverage report was uploaded using the following syntax:
# Your Codecov Uploader approach here
# See more at https://docs.codecov.com/docs/codecov-uploader
./codecov -f ui_cov.xml -F ui
This upload command specifies both the path to the coverage file for files under the ui flag and the appropriate flag name (ui in this case). Once uploaded, coverage was calculated to the ui_1.py and ui_2.py files in the same manner as any report upload.
Since no report is uploaded for the unit
flag, which has a setting of carryforward: true
, Codecov reaches back to Commit 0 to carry forward the coverage for all files covered by the unit
flag. As a result, those files have coverage information for Commit 1 that is equivalent to Commit 0.
Finally, since the enterprise
flag has a setting of carryforward: false
no coverage is carried forward for any file covered by the enterprise
flag. Since no coverage report was uploaded for the enterprise
flag for Commit 1, the ent_1.py
and ent_2.py
files have 0% coverage for Commit 1.
A Carryforward Flag without a specific path?
You can use a regular expression as the "path", as in the example below.
This would be most common in a set of Integration or End-to-End tests run across the entire repo, but not run on every commit.
If, for example, you have a set of integration tests that test the full repository, but only run on specific commits or pull requests, you can do something like the following as the paths:
setting
- Unit tests have specific paths
- Integration tests run on all paths
# Example of a carryforward integration test suite for
# for the full repo only run on *some* of the commits
flags:
unit:
paths: #note, accepts globs, not regexes
- app/ORM/Models
carryforward: false
integration:
paths:
- ".*"
carryforward: true
Carryforward Flags in the Codecov UI
Once your team starts using carryforward flags they will appear in the UI just like normal flags, with the addition of distinctive iconography to set them apart from Codecov's standard flags. For example, carryforward flags can be toggled when viewing source:
Additionally, any coverage that is carried forward will be shown in the Build tab of a the commit that carried the coverage forward. The commit from which the coverage was carried forward is also referenced wherever carryforward flags are shown, like so:
Advanced: Configuring Carryforward Flags in the Code Host
Carryforward Flags in the Pull Request Comment
Carryforward Flags do not show in the pull request comment by default
By default, any flags that are carried forward will not show up in the pull request / merge request in your code host (Github, Gitlab, or Bitbucket) in the status checks and/or the pull request comments.
Instead, only flags that have coverage uploaded on the most recent commit will show in the PR Comments.
By default flags that are carried forward will not appear in the Pull Request Comment.
If you would like Carryforward Flags to appear in PR comments, use the following YAML configuration under show_carryforward_flags
.
The possible settings for:show_carryforward_flags
are:
-
false
[default]: flags with carried forward coverage will not show up on the flags table -
true
: if any flag coverage was carried forward, the flags table will have an additional column labeled “Carriedforward” indicating which flags had coverage carried forward
Example YAML:
comment:
layout: "diff, flags, files”
behavior: default require_changes: false
# if true: only post the comment if coverage changes
require_base: no # [yes :: must have a base report to post]
require_head: yes # [yes :: must have a head report to post]
branches: # branch names that can post comment
- "master"
show_carryforward_flags: false
Updated about 1 month ago