Group coverage reports based on type of tests or sub-projects/teams.
Flags allow you to isolate and categorize coverage reports for different tests and features in your project. This is particularly helpful if:
- You have multiple types of tests (e.g., unit, integration, frontend, backend, etc)
- You're employing a monorepo setup where you'd like to encapsulate each project's test coverage independently.
To see the "flags" feature in action, you can also take a look at this video tutorial: https://about.codecov.io/product/feature/flags/
As a minimal example, consider a project called
Monorepo X that has the following folder structure:
Monorepo X /front-end /back-end /mobile
We'll refer back to this
Monorepo X example throughout the document.
Step 1: Flag Creation in Uploader
To apply a flag you must submit coverage reports with the
-F flagname included in the upload command. (Note: remember, Codecov always recommends the Uploader in your CI)
It is important to note that for flagging to work, separate coverage reports must be uploaded and flagged per project in the mono repo. For example, in a CI run for the
Monorepo X project example, assume that coverage for moduleA is written to
Then that report would be uploaded as follows:
./codecov -t <token> -f tests/front-end/output/coverage.xml -F moduleA
Now, your flag level coverage is being uploaded to Codecov.
# Ideal example of Flags matched to uploads # Your Codecov Uploader approach here # See more at https://docs.codecov.com/docs/codecov-uploader # example running unittests only py.test --cov=./ -k tests/unittests/ ./codecov -c -F unittests # example running integration tests only py.test --cov=./ -k tests/integration/ ./codecov -c -F integration # example running ui tests only npm test ./codecov -c -F ui
One-to-One Relationship of Flags to Uploads
Specifying multiple flags for a single report upload can result in erroneous coverage unless the contents of that report fully encompass each flag.
For example, one report that contains coverage information for moduleA and moduleB, can be uploaded in a manner that is valid but technically incorrect:
# Not ideal example of many-to-one Flags to upload ./codecov -t <token> -f coverage.xml -F front-end -F back-end -F mobile
This will apply the entire coverage of the report to both flags, resulting in incorrect coverage. Specifically, the
mobile flags will display the coverage of the entire uploaded report, not just the subset of the report that happens to cover the files under their purview.
Flags Must consist only of alphanumeric characters, '_', '-', or '.' and not exceed 45 characters
Flag names will be validated against the following regex.
Step 2: Flag Management in YAML
Once the uploader step is done (above), Flag-level coverage is being sent to Codecov, but, to unlock the full feature set of Codecov, you'll additionally to add Flag Management to your repository-level YAML.
Here are just a few features unlocked by adding Flag definitions to your YAML.
- Flags in PR Comments and Status Checks in your Github, Gitlab, Bitbucket instance
- Overlay source code coverage in UI by Flag
- Carryforward Flags for partial test runs
Two approaches to Flag Management
Recommended: Automatic Flag Management
Codecov's recommended approach to Flags is based on YAML settings under
A. Relies on a set of
default_rules: to automatically ingest uploaded flags, and manage Flags in an ongoing manner
B. If you have flags that don't fit the
default_rules:, Universal Flag Settings allows for customization for any flags using
Here is an example of
default_rules: in the YAML
flag_management: default_rules: # the rules that will be followed for any flag added, generally carryforward: true statuses: - type: project target: auto threshold: 1% - type: patch target: 90% individual_flags: # exceptions to the default rules above, stated flag by flag - name: feature_1 #fill in your own flag name paths: - src/feature_1 #fill in your own path. Note, accepts globs, not regexes carryforward: true statuses: - type: project target: 20% - type: patch target: 100% - name: feature_2 #fill in your own flag name paths: - src/feature_2 #fill in your own path. Note, accepts globs, not regexes carryforward: true statuses: - type: project target: 20% - type: patch target: 100%
Changing your YAML? A reminder to always validate before you deploy https://api.codecov.io/validate
Advanced: Bespoke Flag Management
If you are going to use Bespoke Flag Management, you must add every flag individually to your YAML, which can be quite manual.
To generate a flag and individual coverage gates per project in the mono repo the
codecov.yml can be structured as follows:
# Setting coverage targets per flag coverage: status: project: default: target: 90% #overall project/ repo coverage front-end: target: 60% flags: - front-end back-end: target: 100% flags: - back-end mobile: target: 80% flags: - mobile # adding Flags to your `layout` configuration to show up in the PR comment comment: layout: "reach, diff, flags, files" behavior: default require_changes: false require_base: yes require_head: yes branches: null # New root YAML section = `flags:` # This is where you would define every flag from your # uploader, and update when new Flags added flags: front-end: paths: #note, accepts globs, not regexes - src/front-end/code.js carryforward: false back-end: paths: #note, accepts globs, not regexes - src/back-end/api_code.py carryforward: true mobile: paths: #note, accepts globs, not regexes - src/new/mobile/app_code.java carryforward: true
This yaml configuration will fail a Pull Request if...
• ...the entire Monorepo project is below 90% covered (this is the
default specification of the yaml above)
• ...the front-end coverage is less than 60%
• ...the back-end (API) coverage is less than 100%
• ...the mobile coverage is less than 80%
front-end flag can be used for any number of reports (unit tests, integration tests, etc) by flagging each report with
front-end. The reports will be merged and contribute to the project's total coverage and the total coverage of the files covered under the
Remove old reports after uploading (optional)
-cargument to clear the workspace of all coverage reports before running you next set of tests.
Tracking Flag Coverage Over Time
The 'Flags' tab of your main repo navigation bar will allow you to visualize your historical flags data. This feature allows you to not just label portions of your code base, but use that labeling to track how that coverage is improving over time.
When you first click the tab you will need to "Enable Flag Analytics," this lets us know you'd like us to backfill your historical data. This process can sometimes take awhile, depending on the number of flags you have and how often you make a commit.
Once flag data has been backfilled you will be able to sort by flag name and filter to the date range of data you'd like to view. Forthcoming will be an ability to both search for and select only certain flags to view as well as other functionality to make this an easy to reference dashboard for your flag coverage. If you have feedback on this new feature, please let us know we'd love to hear what you think on this Github issue.
Additional Flag Use Cases
If you do not test all of your repo code on each commit, Codecov uses a feature called Carryforward Flags to help only update coverage on tests that were run. Carryforward Flags is built on top of basic Flags. We use (and need) a full upload of all flags when carry forward is being used to initially set the coverage.
Carryforward Flag Usage
It is highly recommended to read the full documentation on carryforward flags before using them in your project.
Carryforward Flags are used by appending your YAML with
flags: ui: paths: #note, accepts globs, not regexes - ui_1.py - ui_2.py carryforward: true unit: paths: - unit_1.py - unit_2.py carryforward: true enterprise: paths: - ent_1.py - ent_2.py carryforward: false # If no Carryfoward flag specified in YAML, the # default configuration is false.
Create custom notifications
Flags can be used to create custom notifications to your repository provider. This makes it easy to see per-flag coverage information alongside pull requests.
You can specify Flags in your Codecov Yaml for statuses and all notifications. Note that a flag must be specifically stated in the
status: section of the YAML to have a custom notification associated.
coverage: status: project: default: off frontend: flags: - frontend backend: target: 50% flags: - backend api: target: 89% flags: - api flags: # filter the folder(s) you wish to measure by that flag backend: # only include files in the backend folder paths: #note, accepts globs, not regexes - app/backend/ frontend: paths: - app/frontend/ api: paths: - app/api/ tests: paths: - tests/
Hide Builds (e.g., nightly builds)
Codecov provides a strategy to isolate specific builds from the master report while maintaining the report's integrity. When reports are not joined into the master report, they will be ignored for comparison, although they will remain accessible for source overlay, API, badges, and graphing.
A nightly build is an example of this feature. What follows is a Yaml configuration for a nightly build.
flags: nightly: joined: false
Now that we have configured the flag
nightly to not join into the master report, let's upload a report flagged as
# Your Codecov Uploader approach here # See more at https://docs.codecov.com/docs/codecov-uploader ./codecov -F nightly
-F nightly we mark all the coverage report data for this build as
nightly coverage data.
Updated 2 months ago