Status Checks

Project Status

The codecov/project status measures overall project coverage and compares it against the base of the pull request or parent commit.

coverage:
  status:
    project:
      default:
        # basic
        target: auto
        threshold: 0%
        base: auto 
        flags: 
          - unit
        paths: 
          - "src"
       # advanced settings
        branches: 
          - master
        if_ci_failed: error #success, failure, error, ignore
        informational: false
        only_pulls: false

📘

New to the Codecov yaml?

View the codecov.yml documentation here.

Basic Configuration

target

auto | <number>
Choose a minimum coverage ratio that the commit must meet to be considered a success.

• auto will use the coverage from the base commit (pull request base or parent commit) coverage to compare against.
• <number> you can specify a target of an exact coverage number such as 75% or 100% (string, int, or float accepted).

threshold

<number>
Allow the coverage to drop by X%, and posting a success status.

base

[Deprecated as of July 2020 -- Codecov's product roadmap includes a novel way to manually pick base of pull requests]

flags

Flags are a list of user defined Flags, and the impact on their coverage. You can specify an array of flags as follows:

coverage:
  status:
    project:
      default:
        #...
        flags:
          - flag1
          - flag2
          - ...

and the combined coverage from those flags will be reported. Before using flags, it is highly recommended to read more about their use.

paths

Similar to flags, an array of paths and/or regular expressions can be provided and the status will report the combined coverage for the files that match the path name / regular expression.

Advanced Configuration

While Codecov defaults should be sufficient for most use cases, below are some advanced features to personalize your experience even more.

branches

The branches that, when used, will trigger this status.

if_not_found

Settings are 'success' and 'failure' the default is 'success'.

• failure: the status will fail if there is no report for the head
• success: the status will pass if there is no report for the head. Use this on commits / PRs where you won't be uploading coverage but still want codecov status checks to pass.

informational

Use Codecov in informational mode. Default is false. If true is specified the resulting status will pass no matter what the coverage is or what other settings are specified. Informational mode is great to use if you want to expose codecov information to other developers in your pull request without necessarily gating PRs on that information.

only_pulls

Only post a status to pull requests, defaults to false. If true no status will be posted for commits not on a pull request

if_ci_failed

Settings are 'success' and 'error' the default is 'error'.

Options are:

• error: Will set the status to success only if the CI is successful
• success: Will set the status to success even if the CI fails

flag_coverage_not_uploaded_behavior

Determines how we handle status checks for which no flag coverage has been newly uploaded on a commit. This includes status checks where flags have carriedforward coverage (since coverage was no newly uploaded), as well as flags for which coverage is missing entirely.
Options are:

• include: (default) All the status checks defined in the YAML file will be processed and sent as normal.
• exclude: Status checks that haven't newly uploaded any flag coverage will not be sent.
• pass: Status checks that haven't newly uploaded any flag coverage will be passed automatically.

This is useful if, for example, you're working on a project in a monorepo and don't want to see status checks related to other projects.
Note that this only applies to status checks that specify flags. If there are no flags on a check, the status check will be processed as usual.

removed_code_behavior

These behaviors allows users to specify what should happen to the project status in case changes causes overall coverage to drop, but are part of good development practices.

removed_code_behavior motivating example:
Let's assume you start with a repo 9 out of 10 lines of code covered: 90% project coverage (the "base" of the PR).

Then, you open a pull request that adds 1 covered line, and removes 3 covered lines. No other lines are changed.

The new project coverage %=
9 covered lines + 1 new covered line - 3 covered lines: 7 covered lines
/
10 total lines + 1 new line - 3 existing lines: 8 total lines
New project coverage = 7/8 = 87.5%

removed_code_behavior options (with mathematical examples):

• fully_covered_patch: If the patch coverage is 100% and there are no unexpected changes, pass the project status. This is the default behavior;

From the example: the project check will always pass because the patch (1 added line) is 100% covered.

• False or off: don't apply any special behavior to project statuses;

From the example: the project status check (if status threshold: is default/not set or automatic) will fail because project coverage has gone down from 90% to 87.5%, a reduction of 2.5%.

• removals_only: If the changes consist of only removing code (i.e. no additions, no unexpected changes), pass the project status;

From the example: The project status check (if status threshold: is default/not set or automatic) will fail because 1 or more lines has been added and coverage has gone down from 90% to 87.5%, a reduction of 2.5%.

• adjust_base: consists in adjusting the BASE coverage by recalculating it, and then comparing it to the unchanged HEAD. The adjustment is to remove misses, hits and partials from lines that were removed in the diff from BASE (as if all removals were made already in BASE) and re-calculate coverage (coverage = hits / (hits + misses + partials)). This behavior is skipped if target coverage is defined.

From the example:
The project status check (if status threshold: is default/not set or automatic) will pass because adjusted base coverage =

9 starting lines in base MINUS 3 removed lines in the head of the PR = 6 covered lines in adjust_base
/
10 starting lines in base MINUS 3 removed lines in the head of the PR = 7 covered lines in adjust_base
Adjusted base coverage % = 85.7%

Therefore project coverage has increased to 87.5% (7/8 covered lines) from adjusted_base of 85.7%, an increase of 1.8%.

The example below shows where in your YAML you would set this particular behavior.

coverage:
  status:
    project:
      default:
        target: auto
        removed_code_behavior: fully_covered_patch # off, removals_only, adjust_base

📘

There's an explanation for why the status passed

If your project status passed because of removed_code_behavior there will be an explanation text at the end of the message. You might have to hover on the text to see the entire message.

Excluding status checks for unrelated projects (Example)

Below is an example of using the flag_coverage_not_uploaded behavior to omit status checks from unrelated projects.

coverage:
  status:
    default_rules:
      flag_coverage_not_uploaded_behavior: exclude # don't send status checks that don't have flag coverage uploaded
    project:
      projectA:
        target: auto
        flags:
          - projectA-unit
          - projectA-integration
      projectB:
        target: auto
        flags:
          - projectB

In this example, if a PR is opened in projectB then the status check for projectA won't be sent, as long as no coverage for projectA was uploaded on the PR.
Note that this requires that the CI process in projectB only uploads to Codecov using the projectB flag, and does not upload anything under the projectA flag. If your CI runs all tests and uploads coverage for every flag specified in the YAML file on every commit, then all status checks will appear as normal.

Excluding tests (Example)

Below, is an example of using multiple project statuses that measure different aspects of your project.

coverage:
  status:
    project:
      default: false  # disable the default status that measures entire project
      tests:  # declare a new status context "tests"
        target: 100%  # we always want 100% coverage here
        paths: "tests/"  # only include coverage in "tests/" folder
      app:  # declare a new status context "app"
        paths: "!tests/"  # remove all files in "tests/"

Now you will see two unique status contexts from Codecov: codecov/project/tests and codecov/project/app.

Splitting up projects (Example)

You may have a code base that has multiple application components that you would like to monitor independently. Codecov provides a very simple way create statuses for each component.

coverage:
  status:
    project:
      users:
        paths:
          - tests/users
          - app/components/user*
      products:
        paths:
          - tests/products
          - app/components/product*

Patch Status

The codecov/patch status only measures lines adjusted in the pull request or single commit, if the commit is not in a pull request. This status provides an indication on how well the pull request is tested.

coverage:
  status:
    patch:
      default:
        # basic
        target: auto
        threshold: 0%
        base: auto 
        # advanced
        branches: 
          - master
        if_ci_failed: error #success, failure, error, ignore
        only_pulls: false
        flags: 
          - "unit"
        paths: 
          - "src"

To illustrate the usage of the patch status, let's go through this exercise.

  def divide(x, y):
+     if y <= 0:
+         raise ValueError("y must be greater than 0")
      return x * y

The resulting codecov/patch status of this commit would be 0% covered because no tests are created for this method. Even though the project coverage is 72% (the entire code base not shown), this patch status will only measure lines added.

To make another commit on this pull request, adding tests, proceed as follows.

+ def test_divide_by_1(self):
+     assert divide(10, 1) == 10

Running the tests will result in a patch coverage of 50% covered because we have not yet tested the behavior of dividing by zero. Let's add another test.

 def test_divide_by_1(self):
     assert divide(10, 1) == 10

+ def test_divide_by_zero(self):
+     with self.assertRaises(ValueError)
+         divide(1, 0)
+

Now Codecov will report a codecov/patch status of 100% covered for this pull request. This indicates that the pull request adjusted code is properly executed by tests.

Disabling a status

You may choose to disable the default statuses Codecov posts by using the following yaml configuration.

📘

Enabling a status

Note that you can not enable a status by setting it to yes, you need to define it by using the structure described here

coverage:
  status:
    project: off
    patch: off

Changes Status

Codecov will detect changes in coverage that are NOT included in the commit/pull diff, and report these changes as a commit status.

Let's take this example to illustrate what an Unexpected Coverage Changes would look like.

As shown above, we have 100% coverage. Now let's make a change to this code base.

Our CI will run and result in the following:

Lines 1 and 2 are considered "changes" in Codecov. This status would detect these changes and report them to the commit status.

default_rules

A top-level default_rules field can be added to specify some behaviors to be applied by default to every status check, unless the status check definition itself explicitly sets a different behavior.
This can be useful for setting a behavior at a global level for all checks without needing to manually go through the definition of every individual checks. This can also be useful if you want to specify a global behavior but also make exceptions for individual checks.

Example:

coverage:
  status:
    default_rules: # this behavior will be applied to all checks by default
      flag_coverage_not_uploaded_behavior: exclude 
    project:
      projectA:
        flags:
          - projectA
      projectB:
        flags:
          - projectB
      projectC:
        flag_coverage_not_uploaded_behavior: include # this behavior will be applied only to this status check
        flags:
          - projectC

In this example, the exclude behavior for the flag_coverage_not_uploaded_behavior setting will be applied to projectA and projectB, but the include behavior will be applied to projectC because that project's YAML configuration explicitly sets a different value for that setting.

Fields that can be set under default_rules

Not all the status check configuration options can be set at a global level like this with default_rules. Currently only the following fields are supported:

• flag_coverage_not_uploaded_behavior