Unexpected Coverage Changes


Looking for additional help?

See our video tutorial on diagnosing unexpected coverage changes at the bottom of this page.


We have found there are many reasons why coverage may change in unexpected ways.

Coverage changes are coverage to lines of code that are not adjusted in the pull/commit diff. This can occur for a number of reasons.

Codecov will analyze the pull/commit diff, detecting lines of code that changed without being adjusted. A simple example of this is removing tests, which is shown below.

## Unchanged source
def string_len(string):
   return len(string)

## Removed test
- def test_my_method():
-   assert string_len('jedi') == 4

In the code above, the developer removed a single test which changes the coverage of the method string_len from being hit to missed.

This change in coverage will be surfaced in the Codecov App through our Changes page.

Reasons for unexpected changes

There are a number of reasons we have found that cause line coverage to change unexpectedly. These can be due to:

  1. Adding or removing tests.
  2. Failing to upload coverage reports, or a different number of reports between head and base
  3. Time sensitive tests.
  4. Missing coverage reports or failed builds.
  5. Dependencies changed resulting in a different execute plan.
  6. Encrypted variables may prevent some execution paths.
  7. In the case of changes to a PR, the base or head commits may not have had an uploaded report, or may not be the commit you expect

How to find the root cause

Most unexpected changes occur in the raw coverage report before Codecov processes the uploaded coverage file.

How to review the the coverage report for unexpected changes:

  1. Proceed to the Builds tab when viewing a single commit.
  2. Click the Download button on one or more of your uploads.
  3. Scroll through the file network and discover your uploaded reports.
  4. Review the report, looking for the line(s) that are not reported accurately.
Scroll through the data to find all the uploaded reports.Scroll through the data to find all the uploaded reports.

Scroll through the data to find all the uploaded reports.

Codecov stores the raw, unprocessed report data precisely for this issue. Reports are archived for 30 days before removal.

Video Tutorial

Example repository from video: https://github.com/kensho-technologies/graphql-compiler/pull/806

Did this page help you?