Unexpected Coverage Changes

📘

Looking for additional help?

See our video tutorial on diagnosing unexpected coverage changes at the bottom of this page.

Introduction

There are many reasons why coverage may change in unexpected ways.

Centrally, these occur in files that are not directly changed in the pull/commit diff. This can occur for a number of reasons.

Codecov will analyze the pull/commit diff, detecting lines of code that changed without being adjusted. A simple example of this is removing tests, which is shown below.

## Unchanged source
def string_len(string):
   return len(string)

## Removed test
- def test_my_method():
-   assert string_len('jedi') == 4

In the code above, the developer removed a single test which changes the coverage of the method string_len from being hit to missed.

This change in coverage will be surfaced in the Codecov App through our Changes page.

Reasons for unexpected changes

The main reason to expect a coverage change on a file not affected by the code change is: Adding or removing tests.

There are a number of reasons that cause line coverage to change unexpectedly in Codecov (called out in our Pull Request page as "Indirect changes"). These can be due to:

1. Which tests are triggered change and/or the results of tests change between test runs (e.g., tests result during the base commit differ from the head commit of a PR/MR).

   a. Different tests run: For example, Dependencies changed resulting in a different execute plan or encrypted variables may prevent some execution paths.

   b. Flaky tests: For example, different error handling paths result in some exception handlers being tested tested and some not, oftentimes the root of "flaky" tests or time-sensitive tests (e.g., one case before 2pm UTC, a different case after 2pm UTC).

   c. Due to long running test builds or dynamic test selection, you are intentionally not running all tests on every commit. If this is the case, please check out our feature Carryforward Flags

2. Differences in what coverage report uploads Codecov received (e.g., the individual CI builds are different between base commit and head commit)

   a. Failing to upload coverage reports, or a different number of reports between head and base

   b. Failed builds leads to missing coverage reports.

   c. An unexpected commit serves as the base (or head) of the comparison for Codecov, which leads to Codecov to errantly compare two commits.

How to find the root cause

Most unexpected changes occur in the raw coverage report before Codecov processes the uploaded coverage file.

How to review the the coverage report for unexpected changes:

1. Open the head and base commits for this PR, seen below in the "Source" section surrounded by a green box below.

3. On the right hand side of a commit page you will find the raw coverage reports listed in order of upload
4. The first piece to investigate is the number of reports themselves. Are they the same between the two commits? If they're different, that's likely the cause of the discrepancy. In the example above, you can see in the highlighted box there's "2 errored, 14 successful" - you want to see an identical result between both commits to rule this out.
5. Click the Download button on one or more of your uploads - if you're seeing coverage changes on a specific flag, it might be easiest to focus on the relevant reports to that flag.
6. In your downloaded file, ctrl+f on "# path=" to take you past your file list and directly to the beginning of your raw coverage reports.

7. Review the raw coverage reports from one commit to the next looking for discrepancy in coverage, if you have a hit on the file from one commit, but not the file from the other commit, that's going to be the source of your coverage change.
8. You'll next need to review the specified lines of code with different coverage reports from both commits to determine the root cause.

Codecov stores the raw, unprocessed report data precisely for this issue. Reports are archived for 30 days before removal.

Mismatching BASE and HEAD Commit Upload Counts

A common cause of unexpected coverage changes is unintentionally uploading a different number of coverage reports for the BASE and the HEAD of a pull request. In the event that this happens, the Compare page will show the following warning:

Unless you are using Carryforward Flags, it is fairly uncommon for a different number of reports to uploaded for any given commit. If this is the case, it can be a sign of the following in either the BASE or HEAD commit:

• Failed tests resulting in some number of coverage reports not being uploaded.
• Flaky tests resulting in a differing number of coverage reports being uploaded between different commits.
• An issue with Codecov that occurred during the commit's CI run, leading to an error receiving the coverage report(s) during upload.

🚧

This warning will not trigger if Carryforward Flags are Used

Since it is incredibly common for upload counts to differ per commit when using Carryforward Flags, this warning is suppressed for any PR that has uploads carried forward on either the BASE or the HEAD

In all three of these cases, the warning can be an indication that the CI run(s) of both the BASE and HEAD commit should be reviewed to ensure those runs completed properly and uploaded to Codecov as expected.

In some cases, a differing number of uploads between the BASE and HEAD of the PR can and should be expected, such as when using Carryforward Flags. For example, if enough tests are added/removed between the BASE and HEAD to facilitate adding/removing entire coverage report uploads, then the number of uploaded reports on the BASE and HEAD will rightfully differ. If a different number of BASE and HEAD commits for a particular PR are expected, this warning can be disregarded.

Video Tutorial

Example repository from video: https://github.com/kensho-technologies/graphql-compiler/pull/806