Unexpected Coverage Changes

📘

Looking for additional help?

See our video tutorial on diagnosing unexpected coverage changes at the bottom of this page.

Introduction

There are many reasons why coverage may change in unexpected ways. Codecov analyzes the pull/commit diff, detecting coverage changes on both lines of code that changed and lines that were not changed.

Indirect coverage changes are differences of coverage to lines of code that are not adjusted in the pull/commit diff. This can occur for a number of reasons. In the UI we call these changes "indirect changes" - on the pull request page you will see an "indirect changes" tab and navigating there will bring you a list of all of the files impacted by indirect changes. More often than not an unexpected coverage change is an indirect change.

A simple example of this is removing tests, which is shown below.

## Unchanged source
def string_len(string):
   return len(string)

## Removed test
- def test_my_method():
-   assert string_len('jedi') == 4

In the code above, the developer removed a single test which changes the coverage of the method string_len from being hit to missed.

This change in coverage will be surfaced in the Codecov App through our Changes page.

Reasons for indirect changes

There are a number of reasons that cause line coverage to change unexpectedly. These can be due to:

1. Adding or removing tests.
2. Failing to upload coverage reports, or a different number of reports between head and base
3. Time-sensitive tests (e.g., one case before 2pm UTC, a different case after 2pm UTC).
4. Missing coverage reports or failed builds.
5. Dependencies (e.g. package.json, requirements.txt) changed resulting in a different execute plan.
6. Encrypted variables (e.g. secrets, tokens) may prevent some execution paths.
7. In the case of changes to a PR, the base or head commits may not have had an uploaded report, or may not be the commit you expect
8. Different error handling paths: sometimes exception handlers are tested and some are not, oftentimes the root of "flaky" tests
9. Due to long running test builds or dynamic test selection, you are intentionally not running all tests on every commit. If this is the case, please check out our feature Carryforward Flags

👍

Indirect Coverage Changes

Almost every example above will manifest as an indirect coverage change in the "indirect changes" portion of the pull request page shown in the screenshot below.

How to find the root cause

Most unexpected changes occur in the raw coverage report before Codecov processes the uploaded coverage file.

How to review the the coverage report for unexpected changes:

1. Open the head and base commits for this PR, seen below in the "Source" section surrounded by a green box below.

2. On the right hand side of a commit page you will find the raw coverage reports listed in order of upload
3. The first piece to investigate is the number of reports themselves. Are they the same between the two commits? If they're different, that's likely the cause of the discrepancy. In the example above, you can see in the highlighted box there's "2 errored, 14 successful" - you want to see an identical result between both commits to rule this out.
4. Click the Download button on one or more of your uploads - if you're seeing coverage changes on a specific flag, it might be easiest to focus on the relevant reports to that flag.
5. In your downloaded file, ctrl+f on "# path=" to take you past your file list and directly to the beginning of your raw coverage reports.

6. Review the raw coverage reports from one commit to the next looking for discrepancy in coverage, if you have a hit on the file from one commit, but not the file from the other commit, that's going to be the source of your coverage change.
7. You'll next need to review the specified lines of code with different coverage reports from both commits to determine the root cause.

Codecov stores the raw, unprocessed report data precisely for this issue. Reports are archived indefinitely unless otherwise specified in YAML.

Video Tutorial

Example repository from video: https://github.com/kensho-technologies/graphql-compiler/pull/806