CI Provider Relationship

Codecov, like other static analysis tools, is an important step in the modern developers workflow.

Codecov does not run your test suite. That is the job of the CI Provider. Codecov gathers coverage reports and other key data for static analysis.

Codecov does not have any special arrangements with CI providers. We work with all providers as much as possible to create an easy and seamless integration with all the CI providers.

Checking CI status

Codecov checks all your CI statuses on every commit to make sure the coverage results are accurate. This process is very important to the inner working of Codecov and data accuracy.

If CI is pending, then the coverage results...

1. may be incomplete as there could be more uploads to Codecov for this commit;
2. should not be used to compare with other commits, as the results may not be complete.

If CI fails, then the coverage results are...

1. not complete because not all tests were executed;
2. invalid because different execution paths were taken due to exceptions and failed tests;
3. not representing a successful CI test case, and comparing reports would almost certainly lead to inaccurate results.

If CI passes, then the coverage results are...

1. merged and accurate;
2. able to be compared to previous commits;
3. sent to all notifications (Slack, etc.) once all reports are processed and merged.