Impact Analysis

About the Product

Impact Analysis uses information obtained during your application's runtime to provide contextual information to Pull Requests. Impact Analysis currently supports:

Critical Changes
1a. Labelling Pull Requests as Critical, such that important PRs in a list of currently open pull requests is emphasized.
1b. Labelling changed files as Critical in the Codecov PR Comment, such that important changes are the file level are obvious to the PR author and reviewer(s).

Impacted Entrypoints
2. Adding an Impacted Entrypoints table to the Codecov PR Comment, such that changes in a pull request are mapped directly to the most user facing aspect of the application

Impact Analysis was formerly marketed as Runtime Insights

📘

Impact Analysis is available in Open Beta

Impact Analysis is currently in Open Beta (full announcement here). It is a free add-on to Codecov during Open Beta, which is expected to run through September 1st, 2022.

What Languages are Supported?

Currently, not every language is able to benefit from Impact Analysis. The current languages and/or frameworks are supported:

• PHP (7.4 or higher) applications using the Laravel Framework
• NodeJS (15.1 or higher)
• Python (3.6 or higher)
• Ruby (v3.1 or higher)

How Do I use Impact Analysis?

📘

Looking to set up Impact Analysis?

It is most useful to select the Quick Start language from the list above in "What Languages are Supported" or in the the dropdown on the left under "Impact Analysis"

Below is a set of general needs of Impact Analysis in any language

In order to function, Impact Analysis requires the following:

1. A profiling token used to identify your repository and authenticate API requests. You can find your token in your repo settings page:

https://app.codecov.io/[gh/gl/bb]/[org_name]/[repo_name]/settings

2. A repository using one of our supported languages that is currently running in a production context.

3. Varying dependencies that are determined by the language and/or framework being used.

Specific installation instructions will be provided for your language of choice upon acceptance into the Early Access program.

codecov.yml Configuration

To see Impacted Entrypoints in your pull request comments, "betaprofiling" must be added to the comment.layout section of your codecov.yml, as follows:

comment: 
  layout: "reach,diff,flags,tree,betaprofiling"

If you wish to add "Critical" markers to impacted files, you must update your codecov.yml as follows:

comment: 
  show_critical_paths: true

These settings are also supported at the Global YAML level.

Impact Analysis FAQ

What is a Critical Change?

A Critical Change is any code change in a Pull Request that impacts code that is frequently executed in a production/runtime context. More specifically, if a line of code executed more than <> the average of all executed lines is edited, it is considered a Critical Change.

Can I use Impact Analysis with an open source repository?

In order to use Impact Analysis you must have running production infrastructure to send Open Telemetry spans to Codecov. So, for example, an open source library that others ingest as a dependency would not make sense here.

However, on the back of this feature, we are working on something for open source maintainers, so hold tight.

What are the security considerations of using Impact Analysis?

Impact Analysis was built using the emerging Open Telemetry standard. It’s facilitated via a runtime dependency of Open Telemetry. That dependency is added directly to your code, similar to a tool like Sentry, Segment, or Raygun.

Therefore, the security surface area of Impact Analysis is the same as using Open Telemetry generally.

What kind of information is collected by Impact Analysis?

Once deployed, the Impact Analysis dependency sends to Codecov: lines of source executed by users, including file path, file name, line number, and execution count (but not actual source code, similar to a coverage report). In the case of HTTP requests, the request route and HTTP verb.

Where does Codecov store Impact Analysis data?

For Codecov SaaS customers, in the same GCP environment alongside code coverage data uploaded by customers.

Can a customer’s Impact Analysis / OpenTelemetry data be deleted upon request?

Yes, in the same fashion that customer’s code coverage data may be requested for deletion.

Who can access Impact Analysis data uploaded to Codecov?

Currently customers in open beta for Impact Analysis do not have the ability to download span data that has been uploaded. Allowing for downloading will be a potential feature in the future.

What software is required to run the consumer Impact Analysis libraries?

In order to use Impact Analysis, an Impact Analysis consumer library must be installed as a production-level dependency along with any required third-party dependencies required by the library. Specific dependencies vary based on the language of the Impact Analysis library in use; however, key requirements are specified in a dependency manifest file based on the language of the Open Telemetry instrumentation. An exception to this is the PHP consumer library’s requirement of PCOV which must be installed independently.