Static Analysis for Automated Test Selection

Details about Static Analysis

What is Static Analysis?

Static Analysis is the process used by Codecov to extract information from your code. We never store the code itself, only certain metrics and relationships between lines of code.

Static Analysis is powered by TreeSitter, a well-known and performant parser generator tool. This process happens in your CI. The processing results are uploaded to Codecov.

For every file, much like git does, we hash the contents of it and use that to index the file snapshot. So if we have static analysis information of your code base you only need to upload new information for the files that have been changed effectively. This means that across multiple commits only few files will need to be uploaded.

To use ATS Codecov needs the static analysis information to exist for the two commits being compared. If you want to use ATS we recommend you run static analysis in all your commits. Check Getting Started with ATS - General guide for more details on integrating ATS.

Supported Languages

List of languages supported by static analysis.

  • Python
  • JavaScript

The static-analysis CLI command

To use the static analysis you have to use the static-analysis command of the Codecov CLI.

This command searches for files to be analyzed, runs the analysis locally. Then it checks with Codecov what files need to be uploaded and uploads the results that need to be updated to Codecov. If the file snapshot saved in Codecov is up-to-date it will not be uploaded again (unless --force is used).

Usage: codecovcli static-analysis [OPTIONS]

Options:
  --foldertosearch PATH      Folder to search
  --numberprocesses INTEGER  number of processes to use
  --pattern TEXT             file pattern to search for
  --force / --no-force
  --commit-sha TEXT          Commit SHA (with 40 chars)  [required]
  --folders-to-exclude PATH  Folders not to search
  --token TEXT               The static analysis token (NOT the same token as
                             upload)  [required]
  -h, --help                 Show this message and exit.

Above are the list of options for the static-analysis command.

  • The default --pattern is *
  • --force option will upload results of all files to Codecov, including those already saved. Default is False
  • If you are running this command locally we recommend using --folders-to-exclude=.venv, to exclude files from your virtual env from the search
  • The CLI will look in CODECOV_STATIC_TOKEN for the value --token if one is not specified.

Speeding up static analysis

You can speed up the static-analysis command by restricting the number of files that it analyses. By using a combination of foldertosearch, folders-to-exclude and pattern.

Let's consider the example below.

my project
ā””ā”€ā”€ apps/
    ā”œā”€ā”€ my_app/
    ā”‚   ā””ā”€ā”€ tests
    ā”œā”€ā”€ my_other_app/
    ā”‚   ā””ā”€ā”€ tests
    ā”œā”€ā”€ shared/
    ā”‚   ā””ā”€ā”€ tests
    ā””ā”€ā”€ experimental

Suppose that I'm only using ATS in my_app, and that my_app uses some of the code in shared. In this case I could use static-analysis only in my_app and its subfolders and shared. One option to accomplish that could be as below.

$ codecovcli static-analysis \
	--folders-to-exclude=apps/my_other_app
	--folders-to-exclude=apps/experimental