Automated Test Selection with Github Actions

You can easily add Codecov's Automated Test Selection to your Github Actions, provided you're already setup with codecov.

This Action is currently in beta. Currently, it has only been tested on linux and macos builds using python and pytest.

If you have feedback or issues with running this action, please don't hesitate to let us know by creating a Github Issue.

Usage

1. Update the checkout step in GitHub actions to include fetch-depth: 0

      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

2. Add in a Codecov upload token (CODECOV_TOKEN) and a Static Analysis token (CODECOV_STATIC_TOKEN) secrets from the Codecov UI to GitHub.
   You can find the CODECOV_STATIC_TOKEN as the Static analysis token.

Set the Static analysis token to CODECOV_STATIC_TOKEN in your repository secrets.

3. Update your codecov.yml by adding the following

flag_management:
  individual_flags:
    - name: smart-tests
      carryforward: true
      carryforward_mode: "labels"
      statuses:
        - type: "project"
        - type: "patch"

cli:
  plugins:
    pycoverage:
      report_type: "json"

4. If pytest-cov is not a dependency, add it to your requirements.txt file, or run the following after you install your python dependencies in your GitHub Actions workflow.

- name: Install pytest
  run: pip install pytest-cov

5. Add the Codecov ATS Action to your CI. This should happen after you install python dependencies, but before you run tests.

- name: Run ATS
  uses: codecov/codecov-ats@v0
  env:
    CODECOV_STATIC_TOKEN: ${{ secrets.CODECOV_STATIC_TOKEN }}
    CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
# This is an example, do not copy below.
# - name: Run tests.
#   run: pytest ...

6. Update your pytest run to include the tests selected from ATS. You will need to read the list of tests that were selected to run by ATS. These tests are exported to codecov_ats/tests_to_run.txt.

- name: Run tests and collect coverage
  run: |
    cat codecov_ats/tests_to_run.txt | xargs pytest --cov app

7. If you are not already using the Codecov CLI to upload coverage, you can update the Codecov Action to v4-beta

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4-beta
  env:
    CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
  with:
    flags: smart-tests
    plugins: pycoverage,compress-pycoverage

8. Run your CI! On your first run, Codecov will not have any labels data and will have to run all tests. However, once all following commits or pull requests are rebased on top of this commit, you should be able to see the benefits of ATS.