Release notes for Codecov v4.6.1
[Self-hosted Only] Ensure You have Upgraded to Codecov v4.6.0 Before Using v4.6.1
Starting with Codecov v4.6.0, new infrastructure is required to successfully run Codecov Self-Hosted. Please refer to the v4.6.0 changelog for more information.
[Self-hosted Only] Backup your Database Before Upgrading
Codecov v4.6.1 introduces a new database migration mechanism. This new approach allows us to make database and migration changes much more efficiently and effectively. While it has been tested heavily internally, and is currently in use in our cloud offering, we recommend backing up your Codecov database before upgrading to v4.6.1.
New Features
Beta Release of the New Codecov Uploader (Self-Hosted Only)
The Codecov Bash Uploader will be Deprecated in Self-Hosted
Future releases of Codecov Self-Hosted will contain both the Bash Uploader and the Codecov Uploader. The Bash Uploader will be deprecated on the first Self-Hosted release that falls after Feb 1, 2022.
Codecov v4.6.1 includes the new Codecov Uploader. This various versions of the Uploader are accessbile via:
https://<your-enterprise-url>/uploader
To use this new uploader you can follow the instructions for Self-Hosted in our public documentation: https://docs.codecov.com/docs/codecov-uploader#self-hosted-use-of-the-uploader.
More information about Codecov's new Uploader can be found in this blog post.
Golang: Count Partials as Hits
Golang's native coverage processor counts partial lines as full hits. Codecov's default behavior in the event of partial coverage is to store the line as a partial. This behavior will ultimately reduce the coverage total since Codecov calculates the coverage percentage as: hits / (hits + misses + partials) * 100.
Therefore, due to the presence of partials, Golang's coverage calculation will generally be higher that Codecov's for any given project. This discrepancy can be confusing for Golang developers, particularly in cases where the lower coverage calculation for Codecov can result in failed patch and project statuses on Pull Requests.
To remedy this discrepancy, Golang teams can now add the following to their Codecov yaml:
parsers:
go:
partials_as_hits: true #false by default
This setting can be provided in the repo, team, or install codecov.yml file.
Rust Notification Processing (Self-Hosted Only)
In our cloud offering, Codecov has replaced key parts of our notification workflow with Rust to prevent performance-based bottlenecks that we were encountering with our original Python implementation. Codecov v4.6.1 opens up our Rust processing to Self-Hosted users. You can enable Rust based notification processing in the worker by adding the following environment variable:
RUST_ENABLE_RATE=1
Setting this variable to 0, or removing it altogether will disable Rust processing.
Our Rust notification processing has been shown in to reduce timeouts and errors, as well as increase general performance in workflows use cases that are notification heavy (e.g, many flags, status, or coverage report uploads).
Upload Processing Delay Configuration (Self-Hosted Only)
Codecov processes reports via a queue as they come into the system. In some cases, particularly those that upload many reports (>50) reports for a single commit in rapid succession, report processing can fail The core issue is that Codecov is attempting to process a report for a single commit as many more are entering the system, leading to locking issues for shared resources.
When this problem arises, it can be mitigated by modifying the worker's processing delay for a single commit. This delay allows many reports to be ingested for a single commit before processing begins, reducing the need for multiple worker processes to wait for shared resources.
In many cases, the default processing delay will be sufficient; however, if you are experiencing processing problems particularly for commits that upload many coverage reports in a short time interval, the following install is recommended as a tool to help.
The change would be made at the self-hosted install-level yaml (not the repo-level yaml).
# Added to the install-level yaml file, not the repo yaml
setup:
upload_processing_delay: 30 #time in seconds, default is None.
If your use case requires this setting, it will likely take a bit of experimentation to determine the appropriate upload_processing_delay for your use case. It may helpful to measure the amount of time that transpires between the start and end of coverage report upload to Codecov in your CI pipeline for all coverage reports, and set the upload_processing_delay to that amount of time in seconds. This approach will ensure that all coverage reports are uploaded for a commit before processing starts.
You can, and should, optimize this parameter after finding a setting that works. As the upload processing delay is applied to all uploads and commits, not just those that upload a large number of reports.
/upload Migration to API Service (Self-Hosted only)
The endpoint for handling uploads to Codecov has moved to the API service, which itself was introduced in Codecov v4.6.0. This change impacts both the v2 and v4 versions of the endpoint. If you have successfully upgraded to Codecov v4.6.0 already, there should be no additional configuration changes required.
Moving /upload to the new API service provides the following:
- Improved logging in the event of failure conditions
- Generally more performant and reliable upload handling
Logs that would normally be accessible on the Web service during upload time are now viewable in the API service logs. If you use an external logging service (e.g., Splunk, Datadog, an external ELK stack, etc) it will be important to ensure that your logging system is properly capturing logs for the API service.
v4.6.1 Full Changelog
New
- (Self-hosted only) [Beta] Users can now leverage Codecov's new Uploader in Self-Hosted.
- (Self-hosted only) [Beta] Users can now opt to have notifications handled by Rust instead of our default Python implementation. This feature is experimental but has been shown to dramatically improve performance, reduce timeouts, and prevent certain classes of errors in some cases. This is a beta feature that will be enabled by default in a future release (see Rust Notification Processing) .
- Users can now enable Codecov to count partial coverage in Golang as hits. This functionality was introduced to help Codecov's coverage output remain consistent when compared to Golang's native output (see Golang: Count Partials as Hits below).
- (Self-hosted only) Codecov's processing delay for uploaded coverage reports is now configurable (see Upload Processing Delay Configuration).
- (Self-hosted only) Updated internal database migration mechanism, this should provide more straightforward, performant, and reliable migrations moving forward. Migrations are now run from the api container on startup.
- (Self-hosted only) Added functionality for rotating the application's encryption key in the codecov.yml (see Encryption Key Rotation).
- (Self-hosted only) Both v2 and v4 versions of the
/uploadendpoint have moved to the API service. - Improved the nav header and general top level navigation in the UI.
- The Codecov Pull Request Comment will now display a warning message when the current pull request head differs from the head of the branch itself.
- Codecov will now attempt to use multiple base paths to match files in an uploaded report if more than one obvious base path is present in the report. This should lower instances of report processing failures that occur due to path matching errors.
Fixes
- Fixed a bug where the avatar in the header could sometimes be displayed with scroll bars.
- Codecov will now properly set the updatestamp of a pull request on subsequent update webhook events.
- Fixed a subtle race condition where the PR comment can be sent using the wrong (usually older) head commit,
- The Slack notifier will now send notifications in the instances that coverage on a PR cannot be meaningfully compared.
- Fixed a bug where the report joining configuration was occasionally ignored during processing.
- Fixed an issue where Salesforce APEX coverage report processing would sometimes evaluate to a
nullcoverage value. - Fixed a bug that resulted in startup failures in the event that no Redis host password was provided.
- Fixed a Bitbucket login issue that was preventing some users from logging in properly.
- Fixed an issue with the Node coverage processor that occasionally resulted in fractions being treated as partials.
Minor
- Updated self-hosted startup output to provide more helpful messages in the event of database or general connectivity issues.
- Updated enterprise license warning messages in the UI to be more clear and meaningful.
- Added a default avatar for Bitbucket users in the event that the true Bitbucket avatar cannot be fetched.
- Added more helpful system-level logging when database errors are encountered.
- Added the ability for Codecov to compensate for a known gcov bug that produces negative execution counts.
Release Update
Note: due to a user impacting bug in v4.6.1 a point release was issued and pushed over the existing v4.6.1 and latest-stable tags for the worker and api. The old and new digests are indicated below:
codecov/enterprise-worker:v4.6.1:
- old sha256:667886d1dba0536fceaee197fca5e1b2b153396a88375dde3ba6ba9a22d92c7b
- current sha256:749957b38246488925d5304e1ff64b582c7df3b86878229168fd60cb8a974d8b
codecov/enterprise-api:v4.6.1
- first: sha256:777347a80150ea7fae61ccca53eb42ad7259319f3cfda753e01dfeec96466889
- second: sha256:cff17c0ffabacdd6a97b3fd8ea34d38432b8932b6c7bef34d8a8568b763236f3
- third: sha256:6e5b5a939c2d91967b4fbefb20fa638ad95a30debf5dc2bcad059556fce2ae7a
- fourth: sha256:d6f37a81f107cd057dfc14c08dfba46f42f8d115913cb9c83e84598eba8cc905
- latest: sha256:e5ec359db0bddb864a8712d91e894ade3aba7cdf787574ab79050bdca680fa84
codecov/enterprise-web:v4.6.1:
- old: sha256:c5c53327250343f6bf27b7c8f862825f9d70c9d3cc04d5b4426d0b6de8231791
- current: sha256:a29267aa299c7808b9d5b662f6275f0a87d8fde0670fcba485a0edab2cf6504c