GuidesAPIChangelogSupportStatusCodecov HomeTry Codecov for Free
These docs are for v4.6. Click to read the latest docs for v2023.

Self-Hosted Deployment Strategies

This page outlines three types of Codecov Self-Hosted deployment that can be used to meet a wide variety of user needs.

Suggest Edits

📘

For use with 4.4.8+

These deployment strategies assume you're installing v4.4.8 or later. Earlier versions of Codecov Self-Hosted need different considerations when deploying.

Please contact [email protected] if you are deploying an earlier version of Codecov Self-Hosted

Introduction

Being a containerized service, there are numerous strategies that can be utilized to deploy Codecov Self-Hosted successfully. How Codecov Self-Hosted is deployed ultimately depends on a combination of available resources, uptime needs, and planned usage.

This documentation is not intended to be an all-encompassing guide for deploying Codecov Self-Hosted. It should, however, serve to outline a few popular deployment strategies that can meet many common requirements of modern on-premises deployments.

Generally, deployment strategies fall along a spectrum of very easy to deploy but with low availability, to more challenging to deploy with high availability. Where a particular use case falls along this spectrum can be helpful in determine which deployment strategy should be used.

Unlike other reference pages in this documentation, which can be fairly easily skimmed, it is recommended to read this page in its entirety if you're considering a non-trivial deployment of Codecov Self-Hosted.

Deployment Types

This document will outline three deployment strategies, one of each of the following types:

  • Minimal Configuration, minimal availability -- This is great for trial and proof of concept style deployments where an organization is still attempting to determine if Codecov is right for the organization's needs.
  • Moderate Configuration, high availability -- This is great for small to midsize deployments that prioritize high availability now, but not necessarily incredibly large scale. This deployment is right for an organization that has decided to use Codecov, wants to ensure high uptime to end users, and desires the ability to move to a larger scale deployment in the future with no data loss and minimal additional effort.
  • High Configuration, high availability, high scale -- This is great for deployments that have to scale to meet the needs of many users within extremely large organizations. This deployment strategy is the one employed by Codecov's SaaS offering: codecov.io.

The remainder of this document will outline each of these deployment types in detail, as well as provide any reference materials necessary for configuration and setup.

Deployment Strategy I: Minimal Configuration, Minimal Availability

🚧

Trial Purposes Only

For all but the smallest of organizations, it is recommended to use this Deployment Strategy for trial purposes only.

The most straightforward deployment is to deploy Codecov Self-Hosted on a single server using docker-compose with no external services. If your goal is simply to evaluate Codecov Self-Hosted, you have no internal requirement to preserve the data generated by Codecov, and plan to start fresh with a more robust deployment if you purchase Codecov; then this is the recommended Deployment Strategy.

The greatest benefit to this deployment is that a single engineer should be able to produce a usable version of Codecov Self-Hosted in two hours or less. Therefore, it's great for testing out the product.

This Deployment Strategy utilizes Docker volumes for persistence of the database, archive file/object storage, and caching. Therefore, loss of the underlying server would result in total data loss. For this reason, it is recommended to use this deployment only for the trial and proof of concept stage of using Codecov Self-Hosted. However, included with this documentation are recommended modifications to make the data generated by this deployment persistent. It is recommended to follow those instructions if you with to upgrade your deployment in the future without any potential data loss.

How to Deploy

The easiest way to use this Deployment Strategy is to follow the Deploy with Docker Compose guide, and then move on to Self Hosted Configuration.

Modifying to Use Cloud Services

🚧

Using Cloud Services does not Guarantee Uptime

The below modification only ensures that data generated by Codecov Self-Hosted will persist past the lifetime of the server on which Codecov Self-Hosted is deployed.

If you require high availability / guarantees on uptime, please do not use this deployment.

This Deployment Strategy can be modified to support smaller deployments and/or to provide data persistence. This is useful if you do not want to allocate a large amount of resources to the installation, or if you would prefer to maintain data generated a trial/proof of concept period in the event that your organization decides to deploy Codecov permanently.

The following modifications will guarantee data persistence in the event that Codecov Self-Hosted's underlying server fails. However, this deployment provides no high availability guarantee, and it is recommended to use a different deployment to support that use case.

It is possible to use this deployment with cloud services in order to guarantee data persistence in the event of a loss of the underlying server running Codecov Self-Hosted. To do this the implementer must:

  1. Create a Postgres 10 database in their cloud provider (e.g., RDS, Google Cloud SQL)
  2. Create a cloud storage bucket (e.g., using S3 or Google Cloud Storage) for uploaded report storage.
  3. Create a redis server for dedicated caching (e.g., using Elasticache)
  4. Update the codecov.yml to use these services.
  5. Remove the postgres and redis blocks from the docker-compose.yml

Example configuration files are provided below. These files assume the user is deploying on a single EC2 instance in Amazon Web Services using GitHub as the repository provider:

setup:
  # Replace with the http location of your Codecov
  # https://docs.codecov.io/docs/configuration#section-codecov-url
  codecov_url: "<codecov-url>"
  codecov_api_url: "<codecov-url>" #Codecov v4.6 and later
  api_allowed_hosts: # Codecov version v4.6 and later
     - "<codecov-url>"
  # Replace with your Codecov Self-Hosted Key
  # https://docs.codecov.io/docs/configuration#section-enterprise-license
  enterprise_license: "<license-key>"
  # Replace with a random string
  # https://docs.codecov.io/docs/configuration#section-cookie-secret
  http:
    cookie_secret: "<your-cookie-secret>"
github:
  client_id: "<client-id>"
  client_secret: "<client-secret>"
  global_upload_token: "<upload-token>"
services:
  database_url: "<postgres-rds-url>"
  redis_url: "<redis-elasticache-url>"
  minio:
     host: s3.amazonaws.com
     port: 443
     bucket: <bucket-name>
     region: <bucket-region>
     verify_ssl: true
     access_key_id: <aws-iam-access-key>
     secret_access_key: <aws-iam-secret>
     #or
     #iam_auth: true

Note that if you're using GCP with Google Cloud Storage, your minio block should read as follows:

#for GCS
services:
    minio:
       host: storage.googleapis.com
       port: 443
       verify_ssl: true
       bucket: <bucket-name>
       region: <bucket-region>
       access_key_id: <gcs-hmac-key>
       secret_access_key: <gcs-hmac-secret>

You will need to do some additional work to setup Google Cloud Storage. You can find those instructions in the v4.4.8 changelog

With this configuration you can completely remove minio from your docker-compose.yml:

version: "3"

services:
  traefik:
    image: traefik:v1.7-alpine
    command:
      - --api
      - --docker
      - --docker.watch
      - --docker.constraints=tag==web
      - --metrics
      - --metrics.prometheus

      # NOTE: Toggle the lines below to enable SSL
      - --entryPoints=Name:http Address::80 Compress::true
      - --defaultEntryPoints=http
      # - --entrypoints=Name:http Address::80 Redirect.EntryPoint:https Compress::true
      # - --entryPoints=Name:https Address::443 TLS:/ssl/ssl.crt,/ssl/ssl.key Compress::true
      # - --defaultentrypoints=http,https

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/null:/traefik.toml:rw
      # NOTE: Provide the SSL certs in the ./config folder below
      # - ./config:/ssl
    ports:
      - "80:80"
      - "8080:8080"
      - "443:443"
    networks:
      - codecov
    depends_on:
      - web
  web:
    image: codecov/enterprise-web:v4.5.9 #or newer
    command: web
    volumes:
      - ./codecov.yml:/config/codecov.yml:ro
    ports:
      - "5000"
    labels:
      - "traefik.tags=web"
      - "traefik.backend=web"
      - "traefik.port=5000"
      - "traefik.frontend.rule=PathPrefix: /"
    environment:
      - STATSD_HOST=statsd
      - DATADOG_TRACE_ENABLED=false
    networks:
      - codecov
    depends_on:
      - statsd

  worker:
    image: codecov/enterprise:v4.4.8 #or newer
    command: worker
    volumes:
      - ./codecov.yml:/config/codecov.yml:ro
    environment:
      - STATSD_HOST=statsd
      - DATADOG_TRACE_ENABLED=false
    networks:
      - codecov
    depends_on:
      - statsd
  
  statsd:
    image: prom/statsd-exporter:v0.6.0
    command: -statsd.listen-udp=:8125 -statsd.listen-tcp=:8125
    ports:
      - "8125"
      - "9102"
    networks:
      - codecov

networks:
  codecov:
    driver: bridge

Note that the AWS Access Key and AWS Secret Key must be the identifying credentials for an IAM user that can specify the AWS bucket specified in the minio block of the above codecov.yml.

Scaling on a Single Server

Thanks to the use of Traefik, it is possible to scale both of codecov's services, the web and worker containers, on a single server. You should consider scaling this way if:

  • There is sufficient headroom on the underlying server
  • Report processing is lengthier than desired (e.g., status checks are taking longer than usual to complete on Pull Requests)
  • Web requests are taking longer than desired to complete (e.g., individual pages are taking awhile to load when viewing Codecov's web front end in the browser.)

Scaling is straightforward with: docker-compose scale <service-name>=<container count>. For example, to add more workers: docker-compose scale worker=3. Since Traefik is used as a reverse proxy in this deployment, new web containers should be automatically detected and used. Worker containers communicate via Redis, and should automatically begin processing tasks after startup.

Erroneous Modifications

❗️

The Following Configuration Changes Should be Avoided

Despite outlining the modifications in detail, it is highly recommended to avoid performing the below steps. These modifications are described because they are both performed often by users who are new to Codecov Enterprise and guaranteed not to work without unknown quantities of additional effort.

Given the simplicity of this deployment, it is tempting to do the following:

  1. Modify the docker-compose.yml and codecov.yml to use cloud services
  2. Spin up multiple EC2 instances using this configuration
  3. Put a load balancer in front of all the instances
  4. Proclaim that you have a high availability deployment.

This approach will not work. The reason being that this Deployment Strategy handles its own proxying between http/https based services using Traefik. Traefik is used in this case because it provides enormous convenience when deploying scalable/multiple services on a single server. However, if you place an external load balancer in front of multiple servers each running Traefik, the reliability of the underlying services is not guaranteed. Unless you're comfortable modifying/removing Traefik, and supplying your own reverse proxying solution for the web and minio containers, it is not recommended to try the above modification.

Deployment Strategy II: Moderate Configuration, High availability

👍

Suitable for Most Self-Hosted Users

This Deployment Strategy is the one generally recommended by Codecov. It strikes an acceptable balance between implementation difficulty, scalability, and availability.

This Deployment Strategy is very similar to Deploy Strategy I with the cloud services modification. However, it also provides an additional high availability / uptime guarantee by running multiple copies of each Codecov Self-Hosted service and load balancing them.

Use this Deployment Strategy if you require scalability, moderate ease of implementation and maintenance, and high availability to end users. This is the Deployment Strategy that is suitable for most Codecov Enterprise users.

The basic approach is the following:

  1. Decouple the services into the following: web and worker, minio; and deploy each on their own instance
  2. Place a load balancer in front of the instances running web and worker containers
  3. Optionally place a load balancer in front of the instances running minio. This is required if you intended to horizontally scale minio and/or require high availability of minio.
  4. Create the required cloud services, and update the codecov.yml to use those services.
  5. Modify docker-compose.yml.

In practical terms, you will need to create a single codecov.yml and two to three different docker-compose.yml's to support this deployment.

An example deployment in AWS would do the following:

  1. Create at least three EC2 instances: two for web and worker containers, one for minio. Though more can be created if desired.
  2. Create an Application Load Balancer for the web and worker containers. Optionally, one can be created for minio.
  3. Create RDS (postgres 10+), Elasticache (redis), and an S3 bucket for the deployment.
  4. Split the standard docker-compose.yml file into a web+worker file and a minio file and deploy them to the appropriate instances.

Example configuration files are shown assuming a deployment into Amazon Web Services:

setup:
  # Replace with the http location of your Codecov
  # https://docs.codecov.io/docs/configuration#section-codecov-url
  codecov_url: "<load balancer path>"
  codecov_api_url: "<load balancer path>" #Codecov v4.6 and later
  api_allowed_hosts: # Codecov version v4.6 and later
     - "<load balancer path>"
  # Replace with your Codecov Self-Hosted License key
  # https://docs.codecov.io/docs/configuration#section-enterprise-license
  enterprise_license: "<license-key>"
  # Replace with a random string
  # https://docs.codecov.io/docs/configuration#section-cookie-secret
  http:
    cookie_secret: "<your-cookie-secret>"
github:
  client_id: "<client-id>"
  client_secret: "<client-secret>"
  global_upload_token: "<upload-token>"
services:
  database_url: "<postgres-rds-url>"
  redis_url: "<redis-elasticache-url>"
  minio:
     host: s3.amazonaws.com
     port: 443
     bucket: <bucket-name>
     region: <bucket-region>
     verify_ssl: true
     access_key_id: <aws-iam-access-key>
     secret_access_key: <aws-iam-secret>
     #or
     #iam_auth: true

The web and worker containers can be merged into a single docker-compose.yml and deployed. Note that, most importantly, traefik has been removed:

version: "3"

services:
  web:
    image: codecov/enterprise:v4.4.8 #or newer
    command: web
    volumes:
      - ./codecov.yml:/config/codecov.yml:ro
    ports:
      - "8000:5000"
      - "80:5000"
    environment:
      - STATSD_HOST=statsd
    depends_on:
       - statsd
    restart: always
  worker:
    image: codecov/enterprise:v4.4.8 #or newer
    command: worker
    volumes:
      - ./codecov.yml:/config/codecov.yml:ro
    environment:
      - STATSD_HOST=statsd
      - DATADOG_TRACE_ENABLED=false
    depends_on:
      - statsd
  statsd:
    image: prom/statsd-exporter:v0.6.0
    command: -statsd.listen-udp=:8125 -statsd.listen-tcp=:8125
    ports:
      - "8125"
      - "9102"

If desired, you can split the web and worker docker-compose into separate docker-compose.yml files and deploy them separately. The worker does not require any sort of HTTP access to function properly, as it only communicates via Redis.

Deployment Strategy III: High Configuration, High Availability, High Scale

📘

Recommended for High Scale

The following deployment is a near replica of what is used for Codecov's SaaS offering: codecov.io. It is recommended when high availability and high scale are required, and maintainers don't mind taking on the additional burden of Kubernetes and Terraform.

This Deployment Strategy is recommended for the most demanding of Codecov Self-Hosted deployments. It is managed using Terraform and requires Kubernetes. Codecov has provided an terraform configuration files supporting this deployment for all three major cloud providers: Azure, AWS, and Google Cloud Platform. Those resources and supporting documentation can be found on GitHub

A Note on System Requirements and Resource Needs

There are two main components to consider when determining the performance of the underlying server(s) that will support Codecov Self-Hosted: Web traffic to the Codecov Self-Hosted frontend, Report processing demands.

Since every organization is different, it's impossible to make hard and fast recommendations that will meet every deployment's needs.

As a general building block, Codecov uses Google's n1-standard-4 for deployment of web containers and the n1-standard-8 for deployment of worker containers (see: Google Machine Types) .

By far the performance bottleneck on Codecov will be coverage report processing time. Therefore it's helpful to consider resource needs in terms of number of reports processed per minute. If your install will initially process 25 or less reports per minute (i.e., 25 or fewer commits running through a codecov enabled CI/CD per minute), it is recommended to deploy the equivalent of at least 1 n1-standard-8 for all infrastructure when using Deployment Strategy I; and 2 n1-standard-8's for web+worker (to provide high availability) and 1 n1-standard-4 for minio when using Deployment Strategy II. Once deployed, the performance of each instance can be monitored and scaled up or down as needed.

This is generally a good starting point, and is straightforward to scale horizontally when using Deployment Strategy II. If larger deployments are needed, the Codecov team is happy to provide more detailed recommendations depending on specific needs.

V4.6.X API container customization

The web container proxies API traffic to the API container. If you wish to change the API container host or port you need to specify environment variable(s) on the web container.

Kubernetes

spec:
      containers:
      - image: codecov/enterprise-web:latest-stable
        env:
          - name: CODECOV_API_HOST
            value: "your-custom-name (default: api)"
          - name: CODECOV_API_PORT
            value: "your-custom-port (default: 8000)"

Docker Compose

web:
    image: codecov/enterprise-web:latest-stable
    environment:
      - "CODECOV_API_HOST=your-custom-host (default: api)"
      - "CODECOV_API_PORT=your-custom-port (default: 8000)"

Updated about 2 years ago

What’s Next