Since our security event in April, we have made numerous changes to our security posture at Codecov. Some of these changes were made immediately in response to the security event itself, others we made proactively to address security issues we have not yet experienced.
In no particular order, here are some of the ways Codecov has reinforced its security posture over the past two months.
Immediate Security Changes
In response to the security event, Codecov took measures to immediately remediate the situation.
This included actions such as rotating all of our internal keys, enhancing our monitoring on Google Cloud Storage assets, formation of a security task force, contracting a third-party forensics team, updating our Docker image deployment process, modifying our bash uploader validation implementation, and much more.
All of the immediate security changes made in response are fully discoverable in our security event post-mortem.
Security Policy Modifications
While we took many immediate actions, the security event forced us to revisit our internal security policies with a more critical eye.
We now have revised policies, procedures, and tooling around:
- Key Generation, Usage, and Rotation
- Software Distribution and Signing
- Enhanced Incident Response
Again, all of these new policies are highlighted in our post-mortem. We will continue to evolve and strengthen these policies on an ongoing basis.
Bash Uploader Validation
It was a customer that initially alerted us to the security incident in April. They were using the method of validation mentioned in our docs.
An observation that we made after the security incident is that this method of validation was not as visible as it could be. On top of this, we had not done everything we could to ensure we were using the method of validation ourselves. For example, our integrations with other vendors that relied on the bash uploader were not all using the validation method provided in our docs.
In response to this, we made several substantial changes that included:
- Making the validation process more visible in our documentation.
- Ensuring all integrations like our Circle CI Orb, Bitrise Step, and Github Action relied on proper SHASUM validation.
- Authoring an in-depth article documenting how to properly validate the Bash Uploader.
We are also proactively providing GPG key signature validation and SHASUM verification capabilities with our newly released Codecov Uploader.
New Codecov Uploader
For the last 8 months, Codecov has been developing a new uploader that does not rely on the bash script that we currently provide to our customers. We have been using an Alpha version of this uploader internally for quite some time.
This new Codecov Uploader uses NodeJS and is shipped as a static binary executable on the Windows, Linux, Alpine Linux, and macOS operating systems. It is easier to maintain, more secure, and brings the capabilities of our many language-specific uploaders under one roof.
Today we are happy to announce that the new Codecov Uploader is entering Beta status and is available for public use.
Hiring for Security
Although we are a small team, and we have historically taken on security efforts collectively, this event made it clear that we need to invest in an individual or individuals to properly steward Codecov’s security efforts.
After meeting with many prominent security professionals, we are opening two separate roles around product and infrastructure security at Codecov. You can find these roles posted on our careers page.
