Introduction
Codecov 4.5.0 marks a major step forward for Codecov’s functionality, reliability, and usability. It is the culmination of more than 12 months of hard work to dramatically overhaul and refactor Codecov’s core components and is the first step of a larger plan to fully refactor and reimagine the entire Codecov product.
This release does not represent a full refactor of all of Codecov’s services, but rather marks the first point when we can release a major aspect of our refactor to enterprise users. Codecov will continue to evolve rapidly with subsequent 4.5.x version releases and will move to version 5.0.0 when the entire refactor is complete.
However, enterprise users are still highly encouraged to move to 4.5.x instead of waiting for 5.0.0. Far from being an intermediate release, 4.5.x will deliver tremendous value to end users all along the way.
Historical Context
The first commit to Codecov.io was made on Aug 3, 2014. In the intervening years, much has changed in the world of web software, and while Codecov leads the charge in code coverage, it has not always been on the cutting edge technologically. It is a business imperative to disregard technical debt in the short term if it helps to speed the discovery of product market fit in the long term. As a software product, Codecov was no different in this regard.
The first six commits to Codecov, from initial commit to released beta.
Codecov released that beta in only three days.
After more than five years of iteration, active development, and incredible customer growth, the time had come for us to pay the technical debt Codecov had accumulated over its lifetime. Therefore, in early 2019, we set off on a dramatic refactor operation while simultaneously continuing to ship features and improvements to our active customers.
At a high level, Codecov was in the following position as a product:
- Codecov was a large software monolith written primarily in Python. Page rendering, offline processing, interaction with external APIs, and all other business logic was handled out of the same repository. While this isn’t necessarily a negative, it was something we wanted to change.
- A large amount of quick decision making had resulted in a codebase that was incredibly tough for newcomers to the codebase to understand.
- Much of the library and framework choices were outdated, making it difficult to attract new engineering talent or get ongoing support in a timely fashion.
- Codecov has an aggressive 2020 product roadmap, and technical debt was making it more and more difficult to ship new features quickly and confidently.
- Bugs and corner cases were very difficult to reason through, track down, and remedy.
Projecting forward, we felt like the longer we put off a refactor to reclaim technical debt, the more insurmountable the task would become.
Refactor Goals
How we approached the refactor as an engineering team and as a business is a separate post in and of itself. But it’s important to outline the goals of this refactor at a high level as follows:
- Leverage common frameworks and software patterns, to make the codebase much easier to pick up for newcomers and make building new features more straightforward.
- Break the monolith into as few logical units as possible to help make a refactor more tractable, but still provide some separation of concerns.
- Squash as many bugs as possible, and make it possible to quickly diagnose and resolve new ones.
- Deliver value from the refactor quickly, and transition to using refactored code as soon as possible in production.
- Ensure that full knowledge and expertise of Codecov’s many systems was being distributed to as many engineers as possible along the way.
- Keep delivering features and improvements even as we were actively refactoring the codebase.
- As much as possible, modernize the initial software engineering choices made in 2014 to reflect the current state of the art in the development of SaaS software.
- Impact end users as minimally as possible and make the enterprise upgrade process as painless as possible.
What’s being Delivered with 4.5.0
The most noticeable shift to enterprise is that Codecov is now delivered as two docker images instead of a single one used in a multi-modal fashion. Consider the current 4.4.x docker compose based deployment:
web:
image: codecov/enterprise:v4.4.12
command: web
volumes:
- ./codecov.yml:/config/codecov.yml:ro
- archive-volume:/archive
ports:
- "5000"
worker:
image: codecov/enterprise:v4.4.12
command: worker
volumes:
- ./codecov.yml:/config/codecov.yml:ro
- archive-volume:/archive
Codecov 4.5.0 will deliver web and worker as two separate Docker images:
web:
image: codecov/enterprise-web:v4.5.0
command: web
volumes:
- ./codecov.yml:/config/codecov.yml:ro
- archive-volume:/archive
ports:
- "5000"
worker:
image: codecov/enterprise-worker:v4.5.0
entrypoint: sh worker.sh
volumes:
- ./config/development.yml:/config/codecov.yml:ro
- ./worker:/app/
This is done to leverage Codecov’s new asynchornous queue processor: worker, which is the first fully refactored service that will ship in Codecov 4.5.0.
This refactored worker will bring the following changes:
- Enterprise support for Carryforward Flags
- Dramatically improved logging, providing much more clarity into the operation of the worker to install maintainers
- Dramatically improved repository and team yaml parsing and validation, preventing many of the common errors encountered when creating team and repo yamls and exposing those common errors in logs when they occur
- Improved support for different and newly emerging software development strategies. Our approach to reconciling the git history as created via reverse merging, squash merging, and rebasing has been overhauled and is greatly improved.
- Far fewer bugs around issues that arise from use of the Team Bot and team and repository level yamls.
- Improved performance of the worker
- More worker metrics captured by statsD. These metrics can be piped to
Prometheus,Grafana,Datadog, etc.
How to Upgrade to 4.5.0
One of our core goals was to make the transition from 4.4.x -> 4.5.0 as painless as possible. For a vast majority of 4.4.x users, this upgrade will be as simple as pulling new images for the web and worker and restarting.
If you’re a Codecov Enterprise user that is still on 4.3.x, the upgrade path is a bit more involved, and we encourage any users in this scenario to reach out to us directly for assistance in upgrading.
For more information on the upgrade, view the release notes.
What if I Use codecov.io?
There’s no need to do anything. Codecov’s new worker has been in place on production for weeks, and currently processes 100% of all production job queues. If you’re using codecov.io, you have immediate access to all of the above benefits already.
What’s Next?
Codecov has big improvements in store for this refactor, and we will use 4.5.x release major fixes, features, and changes as we move toward Codecov Enterprise 5.0.0. We encourage all of our enterprise customers to move to 4.5.0 so they can take advantage of our numerous benefits and improvements as we make our way to 5.0.0 and continue pushing the envelope on what code coverage can do for modern software development teams.
What about 4.4.x and earlier?
Codecov Enterprise 4.4.x will receive six months of vulnerability and critical bug patching only. Since the migration path is, in many cases, incredibly straightforward, customers will be encouraged to upgrade as opposed to waiting for patched releases for 4.4.x. However, critical issues will be addressed until Nov 1, 2020.
All previous versions of Codecov Enterprise, including 4.3.9 and 4.3.10, will see an end of life on Nov 1, 2020.
