Announcing Impact Analysis in Open Beta

Please visit our docs for relevant setup steps for each language:

• Python
• Ruby
• NodeJS
• Laravel

There are five steps to integrate Impact Analysis into your project (requires a Codecov account): 

1. Integrate the appropriate open source dependency into your code (see instructions for: Python, Ruby, NodeJS, and Laravel).
2. Copy the Impact Analysis token from the Codecov Settings page for your repository.
3. Use the token as directed by the particular language/framework’s implementation.
4. Update the repository’s codecov.yml with the proper configuration.
5. Deploy to production.

The following languages (and respective minimum versions) are supported: 

• PHP (7.4 or higher) applications using the Laravel Framework.
• Python (3.6 or higher)
• Ruby (v3.1 or higher)
• NodeJS (15.1 or higher)

Impact Analysis makes it significantly faster to answer the question, “what should I be testing next?” Here are just a few specific use cases where Impact Analysis can be a game changer:  

• Teams beginning a testing initiative on a deployed application who don’t know where or how to focus their testing strategy.
• Teams fielding an MVP who want to add tests to improve overall reliability but don’t know where to start.
• Teams porting a legacy application to a new codebase who don’t know where to direct their migration and testing efforts.
• Teams maintaining a codebase that’s grown beyond the point where individual developers know the landscape of the system in detail.

In order to use Impact Analysis you must have running production infrastructure to send Open Telemetry spans to Codecov. So, for example, an open source library that others ingest as a dependency would not make sense here.

However, on the back of this feature, we are working on something for open source maintainers, so hold tight.

Impact Analysis is a direct response to the engineering practices that many of the world’s pre-eminent software companies are already using.

For example, one of our early adopter development teams at a large company would often conduct lengthy internal surveys to tap their own knowledge and determine what parts of their code were the most critical to end users. Once known, these teams would build their own internal processes from scratch to ensure that developers knew when they were making changes to this critical code. While this work is extremely helpful to development teams, it’s arduous, takes months to perform, is subject to developer bias, and can rapidly become out of date. What once took months of manual work is now done automatically and continuously kept up to date.

Thank you to the private early access customers who have already set up Impact Analysis and are seeing value from the tool.

Additionally, internally at Codecov, we’ve used Impact Analysis for months to understand the impacts of our own code changes. Our teams have used it to refine their review strategy by more closely scrutinizing critical code changes and improving or adding tests to ensure our most critical code is adequately covered.

Impact Analysis was built using the emerging Open Telemetry standard. It’s facilitated via a runtime dependency of Open Telemetry. That dependency is added directly to your code, similar to a tool like Sentry, Segment, or Raygun.

Therefore, the security surface area of Impact Analysis is the same as using Open Telemetry generally.

Security FAQs:

1. What kind of information is collected by Impact Analysis?

   Once deployed, the Impact Analysis dependency sends to Codecov: lines of source executed by users, including file path, file name, line number, and execution count (but not actual source code, similar to a coverage report). In the case of HTTP requests, the request route and HTTP verb.

2. Where does Codecov store Impact Analysis data?

   For Codecov SaaS customers, in the same GCP environment alongside code coverage data uploaded by customers.

3. Can a customer’s Impact Analysis / OpenTelemetry data be deleted upon request?

   Yes, in the same fashion that customer’s code coverage data may be requested for deletion.

4. Who can access Impact Analysis data uploaded to Codecov?

   Currently customers in open beta for Impact Analysis do not have the ability to download span data that has been uploaded. Allowing for downloading will be a potential feature in the future.

5. What software is required to run the consumer Impact Analysis libraries?

   In order to use Impact Analysis, a Impact Analysis consumer library must be installed as a production-level dependency along with any required third-party dependencies required by the library. Specific dependencies vary based on the language of the Impact Analysis library in use; however, key requirements are specified in a dependency manifest file based on the language of the Open Telemetry instrumentation. An exception to this is the PHP consumer library’s requirement of PCOV which must be installed independently.

Thank you! Since this is a beta period, feedback is highly encouraged. It can be provided here in our public GitHub Issue “Your Feedback about Impact Analysis”.

Sure thing! Contact us here: https://about.codecov.io/contact/.

Absolutely. We’d be happy to share more. You can schedule a demo here: https://about.codecov.io/demo/.

Codecov utilizes Open Telemetry to collect occasional samples of running production spans. In our own utilization and testing, we observed <1% impact on running production infrastructure, and often below <0.1% performance.

Impact Analysis is free to use throughout the open beta period, slotted to run until September 1st, 2022.

Beyond the open beta period, Impact Analysis is planned to be included as part of the Codecov Enterprise package https://about.codecov.io/pricing/.

Good memory! That was the previous name of this same product, now called Impact Analysis. Input on the name or anything else? We’d love to hear from you “Your Feedback about Impact Analysis”.