No one likes digging through logs to figure out why a test failed. Even less fun: wasting time on flaky tests. That’s why we made Test Analytics – a new set of tools that call out flaky tests and tell you exactly why a test failed all in the PR comments. It’s in beta now and free to use while in beta.*
Say no more. Take me to the docs!
When a test fails, and you’re not entirely sure it’s a flaky test, what do you do? You re-run it, right? What a total waste of time! Refreshing your browser every 30 seconds to see if the test passed the second time even though you already know this process is a roughly 30-minute wait-and-see process. Flaky tests make your entire CI pipeline unpredictable and keep you from mashing the merge button and moving on to the next issue/feature/task.
With Test Analytics, you can spend less time debugging test failures because Codecov will tell you why they failed or if they’re flaky as soon as the test fails in your PR. Better yet, now you can do your team a massive favor and fix those flaky/broken tests to save everyone the headache. If you’re looking for more, you can get a full list of frequently failed tests and flaky tests in Codecov – add those to your backlog and then humble brag about fixing them.
Test Analytics in the PR Comment
The updated Codecov PR comment tells you if your PR is failing due to an issue with your proposed code changes, or due to a systemic flaky test. Without opening another tab or window, see the line(s) failing, commit a fix, or simply re-run the test and 🤞it passes on the second or third try.
Tests Dashboard in the Codecov UI
To help you plan your test suite optimizations, we also added Test Analytics into the Codecov UI. Here, you’ll see a simple summary of the tests in your test suite, including average duration, failure rate, commits failed, and time last run. This will help you understand just how much your tests may be impacting CI performance – and therefore developer experience.
CI taking too long? Sort by Average Duration and see what tests run the longest. Or, sort by Failure Rate to see what tests may be unreliable.
Getting Started with Test Analytics
If you’re already using Codecov, all you need to do is:
- Instruct your test runner to output a test result file in `junit xml` format
- Upload test results using the Codecov Test Results Action, or the CLI
- Create a new commit and you’re good to go!
You can find all the information, instructions, and prerequisites you need in the Test Analytics Guide. Looking for a real working example? Take a look at how we set up Test Analytics on Codecov’s Front-End project.
*Just a heads up, Test Analytics is in beta and free to use during this period. When we GA Test Analytics, failed test detection will be included on every plan and flaky test detection will be available on our Pro and Enterprise plans.
Not using Codecov yet? Get started.
Join the Conversation
Test Analytics is now available for all Codecov users in Open Beta. Experience the difference it makes in streamlining your test management process and accelerating your code review cycles. Say goodbye to tedious test failure diagnosis and hello to a smoother development workflow.
If you have thoughts or suggestions, don’t hesitate to share them with us on GitHub.
