Hello 👋 My name is Isaac Halvorson. I work on the mobile infrastructure team at Tandem Diabetes Care, an insulin pump manufacturer. Tandem pumps are one of the few on the market that can be controlled by a mobile app. Our customers need our products and services to be reliable and performant at all times. We have little margin for error and need to ship the highest quality code we can.
“We love that it can track and report on our code coverage in total and on pull requests. Codecov helps us understand what parts of our codebase are covered and also helps us keep track of trends.“
I focus on developer automation and tooling of Tandem’s mobile apps. One of the newer tools we’ve been using at Tandem is Codecov. We love that it can track and report on our code coverage in total and on pull requests. Codecov helps us understand what parts of our codebase are covered and also helps us keep track of trends. For example, if a pull request would introduce a greater than 1% change to our total code coverage, merging it will be blocked until that is addressed.
The Problem
One of my focuses recently was finding ways to optimize our CI workflows. I noticed that for our iOS app, the Codecov step of our workflow was taking much longer than expected, so I decided to find out why, and if there was anything we could do to improve it.
Xcode collects code coverage data and can display it in the IDE for developers. It can also be exported as an .xcresult file when using xcodebuild from the command line, like so:
xcrun xcodebuild test \
-project tconnect.xcodeproj \
-scheme tconnect \
-testPlan tconnect \
-destination "platform=iOS Simulator,name=iPhone 14" \
-derivedDataPath DerivedData \
-resultBundlePath artifacts/ResultBundle.xcresultThese .xcresult files are great, and can be useful in lots of different ways, but like many things Apple, they can be difficult to use outside the Apple ecosystem.
xcresult files are a binary format, and while it can be converted into a JSON representation using the xccov binary included with Xcode, the resulting JSON is not in the standard coverage formats that Codecov can ingest. It also has been known to change without warning with new Xcode releases.
And so, in order for Codecov to use the coverage results from Xcode, they have to be converted into another format. Codecov’s official GitHub Action can do the conversion, but it handles this conversion by analyzing the coverage for each file one by one, which can take up to a second for each file. This is a fine enough approach for some projects, but when working with a large codebase like ours, that can take quite some time.
Enter xcresultparser
xcresultparser is an open-source Swift tool that can parse .xcresult files, and convert them into various other formats. One of these formats is Cobertura XML, which Codecov supports!
The big advantage xcresultparser brings is, because it is a compiled program and not a script, it can utilize multiple threads to do the conversion. This speeds up the conversion process immensely.
After running the xcodebuild command above to generate the .xcresult file, we tell xcresultparser to convert it like so:
xcresultparser \
--output-format cobertura \
"artifacts/ResultBundle.xcresult" >"artifacts/coverage.xml"And finally, we tell the Codecov GitHub Action to upload that .xml file instead of the.xcresult file.
Results
So, just how much time savings are we seeing?
| Total Build Time Before | Total Build Time After | Delta | |
|---|---|---|---|
| App Unit Tests | 18m 9s | 16m 12s | 1m 57s |
| Packages Unit Tests | 22m 8s | 15m 16s | 6m 52s |
We run these builds in parallel, so the total real-time savings for each build is the delta of the longer Packages Unit Tests build: about 7 minutes! This might not seem like much, but when you factor in that we’re running these builds upwards of 20 times a day, it’s a considerable time (and cost) savings. That’s over 2 hours of total developer time saved per day; almost 12 hours per week!
Bonus
While implementing xcresultparser for our project, I learned that it can also print a summary of test results to the command line. For our Packages Unit Test build, we run 8 separate test suites in serial, so if a test fails further up the log, it can be difficult to find it.
Now, at the end of each test run, we print out a summary like so:
xcresultparser \
--output-format cli \
--failed-tests-only \
"artifacts/ResultBundle.xcresult"This produces output that looks like this:
Now it’s very easy for our developers to see which specific tests had failures just by looking at the end of the test log in CI.
Conclusion
Using xcresultparser has improved the lives of our developers quite a bit. And the fact that it is open source means that we as a developer community can help improve it for the benefit of ourselves and others!
