The case for tests is clear every time a piece of code is shared. Whenever a shiny new codebase is handed to someone who can only focus on a subsection of it, tests enforce metadata about the code, declaring that it takes certain inputs and returns certain values. Developers can have the confidence to experiment and verify that their changes will work.
On the other hand, poorly written tests can become a hassle, an extra step to get modified every time code is refactored. To ensure that your tests can guide your product to reliability and stability, consider these ten rules for thoughtful, goal-oriented test writing.
1. Treat Your Tests Like a Product
Your tests require the same care you use for the rest of your code base. This means using comments, code review, and planning architecture. Plan for tests to be maintained and set aside some time for that maintenance. Like any other product, they should be regularly evaluated and improved. They take development time and should be designed with the goal of supporting and enhancing business critical infrastructure.
2. Don’t Repeat Yourself
The best practices you apply to your production codebase should also apply to your test suites. The first step toward maintaining code is keeping it organized, which helps avoid repetition. Be familiar with patterns for modularizing test cases, like factories, traits, and abstract classes. For example, this Python frontend test checks the title of the Codecov homepage and feature pages for matching text.
import selenium
from selenium.webdriver import Firefox
with Firefox() as driver:
url = "https://about.codecov.io/"
titleText = "Codecov"
driver.get(url)
assert titleText in driver.title
print url + " is working."
url += "product/features/"
driver.get(url)
assert titleText in driver.title
print url + " is working."
These tests work, but this code is prepared to break. The url variable is particularly fragile. Abstracting the page visit process will reduce duplication and make it less risky to add future tests.
def getBaseUrl():
return "https://about.codecov.io/"
def verifyPageTitle(driver, urlSubdirectory, checkText):
url = getBaseUrl() + urlSubdirectory
driver.get(url)
assert checkText in driver.title
print url + " is available."
with Firefox() as driver:
verifyPageTitle(driver, "", "Codecov")
verifyPageTitle(driver, "product/features", "Codecov")
verifyPageTitle(driver, "resources/", "Resources")
3. Know the ROI of Your Tests
Obviously, the code you write for a business has the end goal of making money. Similarly, tests should be written with the end goal in mind, targeted at a desired customer behavior or attitude. So a test’s first job is to protect the infrastructure supporting your business, often from getting accidentally chopped down by well-meaning developers.
The code in our previous example is used to test customer-facing pages. If those customers hit a 404 error, they may walk away from the service. A developer-facing page that’s down has a built-in internal reporting mechanism. A good goal is to test both, but it’s important to prioritize testing the things that impact the livelihood of your business.
4. Measure and Monitor Your Coverage
Code coverage is a metric that helps you track and understand the effectiveness of your tests. It’s a line-by-line analysis of what code is run by a test suite. Code coverage tools, like Codecov, can tell you exactly which pieces of code are covered by tests, and include analysis about partial coverage, like a test that only handles one branch of an if statement. Code coverage metrics help you understand where tests are present, and direct your efforts toward sections of code that are lacking coverage. Code coverage also allows you to track coverage over time, so you can report on and measure the growth of your testing program.
5. Create Deterministic, Idempotent Tests
These simple unit tests have an issue—they rely on state that exists outside of the test suite, so they have to run in a specific order. This can cause frustrating issues, particularly in situations where you’re running tests in parallel.
def isNumberEven(number):
if number % 2 == 0:
return True
else:
return False
def testOdd():
assert not isNumberEven(input)
def testEven():
assert isNumberEven(input)
input = 1
testOdd()
input +=1
testEven()
Tests should either always pass or always fail on any machine. Code is deterministic if it always returns the same result when passed the same variables; it doesn’t rely on state. If it always returns the same results whether it’s run once or a hundred times in a row, the code is idempotent. Idempotent tests don’t modify state.
To improve our tests, we can remove the global state variable input and move it into the unit test functions so they don’t make references or calls to any external state. These changes allow us to run our tests in any order, making it easier to understand what might be failing and making the test suite less brittle.
def testOdd():
assert not isNumberEven(1)
def testEven():
assert isNumberEven(2)
testOdd()
testEven()
6. Reduce the Use of Conditions in Tests
One strategy for building idempotent, deterministic tests is to avoid conditionals. Your tests should be simple and to the point, generally providing an input and expecting an output many times in a row. By the time input gets to a test function, you should know exactly what it will be. If statements guessing at state are often an indicator that code should be broken down into multiple test cases.
7. Tests Should Work Alone or in Parallel
Testing suites grow along with your code base. When a testing suite is small, running it on every commit isn’t a large buden, but you want to avoid a situation where a test suite becomes a drag for rapid deployments. This is when writing idempotent tests really pays off, because idempotent tests can be run individually or in parallel on multiple machines. This allows you to only test the parts of your code that were changed, and allows you to distribute testing resources across multiple machines, which is very helpful if you have a complicated deployment process or need your tests to return results quickly. Planning tests that work alone means you can refactor tests without fear of unintended consequences for other tests and allows you the flexibility to optimize how tests are executed.
8. Refactor Your Tests Regularly
Your tests are code and should be reviewed and refactored regularly. When code is refactored and functions are changed, added, or removed, data passed into functions often picks up a new parameter. Make sure you schedule development time to handle this in your test suites. Plan when you’ll review code and survey your test suites on a regular basis to make sure they’re still relevant and useful.
One of the benefits of monitoring code coverage is becoming more aware of unused code. Make it a priority to clear away the underbrush of unused code during refactoring in order to take advantage of this insight.
9. Minimize Waits
Tests that require “waiting” or “sleeping” are likely to cause issues as services interact differently on different machines. Adding waits to a test suite can culminate in a slow-moving test suite, especially if you run tests often. Testing asynchronous code sometimes appears to require wait or sleep steps, but try to incorporate a library designed to handle asynchronous waits in your language, or mock your classes to make them operate synchronously.
Dates are also hard to test. Consider using static dates as test inputs; it’s much more likely to mimic the actual input of your function, and it’s easier to test edge cases and irregularities than using a language’s built-in date function.
10. Remember to Test Boundary Layers of Your System
The external services your code relies on will fail, so your tests should account for this. Huge swaths of modern code rely on other systems to function, so if your code depends on external services, like API calls, you can anticipate and prevent problems by testing the boundary layers.
Having tests for these scenarios allows you to exert some control over code you might not have written. These situations often involve mocking external code, so plan how you’ll do this ahead of time. Decide if you’re able to partially mock a service by subclassing, or if you’ll have to fully mock it and write an implementation in your tests. Be sure to abstract this code so it can keep up with the times.
Conclusion
These ten rules for writing tests boil down to a few particular behaviors: treat your tests with the care and consideration you bring to the rest of your codebase, research and plan how to architect them, consider your priorities, and use your tests to support your goals.
Codecov’s code coverage tool makes following these rules easier, allowing you to visualize code coverage, monitor commits, and see how every commit changes coverage in real time. Teams under five developers can use the platform for free, so check it out here.
