Getting Started with Contract-Based Testing

Testing is a fundamental aspect of software development. It helps to enhance the quality of the product by confirming the software works and catching any potential bugs before the product is released to customers.

Contract-based testing, in particular, can help you speed up your tests on a microservice-oriented architecture without giving you too many headaches.

In this article, you will learn how contract-based testing compares to other types of testing and why it can be the best solution for your software.

Levels of Testing

There are multiple levels of testing that you can use on your product.

• Unit tests are generally the most common tests added to a project. The goal is to validate the code at the function or method level. If you have a sum function, then you want to check that 5 + 5 = 10. It’s generally easy to write and maintain such tests.

/**
 * the function to test
 */

const sum = (a, b) => {
  return a + b;
};


/**
 * the unit test
 */

test("adds 5 + 5 to equal 10", () => {
  expect(sum(5, 5)).toBe(10);
});

• Integration tests (or system tests) check interfaces between components. You can test a whole class or service, which generally involves mock to simulate an external interface that cannot be reproduced in the testing environment. It’s a bit harder to write integration testing as more code is involved, and the maintenance can be more costly. A lot of code is tested at once, so tracking down the issue can take some time.

• End-to-end (E2E) tests are the most complete tests, as the goal is to simulate being a final user of the product. You generally need to build a complete E2E environment with all the components of your application (all services, backend storage, etc.). You can work with tools like Postman to simulate REST calls or Cypress to simulate usage via a web application interface. Generally, you’ll write fewer E2E tests because they cost a lot in both running time and maintenance time.

Mike Cohn introduced the concept of the test pyramid in his book Succeeding with Agile. It’s a nice visual metaphor for understanding the testing levels.

Unit tests are easy to write and maintain, so you will have a lot of them (base of the pyramid). You need fewer integration tests, and finally just a few end-to-end tests.

This pyramid can help you to invest the right amount of time at the right level of testing. For further information about the test pyramid, read this article.

What Is a Microservice-Oriented Architecture?

A microservice-oriented architecture is the opposite of a more traditional, monolithic approach. Instead of building a single piece of software like an application running on a server, you can build a collection of services that are loosely coupled. Microservice architecture offers advantages like a smaller codebase and better flexibility and scalability.

But microservices present some challenges to testing. You can test each service in isolation (as with integration tests), or you can test your whole stack via E2E testing.

Unfortunately, testing each service separately doesn’t guarantee that the application will behave correctly for the user. And if service A is relying on a mock of service B in version 1.4.0, but service B is switching to 1.5.0 with a different API implementation, you can break your production without any issues at this level.

E2E testing requires you to build a complete environment with all required services, and the tests can take multiple seconds or minutes to complete depending on the complexity. Because there are many layers, you can end up with a lot of issues, and it will be difficult to track down which components have failed.

This is why contract-based testing is so common for microservice architecture.

Contract-Based Testing

Contract-based testing (CBT) is not a new methodology, but the concept is easy to understand in a microservice world. Let’s say you’re running a simple system with only two microservices, A and B:

A is consuming the service B. A is the consumer and B is the producer. The dialogue between the services is a simple HTTP REST call involving an information exchange.

A is requesting information about a user:

GET /users/julien

B is providing information about the user:

{
    slug: "julien",
    fullname: "Julien Bras",
    twitter: "_julbrs"
}

This dialogue is a contract. B is expecting a HTTP query with a specific path (/users/{slug}), and A is expecting the answer as a JSON object with the keys slug, fullname, and twitter.

The idea behind CBT is to rely on this contract, testing each side with the information in the contract:

Each test is simple and isolated (involving only one service), and you just have to test each side of each relation. This testing works equally well with complex relations (like a service with multiple services linked or a web UI that is consuming services).

On the test pyramid, you can place CBT between E2E tests and integration tests.

This example uses a classic REST API, but the testing works the same with other protocols (SOAP or messaging queue). The most important piece of data here is the contract that needs to be shared between the consumer and the provider. In order to get accurate testing and avoid the dependency isolation, it’s important to store the contract information in a central repository that can be accessed during the tests.

Consumer Driven or Provider Driven?

In the CBT there are two main approaches for contract definition.

Consumer Driven

If the contract can be defined by the consumer, it’s a consumer-driven contract (or CDC). The consumer must define exactly what it needs from the provider. This seems to be a more common approach, used by popular tools including Pact.

This can be a challenging option, though, if you have a provider that is consumed by multiple consumers. Each consumer will generate a contract, and the provider must implement a solution that will be valid for all contracts.

Provider Driven

In this scenario, the provider defines the contract and each consumer must comply with the contract. No adaptation is possible.

Your approach may vary depending on your product. This article will focus on consumer-driven contracts to keep it simple.

Tools in Practice

It’s possible to implement this type of methodology from scratch, but you will face some challenges:

• How to describe a contract efficiently
• How to share contracts between consumers and producers
• How to quickly discover the contracts based on existing interfaces

Here are the most common tools used for CBT:

• Pact describes itself as “a code-first tool for testing HTTP and message integrations using contract tests” here. Pact supports a large set of languages, and it tests that the service is working as expected by the contract. It works generally with a Pact Broker, the central repository that is used to share contracts. There is a deployable, open-source version of Pact Broker, or you can rely on a managed Pact Broker named Pactflow.
• Spring Cloud Contract was originally designed for JVM-based applications but has expanded to other types of projects as well.

CBT for Microservices

The following is a basic example of CBT using Pact on two Node.js microservices. The producer will expose a /users/ endpoint that will provide a list of users, and the consumer will read this endpoint.

Build the Consumer

Start by building the consumer so Pact can generate the contract.

In a folder initialize the consumer, named client, and import the needed libraries:

mkdir client
cd client
yarn init
yarn add dotenv express superagent
yarn add @pact-foundation/pact chai chai-as-promised mocha

Now add the following in the package.json to start the application and test it:

"scripts": {
    "test": "mocha",
    "start": "node index.js"
  }

Create a lib/user.js class to handle users:

class User {
  constructor(slug, fullname, twitter) {
    this.slug = slug;
    this.fullname = fullname;
    this.twitter = twitter;
  }

  toString() {
    return `User ${this.slug}`;
  }
}

module.exports = {
  User,
};

Create a lib/userClient.js to fetch users on the API:

const request = require("superagent");
const { User } = require("./user");

const fetchUsers = () => {
  return request.get(`http://localhost:${process.env.API_PORT}/users`).then(
    (res) => {
      return res.body.reduce((acc, u) => {
        acc.push(new User(u.slug, u.fullname, u.twitter));
        return acc;
      }, []);
    },
    (err) => {
      throw new Error(`Error from response: ${err.body}`);
    }
  );
};

module.exports = {
  fetchUsers,
};

The index.js is just using the fetchUsers() function:

const { fetchUsers } = require("./lib/userClient");

require("dotenv").config();

(async () => {
  // just returning the users from the API
  users = await fetchUsers();
  console.log(users);
})();

In pact.js, define the configuration of Pact (see here for details).

Finally, the most important file is test/user.spec.js, which defines the test. A mock service is started by Pact to simulate the provider. After the test, the provider.finalize() will write the contract file in pacts/userweb-userapi.json.

const chai = require("chai");
const chaiAsPromised = require("chai-as-promised");
const { eachLike } = require("@pact-foundation/pact").Matchers;
const { User } = require("../lib/user");
const expect = chai.expect;
const { fetchUsers } = require("../lib/userClient");
const { provider } = require("../pact");
chai.use(chaiAsPromised);

describe("Pact with User API", () => {
  // Start the mock service on a randomly available port,
  // and set the API mock service port so clients can dynamically
  // find the endpoint
  before(() =>
    provider.setup().then((opts) => {
      process.env.API_PORT = opts.port;
    })
  );
  afterEach(() => provider.verify());

  describe("given there are users", () => {
    const userProperties = {
      slug: "julien",
      fullname: "Julien Bras",
      twitter: "_julbrs",
    };

    describe("when a call to the API is made", () => {
      before(() => {
        return provider.addInteraction({
          state: "there are users",
          uponReceiving: "a request for users",
          withRequest: {
            path: "/users",
            method: "GET",
          },
          willRespondWith: {
            body: eachLike(userProperties),
            status: 200,
            headers: {
              "Content-Type": "application/json; charset=utf-8",
            },
          },
        });
      });

      it("will receive the list of current users", () => {
        return expect(fetchUsers()).to.eventually.have.deep.members([
          new User(userProperties.slug, userProperties.fullname, userProperties.twitter),
        ]);
      });
    });
  });

  // Write pact files to file
  after(() => {
    return provider.finalize();
  });
});

You can now launch the test by running yarn test:

Build the Provider

The provider part is also a simple Express.js service. Mimic what you’ve done for the client:

mkdir service
cd service
yarn init
yarn add express cors
yarn add @pact-foundation/pact chai chai-as-promised mocha get-port@^5.1.1

Add the following in package.json to start the application and test it:

"scripts": {
    "test": "mocha",
    "start": "node index.js"
  }

Put static data in data/users.js:

module.exports = [
  {
    slug: "julien",
    fullname: "Julien Bras",
    twitter: "_julbrs",
  },
  {
    slug: "tom",
    fullname: "Tom Sawyer",
    twitter: "",
  },
];

This is the core of the application. Put it in app.js:

const express = require("express");
const cors = require("cors");
const app = express();
app.use(cors());

const users = require("./data/users");

app.get("/users", (req, res) => {
  res.json(users);
});

app.get("/user/:slug", (req, res) => {
  const slug = req.params.slug;
  const user = users.find((user) => user.slug === slug);
  if (user) {
    res.json(user);
  } else {
    res.sendStatus(404);
  }
});

module.exports = {
  app,
  users,
};

This index.js file will fire the express server on port 5000 (you can test it with the client part):

const { app } = require("./app");

const port = 5000;

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`);
});

The file pact.js will configure Pact for the tests (see here for more details).

Finally, place the test file in test/user.spec.js:

const Verifier = require("@pact-foundation/pact").Verifier;
const chai = require("chai");
const chaiAsPromised = require("chai-as-promised");
const getPort = require("get-port");
const { app } = require("../app.js");
const { providerName, pactFile } = require("../pact.js");
chai.use(chaiAsPromised);
let port;
let opts;

// Verify that the provider meets all consumer expectations
describe("Pact Verification", () => {
  before(async () => {
    port = await getPort();
    opts = {
      provider: providerName,
      providerBaseUrl: `https://localhost:${port}`,
      pactUrls: [pactFile], // if you don't use a broker
      //   pactBrokerUrl: "https://test.pact.dius.com.au/",
      //   pactBrokerUsername: "dXfltyFMgNOFZAxr8io9wJ37iUpY42M",
      //   pactBrokerPassword: "O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1",
      publishVerificationResult: true,
      tags: ["prod"],
      providerVersion: "1.0." + process.env.HOSTNAME,
    };

    app.listen(port, () => {
      console.log(`Provider service listening on https://localhost:${port}`);
    });
  });

  it("should validate the expectations of User Web", () => {
    return new Verifier()
      .verifyProvider(opts)
      .then((output) => {
        console.log("Pact Verification Complete!");
        console.log(output);
      })
      .catch((e) => {
        console.error("Pact verification failed :(", e);
      });
  });
});

Here you’re using a contract file (pactUrls), not a broker, so you must copy the file client/pacts/userweb-userapi.json to service/pacts/userweb-userapi.json. Now you can test with yarn test:

As you can see, it passed, and the contract file was read during the process.

For a complete reference, use this GitHub repository that contains both services.

Advantages of CBT

There are several advantages to implementing contract-based testing in your applications.

No Specific Deployment Needed

Unlike with E2E testing, you don’t need to deploy a full stack to run CBT. You can easily run it on your local machine or inside your CI process. The tests are more or less like unit tests.

Faster to Run

The tests are generally faster than E2E testing, because they don’t require network interaction or storage or data backend. You will also get positive or negative feedback more quickly.

Easier to Debug

You can more quickly find the root causes of a bug with CBT. When an E2E test fails, any part of the stack can be the root cause, making detection harder.

Better Documentation

CBT helps you to formalize the various relations (contracts) between your services. It also prevents the dependency issue when services are shipped to production by different teams. CI can prevent the release if a contract is broken.

Reveals Unused Interfaces

CBT helps identify an unused interface. If the interface is not tested, then it’s probably not used by any of the consumers where you have implemented CBT. You can optimize the whole platform by removing unused interfaces and associated code.

In the example in the previous section, the provider contains a /user/{slug} interface to return a single user. As it’s not checked via any valid contract, it’s an unused interface.

Challenges of CBT

There are some things to keep in mind when writing contract-based tests. Here are some tips.

1. Test only the contract, not functionality. You may be tempted to test not only the interface and the dialogue between the consumer and the producer, but also the response of the producer. To detect that, you would need to combine CBT with other levels of testing like integration tests. You can find more information in this article.
2. Try to be contract first. It’s better to work first on the contract, then on the implementation. The contract will be used during the development and then after to write the tests.

This type of testing can pose difficulties. Here are some main challenges and how to handle them:

1. Infrastructure can be complex. CBT requires some adaptation and coordination from developer teams. You will probably need to implement a system to share the contracts between the services. The CI will be more complex, too, so that a service that isn’t passing API tests isn’t shipped to production. Do not reinvent the wheel—try to rely on existing, proven solutions.
2. Use the right paradigm. As previously mentioned, you must decide if the contract is written by the consumer or the producer. My recommendation is to start with the consumer-driven contract paradigm, as it requires fewer details for the consumer and is generally easier to work with.

Keep in mind that CBT is only one layer of your test pyramid and may not catch all errors. You can read more in this article.

Conclusion

CBT can be a nice addition to your test tool kit if you are managing a microservice application. When used well, it can replace an important part of existing E2E tests. Give it a try and see how it performs on your platform.