Writing web services or applications that consume APIs is a common part of web development. Be it either a web service that is developed internally or by using an existing API from a 3rd party.

When writing tests against an external API one can face a number of issues

• tests fail due to connectivity issues
• service has a limited hit rate and responds with errors after a while
• service does not exist yet or is incomplete
• authentication is not possible or access is restricted
• no development / staging server available
• communication with the API leads to high payload or has slow response times
• interacting with the API directly might lead to side effects in the service

To avoid these issues we want our tests to not hit any 3rd party API while still respecting the API functions available to us. Furthermore our test suite needs to pass in a repeatable manner, fast and without any side effects.

When we develop against an external API a common way to communicate with it is by using a dedicated library from either a 3rd party or by writing one ourselves. Typically there is already a ruby library available to us if the development is for a known platform such as twitter. But even when it's for an internal API it makes sense to group the functionality into a single library to provide reusability or to offer others a convenient way to interact with our API.

There are a couple of solutions available that differ depending on the context and the requirements of the API.

Example

To illustrate the different ways on how to test an API we use the following example. We use a client library to request a list of todos from an external service. First we assume there is some client that has a method to return a list of todos.

class ApiClient
  def fetch_todos
    # sends a HTTP request and returns a JSON response
    ...
  end
end

Then our initial test might look like the following:

require 'spec_helper'
describe 'ToDo API request' do
  it 'returns a list of todos' do
    client = ApiClient.new

    response = JSON.parse(client.fetch_todos)

    expect(response).to be_an_instance_of(Hash)
    expect(response['todos']).to be_an_instance_of(Array)
    expect(response['todos'].size).to > 0
  end
end

In this example an ApiClient instance is created and a list of todos is requested. The specific implementation is not important. Let's assume the ApiClient can be configured correctly and it sends a request to the external service over HTTP and responds with a JSON string.

Stubs and Mocks

The ruby community has a strong focus on testing as part of the development cycle. Among the most often used techniques are stubs and mocks. These are usually used when a class under test interacts with other parts of the system and depends on their outcomes, e.g. writing to a file, sending data to a queue etc. From the perspective of the class under test we are more interested in the interaction with the other entities as long as they behave as expected.

Stubs and mocks are a good way to accomplish this. To apply this to the above example we might want to encapsulate the client in a wrapper object, for example a class called TodoRepository.

class TodoRepository
  def initialize(client)
    @client = client
  end

  def get_todos
    @client.fetch_todos
  end
end

There are some good reasons why we want to encapsulate a client, especially from a 3rd party:

• it minimizes the dependency & interaction with the API client, especially if it's a 3rd party library
• if only a subset of the client functionality is used by our code we only expose what is really needed
• use of the more general client can become more specific to our domain by not exposing details of the client. Instead of handling hashes or strings directly we can introduce domain specific objects, e.g. a ToDo class
• our code becomes more robust against changes in the client, the wrapper acts as a safeguard
• we gain control over logical aspects of the client, for example if the client sends a request when instantiating it, we can delay this
• it becomes easier to test, it allows us replace the client

The example above can be adjusted to the following version where the client method is stubbed.

require 'spec_helper'
describe TodoRepository do
  it 'returns a list of todos' do
    todos  = [{ 'subject' => 'A to do for today' }]
    client = ApiClient.new
    allow(client).to receive(:fetch_todos).and_return(todos)

    response = TodoRepository.new(client).get_todos

    expect(response).to eq todos
  end
end

This is a partial stub, it creates a concrete intsance of the ApiClient class and stubs the method fetch_todos which returns a list of hashes. This might not be feasible for example if a HTTP connection is opened in the class constructor already. The TodoRepository.get_todos method could also return a list of Todo objects to be more domain specific. It might also handle client specific exceptions in a consistent manner and throw domain specific ones.

Another way is to use a test double (mock) as the client.

require 'spec_heler'
describe TodoRepository do
  it 'returns a list of todos' do
    todos  = [{ 'subject' => 'A to do for today' }]
    client = double('client', :fetch_todos => todos)

    response = TodoRepository.new(client).get_todos

    expect(response).to eq todos
  end
end

This injects the mock into the TodoRepository object. It requires that the public methods (its interface) which are accessed conforms to the one from the ApiClient class. If they differ it can lead to passing tests while the real client would fail, therefore it is necessary to update the client's interface.

As we can see stubs and mocks are useful when we are able to isolate the client. Sometimes this cannot be easily done, for example if the use of the client library is spread over a lot of different locations or we have to deal with an existing code base where changes cannot be easily introduced or refactoring takes a lot of time. If we develop the API client ourselves stubs and mocks are a good way to test the interaction within our application, but we still would like to test the interaction with the external API directly and therefore need other techniques as well.

Using stubs and mocks for a 3rd party client might become a tedious task, especially if the behavior is too complex or too many public API functions are used. Then mocking the client or stubbing its methods can lead to a lot of effort and maintenance work. There are a couple of projects specifically designed to offer fake implementations of existing libaries for use in test environments, e.g. see fakeredis or pusher-fake. These fake implementations behave exactly as the original clients but can be used in tests without depending on external services.

WebMock

Another approach to stub requests and responses is to use the WebMock gem. It is specifically designed to stub HTTP requests in Ruby and works with all common HTTP libraries and test frameworks. WebMock exchanges specific HTTP requests that matches certain criteria, e.g. request type, URL, query arguments, headers and returns a defined HTTP response. Instead of sending a HTTP request to the external service, a defined response is returned locally. Stubbed requests work on the communication layer where HTTP requests and responses are handled.

The first thing we can do when using the WebMock framework is to disable all net connections but the localhost.

# in spec_helper.rb
require 'webmock/rspec'
WebMock.disable_net_connect!(:allow_localhost => true)

This raises exceptions for all net connections, still allows communcation via localhost and is a good way to find out what requests are send to external services in our tests. An exception also provides details of how the specific request can be stubbed.

WebMock::NetConnectNotAllowedError:
[...]
You can stub this request with the following snippet:

stub_request(:get, "http://www.example.com/todos").
    with(:headers => {
           'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
           'Content-Type'=>'application/json',
           'User-Agent'=>'Faraday v0.9.1'
         }).
    to_return(:status => 200, :body => "", :headers => {})

This stubs the HTTP request (type GET) for the URL http://www.example.com/todos. Let's stub the request in the test of our example.

describe ApiClient do
  before do
    response = '[{ "subject": "a todo for you" }]'
    stub_request(:any, /www.example.com/).to_return(:body => response, :status => 200, :headers => {})
  end

  it 'returns a list of todos' do
    client = ApiClient.new(:host => 'www.example.com')
    response = client.fetch_todos

    expect(response).to eq response
  end
end

This maps the HTTP request of the ApiClient when it calls the host www.example.com to the stub and returns the defined response. The :any request type means all request types are stubbed (POST, PUT, DELETE, etc). This technique also provides a good way to check error responses. It can be complemented by a list of fixtures for different requests, where a stubbed request loads a JSON string from a file and returns it. A comprehensive and informative example of how an API can be tested with stubbed requests and fixtures is the twitter gem. Furthermore stubbing requests can also be used to build a client that acts correctly when an API function returns different status codes for various query arguments or supports different formats.

WebMock is a powerful library when developing a client against an (external) API or when the code has a lot of dependencies to different services. In the former case it makes no sense to stub the client directly, because it is the object under test. In other cases stubbing everything might be tedious or error prone, because a lot of complexity is involved to produce a response. Therefore it can be advantageous to stub HTTP requests with WebMock and to serve fixtures.

VCR

Another popular library when testing HTTP dependencies is the VCR gem. VCR records the responses of HTTP requests once and replays them later when the same request happens again. The responses along with the requests are stored in so-called cassettes. The first time a cassette is recorded the HTTP request hits the external API. Once the recording is stored the same HTTP request will replay the recording and serve the response.

This allows a couple of things:

• improves test speed with recordings
• allows to run the tests repeatedly and offline (assuming no new requests are made)
• contains the same headers and body as the real request
• reflects HTTP communication most closely

VCR has support for all common HTTP libraries and test frameworks and can be easily configured.

# spec_helper
require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = 'fixtures/.cassettes'
  c.hook_into :webmock
end

The given configuration stores the cassettes in fixtures/.cassettes and uses WebMock to stub the HTTP requests with the recordings. It integrates well with WebMock and supports metadata information useful when specifying tests.

To test our previous example with VCR add a metadata argument to the test description:

require 'spec_helper'

describe ApiClient, :vcr => true do
  it 'returns a list of todos' do
    client = ApiClient.new(:host => 'www.example.com')
    response = JSON.parse(client.fetch_todos)

    expect(response).to be_an_instance_of(Hash)
  end
end

When the test runs for the first time our ApiClient hits the real external API, records the response and when running subsequent tests replays it.

VCR comes with a couple of features which are very useful when developing a client for an external API. One useful feature is to see if there are any unused recordings. To raise an error and see which recordings are not used, instead of setting :vcr => true provide the argument :vcr => { :allow_unused_http_interactions => false }. It lists all HTTP requests for which a recording exists but are not called from the test suite.

VCR supports different record modes that define how requests are recorded and replayed. The default mode :once will replay previously recorded requests, record new ones for which no cassette is available, but will raise an error if there is a new request for which a cassette already exists. The latter case can happen if different test cases stub the same request again and is a good indicator where to group tests to use the same setup. The mode :new_episodes will record all new HTTP requests as well. The :none mode rejects all new HTTP recordings not previously recorded and will raise an error otherwise.

Sometimes an external API changes, for example a path changes or the response differs. In these cases the previously recorded VCR cassettes would still pass all tests. It is therefore good practice to regularly update the cache and record the requests again. One way to do this is to specify the option re_record_interval in the configuration:

VCR.configure do |c|
  # ...
  c.default_cassette_options = {
    re_record_interval: 60 * 30  # in seconds
  }
end

Another option is to set the record mode to :all temporarily which re-records everything.

When the VCR cassettes are checked into the version control systems such as git, no sensitive data (username, password) should be stored inside the recordings. A good way is to filter sensitive data from the recordings and have them replaced with real values before sending any requests. For example during the VCR configuration set the following filter entries:

VCR.configure do |c|
  # ...
  c.filter_sensitive_data('<USERNAME>') { ENV['API_USERNAME'] }
  c.filter_sensitive_data('<PASSWORD>') { ENV['API_PASSWORD'] }
end

This configuration replaces <USERNAME> and <PASSWORD> with values from environment variables. It is suggested to use enclosing brackets for better indication of which variables need to be replaced. If everything is set up correctly the cassettes contain the placeholders instead of the user's credentials.

VCR is a good option when hitting the external API from the test suite is not a problem, e.g. staging server. If the client or the API is still in development, VCR recordings might be a good choice between running tests in a fast and repeatable manner while still recognizing changes.

Fake Service

The last option described here is how to use a fake service to test an external API. A fake service is similar to a mock in a way where all requests are stubbed. But instead of stubbing requests inside the tests a small web application is started and HTTP requests are redirected to it. In this way the fake service mimics the external API in its behavior and responses, substituting the real service with a version running locally. This means building a fake application takes a bit more development effort and maintainance work then stubbing requests, but the tests become cleaner. It also works nicely with the other described options.

A fake application might be a good fit for the following reasons

• only a small subset of the external API is used
• the external API is stable enough (e.g. versionised), consistent and well understood
• the API is well documented with samples, but not available fully or still in development
• external API is a RESTful service
• adjusting the existing test suite means a lot of effort, due to extensive calls to external APIs
• test data needs to be set up and you want to have control over it

We use the web framework Sinatra to write a fake application. It is easy to set up and offers a simple DSL for writing a web application. In combination with fixtures a web service can serve responses that are read from file in a convenient way.

require 'sinatra/base'

class FakeTodoService < Sinatra::Base
  get '/todos' do
    get_response 'todos.json'
  end

  private

  def get_response(filename, status_code = 200)
    content_type :json
    status status_code
    load_fixture(filename)
  end

  def load_fixture(filename)
    File.open(File.join('fixtures', filename), 'r').read
  end
end

The fixture for the /todos API endpoint looks like

{
  "todos": [ {
    "subject": "a todo from a fixture"
  } ]
}

This defines an application (FakeTodoService) with a single resource path /todos (as a GET request) that returns the content of the JSON file todos.json. Mimicking the paths and resource handling of the external API in this way enables us to define fixtures for all API endpoints. Instead of matching the public interface of a client, a fake service matches the API interface of the service. Defined in this way the client used in our application does not need any modifications or adjustments to communicate with our fake service. But this requires a good understanding of the external API either by having good documentation or well defined responses. With Sinatra we are also able to parse URI queries or HTTP headers accordingly, e.g. to support pagination or setting headers such as Content-Type to request different document types. To use the fake application in our tests we need to configure RSpec accordingly.

# in spec_helper.rb
RSpec.configure do |c|
  # [..]
  c.before(:each, :api => :external) do
    stub_request(:any, /www.example.com/).to_rack(FakeTodoService)
  end
end

This redirects all HTTP requests that would hit www.example.com to our fake application and only activates the service when tests are marked with the :api metadata tag. By marking tests with metadata we ensure that only those tests are run against the fake service. They are also useful to group tests with similar requirements.

Our test example then looks as follows:

require 'spec_helper'
describe 'ToDo API request', :api => :external do
  it 'returns a list of todos' do
    client = ApiClient.new(:host => 'www.example.com')

    response = JSON.parse(client.fetch_todos)

    expect(response).to be_an_instance_of(Hash)
    expect(response['todos']).to be_an_instance_of(Array)
    expect(response['todos'].size).to > 0
  end
end

This returns the JSON response from the fake application when calling the /todos path. As you can see this version matches very closely our initial test.

There are a couple of things to be aware of when introducing a fake service

• it increases the maintainance overhead
• the fake service can become quite complex in itself, a problem similar to mock objects
• extending the scope / usage of the client might require unobvious adjustments in the fake service
• when the external API changes, the expected behavior of the external API and the actual behavior of the fake service might differ, resulting in false positives
• when it's easier and cleaner to stub a request in the tests WebMock is often the better choice, especially if the API request returns different responses for different query arguments
• when using JRuby it might slow down the initial start up time

The main advantage of a fake service is that it can get introduced at any time and the existing test suite does not require all too many modifications. Once in place refactorings are easier to do and can be later substituted with stubbed requests via WebMock. It can be mixed easily with the other approaches described above, for example while the majority of API tests use the fake service, some requests can still be stubbed with WebMock or the other way around. It is a useful tool for certian situations.

Conclusion

Writing applications that use external APIs is a common use case when developing services. Depending on the context, complexity and control in regard to the service and the client we have different options of how to test the API.

Developing an internal client for an external API offers the chance to write it in a way where we can easily mix WebMock, VCR or a fake service. On the other hand if we use a 3rd party client to communicate with an external service it is best to encapsulate the functionality and only expose what is really needed. Testing external APIs also becomes easier when the used client is a popular library that is already extensively tested. In some cases we have specific mock libraries available to us that have the exact same API and mimick the functionality without relying on HTTP communication directly. A fake service is useful when it is easy to serve defined responses as for RESTful APIs or it's easier to introduce than to adjust the existing test suite. Alternatively VCR can be used if the logic of the external API is more complex and if it's ok to send HTTP requests every now and then.