APIs are everywhere these days. Even if you don't realize it, a lot of the software you use daily connects to APIs. The software world is slowly becoming API-centric, powering everything from small side projects to large enterprise applications. More services than ever offer APIs to allow anyone access to their data instead of having to rely on their interfaces. They indeed are everywhere.
If you're doing any software development these days, chances are that you have developed or maintained an API yourself. It's especially prevalent in modern web development, where more organizations gravitate away from single-repo monolithic applications and towards separate backends and frontends.
The most common web APIs created these days are REST APIs. Lots of web frameworks have support for REST APIs baked in, like Ruby on Rails and Express. Frameworks that don't include direct support have libraries that can easily do the heavy lifting for you, like Django REST framework.
Given the prevalence and importance of APIs in today's software development world, developers and testers alike need to keep their APIs in tip-top shape. The need to test APIs is higher than ever.
Typical ways to test an API
There are multiple ways to test an API, depending on your situation. Some people focus on white-box testing as their primary form of verification. However, this relies on having access to the internals of the API.
Black-box testing seems to be a more conventional approach. It doesn't require access to the source code and allows testers to do their work without needing to dig into the guts of the application.
Often, REST APIs are a part of UI testing. While these tests verify that the API works well, there's a high risk involved. These tests have many moving parts during the test process, which can lead to unstable or flaky tests. If something fails, you won't know whether the issue lies in the API, the application interface, or something else.
Ideally, you'd want to test an API in isolation. Not involving the application interface or an unrelated component reduces any common issues that tend to crop up. All you need is to make a request with specific data, and verify that the response is what you expect.
To accomplish this, you can choose one of the many tools that exist to create tests for APIs. Postman is one of the most common testing applications for APIs, along with its command-line companion Newman. Another popular tool that many testers refer to is SoapUI. These are just a few tools out of dozens out there.
End-to-end testing for APIs using APId
Recently, I stumbled upon a new tool called APId for testing APIs. I took it for a test run and enjoyed using it, so I wanted to share my experiences here.
In a nutshell, APId is a new tool that allows you to write end-to-end tests without tying it to any specific environment. It's a small binary that reads and executes your tests from a YAML file. Tests are grouped into transactions, each containing one or more steps. These steps are used to make requests to an API endpoint and optionally validate its response.
Since the tool was built specifically with API testing in mind, it's pretty simple to use for validating most of what REST APIs do. When you make a request, you can send data through the request body or submit additional headers. Then you can validate the status code of the response and verify the response body. It doesn't do much more than that. In most cases, it's enough for testing the API.
There are a few other neat things about APId that are useful:
- To use APId, all you need is to download a binary and execute it. It's not tied to a specific programming environment, nor do you need to install dependencies to have it work.
- Since there are no external dependencies required, it's straightforward to integrate the tool into any project or continuous integration system.
- The format of the tests are in YAML. Some people don't like this format, but in my opinion, YAML is very readable, and many developers are familiar with the format. If someone isn't too familiar with YAML, the format is comfortable for anyone to pick up.
Note that APId is just another testing tool to help with writing end-to-end tests. It doesn't replace any existing tool. If you're using an existing tool like Postman or SoapUI, there's no reason to switch. This article is for introducing a new tool as a potential option for a new or future project.
APId by example
To demonstrate how APId works in standard REST APIs, I created a new application called Airport Gap.
Airport Gap is a RESTful API that provides access to a database of airports, calculate distances between airports, and allows you to save your favorite airports. Read the API documentation for more information. The primary reason is to help others use it as a base to improve their API automation testing skills. It's an excellent candidate to show what APId can do.
Installing APId is as simple as it gets. Go the APId Releases on GitHub and download the appropriate binary file for your operating system and architecture. That's all you need. You can execute the binary from the command line, as long as it has the proper execute permissions as specified in the documentation. To make it easier to run, place the APId binary in any directory in your PATH so that you can execute the binary in any other directory.
Let's create our first test using APId. To begin, we need to create the YAML file that APId uses to execute the tests. By default, the APId binary expects a file called apid.yaml in the current directory where you're running the tests. Create an empty file named apid.yaml in any directory, which we'll use for all of our tests.
A simple test to write is to make a request to an endpoint and check its response. Using Airport Gap, let's use the endpoint to search for a single airport and verify its response. This process is represented by writing the following test in our currently-empty apid.yaml file:
version: 1
variables:
api_url: "https://airportgap.dev-tester.com/api"
transactions:
- id: "GET /airports/:id"
steps:
- id: "Get information about a single airport"
request:
method: GET
endpoint: "{{ var.api_url }}/airports/KIX"
expect:
code: 200
body:
type: json
content: |
{
"data": {
"attributes": {
"altitude": 26,
"city": "Osaka",
"country": "Japan",
"iata": "KIX",
"icao": "RJBB",
"latitude": "34.427299",
"longitude": "135.244003",
"name": "Kansai International Airport",
"timezone": "Asia/Tokyo"
},
"id": "KIX",
"type": "airport"
}
}
Let's break this file down by section:
version: This line tells APId which format to expect when reading the rest of the file. As far as I can tell, only version 1 is available, so there's nothing else to configure here.variables: APId allows you to configure variables to use throughout your tests. Variables are scoped depending on where they're defined. In this example, we scope theapi_urlvariable globally, meaning that all of the transactions and steps for the tests can access to it. Variables are accessed through special templates, as you'll see later on. Read more about variables in the APId documentation.transactions: Transactions are the primary way to group tests in APId. Your entire test suite consists of multiple transactions, each containing the steps needed to run the test. Think of transactions as separate scenarios when writing your tests.id: The ID is your transaction identifier. You can set the ID to any string, but we'll see later how to use the ID in different scenarios.steps: As mentioned, each transaction contains one or more steps. All steps make a request to an endpoint. You can also specify locally-scoped variables and make assertions as necessary.request: Here, you define what request you want to make to the API. You're required to specify themethodandendpointfor the request. But there are optional parameters like specifying the body and headers. Also, note the use of theapi_urlvariable as a template. This template replaces that section with our API URL.expect: Your assertions for the request you specified in the step. In this example, we're checking that the response's status code is 200 (OK) and verifying the body that the response returned. As we'll also see later, you don't always need to have assertions for steps.
To run this test, all you need to do is run apid check in the directory where the apid.yaml file is from the command line. If the binary is set up correctly in your PATH and the formatting of the test is correct, you'll see the results of the test pop up:
$ apid check
GET /airports/:id:
OK Get information about a single airportAnd with that, you've completed your first APId test! That's all there is to it. This simple test should give you a better idea of how you can use this tool for any API.
If you're receiving any errors when running this test, double-check the apid.yaml file for typos and proper formatting. If everything looks fine, leave a comment below or send me a message so I can help you out.
This test is straightforward - ping an API endpoint, verify its response. But what if you want to send more data to an endpoint? For instance, Airport Gap has an endpoint that requires you to send two airport IDs, and the response returns a calculation of the distance between airports.
APId makes this action simple, too. Let's write another test, this time sending specific data through the request body. Open the apid.yaml file, and at the end of the previous test, add a new set of steps. Make sure you have the proper formatting for the YAML file.
- id: "POST /airports/distance"
steps:
- id: "Calculate the distance between two airports"
request:
method: POST
endpoint: "{{ var.api_url }}/airports/distance"
headers:
"Content-Type": "application/json"
body: |
{
"from": "KIX",
"to": "SFO"
}
expect:
code: 200
body:
type: json
content: |
{
"data": {
"id": "KIX-SFO",
"type": "airport_distance",
"attributes": {
"from_airport": {
"altitude": 26,
"city": "Osaka",
"country": "Japan",
"iata": "KIX",
"icao": "RJBB",
"id": 3158,
"latitude": "34.427299",
"longitude": "135.244003",
"name": "Kansai International Airport",
"timezone": "Asia/Tokyo"
},
"to_airport": {
"altitude": 13,
"city": "San Francisco",
"country": "United States",
"iata": "SFO",
"icao": "KSFO",
"id": 2672,
"latitude": "37.618999",
"longitude": "-122.375",
"name": "San Francisco International Airport",
"timezone": "America/Los_Angeles"
},
"kilometers": 8692.066508240026,
"miles": 5397.239853492001,
"nautical_miles": 4690.070954910584
}
}
}The test is almost the same as our first test, with a few new things to point out in the request section. Since the route for the endpoint is expecting a POST request, the method value has to change to reflect that. The other main changes relate to the data we want to send over to the API.
The first thing is that we need to tell Airport Gap that the request body we're sending it is in JSON. To do this, we need to send the Content-Type: application/json header, using the headers field in the request section.
Once the header is in place, we can send the data along with the request. Sending the data is accomplished using the body field in the request section. APId expects a string in this field, so we send it a JSON string with the required parameters that the API needs.
With the test written, you can once again run apid check from the command line. The results of the new test appear, showing two test executions:
$ apid check
GET /airports/:id:
OK Get information about a single airport
POST /airports/distance:
OK Calculate the distance between two airportsValidating API responses
The tests we've written so far are basic, but you might notice that with only two tests, the apid.yaml file is beginning to get a bit long. APIs usually have a lot more endpoints to test, and some of these endpoints return lots of data.
The main reason our tests are currently long is that we're validating the entirety of the API response body. For some scenarios, like our example of getting information about a single airport, checking the entire body is acceptable. It's a small request, and it contains the basic structure of most responses in the API.
However, APIs often have endpoints that return a list of data. These responses are usually lengthy and contain the same structure. It doesn't make sense to test the entire body of these kinds of responses. For instance, Airport Gap has an endpoint that returns a paginated list of 30 airports per request. If you added the response to the test, it would add hundreds of lines and make the tests difficult to maintain.
Thankfully, APId doesn't require you to verify the entire body of the API response you're testing. You can optionally tell APId to not be as strict with validating the body, and only check that specific fields exist. There's more detailed information on their documentation, but let's see it in action.
We're going to add a test that calls an endpoint which returns a paginated list of airports. In this example, there's no need to check the actual data. All we want to validate is that the API returns specific fields. In the apid.yaml file, add the following test:
- id: "GET /airports"
steps:
- id: "Get all airports, including pagination"
request:
method: GET
endpoint: "{{ var.api_url }}/airports"
expect:
code: 200
body:
type: json
exact: false
content: |
{
"data": [
{
"id": "",
"type": "",
"attributes": {}
}
],
"links": {
"self": "",
"first": "",
"last": "",
"next": "",
"prev": ""
}
}The assertion in this test might look strange, but it's a valid test. In this test, we tell APId that we don't want to validate the exact content of the body by setting exact: false in the expect section. For the content, we only have the basic structure of the API response, noting the fields and their data types. We don't have to specify any values for the fields since we're just checking the structure.
Running apid check in the command line tells you that this is a valid test:
GET /airports/:id:
OK Get information about a single airport
POST /airports/distance:
OK Calculate the distance between two airports
GET /airports:
OK Get all airports, including paginationYou should try to verify exact responses as much as possible for more confidence that the API is working well. But these types of tests can be useful when you only want to check the structure of a JSON response.
Testing endpoints requiring authentication
So far, the tests we have focus on publicly accessible endpoints. For these requests, all you need is the endpoint and any required parameters. But many REST APIs have endpoints that require some form of authentication to return a successful response. There are different authentication schemes for APIs, like JSON Web Tokens and OAuth.
Airport Gap uses a simple form of token authentication for secure endpoints. Each Airport Gap account has an access token associated with it upon account creation. When making a request to the API, the request header includes the token, which the API verifies to grant access to the requested endpoint. It's one of the simplest methods for authenticating APIs.
Testing APIs requiring authentication varies by application. In the case of Airport Gap, all you need is a token to send along with your request. There are a few ways of going about this.
The fastest way to test secure endpoints is to include the token directly in your test. You can set the token as a variable, as mentioned before, or hard-code it into the test itself. This approach gets you up and running quickly, but it makes your test brittle. For instance, if you regenerate your access token for any reason, your test will break.
A better approach is to have a step that fetches your token for you inside the test and use it in subsequent steps. APId provides a straightforward way of doing this by storing responses as variables.
Let's create another transaction in the apid.yaml file. Airport Gap has an endpoint that receives an account's email and password and returns that account's token. We can begin with that step, storing the returned token as a variable to use later.
- id: "GET /favorites"
steps:
- id: "authentication"
request:
method: POST
endpoint: "{{ var.api_url }}/tokens"
headers:
"Content-Type": "application/json"
body: |
{
"email": "{{ env.AIRPORT_GAP_EMAIL }}",
"password": "{{ env.AIRPORT_GAP_PASSWORD }}"
}
export:
token: response.body.tokenYou might notice that we're using different variables in the body for the request. These variables are environment variables. The reason for using environment variables here is to avoid setting your sensitive account information inside the test itself and set it for your specific environment. This method is highly useful when running in other places like a continuous integration system.
Locally, you can set these variables by appending the AIRPORT_GAP_EMAIL and AIRPORT_GAP_PASSWORD variables to the apid check command in your command line (for example, AIRPORT_GAP_EMAIL=<your email> AIRPORT_GAP_PASSWORD=<your password> apid check). This article on Trello's website goes into more detail about environment variables.
One new section introduced in this test is the export section. APId uses this to allow you to grab part of a response, store it in a variable, and use it in other steps. Storing these variables makes it useful in situations where you want to fetch a token and use later like we're doing here. It's important to note that the value given to the id of the step is used to access the exported variable, as you'll see soon.
Another thing to note in this step is that you're not validating the response of the API request, as done in previous tests. While you can add an assertion to verify that the response body is correct, there's not a need to handle that here. All you want is to get a token and store it for future tests.
Once you have your token, you can begin using it to call secure endpoints. In the same transaction, let's create a new step in the same transaction that calls an endpoint that requires authentication:
- id: "Get all favorites for the account"
request:
method: GET
endpoint: "{{ var.api_url }}/favorites"
headers:
"Authorization": "Bearer token={{ authentication.token }}"
expect:
code: 200
body:
type: json
exact: false
content: |
{
"data": [
{
"id": "",
"type": "",
"attributes": {}
}
],
"links": {
"self": "",
"first": "",
"last": "",
"next": "",
"prev": ""
}
}This test is almost identical to the test that verifies the endpoint that returns the paginated list of airports. There is one significant difference in this request, however. In the request, we're sending a header to authenticate our request. In the previous step, we stored the token returned from the request into a variable. In this step, we use the token through a variable in the Authorization header.
The variable is accessed using the name of the step and the name of the key assigned when exporting. In this scenario, the variable's name is authentication.token. The token is used as part of the Authorization header, as specified by the authentication scheme. This header is all that's needed for Airport Gap to authenticate an API user.
Here's the full transaction with the step to authenticate the user, and the step to request the protected endpoint:
- id: "GET /favorites"
steps:
- id: "authentication"
request:
method: POST
endpoint: "{{ var.api_url }}/tokens"
headers:
"Content-Type": "application/json"
body: |
{
"email": "{{ env.AIRPORT_GAP_EMAIL }}",
"password": "{{ env.AIRPORT_GAP_PASSWORD }}"
}
export:
token: response.body.token
- id: "Get all favorites for the account"
request:
method: GET
endpoint: "{{ var.api_url }}/favorites"
headers:
"Authorization": "Bearer token={{ authentication.token }}"
expect:
code: 200
body:
type: json
exact: false
content: |
{
"data": [
{
"id": "",
"type": "",
"attributes": {}
}
],
"links": {
"self": "",
"first": "",
"last": "",
"next": "",
"prev": ""
}
}Given a valid Airport Gap account email and password, you can run the test and verify that the authentication for the endpoint is valid:
$ apid check
GET /airports/:id:
OK Get information about a single airport
POST /airports/distance:
OK Calculate the distance between two airports
GET /airports:
OK Get all airports, including pagination
GET /favorites:
OK authentication
OK Get all favorites for the accountEnd-to-end flow with APId
The four transactions demonstrated above are enough to run through most scenarios needed to test an API. These examples test different parts in isolation, but you can also use APId for more extended end-to-end tests. Let's write a full end-to-end flow for Airport Gap to put together everything learned so far.
For this end-to-end test, we'll test the following scenarios for Airport Gap using the API:
- Authenticate a user and receive the account's token.
- Create a new favorite resource for the account.
- Fetch the newly created resource.
- Update the resource.
- Fetch the resource to verify the update.
- Delete the resource.
- Verify that the resource is no longer accessible.
Here's the entire test covering this flow:
- id: "End-to-end test flow"
steps:
- id: "authentication"
request:
method: POST
endpoint: "{{ var.api_url }}/tokens"
headers:
"Content-Type": "application/json"
body: |
{
"email": "{{ env.AIRPORT_GAP_EMAIL }}",
"password": "{{ env.AIRPORT_GAP_PASSWORD }}"
}
export:
token: response.body.token
- id: "create_favorite"
request:
method: POST
endpoint: "{{ var.api_url }}/favorites"
headers:
"Content-Type": "application/json"
"Authorization": "Bearer token={{ authentication.token }}"
body: |
{
"airport_id": "KIX",
"note": "My local airport"
}
expect:
code: 201
export:
favorite_id: response.body.data.id
- id: "Fetch the newly created favorite"
request:
method: GET
endpoint: "{{ var.api_url }}/favorites/{{ create_favorite.favorite_id }}"
headers:
"Authorization": "Bearer token={{ authentication.token }}"
expect:
code: 200
body:
type: json
content: |
{
"data": {
"attributes": {
"airport": {
"altitude": 26,
"city": "Osaka",
"country": "Japan",
"iata": "KIX",
"icao": "RJBB",
"id": 3158,
"latitude": "34.427299",
"longitude": "135.244003",
"name": "Kansai International Airport",
"timezone": "Asia/Tokyo"
},
"note": "My local airport"
},
"id": "{{ create_favorite.favorite_id }}",
"type": "favorite"
}
}
- id: "Update favorite"
request:
method: PATCH
endpoint: "{{ var.api_url }}/favorites/{{ create_favorite.favorite_id }}"
headers:
"Content-Type": "application/json"
"Authorization": "Bearer token={{ authentication.token }}"
body: |
{
"note": "My local airport, goes everywhere in the world!"
}
expect:
code: 200
- id: "Verify updated favorite"
request:
method: GET
endpoint: "{{ var.api_url }}/favorites/{{ create_favorite.favorite_id }}"
headers:
"Authorization": "Bearer token={{ authentication.token }}"
expect:
body:
type: json
content: |
{
"data": {
"attributes": {
"airport": {
"altitude": 26,
"city": "Osaka",
"country": "Japan",
"iata": "KIX",
"icao": "RJBB",
"id": 3158,
"latitude": "34.427299",
"longitude": "135.244003",
"name": "Kansai International Airport",
"timezone": "Asia/Tokyo"
},
"note": "My local airport, goes everywhere in the world!"
},
"id": "{{ create_favorite.favorite_id }}",
"type": "favorite"
}
}
- id: "Delete favorite"
request:
method: DELETE
endpoint: "{{ var.api_url }}/favorites/{{ create_favorite.favorite_id }}"
headers:
"Authorization": "Bearer token={{ authentication.token }}"
expect:
code: 204
- id: "Ensure favorite is deleted"
request:
method: GET
endpoint: "{{ var.api_url }}/favorites/{{ create_favorite.favorite_id }}"
headers:
"Authorization": "Bearer token={{ authentication.token }}"
expect:
code: 404This kind of test shows how simple it is to run an entire flow through multiple endpoints in an API. These end-to-end tests are useful to ensure different pieces of the application work as expected.
The full apid.yaml file is available here.
APId shortcomings
While APId is very useful for creating API-specific tests, I did find plenty of shortcomings to the tool in its current state:
- I couldn't find a way to reuse common steps for different transactions. For instance, for transactions that need an authentication token, you'll need to write the same test step to fetch the token and store the variable.
- The YAML file can get challenging to manage the more tests you write. There's no easy way to organize tests besides grouping them into transactions. You can technically split tests into multiple YAML files and run them separately using the
--configflag when runningapid check. But this requires multiple calls to the binary, which is not convenient. - Assertions are very limited. You can only check the status code and the body of the response, nothing else. Currently, there's no way to check other essential portions of the response like the headers.
- Debugging is difficult. Admittedly, YAML is finicky when it comes to formatting. It was often the primary source of any problems I encountered. But other times, I found the error messages to be a bit obscure. I also had some trouble figuring out if my exported variables were correct since I couldn't find how to check currently set variables in a test.
- APId is a small project involving just one or two developers, based on social media comments left by one of the authors. While APId is useful, it's tough to bet part of your long-term testing strategy on a small side project. We don't know whether this project will receive maintenance and new features in the future, or if it will slowly fade away as many side projects do.
As of the time of posting this article, APId is not open source, so you're at the whims of the developers for any bug fixes and updates to the tool. The project's creator mentioned on a Reddit thread that they might make the binary open-source, but there are no guarantees.The project is now open-source and available on GitHub.
Summary
End-to-end testing with APIs can be tricky, but if you have an API that's on the smaller side of things, APId is a helpful tool to consider for testing purposes.
Given that it's a new tool, it does have some gotchas and shortcomings to manage. Some of those issues are dealbreakers for some teams and can make it difficult to recommend using this for larger projects.
Even with its shortcomings, it's easy to set up and contains most of what you need to test a RESTful API thoroughly. Chances are you won't need to check more than what it currently provides. Hopefully, the project continues to mature and become a reliable alternative for all sorts of APIs.
If you have any trouble running the tests, or if you have any thoughts or questions about APId, please leave your comments below!
Disclaimer: I am in no way involved with APId or its creators. It's just a new tool I discovered and wanted to share with others.
