Note: This article was originally published on the official Artillery blog.
The popularity of GraphQL for building and querying APIs has been on the rise for years with no sign of slowing down. Developers can now build robust and flexible APIs that give consumers more choices when accessing their data. This flexibility, in turn, can help with application performance compared to traditional RESTful APIs. By providing API consumers the flexibility to fetch the data it needs, it avoids the overhead of processing unnecessary information.
Despite all the advantages GraphQL provides to developers, it's also easy to introduce potential performance issues when constructing APIs:
- Its flexibility can allow clients to craft queries that are inefficient when fetching server-side data, especially when using nested queries that need to get data from multiple database tables or sources. This problem is more prevalent when using ORMs with GraphQL, since it can mask any inefficiencies in your database calls.
- Since GraphQL allows consumers to build different queries as they need, it's difficult to implement caching effectively. Since each request made to the API is potentially different, developers can't use standard caching techniques. Some libraries provide built-in caching, but it may not work in all scenarios.
Most developers building GraphQL APIs eventually run into one or both of these issues. While these performance problems are simple to deal with, they're also easily overlooked. It's also not uncommon to find yourself with a sluggish API after deploying to production when dealing with real data. That's why load testing your GraphQL APIs before and after deploying is more important than ever.
With Artillery, you can create test scripts that can test common workflows for GraphQL to smoke out any potential performance issues that can creep up when you least expect. Running regular load tests on your APIs both during development and against your production servers will keep your APIs performant and resilient for your customers. This article shows you how to use Artillery to keep your GraphQL services in check.
GraphQL API example
For this article, let's assume we have a GraphQL API with a data store, some queries to fetch this data, and mutations to manage data manipulation based on this diagram:
The data store is a database consisting of two tables to store user information and messages associated to that user that performs standard CRUD operations. There's a single query to fetch a specific user by their data store ID. The GraphQL API also has some mutations to create, update and delete a user, and create messages associated to a single user.
Let's imagine that most consumers of the GraphQL API perform two primary flows after creating and logging in as the new user. The first flow will create new messages through the API and later fetch the user's account information along with their messages. The other flow handles updating a user. We want to make sure these flows remain performant under load for whoever accesses the API. Let's create a few load tests using Artillery.
Creating our first GraphQL load test
Most GraphQL APIs handle requests via standard HTTP GET and POST methods, meaning we can easily use Artillery's built-in HTTP engine to make requests for load testing. Both methods work similar, with some differences related to how HTTP works. For instance, GET requests are easier to cache while POST responses aren't cacheable unless they include specific headers. For simplicity in our load tests, we'll use the POST method to make our GraphQL API requests.
When served over HTTP, a GraphQL POST request accepts a JSON-encoded body with a query field containing the GraphQL query to execute on the server. The body can also contain an optional variables field to set and use variables in the specified GraphQL query. We'll use both of these fields for our load test.
For our first test, we'll go through one of the flows described above:
- First, we'll create a new user record in through the GraphQL API and capture their email address from the GraphQL response. To ensure we don't run into uniqueness constraints when creating users during the test, we'll generate randomized usernames and email address using Artillery's
$randomString()method. - Next, we'll log in as that user to test the login process and capture their ID from the data store.
- With the user's ID in hand, we'll then use the
loopconstruct to create 100 new messages associated to the user. - After creating those messages, we'll make a request to return the user data and their messages to verify how the API performs when fetching associated data between different database tables.
- Finally, to clean up the database, we'll delete the user at the end of the flow, which will also delete the user's associated messages.
Let's see how this test script looks for executing this flow:
config:
target: "https://graphql.test-app.dev"
phases:
- duration: 600
arrivalRate: 50
scenarios:
- name: "Create and fetch messages flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"
- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"
- loop:
- post:
url: "/"
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: "{{ userId }}"
body: "Message Body {{ $loopCount }}"
count: 100
- post:
url: "/"
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: "{{ userId }}"
- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"For this example, we have our GraphQL API set up at https://graphql.test-app.dev. The Artillery test will run for ten minutes, generating 50 virtual users per second. In our scenario, we have our defined flow with a few POST requests made to the root URL (/) where the GraphQL API handles our requests.
Each step in our flow section works in a similar way. We'll have a JSON request to the GraphQL server with the query and variables field (where necessary). The query field contains the GraphQL query or mutation that's sent to the server for processing. To make the body more readable in our test script, we'll use a pipe (|) when sending the data.
Each query and mutation in our script is set up to handle variables, since we'll use them for keeping track of the dynamic data generated throughout the test phase. That's where the variables field comes into play. This field can set up the variables we'll use through a standard JSON object. The GraphQL server will handle setting up the values where specified in a query or mutation.
For instance, in the step where we create a user, the createUser mutation expects an input argument, which is an object containing the username, email and password fields. The variables field will construct the input object using fields required in the mutation argument and pass it to the query. This allows us to dynamically generate unique usernames and emails for each virtual user.
Assuming we save the test script in a file called graphql.yml, we can run it with Artillery using artillery run graphql.yml. After Artillery completes all the requests, you'll see how the GraphQL API performs under this load:
All VUs finished. Total time: 10 minutes, 39 seconds
--------------------------------
Summary report @ 15:31:12(+0900)
--------------------------------
vusers.created_by_name.Create and fetch messages flow: ...... 30000
vusers.created.total: ....................................... 30000
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 947.6
max: ...................................................... 20767.2
median: ................................................... 1465.6
p95: ...................................................... 20136.3
p99: ...................................................... 20136.3
http.request_rate: .......................................... 29/sec
http.requests: .............................................. 3120000
http.codes.200: ............................................. 3120000
http.responses: ............................................. 3120000
http.response_time:
min: ...................................................... 173
max: ...................................................... 4870
median: ................................................... 186.8
p95: ...................................................... 1312
p99: ...................................................... 3899
This example shows how straightforward it is to create a fully-fledged load test for GraphQL services. Artillery's HTTP engine has everything you need out of the box to check the most important actions of your GraphQL API. It's a simple yet powerful way to ensure your GraphQL services are resilient to any spikes in traffic and remain performant throughout. Although the performance of our GraphQL service aren't close to what we'd like, it gives us an excellent starting point for improvement.
Expanding the GraphQL load test
To expand our load test, let's add another scenario to further validate the GraphQL API's performance. For this flow, we want to validate that the service properly handles password updates. For security purposes, the GraphQL service uses bcrypt hashing to ensure we don't store plain-text passwords in the database. Depending on how a service uses bcrypt, the hashing function can slow down password-related functionality, so it's a good candidate to check for performance purposes.
The second scenario for our test goes through the following steps:
- We'll create a new user record and log in as that user, the same way as we did in the first scenario.
- Next, we'll log in as that user to test the login process and capture their ID from the data store.
- With the user's ID in hand, we'll call the mutation for updating a user's database record to update their password. This step helps us verify that the request works as expected along with testing its performance.
- Just like before, we'll delete the user at the end of the flow to keep the database clean of any test records.
The full test script with both flows is as follows:
config:
target: "https://graphql.test-app.dev"
phases:
- duration: 600
arrivalRate: 50
scenarios:
- name: "Create and fetch messages flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"
- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"
- loop:
- post:
url: "/"
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: "{{ userId }}"
body: "Message Body {{ $loopCount }}"
count: 100
- post:
url: "/"
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: "{{ userId }}"
- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"
- name: "Update password flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"
- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"
- post:
url: "/"
json:
query: |
mutation UpdateUserMutation($updateUserId: ID!, $input: UpdateUserInput) {
updateUser(id: $updateUserId, input: $input) {
id
}
}
variables:
updateUserId: "{{ userId }}"
password: "my-new-password"
- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"Running the test script with Artillery as we did previously (artillery run graphql.yml) shows the generated VUs going through both scenarios:
All VUs finished. Total time: 11 minutes, 20 seconds
--------------------------------
Summary report @ 15:43:19(+0900)
--------------------------------
vusers.created_by_name.Update password flow: ................ 18469
vusers.created.total: ....................................... 30000
vusers.created_by_name.Create and fetch messages flow: ...... 11531
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 899.1
max: ...................................................... 30451.4
median: ................................................... 1300.1
p95: ...................................................... 29445.4
p99: ...................................................... 29445.4
http.request_rate: .......................................... 28/sec
http.requests: .............................................. 1966900
http.codes.200: ............................................. 1966900
http.responses: ............................................. 1966900
http.response_time:
min: ...................................................... 172
max: ...................................................... 5975
median: ................................................... 186.8
p95: ...................................................... 1620
p99: ...................................................... 4867
As seen in our less-than-ideal performance results, GraphQL helps developers build flexible APIs with ease, but that flexibility comes at a cost if you're not careful. It's really easy to accidentally introduce N+1 queries or implement ineffective caching that bring your GraphQL API to a grinding halt with just a handful of consumers. This example test script gives you an idea on how to use Artillery to run robust load tests against your GraphQL services to find any bottlenecks or other issues during periods of high traffic.
The example GraphQL application and Artillery test script used for this article is available in the Artillery Examples repository on on GitHub.
