Capti

Capti is a lightweight end-to-end testing framework for testing REST APIs. You define your tests in YAML format as an HTTP request and expected response. Leverage the flexible features of Capti, such as matchers and variables, to write high-quality, customized test suites for your applications. Configure automation and scripting to build CI flows to test your endpoints or contract dependencies in staging or production.

  - test: Get recipe
    description: Should be able to get recipe information
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    expect:
      status: 2xx
      body:
        id: ${RECIPE_ID}
        name: Guacamole
        ingredients: $exists

Features

  • Define test suites to model the behavior of your users.
  • Write HTTP endpoint tests that make HTTP requests and assert expected responses.
  • Use various matchers in your tests to make pattern-based assertions.
  • Define and extract variables from responses to reuse repeated values and test authentication and stateful resource flows.
  • Provide setup and teardown scripts for your tests and test suites, enabling CI workflows.

Next Steps

Installation

There are currently a few different ways you can install Capti.

Global NPM Install

Installing globally is the easiest way to get started with Capti if you already have Node/NPM installed.

$ npm install -g capti

This will make Capti available for you to use in any project at the command line. To verify that your installation succeeded, you can run:

$ capti --version

Note: If your installation did not succeed, please report the issue so that any necessary fixes can be made.

Local NPM Install

Installing Capti locally in an NPM project is useful when you share your project with other developers and don't want them to have to globally install anything. It's also useful if you plan to use Capti for continuous integration in your project.

To install locally in an NPM project, cd into your project directory and run:

$ npm install --save-dev capti

Note: If your installation did not succeed, please report the issue so that any necessary fixes can be made.

This will save Capti as a development dependency (meaning that it won't bundle into your final build). To use Capti in your project, first create a new folder in your project directory tests/ (you can name it whatever you want). Then open your package.json and add the following script:

{
    "scripts": {
        "test:capti": "capti --path ./tests"
    }
}

The to run the tests your project, all you need to do is run:

$ npm run test:capti

Binary Executable

If you want to install Capti on the command line but you do not want to use NPM or don't have Node installed, you can download the binary executable for your platform/architecture and manually add it to your PATH.

To access the binary executable downloads, head over to the project's GitHub Releases and download the latest version.

Note: If you don't see your platform/architecture as a download option, and you think it should be available, feel free to put in an enhancement issue and we can look at possibly adding support for your architecture.

Instructions for adding the binary to your PATH environment variable varies system-to-system. Please expand the section below for your platform.

Unix (Linux / MacOS)
  1. Start by moving the downloaded binary to your usr/local/bin directory.
$ mv capti /usr/local/bin/capti

  1. Open your shell config file .bash-profile if you use bash, or .zshrc if you use zsh (default on MacOS) using your favorite text editor.

  2. Add the following line to the end of the file:

export PATH="/usr/local/bin/capti:$PATH"
  1. Restart your shell, and you should be good to go. You can verify by running:
$ capti --version
Windows

  1. Move the binary to a convenient location. I recommend C:/Program Files/Capti/capti.exe.

  2. Right-click on the Start button > System > About > Advanced system settings > Environment Variables

  3. Edit the PATH Variable:

  • In the Environment Variables window, under "System variables" (for all users) or "User variables" (for the current user only), find and select the PATH variable, then click Edit.
  • In the Edit Environment Variable window, click New and add the path to the folder that contains your binary. For example, C:\Program Files\MyBinary.
  • Click OK to close each window.
  1. Restart any open command prompts or applications.

Getting Started

Prerequisites

  • A REST API project with at least one HTTP endpoint to test
  • Capti installed in the project or globally on your machine. See the installation instructions.

Note: If you just want to see how Capti works and you have it installed globally, you can use a resource like JSON Placeholder and test fake HTTP endpoints.

Setting up

For the examples in this setup, a NodeJS project will be referenced, but Capti is not limited to use in NodeJS projects. You can use it with Django, Ruby on Rails, Laravel, or any other framework for writing backends.

The project in this example is a simple ts-node Express backend with one endpoint - /hello. This is the entire application:

// src/index.ts
import express from "express";

const app = express();

app.get("/hello", (req, res) => {
  res.send("Hello World!");
});

app.listen(3000, () => {
  console.log("Server is running on port 3000");
});
  1. Create a directory at the root of your project that will contain all your Capti tests. In this directory, create a new file hello.yaml. The directory structure will look something like this:
.
├── src/
│   └── index.ts
├── tests/
│   └── hello.yaml
├── .gitignore
├── package-lock.json
├── package.json
└── tsconfig.json
  1. Add Capti to the project with npm install --save-dev capti, and then add the following script to package.json:
{
    "scripts": {
        "test:capti": "capti --path ./tests"
    }
}

Note: If you are not using Node, you can skip this step and instead when you want to use Capti, just use the command capti --path ./tests instead of npm run test:capti.

Writing your first test

  1. Open hello.yaml in your favorite text editor. Start by adding the following fields:
suite: "Hello endpoint tests"
description: "Tests the various HTTP methods of the /hello endpoint."
  • suite is the title of this test suite. Every file consists of a single test suite which groups together related tests.
  • description is an optional field that describes the test suite in more detail.
  1. Next, add the tests array, which will consist of each of the tests we want to run.
suite: "Hello endpoint tests"
description: "Tests the various HTTP methods of the /hello endpoint."

tests:
  - test: "Get hello"

Note: If you're unfamiliar with YAML syntax, take a moment to skim over the YAML specification. While Capti does not use all of the advanced features of YAML, such as tags, knowing the basic syntax around arrays (sequences), objects (mappings), and primitive values can help you better understand what to expect when writing your tests.

  1. Let's write our first test. We need to add a request field first, which consists of the method, url, optional headers, and optional body of our HTTP request.
tests:
  - test: "Get hello"
    request:
      method: GET
      url: http://localhost:3000/hello
  1. To finish the first test, we need to add the expect field, which contains the HTTP response we expect to get back.
tests:
  - test: "Get hello"
    request:
      method: GET
      url: http://localhost:3000/hello
    expect:
      status: 200
      body:
        message: "Hello, world!"

We defined a status, which represents the HTTP Status that we expect our server to return, and we defined a body with one property message whose value should exactly equal "Hello, world!".

Note: Most people are used to seeing response bodies in JSON format - writing them in YAML might take some getting used to. For reference, the above YAML syntax for body is equivalent to { "message": "Hello, world!" } in JSON.

Running your first test suite

Let's start up our server and try this out. Start your server with npm start (or whatever start command you usually use). To run our first test, we can run npm run test:capti (or if you installed globally, run capti -p ./tests).

✗ [Hello endpoint tests] Get hello... [FAILED]
→ Body does not match.
  Mismatched types
    expected:
    {
      "message": "Hello, world!"
    }

    found: "Hello world!"

Uh oh - we have screwed up. From the messages, we can see we were expecting a JSON object, but got a string instead.

Here's the issue in our code:

// src/index.ts
app.get("/hello", (req, res) => {
  res.send("Hello World!");
});

This is why we write tests. Let's update it to the correct format.

// src/index.ts
app.get("/hello", (req, res) => {
  res.json({ message: "Hello World!" });
});

✗ [Hello endpoint tests] Get hello... [FAILED]                                                                                                                                    → Body does not match.
  Assertion failed at "Hello, world!" == "Hello World!"
  Mismatch at key "message":
    expected: "Hello, world!"
    found: "Hello World!"

Still not quite right, but as you can see - the messages from Capti give us all the info we need to fix our endpoint. Clearly we can see that we are missing a comma and we have uppercased 'W'. Let's update the server one more time.

== Hello endpoint tests =======

✓ Get hello

Passed: 1 | Failed: 0 | Errors: 0 ▐  Total: 1

== Results Summary =======

Total Tests: 1

Total Passed: 1
Total Failed: 0
Total Errors: 0

Now we have a passing test. The two summaries you see are one for the "Hello endpoint tests" test suite (there will be more once you add more test suites) which shows our test "Get hello" has passed. We also have the Results Summary, which shows the test results for all tests.

Conclusion

Hopefully, with this quick guide, you can see where to go from here. Start writing more tests - for each of your endpoints. Take some time to learn more about how to write good tests with Capti, including:

  • How to use matchers to match values based on certain conditions
  • How to write setup and teardown scripts and have Capti execute them for you
  • How to use variables to write DRY tests
  • How to extract variables from responses (such as JWT tokens) and carry them over to subsequent tests

Writing Tests

Example

  - test: Get recipe
    description: "Should be able to get recipe information"
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    expect:
      status: 2xx
      body:
        id: ${RECIPE_ID}
        name: Guacamole
        ingredients: $exists

Let's review each component of this example.

  • test - This is the name of the test as it will appear in your terminal. Keep this short.
  • description - This is an optional description that describes your test in more detail.

Note: As of version 0.1.0, description does not actually do anything and is akin to a comment. However, in future updates it will hopefully be integrated with any kind of test report output.

  • request - The HTTP request that you want Capti to make.
    • method - The HTTP method - one of "GET", "POST", "PATCH", "PUT", or "DELETE"
    • url - The URL to which the request should be made. In the example, variables are used to substitute in parts of the URL. See the section on variables for more information.
  • expect - The HTTP response that you expect to get back.
    • status - The HTTP Status Code you expect your endpoint to return.
    • body - The response body you expect to get back - in this case the body has three fields id, name, and ingredients. id is being matched to a variable, name must exactly match the word "Guacamole", and ingredients is using the $exists matcher to verify that the ingredients field is non-null. Please see the matcher section for more information on matchers.

Implicit Matching

Omitted fields in your expect definition are matched implicitly with the response. This means that you can focus on testing individual components of the response body by only defining a subset of the response. For example, if the response body returned from your server looks like this:

{
  "__v": 0,
  "_id": "65d204a107b727ac2667e82f",
  "displayName": "john-smith",
  "email": "testuser3@test.com",
  "id": "65d204a107b727ac2667e82f"
}

And all you care about is making sure the displayName field is correct, you can simply define your expect definition like so:

  expect:
    body:
      displayName: john-smith

This test will pass. You do not need to include the other fields to get a passing test. In fact, if your test specifies an empty expect definition, it will always pass unless the test throws an error.

Note: In some cases, you do want to ensure a field is absent from the response. For example, say you want to make sure the password field does not exist in the body. For this, you can use the $absent matcher. Review Matchers for more information on the $absent matcher and other matchers.

Matchers

You can specify exact values in the expect section of each test, or tests can also be configured with special matchers.

tests:
  - test: "get hello"
    description: "hello endpoint responds with some type of greeting"
    request:
      method: GET
      url: "http://localhost:3000/hello"
    expect:
      status: 2xx # match any 200 level status code
      headers:
        Content-Type: application/json # exact match
      body:
        message: $regex /[Hh]ello/ # match based on regex
        currentTime: $exists # match anything as long as it is present

For more information on matchers, review the matchers chapter.

Variables

Static variables can be defined for each test suite with the variables option, and then used in test definitions like ${this}. When the test is run, each variable will be expanded in place.

suite: "User Signup"
description: "Confirms that protected routes cannot be accessed until the user signs up."
variables:
  BASE_URL: "http://localhost:3000"
  USER_EMAIL: "testuser2@test.com"
  USER_PASSWORD: "F7%df12UU9lk"

tests:
  - test: "Sign in"
    description: "The user should be able to sign in with email and password"
    request:
      method: POST
      url: "${BASE_URL}/auth/signin"
      headers:
        Content-Type: application/json
      body:
        email: ${USER_EMAIL}
        password: ${USER_PASSWORD}
    expect:
      status: 2xx
      body:
        id: $exists
        email: ${USER_EMAIL}
        password: $absent

Environment variables can be referenced in the same way. If a variable is set in both your local environment and in the variables section, the value specified in the variables section will take precedence.

Variables can also be "extracted" from responses and used in subsequent tests by defining an extract section for a test.

tests:
  - test: "sign in"
    description: "Sign in as the test user"
    request:
      method: POST
      url: "${BASE_URL}/auth/signup"
      headers:
        Content-Type: application/json
      body:
        email: ${USER_EMAIL} # email and password defined as variables in the test suite for easy reuse throughout tests
        password: ${USER_PASSWORD}
    expect:
      status: 2xx
    extract:
      headers:
        Authorization: Bearer ${JWT_TOKEN} # extracts the token variable from the response
      body:
        userId: ${USER_ID} # extracts the user id generated by the database

  - test: "access protected route"
    description: "After signing in, the user can get their profile data"
    request:
      method: GET
      url: "${BASE_URL}/profile/${USER_ID}" # extracted variables can be used just like any other
      headers:
        Authorization: Bearer ${JWT_TOKEN} # great for auth flows
    expect:
      status: 2xx
      body:
        firstName: $exists
        lastName: $exists
        imageUrl: $regex /.*\.png$/

Note: Each suite manages its own cookies internally, enabling session authentication to work automatically within a suite. There is no need to extract cookies to carry over between requests.

For more information on variables, please review the variables chapter.

Configuration

There are many ways to configure Capti for your testing needs. You can configure various options for:

For individual tests, you can configure how your test is run and evaluated. For test suites, you can set up variables, determine whether tests should run sequentially or parallel, and establish any scripts to run before and after each test. And lastly, you can establish a global config file that runs scripts before and after all test suites are run.

Config File

Capti config files enable you to set some settings that apply to all your test suites and infuence how test suites are processed. Many of the same configuration options are available on a per-suite basis as well. One useful component of the configuration is specifying scripts that should run before and after your tests - for example, starting your server.

Setup

To create a config file, simply include a file named capti-config.yaml in your tests folder. This will automatically be parsed as a configuration for Capti.

Custom Config

If you would instead prefer to name your config differently, or include the config in a location separate from your tests, you can specify the --config or -c argument when running Capti. For example, say you want to keep your config file in a separate directory in your project:

.
├── src/
│   └── index.ts
├── tests/
│   └── hello.yaml
├── config/
│   └── capti.yaml
└── .gitignore

You can configure your script to run

$ capti --path ./tests --config ./config/capti.yaml

Config Contents

Setup

You can define the key setup in your config file to describe a series of global setup scripts that run before and after all test suites.

Environment Variables

You can define the key env_file with a path to your project's .env file. This enables you to reference variables from your .env file in your tests as variables.

Setup Scripts

If you would like Capti to run commands or scripts before executing tests, whether for continuous integration workflows or just for convenience, you can specify scripts to run and optional wait_until parameter to determine when to continue with executing your tests or additional scripts.

Adding Scripts

Setup scripts should be listed in sequential order under before_all or after_all in your config file.

# tests/capti-config.yaml

setup:
  before_all:
    - script: npm start
      description: start server
      wait_until: output 'Server running on port 3000'

You can also additionally include before_each and after_each scripts in your individual test suites. Keep in mind that config-level scripts will all execute first.

# tests/hello.yaml
suite: /hello endpoint tests
description: tests the various HTTP methods of the /hello endpoint
setup:
  before_all:
    - script: echo 'starting hello suite'
  before_each:
    - script: ./scripts/reset-test-db.sh
      desription: reset test db
      wait_until: finished

Wait Until Options

There are a few different options to choose from when deciding how to wait for your scripts to finish. By default, if wait_until is not included, execution will immediately continue with your script running in the background. This is not always what you want - for example when starting a server, you need to give it time to fully spin up before you start testing its endpoints.

  • wait_until: finished - This executes the command/script/program and waits synchronously for it to finish before proceeding.
  • wait_until: 5 seconds - This executes the script and then waits for the specified number of seconds before continuing.
  • wait_until: port 3000 - This executes the script and waits for the specified port to open. If the port already has an open connection, the script will not execute.
  • wait_until: output 'Server listening on port 3000 - This executes the script and then waits for the specified console output from your server. This is useful in some cases where the port may be open but the server is still not quite ready to take requests.

Examples

Here is a simple cross-platform script to start a server and check that the port connection is open before proceeding.

setup:
  before_all:
    - description: start app server
      script: NODE_ENV=test && npm start
      wait_until: port 3000

Here is an example from a project that uses Docker Compose to spin up both a database and a server. This Unix script checks if Docker Compose is already running, and if not - starts it. If it is already started, the wait_until output is still detected because of the call to echo the same output text. Note - make sure you update the output text to match your server's log message when it becomes ready.

setup:
  before_all:
    - description: "Start db and server"
      wait_until: output "Listening on 3000"
      script: >
        if ! docker-compose ps | grep -q " Up "; then
            docker-compose up
        else
            echo "Listening on 3000"
        fi

Considerations

Running setup scripts is entirely optional and merely provides a convenience for development. It is not necessary and you may instead choose to start your server manually first and then run Capti.

Why doesn't Capti just integrate directly with the server? The goal of Capti is to provide the convenience of platform-agnostic test suites to run with your project, without directly coupling with your server (and behaving more like a user). If you want a framework that more tightly integrates with your server, you can look into a tool like supertest for NodeJS or MockMvc with Java/Spring.

Suite Configuration

You have a few options to work with when configuring your test suites. You can define setup scripts that execute before or after your tests, specify whether tests should run in parallel or sequentially, and specify static variables to be used throughout your test suite.

Setup Scripts

See setup scripts for more information on how to create setup scripts. These scripts execute command line programs or utilities before and after your tests, and can be useful if you want to do some specific configuration, like resetting a database, before testing.

Parallel Testing

In general, you should prefer sequential testing over parallel testing. Capti test suites are each meant to simulate individual users interacting with your application, and a user would not typically be visiting multiple endpoints concurrently.

Note: When you have multiple test suites defined, the test suites will always run concurrently. Each suite should be designed to simulate a user, and multiple users should be able to interact with your API concurrently and deterministically. Your suites should never rely on the state of other test suites. The individual tests in a suite should, in the majority of cases, run sequentially.

There are some cases in which the tests within a suite should run in parallel. One example would be if you are grouping together multiple tests of several different public and stateless endpoints. In these cases, you can specify in your test suites that all tests should run in parallel with parallel: true.

suite: "Published recipes"
description: "This suite tests multiple sequential accesses to the public endpoints returning published recipe information."
parallel: true

Note: You cannot extract variables when specifying parallel: true. Referencing an extracted variable in a later request is not possible when all requests run concurrently.

Variables

You can define static variables to be used throughout the tests in your suites with the variables: mapping. These variables will expand to the specified value, sequence, or mapping when they are used. You can learn more in the variables chapter.

suite: "Create Recipe"
description: "This suite involves creating a new recipe and fetching its information."
variables:
  BASE_URL: http://localhost:3000
  USER_EMAIL: recipe1@tests.com
  USER_PASSWORD: abc123!

Test Configuration

In addition to defining a request and an expect mapping for each test, you can also define the following settings.

By setting print_response: true on your tests, the complete response status, headers, and body will be printed to the console when your test is run. This can be useful for debugging a failing test.

== Response: (Sign up) =======

  Status: 200

  Headers:
    ▹ "x-powered-by": "Express"
    ▹ "content-type": "application/json; charset=utf-8"
    ▹ "content-length": "130"
    ▹ "etag": "W/"82-l0Mhda3RFUb75lW/cRtznG5a9jI""
    ▹ "set-cookie": "connect.sid=s%3A0D8I6wmav5gUclgFPWA9u9WvCQ4oSNo7.u7xk7r6XkMbMdwsVtwArBZ1Q0DFT0pzo72tWRuh9JA8; Path=/; HttpOnly"
    ▹ "date": "Sat, 17 Feb 2024 21:55:29 GMT"
    ▹ "connection": "keep-alive"
    ▹ "keep-alive": "timeout=5"

  Body:
    {
      "email": "testuser3@test.com",
      "displayName": "john-smith",
      "_id": "65d12b5182456857b2b9c8ce",
      "__v": 0,
      "id": "65d12b5182456857b2b9c8ce"
    }

==============================

Should Fail

Setting should_fail: true on your test, as expected, will assert that the test should fail. In most cases, however, you should be able to acheive this functionality with the right matchers in your expect definition.

This example uses the should_fail attribute to ensure the test does not pass with a successful status.

  - test: "Protected route"
    description: "Attempting to access protected route without signin or signup"
    should_fail: true
    request:
      method: GET
      url: "${BASE_URL}/recipes"
    expect:
      status: 2xx
      body:
        recipes: $exists

However, a more declarative and idiomatic pattern would be to use matchers to assert the expected 400-level status code and absent request body information. This also enables asserting the correct error status code - in the case that the endpoint actually returns a 404 or 500-level status, the above test would pass, whereas this test would still detect the error and fail.

  - test: "Protected route"
    description: "Attempting to access protected route without signin or signup"
    request:
      method: GET
      url: "${BASE_URL}/recipes"
    expect:
      status: 403
      body:
        recipes: $absent

Matchers

Matchers are what make Capti so flexible when writing tests for HTTP endpoints. Often we don't know exactly what we're going to get back from the server (e.g. a unique ID value generated by the database). Using matchers, we can verify that the the response returns any value, or returns values that follow some sort of pattern or structure.

Here is an example test that makes heavy use of various matchers:

tests:
  - test: "get hello"
    description: "hello endpoint responds with some type of greeting"
    request:
      method: GET
      url: "http://localhost:3000/hello"
    expect:
      status: 2xx # match any 200 level status code
      headers:
        Content-Type: $regex /(json)+/ # match the provided regular expression
      body:
        message: $length >= 1 # ensure the message is not an empty string
        currentTime: $exists # match anything as long as it is present

Format

Matchers are always a keyword prefixed with a $ symbol, and followed by any number of arguments. The arguments, in some case, are also valid as matchers themselves, thus some matchers can be nested. Here are some examples:

# this matcher takes no arguments
$exists 

# this matcher takes one argument - a regex string used to match the response
$regex /[Hh]ello/ 

# this matcher checks whether the array at this position 
# contains an object with the property "id"
$includes { "id": "$exists" } 

# includes can also check any other valid YAML/JSON object, value, or matcher
$includes 5
$includes "hello, world!"
$includes $includes 5

# logical matchers can also be used to match multiple values conditionally or invert match results
$and ["$length 3", "$includes 5"]
$not $regex /[Ee]rror/

Matchers List

Basic Matchers

These matchers provide the base functionality for matching the most common assertions you will make.

  • $exists - matches anything except null or missing values.
  • $absent - matches null or missing values only. Great for asserting that a field like "password" should not be included in your response.
  • $regex - compares the response value to a provided regular expression argument and matches any occurrences of that expression.

Array Matchers

Matching with arrays can be very tricky with Capti. Capti cannot easily determine which element should match which, so by default all elements are matched in order. However, you may not know the exact order when writing your assertions. Array matchers provide flexibility in matching array elements.

  • $empty - asserts that the array is empty. This matcher also works for objects or strings.
  • $length - allows you to match the length of the array using exact values or comparisons. Also works with strings and objects.
  • $includes - asserts that the specified value or matcher matches at least one item in the array.
  • $all - asserts that every item in an array matches the specified matcher argument

Logical Matchers

Logical matchers use basic logic concepts to allow to match multiple possible values and conditions in your responses.

  • $and - provide multiple matchers as arguments to assert that your response matches all the arguments.
  • $or - provide multiple matchers as arguments and assert that at least one argument matches your response.
  • $not - invert the match result of the provided argument.
  • $if - evaluates a secondary match conditionally based on the result of an initial match

$exists

The $exists matcher checks that an object or value exists in the response.

More specifically, it will match everything except:

  • null
  • Values where the key/value pair is completely missing

Example

This is example is calling a POST endpoint to create a recipe. When the recipe is returned, it will have been given an id property by the database. We do not know what this id will be, and therefore we cannot assert that it should be any particular value. We do, however, want to make sure it exists in the response. This is a perfect use case for $exists.

  - test: Create new recipe
    description: Create a new recipe
    request:
      method: POST
      url: http://localhost:3000/recipes 
      headers:
        Content-Type: application/json
      body: ${RECIPE}
    expect:
      status: 2xx
      body:
        id: $exists

$absent

The $absent matcher is the inverse of $exists - where $exists matches any non-null value, $absent only matches null or missing values.

More specifically, it will match:

  • null
  • Values where the key/value pair is completely missing

Example

In the example below, we are calling an endpoint to get user information from the server. In our expect defintion, we assert that the password field should be null or missing, and we didn't accidentally include the password hash with the response after fetching user data from the database.

tests:
  - test: "Get user info"
    request:
      method: GET
      url: http://localhost:3000/user/${USER_ID}
    expect:
      email: test@test.com
      password: $absent

$regex

The $regex matcher uses a provided regular expression to determine whether a match is found in the specified response field.

Usage

The $regex matcher takes one argument, a regular expression wrapped in forward slash characters.

$regex /<regular expression>/

Note: Unless you include start/end matchers ^ and $ in your regular expression, $regex will match any contained instances of the value. The $regex matcher also only matches strings - any attempt to match another value type will simply result in the test failing.

Example

This example matches the description fields returned by the response body and matches any description which contains one or more matches of the word "guacamole" with a case-insensitive 'G'.

  - test: Recipe description mentions guacamole
    description: "Not sure why, but the description should mention guacamole at least once"
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    expect:
      status: 2xx
      body:
        id: ${RECIPE_ID}
        name: Guacamole
        description: $regex /([Gg]uacamole)+/

$length

The $length matcher can be used to assert the length of arrays, objects, and strings. You can assert using exact values, or a custom value matcher.

Usage

$length <arg>

Valid arguments include:

$length 5 # length is exactly 5
$length > 5 # length is greater than 5
$length >= 5 # length is greater than or equal to 5
$length < 5 # length is less than 5
$length <= 5 # length is less than or equal to 5
$length == 5 # length is exactly 5, same as first example

Example

tests:
  - test: At least two comments
    description: The comments field should have at least two comments
    request:
      method: GET
      url: http://localhost:3000/post/${POST_ID}
    expect:
      post:
        comments: $length >= 2

$empty

The $empty matcher checks that an object, array, or string has length 0.

Because of Capti's implicit matching, you cannot simply write something like this:

  expect:
    body:
      comments: []

The goal here is to assert that the comments array is empty, however because of implicit matching, this will match any number of comments.

To properly assert that there are no comments in this array, you can use the $empty matcher instead.

  expect:
    body:
      comments: $empty

You can also use $empty to match empty objects or strings. Using $empty is identical to writing $length 0.

$includes

The $includes matcher is used to verify that a specific value exists in an array.

Usage

$includes <value or matcher>

The $includes matcher takes one argument, which is a value or matcher that should be found in the array. It can be used to match primitive values:

$includes 5
$includes "Hello, world!"
$includes true

It can be used to match objects or other arrays, following the same implicit matching rules used in normal expect definitions. However, because of limitations of YAML, these objects or arrays must be defined as JSON wrapped in quotes, or separately as variables.

$includes "{ "id": "1A2B3C" }"
$includes "[1, 2, 3]"

It can also be used to match other matchers. For example, checking if a string that matches a regex using the $regex matcher:

$includes $regex /[Hh]ello/
# would match ["A", "B", "hello"]

Or matching another $includes to search nested arrays for a value:

$includes $includes 5
# would match [[1, 2, 3], [4, 5, 6]]

Examples

Here is a simple example that checks if an object with the specified "id" property is included in a returned data array.

tests:
  - test: Recipe included in list
    request:
      method: GET
      url: http://localhost:3000/recipes
    expect:
      body:
        data: $includes "{ "id": "${RECIPE_ID}" }"

For a more specific check, the expected item can first be defined as a local variable in the test. The expected value can still be defined as YAML and later will be expanded to the full value when the test is run.

  - test: Recipe included in list
    request:
      method: GET
      url: http://localhost:3000/recipes
    define:
      RECIPE:
        name: Guacamole
        description: A delicious classic guacamole recipe.
        ingredients:
          - 3 avocados
          - 1/2 red onion
          - 1 lime
          - 1 green bell pepper
          - 1 jalapeno pepper
          - 1/2 tsp cumin
          - 1/2 tsp salt
          - 1/2 tsp red pepper
        instructions: [
          "Roughly chop onion, green pepper, and jalapeno and add to food processor. Pulse 2-3 times.",
          "Cut and remove seeds from avocados, use spoon to scoop and mash in a bowl.",
          "Mix in the vegetables, seasonings, and lime juice squeezed from a fresh lime." ]
    expect:
      body:
        data: $includes ${RECIPE}

Alternatively, you may instead choose to define this variable as a suite variable, so that it can be used to create the resource as well.

$all

The $all matcher is used to assert that every value in an array returned in the response matches a certain condition. This can be useful for asserting that every object in a list of data has an id property, or verify that a user can only see data they are authorized to view.

Usage

The $all matcher takes one argument, a matcher that is compared to every item in the corresponding array. If any of those items fail to match, the test fails.

$all <matcher>

Example

This test asserts that every recipe returned from the '/recipes' endpoint contains a userId value that matches the current user (which is defined by the variable ${USER_ID}, presumed to have been extracted from an earlier test).

  - test: All recipes belong to user
    description: Ensure that the endpoint only returns recipes that belong to the current user
    request:
      method: GET
      url: ${BASE_URL}/recipes
    expect:
      status: 2xx
      body: 
        data: '$all { "userId": "${USER_ID}" }'

Here is another similar test that asserts that every recipe in the list has at least three ingredients and three instructions. This test defines a local variable to represent the expected match for each item in the list, which in turn uses a $length matcher to verify the associated arrays contain at least three items.

  - test: Recipe constraints upheld
    description: Verify that all recipes have at least three ingredients and three instructions.
    request:
      method: GET
      url: ${BASE_URL}/recipes
    define:
      EXPECTED_AMOUNTS:
        ingredients: $length >= 3
        instructions: $length >= 3
    expect:
      status: 2xx
      body: 
        data: $all ${EXPECTED_AMOUNTS}

$not

The $not matcher simply matches the opposite of the provided matcher or value.

Examples

  - test: Recipe not guacamole
    description: This recipe should not be named 'Guacamole'
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    expect:
      body:
        title: $not Guacamole

The $not matcher can match other matchers as well. This example uses a $regex matcher to ensure that a field in the response does not contain any quotation marks.

  - test: No quotes in name
    description: The recipe title should not have quotes in the name
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    expect:
      body:
        title: $not $regex /\"+/

In this more complex example, the $not matcher negates an $includes matcher to confirm that an object containing the specified id does not appear in the array.

  - test: Deleted recipe gone
    description: The now-deleted recipe should no longer appear in the list of recipes
    request:
      method: GET
      url: ${BASE_URL}/recipes/all
    expect:
      status: 2xx
      body:
        data: '$not $includes { "id": "${RECIPE_ID}" }'

$and

The $and matcher is a logical matcher that accepts an array of possible values or matchers and confirms that all of the provided items match the result. The matchers to compare to the response value must either be in JSON format as an array, or they can be defined as variables using YAML sequence syntax.

Examples

This example uses an $and matcher along with $includes and $regex matchers to confirm that the ingredients array contains at least avocado and lime. This uses JSON string syntax to feed the additional matchers as an argument to $and.

  - test: Proper guacamole
    description: Guacamole must have at least avocados and lime ingredients
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    expect:
      status: 2xx
      body:
        ingredients: '$and ["$includes $regex /avocado/", "$includes $regex /lime/"]'

Alternatively, and for clarity, you can provide the matchers as a variable - avoiding the need for JSON syntax and additional quotation marks in your tests.

  - test: Proper guacamole
    description: Guacamole must have at least avocados and lime ingredients
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    define:
      MIN_GUAC_INGREDIENTS:
        - $includes $regex /avocado/
        - $includes $regex /lime/
    expect:
      status: 2xx
      body:
        ingredients: $and ${MIN_GUAC_INGREDIENTS}

You are not limited to just two arguments when using the $and matcher. You can provide as many as you would like.

  - test: The best guacamole
    description: Guacamole must have avocados, lime, onion, jalapeno, and cilantro at least
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    define:
      MIN_GUAC_INGREDIENTS:
        - $includes $regex /avocado/
        - $includes $regex /lime/
        - $includes $regex /onion/
        - $includes $regex /jalapeno/
        - $includes $regex /cilantro/
    expect:
      status: 2xx
      body:
        ingredients: $and ${MIN_GUAC_INGREDIENTS}

$or

The $or matcher is a logical matcher that compares the response value to an array or sequence of possible values or matchers, allowing the test to pass if any of the provided values or matchers match the response. The matchers to compare to the response value must either be in JSON format as an array, or they can be defined as variables using YAML sequence syntax.

Examples

This example uses an $or matcher along with $includes and $regex matchers to confirm that the instructions array contains either the word 'mash' or 'stir'. This uses JSON string syntax to feed the additional matchers as an argument to $and.

  - test: Mash or stir
    description: Guacamole recipe should instruct you to either mash or stir the avocados.
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    expect:
      status: 2xx
      body:
        instructions: '$or ["$includes $regex /[Mm]ash/", "$includes $regex /[Ss]tir/"]'

Alternatively, and for clarity, you can provide the matchers as a variable - avoiding the need for JSON syntax and additional quotation marks in your tests.


tests:
  - test: Mash or stir
    description: Guacamole recipe should instruct you to either mash or stir the avocados.
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    define:
      GUAC_REQUIRED_INSTRUCTIONS:
        - $includes $regex /[Mm]ash/
        - $includes $regex /[Ss]tir/
    expect:
      status: 2xx
      body:
        instructions: $or ${GUAC_REQUIRED_INSTRUCTIONS}

You are not limited to just two arguments, you may specify as many as you would like and $or will retrun true as long as at least one is true.

tests:
  - test: Required kitchen tools
    description: Guacamole recipe should instruct you to use a fork, spoon, or a molcajete to mix the guacamole.
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    define:
      GUAC_REQUIRED_TOOLS:
        - $includes $regex /[Ff]ork/
        - $includes $regex /[Ss]poon/
        - $includes $regex /[Mm]olcajete/
    expect:
      status: 2xx
      body:
        instructions: $or ${GUAC_REQUIRED_TOOLS}

$if

The $if matcher, as expected, evaluates a secondary match based on the result of a previous match. The argument to the $if matcher is an array/sequence of either two or three items.

If two items are provided, the statement is evaluated as an if/then match. If the first item evaluates to true, then the second item must also evaluate to true. If the first item evaluates to false, then the second item is never evaluated and the whole statement returns true, passing the test.

ifthenresult
truetruetrue
truefalsefalse
falsetruetrue
falsefalsetrue

If three items are provided, the statement is evaluated as an if/then/else match. The first and second items are evaluated the same way as an if/then match, except that if the first item returns false, then the third item must evalaute to true for the test to pass.

ifthenelseresult
truetruetruetrue
truetruefalsetrue
truefalsetruefalse
truefalsefalsefalse
falsetruetruetrue
falsetruefalsefalse
falsefalsetruetrue
falsefalsefalsefalse

Example

This example uses the $if matcher, along with the $regex and $includes matchers, to assert that if any recipes in the list contain the word "guacamole" in the name, that they also contain "avocado" somewhere in the list of ingredients.

The example uses a local variable to provide the $if arguments in a YAML sequence. Each item in the sequence is an object with either the property name or ingredients.

  - test: Guac recipes have avocado
    description: Any guacamole recipes in the list should contain avocado as an ingredient
    request:
      method: GET
      url: ${BASE_URL}/recipes
    define:
      RECIPE_IF:
        - name: $regex /[Gg]uacamole/
        - ingredients: $includes $regex /[Aa]vocado/
    expect:
      status: 2xx
      body: 
        data: $includes $if ${RECIPE_IF}

Variables

Variables are one component of Capti test suites that provide a lot of power. They enable features such as testing authorization flows, embedding complex mappings or sequences as matcher arguments, or simply reducing boilerplate and making your tests more DRY.

Variable Types

Variables can be defined statically as part of the suite or test configuration, or dynamically with extract definitions. They can also be pulled from the environment or from a .env file.

  • Global Variables - Defined at the suite level under the field variables. They are available to all tests within the suite. They are most useful for defining values that you intend to repeat multiple times throughout your tests.
  • Extracted Variables - These are extracted from your tests an can be used in any subsequent tests. They are useful for comparing dynamic values such as unique identifiers for resources or authentication tokens.
  • Local Variables - These are defined at the test level under the field define. They are only valid for that single test. They can be useful for specifying complex structures to be used as matcher arguments, or they can be used to override suite variables for a single test.
  • Environment Variables - These are pulled from your shell environment or from a env file. They are useful for synchronizing values in your tests that are also available to your server or other services.

Variable Precedence

Variables can be defined as any or all of the above types, and certain types take precedence over others. In order from highest precedence:

Local -> Extracted / Global -> Environment (.env) -> Environment (Shell)

Local variables will be applied first. Next, global or extracted variables are used (extracting a variable with the same name as a global variable permanently overrides the global variable). Lastly, if no variable is found in the local or global space, the env file and shell environment are searched - with env file variables taking precedence over shell environment as well.

It's generally advised to avoid clashing variable names anyway.

Simple Values

The basic usage of variables is to reduce repetitive values, such as a BASE_URL for your endpoints.

suite: "Create Recipe"
description: "This suite involves creating a new recipe and fetching its information."
variables:
  BASE_URL: http://localhost:3000

tests:
  - test: Fetch recipes
  description: List of public recipes contains at least one recipe
  request:
    method: GET
    url: ${BASE_URL}/recipes
  expect:
    status: 2xx
    body: 
      data: $length >= 1

Local Variables

Local variables are defined for a single test. They are primarily useful when using complex matcher arguments and you need to define nested structures to be used in your assertions.

In this example, an array of matchers is defined as a local variable so that it can easily be used with the $and matcher, which accepts an array of matchers as an argument.

tests:
  - test: Required kitchen tools
    description: Guacamole recipe should instruct you to use a fork, spoon, or a molcajete to mix the guacamole.
    request:
      method: GET
      url: ${BASE_URL}/recipes/${RECIPE_ID}
    define:
      GUAC_REQUIRED_TOOLS:
        - $includes $regex /[Ff]ork/
        - $includes $regex /[Ss]poon/
        - $includes $regex /[Mm]olcajete/
    expect:
      status: 2xx
      body:
        instructions: $or ${GUAC_REQUIRED_TOOLS}

Local variables have precedence over suite variables, so if desired you can override suite variables for a single test by specifying a variable with the same name in the define field. Alternatively, you could just use a different variable.

Complex Variables

Beyond simple values, variables can be used to define entire mappings or sequences, and even matchers. Variables can reference other variables, and can be used in place of entire request mappings or response body mappings. Additionally, some matchers that accept embedded mappings or sequences (such as $includes), it can be more ergonomic to predefine these mappings as variables.

Nesting Variables

You can reference, or nest, variables within other variables. Nested variables are recursively resolved and expanded.

suite: "Create Recipe"
description: "This suite involves creating a new recipe and fetching its information."
variables:
  BASE_URL: http://localhost:3000
  RECIPE_URL: ${BASE_URL}/recipes

Mappings or Sequences

You can define entire mappings (objects) or sequences (arrays) as variables and use them in your tests. In the example below, and entire RECIPE object is defined as a variable and then used as the request body for creating the recipe, and later used as an expect definition when verifying the correct recipe is returned.

suite: 'Create Recipe'
description: 'This suite involves creating a new recipe and fetching its information.'
variables:
  RECIPE_URL: http://localhost:3000/recipes
  USER_EMAIL: recipe1@tests.com
  USER_PASSWORD: hG98s4%%phG
  RECIPE:
    name: Guacamole
    description: >
      A delicious classic guacamole recipe.
    time: 10
    servings: 6
    ingredients:
      - 3 avocados
      - 1/2 red onion
      - 1 lime
      - 1 green bell pepper
      - 1 jalapeno pepper
      - 1/2 tsp cumin
      - 1/2 tsp salt
      - 1/2 tsp red pepper
    instructions:
      [
        'Roughly chop onion, green pepper, and jalapeno and add to food processor. Pulse 2-3 times.',
        'Cut and remove seeds from avocados, use spoon to scoop and mash in a bowl.',
        'Mix in the vegetables, seasonings, and lime juice squeezed from a fresh lime.',
      ]

tests:
  - test: Create new recipe
    description: Create a new recipe
    request:
      method: POST
      url: ${RECIPE_URL}
      headers:
        Content-Type: application/json
      body: ${RECIPE}
    expect:
      status: 2xx
      body: ${RECIPE}
    extract:
      body:
        id: ${RECIPE_ID}

  - test: Get recipe
    description: 'Should be able to get recipe information'
    request:
      method: GET
      url: ${RECIPE_URL}/${RECIPE_ID}
    expect:
      status: 2xx
      body: ${RECIPE}

Embedded Matcher Arguments

When using a matcher like $includes that expects any value, mapping, or sequence as an argument - defining something complex like an object requires using JSON strings (because of the limitations of YAML nesting in strings). Instead of defining a complex JSON string and wearing out your " key, you can define the object you expect to find in the array as a variable.

For this example, reference the previous example's use of the RECIPE variable.

  - test: Recipe in list
    description: Recipe should be visible when listing all recipes
    request:
      method: GET
      url: ${RECIPE_URL}/all
    expect:
      status: 2xx
      body: 
        recipes: $includes ${RECIPE}

At runtime, the ${RECIPE} will be expanded to the full mapping as defined above, and will then proceed to be correctly parsed as an object to be matched.

Extracting Variables

Variable extraction enables your tests to adjust dynamically to whatever scenario you've decided to test. You can extract auto-generated IDs from your database, or JWT tokens for use in making subsequent authorized requests.

Simple Extraction

To extract a variable from a repsonse, create an extract mapping definition on your test. The extract definition follows the exact same layout as expect, except variables placed here will be populated, instead of expanded.

In this example, the USER_ID which was auto-generated by the db is extracted, as well as the authorization token.

  - test: "Sign up"
    description: "Signing up as a new user for the site"

    request:
      method: POST
      url: "${BASE_URL}/auth/signup"
      headers:
        Content-Type: application/json
      body:
        email: ${USER_EMAIL}
        displayName: john-smith
        password: ${USER_PASSWORD}

    expect:
      status: 2xx
      headers:
        Content-Type: $regex /json/
      body:
        id: $exists
        email: ${USER_EMAIL}
        displayName: john-smith
        password: $absent

    extract:
      headers:
        Authorization: Bearer ${AUTH_TOKEN}
      body:
        id: ${USER_ID}

Those collected variables can then be used in subsequent requests. In the following example, the USER_ID we collected is used as a URL path param, and the AUTH_TOKEN we collected is used to authenticate the request.

  - test: "Access profile data"
    description: "After signing in, the user can get their profile data"

    request:
      method: GET
      url: ${BASE_URL}/user/${USER_ID}
      headers:
        Authorization: Bearer ${AUTH_TOKEN}

    expect:
      status: 2xx
      body:
        firstName: $exists
        lastName: $exists
        imageUrl: $regex /.*\.png$/

Embedded Extraction

Notice in the above extract example that ${AUTH_TOKEN} is placed after "Bearer" - this ensures that only the token is extracted, and not the word "Bearer". That's not particularly useful in this case since the word "Bearer" is still going to be used in subsequent tests anyway, however there are some cases where needed values are embedded in long strings of content. This extraction works as if the entire string were a regex with your variable as a capture group.

For example, if your extract definition has the following:

  extract:
    body:
      message: The quick ${COLOR} fox jumped over the ${ADJECTIVE} dog.

And the actual response looks like this:

  body:
    message: The quick brown fox jumped over the lazy dog.

Then the resulting values for ${COLOR} and ${ADJECTIVE} will evaluate to "brown" and "lazy", respectively.

Considerations

  • Extracted variables cannot be used in suites with the configuration option parallel: true set. This is because tests running in parallel cannot reference variables extracted from each other.
  • Currently, extracted values can only be strings. Unlike statically defined variables, you cannot extract entire mappings or sequences from a response.
  • Using extracted variables in subsequent tests creates an inherent dependency of those tests on the test which performs the extraction. If a test with an extract definition fails, all its dependent tests will fail as well. Keep this in mind when you see many failures - it may just be one test causing the issue.

Environment Variables

By default, if you specify a variable in your tests but the declaration for that value cannot be found, Capti will then default to searching your local environment for that variable.

For example, if you have a SERVER_URL variable defined in your local environment, you can use that like any other variable:

  - test: Get a user
    request:
      method: GET
      url: ${SERVER_URL}/users
    expect:
      status: 200

However, since variables defined in your test suite take precedence, if you were to define the SERVER_URL in your suite configuration, that value will be used instead.

suite: User endpoint tests
variables:
  SERVER_URL: http://localhost:4000

tests:
  - test: Get a user
    request:
      method: GET
      url: ${SERVER_URL}/users
    expect:
      status: 200

Env File

If you want to load variables into your environment from a .env file in your project, you can specify the path to your .env file in your global config. These variables are not loaded by default.

Just as with environment variables that already exist in your terminal environment, variables loaded from .env files will never overwrite variables defined in your test suites.

Note: Currently, variables loaded from .env are not available when declaring variables in your test suites, so you cannot compose static variables from .env variables. This is expected to change in the future.

Contributing

If you're interested in contributing to Capti, please reach out! I'd love to collaborate on new ideas and the future of Capti tests. Feel free to help with any of the below items, or by creating an issue.

Development

If you're even just a little familiar with Rust, I'd love your help in developing components of Capti and fixing bugs. Especially if you are experienced in working with serde or reqwest.

Additionally, if you'd like to help write some Node/Express/TypeScript code for the test app so that more scenarios can be used for examples and testing, that would be a huge help.

The future of Capti may involve elements such as custom LSPs, syntax highlighting for YAML files, VSCode extensions, etc. If you have experience with any of these, I would appreciate your help.

Testing

Want to help test Capti? If you happen to have a few REST APIs laying around that you've developed over the years, use Capti to create some tests for them. Try testing various use cases and see if you come across any limitations or issues.

Feature Suggestions

Have an idea that might improve Capti? I'm all ears. I have a lot of ideas I'd like to implement in the future, but I'm always open to new and thoughtful suggestions - especially from users of the framework.

Reporting Issues

Capti is not a very mature technology, and therefore issues are bound to occur. Taking the time to report them when they happen will help Capti to become a more stable and mature testing framework.

Issues

To report a problem that you have had, please open an issue on GitHub. Please review the following types of issues and include as much of the requested information as possible.

Installation Issues

NPM Installation

If your NPM install did not succeed, please open an issue. Mention your CPU architecture, your platform (Windows, Linux, MacOS), and try to include the following:

  • CPU Architecture (x86, ARM, etc.)
  • Platform (Linux, MacOS, Windows)
  • Log file or files located in node_modules/capti/logs
  • Any logs from NPM, if available