​
The Goal

Imagine you are updating a RESTlet. The RESTlet is responsible for creating item fulfillments. You want to test that your update works. What do you do? You probably:
  1. Create a sales order in a Pending Fulfillment status
  2. Call the RESTlet with a specific payload
  3. Examine the sales order status to ensure it has moved to Pending Billing. You maybe even open the related item fulfillment and ensure the correct location is set. And if it has a Pending Billing status (and correct fulfillment location), then you conclude that your code worked and “passed your test”.
Our goal is to store a definition of these manual steps, so that we can run them automatically. We call this definition a “test”. We consider each of the numbered steps above a “test step”.

​
Core Architecture

  1. Step
    • Setup
    • Execute
    • Evaluate
  2. Test
  3. Run
  4. Group Run
A Test is the smallest unit of work you can call. When you call a test it performs each of its Steps in subsequent order. When a test finishes, it creates a record called a Run that stores data, such as whether the test passed, as well as notes on what happened during the test, such as records created, RESTlet responses, and errors. You can also several tests at once, for instance you may want to run your entire testbed overnight. When several tests run together, they are create a Group Run. A Group Run is a collection of Runs and stores information such as the percentage of tests that passed and a list of tests that failed. The below picture shows a group run, where each line represent a run and the Test column represents the test that was run.

​
Steps

Steps are the most complex part of Assurance and where you will spend the most of your time. To make them easier, we built a Step Builder app to make it easier to use. The important thing to know is that steps come in 3 flavors: Setup, Execute, and Evaluate. These map onto the 3 classic steps of software testing: Arrange, Act, and Assert. In the example above, Step #1 is a Setup step because the developer is creating data. Step #2 is an Execute step because the developer is calling a script, in this case their RESTlet. And Step #3 is an Evaluate step because the developer is checking the results of their RESTlet. The Evaluate step is where a test passes or fails. If every field matches the expected value, the test passes. If any field does not match, the test fails. Below is a screenshot of Step #2 in Step Builder.

​
Test Runner

Behind the scenes, when you call a test (or group of tests), a map/reduce is triggered that does all of the work. Therefore, to run a test you can:
  1. Call the map/reduce directly (ad-hoc or scheduled). Use script parameters to define which test(s) to return.
  2. Click the “Run” button on a test (or on a tag - more on them later) which will call the map/reduce for you.
  3. Call a pre-built endpoint to run this via API call, such as from the terminal or GitHub action - see “Operationalize Your Tests - Running Your Tests” for details.
Below you can see the parameters for calling the map/reduce directly.