We believe that the entire way of thinking about tests as fundamentally a collection of imperative scripts that may or may not have shared blobs of setup code critically limits the capabilities of working with those those tests at runtime. Instead we propose enforcing a more functional separation of actions and assertions.
What does this look like?
When writing tests written in the conventional style established by unit test frameworks, or even the more expressive BDD-style frameworks inspired by RSpec like Jest and Mocha, we often come up against a problem of expressiveness and good practices vs performance. This is especially problematic in acceptance tests where run time is generally slow.
Let’s take a look at a common way of writing a test in the traditional way to explore the problem.
describe("logging in", () => {
it("can successfully logs in and log out", async () => {
await createUser({ email: "[email protected]" });
await fillIn("Email", { with: "[email protected]" });
await clickButton("Log in");
expect(headline("Welcome to the App")).to.exist();
expect((text("Signed in as Jonas")).to.exist();
clickButton("Log out");
expect(headline("Welcome to the App")).to.exist()
expect(text("Signed in")).not.to.exist();
});
});
Even in this relatively simple example, there are a few problems.
-
It is good practice to write tests in a setup->action->assert style, but we are not following this practice here. While it would be “purer” to write the second part of the test (logging out) as a separate test, it would also be much slower, since we’d need to repeat the first part.
-
The describe
/it
terminology does nothing to make this test more understandable. While it works well, and makes sense for unit tests, in this case there really isn’t anything to describe, and “it successfully logging in” doesn’t make any sense.
-
The steps in the example could really use some elaboration. What happens on the first line? We are setting up a user for the example, but this is not explicitly stated. While in this simple example it is fairly easy to discern a logic to the flow, this will not always be the case if the examples become more complicated.
-
A good practice is to write a single assertion per example. The main purpose of this is so that the assertion can be adequately described, and a failed assertion be shown with its proper context. However, writing examples this way is prohibitively slow, since we’d have to rerun the whole example for each assertion.
Completely Rethinking it
We propose a completely new style of writing examples which solves these problems by imposing more structure onto the way tests are expressed. We are calling these tree style tests, because the structure of the examples forms a tree. Branches of the tree represent actions to be taken and the leaves represent assertions.
In this model, assertions must be side-effect free, that is an assertion cannot change the state of the test in any way. This makes it possible to execute multiple assertions after a single action, and even to proceed with further actions after executing an assertion.
To further break down the examples into smaller units, the action taken by a branch node in a tree may be broken down into multiple steps, and there are two distinct kind of steps, setup and action steps.
To get away from the technical jargon somewhat, we have chosen to call setup steps “Given” steps, actions steps “When” steps, and assertion steps “Then” steps. This follows the terminology of tools such as cucumber, rspec-given and similar BDD-style frameworks.
If we imagine the example as a data structure, it could look somewhat like this:
Example {
givens: GivenSteps[],
whens: WhenStep[],
thens: ThenStep[],
children: Example[]
}
Let’s look at the example above and imagine what it could look like as a tree:
![unnamed](https://user-images.githubusercontent.com/4205/70252516-2b03af00-1747-11ea-87ec-d6251b388ca1.png)
In pseudo-code this could look as follows:
example("logging in", () => {
given("a valid user", async () => {
await createUser({ email: "[email protected]", password: "1234"});
});
when("I fill in valid credentials", async () => {
await fillIn("Email", { with: "[email protected]" });
await fillIn("Password", { with: "1234" });
});
when("I press the 'Log in' button", async () => {
await clickButton("Log in");
});
then("I should be on the home page of the application", () => {
expect(headline("Welcome to the App")).to.exist();
});
then("I should be signed in", () => {
expect(text("Signed in as Jonas")).to.exist();
});
example("logging out", () => {
when("I log out", async () => {
await clickButton("Log out");
});
then("I should be on the home page of the application", () => {
expect(headline("Welcome to the App")).to.exist()
});
then("I should not be signed in", () => {
expect(text("Signed in")).not.to.exist();
});
});
});
While this example is significantly longer than the original, it is much clearer about the steps being performed, the assertions being executed and the hierarchy of the examples.
By making the actions of a test an explicit, 1st class entity, this allows us to make several game-changing optimizations when actually running the tests.
Don't run redundant actions (setup).
Under classic runners, we need to run the entire setup chain for every single assertion. In this hypothetical example, that would be a total of ten actions: one set for each then
declaration.
before
assertion 1: logging in/I see the headline "Welcome"
- fill in credentials
- when I push the "login button"
assertion 2. logging in/the I see the text. "Signed In"
- fill in credentials
- when I push the "login button"
assertion 3. logging in/logging out: then I see the headline "Welcome"
- fill in credentials
- when I push the "login button"
- when I push the "log out button"
assertion 4. logging in/logging out/then I do not see the text "Signed In"
- fill in credentials
- when I push the "login button"
- when I push the "log out button"
However, because we are now orienting our tests around a specific sequence of actions contained in separate code, we only need to run each sequence of actions one time to put an application into a particular state, and then make any number of pure assertions against that state. In this case, it means we only have to run five actions.
after
Sequence 1
actions
- fill in credentials
- when I push the "login button"
assertions
- I see the headline "Welcome"
- I see the text. "Signed In"
Sequence 2
actions
- fill in credentials
- when I push the "login button"
- when I push the "logout button"
assertions
- I see the headline "Welcome"
- I do not see the text. "Signed In"
Even with this trivial test suite, we've cut the amount of actions (slow code) that needs to be run by half. Basically, it's a a geometric level of savings which will yield gigantic cost reduction for larger, more complicated test suites.
Fail super fast
Another advantage of this approach is that failures at a high level in a tree will automatically fail the rest of the tree, rather than the test suite attempting to execute the same bound-to-fail code again and again. This shortens the feedback cycle in case of errors. This is especially valuable with acceptance tests, where due to synchronization issues, we will often have to wait a long time before deciding to fail an assertion.
Requirements
The pseudo code outlined above is one example of how we might declare, but it's the tree nature of the suite and the hard separation of side-effects and assertions that gives us the key runtime capabilities. What it actually looks like is still up in the air but nailing down a draft of it is the goal of this issue.
Given that, there are still some high-level constraints to observe:
No side effects
Classic runners all express test suites by mutating a shared global "root test case". E.g.:
describe('a context, () => {
beforeEach(() => {
});
it('does stuff', () => {
expect(thing).to.beACertainWay();
});
});
@bigtest/suite
is a test suite description language which will declare test suites as immutable data structures. E.g.
import { describe } from '@bigtest/suite';
export default describe('context, () => {
})
Decoupled from runtime semantics
Classic test runners treat the test suite declaration also as the runtime data about executing a test trial. @bigtest/suite
will be a separate package that contains only the suite declaration syntax. (It can live in the server for the time being though)
Specifically, it will not impose any constraints on how a test is actually run. At the highest level, test modules are just functions that return a value (the suite)
Among other things, if we are just working with a tree value, then we can experiment with other syntaxes in the future on how to build that tree value.
Gracefully pass values from actions to assertions
In Mocha and Jest, if you want to pass data from the actions down to child actions or child assertions, you need to declare a mutable variable, assign to it during the action, and then read it during your assertion. For example:
describe('view profile', () => {
let user = null;
beforeEach(() => {
user = createUser();
});
beforeEach(async () => {
await visit(`/users/${user.id}`);
});
});
Rather, we need a way for data to flow naturally down the tree. One way might be to pass the return value of each action downwards:
describe('view profile', () => {
beforeEach(() => {
return { user: createUser() };
});
beforeEach(async ({ user }) => {
await visit(`/users/${user.id}`);
});
});