udibo / test_suite Goto Github PK

Below is an example of a test created with a string and function. If you're not making use of other TestDefinition options, it is easier to write the test like this and the function's body will have less indentation.

test("example", () => {
  assert(true);
});

Currently if you want to focus or skip the test, you have to rewrite it to use a TestDefinition.

test({
  name: "example",
  only: true,
  fn() {
    assert(true);
  },
});

I'd like this module to provide shorthands to enable focusing or skipping tests without modifying the call signature.

The first option I thought of is adding functions onto the test function.

test.only("example 1", () => {
  assert(true);
});
test.ignore("example 2", () => {
  assert(true);
});

Another option would be to export separate test functions. The downside to this way is that it would require importing the functions.

testOnly("example 1", () => {
  assert(true);
});
testIgnore("example 2", () => {
  assert(true);
});

Instead of having describe return the TestSuite, have it return a symbol that maps to a TestSuite object instead so that it is never directly exposed. Then update everywhere that has a suite argument to take that symbol instead of the TestSuite object.

Currently this module requires hooks to be registered first before calling describe/it globally.

At the global level this was needed because I need to determine if Deno.test should be called once at the top level with all the hooks and top level test cases being called as steps or if the top level tests should be registered with Deno.test instead. If there is a way to delay calling Deno.test until all test cases are registered using describe/it, I could make the determination right before the tests start running. That would make it possible to call the global level hook functions and describe/it blocks in any order.

Currently this module has 2 different ways of writing test suites, either with TestSuite and test or describe and it. Initially I wrote TestSuite and test syntax as an easy way to write grouped tests without having to nest them in callbacks for each group layer like you currently have to with describe. If you have multiple layers of tests, you would have a lot of indentation in front of each line of your test. With TestSuite and test, all tests have the same base indentation level, you just pass in what group they belong to as an argument.

If I change the describe/it functions to optionally take a TestSuite or DescribeBlock or Symbol representing the test group and update describe to return it for passing into other describe/it function calls to make them children of it.

If a test suite doesn't have any setup, teardown, or test definition options besides names, accept creating a test suite with just a name string argument.

const userSuite: TestSuite<void> = new TestSuite("user");

A file could be treated as a nameless suite. The beforeEach, afterEach, beforeAll, afterAll hooks can be made to work for the whole file. They would need to be called before any tests are created. This should work since each file is run separately by deno's test runner.

Deno 1.19 added better explanations for leaking ops which are shown when using Deno.test but not when using describe/it

With Deno.test:

Test case is leaking async ops.

- 1 async operation to sleep for a duration was started before this test, but was completed during the test. Async operations should not complete in a test if they were not started in that test.

With describe/it:

AssertionError: Test case is leaking async ops.
Before:
- dispatched: 5
- completed: 4
After:
- dispatched: 36
- completed: 36
Make sure to await all promises returned from Deno APIs before
finishing test case.
    at assertEquals (https://deno.land/[email protected]/testing/asserts.ts:269:9)
    at assertOps (https://deno.land/x/[email protected]/test_suite.ts:149:3)
    at async fn (https://deno.land/x/[email protected]/test_suite.ts:482:28)
    at async testStepSanitizer (deno:runtime/js/40_testing.js:432:7)
    at async Object.exitSanitizer [as fn] (deno:runtime/js/40_testing.js:415:9)
    at async runTest (deno:runtime/js/40_testing.js:673:7)
    at async Object.runTests (deno:runtime/js/40_testing.js:786:22)

Deno recently added the ability to shuffle tests.

https://deno.com/blog/v1.12#improvements-to-test-runner

Currently the beforeAll and afterAll hooks end up being included in the first and last tests added to a test suite respectively. With shuffle, those might not be the first and last tests to run in a test suite. To work around this, each test should be checking if it is the first or last test in all the parent test suites, then call the beforeAll or afterAll hooks if they are the first or last to run. To determine if a test is first, we could create a map of the test suites with the count of how many have started and how many were registered. when 0 have started, that would mean it's the first and that the beforeAll hook should be called first. When the number of started tests match the number of registered tests, the afterAll hook should be called for the test suite.

There currently isn't any information about or examples of test focusing and ignoring in the readme file. It should have information about it and an example to make it more clear to users that the capability is available.

Currently the only flag will only apply to the top level test it is nested in. That means that tests before and after it at the top level will still run without having an only flag. That is because the top level tests get registered immediately and I cannot modify the definition after it is registered.

If there was or is a way to detect when tests will start running, I could delay actually calling Deno.test until right before it's needed, that would allow me to add only flags to the top level tests that need them.

Currently the tests verify that this module will call Deno.test and t.step with the correct arguments. To be able to add this module to deno std, I would need to remove my usage of the mock module for verifying behavior and instead do integration testing.

denoland/std#1991

Deno does not have subtests yet but may sometime soon. Here is a discussion about the design that is most likely. Once implemented into deno, TestSuite should be updated to use it. This will make the output cleaner for test groups and simplify test setup/cleanup.

denoland/deno#10771

Hey @KyleJune !

so i'm walking through your module and i noticed the following in the README section (https://github.com/udibo/test_suite#flat-test-grouping) errors out:

import {
  getUser,
  resetUsers,
  User,
} from "https://deno.land/x/[email protected]/example/user.ts";

found that it should be this?

- } from "https://deno.land/x/[email protected]/example/user.ts 
+ } from "https://deno.land/x/[email protected]/examples/user.ts

The describe and it functions both have versions for easily focusing or ignoring them. The each function was added in test_suite v0.8.0 but does not have equivalent functions for easily focusing or ignoring them. Currently the only way to focus or ignore each cases is to use the test definition call signature so that you can pass an ignore or focus parameter.

My change to replace TestSuite with a symbol broke it being able to infer the type of the ItDefinition's fn function to infer the type of T from the suite. It's inconvenient to have to declare the generic type for each it call, so wrap the symbol in a basic interface so that T can be inferred from the suite.

The describe and it functions have call signatures that currently do not have direct test coverage. Add test coverage for those call signatures.

Currently I restrict each hook to being called once per describe block and once globally, meaning you cannot have multiple beforeEach calls in a single describe block. It would be easy to support having multiple beforeEach calls, it would just require adding additional test coverage for verifying it works correctly.

Deno.test's TestDefinition has an optional sanitizeExit arg but TestSuite currently does not make use of it. TestSuite should support all of the options that Deno.test supports.

I added tests for hooks on top level suites but still need test coverage for hooks on multi level suite tests. They should verify that it will call parent suite hooks if they are defined.

Currently TestSuite depends on https://deno.land/x/[email protected] for a Vector class. Vector is basically a wrapper around JavaScript Array to make it more efficient to shift/unshift values from it. After refactoring this module in v0.10.0, those functions are no longer used, so there is no benefit to using it over Array besides the delete helper function which can be easily replaced with splice.

This will need to be done for this module to get added to deno std.