Run subset of tests and address performance issues about openactive-test-suite HOT 9 CLOSED

Additionally the test suite currently takes around 60 seconds to startup, before it runs any tests.

It outputs jest and then appears to wait for 30 seconds, before outputting Found 7 test suites and then waiting for a further 30 seconds.

Can we improve performance here? What are we doing to slow things down?

from openactive-test-suite.

Also we should make parallel execution of tests configurable, as this can have the effect of slowing down testing for a larger number of tests against a database

from openactive-test-suite.

This is already configurable, just needs documenting. Defaults are in the package.json, and can also be changed via commandline args.

npm run test -- --maxWorkers 1

from openactive-test-suite.

Running a subset of tests can be done just by specifying the test path, this can be a directory (run everything within).

npm run test -- test/flows/book-and-cancel

(run all book and cancel flows).

all the way down to specifying an individual test file:

npm run test -- test/flows/book-and-cancel/book-and-customer-cancel-success/book-free-test.js

from openactive-test-suite.

Ok great! Can we also use wildcards?

The usecase I'm thinking of is where we have (following discussions this morning and notes in #34):

• test/flows/book-and-cancel/book-and-customer-cancel-success/book-free-test-scheduledsession.js (booking a SessionSeries)
• test/flows/book-and-cancel/book-and-customer-cancel-success/book-free-test-slot.js (booking a Slot)
• test/flows/book-and-cancel/book-and-customer-cancel-success/book-free-test-slot-and-scheduledsession.js (booking multiple OrderItems - a SessionSeries and a Slot together - as per #34 (comment))

and we perhaps only want to run those that only involve "ScheduledSession" (because they haven't implemented Slots in their system)

Unless there's another approach we can take here?

from openactive-test-suite.

Need to allow a subset of the tests to be run for only SessionSeries, or FacilityUse, or both or more depending on what feeds have been implemented by the booking system

from openactive-test-suite.

Trying to figure out whether we can implement some sort of filtering system - being able to define them much like rspec would be nice: https://relishapp.com/rspec/rspec-core/docs/hooks/filters

from openactive-test-suite.

@ylt to scope this issue down to something achievable within a reasonable timebox, could we just cover the following two in a config file:

Configuration spec

The test suite can be configured to test "optional" features, by indicating whether they are either:

• implemented - the tests will run to confirm proper implementation of the feature
• not-implemented - the tests will run to confirm the feature is correctly advertised as "not implemented"
• disable-tests - disable all tests for this feature (only recommended during development)

Additionally, bookable opportunity types can be configured, to indicate which types the implementation is expected to support:

• sessions (dual feeds of SessionSeries and ScheduledSession)
• facilities (dual feeds of FacilityUse and Slot)
• events (feeds of Event)
• headline-events (feeds of HeadlineEvent with embedded Event)
• courses (feeds of CourseInstance)

Configuration example

An example config file could look like this:

{
  "features": {
    "opportunity-feed": "implemented",
    "dataset-site": "implemented",
    "availability-check": "not-implemented",
    "simple-book-free-opportunities": "not-implemented",
    "simple-book-with-payment": "not-implemented",
    "payment-reconciliation-detail-validation": "not-implemented",
    "booking-window": "not-implemented",
    "customer-requested-cancellation": "not-implemented",
    "seller-requested-cancellation": "not-implemented",
    "seller-requested-cancellation-message": "disable-tests",
    "cancellation-window": "implemented",
    "seller-requested-replacement": "implemented",
    "named-leasing": "implemented",
    "anonymous-leasing": "implemented"
  },
  "opportunity-types": {
    "sessions": true
    "facilities": true
    "events": false
    "headline-events": false
    "courses": false
  }
}

Logic example

An example of how this should work:

If "simple-book-with-payment": "implemented", the following test should be run:

• within openactive-integration-tests/test/features/payment/simple-book-with-payment/implemented

If "simple-book-with-payment": "not-implemented", the following test should be run:

• within openactive-integration-tests/test/features/payment/simple-book-with-payment/not-implemented

If "simple-book-with-payment": "disable-tests"

• neither tests should be run

Depending on the boolean values of opportunity-types, the array in the following should be populated:

• openactive-integration-tests/test/features/payment/simple-book-with-payment/implemented/book-random-test.js

from openactive-test-suite.

All the issues mentioned here have been resolved / implemented

from openactive-test-suite.