Great convergence effort/suggestion: Making QUnit proxy to node.js and deno testing APIs in these environments about qunit HOT 2 OPEN

@izelnakri Hey. Thanks for reaching out.

Let me echo some of this back to make sure I understand the broader idea and the specific proposal. The idea is to provide multiple implementations of the QUnit API (one for nodejs, one for browser, one for deno). For Deno and Node.js, these would not use the current test runner, but rather proxy to the new test runners built-in to Deno and Node.js, register each module and their tests there, and have that layer responsible for selecting, filtering, and running tests, and also let that layer control the output/results (i.e. serializing and formatting values for assertion failures, and TAP reporter).

The overaching purpose would be that applications and libraries that consider themselves isomorphic, can write their tests once and run them in all three kinds of environments.

Correct me if I got any of that wrong or otherwise differently from what you're thinking. So far, that sounds awesome to me. For test suites, things would be a lot simpler if test suites use ESM for discovering the source code imports, as that way they'll naturally work in both the browser and in Node.js/Deno.

Today most test runners require the HTML entrypoint and the CLI entrypoint to both sepecify the list of source files and test suits (though Karma and QUnit CLI both support glob to make this easier), and then there is Airtap which gets around this by standardising on CJS/ESM and runnning browserify for the browser target (though I dislike that as it adds overhead and complexity to both source and test code which make it harder to debug and less like how it runs normally in the browser).

If we add an options argument to QUnit.module() and QUnit.test(), the part that I'm not yet understanding, is what this would represent in the API. If it is a transparent pass-through to specific options that exist only in Deno, or only in Node.js, would that take away from the universality? E.g. if you set concurrency in your test, what would this do in the browser today? But.. maybe this is okay if the options are only management tweaks and e.g. not the kind of option that would be functionally observable and change behaviour such that it makes a test pass or fail depending on it.

Thinking more broadly about the strategy, would you mind drawing a comparison between a dream outcome and say what we have today with alternate approaches that run QUnit tests in Node + Browser. What benefits are you most excited about? E.g. something we can make possible and/or much easier.

from qunit.

Glad to hear you find this approach worthy for evaluation @Krinkle ! :)

If we add an options argument to QUnit.module() and QUnit.test(), the part that I'm not yet understanding, is what this would represent in the API. If it is a transparent pass-through to specific options that exist only in Deno, or only in Node.js, would that take away from the universality? E.g. if you set concurrency in your test, what would this do in the browser today? But.. maybe this is okay if the options are only management tweaks and e.g. not the kind of option that would be functionally observable and change behaviour such that it makes a test pass or fail depending on it.

options second argument module and test functions would indeed represent all the configuration that can be done to a specific test or module. Ideally they should use the defaults so they can be overriden by cli arguments: qunit test.ts --concurrency=0. This is needed because in the ideal world, the default would be that every test case is concurrent and parallel by default. Elixir programming language achieves this by running test suite by a random seed value and all the relevant test cases in that order, seed=0 means running everything in serial order like we do now by default. In node.js it functions takes its concurrency configuration from the above module unless specifically provided a concurrency: true option.

I imagine the options type interface difference between the node and deno test runners will converge, as both projects work together more. They do get inspired & evolve based on one another so I don't mind the mismatch for now, both configurations should be available in the same options argument.

Thinking more broadly about the strategy, would you mind drawing a comparison between a dream outcome and say what we have today with alternate approaches that run QUnit tests in Node + Browser. What benefits are you most excited about? E.g. something we can make possible and/or much easier.

I'm going to build QUnitX based on this strategy going forward and make it the test runner for my @memoria ORM project, it already runs in node.js and browser environments but soon I'll make some changes to make this shimming/proxying possible, also to make it work/tested on deno.

I'm mostly excited about having one flexible testing API for all environments & make tests completely concurrent/independent by default in JS. This will certainly allow us to deprecate bunch of internal packages in qunit as well eventually.

Testing is currently too hard as you mentioned especially in the frontend and even the big companies don't do enough testing or high quality testing. I see this as a major problem in the software industry now, if we can make it easier then adoption will certainly increase, or at least increase in areas/fields where it really matters. Even new hip alternatives like vitest considers browser mode a second class citizen and lacks many features QUnit already has for years, so this approach should help QUnit adoption as well. We don't need to add any more API than those options arguments to module and test, maybe deprecate more public API eventually.

I'll keep you posted about the @memoria/adapters test suite with the changes once I implement the shimming/proxying!

from qunit.