modernweb-dev / web Goto Github PK

We support multiple browsers launched by a single browser launcher, but not multiple browser launchers.

We should handle cases where test code reloads the page (for example by setting window.location).

We should into the DX of authoring tests in TS:

• TSC + WTR setup
• Compile TS in EDS (perhaps using sucrase?)
• Source mapped stack traces
• Source mapped click to test file

mocha/mocha.js is pretty big and it has a lot of comments etc. we should look into whether we can shake off some kbs and bundle it with the package. This way it is just one request, and we strip off all the comments etc.

We should allow setting the protocol (http, https) from the config.

Local chrome launcher takes a long time to startup. Maybe we can use some flags to improve it, or file an issue.

It would be great if you don't need to restart the test runner if you add/remove test files.

We would need to watch the file system based on the globs provided, and carefully add/remove tests from the test runner.

I tried to add branch protection, but it prevents the CI from pushing to github. We should figure out how we can have both.

When running in coverage mode there is a noticeable slowdown. We need to check if we can exclude files that don't need coverage from babel.

When using task runners the terminal is not interactive. We should still allow watch mode in these cases.

When changesets updated the package.json, it reformats the package json which conflicts with prettier.

When rerunning tests we clear the terminal and then quickly fill it again. This causes a brief flicker, we're basically a bit too fast. We should look into whether adding an artificial delay can help.

The test runner has a lot of dependencies. To make the package lighter to install and use, we should bundle it for publishing. We will need to review if there are some packages which we don't want to bundle.

A lot of rollup packages take this approach as well.

When using puppeteer to close a browser it takes around 30sec for it to unwind and exhaust the event loop. We don't notice this in the CLI, as we kill the process in manually. But this is noticeable in the tests for the puppeteer and chrome launcher.

We should have an auto generated package listing in the main readme, showing the latest versions.

In a non interactive terminal (such as a CI or in lerna) we still keep logging progress. We should log less to the terminal.

Chrome: |███████████████████████████▏  | 104/114 test files | 1312 passed, 20 failed

Running tests...
 
Chrome: |███████████████████████████▎  | 105/114 test files | 1341 passed, 21 failed
 
Running tests...
 
Chrome: |███████████████████████████▍  | 106/114 test files | 1391 passed, 21 failed
 
Running tests...
  
Chrome: |████████████████████████████▏ | 107/114 test files | 1392 passed, 21 failed
 
Running tests...
  
Chrome: |████████████████████████████▎ | 108/114 test files | 1394 passed, 21 failed

Running tests...
 
Chrome: |████████████████████████████▎ | 109/114 test files | 1395 passed, 21 failed
  
Running tests...
  
Chrome: |████████████████████████████▍ | 110/114 test files | 1396 passed, 21 failed
 
Running tests...
  
Chrome: |█████████████████████████████▏| 111/114 test files | 1397 passed, 21 failed
  
Running tests...
  
Chrome: |█████████████████████████████▎| 112/114 test files | 1403 passed, 21 failed
  
Running tests...
  
Chrome: |█████████████████████████████▎| 113/114 test files | 1411 passed, 21 failed
 
Running tests...
  
Chrome: |██████████████████████████████| 114/114 test files | 1426 passed, 21 failed
  
Finished running tests in 15s with 21 tests failed.

I had to add @types/parse5 because it is missing on a package from open-wc:

node_modules/polyfills-loader/index.d.ts:20:49 - error TS7016: Could not find a declaration file for module 'parse5'. '/Users/ri04pi/Workspace/code/mine/web/node_modules/parse5/lib/index.js' implicitly has an 'any' type.
  Try `npm install @types/parse5` if it exists or add a new declaration (.d.ts) file containing `declare module 'parse5';`

20 export const getScriptFileType: (script: import("parse5").Node) => import("./src/types").FileType;
                                                   ~~~~~~~~


Found 1 error.

We should allow connecting any custom to the browser runner, and reporting the results using the user agent string.

allow configuring launch options

When logging objects to the console we just do a JSON.stringify which loses a lot of information. We should look into a better way of serialization things like DOM elements, because right now they just turn into {}.

Preferably using an existing library. We may be able to take some inspiration from how storybook does this to send data between iframes.

We should allow using firefox in the browsers flag for puppeteer

Syntax errors are now logged by es-dev-server, so they are not deduped cross browsers. We should pull this logic into WTR and dedupe them.

When the test framework resolves to a file outside the root directory, we should throw an error.

We used to have this logic in es-dev-server, but I'm not seeing it happen. We should investigate.

Through the config, but also with flags.

Right now when you set the test coverage configuration, it always runs test coverage. We should allow setting the coverage thresholds without actually doing test coverage. We can do this by splitting the options between coverage: true and coverageThreshold or something like that.

When focusing a test file, coverage is printed multiple times

Add documentation on common test requirements:

• Test fixtures
• Stubbing/spying
• Mocks
• Mocking network calls
• Mocking es modules

Ticked boxes are added.

I'm not entirely certain yet, but I think I'd like to rename test-runner-framework to test-runner-lib or test-runner-browser-lib.

When either the test framework implementation or generic core can't be loaded we don't log anything to the terminal.

We should split the config into a user config, with a lot of optional values, and a parsed config with a lot of required values. This will make it easier on both sides to use safe types.

Since the 0.5.2 I have this error:

(without specific conf, it's ok on 0.5.1)

And thx for this tool.

When you are using some kind of build step (ex. TS), errors will report the compiled JS file location. It would be great if we can support source maps in the error location.

We should allow configuring mocha, for example to configure TDD.

We should allow setting an html file as input file. This will just open that HTML file in the browser, and expect it to ping back the results.

This is useful for being in control of everything that happens in the browser, and for special tests that require the page to be set up in a specific way (for example polyfills).

Everywhere we catch an error, we should check if it's actually an object. Some code might throw strings or undefined.

See also LarsDenBakker/web-test-runner#54

We have a lovely config option in the CLI, and it is ignored. When config is specified, we should read the config from that path.

Fixes #76

It could theoretically happen that a file tracked by es-dev-server changes and we don't know about it. In that case, we should just rerun all the tests.

Istanbul can generate a HTML test report. We should add this, and print a clickable link to the report if your coverage fails.

The test runner puts a lot of load on es-dev-server. We should investigate if there are slow parts, and improve them.

We need to add support for the help command. It's tricky because both test-runner and test-running-cli do things with the command line args. We need to make sure to handle both.