catchpoint / webpagetest.crawler Goto Github PK

This is already super helpful, but I think it opens up more possibilities and probably better overall ergonomics if this was available to run like this (for example):

index.js -level 3 -url_limit 10 -key #### -whiteListUrls whiteListUrls.txt

A few of those could take either config files or just params (like, whiteListUrls could potentially be just a list of urls passed directly, and I could see a config param to pass in a config file versus using level, url_limit, etc.

The end goal though is that when this is set, we could wrap this up as an npm module so people could eventually have something like:

npx wpt-crawler -level 3 .....

Which would make it easier to drop into other workflows.

What do you think, @abdulsuhail?

whiteListUrls doesn't quite match what you're doing there and implies that the ability to black list other URLs. Maybe just call it urls or startingUrls?

Right now, looks like once 5 URLs are submitted, sleep is used to hold off for a bit and reset the submission count, presumably to avoid having too many concurrent tests running. https://github.com/abdulsuhail/wpt-crawler/blob/main/index.js#L100

It's a decent rough gauge, but even that doesn't ensure that any of the tests have completed before things pick back up again.

Could this be changed to watch for whenever a test is completed? Ex: submit 5, wait, a test completes, so another gets submitted, etc.