Not all images in srcset are scraped about browsertrix-crawler HOT 6 CLOSED

One thing to try is to update the autofetch script to cause an additional wait as soon as any autofetching is started, similar to videos.. Need to test if that would be sufficient..
I dont know if networkidle0 is always the best option, I think I've seen some pages work better with waiting for load event..

from browsertrix-crawler.

It turns out its possible to wait for both networkidle0 and load events simply by passing an array to waitUntil.
The command-line option defaults to load,networkidle0 and I think that should do it for this use case.
Tested with the example linked and the srcsets seem to be getting captured.

from browsertrix-crawler.

Thanks for testing this in the browser! That does seem very strange, perhaps there is a timing issue and it moves to next page too quickly before they can be fetched.. There is not anything waiting to ensure the autofetcher is fully finished..

Could you try running with --waitUntil networkidle0 to see if that makes a difference? If not, maybe need to have additional waiting options.

from browsertrix-crawler.

Ah, that's it! Worked with networkidle0.

from browsertrix-crawler.

What should be done here? Should this working configuration be the default one?

from browsertrix-crawler.

@ikreymer Thank you. Would that be possible to make it available soon in a release?

from browsertrix-crawler.