Which spider to use about zimit HOT 3 CLOSED

@ikreymer @pirate @ivanistheone @tim-moody What kind of alternatives would you recommend to consider?

from zimit.

@kelson42 I think if you have already settled on warc files then you have already decided on the spider.

@ivanistheone strategy of crawling before you can run makes sense to me. I have used httrack, wget, and other tools to scrape the cdc site. The problem is that it is huge (I think it approaches 1TB) and the scrape runs out of resources before finishing. I ended up leaving out all the videos and even the pdfs and other docs.

So I want know what to leave out without downloading the whole site. BasicCrawler crawls the whole site (mostly) but only gets head for everything but html. That way you can size the job before executing it.

I am in the process of expanding BasicCrawler to include more content-types. The output is not a scraped site but a set of json files that can be analyzed before scraping. I looked at the warc extractor some time ago but found the files large and the server quite slow. If you can now convert the output to zims that would be useful. I was never able to make a zim from the cdc download because it was too unruly, so you may find that not all warcs will convert to zims.

from zimit.

The decision has been made to use https://github.com/webrecorder/browsertrix

from zimit.