Progression indicators about zimit HOT 9 CLOSED

@rgaudin @ikreymer I believe this topic to be touchy to discuss immediatly via Github. Please talk about it in the weekly meeting before commenting here.

from zimit.

Would be good to have a plan here.

from zimit.

Make sure we can still access those easily now that we've split the project into the crawler and this.

from zimit.

The data is accessible and can easily be exposed. Maybe easiest is to output stats to a JSON file, eg.

For example, zimfarm can add:
-v ./job-stats-{id}.json:/output/stats.json

and the crawler can periodically write stats to /output/stats.json

Then, zimfarm can simply display the contents.

from zimit.

@rgaudin @ikreymer AFAIK this is the last ticket to implement to finish the work at the spider level. Would you be able please to secure this ticket is implemented and closed properly soon?

from zimit.

OK, as discussed, I think we can go with a JSON file for tracking the progress. We're discussing this here but this is mostly about modifying the browsertrix crawler

We can start with a very simple format:

{
    "total": 832,
    "progress": 43
}

Total would be the found number of links (crawler's items to process) and progress would be the number of those already crawled.

This could be called crawl-progress.json and be placed on the output folder that was requested (so /output/crawl-progress.json in the zimfarm).

I think it might be good to enable that via some option (that we'd set on zimit). Possibly, we could pass it the expected path.

Next, we'll do something similar with warc2zim and figure out how we'll expose it in zimit but what we need your help we @ikreymer is the crawler part.

from zimit.

Sounds good. At the browsertrix level, I think it would be an option like: --statsOutput /output/crawl-progress.json.
Can just pass that through from zimit.

from zimit.

👍

from zimit.

@ikreymer Hi, any news on this? The feature is need next week to be able to have the progress bar within the Zimit public frontent.

from zimit.