Create sanity-check Zimfarm image about zimfarm HOT 8 CLOSED

@Popolechien I think I have a ticket for that somewhere... but we need definitly to setup an quality insurance system. The idea is to add this validation step one time the files are uploaded to the warehouse.

@automactic The docker itself should be really is: monitor a directory, check the new files with zim-check, if returns no error, then move the file to make it really available to download. Otherwise "to be defined".

from zimfarm.

@kelson42, the zimfarm warehouse is a SFTP server. It cannot do stuff like monitor dir and test new files.

What might be a good idea is to introduce the concept of staging. SFTP server move files from workers to staging, then a dedicated monitor will kick off testing jobs for new files in staging. After tests passed, move them to production.

from zimfarm.

@kelson42 How are we planning to test zim files? For situation like above, there doesn't seems to be an obvious way to automate the test

from zimfarm.

@automactic We'll need to have a human step in there. For these wikis I'm also thinking of contacting the mods to ask if they'd be ok with us having a simplified landing page (like we already do on a few Wikipedia).

from zimfarm.

@Popolechien It seems to me quite unrealistic, because of human resource bottleneck, to have a human review of many thousands of new ZIM a month. On the top of that, this is something which can be automated, so for us, probably something we could/should do

@automactic I do not have talked about the "warehouse" container. IMO the warehouse container is fine to receipt the ZIM files from the distributed workers. Just take care that we have a way to easily know if a file is fully uploaded or not on the fs. We need that because one time a file will be uploaded, the "sanity check" container (still to build) will run zim-check against that file and then move it to final destination. To conclude the warehouse and the sanity-check container will share a Docker volume.

from zimfarm.

@kelson42 Of course not every ZIM, but just the new ones for their very first deployment. I don't know how many new contents we publish yearly, but I'd be surprised at this stage that it's more than a handful.

from zimfarm.

@Popolechien OK, then I do that already.

from zimfarm.

This will be handled outside the zimfarm project. See kiwix/container-images#30

from zimfarm.