Crawler SIGTERM by itself while running about browsertrix-crawler HOT 4 CLOSED

It's hard to say exactly, as its not easy to repro, but I suspect its a running-out-of-memory issue (probably RAM). At least that's what searching for 'error waiting for container: EOF' seems to suggest.
You could try running with less workers to see if it happens again? And then try docker inspect on the container to see if there is more info on why it was SIGTERMed.

The next release will make it a bit easier to restart the crawl. With the 0.5.0, the container will automatically try to save the crawl config + current state of the crawl to the crawls directory such that it can be restarted again. You could try running with that version, currently on the: https://github.com/webrecorder/browsertrix-crawler/tree/save-state branch.

from browsertrix-crawler.

Thanks. I am running jobs using 8 workers. Is that considered to be a lot / too many, or is it within the acceptable range for browsertrix (and therefore, should not crash)?

from browsertrix-crawler.

I can report that I've seen a strange SIGTERM as well, on an Ubuntu snap-based Docker deployment. Of course, when I added extra logging and restarted the daemon, it didn't happen! So I guess one possibility is to try restarting the docker daemon if it happens again? Will update if I find more info..

from browsertrix-crawler.

Haven't been able to repro since, but I suspect it was oomkilled. We now have Period State Saving which can help if this happens in the future. Closing, since exact repro is uncertain and not sure its fully solvable if OOM.

Comment if there is a specific repro.

from browsertrix-crawler.