Add pause, stop, resume about abot HOT 8 CLOSED

A few ideas...

var pausedCrawler = manager.Pause(crawler);
manager.Resume(pausedCrawler);

from abot.

Made all classes in Abot project Serializable so others may implement a pause/resume

from abot.

More talk about this here...
https://groups.google.com/forum/#!topic/abot-web-crawler/KiYhgjaESNU

from abot.

Hi sjdirect, if you don't mind I ask, I remember you have a file based Url and Crawl repository in Google repository before, or is it my memory corrupted or fragmented. I hope I understand this issue correctly, but as long as Abot resume previous unfinished task, that will satisfy this ticket? i.e. stop / resume and pause is consider as stop. I am working on a simple Mongodb based Scheduler for Abot, I will upload to Github later, but then again, if you don't mind it is in F#. Thanks.

from abot.

Yes there use to be a file based crawl repo but it was overly complex and under performant. A lightweight version of it MAY be created if I decide that is the best way to pause the crawl.

from abot.

Hi Steven, I added a new repository using Redis as scheduler store (I thought about using MongoDB before, but Redis gives much better performance as it runs in memory). That should allow the crawler to start and stop without losing track of previous progress. Hope this will help. Cheers. https://github.com/mnta/Abot.Redis.Scheduler

from abot.

Thanks, i'll take a look at this when I cross that bridge

On Sat, Oct 17, 2015 at 5:18 AM, mnta [email protected] wrote:

Hi Steven, I added a new repository using Redis as scheduler store (I
thought about using MongoDB before, but Redis gives much better performance
as it runs in memory). That should allow the crawler to start and stop
without losing track of previous progress. Hope this will help. Cheers.
https://github.com/mnta/Abot.Redis.Scheduler

—
Reply to this email directly or view it on GitHub
#45 (comment).

from abot.

You can use the Pause/Resume feature of AbotX. AbotX build on top of abot. Closing issue.

from abot.