Automate the tracking of Dark Visitors about ai.robots.txt HOT 4 OPEN

@cdransf: please could you add a "help wanted" label to this issue? You never know, someone might automate this for us.

from ai.robots.txt.

@glyn added! I don't see a feed on the site, so this may be manual (I'm signed up for their emails).

from ai.robots.txt.

Dark Visitors now offers an API, which may be helpful. It does require a sign up for a token.

from ai.robots.txt.

The API only offers a subset of list so far. e.g. It does not include uncategorized ones. Fortunately, the site itself does not prevent any bots at all at this point.

I managed to add this feature to my own repo with a python script and commit hooks. It caches the response in a sqlite database as well.

If such added dependencies are fine with you, I can create a PR to help this. But I need to understand your preferences in terms of:

1. How often, where, and how it should be executed;
2. What categories of bots should be disallowed by default;
3. Should it modify files other than the robots.txt;

and any other ideas you might have.

Many thanks for this repo!

from ai.robots.txt.