jaydenivanovic / robotstxtrun Goto Github PK
View Code? Open in Web Editor NEWPackages up Googles Robots.txt Parser and Matcher Library (https://github.com/google/robotstxt) into a docker container, and adds a small bash wrapper to assert URLs are allowed or disallowed via a JSON object. Useful for running assertions locally in a dev environment.