bw-tom / robots Goto Github PK
View Code? Open in Web Editor NEWThis project forked from brandwatchltd/robots
Support for Robots Exclusion Protocol, including parsing and matching against robot.txt directives.
License: BSD 3-Clause "New" or "Revised" License