lichao-mobanche / robotstxt Goto Github PK
View Code? Open in Web Editor NEWThis project forked from google/robotstxt
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
License: Apache License 2.0