- countio: The wrapper struct to enable byte counting for std::io::Read and std::io::Write and its asynchronous variants from futures and tokio.
- robotxt: The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
- sitemapo: The implementation of the Sitemap (or URL inclusion) protocol with the support of txt, xml formats and video, image, and news extensions.
spire-rs / kit Goto Github PK
View Code? Open in Web Editor NEW๐ Libraries for retrieving and parsing robots.txt & sitemaps.
License: MIT License