ai-robots-txt / ai.robots.txt Goto Github PK

We could prime some questions in a FAQ from the hacker news discussion. The main one is along the lines of "Why would an AI web crawler respect robots.txt?"

If we wanted to be brave, we could enable a wiki for this repo!

img2dataset is software that spiders sites for AI training. It's not run by a specific company so hits can come from anywhere.

It claims to honor robots.txt with the "img2dataset" user agent token, and X-Robots-Tag or HTML <meta> directives "noai" and "noimageai".

tl;dr: independent body suggests adding new file to site root directory to ask organizations that use AI for text and data mining (TDM) to NOT use site data for training purposes.

From their FAQ:

An ai.txt file is a simple text file placed in the root directory (or .well-known/) of your website that communicates with data miners. It provides instructions on whether the text and media files hosted on your domain can be used to train commercial AI models.

An example file I created:

# Spawning AI
# Prevent datasets from using the following file types

User-Agent: *
Disallow: /
Disallow: *

PS: Thanks for this project 💖

Resources:

• ai.txt: A new way for websites to set permissions for AI
• About Spawning AI

The idea is to scrape the content of Dark Visitors using a bot and generate PRs for this project. A bit like dependabot.

Help populate the table-of-bot-metrics.md to clarify bot activity.