Google is proposing that robots.txt become a standard requirement for all AI providers. The move is intended to address the fact that many AI crawlers currently disregard the instructions published in websites’ robots.txt files.
The file is commonly used by website operators to communicate which areas of a site automated crawlers may access. According to the available information, Google’s proposal would seek to establish a more consistent approach among AI providers by making compliance with robots.txt a compulsory standard.
The proposal reflects growing concern among website operators about how AI crawlers access online content. The source does not provide further details about the proposed rules, their implementation or a timetable for adoption.
Sources: Golem.de.