AI companies are working on a new way to help publishers block certain web crawlers that collect online data, according to the provided update. The move is aimed at giving website owners more control over how their content is accessed and used.
At present, publishers must use multiple blocking commands to stop different bots from crawling their sites for data collection. The process can be cumbersome because each bot may need to be blocked separately.
The proposed approach is expected to simplify that process by offering a more unified way to restrict scraping activity. No further details were provided on when the tool will be available or which companies are involved.
