The rapid progress in artificial intelligence in recent months is partly due to training on vast data sets of text and images, scraped for free from the internet. Although automated web scraping by search engines has been accepted by website owners for decades, the economic shift being brought about by AI has triggered a rethink.
At a basic level, search engines offer an exchange to website owners: let us scrape to compile the information and serve useful results, and we will send traffic …