The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them from “accessing content without permission or compensation,” according to an announcement on Tuesday. With the change, Cloudflare will start asking new domain owners whether they want to allow AI scrapers, and will even let some publishers implement a “Pay Per Crawl” fee.
The Pay Per Crawl program will let publishers set a price for AI scrapers to access their content….
