The unspoken deal that held the web together for twenty years was simple: search engines crawled your content and, in return, sent you visitors. AI crawlers have broken that bargain. They crawl to train models and to answer users directly, often without sending traffic back to the original source. In July 2025, Cloudflare — the intermediary for an enormous slice of internet traffic — decided to step in.

Blocking by default

Cloudflare began blocking AI crawlers by default on new domains joining its network. Instead of having to discover and manually ban each bot, site owners now explicitly decide whether to let AI companies access their content. It is a philosophical shift: from "everything is allowed unless you block it" to "everything is blocked unless you allow it."

What is "pay per crawl"?

The more ambitious move is pay per crawl, launched in beta. The idea is to create a marketplace where publishers can charge AI companies for access to their content. Technically, it builds on an HTTP status code that almost nobody used: 402 Payment Required. When an AI crawler requests a page, the server can respond with a 402 and a set of terms; if the AI company agrees to pay, it gains access.

For the first time, a small publisher has a lever to monetise AI crawling without negotiating individual contracts with each lab.

What it means for publishers

For anyone publishing content, this opens up three choices:

• Allow for free: you bet on visibility in AI responses, even if you get no clicks.
• Block: you protect your content from training and from responses that do not credit you.
• Charge: you try to capture part of the value models generate from your work.

There is no universal right answer. A news outlet, an online shop and a personal blog each have very different incentives.

The reasonable doubts

It is worth approaching this with a critical eye. The system only works if crawlers identify themselves honestly: a bot that spoofs its user-agent or crawls from outside Cloudflare's network slips through the controls entirely. It is also unclear whether the major AI labs will pay at scale, or how pricing will be set. This is an early market move, still evolving — not a closed and settled system.

robots.txt is still your foundation

Pay per crawl is managed from the Cloudflare dashboard, but it does not replace robots.txt — it complements it. Your robots.txt file remains the portable, publicly readable declaration of which bots you accept, regardless of your CDN provider. Before you start charging or blocking, it is worth knowing exactly what your current configuration looks like.

How Spider can help

Deciding whether to allow, block or charge AI bots first requires knowing what they see when they visit your site. Spider.es analyses your domain against the leading AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and many more — and shows you, one by one, whether your robots.txt and response headers let them through. It is the right starting point for making an informed decision before touching your Cloudflare configuration.