Cloudflare’s Crawler Classification Puts Publishers Between Discovery and Training

Cloudflare’s new purpose-based crawler defaults expose a structural impossibility: you can permit a crawler, but you cannot permit its intentions.

What Happened

Effective today, Cloudflare has introduced a purpose-based crawler classification system and attached a new default to it. The company defines three categories in its own words: Search is “any behavior that collects or indexes your content, so it can answer questions about it later”; Agent is “automated behavior that is acting, usually in real time, on a person’s behalf, to get something done right now”; and Training is “a crawler taking your content to train or fine-tune a model.” On pages that display advertising, the default as of September 15, 2026, blocks Training and Agent crawlers while leaving Search crawlers permitted.

Cloudflare’s own post scopes this change to all new domains onboarding to Cloudflare. Some trade coverage has reported a wider application — to new customers and existing free-tier customers — though that broader reach has not been confirmed by Cloudflare directly and should be treated as reported rather than established. Site owners who wanted no change to the treatment of Training crawlers that also crawl for Search purposes could opt out through their Security settings ahead of the deadline.

The complication Cloudflare names directly is the one that matters most analytically. Googlebot, Applebot, and BingBot are identified as multi-purpose crawlers that combine Search with Training. Under the new default, if a publisher blocks Training, such a crawler is, in Cloudflare’s framing, “blocked according to all of their behaviors” — which means the search function is blocked alongside the training function. Cloudflare’s own July 2026 data showed that mixed-use crawlers blending search, agent use, and training represented more than 36% of crawler activity; that figure is a floor, not a precise count of a static population.

The Structural Read

The taxonomic work Cloudflare has done here is genuinely useful. Naming crawlers by their downstream function — not by the company that operates them — is a more honest framing than the bot-allowlist model that preceded it. But the taxonomy runs into a protocol reality the moment it meets a mixed-use crawler: purpose lives in what the operator subsequently does with the bytes, somewhere else entirely. A publisher can permit a crawler. A publisher cannot permit a crawler’s intentions. For the more than 36% of traffic that is already mixed-use, “allow search, refuse training” is simply not an available option, however sensible it sounds when stated as a preference. The real choice is between being found and being trained on — and it is one choice, not two.

Everything that flows from this is what Cloudflare’s system, operating honestly at the network layer, has to treat as an honour system dressed as a control. The operator on the other side of the fetch decides how the bytes are used. The publisher set a preference. The two things happen at different layers, at different times, with no enforcement mechanism connecting them for mixed-use traffic.

The second structural observation concerns bundling, and it needs to be stated carefully. Operators that run a general search index and a frontier model behind a single crawler identity have made refusal expensive in a way no independent AI company can match. An AI-only crawler presents a publisher with a clean decision carrying a bounded cost: refuse it, and nothing visible is lost. A crawler that also carries discovery makes refusal cost traffic — which for an advertising-supported publisher means revenue. The asymmetry is entirely about who else is in the bundle.

This is a structural consequence of operating both functions from a single crawler identity, not an allegation about anybody’s intentions. Nothing here suggests that Google, Apple, or Microsoft combined these behaviours in order to deter refusal. Nothing here suggests conduct that is improper, anticompetitive, or unlawful. The observation is about the position those firms occupy, not the motive behind it.

The third observation is about where Cloudflare chose to draw the line, and it is the most revealing design decision of the three. The default does not apply to all content. It applies to pages carrying ads. That locates the boundary where the economic injury is — not where the copyright argument is. It converts a slow, abstract, and genuinely unresolved intellectual-property dispute into a concrete revenue-protection feature a publisher can switch on this afternoon. The questions of whether training on published content is lawful, what fair use covers, and how copyright applies to AI training data are contested and remain unsettled; nothing here states any legal conclusion about them. But the design implicitly concedes something worth noticing: content that does not monetise through human attention is not what this particular default defends. This is an advertising problem being solved, not an authorship one.

Three Implications