Cloudflare Orders AI Companies to Separate Crawlers by September 15 or Be Blocked
Cloudflare has given AI companies until September 15, 2026 to separate web crawlers — search bots separately from bots for training AI and agents. Those who fail to comply will be blocked by default on publisher websites. The policy aims to force AI companies to pay for content rather than take it for free.
AI-processed from TechCrunch; edited by Hamidun News
Cloudflare on July 1, 2026 introduced new requirements for AI companies: they are required to separate their web crawlers — search bots and bots for training AI models must be different entities — by September 15, 2026.
Why is Cloudflare separating AI company crawlers?
The goal is to force AI companies to pay for content instead of using publisher data without compensation.
What will happen to AI companies that fail to meet the requirement?
Cloudflare infrastructure will begin blocking them by default on publisher sites.
Key Questions Answered
Why is Cloudflare separating AI company crawlers?
The goal is to force AI companies to pay for content instead of using publisher data without compensation.
What will happen to AI companies that fail to meet the requirement?
Cloudflare infrastructure will begin blocking them by default on publisher sites.
Delegate Deep Research, Coding & Multi-Step Workflows to Nomad Agent
Deploy autonomous task execution with Nomad Agent (agent.nomadnet.ai) — automated web research, data synthesis, code generation, and MCP tool orchestration.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.