AI Crawler and Agent access policies let you control AI crawler and agent traffic by behavior. You can set actions for Search, Agent, and Training traffic, and customize access for the top Personal Agents.
Cloudflare offers configuration for the most common AI behaviors:
- Search: crawlers that collect or index your content to answer questions about it later.
- Agent: automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents.
- Training: crawlers taking your content to train or fine-tune a model, including mixed-purpose crawlers that are used both for Training and for Search.
Each policy applies to Verified bots with that behavior and matching unverified bots.
Choose an action for each behavior. Available actions are:
| Action | Effect |
|---|---|
| Block | Block matching requests across the zone. |
| Block on pages with ads | Block matching requests on pages where Cloudflare detects ads. |
| Allow (do not block) | Do not block matching requests. |
| Disallow | For Training only, tell crawlers not to access your content through robots.txt. |
robots.txt directives are voluntary. Crawlers can ignore them. When Bot Preference Sync is enabled, Disallow also blocks AI training crawlers at Cloudflare's edge. Mixed-purpose crawlers used for both Search and Training are excluded from this edge block.
To configure these policies for an existing zone:
-
In the Cloudflare dashboard, go to Security > Settings.
Go to Settings ↗ -
Filter by Bot traffic, then select AI Crawler and Agent access policies.
-
Select an action for each behavior. For Agents, you can also granularly customize access for the top Personal Agents.
-
Select Save.
For new domains, Search, Agents, and Training use Allow (do not block) by default. You can change these defaults after onboarding in Security > Settings > AI Crawler and Agent access policies.
Training actions also apply to mixed-purpose crawlers used for both Search and Training.
Personal Agents inherit the Agents policy by default. Changing the Agents policies updates each Personal Agent. You can then choose individual actions again. Cloudflare recommends Allow (do not block) for Personal Agents.
You can also configure Personal Agent policies during zone onboarding.