---
title: AI Crawl Control with Cloudflare WAF
description: Use AI Crawl Control alongside WAF custom rules.
image: https://developers.cloudflare.com/core-services-preview.png
---
> Documentation Index
> Fetch the complete documentation index at: https://developers.cloudflare.com/ai-crawl-control/llms.txt
> Use this file to discover all available pages before exploring further.
[Skip to content](#%5Ftop)
# AI Crawl Control with Cloudflare WAF
AI Crawl Control works alongside other Cloudflare products, such as Cloudflare [Web Application Firewall (WAF)](https://developers.cloudflare.com/waf/). WAF checks incoming web and API requests, and filters undesired traffic based on rules. [WAF custom rules](https://developers.cloudflare.com/waf/custom-rules/) allow you to perform certain actions such as enforcing `robots.txt`.
## Order of precedence
* AI Crawl Control uses WAF custom rules to block the selection of AI crawlers the site owner has decided to block.
* AI Crawl Control's pay per crawl feature takes place after WAF.
graph LR
A[Traffic] --> B[WAF custom rules
AI Crawl Control: Crawler blocks]
B --> C[Cloudflare
Bot Solutions]
C --> D[AI Crawl Control:
Pay Per Crawl]
classDef highlight fill:#F6821F,color:white
For this reason, if you plan on using AI Crawl Control to manage AI crawlers, you may wish to modify your existing WAF custom rules such that it does not affect AI crawlers. This will allow you to manage AI crawlers only from AI Crawl Control, thereby streamlining your workflow.
How AI Crawl Control uses WAF custom rules
When you block AI crawlers via AI Crawl Control (either all or some), you are using **one** WAF custom rule to block those AI crawlers.
If you choose to allow all AI crawlers, AI Crawl Control does not utilize any WAF custom rules.
Depending on the type of account you have, you may have a limited number of WAF custom rules.
## Examples of using WAF vs AI Crawl Control
Consider the following examples.
### Traffic from a restricted country vs pay per crawl
You may have both of the following features enabled:
* [WAF custom rule to block traffic from specific countries](https://developers.cloudflare.com/waf/custom-rules/use-cases/block-traffic-from-specific-countries/)
* AI Crawl Control's [pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers when they request access to your content
Since WAF custom rules are enforced before pay per crawl, traffic (including AI crawlers) from your blocked countries will continue to be blocked, even if they provide the [required headers](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/#1-identify-payment-requirements) for pay per crawl.
### Allowed search engine bots via WAF custom rule vs pay per crawl
You may have both of the following features enabled:
* [WAF custom rule to allow search engine bots](https://developers.cloudflare.com/waf/custom-rules/use-cases/allow-traffic-from-verified-bots/)
* AI Crawl Control's [pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) to charge all AI crawlers when they request access to your content (including search engine bots).
Since custom rules are enforced before pay per crawl:
* Only search engine bots will be able to access your site (enforced by custom rule).
* The search engine bots will then be charged for access to your content (enforced by AI Crawl Control's pay per crawl).
Note
This example only serves to highlight the order of precedence between WAF and AI Crawl Control.
Practically, it may be beneficial to allow well-behaved search engine bots to access your content to ensure your content is indexed.
### Troubleshoot allowed bots
If you have set certain AI crawlers to **Allow** in AI Crawl Control, but they are still being blocked, check for upstream WAF custom rules that may be blocking them. Since the AI Crawl Control rule only includes blocked bots, allowed bots may still be affected by other security rules that execute before the AI Crawl Control rule.
These upstream rules will affect traffic but may not be visible in AI Crawl Control analytics. Review your WAF custom rules to identify and modify any rules that may be blocking AI crawlers you intend to allow.
### Troubleshoot blocked bots
If you have set certain AI crawlers to **Block** in AI Crawl Control, but they are still accessing your content, check for upstream rules that may be bypassing the AI Crawl Control rule. Since the AI Crawl Control rule is added at the end of existing WAF custom rules, the following types of rules may allow bots to bypass the block:
* **Skip rules** that bypass WAF custom rules
* **Redirect rules** that change the request path
* **Transform rules** that modify the request
To ensure blocked bots are properly blocked, move the AI Crawl Control rule to the top of your WAF custom rules, so it executes before other rules.
### Conflict in AI crawler blocking logic
You may have both of the following features enabled:
* A WAF custom rule which blocks all bots.
* AI Crawl Control selection which allows certain AI crawlers.
In this scenario, you have two custom rules, each directing a different logic for handling AI crawlers. To resolve this issue:
* [ New dashboard ](#tab-panel-4338)
* [ Old dashboard ](#tab-panel-4339)
1. In the Cloudflare dashboard, go to the **Security rules** page.
[ Go to **Security rules** ](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules)
2. Filter by _Custom rules_.
3. Identify your custom rule and the AI Crawl Control rule.
4. Drag the rule you wish to prioritize to the top, or modify your custom rule to ensure it does not conflict with your AI Crawl Control configurations.
1. Log in to the [Cloudflare dashboard ↗](https://dash.cloudflare.com/), and select your account and domain.
2. Go to **Security** \> **WAF** \> **Custom rules** tab.
3. Identify your WAF custom rule and the AI Crawl Control rule.
4. Drag the rule you wish to prioritize to the top, or modify your WAF custom rule to ensure it does not conflict with your AI Crawl Control configurations.
## Extending the AI Crawl Control WAF rule
For most use cases, managing crawlers directly in AI Crawl Control is recommended. However, the underlying WAF rule supports additional customization for scenarios the dashboard does not cover.
The AI Crawl Control rule is named **AI Crawl Control** and can be found under **Security** \> **WAF** \> **Custom rules**.
Common additions include:
* Path-based exceptions, such as allowing a blocked crawler to access specific sections of your site by adding an `AND` clause that excludes certain paths
* Extra user agents or detection IDs for crawlers not listed in AI Crawl Control
* Additional expression clauses to restrict blocking to specific hostnames or other request properties
Any additions you make are preserved when you subsequently update crawler actions in AI Crawl Control. If the expression has been modified in a way AI Crawl Control cannot parse, a warning banner will appear on the **Crawlers** page. Select **View rule in WAF** in the banner to inspect or correct the rule.
Warning
Changes made directly to the WAF rule are not reflected back in the AI Crawl Control dashboard. Use the **Crawlers** tab as your primary interface, and reserve direct WAF edits for cases where the dashboard does not cover your requirements.
```json
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-crawl-control/","name":"AI Crawl Control"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-crawl-control/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-crawl-control/configuration/ai-crawl-control-with-waf/","name":"AI Crawl Control with Cloudflare WAF"}}]}
```