---
title: Bulk get robots.txt rules
---

[Skip to content](#%5Ftop) 

[API Reference](https://developers.cloudflare.com/api)

[AI Audit](https://developers.cloudflare.com/api/resources/ai%5Faudit)

[Robots](https://developers.cloudflare.com/api/resources/ai%5Faudit/subresources/robots)

Copy Markdown

Open in **Claude**

Open in **ChatGPT**

Open in **Cursor**

---

**Copy Markdown**

**View as Markdown**

# Bulk get robots.txt rules

POST/zones/{zone\_id}/ai-audit/robots/bulk

Fetches and parses robots.txt files for multiple domains within a zone in a single request. Each domain must belong to the specified zone. Results are keyed by hostname.

##### Security

API Token

The preferred authorization scheme for interacting with the Cloudflare API. [Create a token](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/).

**Example:**`Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY`

API Email + API Key

The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

**Example:**`X-Auth-Email: user@example.com`

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

**Example:**`X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194`

##### Accepted Permissions (at least one required)

`Zone Settings Write` `Zone Settings Read`

##### Path ParametersExpand Collapse 

zone\_id: string

##### Body ParametersJSONExpand Collapse 

body: array of string

Array of domain hostnames to fetch robots.txt for. Each domain must end with the zone name. Maximum 25 domains per request.

##### ReturnsExpand Collapse 

errors: array of [ResponseInfo](https://developers.cloudflare.com/api/resources/$shared#%28resource%29%20%24shared%20%3E%20%28model%29%20response%5Finfo%20%3E%20%28schema%29) { code, message, documentation\_url, source } 

code: number

minimum1000

message: string

documentation\_url: optional string

source: optional object { pointer } 

pointer: optional string

messages: array of [ResponseInfo](https://developers.cloudflare.com/api/resources/$shared#%28resource%29%20%24shared%20%3E%20%28model%29%20response%5Finfo%20%3E%20%28schema%29) { code, message, documentation\_url, source } 

code: number

minimum1000

message: string

documentation\_url: optional string

source: optional object { pointer } 

pointer: optional string

success: boolean

result: optional map\[object { userAgents, sitemaps, status } \]

Map of hostname to parsed robots.txt rules.

userAgents: map\[object { allow, disallow, contentSignals, crawlDelay } \]

Map of user-agent string to its parsed rules.

allow: array of string

List of allowed path patterns.

disallow: array of string

List of disallowed path patterns.

contentSignals: optional object { "ai-input", "ai-train", search } 

Content signal directives from robots.txt.

"ai-input": optional "yes" or "no"

Whether AI input usage is permitted.

One of the following:

"yes"

"no"

"ai-train": optional "yes" or "no"

Whether AI training is permitted.

One of the following:

"yes"

"no"

search: optional "yes" or "no"

Whether search indexing is permitted.

One of the following:

"yes"

"no"

crawlDelay: optional number

Crawl delay in seconds.

sitemaps: optional array of string

List of sitemap URLs found in robots.txt.

status: optional number

HTTP status code from fetching the robots.txt file.

### Bulk get robots.txt rules

HTTP

HTTPHTTP

TypeScriptTypeScript

PythonPython

GoGo

TerraformTerraform

```
curl https://api.cloudflare.com/client/v4/zones/$ZONE_ID/ai-audit/robots/bulk \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
    -d '[
          "example.com",
          "blog.example.com"
        ]'
```

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}
```

##### Returns Examples

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}
```