---
title: Robots
---

[Skip to content](#%5Ftop) 

[API Reference](https://developers.cloudflare.com/api)

[AI Audit](https://developers.cloudflare.com/api/resources/ai%5Faudit)

Copy Markdown

Open in **Claude**

Open in **ChatGPT**

Open in **Cursor**

---

**Copy Markdown**

**View as Markdown**

# Robots

##### [Get robots.txt rules](https://developers.cloudflare.com/api/resources/ai%5Faudit/subresources/robots/methods/get)

GET/zones/{zone\_id}/ai-audit/robots

##### [Bulk get robots.txt rules](https://developers.cloudflare.com/api/resources/ai%5Faudit/subresources/robots/methods/bulk%5Fget)

POST/zones/{zone\_id}/ai-audit/robots/bulk

##### ModelsExpand Collapse 

RobotGetResponse object { userAgents, sitemaps, status } 

Parsed robots.txt rules for a single domain.

userAgents: map\[object { allow, disallow, contentSignals, crawlDelay } \]

Map of user-agent string to its parsed rules.

allow: array of string

List of allowed path patterns.

disallow: array of string

List of disallowed path patterns.

contentSignals: optional object { "ai-input", "ai-train", search } 

Content signal directives from robots.txt.

"ai-input": optional "yes" or "no"

Whether AI input usage is permitted.

One of the following:

"yes"

"no"

"ai-train": optional "yes" or "no"

Whether AI training is permitted.

One of the following:

"yes"

"no"

search: optional "yes" or "no"

Whether search indexing is permitted.

One of the following:

"yes"

"no"

crawlDelay: optional number

Crawl delay in seconds.

sitemaps: optional array of string

List of sitemap URLs found in robots.txt.

status: optional number

HTTP status code from fetching the robots.txt file.

RobotBulkGetResponse \= map\[object { userAgents, sitemaps, status } \]

Map of hostname to parsed robots.txt rules.

userAgents: map\[object { allow, disallow, contentSignals, crawlDelay } \]

Map of user-agent string to its parsed rules.

allow: array of string

List of allowed path patterns.

disallow: array of string

List of disallowed path patterns.

contentSignals: optional object { "ai-input", "ai-train", search } 

Content signal directives from robots.txt.

"ai-input": optional "yes" or "no"

Whether AI input usage is permitted.

One of the following:

"yes"

"no"

"ai-train": optional "yes" or "no"

Whether AI training is permitted.

One of the following:

"yes"

"no"

search: optional "yes" or "no"

Whether search indexing is permitted.

One of the following:

"yes"

"no"

crawlDelay: optional number

Crawl delay in seconds.

sitemaps: optional array of string

List of sitemap URLs found in robots.txt.

status: optional number

HTTP status code from fetching the robots.txt file.