---
title: Robots
---

[Skip to content](#%5Ftop) 

[API Reference](https://developers.cloudflare.com/api/python)

[AI Audit](https://developers.cloudflare.com/api/python/resources/ai%5Faudit)

Copy Markdown

Open in **Claude**

Open in **ChatGPT**

Open in **Cursor**

---

**Copy Markdown**

**View as Markdown**

# Robots

##### [Get robots.txt rules](https://developers.cloudflare.com/api/python/resources/ai%5Faudit/subresources/robots/methods/get)

ai\_audit.robots.get(RobotGetParams\*\*kwargs)  \-> [RobotGetResponse](https://developers.cloudflare.com/api/python/resources/ai%5Faudit#%28resource%29%20ai%5Faudit.robots%20%3E%20%28model%29%20robot%5Fget%5Fresponse%20%3E%20%28schema%29)

GET/zones/{zone\_id}/ai-audit/robots

##### [Bulk get robots.txt rules](https://developers.cloudflare.com/api/python/resources/ai%5Faudit/subresources/robots/methods/bulk%5Fget)

ai\_audit.robots.bulk\_get(RobotBulkGetParams\*\*kwargs)  \-> [RobotBulkGetResponse](https://developers.cloudflare.com/api/python/resources/ai%5Faudit#%28resource%29%20ai%5Faudit.robots%20%3E%20%28model%29%20robot%5Fbulk%5Fget%5Fresponse%20%3E%20%28schema%29)

POST/zones/{zone\_id}/ai-audit/robots/bulk

##### ModelsExpand Collapse 

class RobotGetResponse: …

Parsed robots.txt rules for a single domain.

user\_agents: Dict\[str, UserAgents\]

Map of user-agent string to its parsed rules.

allow: List\[str\]

List of allowed path patterns.

disallow: List\[str\]

List of disallowed path patterns.

content\_signals: Optional\[UserAgentsContentSignals\]

Content signal directives from robots.txt.

ai\_input: Optional\[Literal\["yes", "no"\]\]

Whether AI input usage is permitted.

One of the following:

"yes"

"no"

ai\_train: Optional\[Literal\["yes", "no"\]\]

Whether AI training is permitted.

One of the following:

"yes"

"no"

search: Optional\[Literal\["yes", "no"\]\]

Whether search indexing is permitted.

One of the following:

"yes"

"no"

crawl\_delay: Optional\[float\]

Crawl delay in seconds.

sitemaps: Optional\[List\[str\]\]

List of sitemap URLs found in robots.txt.

status: Optional\[int\]

HTTP status code from fetching the robots.txt file.

Dict\[str, RobotBulkGetResponseItem\]

Map of hostname to parsed robots.txt rules.

user\_agents: Dict\[str, RobotBulkGetResponseItemUserAgents\]

Map of user-agent string to its parsed rules.

allow: List\[str\]

List of allowed path patterns.

disallow: List\[str\]

List of disallowed path patterns.

content\_signals: Optional\[RobotBulkGetResponseItemUserAgentsContentSignals\]

Content signal directives from robots.txt.

ai\_input: Optional\[Literal\["yes", "no"\]\]

Whether AI input usage is permitted.

One of the following:

"yes"

"no"

ai\_train: Optional\[Literal\["yes", "no"\]\]

Whether AI training is permitted.

One of the following:

"yes"

"no"

search: Optional\[Literal\["yes", "no"\]\]

Whether search indexing is permitted.

One of the following:

"yes"

"no"

crawl\_delay: Optional\[float\]

Crawl delay in seconds.

sitemaps: Optional\[List\[str\]\]

List of sitemap URLs found in robots.txt.

status: Optional\[int\]

HTTP status code from fetching the robots.txt file.