---
title: Get robots.txt rules
---

[Skip to content](#%5Ftop) 

[API Reference](https://developers.cloudflare.com/api/python)

[AI Audit](https://developers.cloudflare.com/api/python/resources/ai%5Faudit)

[Robots](https://developers.cloudflare.com/api/python/resources/ai%5Faudit/subresources/robots)

Copy Markdown

Open in **Claude**

Open in **ChatGPT**

Open in **Cursor**

---

**Copy Markdown**

**View as Markdown**

# Get robots.txt rules

ai\_audit.robots.get(RobotGetParams\*\*kwargs)  \-> [RobotGetResponse](https://developers.cloudflare.com/api/python/resources/ai%5Faudit#%28resource%29%20ai%5Faudit.robots%20%3E%20%28model%29%20robot%5Fget%5Fresponse%20%3E%20%28schema%29)

GET/zones/{zone\_id}/ai-audit/robots

Fetches and parses the robots.txt file for a zone or a specific subdomain within the zone. Returns parsed user-agent rules, content signals, and sitemaps.

##### Security

API Token

The preferred authorization scheme for interacting with the Cloudflare API. [Create a token](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/).

**Example:**`Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY`

API Email + API Key

The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

**Example:**`X-Auth-Email: user@example.com`

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

**Example:**`X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194`

##### Accepted Permissions (at least one required)

`Zone Settings Write` `Zone Settings Read`

##### ParametersExpand Collapse 

zone\_id: str

subdomain: Optional\[str\]

Optional subdomain to fetch robots.txt for. If omitted, fetches robots.txt for the zone apex domain.

##### ReturnsExpand Collapse 

class RobotGetResponse: …

Parsed robots.txt rules for a single domain.

user\_agents: Dict\[str, UserAgents\]

Map of user-agent string to its parsed rules.

allow: List\[str\]

List of allowed path patterns.

disallow: List\[str\]

List of disallowed path patterns.

content\_signals: Optional\[UserAgentsContentSignals\]

Content signal directives from robots.txt.

ai\_input: Optional\[Literal\["yes", "no"\]\]

Whether AI input usage is permitted.

One of the following:

"yes"

"no"

ai\_train: Optional\[Literal\["yes", "no"\]\]

Whether AI training is permitted.

One of the following:

"yes"

"no"

search: Optional\[Literal\["yes", "no"\]\]

Whether search indexing is permitted.

One of the following:

"yes"

"no"

crawl\_delay: Optional\[float\]

Crawl delay in seconds.

sitemaps: Optional\[List\[str\]\]

List of sitemap URLs found in robots.txt.

status: Optional\[int\]

HTTP status code from fetching the robots.txt file.

### Get robots.txt rules

Python

HTTPHTTP

TypeScriptTypeScript

PythonPython

GoGo

TerraformTerraform

```
import os
from cloudflare import Cloudflare

client = Cloudflare(
    api_token=os.environ.get("CLOUDFLARE_API_TOKEN"),  # This is the default and can be omitted
)
robot = client.ai_audit.robots.get(
    zone_id="zone_id",
)
print(robot.user_agents)
```

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "userAgents": {
      "foo": {
        "allow": [
          "string"
        ],
        "disallow": [
          "string"
        ],
        "contentSignals": {
          "ai-input": "yes",
          "ai-train": "yes",
          "search": "yes"
        },
        "crawlDelay": 0
      }
    },
    "sitemaps": [
      "string"
    ],
    "status": 0
  }
}
```

##### Returns Examples

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "userAgents": {
      "foo": {
        "allow": [
          "string"
        ],
        "disallow": [
          "string"
        ],
        "contentSignals": {
          "ai-input": "yes",
          "ai-train": "yes",
          "search": "yes"
        },
        "crawlDelay": 0
      }
    },
    "sitemaps": [
      "string"
    ],
    "status": 0
  }
}
```