---
title: Robots TXT
---

[Skip to content](#%5Ftop) 

[API Reference](https://developers.cloudflare.com/api/typescript)

[Radar](https://developers.cloudflare.com/api/typescript/resources/radar)

Copy Markdown

Open in **Claude**

Open in **ChatGPT**

Open in **Cursor**

---

**Copy Markdown**

**View as Markdown**

# Robots TXT

#### Robots TXTTop

##### [Get top domain categories by robots.txt files parsed](https://developers.cloudflare.com/api/typescript/resources/radar/subresources/robots%5Ftxt/subresources/top/methods/domain%5Fcategories)

client.radar.robotsTXT.top.domainCategories(TopDomainCategoriesParams { date, format, limit, 2 more } query?, RequestOptionsoptions?): [TopDomainCategoriesResponse](https://developers.cloudflare.com/api/typescript/resources/radar#%28resource%29%20radar.robots%5Ftxt.top%20%3E%20%28model%29%20top%5Fdomain%5Fcategories%5Fresponse%20%3E%20%28schema%29) { meta, top\_0 } 

GET/radar/robots\_txt/top/domain\_categories

##### ModelsExpand Collapse 

TopDomainCategoriesResponse { meta, top\_0 } 

meta: Meta { confidenceInfo, dateRange, lastUpdated, 2 more } 

Metadata for the results.

confidenceInfo: ConfidenceInfo | null

annotations: Array<Annotation\>

dataSource: "ALL" | "AI\_BOTS" | "AI\_GATEWAY" | 22 more

Data source for annotations.

One of the following:

"ALL"

"AI\_BOTS"

"AI\_GATEWAY"

"BGP"

"BOTS"

"CONNECTION\_ANOMALY"

"CT"

"DNS"

"DNS\_MAGNITUDE"

"DNS\_AS112"

"DOS"

"EMAIL\_ROUTING"

"EMAIL\_SECURITY"

"FW"

"FW\_PG"

"HTTP"

"HTTP\_CONTROL"

"HTTP\_CRAWLER\_REFERER"

"HTTP\_ORIGINS"

"IQI"

"LEAKED\_CREDENTIALS"

"NET"

"ROBOTS\_TXT"

"SPEED"

"WORKERS\_AI"

description: string

endDate: string

formatdate-time

eventType: "GENERAL" | "OUTAGE" | "PARTIAL\_PROJECTION" | 2 more

Event type for annotations.

One of the following:

"GENERAL"

"OUTAGE"

"PARTIAL\_PROJECTION"

"PIPELINE"

"TRAFFIC\_ANOMALY"

isInstantaneous: boolean

Whether event is a single point in time or a time range.

linkedUrl: string

formaturi

startDate: string

formatdate-time

tags?: Array<string\>

level: number

Provides an indication of how much confidence Cloudflare has in the data.

dateRange: Array<DateRange\>

endTime: string

Adjusted end of date range.

formatdate-time

startTime: string

Adjusted start of date range.

formatdate-time

lastUpdated: string

Timestamp of the last dataset update.

formatdate-time

normalization: "PERCENTAGE" | "MIN0\_MAX" | "MIN\_MAX" | 5 more

Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

One of the following:

"PERCENTAGE"

"MIN0\_MAX"

"MIN\_MAX"

"RAW\_VALUES"

"PERCENTAGE\_CHANGE"

"ROLLING\_AVERAGE"

"OVERLAPPED\_PERCENTAGE"

"RATIO"

units: Array<Unit\>

Measurement units for the results.

name: string

value: string

top\_0: Array<Top0\>

name: string

value: number

#### Robots TXTTopUser Agents

##### [Get top user agents on robots.txt files](https://developers.cloudflare.com/api/typescript/resources/radar/subresources/robots%5Ftxt/subresources/top/subresources/user%5Fagents/methods/directive)

client.radar.robotsTXT.top.userAgents.directive(UserAgentDirectiveParams { date, directive, domainCategory, 4 more } query?, RequestOptionsoptions?): [UserAgentDirectiveResponse](https://developers.cloudflare.com/api/typescript/resources/radar#%28resource%29%20radar.robots%5Ftxt.top.user%5Fagents%20%3E%20%28model%29%20user%5Fagent%5Fdirective%5Fresponse%20%3E%20%28schema%29) { meta, top\_0 } 

GET/radar/robots\_txt/top/user\_agents/directive

##### ModelsExpand Collapse 

UserAgentDirectiveResponse { meta, top\_0 } 

meta: Meta { confidenceInfo, dateRange, lastUpdated, 2 more } 

Metadata for the results.

confidenceInfo: ConfidenceInfo | null

annotations: Array<Annotation\>

dataSource: "ALL" | "AI\_BOTS" | "AI\_GATEWAY" | 22 more

Data source for annotations.

One of the following:

"ALL"

"AI\_BOTS"

"AI\_GATEWAY"

"BGP"

"BOTS"

"CONNECTION\_ANOMALY"

"CT"

"DNS"

"DNS\_MAGNITUDE"

"DNS\_AS112"

"DOS"

"EMAIL\_ROUTING"

"EMAIL\_SECURITY"

"FW"

"FW\_PG"

"HTTP"

"HTTP\_CONTROL"

"HTTP\_CRAWLER\_REFERER"

"HTTP\_ORIGINS"

"IQI"

"LEAKED\_CREDENTIALS"

"NET"

"ROBOTS\_TXT"

"SPEED"

"WORKERS\_AI"

description: string

endDate: string

formatdate-time

eventType: "GENERAL" | "OUTAGE" | "PARTIAL\_PROJECTION" | 2 more

Event type for annotations.

One of the following:

"GENERAL"

"OUTAGE"

"PARTIAL\_PROJECTION"

"PIPELINE"

"TRAFFIC\_ANOMALY"

isInstantaneous: boolean

Whether event is a single point in time or a time range.

linkedUrl: string

formaturi

startDate: string

formatdate-time

tags?: Array<string\>

level: number

Provides an indication of how much confidence Cloudflare has in the data.

dateRange: Array<DateRange\>

endTime: string

Adjusted end of date range.

formatdate-time

startTime: string

Adjusted start of date range.

formatdate-time

lastUpdated: string

Timestamp of the last dataset update.

formatdate-time

normalization: "PERCENTAGE" | "MIN0\_MAX" | "MIN\_MAX" | 5 more

Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

One of the following:

"PERCENTAGE"

"MIN0\_MAX"

"MIN\_MAX"

"RAW\_VALUES"

"PERCENTAGE\_CHANGE"

"ROLLING\_AVERAGE"

"OVERLAPPED\_PERCENTAGE"

"RATIO"

units: Array<Unit\>

Measurement units for the results.

name: string

value: string

top\_0: Array<Top0\>

name: string

value: number

fully?: number

partially?: number