---
description: Browser Run crawl results now identify each URL outcome and report robots.txt availability by origin.
title: Crawl results include URL outcomes and robots.txt diagnostics
image: https://developers.cloudflare.com/changelog/post/2026-11-03-crawl-result-diagnostics/og.png?v=261b213e09f37076
---

[Skip to content](#main-content)

[View RSS feeds](https://developers.cloudflare.com/fundamentals/new-features/available-rss-feeds/) [ Subscribe to RSS](https://developers.cloudflare.com/changelog/rss/index.xml)

[Back to all posts](https://developers.cloudflare.com/changelog)

November 3, 2026

## Crawl results include URL outcomes and robots.txt diagnostics

[Browser Run](https://developers.cloudflare.com/browser-run/)

Copy as Markdown| [View as Markdown](https://developers.cloudflare.com/changelog/post/2026-11-03-crawl-result-diagnostics/index.md)| [Agent setup](https://developers.cloudflare.com/agent-setup/)

Browser Run [`/crawl` results](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#request-results-of-the-crawl-job) now include more detailed diagnostics. Each URL record can include an `outcome` that identifies the specific processing result, such as `success`, `robots_disallowed`, `timeout`, or `page_error`.

The job-level `robots` array reports the `robots.txt` state for up to 50 origins encountered by the crawl. Each entry includes the origin, whether usable rules were available, and the HTTP status when the crawler received a response.

```json
{
	"robots": [
		{
			"origin": "https://example.com",
			"state": "available",
			"status": 200
		}
	],
	"records": [
		{
			"url": "https://example.com/docs/",
			"status": "completed",
			"outcome": "success"
		}
	]
}
```

For field definitions and all possible outcome values, refer to [URL outcomes](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#url-outcomes) and [robots.txt diagnostics](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#robotstxt-diagnostics).

```json
{"@context":"https://schema.org","@type":"BlogPosting","@id":"https://developers.cloudflare.com/changelog/post/2026-11-03-crawl-result-diagnostics/#page","headline":"Crawl results include URL outcomes and robots.txt diagnostics","description":"Browser Run crawl results now identify each URL outcome and report robots.txt availability by origin.","url":"https://developers.cloudflare.com/changelog/post/2026-11-03-crawl-result-diagnostics/","inLanguage":"en","image":"https://developers.cloudflare.com/changelog/post/2026-11-03-crawl-result-diagnostics/og.png?v=261b213e09f37076","dateModified":"2026-11-03","datePublished":"2026-11-03","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"}}
```