Skip to content
Cloudflare Docs

Advanced configuration

Disable Pay Per Crawl by URI pattern

You may want to offer free access to certain pages while charging for others:

  • Allow free access to homepages, category pages, or navigation to help crawlers discover paid content.
  • Exclude functional pages like login, search, or API endpoints that don't contain chargeable content.
  • Start with Pay Per Crawl on a small section of your site before expanding.
  • Offer free access to promotional or archived content while charging for premium articles.

To get started, use Configuration Rules to exclude specific URI patterns from charging.

  1. Go to Rules > Overview in the Cloudflare dashboard.

    Go to Overview

  2. Select Create rule > Configuration Rule.

  3. When incoming requests match: Set your URI pattern.

    • Field: URI Full
    • Operator: wildcard
    • Value: https://*example.com/public/*

  4. Select Disable Pay Per Crawl > Add

  5. Select Deploy.

Example patterns:

  • Free homepage: URI Full equals https://example.com/
  • Free directory: URI Full wildcard https://*example.com/public/*

Additional resources