Channels
Should you block AI crawlers from your website?
Should you block GPTBot, ClaudeBot or PerplexityBot? The trade-off between training-data control and AI-search visibility, plus a decision table by content type.
- 3 August 2026
- Free to read
- No signup
Should I block AI crawlers like GPTBot from my website?
For most public marketing and content pages, blocking AI crawlers means you can never be cited in ChatGPT, Perplexity or AI Overviews, which is a real cost if discoverability matters. Block them selectively for genuinely proprietary content (pricing tiers meant only for logged-in users, unpublished data) and allow them everywhere you want to be found and quoted.
Last updated 3 August 2026
| Content type | Recommendation | Reason |
|---|---|---|
| Blog posts, guides, answer pages | Allow | Being cited is the goal; blocking removes any chance |
| Pricing and comparison pages | Allow | Buyers ask assistants about pricing directly |
| Proprietary datasets, paid reports | Consider blocking | Training/reuse without compensation is a real cost |
| Logged-in customer content | Blocked by default | Not public, shouldn't be crawlable anyway |
| Legal/compliance pages | Allow | Low risk, occasionally useful to be cited accurately |
Do this
The steps, in order.
- Step 1
List the crawlers you're deciding about
GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and Applebot-Extended are the main ones. Each can be allowed or blocked independently in robots.txt — decide per bot, not as a single yes/no.
- Step 2
Separate 'training use' from 'answer citation'
Some bots are used both to train models and to fetch pages live for an answer; blocking a bot can remove you from both. If your concern is only training-data use, check whether the provider offers a narrower opt-out before blocking crawling entirely.
- Step 3
Default to allow for anything you'd want ranked on Google
If a page is meant to be found and read publicly, blocking AI crawlers is inconsistent with that goal — it removes an entire discovery channel for the same reasons you wouldn't block Googlebot.
- Step 4
Block selectively for content with real commercial value if scraped whole
Original research, licensed data, or paywalled reports are reasonable candidates to block, since being 'quoted' there mostly just means being reproduced without payment or a click.
- Step 5
Revisit the decision periodically
Crawler policies and the value exchange (attribution, referral traffic) are changing quickly. Review your robots.txt rules every 6–12 months rather than setting them once and forgetting them.
Worth knowing
The bits people get wrong.
There's a real tension here: allowing crawlers gives your content a chance to be cited and discovered by a fast-growing set of users, but it also lets a model reuse the content with far less certainty of attribution or a click-through than a traditional search result. For most marketing sites trying to build awareness, the discoverability upside outweighs the reuse cost.
Publishers with a direct monetisation model tied to page views (ad-supported media, subscription content) face a different calculation than a B2B software company whose content exists to generate leads. If your revenue depends on the page view itself, blocking is more defensible; if the page exists to be found and to build trust, blocking works against its purpose.
Questions
Follow-up questions.
How do I block a specific AI crawler in robots.txt?
Add a User-agent block naming the bot (e.g. "User-agent: GPTBot") followed by "Disallow: /" for a full block, or specific paths for a partial one. Each crawler needs its own User-agent line; a blanket "User-agent: *" rule also affects Googlebot.
Does blocking AI crawlers hurt my Google ranking?
Blocking GPTBot, ClaudeBot or PerplexityBot doesn't directly affect Google Search ranking, since that's controlled by Googlebot and Google-Extended separately. It does remove you from AI Overviews if you block Google-Extended specifically, so check which bot you're targeting.
Can I allow crawling but stop my content being used for AI training?
Not fully with robots.txt alone — it's a blunt allow/disallow tool, not a licensing mechanism. Some providers offer separate opt-outs or licensing terms for training use distinct from retrieval; check each provider's current policy rather than assuming robots.txt covers it.
Elsewhere on gtm.help
Keep exploring.
Get started
Want this answered for your business?
Paste your URL and you'll get a ranked plan, costed in hours and dollars, in about a minute.