Channels
Should you block AI crawlers from your website?
Should you block GPTBot, ClaudeBot or PerplexityBot? The trade-off between training-data control and AI-search visibility, plus a decision table by content type.
- 3 August 2026
- Free to read
- No signup
Add your website — see your ranked plan in about a minute, free.
PREVIEW FREE — NO CARD7-DAY TRIAL — CARD TO START
Should I block AI crawlers like GPTBot from my website?
For most public marketing and content pages, blocking AI crawlers means you can never be cited in ChatGPT, Perplexity or AI Overviews, which is a real cost if discoverability matters. Block them selectively for genuinely proprietary content (pricing tiers meant only for logged-in users, unpublished data) and allow them everywhere you want to be found and quoted.
Last updated 3 August 2026
| Content type | Recommendation | Reason |
|---|---|---|
| Blog posts, guides, answer pages | Allow | Being cited is the goal; blocking removes any chance |
| Pricing and comparison pages | Allow | Buyers ask assistants about pricing directly |
| Proprietary datasets, paid reports | Consider blocking | Training/reuse without compensation is a real cost |
| Logged-in customer content | Blocked by default | Not public, shouldn't be crawlable anyway |
| Legal/compliance pages | Allow | Low risk, occasionally useful to be cited accurately |
Do this
The steps, in order.
- Step 1
List the crawlers you're deciding about
GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and Applebot-Extended are the main ones. Each can be allowed or blocked independently in robots.txt — decide per bot, not as a single yes/no.
- Step 2
Separate 'training use' from 'answer citation'
Some bots are used both to train models and to fetch pages live for an answer; blocking a bot can remove you from both. If your concern is only training-data use, check whether the provider offers a narrower opt-out before blocking crawling entirely.
- Step 3
Default to allow for anything you'd want ranked on Google
If a page is meant to be found and read publicly, blocking AI crawlers is inconsistent with that goal — it removes an entire discovery channel for the same reasons you wouldn't block Googlebot.
- Step 4
Block selectively for content with real commercial value if scraped whole
Original research, licensed data, or paywalled reports are reasonable candidates to block, since being 'quoted' there mostly just means being reproduced without payment or a click.
- Step 5
Revisit the decision periodically
Crawler policies and the value exchange (attribution, referral traffic) are changing quickly. Review your robots.txt rules every 6–12 months rather than setting them once and forgetting them.
Worth knowing
The bits people get wrong.
There's a real tension here: allowing crawlers gives your content a chance to be cited and discovered by a fast-growing set of users, but it also lets a model reuse the content with far less certainty of attribution or a click-through than a traditional search result. For most marketing sites trying to build awareness, the discoverability upside outweighs the reuse cost.
Publishers with a direct monetisation model tied to page views (ad-supported media, subscription content) face a different calculation than a B2B software company whose content exists to generate leads. If your revenue depends on the page view itself, blocking is more defensible; if the page exists to be found and to build trust, blocking works against its purpose.
Questions
Follow-up questions.
How do I block a specific AI crawler in robots.txt?
Add a User-agent block naming the bot (e.g. "User-agent: GPTBot") followed by "Disallow: /" for a full block, or specific paths for a partial one. Each crawler needs its own User-agent line; a blanket "User-agent: *" rule also affects Googlebot.
Does blocking AI crawlers hurt my Google ranking?
Blocking GPTBot, ClaudeBot or PerplexityBot doesn't directly affect Google Search ranking, since that's controlled by Googlebot and Google-Extended separately. It does remove you from AI Overviews if you block Google-Extended specifically, so check which bot you're targeting.
Can I allow crawling but stop my content being used for AI training?
Not fully with robots.txt alone — it's a blunt allow/disallow tool, not a licensing mechanism. Some providers offer separate opt-outs or licensing terms for training use distinct from retrieval; check each provider's current policy rather than assuming robots.txt covers it.
Related elsewhere on gtm.help
AI digital marketing: what to run, channel by channel
Six channels, what AI genuinely changes in each, and what it quietly makes worse.
Tool guideAI marketing agents: what they do and which are worth paying for
Eight options, real prices, and a straight answer on what agents can safely run unattended.
ComparisonApollo vs Clay
Apollo is one database you search and export from; Clay is a spreadsheet that calls many databases in turn until it fin…
ChecklistHow do you get found by AI assistants like ChatGPT?
Structure content so AI assistants can cite it directly, ranked by return.
Elsewhere on gtm.help
Keep exploring.
Free marketing tools
Calculators and generators that run without a signup.
BrowseGrowth playbooks
Step-by-step plans by industry and goal.
BrowseTool guides
What to use, what it costs, when it earns its keep.
BrowseTool comparisons
Head-to-head picks for the calls people get stuck on.
BrowseMarketing checklists
Task lists by business type and competitive situation.
BrowseGet started
Want this answered for your business?
Paste your URL and you'll get a ranked plan, costed in hours and dollars, in about a minute.
Add your website and see your own ranked plan.