Short answer
Robots.txt Generator with AI Crawler Controls: what it does
robots.txt sits at yoursite.com/robots.txt and tells crawlers which paths they may fetch. Blocking AI crawlers keeps your content out of their training and answers, which also means AI search is less likely to recommend you.
Last updated 3 August 2026
Here's one we ran earlier.
A shop that wants AI search citations but not model training
robots.txt
Copy thisUser-agent: * Allow: / Disallow: /admin User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Three steps, no account.
Hide only what has no search value
Admin, cart, checkout and internal search results. Blocking real pages removes them from Google.
Decide on AI deliberately
AI search crawlers (OAI-SearchBot, PerplexityBot) drive citations. Training crawlers (GPTBot, CCBot, Google-Extended) only feed models.
Point to your sitemap
A Sitemap line helps every crawler find new pages faster.
Common questions.
Does robots.txt stop a page being indexed?
Not reliably. A blocked page can still appear if others link to it. Use a noindex tag for pages that must stay out.
Should I block GPTBot?
Block it if you don't want your content used to train OpenAI models. It doesn't affect ChatGPT search, which uses OAI-SearchBot.
Where does the file go?
At the root of your domain: yoursite.com/robots.txt. Subfolders are ignored.
Does this cost anything?
No. It's free, no account needed, and you can run it as many times as you like.
Use these next.
Schema Markup Generator
Copy-ready JSON-LD for your business, product or FAQ
OpenFree checkerMeta Tag Checker
Check your title and description before Google cuts them
OpenFree toolAEO Question Finder
Find the questions AI search answers in your market
OpenGoing deeper? SEO checklists — marketing checklists from the rest of the library.
Related elsewhere on gtm.help
How do I write content that AI search engines will quote?
AI search tends to quote short, self-contained blocks: a one-sentence direct answer with a number, followed by a table…
PlaybookHow to use AI for content marketing
Where AI genuinely helps, where it quietly hurts you, and the editing pass in between.
ComparisonAhrefs vs Semrush
Ahrefs if your work is links, competitor content and keyword research, and you want the cleanest data. Semrush if you a…
Tool guideAI marketing agents: what they do and which are worth paying for
Eight options, real prices, and a straight answer on what agents can safely run unattended.
Keep exploring.
Growth playbooks
Step-by-step plans by industry and goal.
BrowseTool guides
What to use, what it costs, when it earns its keep.
BrowseTool comparisons
Head-to-head picks for the calls people get stuck on.
BrowseMarketing checklists
Task lists by business type and competitive situation.
BrowseStraight answers
The growth questions founders ask, answered with numbers.
Browse32 free tools, or one plan that uses them all.