/projects//ai_search/readiness/ & /projects//audit/ ).
1. The Evolution of Robots.txt in the AI Era
Historically, robots.txt was used primarily to manage search engine crawlers like Googlebot and Bingbot. In 2026, webmasters must account for dozens of AI data-gathering bots. Blocking GPTBot prevents OpenAI from scraping your site for model training, but also prevents your brand from being cited in ChatGPT Search and conversational answers.
LIVE PLATFORM SCREEN & TELEMETRY
SEO Runway AI Search Readiness & Site Audit — Active Workspace
Below is the exact production screen with live parameters and dummy data illustrating the diagnostic findings and remediation buttons.
2. Validating Robots.txt in SEO Runway
Navigate to AI Search (GEO) → Readiness Audit. SEO Runway automatically parses your robots.txt directives and displays crawler permissions for Googlebot, GPTBot, PerplexityBot, and ClaudeBot, highlighting conflicting directives.
Target System: SEO Runway TECHNICAL Engine
Execution Frequency: Automated Background Queue (UTC Sync)
Tested across enterprise datasets with zero rate limiting or data loss.
3. What robots.txt Can and Cannot Do
robots.txt is a plain-text file at the root of a host that tells crawlers which paths they may request. Search engines and well-behaved bots read it before crawling. It is important to be precise about its limits:
- It controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results (usually without a description) if other pages link to it. To keep a page out of the index, allow crawling and use a
noindexdirective instead. - It is voluntary. Reputable search engines and major AI companies say their crawlers respect it, but scrapers and malicious bots can ignore it. It is a policy statement, not a security control.
- It is public. Anyone can read it, so never use it to "hide" sensitive paths; that advertises them.
- It applies per host and protocol.
www.example.com,example.comand subdomains such asshop.example.comeach need their own file.
4. The Rules That Matter
- User-agent groups. Rules are grouped under
User-agent:lines. A crawler follows the most specific group that matches its name and ignores the rest, so aUser-agent: Googlebotgroup replaces, rather than adds to, theUser-agent: *group for Google. - Allow and Disallow. When rules conflict, Google applies the most specific (longest) matching rule. If an Allow and a Disallow rule are equally specific, Google uses the less restrictive Allow.
- Wildcards.
*matches any sequence of characters and$anchors the end of a URL. For exampleDisallow: /*?sort=blocks any URL containing a sort parameter. - Sitemap lines.
Sitemap:lines with absolute URLs can appear anywhere and apply to all crawlers. - Size and format. Google reads up to 500 KiB of the file. It must be UTF-8 plain text, served with a 200 status at
/robots.txt. - Crawl-delay. Google ignores it (crawl rate is managed automatically), while some other crawlers, including Bing, respect it.
5. Designing a Resilient File
"Resilient" means a file that protects crawl budget and content, doesn't accidentally block important pages, and keeps working as the site changes. A sensible structure:
- A default group for all crawlers that blocks only genuinely non-public or wasteful paths: internal search results, cart and checkout, account areas, endless filter combinations and admin paths.
- Explicit groups for AI crawlers reflecting your policy for each, such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Applebot-Extended.
- No blocks on rendering resources. CSS, JavaScript, images and the API endpoints used to render pages must stay crawlable, or search engines see broken pages.
- Sitemap references for every sitemap index.
- Comments (lines starting with
#) explaining why each rule exists and who owns it.
6. Making AI Crawler Decisions Deliberately
AI-related crawlers fall into roughly three groups, and each deserves its own decision:
- Training crawlers collect content to train models. Examples include GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and the Google-Extended and Applebot-Extended tokens, which control use of your content for Google's and Apple's AI models without affecting search crawling.
- AI search and retrieval crawlers fetch pages to answer questions and cite sources, such as OAI-SearchBot (ChatGPT search) and PerplexityBot. Blocking these can remove you from AI answers that would otherwise cite and link to you.
- User-triggered fetchers, such as ChatGPT-User, fetch a page because a user asked the assistant to look at it.
Many businesses allow AI search crawlers (because citations send traffic and visibility) while making a separate choice about training crawlers. Whatever you decide, write it down, implement it per user agent, and review it as new crawlers appear. Check each company's documentation for current user-agent names, since they change.
7. Testing Before You Deploy
- Use Search Console's robots.txt report to see which version Google last fetched and any parsing errors, and use URL Inspection to check whether specific URLs are blocked.
- Test critical URLs automatically. Keep a list of must-crawl URLs (homepage, top categories, top products, key articles, CSS and JS files) and must-block URLs, and run a robots.txt parser against both in CI whenever the file changes.
- Watch for environment leaks. A staging robots.txt with
Disallow: /deployed to production is one of the most common, and most damaging, SEO incidents. Generate the file per environment and test production's version after every release. - Check server behaviour. robots.txt must return 200 (or 404, meaning "no restrictions"). If it returns a 5xx error, Google may pause crawling the site until it can fetch the file again.
8. Common Mistakes
- Using
Disallowto remove pages from Google's index (use noindex instead, and leave crawling allowed so the directive can be seen). - Blocking URLs that carry noindex tags, so search engines never see the noindex.
- Blocking JavaScript or CSS directories needed for rendering.
- Assuming a Googlebot-specific group inherits rules from the wildcard group.
- Forgetting subdomains, which need their own robots.txt.
- Blocking parameter patterns so broadly that legitimate pages (such as paginated categories) are caught.
9. Maintaining the File Over Time
Treat robots.txt as code: store it in version control, require review for changes, and record the reason for each rule in comments. Review it quarterly against server logs to confirm the rules still match how crawlers behave on your site. New URL patterns, new AI crawlers and site migrations all mean the file should evolve. A five-minute review is far cheaper than discovering months later that a whole section was blocked.
10. Bot Management Beyond robots.txt
Because robots.txt is voluntary, enforce critical policies elsewhere too. CDN and firewall tools can verify known crawlers by IP and reverse DNS, rate-limit unverified automated traffic, and block user agents that ignore your rules. Use logs to confirm that crawlers you've disallowed actually stopped. If one hasn't, block it at the network edge.
11. robots.txt Checklist
- Served at
/robots.txtwith a 200 status on every host and subdomain. - Only non-public and wasteful paths blocked; rendering resources allowed.
- Separate, documented groups for each AI crawler category.
- Sitemap lines included.
- Must-crawl and must-block URL lists tested automatically on every change.
- Environment-specific files, with production verified after each deploy.
- Policies enforced at the CDN or firewall for bots that ignore robots.txt.
12. An Annotated Example File
Here is how a resilient robots.txt for a mid-size e-commerce site might be organised, described line by line rather than copied blindly:
- Default group (
User-agent: *): disallow/cart/,/checkout/,/account/,/searchand parameter patterns such as/*?sort=and/*?sessionid=. Allow everything else, including/static/and/assets/. - AI search group (
User-agent: OAI-SearchBot,User-agent: PerplexityBot): repeat the default disallow rules. Because a named group replaces the wildcard group, every rule you still want applied must be listed again. - AI training group (
User-agent: GPTBot,ClaudeBot,CCBot,Google-Extended,Applebot-Extended): either the same rules as the default group orDisallow: /, depending on your training policy. - Sitemap line:
Sitemap: https://www.example.com/sitemap.xml. - Comments above each group naming the policy owner and the date of the last review.
The key structural lesson is the replacement rule: forgetting to copy the default disallows into a named group is the most common way specific crawlers end up crawling cart, search and filter URLs that everyone else is blocked from.
13. robots.txt During Migrations and Launches
Site launches and migrations are when robots.txt causes the most damage. Before launch, staging sites are often protected with Disallow: /, which must be removed at go-live. Protecting staging with HTTP authentication instead avoids the risk entirely. During domain migrations, keep the old domain's robots.txt permissive, because blocking it prevents search engines from crawling the old URLs and discovering their redirects. After launch, fetch the live file, run your must-crawl URL tests, and check Search Console's robots.txt report within the first hour.
14. Measuring the Effect of robots.txt Changes
Every rule change should have a measurable purpose. After blocking wasteful paths, the share of Googlebot requests going to them in your logs should fall towards zero within a few weeks, and requests to valuable templates should rise. After changing AI crawler access, logs show whether those crawlers stopped or started visiting, and AI visibility tracking shows whether citations changed. If a change produces no measurable effect, either the crawler wasn't visiting those paths anyway or it ignores the file, and both are useful to know.
15. robots.txt, Meta Robots and X-Robots-Tag: Choosing the Right Tool
Three mechanisms control how search engines treat your URLs, and each solves a different problem:
- robots.txt controls whether a URL can be crawled. Use it to stop crawlers wasting time on infinite spaces, internal search and private areas.
- The meta robots tag (
<meta name="robots" content="noindex">) controls whether an HTML page can be indexed and how it appears in results. The page must be crawlable for the tag to be seen. - The X-Robots-Tag HTTP header does the same for non-HTML files such as PDFs, images and documents, or for applying rules to whole directories at server level.
A simple decision rule: if you don't want a URL in search results, use noindex and keep it crawlable. If you don't want crawlers spending time on a URL pattern at all, and it doesn't matter whether a bare URL occasionally appears in results, use robots.txt. For snippet and AI-display controls, Google also supports directives such as nosnippet and max-snippet, which apply to its search features including AI Overviews.
Mixing these tools up is behind many indexing surprises. The most common is a page that is both blocked in robots.txt and tagged noindex: the crawler can't fetch the page, never sees the noindex, and the URL lingers in results. Audit for this combination whenever you add new robots.txt rules.
Want the complete procedural runbook?
Visit our comprehensive step-by-step Knowledge Base for interactive screen maps, configuration parameters, and adjacent module workflows.
Open Knowledge Base Hub →

The point about isolating parameters in the audit crawl depth is pure gold. Saved our engineering team dozens of hours. As a Head of Technical SEO, Rakuten, this provides exactly the architectural depth we look for.