Free GEO tools · AI robots.txt checker
AI robots.txt checker
Enter your site address: the tool reads your robots.txt and shows, for 16 crawlers (Google, Bing and 14 AI crawlers), whether they can crawl your site. It then suggests a corrected file.
Free, 5 uses per day without sign-up. Unlimited with the Solo and Agency plans.
Why check your robots.txt for AI?
The robots.txt file tells crawlers what they are allowed to read. A rule that is too broad (“Disallow: /” for everyone) or a badly chosen anti-AI rule can prevent ChatGPT, Perplexity or Claude from reading your pages, and therefore from citing you. Many hosts and plugins add these rules without warning.
Search crawlers and training crawlers: what is the difference?
Search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot…) read your pages to answer a question and cite the source. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) collect texts to train models. You can block the latter without blocking the former: many major sites make that choice, as our study shows.
Go further: the full GEO audit
The free TraceFlow Lens GEO and SEO audit checks 33 points in 20 seconds (AI crawlers, readability without JavaScript, structured data, sitemap, technical SEO, content) and gives, for each issue, the fix and a prompt ready to paste into your AI agent.
Frequently asked questions
How do I allow ChatGPT in my robots.txt?
Add “User-agent: OAI-SearchBot” then “Allow: /”, and do the same for ChatGPT-User. These two crawlers let ChatGPT read and cite your pages.
Does blocking GPTBot keep me out of ChatGPT?
No. GPTBot collects texts for training. ChatGPT search goes through OAI-SearchBot and ChatGPT-User, which have their own rules.
Where is my robots.txt file?
Always at the root of the site: https://yoursite.com/robots.txt.