Reads the robots.txt of a site from our server and shows, bot by bot, whether the main AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and more) are allowed or blocked, with the rule that decided and any meta tag that restricts them. For founders who want to be found by AI assistants, or who want to keep them out.
How to use it
Enter the address of your site. Any page works, because only the site's address is used. You can leave out https://.
- Type or paste your site's address and press Run check.
- Read the table: each crawler, what it is used for, whether it is allowed, and the rule that decided.
- Check the findings for crawlers you want to let in, such as the ones that power AI search.
- Edit robots.txt if the result is not what you intend, then run the check again.
How Reavlo tests this
This tool runs on our server, because a browser cannot read another site's robots.txt. Nothing about the request is stored or logged: not the address, not the file.
The crawler list. The bots come from the Reavlo ai-user-agents dataset: each token is written as the vendor's own documentation gives it, with a link to that page and the date it was read. The list is reviewed every month. A token that is not documented by its operator is not included, so a bot missing from the table is not covered.
Fetching. One request to /robots.txt and one to the home page of the same site, with a size cap and a time limit. Addresses that point to private or internal networks are refused before any connection is made.
Matching. The file is read as RFC 9309 defines it. A crawler follows the group that names its token; if none does, it follows the group for *. The longest matching rule decides, and when two are equally specific, Allow wins. Each bot is tested against the site's root path. A bot that no rule applies to is shown as allowed by default, not as explicitly allowed.
Meta tags. A meta tag named for a crawler, or the general robots tag, with noindex or none is reported. These tags only reach crawlers that read the page, so they say nothing about whether a crawler is allowed to fetch it.
Purpose. Each bot is labelled by what its operator says it is used for: search, training, a fetch made when a person asks, or ads. Blocking a search bot can keep your site out of that assistant's answers; blocking a training bot does not.
Limits. Robots.txt is a request, not a lock: it works for crawlers that honour it. Some operators say that fetches triggered by a person's request may not follow robots.txt, and the table marks those bots. The tool cannot see rules a firewall or CDN applies to a crawler, and it tests only the root path, so a rule for a deeper path is not reflected.
Questions
Is the address I enter stored?
No. The address goes to our server only to fetch robots.txt and the home page, and neither it nor the files are stored or written to a log. Analytics records only that the tool ran, never the address.
Should I block GPTBot?
That is your choice. GPTBot collects content that OpenAI may use for training, while OAI-SearchBot is the one that powers search results in ChatGPT. Many sites block the first and allow the second.
Does blocking a bot in robots.txt guarantee it stays out?
No. Robots.txt is honoured by crawlers that choose to follow it. Operators of the main AI crawlers document that they do, but a request that a person triggers may be handled differently, and the table marks those bots.
Why is a bot I know missing from the table?
The list holds only crawlers whose tokens are documented by their operator. A name that is not in that documentation is left out rather than guessed.
What does allowed by default mean?
No rule in your robots.txt applies to that bot, so it may fetch everything. The rule column then shows that no rule decided.