Which AI crawlers can read my store?
Emtor reads your robots.txt, shows which AI crawlers it lets read your product pages, the lines that keep one out and the steps to change them, and can ask for your pages with each crawler's name. It never changes the file.
Emtor reads your store's robots.txt and shows, for 12 AI crawlers, whether it lets them read your product pages: 9 that answer shoppers, and 3 that collect pages to train future models. It's on the AI answers page, which you open from the Proof page, under Can AI assistants read your store?
A firewall or bot protection can still turn a crawler away, and robots.txt doesn't show that. When you press Check live, Emtor asks for your home page and one published product's page from our server, once with each crawler's name and once with its own, and shows what came back under Our request.
| Crawler | For |
|---|---|
| OAI-SearchBot | ChatGPT search |
| ChatGPT-User | ChatGPT, opening a page for a shopper |
| PerplexityBot | Perplexity |
| Perplexity-User | Perplexity, opening a page for a shopper |
| Claude-SearchBot | Claude search |
| Claude-User | Claude, opening a page for a shopper |
| Googlebot | Google Search and its AI Overviews |
| Google-Extended | Gemini app answers, and Gemini training |
| Bingbot | Bing and Microsoft Copilot |
| Crawler | For |
|---|---|
| GPTBot | OpenAI model training |
| ClaudeBot | Anthropic model training |
| Applebot-Extended | Apple Intelligence training |
What the results mean
- Allowed means the file lets that crawler read your product pages, and Blocked means a rule in it keeps the crawler out.
- Got through means our request with that crawler's name got your pages. Turned away means it was refused or got an error page, or a bot check instead of the page: Cloudflare's, or the challenge or checkpoint page it was sent to. Cloudflare also puts a script of its own in ordinary pages, so that script alone doesn't count.
- Rate limited means your store answered that there were too many requests. Shopify can answer that to any request it can't verify, as ours are, so it says nothing about the crawler itself.
- Password page means your store's password page answered: no crawler can read your store while the password is on. No answer means no page came back: our request got no answer in time, couldn't connect, was sent somewhere other than your store or met a server error, or the check ran out of time before asking.
- A live result is a sign, not proof. Real crawlers come from their own networks, and a firewall can treat those differently from ours. A crawler turned away here is worth looking into, and one that got through may still be turned away. Emtor's own request, with no crawler's name, shows whether your store lets our server in at all.
- Google-Extended and Applebot-Extended aren't checked live. No crawler sends those names: Google and Apple read them in robots.txt only. Google-Extended is listed with the crawlers that answer shoppers because Google uses it to decide whether Gemini may answer from your pages, not only whether it may train on them. It doesn't change Google Search.
- Keeping training crawlers out is your choice. It doesn't stop assistants from answering about your store.
- No robots.txt lets every crawler in, as the robots.txt standard says. If the file can't be read just then, the page says so instead of guessing.
The lines that keep a crawler out
When your robots.txt keeps a crawler out of product pages, its row says Blocked with the number of the line that does it, and under the table, What keeps them out shows the lines themselves, as your store serves the file: the user-agent line that names the crawler, and the rule. If the file doesn't name the crawler, its rules for every crawler (User-agent: *) apply, and the page says so. If it names the crawler in more than one group, their rules count together, and the longest rule that matches the page decides, as the robots.txt standard says.
The line numbers are those of the file at your store's address. Your theme's robots.txt.liquid writes it, partly from Liquid, so in the template the same words can sit on other lines: look for the words.
Changing it
Emtor never changes your robots.txt, your theme or a firewall. It shows the lines, the steps of whoever writes them, and checks again once you've changed them. For the crawlers that answer shoppers:
- Lines in your theme. The page lists Shopify's steps to edit robots.txt.liquid in the theme's code editor, and Shopify's way back to its own file: deleting the template, which drops every change in it, not only these lines. Shopify warns that it's an unsupported customization, and that incorrect use can cost a store all its traffic.
- Lines Cloudflare adds. If your store's domain goes through a Cloudflare account, Cloudflare can add lines of its own at the start of the file, between “# BEGIN Cloudflare Managed content” and “# END Cloudflare Managed Content”. They aren't in your theme, so the page lists Cloudflare's steps for its AI crawler policies instead.
- Turned away in the live check. Shopify serves its stores through Cloudflare and runs bot protection for them itself, so a Cloudflare bot check can be Shopify's own; the page lists Cloudflare's steps for a domain that goes through a Cloudflare account of your own. Shopify's challenge page is Shopify's own check, and Shopify documents no setting for it that covers the pages crawlers read. Its checkpoint page is the bot protection a Shopify Plus store schedules for a sale, with Shopify's steps to end it early. A plain refusal can't be traced to anyone, and the page says so.
- Password-protected. While your store's password is on, no crawler can read it, whatever robots.txt says. The section says so above the tables, with Shopify's steps to open your store, and still shows what the file itself allows.
Keeping a training crawler out is your choice, so for those the page shows the lines and the help of whoever writes them, and nothing more.
When you've made a change, press Check again. It reads your robots.txt again, runs the live check again if you ran one, and says what changed since the last read on the page, such as a crawler that went from blocked to allowed. It doesn't run an agent test. Your store can be checked once a minute, and nothing is kept: the results show until you reload the page.
On our test store, Shopify's own robots.txt let every one of these crawlers read product pages, so a crawler shown as blocked is kept out by a rule someone added to your store's file, or by Cloudflare's lines.