Are ChatGPT and other AI bots reading your site? How to check
Short answer: You can check in three places: your robots.txt file (a public text file that tells crawlers what they may read), your server logs (the raw list of every request your site received), and a dashboard from your host or CDN (a content delivery network, a service that sits in front of your site). Look for the names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. Google-Extended never shows in logs, because it is only a robots.txt setting. A crawler visit proves a bot read a page. It does not prove you are quoted in answers.
You want a straight yes or no on whether ChatGPT and other AI tools are reading your site, and you want to see it for yourself instead of trusting a guess.
I'm Ranjit, and I run Blur Studio: 15+ years building software, 120 apps taken live, and clients who have gone on to raise $10M+. I build AI systems and processes that help businesses do a lot more with a lot less time. I also work differently from most of what else is out there, more on that further down this page. Say hello.
Yes, you can find out, and you do not need a developer for the first pass. Open your robots.txt, then ask your host for last month's access logs, then search them for the bot names in the table below. If your site sits behind Cloudflare, a service that speeds up and protects many sites, its AI Crawl Control page shows the same thing without a log file. If your site is hosted on Ploy, an AI website builder and host, the analytics Overview page has a Bots card that counts AI crawlers.
This page gives you the exact names to look for, the commands to run, a made-up worked example, a pass and fail test plan and the limits of what a hit means. Where I describe how a bot behaves, I am quoting what its owner publishes. As of November 2026, check each vendor's current page, because bot names and version numbers change.
This page may contain affiliate links from which Blur Studio earns commissions.
Three kinds of AI bot, in plain words
Not every AI bot does the same job, and the difference decides what you should do about it. Each big vendor runs bots in three buckets: one that collects pages that may be used to train models, one that builds a search index, and one that fetches a page because a person just asked a question.
| Company | Training bot | Search bot | Fetches when a user asks |
|---|---|---|---|
| OpenAI (maker of ChatGPT) | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic (maker of Claude) | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity (an AI answer engine) | None named. Perplexity says PerplexityBot is not used to crawl for AI foundation models | PerplexityBot | Perplexity-User |
| Google-Extended (a robots.txt setting only, no bot name in logs) | Googlebot, the normal Google Search crawler | Not part of this page |
What each owner says about robots.txt, as of November 2026:
- OpenAI: disallowing GPTBot signals that your content should not be used for training. Disallowing OAI-SearchBot means you will not appear in ChatGPT search answers, though you can still show as a plain link. ChatGPT-User is triggered by a person, so OpenAI says robots.txt rules may not apply. OpenAI says the settings are independent, so blocking GPTBot does not remove you from ChatGPT search. Search changes can take about 24 hours.
- Anthropic: says its bots honor "do not crawl" rules in robots.txt, supports the non-standard Crawl-delay line, and advises against blocking by IP address, because that can stop the bot from even reading your robots.txt. If your robots.txt names ClaudeBot, that rule wins over the general setting.
- Perplexity: says PerplexityBot surfaces and links sites in search results. Perplexity-User is triggered by a person and generally ignores robots.txt rules.
- Google: says Google-Extended controls whether crawled content is used for Gemini (Google's AI assistant) training and grounding (looking up facts to ground an answer), has no separate user agent string, and does not affect inclusion or ranking in Google Search.
Step 1: read your robots.txt
Type your address plus /robots.txt into a browser, for example yourdomain.com/robots.txt. This tells you what you are currently telling bots, not what they actually did. Look for four things.
- Any line that starts with User-agent followed by a bot name from the table, and what comes under it. "Disallow: /" means the whole site is closed to that bot.
- A general "User-agent: *" block. If it says "Disallow: /", every well-behaved bot is shut out, including search bots you may want.
- Many website platforms and security plugins add AI bot rules for you. If you never wrote any, someone or something else did.
- Note that each subdomain has its own robots.txt. Anthropic says to put its rules in each subdomain's file.
Remember what robots.txt can and cannot do. It is a request, not a lock. The vendors above say they follow it for their crawling bots, but the user-triggered bots may not. If you want the longer story on what to allow, read how to get mentioned by ChatGPT and Perplexity and what answer engine optimization is. This page is only about checking who actually showed up.
Step 2: read your raw server logs
A server log is a plain text file where your web server writes one line for every request, including the time, the page asked for, the response code and a user agent. A user agent is the name a visitor gives itself, and bots put their name in it. This is the only method that shows requests even when the visitor runs no scripts, which matters because bots rarely run the tracking code that Google Analytics depends on.
How to get the logs
- On shared hosting, look in the control panel for "Raw access logs" or "Logs". Many hosts keep only a few days to a few weeks. Check yours.
- On a managed WordPress host (WordPress is a popular site-building system), ask support: "Please send me the raw access logs for the last 30 days for my domain."
- On a hosted builder such as Wix, Squarespace or Shopify, raw logs are usually not offered. Use the dashboard method in Step 3 or the builder's own bot report if it has one. Check your plan.
How to search them
On a Mac or Linux computer, open the terminal in the folder with the log file and run:
grep -i -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User" access.log | wc -l
That prints the number of matching lines. To see which pages they read, drop the count and read the output. To count each bot separately, run the command once per name. If you have no terminal, open the file in a spreadsheet, split the columns on spaces, and use the filter box with "contains" for each name.
Keep these cautions in mind:
- Anyone can type a bot name into a script, so a user agent alone is a claim. OpenAI, Anthropic and Perplexity each publish the IP address ranges their bots use (OpenAI at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json, Anthropic at claude.com/crawling/bots.json). To be sure a hit is real, compare its IP address to those lists.
- If your site is cached by a CDN, your origin server may never see requests the CDN answered. Logs from the CDN edge are more complete than logs from the origin.
- Version numbers in the user agent change, for example GPTBot/1.4. Search for the name, not the number.
Step 3: use a dashboard instead of a log file
If reading log lines sounds unpleasant, a dashboard does the counting for you.
- Cloudflare AI Crawl Control (formerly called AI Audit). Cloudflare's page says it is available on all plans and works without setup on any Cloudflare plan. It shows which AI services access your content, lets you allow or block each crawler, and tracks whether crawlers follow your robots.txt. Cloudflare's own documentation describes filters by date, crawler, operator, hostname and path. Check the Cloudflare page for what your plan shows, because some extras such as referral counts are tied to paid plans.
- Ploy analytics. On a site hosted by Ploy, the Bots card on the site's Overview page counts AI crawlers separately from "agent visits", which are visits where an AI assistant opens a page for a person. Ploy's analytics page lists the AI traffic channel as covering ChatGPT, Perplexity, Claude, Gemini and Copilot. The docs do not state a retention period for bot data, and they do not say whether crawler visits are removed from your other totals, so check with Ploy before relying on those numbers for reports. Analytics is included on every Ploy plan per Ploy, and Ploy's plans (as of November 2026) are Free at $0, Starter at $50 a month, Pro at $300 a month, and Enterprise by quote.
- Google Analytics 4 (GA4, Google's visitor-counting tool) and Search Console (Google's free report on how its search sees your site). Both are free, but neither is built for this. Google says GA4 automatically excludes known bots and spiders and you cannot turn that off or see how much was excluded, so GA4 will not tell you about crawler visits. Search Console reports on Google's own crawling, not on OpenAI or Anthropic.
- An SEO suite such as Semrush (a paid subscription for search research and site audits). These tools usually crawl your site themselves to find problems. That is a report about your pages, not a record of who visited you. Check the vendor page for what each plan includes before you pay for it for this purpose.
What a hit does and does not mean
| You see | It means | It does not mean |
|---|---|---|
| GPTBot or ClaudeBot | A crawler collected a page that may be used for model training | That you will be quoted, or that you are in any index |
| OAI-SearchBot, Claude-SearchBot or PerplexityBot | A search index may include your page | That the page ranks or gets cited for any question |
| ChatGPT-User, Claude-User or Perplexity-User | A person asked an assistant something and it opened your page | That the person saw your site, or that they clicked through |
| No hits at all | Bots were blocked, could not reach you, or your log window is too short | That you are invisible to AI tools |
A crawl is a read, not a citation. To see whether you actually appear in answers, you need a separate check: ask the AI tools your customers' real questions and write down whether your business is named. The guide to getting mentioned has a test plan for that.
A worked example with made-up numbers
These numbers are invented to show the method. Suppose a plumbing company website gets 9,000 requests in a 30-day log. A search for the bot names finds 612 lines. Counting each name separately gives:
| Bot name | Requests (made up) | Share of the 612 |
|---|---|---|
| GPTBot | 240 | 39 percent |
| ClaudeBot | 180 | 29 percent |
| OAI-SearchBot | 96 | 16 percent |
| PerplexityBot | 54 | 9 percent |
| ChatGPT-User | 30 | 5 percent |
| Others | 12 | 2 percent |
The owner then checks which pages the 612 lines asked for. If 70 percent are the blog and none are the "emergency plumber" service page, that suggests the service page is hard to reach, perhaps blocked by robots.txt or buried five clicks deep. If the 30 ChatGPT-User lines point to a pricing page, real people are asking an assistant about prices, and that page is worth tightening. The count itself is not the goal. The pages and the pattern are.
Next the owner checks whether 612 of 9,000 (about 7 percent) matters to the analytics. In a normal analytics tool these requests would be mostly missing, since bots do not run the tracking script, so the owner's visitor graph and the server log will not agree. That gap is the subject of the page on bot traffic in your numbers.
Test plan with pass and fail rules
- Get 30 days of logs or a dashboard export. Pass: at least 14 days of data. Fail: only a few days, so ask the host for more or start saving logs monthly.
- Search for the eight bot names. Pass: you can say how many requests each made. Fail: the log is too large to open. Ask your host for a filtered extract.
- Verify the top two by IP. Pass: the addresses fall in the vendor's published range. Fail: they do not, so treat the traffic as unverified or fake and consider blocking by IP at your firewall.
- Compare to robots.txt. Pass: bots you blocked show no crawling requests. Fail: a blocked bot keeps crawling, so that bot may not follow the rule or the file is not being served. Re-check the file and retest after 24 hours, since vendors say changes take up to about 24 hours.
- List the ten most-read pages. Pass: your money pages (services, pricing, contact) appear. Fail: only old blog posts appear, so fix the internal links and sitemap.
- Repeat monthly. Save the counts in a sheet with the date.
Decision rules: allow, block or ignore
- If you want to appear in ChatGPT search answers, allow OAI-SearchBot. Blocking GPTBot does not change that.
- If you do not want pages used for model training, disallow GPTBot and ClaudeBot. Google-Extended is the equivalent setting for Gemini training. Check Google's page for the exact wording.
- If you are a local business that wants to be recommended, allowing the search and user bots is usually worth more than the privacy of ordinary public pages. That is a judgment, not a rule. Your lawyer should weigh in if your content is licensed from others.
- If a bot hammers your server, use Anthropic's Crawl-delay line for ClaudeBot, or ask your host to rate-limit. Do not block ClaudeBot by IP, because Anthropic advises against it.
- Do not assume a block works against user-triggered fetches. OpenAI and Perplexity both say those may not follow robots.txt.
Who can skip this
Skip the log work if your site is a one-page brochure and you do not care about AI answers. Skip dashboards that need a plan upgrade if all you want is a yes or no, because the free log check above answers it. Skip Ploy for this purpose if your site already lives elsewhere and works. Ploy's bot card only counts requests to sites it hosts, so using it means moving the site, which is a bigger decision than reading logs. If you do not own the host login and cannot get logs, ask your web person for a one-time extract rather than buying a tool.
How I'd approach this
Start with the free method: robots.txt, then a 30-day log, then the grep above. If you already use Cloudflare, switch on AI Crawl Control and read it once a month. If you are rebuilding the site anyway and want these counts in the same place as your pageviews, a host with a built-in bots card, such as Ploy's analytics, saves the log step, but only pick a host for the whole package, not for one chart. If you want a custom monthly dashboard built from your logs, or repairs for what the logs reveal, Blur Studio builds that, see what a custom build includes.
First-week plan. Day 1: open robots.txt and write down any AI bot rules. Day 2: request 30 days of logs. Day 3: run the search, count each bot, verify the top two by IP. Day 4: decide allow, block or ignore for each bucket using the rules above. Day 5: if you changed robots.txt, note the date. Day 7: re-check the counts and save them in a sheet. Then set a monthly reminder. For the follow-on question of how these bots distort your visitor numbers, read the bot traffic page, and for the larger question of who is on your site, read who is visiting my website.
Related guide: Are Competitors Named Instead of You in AI Answers? How to Check.
Related guide: Selling AI Search Reports to Clients: Options for Agencies.
A closely related page covers the same ground from another angle: Make Your Website Readable for AI Agents: Where to Start.
A closely related page covers the same ground from another angle: Ad Ideas for a New Client in an Unfamiliar Niche.
Questions people ask
Is GPTBot crawling my site?
Search your logs for the name GPTBot, or check your Cloudflare AI Crawl Control page or your host's bot report. OpenAI publishes the IP ranges GPTBot uses, so compare any hit to openai.com/gptbot.json before you believe it. Many small sites see few or no hits in a 30-day window.
If I block GPTBot, will I disappear from ChatGPT search?
No. OpenAI says its bots are independent. GPTBot relates to training, and OAI-SearchBot is the one that controls appearing in ChatGPT search answers. Check OpenAI's bots page for its current wording.
Why does Google-Extended never appear in my logs?
Google says Google-Extended is a robots.txt control token with no separate user agent string. Google crawls with its normal crawlers, and the token only tells Google how it may use what it has fetched. It does not affect Search inclusion or ranking.
Does Google Analytics show AI bot visits?
Generally no. Google says GA4 automatically excludes known bots and spiders, and you cannot disable that or see how much was excluded. Use server logs or a CDN dashboard to count bots, and read how to filter bot traffic for the analytics side.
Can robots.txt stop ChatGPT from reading my page?
It can stop GPTBot and OAI-SearchBot if they honor it. OpenAI says ChatGPT-User visits are started by a person, so robots.txt rules may not apply. Perplexity says the same about Perplexity-User. If a page must stay private, put it behind a login.
What does Ploy show about AI crawlers?
Per Ploy's analytics docs, the Bots card on a site's Overview page counts AI crawlers separately from agent visits. The docs do not state a retention period, and they do not say whether crawler visits are excluded from other totals, so ask Ploy before using the numbers in a client report. You can compare plans on the Ploy pricing page.
Does being crawled mean ChatGPT will mention my business?
No. A crawl is a read, not a citation. To test whether you are named, ask the tools real customer questions and record the answers. The guide to getting mentioned includes a test plan.
How often should I check?
Monthly is enough for most small sites. Save the counts per bot in a sheet so you can see change over time. Check again within a day or two after you edit robots.txt, and remember that vendors say changes can take about 24 hours.
More in this topic: Websites: AI answers, AI bots and AI search reports (4 guides) →
Sources
- OpenAI: crawlers and user agents (GPTBot, OAI-SearchBot, ChatGPT-User)
- Anthropic: does Anthropic crawl data from the web
- Perplexity: PerplexityBot and Perplexity-User
- Google Search Central: Google's common crawlers (Google-Extended)
- Cloudflare: AI Crawl Control
- Google Analytics Help: known bot traffic
- Ploy docs: analytics
- Ploy pricing