How do you track GPTBot, ClaudeBot, and PerplexityBot in server logs and configure robots.txt?

Server log inspection is the reliable way to track GPTBot, ClaudeBot, and PerplexityBot visits across your website. AI crawlers fetch raw HTML from Cloudflare, Vercel, AWS CloudFront, or Nginx without running browser JavaScript tags such as Google Analytics 4. Once your logs show which URLs AI bots read, configure /robots.txt by crawler purpose rather than blocking all AI bots together. Disallowing GPTBot, ClaudeBot, and CCBot keeps future pages out of foundation-model training sets, whereas disallowing OAI-SearchBot, Claude-SearchBot, and PerplexityBot removes your domain from live ChatGPT, Claude, and Perplexity citations.

Why did AI crawler volume surge in 2025 and 2026?

AI crawler volume is now a majority share of automated web requests across global CDN networks. Cloudflare Radar reported on July 1, 2026 that non-human requests passed 50% of global internet traffic. AI training crawls accounted for 52% of purpose-identified crawler requests in June 2026, up from 22% in Spring 2025, while mixed-use crawlers generated over 36% of requests.

Separating training scrapes from search indexing is the operational fix for engineering and growth teams. Blocking unverified scrapers cuts origin bandwidth costs while keeping product pages, research studies, and pricing tables open to answer engines.

Why does Google Analytics 4 miss GPTBot, ClaudeBot, and PerplexityBot traffic?

Google Analytics 4 misses GPTBot, ClaudeBot, and PerplexityBot traffic because client-side analytics requires browser JavaScript execution. GA4 relies on a visitor's browser running gtag.js and firing an HTTP beacon to Google collection servers. In a December 2024 study with MERJ, Vercel analyzed 569 million monthly fetches from GPTBot, 370 million from Anthropic's Claude crawlers, and 24.4 million from PerplexityBot.

Zero requests from GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider, or CCBot executed JavaScript in Vercel's dataset. ChatGPT crawlers spent 11.50% of fetches and Claude crawlers spent 23.84% downloading .js files as raw text without running a single script. Any page that renders text only on the client is an empty shell to those crawlers.

GA4 referral reports are a record of human clicks from chatgpt.com, claude.ai, or perplexity.ai, not bot crawls. Cloudflare measured 38,000 crawls per referred visitor for Anthropic and 194 crawls per referred visitor for Perplexity in July 2025. Reading CDN or origin HTTP logs is the only way to see those thousands of unclicked bot visits.

What is ClaudeBot and how do GPTBot, OAI-SearchBot, and PerplexityBot differ?

ClaudeBot is Anthropic's model-training crawler, GPTBot is OpenAI's model-training crawler, and OAI-SearchBot, Claude-SearchBot, and PerplexityBot are search-citation crawlers. Every major AI lab splits its crawlers into distinct user-agent tokens so site owners can opt out of foundation-model training without disappearing from AI search answers.

AI traffic consists of four technical categories in robots.txt and HTTP logs:

• Model-training crawlers. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, and Bytespider collect public pages to train or fine-tune language models. Blocking GPTBot or ClaudeBot opts future pages out of training while leaving search citations untouched.
• Search-citation crawlers. OAI-SearchBot, Claude-SearchBot, and PerplexityBot build the search indexes behind ChatGPT search, Claude search, and Perplexity answers. Blocking OAI-SearchBot or PerplexityBot erases your pages from cited search responses.
• Live user-triggered fetchers. ChatGPT-User, Claude-User, and Perplexity-User fetch one URL when a human asks an assistant about that page. OpenAI and Perplexity document that ChatGPT-User and Perplexity-User often ignore robots.txt because a person requested the fetch, whereas Anthropic states that Claude-User honors robots.txt.
• Usage-control tokens. Google-Extended and Applebot-Extended are robots.txt policy tokens rather than active HTTP crawlers. Googlebot and Applebot perform the actual crawls, and the extended tokens govern how Google and Apple reuse those crawled pages.

Which user-agent tokens map to AI training vs AI search citations?

User-agent tokens map to AI training versus AI search citations across twelve documented bots and policy tokens as of November 2026.

How do you inspect Nginx and CDN logs to see which pages AI bots read?

Inspecting Nginx and CDN logs with command-line tools rg and awk shows which pages AI bots read and which HTTP status codes they receive. Export combined logs from Nginx, Apache, AWS CloudFront, Fastly, Cloudflare Logpush, or Vercel Log Drains. In combined log format, field $7 is the request path, field $9 is the status code, and quote-delimited field $6 is the User-Agent header.

Match the short bot token instead of the full header string. OpenAI and Perplexity update version strings such as GPTBot/1.4 and OAI-SearchBot/1.4 without advance notice.

# 1. Count total hits per AI crawler token
rg -oi 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-externalagent' /var/log/nginx/access.log \
  | tr '[:upper:]' '[:lower:]' \
  | sort | uniq -c | sort -rn

# 2. Audit HTTP status codes 200, 301, 403, and 404 for AI search bots
rg -i 'OAI-SearchBot|Claude-SearchBot|PerplexityBot' /var/log/nginx/access.log \
  | awk '{
      match($0, /(OAI-SearchBot|Claude-SearchBot|PerplexityBot)/, bot);
      print tolower(bot[1]), $9
    }' \
  | sort | uniq -c | sort -rn

Status code distribution is the quickest diagnostic for invisible CDN blocks. When PerplexityBot or OAI-SearchBot receives 403 while /robots.txt allows /, a WAF rule or Cloudflare Bot Fight Mode is blocking the crawler at the network edge. After confirming 200 responses, rank the top crawled paths to see which commercial pages AI assistants read most.

# 3. Rank the top 25 URLs returning HTTP 200 to AI search and user fetchers
rg -i 'OAI-SearchBot|ChatGPT-User|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.log \
  | awk '$9 == 200 {print $7}' \
  | sort | uniq -c | sort -rn | head -n 25

How do you verify real GPTBot, ClaudeBot, and PerplexityBot IP addresses?

Verifying real GPTBot, ClaudeBot, and PerplexityBot IP addresses requires matching each request against the operator's published CIDR JSON files or reverse DNS records. Any scraper can spoof GPTBot or ClaudeBot in an HTTP User-Agent header to bypass rate limits. OpenAI publishes separate IPv4 prefix lists at https://openai.com/gptbot.json, https://openai.com/searchbot.json, and https://openai.com/chatgpt-user.json routed on Microsoft Azure AS8075 without reverse DNS hostnames.

Anthropic publishes its official IP ranges in https://claude.com/crawling/bots.json. Perplexity moved its IP files on October 30, 2026 from perplexity.com to https://www.perplexity.ai/perplexitybot.json and https://www.perplexity.ai/perplexity-user.json. Perplexity instructs Cloudflare and AWS WAF administrators to match each User-Agent against its own IP list using a WAF Skip rule above challenge rules.

Common Crawl is verifiable through both https://index.commoncrawl.org/ccbot.json and reverse DNS resolving to crawl.commoncrawl.org. Sync operator JSON files automatically every 24 hours rather than pasting static IP blocks, and apply CDN rate limits to undocumented Bytespider requests.

How do you configure robots.txt to block AI training while keeping search citations?

Separate User-agent groups in /robots.txt are the standard way to block AI training crawlers while allowing search-citation bots. Under IETF RFC 9309, a crawler that matches its own named User-agent group ignores the wildcard User-agent: * block completely. Omitting Disallow: /api/ inside a named PerplexityBot or OAI-SearchBot block exposes /api/ to those crawlers even when User-agent: * blocks /api/.

# Policy: block model training crawlers, allow AI search citation bots
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /api/
Disallow: /admin/

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml
# LLMs: https://www.example.com/llms.txt

Googlebot stays allowed under User-agent: * because Google Search, Google AI Overviews, and Google AI Mode all depend on Googlebot indexing. Google-Extended is left out of the block group above because Google Search Central documents that disallowing Google-Extended blocks both Gemini model training and grounding links in Gemini Apps. On bot.spot, built by Botdot Oy in Piikkiö, Finland, /robots.txt allows all search and AI crawlers under User-agent: *, blocks /api/, sets Host and Sitemap to https://www.bot.spot/sitemap.xml, and adds an # LLMs: comment pointing to https://www.bot.spot/llms.txt, as explained in our llms.txt guide.

How do edge gateways and AI visibility tracking connect crawl logs to citations?

Edge gateways and prompt tracking are the two layers that turn raw AI crawl logs into measurable pipeline. An HTTP 200 log line from OAI-SearchBot, Claude-SearchBot, or PerplexityBot proves only that a bot fetched your HTML. Static HTML cannot serve real-time inventory, render interactive product comparisons, or submit a validated lead form inside a chat thread.

bot.gateway is your brand's public portal for AI agents, giving ChatGPT, Claude, and other assistants a branded endpoint to browse products, search content, and capture leads with Enterprise rollout agreed with sales. To measure whether crawler access produces citations, bot.radar tracks mention rate, citation rate, and share of voice across ChatGPT, Claude, Perplexity, Gemini, Copilot, Google AI Overviews, and Google AI Mode so growth teams tie technical fixes to answer visibility.

What is the summary checklist for governing AI crawlers?

The summary checklist for governing AI crawlers consists of five repeatable checks across every production domain and subdomain:

1. Parse seven days of CDN or Nginx HTTP access logs with rg and awk to count hits and status codes for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
2. Check crawler source IPs against openai.com/searchbot.json, openai.com/gptbot.json, claude.com/crawling/bots.json, and www.perplexity.ai/perplexitybot.json to filter spoofed scrapers at your WAF.
3. Split /robots.txt directives by job on every subdomain and repeat Disallow: /api/ and Disallow: /admin/ inside each named User-agent block under RFC 9309.
4. Test priority URLs with curl to verify that titles, pricing tables, and JSON-LD render in raw server HTML without client-side JavaScript.
5. Monitor buyer prompts in bot.radar and deploy a public agent portal through bot.gateway to convert AI crawl traffic into cited answers and leads.