Skip to main content

​
New features

​
Crawler analytics

Track requests from identified AI crawlers in connected server logs: At this release, the interface grouped more than 20 crawler and policy labels:
  • OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
  • Anthropic: ClaudeBot, Claude-Web
  • Google: Google-Extended, Googlebot (AI features)
  • Perplexity: PerplexityBot
  • Meta: FacebookBot, Meta-ExternalAgent
  • Microsoft: Bingbot (Copilot), MSNBot
  • Others: Cohere, Applebot, YandexBot, Bytespider, and more
For each crawler:
  • Crawl frequency and trends
  • Pages visited
  • Comparison to previous periods
How to interpret the data:
  • A crawl records a request to your site. It does not prove indexing, model training, or use in an answer.
  • If crawl volume declines, check the selected period, log collection, robots.txt rules, and site availability.
  • Differences by crawler show observed request patterns, not provider preference.
This historical list mixed observable HTTP User-Agent identities with policy or general-purpose identities. Current Crawler Analytics reports supported observable AI-specific User-Agent identities. Google-Extended is a robots-control token, not a crawler User-Agent, and Googlebot does not identify a separate AI-feature visit. Use the current Crawler Analytics guide for supported interpretation.

​
Referrer analytics

Track traffic from AI conversations (powered by GA4): Compare periods week over week and month over month. Differences between platforms describe the connected analytics dataset and may not represent every user or visit.

​
Site diagnostics

Audit supported pages for AI-readiness across four dimensions: Technical score (0-100)
  • Crawlability, robots.txt compliance
  • Schema markup, structured data
  • Canonical tags, XML sitemap
Content score (0-100)
  • Readability, heading structure
  • Content depth, keyword relevance
  • Update frequency, internal linking
AEO score (0-100)
  • Answer Engine Optimization
  • FAQ sections, clear definitions
  • Structured answers AI can extract
Performance score (0-100)
  • PageSpeed Insights integration
  • Core Web Vitals, loading speed
  • Mobile-friendliness
Page-level issues:
  • Each issue shows severity, type, and how to fix
  • Severity and priority labels for triage

​
Improvements

  • Unified Analytics module: All intelligence in one place
  • Period comparisons: Week and month trends
  • Export capabilities: Download supported analytics datasets