Training crawler noise vs search indexing on small sites
Why the Training tab exists, how noisy it is, and when to ignore it for business decisions.
training-botscrawlersseo
5 min read
AI & search crawlers on indie sites
When ChatGPT, Perplexity, and Googlebot hit your docs, and what to do with the counts.
Main guide
AI and search crawler hits on documentation-heavy indie sites
What counts as indexing vs answer fetches vs training noise, and how to read 24h charts without panic.
5 min read
Googlebot crawl bursts after sitemap updates and doc deploys
Expect indexing spikes 6–48h after shipping, how to read them on a 24h chart.
googlebotsitemapindexing
5 min read
OpenAI and Perplexity user-agent patterns on indie marketing sites
Recognize fetchers in logs, separate them from Googlebot, and avoid blocking paying customers by mistake.
openaiperplexityuser-agent
5 min read