Your docs site gets traffic you did not earn on Hacker News. Bots fetch pages for search indexes, answer engines, and opaque training pipelines. Human pageviews stay flat while server logs scream. Founders either ignore bots entirely or treat every hit as “AI SEO”, both waste the signal.
This guide covers AI crawler guides: three lenses (AI answers, Indexing, Training) on one timeline, classified from User-Agent at ingest, no extra script.
Why docs attract disproportionate bot share
- Stable URLs and headings.
- Technical vocabulary models associate with answers.
/docs,/changelog,/apipaths are crawl candy.
Marketing home may be 80% human; docs may be 80% bot. Split properties mentally even on one domain.
AI answers vs indexing
Indexing (Googlebot, Bingbot): build search snippets. AI answers (OpenAI, Perplexity fetchers): retrieve for chat answers. Spikes after you publish major docs may show in both tabs, compare timing.
Training bucket honesty
“Training” classification is heuristic. Unknown aggressive crawlers land here. Do not sue OpenAI based on a graph, use counts to rate-limit or robots.txt decisions.
24h chart reading
Hourly buckets show deploy effects: push sitemap → Google burst 6h later. If human signups lag bot spike, docs are indexed, not yet trusted by people.
Sidebar crawler ranking
Google 1.9k, OpenAI 107, Perplexity 4, typical indie docs skew. Compare week-over-week, not to enterprise blogs.
robots.txt levers
Disallowing all bots kills SEO. Target paths (/internal/) not whole docs. Log before block.
Rate limits and hosting bills
Bot surges cost bandwidth on static docs. CDN caching helps; analytics helps you notice day one.
Pair with human metrics
Docs bot spike + signup paths from /docs → measure human outcome, not bot vanity.
Siblings
Hub
/blog/topics/ai-crawler-visibility
Bots are not customers. They are weather on your docs, measure weather instead of guessing from Cloudflare totals alone.
Classifying at ingest without a second script
Indie stacks should not bolt on a separate “bot analytics” product. User-Agent classification at pageview ingest keeps one timeline: humans, indexing, AI answers, training heuristics. Your marketing site and docs share the same tracker; filter path starts with /docs when reading crawler charts so home hero RPV does not get interpreted as bot weather.
Docs IA decisions bots amplify
When you add fifty API reference pages at once, bots crawl faster than humans read. Expect Googlebot burst within hours and OpenAI/Perplexity fetchers within days on niches with heavy AI search interest. Shipping docs is still correct; panic at hour six is not.
robots.txt vs business outcomes
Blocking GPTBot reduces AI answers tab counts; it does not automatically improve human signups. If human documentation paths before checkout already convert, public docs are a feature. If confidential SDK details leak in docs, block paths not whole properties.
Comparing tabs week over week
| Tab | Question it answers |
|---|---|
| Indexing | Is search infrastructure reading new URLs? |
| AI answers | Are answer products fetching us? |
| Training | Is unknown bulk traffic spiking? |
Week-over-week rate of change matters more than absolute counts on a 2k-visitor site. Doubling OpenAI from 40 to 80 hits is note-worthy; arguing about 40 vs 45 is not.
Hosting and CDN implications
Bot surges raise egress bills on uncached markdown-heavy docs. Cache Cache-Control on static doc builds; bots respect caches differently than humans but still reduce origin load. Analytics noticing day-one spike saves a surprise invoice more than Twitter threads about AI.
Security scanning noise
Some “bots” are vulnerability scanners hitting /wp-admin on your Next.js site. They may land in Training bucket heuristics. Path filters exclude nonsense routes from doc-focused reviews.
Human conversion overlay
Build a simple weekly table:
- Docs bot hits (Indexing + AI answers)
- Human sessions landing on
/docs/* - Paid conversions with doc paths in session (from path funnel guides)
If row three grows with row one, bots and humans may both care about the same new content, good problem.
Press and launch interaction
Tech press linking your API reference drives humans and bots together. Pin the week on marketing pulse guides; do not compare bot charts to quiet baseline without label.
When to escalate to engineering
- Training tab 10× for 7+ days with odd paths (
/.env,/config) - Googlebot flat zero after deploy (accidental
Disallow: /) - AI answers exceed Google on non-doc paths (possible scrape)
Otherwise log monthly and return to RPV work.
Stakeholder translation
“Google read our new webhook docs heavily this week; Search Console clicks may move in 2–4 weeks. Human checkout paths including /docs/webhooks rose 12%, keep docs public.”
That is a complete update. Bot counts alone are not.
Sibling deep dives
Founder weekly ritual (5 minutes)
Monday coffee: glance 24h chart for spikes, check Indexing trend, ignore Training unless 10×. Return to shipping. Crawler visibility informs infra and SEO timing, not quarterly pricing strategy.
Changelog vs reference crawl mix
Changelogs attract existing customers and bots re-fetching on every deploy. Split /changelog from /docs in crawler reports. A changelog-only burst should not trigger “rewrite API docs for SEO”, it is release noise.
International doc locales
/fr/docs and /en/docs double bot volume without doubling human TAM. Locale splits in analytics prevent misreading France bot traffic as France buyer interest.
Log retention and compliance
Raw access logs with IP may have shorter retention than analytics aggregates. If you investigate abuse, export once, then delete, do not turn bot investigation into accidental PII archive.
Integration with launch checklists
Before major launch, add line item: “Expect Indexing burst 24–72h; do not interpret as human launch failure.” Aligns docs team with marketing nerves.
Comparison to Cloudflare “bot fight mode”
Aggressive WAF may block legitimate fetchers and break search previews. Analytics classification is observe-first; firewall is act-second. Change one lever per week.
Long-tail doc pages
Hundreds of thin doc pages create crawl surface area. Bots index them; humans never land. Consolidate or noindex orphan pages when Search Console shows impressions without clicks for quarters, saves crawl budget and clarifies charts.
API versioning pages
/docs/v1 vs /docs/v2 parallel bursts during migration are normal. Pin migration month; compare human paths only on v2 after redirect map stable.
When docs are behind login
Bots stop; humans pre-sale cannot read. RPV may rise on sales-assisted deals while self-serve falls, different business model, not crawler failure.
Closing table: who pays rent
| Audience | Metric |
|---|---|
| Humans | RPV, checkout, trials |
| Googlebot | Indexing tab, GSC later |
| Answer bots | AI answers tab, optional CTAs |
| Training bucket | Hosting, security |
Keep the table taped to your monitor until instincts match.
One sentence for your next standup
“Bot traffic on docs rose after the sitemap push; human signup paths through docs are stable, no action.” If you cannot say that, open the path funnel guide before changing robots.txt. Return to the AI crawler guides when you need the full topic list.