You ship a new /docs/api section Friday night. Saturday morning the Indexing tab shows a wall of Googlebot hits. Humans have not arrived yet. That is often normal crawl queue behavior, not instant SEO success.
AI crawler guides guides.
Sitemap ping habits
Submit sitemap in Search Console after large doc changes. Burst timing correlates with ping + internal link graph depth.
Staging vs production
Accidental staging index creates bot burst on wrong host, password-protect staging; verify host filter in analytics.
24h chart shape
Ramp up, plateau, decay over 72h typical for medium sites. Flatline at zero after burst may mean robots block.
Search Console lag
Clicks trail crawls by days or weeks. Bot chart is leading indicator; RPV is lagging for organic.
Internal links matter
One new doc without links from /docs hub may see weak burst, information architecture is crawl logic.
Compare Bingbot
Smaller burst amplitude. If Google only spikes, Google-specific sitemap issue unlikely, just scale.
Bandwidth
Large PDF docs in sitemap inflate burst bytes. Host PDFs on CDN with cache.
Do not confuse with DDoS
Sustained 10× baseline for weeks ≠ sitemap burst. Check IPs and paths.
Hub
Crawl burst is Google reading your homework. Grades arrive later in Search Console and revenue.
What triggers a burst (checklist)
- New
sitemap.xmlURLs submitted in Search Console - Large docs deploy with internal links from hub page
- Domain migration with redirect maps
- Accidental republish of all pages (build bug)
- New subdirectory
/docs/v2linked from footer
Each deserves a pin note so you remember why Indexing tab jumped.
Shape of a healthy burst
Hour 0–6: ramp as Google discovers URLs. Hour 6–48: peak fetch rate. Day 3–7: decay toward baseline unless you keep publishing. Plateau at 3× baseline for months is not a sitemap burst, investigate crawl budget waste or infinite URL spaces (?page= loops).
Search Console pairing
Indexing burst in analytics → Coverage and Pages reports weeks later. Clicks trail crawls. Do not raise prices because bots read docs; do not cut docs because clicks lag, see training vs indexing for what not to confuse.
Staging leak burst
Staging indexed is a classic burst on staging.yoursite.com while production looks quiet. Password gate staging; noindex on preview deploys; filter hostname in analytics before screaming about SEO win on wrong host.
Sitemap hygiene
- Split sitemaps when >50k URLs (rare on indie).
- Remove 404 URLs from sitemap same day.
- Include
lastmodonly when content actually changed, fake dailylastmodtrains nobody well.
Internal linking drill
New doc without link from /docs hub → weak burst. Add hub link before ping sitemap. Documentation before checkout paths often start at hub navigation humans also use.
Bing and secondary indexers
Bingbot bursts are smaller amplitude; same timing idea. If Google bursts and Bing flat, unlikely you broke Bing, scale difference.
AI fetchers after Google
Order of operations often: Google indexing burst first, then OpenAI/Perplexity fetches on pages Google already deemed real. Pin same week across AI visibility guides.
Bandwidth math (rough)
10k Googlebot fetches × 80kb HTML ≈ 800MB. On CDN with cache, origin sees less. On origin-only docs, burst matters, add cache headers before adding bot block rules.
DDoS differentiation
| Signal | Sitemap burst | Attack |
|---|---|---|
| User-Agent | Googlebot | mixed |
| Paths | known URLs | random |
| Duration | days | hours–weeks |
| Geographic | Google IP ranges | scattered |
Cloudflare “under attack” mode hurts real users, verify before enabling.
Revenue expectations
Organic RPV moves on quarter scale for small sites, not burst week. Pair burst week pin with 28d RPV review next month.
Post-burst actions
- Verify URLs return 200.
- Check Search Console for crawl errors.
- Note burst end date on pin.
- Ship human-facing CTA on top crawled doc page.
- Return to writing, not refreshing Indexing tab hourly.
Soft 404 and burst waste
Pages that return 200 with “not found” content burn crawl budget. Burst hits empty shells; Search Console shows soft 404 later. Fix templates before sitemap ping.
Canonical tags during burst
Duplicate URLs (/docs/page and /docs/page/) split burst across duplicates. Canonical to one URL in HTML; sitemap lists canonical only.
JavaScript rendering
If docs require client render, Googlebot may burst twice, raw HTML pass and rendered pass. Static generation for docs reduces double-fetch and clarifies burst timing.
Monitoring alerts (sensible thresholds)
Alert if Indexing tab >5× 7d median and paths are mostly /docs/* after known deploy, informational Slack, not PagerDuty. Unknown paths trigger security review.
Comparing week of burst to email send
Email send spikes humans; sitemap spikes bots. Both can overlap on launch week. Pin notes disambiguate: “Tue email, Wed sitemap ping.”
Redirect chains
/old-doc → /docs/new → 302 loops stop bursts and hurt SEO. Test redirects with curl before sitemap update.
XML sitemap index files
Large doc sets use sitemap index pointing to child sitemaps. Burst may stagger per child file, multi-day plateau normal.
Log sampling for verification
During burst, sample 20 Googlebot hits: status code, TTFB, cache HIT/MISS. Slow origin extends burst duration and costs money.
Content freshness myths
Updating lastmod daily without content changes does not trick Google into loving you. Ship real doc improvements; ping sitemap when URLs are new or materially changed.
Post-burst human content tasks
Top five crawled URLs this week → add “Related guides” and pricing CTA where appropriate. Bots already validated URL importance; humans benefit from navigation.
Integration with product launches
Launch blog post + new doc section + sitemap ping = triple spike (humans, Google, AI fetchers). Expect crowded charts; use marketing pulse guides narrative.
When burst never comes
No burst after major deploy suggests: robots block, sitemap omit, or site not in Search Console property. Fix discovery before blaming “Google hates us.”
Copy for your launch retro doc
Add a single line under SEO: “Indexing burst expected days 1–3 post-sitemap; GSC clicks reviewed on day 21.” Future launches inherit the template and fewer people refresh the crawler chart at midnight.
Pair burst week with human landing RPV
When Indexing spikes on /docs/api, check whether human landers on the same path rose in the same pin, if yes, ship a clearer upgrade CTA; if no, wait for Search Console before rewriting titles. Crawl leading, clicks lagging, revenue lagging more is normal on indie doc sites.
Pair with path funnels
Converters who touch docs before checkout are the human signal that matters alongside any bot wall on the Indexing tab, filter paid sessions in documentation before checkout.
Sitemap ping timing
Many teams ping Search Console Friday evening and wonder why Monday humans are quiet while bots roar, that is the expected sequence, not a failed launch. Hold product and pricing decisions until human metrics move on their own lag. Document the ping date in the same note as the deploy SHA so burst charts stay explainable months later.