GEO-300 · Lesson 9 of 12 · Module 3: Ready for Agents · 2:58

Fast, reliable responses

Fetchers give up fast. Keep time to first byte low, serve bots from the edge cache, and use Cache-Control, ETag and 304 so unchanged pages cost nothing. Google shrinks its crawl when responses slow down or return 5xx and 429, and a robots.txt that errors can hold crawling back altogether. Rate limits should spare verified bots; redirects should land in one hop; Core Web Vitals (LCP 2.5 s, INP 200 ms, CLS 0.1 at the 75th percentile) keep the human side fast. Monitor with synthetic fetches as the bots.

Frame from lesson 9 of Sophyx Certified Technical GEO Specialist – Professional, "Fast, reliable responses"

What you will learn

  • Measure and cut time to first byte for bot requests
  • Use caching and 304 for unchanged pages
  • Keep errors, rate limits and redirects from shrinking the crawl

The one thing to remember

Fast, cached, and never a 5xx to a verified bot.

Common mistake: Rate limiting by IP, so verified bots get 429 at peak

The crawler reads 429 as "slow down" and fetches less for weeks.

The fix: Rate limit the unverified; give verified bots their own lane and cache their pages at the edge.

Key terms

Time to first byte
How long a fetcher waits before the first byte of the response arrives.
Conditional request
A fetch with If-None-Match or If-Modified-Since, answered with 304 when nothing changed.
Crawl capacity
How much a crawler is willing to fetch from you; it falls with slow responses and errors.

Part of Technical GEO Specialist (GEO-300)

For developers and technical SEOs: build a site AI can fetch, read and act on, and keep it that way.

12 lessons, practice quizzes, a final exam and the Sophyx Certified Technical GEO Specialist – Professional certificate. Regular price USD 129. Your first certification is free. Free with any paid Sophyx plan.

Start the course free
Book a Demo