GEO-300 · Lesson 9 of 12 · Module 3: Ready for Agents · 2:58
Fast, reliable responses
Fetchers give up fast. Keep time to first byte low, serve bots from the edge cache, and use Cache-Control, ETag and 304 so unchanged pages cost nothing. Google shrinks its crawl when responses slow down or return 5xx and 429, and a robots.txt that errors can hold crawling back altogether. Rate limits should spare verified bots; redirects should land in one hop; Core Web Vitals (LCP 2.5 s, INP 200 ms, CLS 0.1 at the 75th percentile) keep the human side fast. Monitor with synthetic fetches as the bots.

What you will learn
- Measure and cut time to first byte for bot requests
- Use caching and 304 for unchanged pages
- Keep errors, rate limits and redirects from shrinking the crawl
The one thing to remember
Fast, cached, and never a 5xx to a verified bot.
Common mistake: Rate limiting by IP, so verified bots get 429 at peak
The crawler reads 429 as "slow down" and fetches less for weeks.
The fix: Rate limit the unverified; give verified bots their own lane and cache their pages at the edge.
Key terms
- Time to first byte
- How long a fetcher waits before the first byte of the response arrives.
- Conditional request
- A fetch with If-None-Match or If-Modified-Since, answered with 304 when nothing changed.
- Crawl capacity
- How much a crawler is willing to fetch from you; it falls with slow responses and errors.
Part of Technical GEO Specialist (GEO-300)
For developers and technical SEOs: build a site AI can fetch, read and act on, and keep it that way.
12 lessons, practice quizzes, a final exam and the Sophyx Certified Technical GEO Specialist – Professional certificate. Regular price USD 129. Your first certification is free. Free with any paid Sophyx plan.
Start the course free