GEO-310 · Lesson 5 of 12 · Module 2: Collecting the Data · 3:08

Reading crawler logs

A crawler read is a line in your server or CDN logs: IP address, time, request, status, bytes, referrer and user agent. Identify AI crawlers by user agent, then verify them against the published IP ranges, because user agents are easy to fake. Separate three kinds: training crawlers, search indexers and user-triggered fetchers, which mean someone asked about you right now. Then analyse: which pages are fetched, with which status codes, how fresh each key page is, and how volume moves over time.

Frame from lesson 5 of Sophyx Certified GEO Analytics Professional, "Reading crawler logs"

What you will learn

  • Read a log line and pick out the crawler
  • Verify AI crawlers against published IP ranges
  • Analyse fetches by page, status and freshness

The one thing to remember

Every fetch is a line. Read the lines.

Common mistake: Counting every "GPTBot" line as OpenAI

Scrapers borrow famous user agents; unverified counts inflate your reads and hide blocks.

The fix: Verify each AI user agent against its published IP ranges and count only verified fetches.

Key terms

User agent
The name a client sends with each request, such as GPTBot or ClaudeBot; easy to fake, so verify it.
User-triggered fetch
A request made because a person asked an assistant something right now, such as ChatGPT-User or Claude-User.
Freshness
Days since a crawler last fetched a page; stale fetches mean stale answers.

Part of GEO Analytics Professional (GEO-310)

For analytics and growth teams: measure AI visibility from crawler reads to conversions, with numbers you can defend.

12 lessons, practice quizzes, a final exam and the Sophyx Certified GEO Analytics Professional certificate. Regular price USD 149. Your first certification is free. Free with any paid Sophyx plan.

Start the course free
Book a Demo