GEO-310 · Lesson 5 of 12 · Module 2: Collecting the Data · 3:08
Reading crawler logs
A crawler read is a line in your server or CDN logs: IP address, time, request, status, bytes, referrer and user agent. Identify AI crawlers by user agent, then verify them against the published IP ranges, because user agents are easy to fake. Separate three kinds: training crawlers, search indexers and user-triggered fetchers, which mean someone asked about you right now. Then analyse: which pages are fetched, with which status codes, how fresh each key page is, and how volume moves over time.

What you will learn
- Read a log line and pick out the crawler
- Verify AI crawlers against published IP ranges
- Analyse fetches by page, status and freshness
The one thing to remember
Every fetch is a line. Read the lines.
Common mistake: Counting every "GPTBot" line as OpenAI
Scrapers borrow famous user agents; unverified counts inflate your reads and hide blocks.
The fix: Verify each AI user agent against its published IP ranges and count only verified fetches.
Key terms
- User agent
- The name a client sends with each request, such as GPTBot or ClaudeBot; easy to fake, so verify it.
- User-triggered fetch
- A request made because a person asked an assistant something right now, such as ChatGPT-User or Claude-User.
- Freshness
- Days since a crawler last fetched a page; stale fetches mean stale answers.
Part of GEO Analytics Professional (GEO-310)
For analytics and growth teams: measure AI visibility from crawler reads to conversions, with numbers you can defend.
12 lessons, practice quizzes, a final exam and the Sophyx Certified GEO Analytics Professional certificate. Regular price USD 149. Your first certification is free. Free with any paid Sophyx plan.
Start the course free