GEO-300 · Lesson 1 of 12 · Module 1: Let AI In · 3:04
AI crawlers and fetchers
AI companies run bots for three jobs: training crawlers, search indexers, and user-triggered fetchers that fetch a page the moment a person asks an assistant something. Each bot names itself with a product token inside its user agent string; Google-Extended and Applebot-Extended are control tokens, not crawlers. A token is only text, so verify a bot before you trust it: the vendor's published IP list, reverse then forward DNS, or a signed request (Web Bot Auth, an IETF draft). Bot names and behaviours are as of October 2026.

What you will learn
- Sort AI bots by job
- Find bot tokens in your logs
- Verify who a bot really is
The one thing to remember
A name is a claim. Verify the proof.
Common mistake: Allowing or blocking bots by user agent alone
Fakes get in, and real bots get shut out.
The fix: Verify first. Then apply the rule for that bot's job.
Key terms
- Product token
- The bot's short name inside its user agent string, such as GPTBot.
- User-triggered fetcher
- A bot that fetches a page because a person asked an assistant right now.
- Reverse DNS check
- Look up the IP's host name, then confirm that name points back to the IP.
- Signed request
- The agent signs each request with a key your CDN can check.
Transcript
The brands in the worked examples are fictional, and the numbers are illustrative.
Gearwick Bikes opened its server logs. GPTBot: forty thousand requests this week.
Should they block it? First, find out if it's really OpenAI.
Technical GEO Specialist. Lesson one: AI crawlers and fetchers.
You will learn to sort AI bots by job, find their tokens in your logs, and verify who they really are.
First, a quick recall from Answer Engine Specialist. AI bots do three jobs. Can you name them?
Training. Search. And fetching a page the moment a user asks.
Every request carries a user agent string. Inside it is a short token: the bot's name.
OpenAI trains with GPTBot, builds its search index with OAI-SearchBot, and fetches for users with ChatGPT-User.
Anthropic follows the same pattern. Perplexity lists a search bot and a user fetcher.
Google-Extended and Applebot-Extended are different. They're tokens, not crawlers. They control how the normal crawl may be used for AI.
And user fetchers act for a person. Some vendors say robots.txt may not apply to them.
Here's the catch. A user agent is just text. Anyone can type GPTBot.
Pause. A request says it's GPTBot. What proves it?
Its IP address. OpenAI publishes the IP ranges its bots use.
There are three ways to check. One: the vendor's published IP list.
Two: reverse DNS. Look up the IP's host name, then confirm it points back. Google, Bing and Apple support this.
Three: signatures. Some agents sign every request, and your CDN can check the key.
Gearwick checked. Only one in four of those GPTBot requests came from OpenAI.
A common mistake: allowing or blocking bots by user agent alone. Fakes get in, and real bots get shut out.
Verify first. Then apply the rule for that bot's job.
Try it this week. Export a week of logs, group the AI requests by token and job, and verify your top three bots.
Before the recap: what do you check before you trust a bot?
Its job. Its token. And its proof: an IP list, reverse DNS, or a signature.
A name is a claim. Verify the proof.
Take the practice quiz. Next: bot policy as code.
Part of Technical GEO Specialist (GEO-300)
For developers and technical SEOs: build a site AI can fetch, read and act on, and keep it that way.
12 lessons, practice quizzes, a final exam and the Sophyx Certified Technical GEO Specialist – Professional certificate. Regular price USD 129. Your first certification is free. Free with any paid Sophyx plan.
Start the course free