AE-200 · Lesson 4 of 12 · Module 2: Be Readable · 2:31
Can AI load you?
Before AI can mention you, its crawlers must load your pages. robots.txt decides which crawlers may fetch which paths. Many AI crawlers read the raw HTML and do not run JavaScript, so content that appears only after scripts run may be invisible. Login walls, errors, slow responses and blocked bots stop them too. Some controls are separate: for example, OpenAI lists GPTBot (training) and OAI-SearchBot (search) separately, and Google-Extended governs use for Gemini, not Google Search.

What you will learn
- Audit robots.txt for AI crawlers
- Check that key facts are in the server HTML
- Find blockers: login walls, heavy scripts, errors and timeouts
The one thing to remember
If AI can't load you, you don't exist.
Key terms
- User agent
- The name a crawler sends with each request, used in robots.txt rules and server logs.
- robots.txt
- A file at the root of your site that tells crawlers which paths they may fetch.
- Server-side rendering
- Sending the finished HTML from the server, so content is readable without running JavaScript.
Part of Answer Engine Specialist (AE-200)
How AI answers are made, and how to get a brand named in them. The core course, and the place to start.
12 lessons, practice quizzes, a final exam and the Certified Answer Engine Specialist certificate. Regular price USD 89. Your first certification is free. Free with any paid Sophyx plan.
Start the course free