GEO-300 · Lesson 2 of 12 · Module 1: Let AI In · 3:03

Bot policy as code

Bot policy is one decision per job, written as code and tested. robots.txt (RFC 9309) groups rules by product token, applies the longest matching path, and can announce your sitemap; most vendors cache it for about a day. Decide per job: search indexers and user-triggered fetchers bring answers and visits; training is a business choice; control tokens such as Google-Extended set AI use without touching the crawl. Your CDN or WAF adds a second layer: allow verified bots by category, block the unverified. Newer signals (Content Signals, the IETF AIPREF drafts) are proposals, and some user fetchers say robots.txt may not apply to them. Keep the policy in version control with a test for every allowed token.

Frame from lesson 2 of Sophyx Certified Technical GEO Specialist – Professional, "Bot policy as code"

What you will learn

  • Write a bot decision table by job
  • Express the table as robots.txt and CDN rules
  • Test the policy before it ships

The one thing to remember

One decision per job. Policy as code, with tests.

Common mistake: A "block all bots" WAF rule to stop scrapers

It returns 403 to the verified search and user fetchers too, so answers cite someone else.

The fix: Block the unverified; allow verified bots by category, and decide training on its own.

Key terms

Decision table
One row per bot job and token: allow, block or limit, and why.
Control token
A robots.txt name that sets how a crawl may be used for AI. It fetches nothing.
Longest match
In robots.txt, the most specific matching Allow or Disallow path wins.
Verified-bot rule
A CDN or WAF rule that acts on a verified bot category, not on a user agent string.
Content signal
A proposed robots.txt line stating search, AI-input and AI-train preferences.

Part of Technical GEO Specialist (GEO-300)

For developers and technical SEOs: build a site AI can fetch, read and act on, and keep it that way.

12 lessons, practice quizzes, a final exam and the Sophyx Certified Technical GEO Specialist – Professional certificate. Regular price USD 129. Your first certification is free. Free with any paid Sophyx plan.

Start the course free
Book a Demo