Technical GEO Engineer

AI companies run bots for three jobs: training crawlers, search indexers, and user-triggered fetchers that fetch a page the moment a person asks an assistant something. Each bot names itself with a product token inside its user agent string; Google-Extended and Applebot-Extended are control tokens, not crawlers. A token is only text, so verify a bot before you trust it: the vendor's published IP list, reverse then forward DNS, or a signed request (Web Bot Auth, an IETF draft). Bot names and behaviours are as of October 2026.

Bot policy is one decision per job, written as code and tested. robots.txt (RFC 9309) groups rules by product token, applies the longest matching path, and can announce your sitemap; most vendors cache it for about a day. Decide per job: search indexers and user-triggered fetchers bring answers and visits; training is a business choice; control tokens such as Google-Extended set AI use without touching the crawl. Your CDN or WAF adds a second layer: allow verified bots by category, block the unverified. Newer signals (Content Signals, the IETF AIPREF drafts) are proposals, and some user fetchers say robots.txt may not apply to them. Keep the policy in version control with a test for every allowed token.

A fetcher receives one HTTP response. Googlebot and Applebot render JavaScript; the crawlers and user fetchers from OpenAI, Anthropic and Perplexity did not in the largest published study (Vercel and MERJ, 2024), and none of them documents rendering as of October 2026, so a page whose facts arrive only through scripts is empty to them. (Browser agents are a different case: Lesson 07.) Test it: fetch the page as the bot with JavaScript off and read the HTML. Serve the facts in that first response with server-side rendering, static generation or pre-rendering, and keep the status honest: a 200 for real pages, a real 404 for missing ones, no consent walls, interstitials or geo-blocks in front of bots, and no "soft 404" shells.

Structured data is a graph, not a sprinkle of snippets. In JSON-LD, give every real-world thing one node with a stable @id, reuse that @id on every page that mentions it, and connect the nodes: the organisation, its website, its locations, its products and offers, its people. Link each entity to its official profiles with sameAs. Keep the markup true to the visible page, because Google ignores or penalises markup for things users cannot see, and validate it on every deploy. Google says no new markup is required for its AI features (as of 2026); the graph still removes ambiguity about who you are.

AI systems quote passages, so write the page as passages. One question per section, answered in the first sentence under a heading that says the question; article and section elements with a clean h1–h6 order; tables for specifications and comparisons; ordered lists for steps; time elements for dates; a lang attribute; link text that names its target. Text inside tabs and accordions counts when it is in the HTML, not when it is loaded on click. Specs in images, PDFs and carousels are invisible: put them in text.

Discovery files tell bots what you have and what changed. A sitemap lists canonical URLs with an honest lastmod (50,000 URLs and 50 MB per file; index files above that), announced with a Sitemap line in robots.txt. IndexNow pushes changed URLs to Bing, Yandex, Naver, Seznam, Yep and Amazon, but not to Google. Feeds (RSS, Atom) and HTTP validators (ETag, Last-Modified) tell fetchers what is fresh. llms.txt is a proposal: a markdown map of your key pages; as of 2026 no major assistant documents reading it when it answers, and Google says its AI features need no AI text files, so publish it cheaply and judge it by the fetches in your own logs.

AI agents (as of October 2026: ChatGPT's cloud browser, Claude's computer use, Google-Agent) act through the page the way assistive technology does: they read the accessibility tree and operate real controls. Give them real forms with labels, buttons that are buttons, links with names, visible state and error messages, keyboard paths with no hover-only menus, and a checkout that does not hide steps behind scripts or unlabelled icons. Let verified agents through your bot defences, and keep consent walls off the task path. The test is the walkthrough: find, compare, act.

Assistants and agents can call you directly. Expose one source of truth as an API: an OpenAPI description, read-only endpoints for facts such as price, stock, hours and specs, and the same data your pages show. The Model Context Protocol (MCP) standardises how an assistant connects: a server offers tools and resources over Streamable HTTP, with OAuth 2.1 for anything personal, and consent before any action. Rate-limit, log and version it, and keep it in sync with the site, because two sources of truth make two different answers. App directories are built on MCP servers (as of October 2026, ChatGPT's plugin directory, which replaced its app directory in July 2026); treat their rules as dated.

Fetchers give up fast. Keep time to first byte low, serve bots from the edge cache, and use Cache-Control, ETag and 304 so unchanged pages cost nothing. Google shrinks its crawl when responses slow down or return 5xx and 429, and a robots.txt that errors can hold crawling back altogether. Rate limits should spare verified bots; redirects should land in one hop; Core Web Vitals (LCP 2.5 s, INP 200 ms, CLS 0.1 at the 75th percentile) keep the human side fast. Monitor with synthetic fetches as the bots.

Turn raw logs into a verified bot feed. Parse CDN or server logs, classify each request by product token and job, then verify: match the IP against the vendor lists you refresh daily, do reverse and forward DNS with a cache, and check signatures where bots sign. Store the result with the status code and the path. Then alert on what matters to an engineer: 403, 429 or 5xx sent to a verified bot, a new token appearing, a key page no longer fetched, robots.txt fetches failing. Keep IP retention short. (GEO-310 reads the same feed for metrics; this lesson builds it.)

A click from an AI answer only shows as an AI visit if the plumbing keeps the source attached. Browsers default to strict-origin-when-cross-origin, sending only the origin across sites and nothing on an HTTPS-to-HTTP step, so serve HTTPS everywhere. ChatGPT appends utm_source=chatgpt.com; a redirect that drops the query string, a canonicalisation step, a consent wall or a geo-redirect to the home page loses it. Make sure landing pages exist, log the referrer and the UTM on the server, and test the path from an answer link to the final page. GA4's AI Assistant channel (as of 2026) can only see what survives.

Everything in this course can be undone by one deploy. Put the checks in CI: robots.txt allow tests per token, a fetch of each key page as a bot with JavaScript off that fails on an empty shell, structured-data validation against the entity graph, sitemap validity and lastmod sanity, a TTFB budget, a referral test, and a check that staging rules never ship. Give each policy an owner, keep a change log, and review the vendor documentation quarterly, because tokens and rules change. GEO is operated, not launched.