GEO-310 · Lesson 4 of 12 · Module 2: Collecting the Data · 3:03

Sampling AI answers

The same prompt gives different answers on different runs, even with settings that sound deterministic, so a single answer is an anecdote. Sample: ask each prompt several times (5 for a quick look, 10 to confirm, 15 to be rigorous), in clean sessions, spread over days, in each assistant you care about. Report every rate with a confidence interval, and call a change real only when the intervals separate. Log every run.

Frame from lesson 4 of Sophyx Certified GEO Analytics Professional, "Sampling AI answers"

What you will learn

  • Plan repeat runs per prompt and per assistant
  • Run prompts in clean, comparable sessions
  • Report rates with confidence intervals

The one thing to remember

One answer is an anecdote. Ten are a measurement.

Common mistake: Asking once, in your own logged-in account

The answer reflects your history and location, and a single run cannot show how often it happens.

The fix: Use clean sessions, repeat each prompt, spread the runs over days, and report an interval.

Key terms

Run
One asking of one prompt in one assistant at one time; the smallest unit you log.
Confidence interval
The range a rate could plausibly sit in, given how many runs you made.
Clean session
A run with no memory, history or personalisation and a fixed location, so runs are comparable.

Part of GEO Analytics Professional (GEO-310)

For analytics and growth teams: measure AI visibility from crawler reads to conversions, with numbers you can defend.

12 lessons, practice quizzes, a final exam and the Sophyx Certified GEO Analytics Professional certificate. Regular price USD 149. Your first certification is free. Free with any paid Sophyx plan.

Start the course free
Book a Demo