GEO-310 · Lesson 4 of 12 · Module 2: Collecting the Data · 3:03
Sampling AI answers
The same prompt gives different answers on different runs, even with settings that sound deterministic, so a single answer is an anecdote. Sample: ask each prompt several times (5 for a quick look, 10 to confirm, 15 to be rigorous), in clean sessions, spread over days, in each assistant you care about. Report every rate with a confidence interval, and call a change real only when the intervals separate. Log every run.

What you will learn
- Plan repeat runs per prompt and per assistant
- Run prompts in clean, comparable sessions
- Report rates with confidence intervals
The one thing to remember
One answer is an anecdote. Ten are a measurement.
Common mistake: Asking once, in your own logged-in account
The answer reflects your history and location, and a single run cannot show how often it happens.
The fix: Use clean sessions, repeat each prompt, spread the runs over days, and report an interval.
Key terms
- Run
- One asking of one prompt in one assistant at one time; the smallest unit you log.
- Confidence interval
- The range a rate could plausibly sit in, given how many runs you made.
- Clean session
- A run with no memory, history or personalisation and a fixed location, so runs are comparable.
Part of GEO Analytics Professional (GEO-310)
For analytics and growth teams: measure AI visibility from crawler reads to conversions, with numbers you can defend.
12 lessons, practice quizzes, a final exam and the Sophyx Certified GEO Analytics Professional certificate. Regular price USD 149. Your first certification is free. Free with any paid Sophyx plan.
Start the course free