GEO-310 · Lesson 9 of 12 · Module 3: Analysis · 2:54
Experiments that hold up
A change followed by a better number proves little: platforms shift, seasons move, and answers vary. Run experiments that hold up: change one thing on a set of pages, keep a control group of similar pages unchanged, measure both before and after (difference-in-differences), use enough runs and weeks to beat the noise, and write the metric, the threshold and the decision date before you start. Judge wins by visits and conversions, not mentions alone.

What you will learn
- Design a page experiment with a control group
- Read a before-and-after with difference-in-differences
- Write a decision rule before the test starts
The one thing to remember
Change one thing. Keep a control. Decide in advance.
Common mistake: Changing every page at once, then reading next month's number
With no control and many changes, you cannot tell what worked, or whether anything did.
The fix: Change one thing on half the pages, keep the rest as a control, and compare the two differences.
Key terms
- Control group
- Pages you leave unchanged, so you can see what would have happened anyway.
- Difference-in-differences
- The change in the test pages minus the change in the control pages.
- Decision rule
- The metric, the threshold and the date, written before the experiment.
Part of GEO Analytics Professional (GEO-310)
For analytics and growth teams: measure AI visibility from crawler reads to conversions, with numbers you can defend.
12 lessons, practice quizzes, a final exam and the Sophyx Certified GEO Analytics Professional certificate. Regular price USD 149. Your first certification is free. Free with any paid Sophyx plan.
Start the course free