AX-300 · Lesson 7 of 12 · Module 3: Make It Reliable · 3:13

Test it with evals

An eval is a repeatable test of the whole agent on real tasks, with the right result written before anything runs: ordinary cases, hard cases, and cases where the agent should not act but stop and ask or refuse. Rules check what rules can (prices, codes, sources, the stop at approval, the step count); a rubric scores quality, applied by an AI grader once it agrees with your own grades on a sample; and a person reads some full runs. Agents vary between runs, so each case runs several times and passes only if it passes every time. The set becomes a regression set, run again after any change to the instructions, the tools or the model. Worked example: 20 cases, three runs each; 16 passed every time at first, and 19 after two fixes.

Frame from lesson 7 of Sophyx Certified AI Agent Builder – Professional, "Test it with evals"

What you will learn

  • Write eval cases with the right result first
  • Score runs with rules, a rubric and a person
  • Re-run the eval set after every change

The one thing to remember

Test whole runs, more than once.

Common mistake: Testing an agent once and calling it done

Agents vary between runs, so one pass can hide a case that fails one time in three, and the first real failure reaches people.

The fix: Run each case several times, and pass it only if it passes every time.

Key terms

Eval
A repeatable test of the agent on real tasks.
Grader
A rule, a rubric or a person that scores a run.
Regression set
Cases you re-run after every change to catch new breaks.

Part of AI Agent Builder (AX-300)

For people who run AI workflows: build an agent with the right tools, tests and guardrails, and approvals where it counts.

12 lessons, practice quizzes, a final exam and the Sophyx Certified AI Agent Builder – Professional certificate. Regular price USD 49.99. Your first certification is free. Free with any paid Sophyx plan.

Start the course free
Book a Demo