AX-300 · Lesson 12 of 12 · Module 4: Ship and Improve · 3:16

Improve it safely

Improve an agent one change at a time (the instructions, a tool, a source or the model), write each change in an improvement record with why and what the evals said, and run the whole eval set before it goes live. Keep every version, and the last good one ready: if a change fails, roll back first, then try a narrower change. A new model from the maker is a change too. Run the agent from a one-page operating guide: owner and deputy, the job card, the access map, approvals, signals and alerts, caps, the stop switch, incident steps and review dates, with a keep, grow or retire decision every quarter. Worked example: “more playful” posts failed four eval cases, one promising next-day delivery, and were rolled back; a narrower change passed and went live.

Frame from lesson 12 of Sophyx Certified AI Agent Builder – Professional, "Improve it safely"

What you will learn

  • Change one thing at a time and re-test
  • Keep a version you can roll back to
  • Write a one-page operating guide

The one thing to remember

Change one thing, then run the evals.

Common mistake: Changing several things at once without re-testing

When results change you can’t tell which change did it, and a new break can reach people before anyone notices.

The fix: Change one thing at a time, record it, and run the evals before it goes live.

Key terms

Improvement record
Each change, why you made it, and what the evals said.
Rollback
Going back to the last version that passed its evals.
Operating guide
One page on how the agent is run, watched and stopped.

Part of AI Agent Builder (AX-300)

For people who run AI workflows: build an agent with the right tools, tests and guardrails, and approvals where it counts.

12 lessons, practice quizzes, a final exam and the Sophyx Certified AI Agent Builder – Professional certificate. Regular price USD 49.99. Your first certification is free. Free with any paid Sophyx plan.

Start the course free
Book a Demo