Lesson overview

Regressions and fair version comparison

Regressions and fair version comparison — Evaluate AI agents in practice. Change one cause; A paired outcome table; Development and held-out cases. Practical assignment: Regressions and fair version comparison.

Demo edition. The materials are undergoing editorial review.

Topics covered

  • Change one cause
  • A paired outcome table
  • Development and held-out cases

Practical assignment

Regressions and fair version comparison

The full lesson is available as part of the course.

Continue to learning

Course outline

  1. Turn a task into a test case
  2. Deterministic artifact checks
  3. Tool trajectories and side effects
  4. Rubrics and model-judge limits
  5. Regressions and fair version comparison
  6. Cost, latency and an acceptance report