Free lesson

Deterministic artifact checks

Deterministic artifact checks — Evaluate AI agents in practice. Parsing and schema; Sets and duplicates; Negative examples for the checker. Practical assignment: Deterministic artifact checks.

Demo edition. The materials are undergoing editorial review.

Topics covered

  • Parsing and schema
  • Sets and duplicates
  • Negative examples for the checker

Parsing and schema

Start with a check that needs no second model. Parse JSON and validate field types. Each row in our list requires a nonempty product string and a nonnegative integer quantity. The string quantity="4" is not the number 4 when the contract requires a number. Do not silently fix the error inside the test: that would evaluate your normalizer rather than the agent’s artifact. If normalization is part of the product, test it as a separately specified stage. Decide how unknown fields are handled; a closed contract may reject them. Parsing catches syntax, while schema validation catches shape. Neither determines whether notebook belongs in the correct list. Record these outcomes separately so formatting failure does not conceal the cause of a content failure. Report the invalid field path without exposing private values.

Sets and duplicates

Searching for notebook in an answer misses an extra pen and a duplicated folder. Compare structured data with expectations. Check row count, product uniqueness and quantities by key. The notebook=4 and folder=0 fixture requires exactly two rows. Converting the list blindly to a Map lets a second folder overwrite the first and hides a duplicate. Check uniqueness before conversion, or compare Map size with list length. The snippet validates an already parsed rows array locally; it does not run an agent, access a network or require a paid judge. Expectations can also be wrong: manually verify quantity<5, including zero and excluding exactly 5. That boundary distinguishes “less than” from “at most” and deserves its own fixture rather than an assumption.

const expected = new Map([["notebook", 4], ["folder", 0]]);
const actual = new Map(rows.map(r => [r.product, r.quantity]));
const pass = rows.length === 2 && actual.size === rows.length &&
  [...expected].every(([name, quantity]) => actual.get(name) === quantity);

Negative examples for the checker

A checker that always returns true looks successful on a correct answer. Test the checker itself with prepared artifacts: a correct array, one containing pen, duplicate folder rows, a string quantity and malformed JSON. Record expectations first. The correct array should pass; each violation should produce a specific failureCode. If duplicates pass, fix the checker before evaluating an agent. Do not relax a criterion merely because the agent frequently fails it; first decide whether the contract is actually too strict for the product. Deterministic checks work well for shape and computable conditions, but cannot establish the completeness of an explanation of a complex document. For an ambiguous criterion introduce an anchored rubric rather than hiding uncertainty behind a precise-looking number. Preserve checker versions alongside evaluation results.

Practical assignment

Deterministic artifact checks

Continue to learning

Course outline

  1. Turn a task into a test case
  2. Deterministic artifact checks
  3. Tool trajectories and side effects
  4. Rubrics and model-judge limits
  5. Regressions and fair version comparison
  6. Cost, latency and an acceptance report