Turn a task into a test case
Turn a task into a test case — Evaluate AI agents in practice. An artifact instead of an impression; Frozen inputs; Define acceptance before the run. Practical assignment: Turn a task into a test case.
Demo edition. The materials are undergoing editorial review.
Topics covered
- An artifact instead of an impression
- Frozen inputs
- Define acceptance before the run
An artifact instead of an impression
“The agent answered well” is not a repeatable criterion. First define what it must return. In the teaching example an agent reads local stock.csv containing pen=12, notebook=4 and folder=0. The user requests JSON listing products with quantity below 5. The expected artifact includes notebook and folder, excluding pen. Ordering is unspecified, so the test must not invent an ordering requirement. Record format, required row set and the prohibition on editing the input file. These are different checks: valid JSON does not establish correct products, and a correct list does not establish absence of a write. If the file is unavailable, expect an explicit availability error instead of invented stock. This produces an inspectable contract without relying on whether a reviewer likes the wording.
Frozen inputs
If stock.csv changes between runs, an answer difference may come from data rather than the agent. Save fixture stock-low-01 with exact CSV bytes, request and expectations. Include parser version and allowed tools. Do not use a live warehouse as the fixture: it introduces a hidden dependency and risks accidental writes. A text file and read_file stub returning its fixed contents are sufficient here. A separate stock-missing-01 returns FILE_NOT_FOUND. Do not confuse an unavailable file with an empty stock list: the latter has available data and may legitimately produce an empty answer. Distinct state names make failures diagnosable. Preserve a fixture digest or immutable repository version so results can be reproduced after the training data is edited. Keep private customer data out of these fixtures.
Define acceptance before the run
Write expectations before seeing the answer; otherwise a polished result can tempt you to adapt the test afterward. For stock-low-01 require parseable JSON, exactly two products with correct quantities, no write_file call and a fixture reference in the report. An unexecuted check is not an agent failure: if the stub breaks before the agent starts, record technical_failure. Returning pen is content_failure; an unstarted case is not_run. These distinctions prevent infrastructure errors from contaminating content results. State the overall acceptance policy, for example that any unauthorized write blocks acceptance regardless of list correctness. That is the training project’s policy, not a universal quality metric. Save it beside the fixture so another author cannot silently change the criterion after observing an inconvenient answer.
Practical assignment
Turn a task into a test case
Continue to learning