Agent Loop Notes

· 9 min read

How to Write Evals for an AI Agent: A Practical Starter Guide

Build your first agent eval suite step by step: pick real tasks, grade outcomes instead of paths, measure consistency with pass^k and read the transcripts.

evalstestingagents

Unit tests tell you whether a function returns the right value. They say very little about whether an agent will book the right meeting, file the right ticket or stop when it should. Agents make a chain of decisions, each one shaped by what came before, and the same input can lead to a different chain on the next run.

An eval suite is the test suite for that behavior. Most teams know they need one and put it off, because it sounds like a research project. It does not have to be. This guide walks through a small, useful first version you can build in an afternoon, then shows how to grow it.

The vocabulary, in one minute

Anthropic's engineering post Demystifying evals for AI agents uses a set of terms that are worth adopting, because they keep conversations about evals precise:

  • Task: one test case, with defined inputs and a definition of success.
  • Trial: one attempt at a task. You run several, because agent output varies.
  • Grader: the logic that scores a trial. A task can have more than one.
  • Transcript: the full record of a trial: outputs, tool calls, reasoning, intermediate results. Also called a trace or trajectory.
  • Outcome: the state of the world when the trial ends.

That last distinction matters more than it looks. The outcome is whether the reservation exists in the database, not whether the agent said "I booked it."

Step 1: Collect 20 to 50 tasks from real failures

You do not need hundreds of tasks to start. Early on, the differences between versions of your agent are large, and a small set catches them. The Anthropic post suggests 20 to 50 simple tasks drawn from real failures as a good starting point, and warns that evals get harder to build the longer you wait.

Good sources for your first tasks:

  • Bugs you already fixed. Each one is a regression test waiting to be written.
  • Support tickets and user complaints.
  • The manual checks you run before every release. If you type the same three prompts into a chat window every time you ship, those are tasks.
  • Edge cases you argued about in code review.

Write each task down with its input and a plain-language statement of what success looks like. If two people on your team would disagree about whether a given result passes, the task is not ready yet.

Step 2: Define success as an outcome you can check

For every task, ask what you could inspect after the run to know it worked. Some examples:

Agent Weak success check Strong success check
Calendar assistant Final message says the meeting was scheduled An event exists at the right time with the right attendees
Coding agent The agent says tests pass You run the test suite yourself after the trial
Support agent Reply sounds helpful Refund issued for the correct order and amount, nothing else changed
Research agent Answer is long and confident Answer contains the specific facts in your reference answer

Checking the environment instead of the agent's claims removes a whole class of false passes. Agents are good at sounding finished.

It also helps to keep each trial isolated. Reset the database, the file system or the mock services between trials, so one run cannot leak state into the next.

Step 3: Pick graders, cheapest first

There are three kinds of graders, and each has a place:

Grader Good for Watch out for
Code (string match, state checks, tests) Anything with a checkable answer Rejecting valid answers phrased differently
Model (an LLM judging against a rubric) Open-ended output, tone, completeness Cost, randomness, needs calibration
Human review Calibrating the other two, subjective quality Slow and expensive

Start with code graders wherever the outcome is checkable. Add a model grader only for the parts of a task a script cannot judge, and give it a narrow rubric: one question per dimension, with an explicit "can't tell" option so it does not guess. Then spot-check a sample of its verdicts by hand. A model grader you have never compared against human judgment is a second untested system.

Grade what the agent produced, not the exact steps it took. A rule like "must call search then fetch then summarize" fails agents that find a better route. If you care about efficiency, measure it separately (tool calls, tokens, turns) instead of hard-coding a path.

Step 4: Run several trials and measure consistency

One run per task tells you what happened once. Run each task several times and you can measure two different things:

  • pass@k: the chance that at least one of k attempts succeeds. Useful when one good answer is enough, such as generating candidate code you will test anyway.
  • pass^k: the chance that all k attempts succeed. Useful for anything customer-facing, where every run has to work.

The gap between them is large. An agent that succeeds 75 percent of the time has a pass^3 of about 42 percent (0.75 × 0.75 × 0.75). Three customers in a row will all get a good result less than half the time. The pass^k metric was introduced in the τ-bench paper to measure how reliably an agent behaves over repeated trials. Even gpt-4o scored below 25 percent on pass^8 in its retail domain.

With n trials per task and c successes, you can estimate both from your data. The pass@k estimator comes from the Codex paper: 1 − C(n − c, k) / C(n, k). The τ-bench paper gives the matching estimate for pass^k: C(c, k) / C(n, k).

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    """Chance that at least one of k sampled trials passes."""
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n: int, c: int, k: int) -> float:
    """Chance that all k sampled trials pass."""
    if c < k:
        return 0.0
    return comb(c, k) / comb(n, k)

# 8 trials, 6 passed
print(round(pass_at_k(8, 6, 3), 3))   # 1.0
print(round(pass_hat_k(8, 6, 3), 3))  # 0.357

Report both. A change that raises pass@k while lowering pass^k means the agent solves more tasks at least some of the time, but solves them less consistently.

Step 5: A minimal harness

You do not need a framework for the first version. Here is the whole loop in plain Python. Replace run_agent with a call into your own agent, and write graders that inspect whatever your agent changes.

import json
from dataclasses import dataclass
from typing import Callable

@dataclass
class Task:
    id: str
    prompt: str
    graders: list[Callable[[dict], bool]]
    setup: Callable[[], dict] = dict

def run_agent(prompt: str, env: dict) -> dict:
    """Call your agent here and return the transcript.

    Act on env in place (or on the services it holds) so graders can inspect the result.
    """
    raise NotImplementedError

def run_suite(tasks: list[Task], trials: int = 5) -> dict:
    results = {}
    for task in tasks:
        passes = 0
        transcripts = []
        for _ in range(trials):
            env = task.setup()                 # fresh state for every trial
            transcript = run_agent(task.prompt, env)
            ok = all(grade(env) for grade in task.graders)
            passes += ok
            transcripts.append({"passed": ok, "transcript": transcript})
        results[task.id] = {"passes": passes, "trials": trials, "runs": transcripts}
    return results

TASKS: list[Task] = []  # add your Task objects here

if __name__ == "__main__":
    results = run_suite(TASKS)
    with open("eval_results.json", "w") as f:
        json.dump(results, f, indent=2, default=str)
    for task_id, r in results.items():
        print(f"{task_id}: {r['passes']}/{r['trials']}")

A grader in this setup is just a function that looks at the environment after the run:

def meeting_was_booked(env: dict) -> bool:
    # env["calendar"] is your test calendar; adapt these calls to its API.
    events = env["calendar"].events_on("2026-11-12")
    return any(
        e.start.hour == 14 and {"dana@example.com"} <= set(e.attendees)
        for e in events
    )

Save every transcript. You will need them in the next step.

Step 6: Read the transcripts

This is the step teams skip, and it is the one that makes the numbers trustworthy. Before you believe a score, open a handful of passing and failing transcripts and check two things:

  1. Did failures fail for real? Sometimes the agent did the right thing and the grader rejected it, for example because a date was formatted differently. That is a grader bug. Anthropic describes a real case: a grader for CORE-Bench rejected "96.12" because it expected "96.124991…".
  2. Did passes pass for real? Sometimes the agent got lucky or the grader was too loose.

A task with a 0 percent pass rate across many trials is usually a broken task, not a hopeless agent. A good habit is to write a reference solution for each task, a run you know is correct, and confirm your graders accept it.

Step 7: Split capability and regression suites

As the suite grows, keep two lists:

  • Capability evals ask what the agent can do. They should start with a low pass rate, so you have room to improve.
  • Regression evals ask whether the agent still does what it used to. They should sit near 100 percent, and a drop in score signals that something broke. Many teams treat that drop as a release blocker.

When a capability task passes reliably, move it to the regression suite. Over time, your regression suite becomes a record of every problem you have solved, and your capability suite stays pointed at what you have not.

Common mistakes

  • Waiting for a big suite. Twenty real tasks this week beat three hundred synthetic ones next quarter.
  • Grading the agent's words. Check the world, not the claim.
  • One trial per task. You will mistake luck for progress.
  • Only positive cases. Include tasks where the right move is to refuse, ask a question or do nothing.
  • Never reading transcripts. A score you have not inspected is a guess.

Where to go next

Once the basic loop works, the most useful additions are cost and latency tracking per task, a model grader for open-ended answers, and running the suite in CI on every prompt or tool change. If your agent's failures are mostly about picking the wrong tool, the next post covers that directly: why your agent calls the wrong tool.

FAQ

How many test cases do I need to start evaluating an AI agent?

Anthropic's engineering team says 20 to 50 simple tasks drawn from real failures is a great start. Early on, each change to an agent has a large effect, so a small set is enough to see it.

What is the difference between pass@k and pass^k?

pass@k is the chance that at least one of k attempts succeeds. pass^k is the chance that all k attempts succeed. An agent with a 75 percent success rate has a pass^3 of about 42 percent.

How do you calculate pass@k from n samples?

Run n trials, count the c that pass, and use the unbiased estimator from the Codex paper: 1 − C(n − c, k) / C(n, k). The τ-bench paper estimates pass^k as C(c, k) / C(n, k).

Why does my agent eval show a 0 percent pass rate?

Anthropic notes that a 0 percent pass rate across many trials most often points to a broken task, not an incapable agent. Read the transcripts, and check that your graders accept a reference solution you know is correct.

Sources

New posts come out regularly. Follow along with the RSS feed.