A unit test wants a function. An agent is a loop that reads a state, picks a tool, writes a side effect, and sometimes lies about what it did. You can still write tests around the pieces — schema validation, tool adapters, the parser that turns a model dump into JSON. Those tests are necessary. They are not the eval.
The eval is: given a real case, did the agent do the job a person would have signed?
I keep a fixture set of cases I already know the answer to. Not toy prompts. Closed claims, messy notes, a missing field, a document that contradicts the structured record. Each case has a rubric, not a golden string. Did it call the right tools, in an order that makes sense? Did it refuse when the evidence was thin? Did the structured output actually parse? Did it invent a policy number?
That last one is the one people skip. Hallucination is easy to miss if your scorer is “the answer looks fluent.” Fluency is the failure mode.
I run the same fixtures against every prompt change, every model swap, every new tool. Offline, on a schedule, with the temperature pinned. If the score moves, I want to know which case and which step. A single aggregate “accuracy” number is how you ship a regression you cannot find.
The uncomfortable part is that a good eval set is a data-modeling problem. You have to decide what “done” means, in public, in a spreadsheet someone else can argue with. That is slower than writing another system prompt. It is also the only reason I trust an agent in a workflow that touches money.
Unit tests stay. They catch the boring breaks. The rubric is what catches the ones that look fine.