How to Run Self-Hosted Evals for AI Agents
Build your own eval loop with real traces, deterministic checks, an LLM judge, and pass rates wey show when a commit drop from 58/60 to 51/60.
Wetín self-hosted evals for AI agents be
Self-hosted evals for AI agents na four things wey you keep for your own repository: file of saved cases, script wey run the agent on top dem, set of checks wey grade each answer, and table of results wey you fit query. No vendor dey needed for any item for that list. The whole loop na just some hundred lines of Python and one SQLite file.
The agent work for the demo because na you pick the five inputs yourself. E spoil for week two because prompt line change, or model change, or tool description change, and no measurement cover any of dem. Eval loop turn "e dey feel worse now" to "pass rate drop from 58 of 60 to 51 of 60 for commit 4f1c9ab".
The loop get four steps, and this guide get one section for each step: collect real traces, promote the interesting ones into cases, grade every case for every change, and store the pass rate beside the commit wey produce am. The same loop work for anything wey you run the agent on, and the self-hosted agent frameworks wey worth running mainly differ for how much of the trace dem hand give you free.
Why agent dey break for week two
Agent na prompt, model, set of tool definitions, and any context wey system retrieve when e dey run. All four fit change even when your application code no change, so normal code review no go see anything to object to.
The most common cause na prompt edit. You add one sentence to stop rude reply. That sentence change behaviour for inputs wey nobody test again, and traces show am clearly: last week's trace for the same question get a create_refund tool call, this week's one no get any, and reply na polite apology instead. Nothing raise error, so no alert fire.
The second cause na model. Record the exact model string wey you send with every run, claude-haiku-4-5-20251001 instead of shorthand wey you just keep for your head, because you fit only diagnose pass rate wey drop on the day you switch models when model dey inside the row.
The third one na tools. If you change the wording of tool description, e fit change when model decide to call am. If your tools dey come through MCP servers wey dey run for VPS, the schema dey for another process, so e fit change under you without any diff for your repository at all. The fourth one na retrieval: the same question hit index wey system rebuild overnight, and answer follow the new document.
Make the golden set from traces wey you already collect
No invent eval cases. Take dem from traffic. If you already dey run self-hosted Langfuse tracing for your agent, every request dey store with its input, tool calls, and output. Na exactly this raw material case need.
Export one window of root observations through the public API. E dey use basic authentication, with your public key as the username and your secret key as the password.
export LF_HOST="https://langfuse.example.com"
curl -sS -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
"$LF_HOST/api/public/v2/observations?limit=50&isRootObservation=true&fromStartTime=2026-07-01T00:00:00Z" \
| jq '.data[0]'Read one record before you write any parser. The rows dey come under data, but the field names wey hold the question and reply depend on how your agent instrument its spans. So map wetin you actually see, instead of wetin you expect. Then write the cases by hand, one JSON object for each line, inside evals/cases.jsonl:
{"id": "refund-double-charge", "tags": ["smoke"], "input": "I was charged twice for order 41822.", "must_call": ["lookup_order", "create_refund"], "must_not_include": ["I cannot help"], "rubric": "The reply confirms exactly one refund for order 41822 and states the amount."}Five rules go keep the set useful to run:
- 40 to 80 cases dey enough to start. If e below 20, one flaky case fit change the pass rate by 5 points, and people go ignore number wey dey jump without reason.
- Every production bug wey you fix become a case that same day. Na this habit dey make the set grow for the correct direction.
- One behaviour for each case. Case wey check refund amount and tone together no go tell you anything when e fail.
- The
idno dey ever change, because na the id dey help compare today's run with last month's run. - Redact before you commit. This file go enter git, so remove customer names and any order numbers wey no belong to you.
Do grade with deterministic checks first, because dem free
Anything wey get correct answer fit use plain assertion. No model call, no cost, no confusion. Deterministic checks dey catch structural regressions, and na dem dey break systems around your agent: JSON no parse, tool no ever get called, forbidden phrase come back, answer no cite any source.
Na one function sabi about your agent. Everything else for the harness dey generic.
import json, os, urllib.request
def run_agent(case):
req = urllib.request.Request(
os.environ["AGENT_URL"],
data=json.dumps({"input": case["input"]}).encode(),
headers={"content-type": "application/json"},
)
with urllib.request.urlopen(req, timeout=120) as resp:
return json.load(resp)
def deterministic(case, result):
text = result.get("output", "")
called = [c["name"] for c in result.get("tool_calls", [])]
failures = []
for tool in case.get("must_call", []):
if tool not in called:
failures.append(f"tool not called: {tool}")
for phrase in case.get("must_not_include", []):
if phrase.lower() in text.lower():
failures.append(f"forbidden phrase: {phrase}")
if len(called) > case.get("max_tool_calls", 12):
failures.append(f"too many tool calls: {len(called)}")
return failuresKeep the tool budget for that list. Agent wey solve one case with 3 calls today and 11 tomorrow don regress, even when final answer correct, because you dey pay for every call e make.
LLM as judge, and the four ways e fit go wrong
Anything wey pass the assertions still need grader wey fit read. An LLM judge na second model call: e receive the question, the agent answer, and one criterion, then e return verdict. Na the only practical way to grade “whether the reply answer wetin user ask”.
Four rules make judge useful:
- Binary verdict, never 1 to 10 score. Scale dey return 7 and 8 for almost everything, so the number no dey change and you learn nothing from am.
- One criterion for each call. Ask about the refund amount, or about the tone, no be both together.
- Give judge the expected answer whenever the case get one. Grading against reference easier well-well than grading in the abstract.
- Force the output shape and parse am strictly.
from anthropic import Anthropic
client = Anthropic() # reads ANTHROPIC_API_KEY from the environment
def judge_prompt(case, output):
return (
"You grade one answer against one criterion.\n"
"Reply with JSON only, in this exact shape:\n"
'{"verdict": "pass", "confidence": "high", "reason": "one short sentence"}\n'
f"Criterion: {case['rubric']}\n"
f"Question: {case['input']}\n"
f"Answer: {output}\n"
"Length is not a criterion. Judge only the criterion above."
)
def judge(case, output, model):
msg = client.messages.create(
model=model,
max_tokens=200,
messages=[{"role": "user", "content": judge_prompt(case, output)}],
)
return json.loads(msg.content[0].text)Now make we look the failure modes. Each one get test wey you fit run this afternoon, and e important to run dem, because unchecked judge dey produce numbers wey look precise but mean nothing.
Length bias. Longer answers dey pass more often. Test am: take ten answers wey judge fail, add two paragraphs of confident filler to each one, without adding any new fact, then judge dem again. If any verdict change to pass, na length bias be that, and na the rubric you need fix.
Self-preference. Judge often dey grade output from im own model family more kindly than output from another one. Test am: use judges from two different families to grade the same 30 answers, then compare the verdicts case by case. For cases wey dem disagree, read the case yourself.
Position bias. If you use judge to compare two answers, A and B, change the order and run am again. If verdict change after the swap, pairwise comparison no safe yet for that rubric.
Rubric drift. Vague criteria dey produce agreeable judges. “Is the answer helpful” go pass almost anything. “Does the answer state the refund amount in dollars” go pass only the thing wey you mean. Rewrite every criterion until e name the fact wey you dey check.
One guard fit cover all four. Keep 30 cases wey you label by hand, and score the judge against your labels every time you change the judge model or judge prompt. If e disagree with you for more than one case out of ten, fix the rubric before you trust any pass rate wey e produce. Judge na code, so version and review am like code.
Grade cheap, then escalate go frontier model
To judge every case with the most expensive model for every commit na how eval bill go pass the agent wey e dey test. Arrange graders by price, then stop once answer don clear.
Di data wey dey behind dis chart
[
{
"label": "Haiku 4.5, Batch API",
"usd_per_1000_judge_calls": "0.90"
},
{
"label": "Haiku 4.5",
"usd_per_1000_judge_calls": "1.80"
},
{
"label": "Sonnet 5",
"usd_per_1000_judge_calls": "3.60"
},
{
"label": "Opus 5",
"usd_per_1000_judge_calls": "9.00"
}
]Those figures assume say each judge call get about 1,200 input tokens and 120 output tokens. Na realistic size for one question, one answer, and one criterion. To judge 1,000 cases go cost 1.80 US dollars with Claude Haiku 4.5, and 9.00 with Claude Opus 5. The difference fit look small until you multiply am. A set of 60 cases, judged for every commit at 40 commits per week, na 2,400 judge calls every week before anybody run the nightly job.
One discount dey work well for eval work. Eval runs no dey interactive, so Batch API cut both input and output prices by half in exchange for asynchronous delivery. Na the first row for the chart. Prompt caching no help judge of this size. The multipliers dey real: five-minute cache write cost 1.25 times the base input price, while cache read cost one-tenth of am. But the cached part of prompt must reach minimum length: 4,096 tokens for Claude Haiku 4.5, 1,024 for Claude Sonnet 5, and 512 for Claude Opus 5. A judge call with 1,200 tokens dey far below the Haiku limit. API go process the shorter prompt without caching and without returning error, so no cache write, no cache hit, and no saving. For Opus 5, na only shared prefix of at least 512 tokens qualify. The fixed instructions for judge_prompt much shorter than that because rubric dey change for every case. These na Anthropic list prices as of October 2026.
The ladder, in order:
- Run deterministic checks for every case. E no get API cost at all.
- Use small model judge for cases wey pass those checks.
- Use frontier judge only where small judge talk say fail, or talk say pass with low confidence.
- Do human review for small sample once every week.
CHEAP = "claude-haiku-4-5-20251001"
STRICT = "claude-opus-5"
def grade(case, result):
hard = deterministic(case, result)
if hard:
return False, "deterministic", "; ".join(hard)
first = judge(case, result["output"], CHEAP)
if first["verdict"] == "pass" and first["confidence"] == "high":
return True, CHEAP, first["reason"]
second = judge(case, result["output"], STRICT)
return second["verdict"] == "pass", STRICT, second["reason"]As of October 2026, Anthropic list both judges as legacy models wey still dey available. The deprecations page give Claude Haiku 4.5 retirement date of not sooner than October 15, 2026. So check that page before you depend on cheap judge for long. Treat changing judge model like any other change: rerun the 30 hand-labelled cases first.
This approach trade some grading accuracy for lower cost, so measure the trade instead of assuming am. Once every month, grade the whole set with strict judge too, then compare the two columns. If dem disagree on more than a handful of cases, your rubric too loose for small model. Na the rubric you need fix. Controlling wetin the agent itself spend na separate work, covered for cost control for an AI agent on a VPS.
Track pass rate over time inside system wey you own
Pass rate wey you no fit link to commit na just feeling. Store one row for every case and every run, with the commit and model inside the row.
CREATE TABLE IF NOT EXISTS results (
run_id TEXT NOT NULL,
ran_at TEXT NOT NULL,
git_sha TEXT NOT NULL,
agent_model TEXT NOT NULL,
case_id TEXT NOT NULL,
passed INTEGER NOT NULL,
graded_by TEXT NOT NULL,
reason TEXT
);SELECT run_id, git_sha, agent_model,
count(*) AS cases,
round(100.0 * sum(passed) / count(*), 1) AS pass_pct
FROM results
GROUP BY run_id
ORDER BY ran_at DESC
LIMIT 10;Load the schema with sqlite3 evals/results.db < evals/schema.sql, then read the trend with sqlite3 -box evals/results.db < evals/passrate.sql. One year of daily runs for 60 cases na about 22,000 rows, so the store no go turn into project by itself. Running SQLite for production on VPS explain the settings wey go start matter if dem share this file between machines.
The runner dey print the same information for person:
run 2026-08-05T09:14:22Z sha 4f1c9ab model claude-sonnet-5 58/60 pass (96.7%)
FAIL refund-double-charge deterministic: tool not called: create_refund
FAIL pto-policy-question judge(opus): reply gives no dollar amountRun the suite on changes wey fit break an agent. This means prompt edits, model changes, and tool changes, instead of every commit anywhere for the repository. A pre-push hook dey cover the fast subset:
cat > .git/hooks/pre-push <<'EOF'
#!/bin/sh
python3 evals/run.py --set smoke || exit 1
EOF
chmod +x .git/hooks/pre-pushFull runs dey slower, so schedule dem. A nightly systemd service and timer for the VPS runs the whole set against the deployed prompt. Na this one dey catch changes wey come from outside your repository, like hosted tool wey its behaviour don change.
Human review, sample, no be exhaustive
Judge dey calibrated against human labels, so person must produce those labels. Read one sample every week: every case wey judge fail, plus ten passes wey you pick randomly. The random passes na the important half, because judge wey don quietly start to pass bad answers go look perfect for any dashboard wey build from judge own verdicts.
Fifteen cases at three minutes each na 45 minutes every week, and e go return corrections to the rubric where you and judge no agree, plus new cases for failure types wey nobody don imagine before. Write the human verdict inside the same table with graded_by set to human, so agreement between judge and human go become query instead of memory.
Wetin fit break for the eval harness itself
anthropic.RateLimitError for the first full run. Sixty cases wey run at once pass the request or token limit for your tier. Limit concurrency to four workers, and move the nightly run go the Batch API.
json.JSONDecodeError: Expecting value: line 1 column 1 (char 0) from the judge. The model reply with prose, or put its JSON inside code fence. Retry am once, then record the case as error. Never allow parse failure count as pass, because suite wey turn errors into passes go climb toward 100% while the agent dey worse.
Flaky cases. The same input pass for one run and fail for the next because the agent dey sample its output. Run the flaky case three times and record the fraction instead of deleting the case. Case wey pass two runs out of three na real robustness bug, and customer go discover am.
Golden set rot. Person edit expected answer to make the suite green. Review diffs to evals/cases.jsonl as carefully as diffs to the agent, because that file na your written definition of correct.
Suite wey never fail. Pass rate wey stay for 100% for one month mean say the set don stop tracking the product. Pull ten recent traces, find the ones wey the agent handle badly, and add dem. Then break something on purpose and confirm say the run go red. This check na mutation testing dey apply to test suite and the only way to know say your set still get teeth.
FAQ
How many cases an AI agent eval set need?
Start with 40 to 80, then grow the set from real failures. If the set get below about 20 cases, one flaky result fit change pass rate by 5 points, so the number no carry useful information again. Once e pass a few hundred, every run cost real money and real time, while each extra case add small coverage. The important measure no be the count. Na the share of known production failure types wey appear in the set at least once.
I fit trust LLM judge to grade my agent?
Only after you don measure am against your own labels. Keep 30 cases wey you grade by hand, and score the judge against dem whenever you change judge model or judge prompt. Judges dey show length bias, where padded answers pass more often. Dem also show self-preference, where output from their own model family dey get kinder grading. You fit test both: add extra text to a failed answer and judge am again, or grade the same answers with judge from another family. If judge disagree with your labels on more than one case in ten, the rubric too vague to use.
Which model suppose grade the evals?
Grade cheap and escalate. Deterministic assertions no cost anything, so run dem first for every case. Small model fit handle clear passes. Only fails and low-confidence verdicts go to frontier model. At list prices in October 2026, judging 1,000 cases cost about 1.80 US dollars with Claude Haiku 4.5 and about 9.00 with Claude Opus 5. Since eval runs dey asynchronous, Batch API cut either figure by half.
Evals fit replace production monitoring?
No, because dem answer different questions. Eval suite tell you whether a change wey you wan release go make fixed set of cases better or worse. Tracing and monitoring tell you wetin real users dey encounter now, including inputs wey no case cover. Dem help each other: traces provide new cases, while eval suite decide whether your fix really work.