Evaluation and quality

How you measure and enforce quality — a judge that scores and regenerates answers, goal verification, runtime checks and alerts, test scenarios, and A/B model comparison.

Quality on Agentheya is measured on your live traffic, against criteria you define — not a lab benchmark. You can score every answer, verify goals, run test scenarios, and compare models. This page covers each.

Feature	When it runs	What it does
Goal checks	After every reply, and when the agent claims a goal is complete	Verifies your success criteria are actually met
Response judge	After every response	Scores the answer against your rubric and can regenerate a weak one
Runtime criteria	On every logged query	Deterministic pass/fail checks you define
Alert rules	On every logged query	Trigger when error rate, latency, or spend cross your thresholds
Scenario runs	On demand	Re-run your test cases against the live agent
A/B comparison	On demand	Compare two models over your real traffic

Goal verification

The success criteria you define on the Purpose page are verified two ways:

• After every reply, all active goals are assessed and each user’s met / partial / unmet state is saved. This never breaks a turn and never flips a goal that was already met.
• When the agent claims a goal is complete, any goal backed by your verification code re-checks against real data. A goal backed by code can’t be passed on the model’s word alone.

Goals & Goal Checks documents the full contract.

The response judge

The judge scores every response 1–10 against a rubric you write. Configure it on the Purpose page:

Setting	Behavior
Enabled	Off by default; nothing runs until you turn it on
Provider / model	Anthropic runs on the platform’s key; a custom provider runs on your own
Criteria	Your rubric. When empty, the default scores accuracy, helpfulness, and adherence to your prompt
Threshold	1–10, default 7. Scores below it are flagged
Max retries	0–5. Above 0, a flagged answer is regenerated

The judge never breaks the answer: if the judge itself fails, the response still ships and an error is noted on the query. Judge cost is tracked per query.

Scores appear in the Judge column on the Activity page, the Judge / Critic panel on the Monitoring page (count, average, pass rate, errors, cost, and a trend), and in each turn’s record. To tune your rubric without spending live traffic, use the test option that runs the judge against a message/response pair you type.

Regeneration

When the judge is enabled with retries above 0, a below-threshold answer is regenerated with the judge’s feedback, up to your configured number of retries. It behaves as one answer:

• On delivered-complete channels, only the final answer is returned. On streaming chat, the whole cycle renders in one bubble — the flagged answer, a divider with the judge’s score and criticism, then the retry.
• Regeneration counts against your existing per-turn limits — it never multiplies your time or tool-loop budget — and it won’t repeat a side effect, so a regenerated turn doesn’t send the same email twice.

Cost stays visible: each attempt writes its own activity row with the judge verdict attached, so you can see exactly what a quality bar costs.

Runtime criteria and alerts

Alongside the judge, deterministic checks run on every logged query. Configure them in the Runtime Criteria cards on the Monitoring page (field reference at Monitoring reference):

Check	Passes when
no_error	The query completed without an error
max_duration	Duration within your threshold
max_tokens	Total tokens within your threshold
min_response_length	Response at least N characters
finish_reason_ok	The model stopped normally
no_tool_errors	No tool errors during the turn
has_retrieval	At least one knowledge hit was used

Each check is required or optional: a failed required check makes the query failure, only optional failures make it partial. The pass rate feeds the Live eval pass tile on Monitoring.

Alert rules watch the same traffic in aggregate — error rate, slow responses, high token usage, consecutive failures, low success rate, and tool failure rate — each with its own severity, window, and threshold. A triggered rule is recorded under Recent Alerts and fires the on_alert hook event so you can page yourself.

Scenario evaluation

For regression testing, run your own scenarios against the live agent through the same path production traffic uses — real tools, real retrieval, real billing. Each scenario can declare:

Field	Check
persona	A simulated user persona
messages	One or more user messages (multi-turn supported)
expected_response_contains	Strings the response must contain
expected_response_excludes	Strings that must not appear
expected_tools_called	Tools that must have been called
forbidden_tools	Tools that must not have been called
judge_criteria	Optional per-scenario judge score
apply_runtime_criteria	Also score with your Runtime Criteria (on by default)

All checks pass → the scenario passes; a mix → partial; all fail → fail.

Scenario runs drive the live agent and spend real credits — a scenario that sends email sends email. The scenario harness is API-only today (no visual editor yet) and is available on the Business plan.

A/B model comparison

The Monitoring page’s A/B model comparison panel compares two models over your real traffic. Pick any two models seen on the agent recently and a window; the panel shows side-by-side query count, average judge score, pass rate, average cost, and average duration. This is observational — the platform doesn’t split live traffic — so it’s most useful right after switching your chat model: run the same traffic for a few days and read the delta.

Retrieval reproduction

Retrieval quality has its own test. The Retrieval Test page runs your query through every retrieval path side by side and can also reproduce the exact production pipeline, so what you see in the test is what a real turn retrieves.

What the platform deliberately doesn’t do

• No standardized benchmark suite. Evaluation is per-agent and traffic-driven — the rubric, criteria, and scenarios are yours, and every number is computed from your real traffic.
• No run-over-run comparison view. Scenario runs persist individually; you compare runs yourself.
• No exact replay. Model outputs are non-deterministic. Pinning a low temperature in Model Controls makes outputs more repeatable, not identical; for systematic re-testing, use scenario runs.