> ## Documentation Index
> Fetch the complete documentation index at: https://agentheya.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation and quality

> How you measure and enforce quality — a judge that scores and regenerates answers, goal verification, runtime checks and alerts, test scenarios, and A/B model comparison.

Quality on Agentheya is measured on your live traffic, against criteria you
define — not a lab benchmark. You can score every answer, verify goals, run
test scenarios, and compare models. This page covers each.

| Feature          | When it runs                                                    | What it does                                                        |
| ---------------- | --------------------------------------------------------------- | ------------------------------------------------------------------- |
| Goal checks      | After every reply, and when the agent claims a goal is complete | Verifies your success criteria are actually met                     |
| Response judge   | After every response                                            | Scores the answer against your rubric and can regenerate a weak one |
| Runtime criteria | On every logged query                                           | Deterministic pass/fail checks you define                           |
| Alert rules      | On every logged query                                           | Trigger when error rate, latency, or spend cross your thresholds    |
| Scenario runs    | On demand                                                       | Re-run your test cases against the live agent                       |
| A/B comparison   | On demand                                                       | Compare two models over your real traffic                           |

## Goal verification

The success criteria you define on the [Purpose page](/sidebar-menu/purpose) are
verified two ways:

* **After every reply**, all active goals are assessed and each user's met /
  partial / unmet state is saved. This never breaks a turn and never flips a
  goal that was already met.
* **When the agent claims a goal is complete**, any goal backed by your
  verification code re-checks against real data. A goal backed by code can't be
  passed on the model's word alone.

[Goals & Goal Checks](/sidebar-menu/goals) documents the full contract.

## The response judge

The judge scores every response 1–10 against a rubric you write. Configure it on
the [Purpose page](/sidebar-menu/purpose):

| Setting          | Behavior                                                                                        |
| ---------------- | ----------------------------------------------------------------------------------------------- |
| Enabled          | Off by default; nothing runs until you turn it on                                               |
| Provider / model | Anthropic runs on the platform's key; a custom provider runs on your own                        |
| Criteria         | Your rubric. When empty, the default scores accuracy, helpfulness, and adherence to your prompt |
| Threshold        | 1–10, default 7. Scores below it are flagged                                                    |
| Max retries      | 0–5. Above 0, a flagged answer is regenerated                                                   |

The judge **never breaks the answer**: if the judge itself fails, the response
still ships and an error is noted on the query. Judge cost is tracked per query.

Scores appear in the **Judge column** on the [Activity page](/sidebar-menu/activity),
the **Judge / Critic panel** on the [Monitoring page](/sidebar-menu/monitoring)
(count, average, pass rate, errors, cost, and a trend), and in each turn's
record. To tune your rubric without spending live traffic, use the test option
that runs the judge against a message/response pair you type.

## Regeneration

When the judge is enabled with retries above 0, a below-threshold answer is
regenerated with the judge's feedback, up to your configured number of
retries. It behaves as one answer:

* On delivered-complete channels, only the final answer is returned. On
  streaming chat, the whole cycle renders in one bubble — the flagged answer, a
  divider with the judge's score and criticism, then the retry.
* Regeneration **counts against your existing per-turn limits** — it never
  multiplies your time or tool-loop budget — and it **won't repeat a side
  effect**, so a regenerated turn doesn't send the same email twice.

<Note>
  Cost stays visible: each attempt writes its own activity row with the judge
  verdict attached, so you can see exactly what a quality bar costs.
</Note>

## Runtime criteria and alerts

Alongside the judge, deterministic checks run on every logged query. Configure
them in the **Runtime Criteria** cards on the
[Monitoring page](/sidebar-menu/monitoring) (field reference at
[Monitoring reference](/reference/monitoring)):

| Check                 | Passes when                          |
| --------------------- | ------------------------------------ |
| `no_error`            | The query completed without an error |
| `max_duration`        | Duration within your threshold       |
| `max_tokens`          | Total tokens within your threshold   |
| `min_response_length` | Response at least N characters       |
| `finish_reason_ok`    | The model stopped normally           |
| `no_tool_errors`      | No tool errors during the turn       |
| `has_retrieval`       | At least one knowledge hit was used  |

Each check is *required* or *optional*: a failed required check makes the query
`failure`, only optional failures make it `partial`. The pass rate feeds the
**Live eval pass** tile on Monitoring.

Alert rules watch the same traffic in aggregate — error rate, slow responses,
high token usage, consecutive failures, low success rate, and tool failure rate
— each with its own severity, window, and threshold. A triggered rule is
recorded under **Recent Alerts** and fires the `on_alert`
[hook event](/sidebar-menu/hooks) so you can page yourself.

## Scenario evaluation

For regression testing, run your own scenarios against the **live agent through
the same path production traffic uses** — real tools, real retrieval, real
billing. Each scenario can declare:

| Field                        | Check                                                 |
| ---------------------------- | ----------------------------------------------------- |
| `persona`                    | A simulated user persona                              |
| `messages`                   | One or more user messages (multi-turn supported)      |
| `expected_response_contains` | Strings the response must contain                     |
| `expected_response_excludes` | Strings that must not appear                          |
| `expected_tools_called`      | Tools that must have been called                      |
| `forbidden_tools`            | Tools that must not have been called                  |
| `judge_criteria`             | Optional per-scenario judge score                     |
| `apply_runtime_criteria`     | Also score with your Runtime Criteria (on by default) |

All checks pass → the scenario passes; a mix → `partial`; all fail → `fail`.

<Warning>
  Scenario runs drive the live agent and spend real credits — a scenario that
  sends email sends email. The scenario harness is **API-only today** (no visual
  editor yet) and is available on the Business plan.
</Warning>

## A/B model comparison

The Monitoring page's **A/B model comparison** panel compares two models over
your real traffic. Pick any two models seen on the agent recently and a window;
the panel shows side-by-side query count, average judge score, pass rate,
average cost, and average duration. This is observational — the platform
doesn't split live traffic — so it's most useful right after switching your
chat model: run the same traffic for a few days and read the delta.

## Retrieval reproduction

Retrieval quality has its own test. The
[Retrieval Test page](/sidebar-menu/retrieval-test) runs your query through
every retrieval path side by side and can also reproduce the exact production
pipeline, so what you see in the test is what a real turn retrieves.

## What the platform deliberately doesn't do

* **No standardized benchmark suite.** Evaluation is per-agent and
  traffic-driven — the rubric, criteria, and scenarios are yours, and every
  number is computed from your real traffic.
* **No run-over-run comparison view.** Scenario runs persist individually; you
  compare runs yourself.
* **No exact replay.** Model outputs are non-deterministic. Pinning a low
  temperature in [Model Controls](/sidebar-menu/purpose) makes outputs more
  repeatable, not identical; for systematic re-testing, use scenario runs.