> ## Documentation Index
> Fetch the complete documentation index at: https://docs.etalon.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Judges

> Rubric-judge transports and what a recorded judge call is evidence of.

# Judges

A pack may declare a `rubric_judge` evaluator. The runner records the judge
model, the prompt or question hash, the prompt version, the rubric version,
and an uncertainty value on every verdict. Reference evaluators remain the
qualification metrics. Inspect, compare, and the HTML report show the
reference route beside the judge when a judge ran.

This is qualification evidence about a recorded judge call. It does not
determine that a regulation, contract, or internal control has been met.

`examples/demo-pack` does not declare a judge. `examples/contact-routing`
declares `routing_judge` with `transport: typesafe`. There is no calibration
UI.

## Transports

`evaluators.yaml` sets `config.transport`. `etalon pack validate` accepts
`endpoint`, `offline-mock`, `typesafe`, and `jev`.

| Transport      | What it does                                                                                                                                                            |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `offline-mock` | Scores with the route taxonomy. Does not open a socket. Used by taxonomy tests.                                                                                         |
| `endpoint`     | Sends the rubric prompt to an OpenAI-compatible chat model. `--judge-endpoint` selects that host and defaults to `--endpoint`. `--judge-model` overrides `judge.model`. |
| `typesafe`     | Calls TypeSafe Jev through `typesafe-sdk`. `jev` is the same transport. Details are in [TypeSafe judge](/docs/typesafe-judge.md).                                       |

`--judge-endpoint` applies to the `endpoint` transport. The TypeSafe client
does not use it. The TypeSafe model id is `--judge-model` when set, otherwise
the pack's `judge.model`.

## Failure

A missing uncertainty, a non-numeric uncertainty, an uncertainty outside 0
to 1, an unusable verdict, an API error, a missing SDK, or a missing
`TYPESAFE_API_KEY` is a judge failure. The failure is not stored as a model
miss and it is not turned into a pass. The run is `INDETERMINATE`.

When the pack lists `judge_uncertainty`, uncertainty at or above
`config.uncertainty_maximum` is also `INDETERMINATE`, unless a hard miss on
a completed capture is already `NOT_QUALIFIED`. The uncertainty is still
written on the decision. `uncertainty_maximum` is a number from 0 to 1.
A pack with a judge must list `judge_uncertainty` in `indeterminate_when`.

Choice confidence, on the TypeSafe transport, is stored and does not fill a
missing uncertainty value.

## Calibration

Calibration cases are not sent to the system under test. Each case carries a
human score and a candidate output. The judge scores that candidate. The run
reports judge–referee agreement, provenance (`expert-authored` or
`synthetic`), and known-failure cases. Agreement below the declared record is
`calibration_invalid`, and the decision is `INDETERMINATE` when that
condition is listed.

## Critical dimensions

A critical risk dimension cannot be scored only by a judge. `etalon pack
validate` rejects that pack. The contact-routing pack keeps critical misroute
on a reference evaluator. A dimension scored only by a judge is still
rejected.

## Fully local worked example

From a source checkout, with the model under test and judge already served on
local OpenAI-compatible endpoints:

```bash theme={null}
pip install -e .
python examples/local_judge_pack.py --output ./local-routing --model YOUR_LOCAL_JUDGE
etalon pack validate ./local-routing
etalon qualify --pack ./local-routing \
  --endpoint http://127.0.0.1:11434/v1 --model YOUR_MODEL_UNDER_TEST \
  --judge-endpoint http://127.0.0.1:8000/v1 --judge-model YOUR_LOCAL_JUDGE \
  --system-prompt ./local-routing/system_prompt.txt \
  --max-tokens 2048 --operator "Local evaluator" --output ./runs
```

The helper creates a separate pack identity, selects `transport: endpoint`, and
recomputes the judge prompt hash. It does not launch or download models. The
local judge must return the JSON verdict described above, including numeric
uncertainty and a rationale. Here the judge independently assesses the ticket
and candidate; deterministic reference evaluators still check the referee label.

Both configured destinations must be within your boundary to claim fully local
execution. No TypeSafe SDK or API key is needed and there is no remote fallback.
The report records judge transport and destination host. For authenticated
endpoints, the current endpoint transport uses the same configured API key for
the model and judge; configure a suitable local gateway if they need separate keys.

This example inherits synthetic calibration candidates. It does not establish
that your local judge is reliable: poor agreement or unusable judge responses
produce INDETERMINATE. Judge uncertainty is self-reported and should not be
interpreted as a calibrated probability. Use a sufficient token budget for both
models; model-under-test request extras are not forwarded to the judge.