Judges

A pack may declare a rubric_judge evaluator. The runner records the judge model, the prompt or question hash, the prompt version, the rubric version, and an uncertainty value on every verdict. Reference evaluators remain the qualification metrics. Inspect, compare, and the HTML report show the reference route beside the judge when a judge ran. This is qualification evidence about a recorded judge call. It does not determine that a regulation, contract, or internal control has been met. examples/demo-pack does not declare a judge. examples/contact-routing declares routing_judge with transport: typesafe. There is no calibration UI.

Transports

evaluators.yaml sets config.transport. etalon pack validate accepts endpoint, offline-mock, typesafe, and jev.

Transport	What it does
`offline-mock`	Scores with the route taxonomy. Does not open a socket. Used by taxonomy tests.
`endpoint`	Sends the rubric prompt to an OpenAI-compatible chat model. `--judge-endpoint` selects that host and defaults to `--endpoint`. `--judge-model` overrides `judge.model`.
`typesafe`	Calls TypeSafe Jev through `typesafe-sdk`. `jev` is the same transport. Details are in TypeSafe judge.

--judge-endpoint applies to the endpoint transport. The TypeSafe client does not use it. The TypeSafe model id is --judge-model when set, otherwise the pack’s judge.model.

Failure

A missing uncertainty, a non-numeric uncertainty, an uncertainty outside 0 to 1, an unusable verdict, an API error, a missing SDK, or a missing TYPESAFE_API_KEY is a judge failure. The failure is not stored as a model miss and it is not turned into a pass. The run is INDETERMINATE. When the pack lists judge_uncertainty, uncertainty at or above config.uncertainty_maximum is also INDETERMINATE, unless a hard miss on a completed capture is already NOT_QUALIFIED. The uncertainty is still written on the decision. uncertainty_maximum is a number from 0 to 1. A pack with a judge must list judge_uncertainty in indeterminate_when. Choice confidence, on the TypeSafe transport, is stored and does not fill a missing uncertainty value.

Calibration

Calibration cases are not sent to the system under test. Each case carries a human score and a candidate output. The judge scores that candidate. The run reports judge–referee agreement, provenance (expert-authored or synthetic), and known-failure cases. Agreement below the declared record is calibration_invalid, and the decision is INDETERMINATE when that condition is listed.

Critical dimensions

A critical risk dimension cannot be scored only by a judge. etalon pack validate rejects that pack. The contact-routing pack keeps critical misroute on a reference evaluator. A dimension scored only by a judge is still rejected.

Fully local worked example

From a source checkout, with the model under test and judge already served on local OpenAI-compatible endpoints:

pip install -e .
python examples/local_judge_pack.py --output ./local-routing --model YOUR_LOCAL_JUDGE
etalon pack validate ./local-routing
etalon qualify --pack ./local-routing \
  --endpoint http://127.0.0.1:11434/v1 --model YOUR_MODEL_UNDER_TEST \
  --judge-endpoint http://127.0.0.1:8000/v1 --judge-model YOUR_LOCAL_JUDGE \
  --system-prompt ./local-routing/system_prompt.txt \
  --max-tokens 2048 --operator "Local evaluator" --output ./runs

The helper creates a separate pack identity, selects transport: endpoint, and recomputes the judge prompt hash. It does not launch or download models. The local judge must return the JSON verdict described above, including numeric uncertainty and a rationale. Here the judge independently assesses the ticket and candidate; deterministic reference evaluators still check the referee label. Both configured destinations must be within your boundary to claim fully local execution. No TypeSafe SDK or API key is needed and there is no remote fallback. The report records judge transport and destination host. For authenticated endpoints, the current endpoint transport uses the same configured API key for the model and judge; configure a suitable local gateway if they need separate keys. This example inherits synthetic calibration candidates. It does not establish that your local judge is reliable: poor agreement or unusable judge responses produce INDETERMINATE. Judge uncertainty is self-reported and should not be interpreted as a calibrated probability. Use a sufficient token budget for both models; model-under-test request extras are not forwarded to the judge.

​Judges

​Transports

​Failure

​Calibration

​Critical dimensions

​Fully local worked example