TypeSafe judge transport

This page is part of the operator guide. The other transports are in Judges. rubric_judge accepts transport: typesafe. jev is the same transport. The contact-routing pack uses typesafe for routing_judge. Critical risk dimensions stay on reference evaluators. A dimension scored only by a judge is still rejected. This is qualification evidence about a recorded judge call. It does not determine that a regulation, contract, or control has been met.

Install

The default Etalon install does not include the TypeSafe SDK.

pip install 'etalon[typesafe]'

etalon[typesafe] installs typesafe-sdk. pip install typesafe-sdk is the same library. The client reads TYPESAFE_API_KEY from the environment. Etalon does not accept the key as a flag and does not write it into the bundle. typesafe-sdk 0.7.1 requires pydantic>=2.12. Etalon accepts pydantic>=2.10.6, so pip install 'etalon[typesafe]' upgrades Pydantic to satisfy the SDK. The hashed requirements.txt lock used by CI stays on pydantic==2.10.6 and does not install the extra. Tests mock the client.

What is sent

One TypeSafeClient.system_one call per case. State holds the rubric, the critical and public labels, the referee label and behaviour, whether the case is ambiguous, the coverage fields, the case input, and the model output. Questions are stable across cases:

Choice verdict: pass or one of critical_misroute, soft_ambiguity, abstain_miss, label_mismatch, behaviour_mismatch.
Noul uncertainty: whether that verdict is uncertain.

The requested model is --judge-model when it is set, otherwise the pack’s judge.model (jev-latest on the contact-routing pack). The evidence model id is response.model, the id Jev reports for the call. prompt_sha256 on this transport is the canonical hash of the question document (instructions and criteria, including the rubric text). It does not cover the case text.

Uncertainty

Uncertainty is the Noul value, a probability from 0 to 1.

Near 0: the Noul says the verdict is clear.
Near 1: the Noul says the verdict is uncertain.
Missing, non-numeric, boolean, or outside 0 to 1: JudgeFailure. The run is INDETERMINATE. Etalon does not replace it with 1 - confidence.

Choice confidence and probabilities are stored on the judge evidence and in the evidence detail. They do not change passed. A choice outside the taxonomy, a probability map that does not match the criteria, or a choice that is not one of the reported probability peaks is also JudgeFailure. An API error, a missing SDK, or a missing TYPESAFE_API_KEY is the same failure. It is not stored as a model miss and it is not turned into a pass.

Live qualify

pip install 'etalon[typesafe]'
export TYPESAFE_API_KEY=...
python3 examples/mock_endpoint.py --port 8000 --pack examples/contact-routing

etalon qualify \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/contact-routing \
  --serving-config examples/contact-routing/serving.yaml \
  --system-prompt examples/contact-routing/system_prompt.txt \
  --model etalon-contact-baseline \
  --operator "Demo Operator" \
  --output runs/

--judge-model jev-1.13.0 pins a versioned id. Omit it to send the pack’s jev-latest alias. The bundle records the id in the response. Without TYPESAFE_API_KEY, the same command still captures the system under test and then marks the run INDETERMINATE because the judge did not return a usable verdict.

Tests

offline-mock is unchanged and does not open a socket. Unit tests pass a mocked client into the judge context. The contact-routing qualify tests replace open_client so CI does not call TypeSafe and does not need a key.

​TypeSafe judge transport

​Install

​What is sent

​Uncertainty

​Live qualify

​Tests