Packs

A pack is the specification a run executes: corpus, rubrics, evaluators, thresholds, decision policy, and requalification triggers. The runner does not choose those bounds. etalon qualify --pack takes the pack directory. etalon pack validate checks that directory against the schema and the runner’s pack rules. This repository is the open-source runner, coyos-ai/etalon. The packs checked in here are public fixtures only, under the runner’s Apache-2.0 licence.

`examples/demo-pack`

Pack id etalon.demo.ticket-routing, version 0.1.1. Synthetic ticket classification. Every case is invented. The pack does not declare a judge. Qualification scoring does not send the calibration corpus to the endpoint. serving.yaml describes the local fixture (engine: etalon-mock). It is not a GPU serving stack. Weight hashes stay unavailable unless you pass --model-index. python3 examples/demo-pack/build_corpus.py rewrites corpus/*.jsonl. The getting started path runs this pack against examples/mock_endpoint.py with --model etalon-demo-baseline.

`examples/contact-routing`

Pack id coyos.fs.contact-routing, version 0.3.1. This tree is a public synthetic fixture for runner demos and tests: contact and ticket intent classification. It stays in examples/contact-routing so the offline demo and CI keep a runnable pack, under the runner’s Apache-2.0 licence. Labels are billing, access, technical, security, and spam. Behaviour is classify, abstain, or escalate. Reference evaluators score the qualification metrics, including the critical misroute dimension. The rubric judge routing_judge uses transport: typesafe. Live use needs TYPESAFE_API_KEY and etalon[typesafe]. See TypeSafe judge. Start the fixture with the pack’s corpus so the scripted replies match the case text:

python3 examples/mock_endpoint.py --port 8000 --pack examples/contact-routing

python3 examples/contact-routing/build_corpus.py rewrites that pack’s corpus. The model ids for this fixture are listed in the repository README.

What a pack directory contains

pack.yaml is the manifest. It names the pack id and version, the intended use, the out-of-scope list, the OpenAI-compatible interface path, coverage dimensions, sampling defaults, reproducibility settings, mandatory fingerprint fields, and the artifact paths. The public fixtures point those artifacts at:

Artifact	Role
`corpus/qualification.jsonl`	Cases sent to the endpoint and scored for the decision.
`corpus/calibration.jsonl`	Held out. Not sent to the endpoint. Used for judge–referee agreement when a judge is declared.
`corpus/challenge.jsonl`	Sent to the endpoint. Reported separately from the qualification metrics.
`rubrics/`	Rubric documents referenced by tests.
`evaluators.yaml`	Evaluator definitions and the tests that bind them to metrics.
`thresholds.yaml`	Metric definitions, aggregation, and minimum sample size.
`qualification.yaml`	Requirements, the critical-failure rule, and `indeterminate_when`.
`triggers.yaml`	`always_requalify`, `review_required`, and `no_material_impact` trigger names.
`mappings.yaml`	Optional metric mappings. The demo fixture lists none.
`methodology.md`	Methodology text copied into `pack_summary.json`.

content_sha256 in pack.yaml may be null on disk. etalon pack validate computes the content hash and prints it. That hash is what the bundle records. Coverage in this runner is value-presence. A pack that sets coverage.model to anything else is rejected. The default, when the field is omitted, is value_presence. A pack that declares a judge must name transport as endpoint, offline-mock, typesafe, or jev, and must list judge_uncertainty under indeterminate_when. A critical risk dimension scored only by a judge is rejected. Reference or deterministic evaluators have to cover it. See Judges.

Validate

etalon pack validate examples/demo-pack
etalon pack validate examples/contact-routing
etalon pack validate examples/demo-pack --json

Exit 0 means the pack matched the schema and the pack rules. Exit 1 means it did not. The text report names the failing path.

Request parameters and response completeness

Optional request_extras in pack.yaml supplies backend-specific JSON fields, for example reasoning_effort: none or chat_template_kwargs: {enable_thinking: false}. qualify --extra-body '{"reasoning_effort":"none"}' merges over pack extras by key (nested objects are replaced, not deep-merged). {} merges nothing and leaves pack extras in place; it does not clear them. The effective extras are recorded under fingerprint.sampling.request_extras and in every captured request. Changes fire the existing sampling_parameters trigger. These are requested settings; the runner cannot prove a backend honored them. Model, messages, sampling, streaming and response-count fields are reserved. Credential-like keys are rejected, including aliases such as x-api-key, openai_api_key, api_token, and bearer, including when nested. Credentials belong in the existing environment configuration, never in extras. For check, resupply the pack or serving sampling configuration plus any --extra-body override. Extras supplied to check replace the extras object; use {} to explicitly clear it. Older bundles without extras mean {}. --extra-body alone is an error: extras cannot attest the other sampling values, and a check that ignored them would hide sampling drift. A serving config may declare request_extras under sampling or at the document root, the same place as pack.yaml. When both are present, the value under sampling is used. Omitting extras while re-observing sampling records {}. finish_reason: length is scored as truncated for applicable tests even if the partial output parses. It remains in the aggregate denominator as an applicable miss. A non-judge evaluator that raises on that partial text is not recorded as required_evaluator_failed, including when another evaluator on the same test was not applicable, so the crash does not also force INDETERMINATE. Etalon neither increases the budget nor retries with a different configuration automatically. The raw response preserves finish reason, reasoning and usage when provided. The json_object evaluator supports optional config.allowed_values, a mapping from field names to nonempty string lists, e.g. allowed_values: {label: [incident, request, question, complaint]}. Invalid values fail with invalid_value. This is separate from whether an allowed label matches the reference.

Challenge gates

Challenge results are informational unless qualification.yaml explicitly adds:

qualification:
  # Existing requirements, critical_failures and indeterminate_when remain required.
  challenge_critical_failures:
    metric: critical_misroute_rate
    maximum: 0.0

This applies an observed failure-rate limit to that metric’s challenge population, separately from the qualification population. One event exceeds a zero maximum. Insufficient challenge samples are INDETERMINATE unless a completed hard failure already decides NOT_QUALIFIED. Reports state when challenge results are informational. Only one judge evaluator is currently supported per pack; validation rejects multiple judges to avoid associating a test with the wrong judge evidence.

Class-specific qualification requirements

Reports include first-pass per-class support, precision, recall, and confusion matrices for qualification and challenge separately. Missing, invalid, truncated, and endpoint-error responses have separate columns. A class with no predictions has undefined precision, not zero or one. Repeated passes are excluded from these descriptive counts. Class names come from the corpus reference labels. To gate qualification on one class, add a reference evaluator and test:

evaluators:
  - id: complaint_recall
    version: 0.1.0
    kind: reference
    type: class_recall
    model_use: none
    config: {class_label: complaint, field: label, expected_field: label}
tests:
  - id: complaint_recall
    metric: complaint_recall
    evaluator: complaint_recall
    rubric: exact_label
    severity: high
    risk_dimensions: [misclassification]
    corpus: [qualification, challenge]

Add this metric to thresholds.yaml:

metrics:
  - id: complaint_recall
    definition: Fraction of reference complaints assigned the complaint label.
    test: complaint_recall
    aggregation: pass_rate
    population: applicable
    minimum_sample_size: 15

Then add matching threshold and mandatory requirement entries in thresholds.yaml and qualification.yaml, with a written methodology rationale. The existing Wilson-bound decision rules apply, so a high overall accuracy cannot compensate for missing a mandatory class requirement. Choose sample sizes and thresholds for the use case; the example minimum is not a recommendation. class_precision uses the same configuration but applies only to responses predicted as that class. Validation requires population: applicable to prevent dilution by other classes.

​Packs

​examples/demo-pack

​examples/contact-routing

​What a pack directory contains

​Validate

​Request parameters and response completeness

​Challenge gates

​Class-specific qualification requirements