> ## Documentation Index
> Fetch the complete documentation index at: https://docs.etalon.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Packs

> What a pack is, and the two public fixtures checked in with the runner.

# Packs

A pack is the specification a run executes: corpus, rubrics, evaluators,
thresholds, decision policy, and requalification triggers. The runner does
not choose those bounds. `etalon qualify --pack` takes the pack directory.
`etalon pack validate` checks that directory against the schema and the
runner's pack rules.

This repository is the open-source runner,
[coyos-ai/etalon](https://github.com/coyos-ai/etalon). The packs checked in
here are public fixtures only, under the runner's Apache-2.0 licence.

## `examples/demo-pack`

Pack id `etalon.demo.ticket-routing`, version `0.1.1`. Synthetic ticket
classification. Every case is invented. The pack does not declare a judge.
Qualification scoring does not send the calibration corpus to the endpoint.

`serving.yaml` describes the local fixture (`engine: etalon-mock`). It is not
a GPU serving stack. Weight hashes stay `unavailable` unless you pass
`--model-index`.

`python3 examples/demo-pack/build_corpus.py` rewrites `corpus/*.jsonl`.

The [getting started](/docs/getting-started.md) path runs this pack against
`examples/mock_endpoint.py` with `--model etalon-demo-baseline`.

## `examples/contact-routing`

Pack id `coyos.fs.contact-routing`, version `0.3.1`. This tree is a
public synthetic fixture for runner demos and tests: contact and ticket
intent classification. It stays in `examples/contact-routing` so the offline
demo and CI keep a runnable pack, under the runner's Apache-2.0 licence.

Labels are billing, access, technical, security, and spam. Behaviour is
classify, abstain, or escalate. Reference evaluators score the qualification
metrics, including the critical misroute dimension. The rubric judge
`routing_judge` uses `transport: typesafe`. Live use needs `TYPESAFE_API_KEY`
and `etalon[typesafe]`. See [TypeSafe judge](/docs/typesafe-judge.md).

Start the fixture with the pack's corpus so the scripted replies match the
case text:

```bash theme={null}
python3 examples/mock_endpoint.py --port 8000 --pack examples/contact-routing
```

`python3 examples/contact-routing/build_corpus.py` rewrites that pack's
corpus. The model ids for this fixture are listed in the repository README.

## What a pack directory contains

`pack.yaml` is the manifest. It names the pack id and version, the intended
use, the out-of-scope list, the OpenAI-compatible interface path, coverage
dimensions, sampling defaults, reproducibility settings, mandatory fingerprint
fields, and the artifact paths. The public fixtures point those artifacts at:

| Artifact                     | Role                                                                                           |
| ---------------------------- | ---------------------------------------------------------------------------------------------- |
| `corpus/qualification.jsonl` | Cases sent to the endpoint and scored for the decision.                                        |
| `corpus/calibration.jsonl`   | Held out. Not sent to the endpoint. Used for judge–referee agreement when a judge is declared. |
| `corpus/challenge.jsonl`     | Sent to the endpoint. Reported separately from the qualification metrics.                      |
| `rubrics/`                   | Rubric documents referenced by tests.                                                          |
| `evaluators.yaml`            | Evaluator definitions and the tests that bind them to metrics.                                 |
| `thresholds.yaml`            | Metric definitions, aggregation, and minimum sample size.                                      |
| `qualification.yaml`         | Requirements, the critical-failure rule, and `indeterminate_when`.                             |
| `triggers.yaml`              | `always_requalify`, `review_required`, and `no_material_impact` trigger names.                 |
| `mappings.yaml`              | Optional metric mappings. The demo fixture lists none.                                         |
| `methodology.md`             | Methodology text copied into `pack_summary.json`.                                              |

`content_sha256` in `pack.yaml` may be null on disk. `etalon pack validate`
computes the content hash and prints it. That hash is what the bundle
records.

Coverage in this runner is value-presence. A pack that sets `coverage.model`
to anything else is rejected. The default, when the field is omitted, is
`value_presence`.

A pack that declares a judge must name `transport` as `endpoint`,
`offline-mock`, `typesafe`, or `jev`, and must list `judge_uncertainty`
under `indeterminate_when`. A critical risk dimension scored only by a judge
is rejected. Reference or deterministic evaluators have to cover it. See
[Judges](/docs/judges.md).

## Validate

```bash theme={null}
etalon pack validate examples/demo-pack
etalon pack validate examples/contact-routing
etalon pack validate examples/demo-pack --json
```

Exit 0 means the pack matched the schema and the pack rules. Exit 1 means it
did not. The text report names the failing path.

## Request parameters and response completeness

Optional `request_extras` in `pack.yaml` supplies backend-specific JSON fields,
for example `reasoning_effort: none` or `chat_template_kwargs: {enable_thinking: false}`.
`qualify --extra-body '{"reasoning_effort":"none"}'` merges over pack extras by
key (nested objects are replaced, not deep-merged). `{}` merges nothing and
leaves pack extras in place; it does not clear them. The effective extras are recorded under
`fingerprint.sampling.request_extras` and in every captured request. Changes
fire the existing `sampling_parameters` trigger. These are requested settings;
the runner cannot prove a backend honored them. Model, messages, sampling,
streaming and response-count fields are reserved. Credential-like keys are
rejected, including aliases such as `x-api-key`, `openai_api_key`, `api_token`,
and `bearer`, including when nested. Credentials belong in the
existing environment configuration, never in extras.

For `check`, resupply the pack or serving sampling configuration plus any
`--extra-body` override. Extras supplied to `check` replace the extras object;
use `{}` to explicitly clear it. Older bundles without extras mean `{}`.
`--extra-body` alone is an error: extras cannot attest the other sampling
values, and a check that ignored them would hide sampling drift.

A serving config may declare `request_extras` under `sampling` or at the
document root, the same place as `pack.yaml`. When both are present, the
value under `sampling` is used. Omitting extras while re-observing sampling
records `{}`.

`finish_reason: length` is scored as `truncated` for applicable tests even if
the partial output parses. It remains in the aggregate denominator as an
applicable miss. A non-judge evaluator that raises on that partial text is not
recorded as `required_evaluator_failed`, including when another evaluator on
the same test was not applicable, so the crash does not also force
INDETERMINATE. Etalon neither increases the budget nor retries with a
different configuration automatically. The raw response preserves finish
reason, reasoning and usage when provided.

The `json_object` evaluator supports optional `config.allowed_values`, a mapping
from field names to nonempty string lists, e.g. `allowed_values: {label: [incident,
request, question, complaint]}`. Invalid values fail with `invalid_value`.
This is separate from whether an allowed label matches the reference.

## Challenge gates

Challenge results are informational unless `qualification.yaml` explicitly adds:

```yaml theme={null}
qualification:
  # Existing requirements, critical_failures and indeterminate_when remain required.
  challenge_critical_failures:
    metric: critical_misroute_rate
    maximum: 0.0
```

This applies an observed failure-rate limit to that metric's challenge population,
separately from the qualification population. One event exceeds a zero maximum.
Insufficient challenge samples are INDETERMINATE unless a completed hard failure
already decides NOT\_QUALIFIED. Reports state when challenge results are informational.

Only one judge evaluator is currently supported per pack; validation rejects
multiple judges to avoid associating a test with the wrong judge evidence.

## Class-specific qualification requirements

Reports include first-pass per-class support, precision, recall, and confusion
matrices for qualification and challenge separately. Missing, invalid, truncated,
and endpoint-error responses have separate columns. A class with no predictions
has undefined precision, not zero or one. Repeated passes are excluded from these
descriptive counts. Class names come from the corpus reference labels.

To gate qualification on one class, add a reference evaluator and test:

```yaml theme={null}
evaluators:
  - id: complaint_recall
    version: 0.1.0
    kind: reference
    type: class_recall
    model_use: none
    config: {class_label: complaint, field: label, expected_field: label}
tests:
  - id: complaint_recall
    metric: complaint_recall
    evaluator: complaint_recall
    rubric: exact_label
    severity: high
    risk_dimensions: [misclassification]
    corpus: [qualification, challenge]
```

Add this metric to `thresholds.yaml`:

```yaml theme={null}
metrics:
  - id: complaint_recall
    definition: Fraction of reference complaints assigned the complaint label.
    test: complaint_recall
    aggregation: pass_rate
    population: applicable
    minimum_sample_size: 15
```

Then add matching threshold and mandatory requirement entries in `thresholds.yaml`
and `qualification.yaml`, with a written methodology rationale. The existing
Wilson-bound decision rules apply, so a high overall accuracy cannot compensate
for missing a mandatory class requirement. Choose sample sizes and thresholds
for the use case; the example minimum is not a recommendation. `class_precision`
uses the same configuration but applies only to responses predicted as that class.
Validation requires `population: applicable` to prevent dilution by other classes.