Evaluation harness

Evaluation harness

laya.evals turns a labelled dataset into a repeatable score, and a baseline into a pass/fail gate, so a quality change is a reviewable diff instead of a hand-check.

The metric math and the dataset parser are pure Python plus numpy and never import torch, so they run with no weights. Running a dataset against a checkpoint needs the checkpoint and takes its normal load time.

Quickstart

# check the format without a model
laya-evals validate research/evals/fixture.jsonl

# score a labelled set on one checkpoint, with thresholds and a baseline
laya-evals run data.jsonl --model english --device cpu \
    --min-accuracy 0.8 --max-ece 0.05 --score-within 0.25 --slice language \
    --json report.json --markdown report.md

# compare a saved report to a baseline
laya-evals compare report.json --baseline baseline.json --tolerance choice_accuracy=0.02

laya eval ... is the same thing through the main CLI, so laya eval validate data.jsonl works too.

Exit codes: 0 on success, 1 when a threshold or a baseline tolerance fails, 2 on a usage error. run prints the overall metrics and any requested slices to stdout, and writes the full report and a Markdown summary when --json / --markdown are given.

Evaluating an ONNX export

run --onnx PATH scores an exported ONNX model through ONNXAgent instead of the torch Router, so an ONNX deployment (including an INT8 copy from scripts/export_onnx.py --quantize) gets gated by the same thresholds and baselines as the torch path:

python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx --quantize
laya-evals run data.jsonl --onnx laya.int8.onnx --max-ece 0.05

--model names the checkpoint the export came from — a Hub id or local path, not a Router short name like english, since there is no Router on this path (default convaiinnovations/laya). Its config and tokenizer are loaded from there. The agent serves one checkpoint, so a dataset row whose model field names a different one fails with a clear error rather than being silently answered by the wrong model; --device does not apply. --batch-size uses the agent’s batch API when it has one and falls back to one call per state otherwise; --sort-by-length is forwarded to that batch API, which the per-state fallback has no group to reorder. The report’s config block records the onnx path.

Measured on research/evals/fixture.jsonl (12 labelled rows, English checkpoint, CPU):

The fp32 export reproduces the torch numbers exactly, and the quantized copy moves score_mae by 0.009 and ece by 0.006 — the kind of drift compare --tolerance is meant to gate.

Dataset format

One JSON object per line (JSONL). Blank lines and lines starting with # are ignored.

research/evals/dataset.template.jsonl has a commented example.

Metrics

Each metric is computed per answer where it applies and aggregated over the dataset:

Add ScoreWithin(0.25) to the evaluator list for a tolerance metric; the default set is choice_accuracy, noul_accuracy, score_mae, mean_confidence, plus ece. From the CLI the same thing is one flag: laya-evals run data.jsonl --score-within 0.25 reports score_within_0.25 beside the defaults, and the flag repeats, so --score-within 0.25 --score-within 0.5 reports both.

A tolerance metric needs a score answer with a numeric label, so on a dataset without one it has no value: run names the metric it could not compute instead of publishing a silent zero, and a --min / --max gate naming that metric fails as missing. The tolerances a run was asked for are recorded in the report’s config block, so a reviewed baseline says which columns it expects.

Batching and timing

--batch-size N scores up to N consecutive rows that share a checkpoint and a question schema in one call. Both timing metrics come from the same measurements and answer different questions: every row of a batch returns when the batch does, so its latency is the whole call, while its cost_per_decision is 1/N of it. Batching therefore raises latency_* and lowers cost_per_decision_* on an unchanged set of decisions, and --max latency_p50_ms=... asks whether requests were served fast, not whether the run was cheap. With no --batch-size the two agree.

compare ignores any *_ms metric unless a tolerance names it, so these never fail a baseline on timing noise. What the harness actually did – the batch size asked for, the runner shape it resolved to, how many rows shared a call, and the largest chunk – is recorded in the report’s config.timing, because the flag alone does not say whether anything was batched. Those counters record the calls issued, not the calls that returned: with on_error=skip, a chunk whose call raised still counts in rows_grouped and max_chunk, next to its entries in config.errored. The two *_ms metrics count only the calls that returned, so a failed call never contributes a latency it did not measure.

Grouping the rows inside a batch

--sort-by-length groups similarly sized rows into the same forward pass, so each pass pads to a shorter maximum instead of to the longest row in it. It is the shape of the calls, not their answers: results come back in the same order and score identically, which is why research/ can report 2.15x over 10,000 tickets with no decision changing.

There has to be more than one pass to reorder, so it takes effect only with a --batch-size N below the number of rows the run groups. config.timing keeps the two claims apart: sort_by_length is what the command line said, sort_by_length_sent is what reached the runner. A run with no --batch-size asks for something that cannot happen, and says so with sent: false; a runner whose predict_batch predates the knob is scored unsorted rather than raising TypeError halfway through a long run.

Slices

compare and run report overall numbers and, for --slice language|model|qid|tag, the same metrics per slice value, so a regression in one language or one question is visible without reading the aggregate. The model slice holds the checkpoint that answered each row: the Router’s own choice per request, or the runner’s model for a runner that does not route.

Run identity

run records what it measured in the report’s config block, so the artifact a reviewer reads is reviewable on its own:

dataset is a path, and a path is not an identity: a dataset can be edited in place, moved, or refetched under the same name, and a CI cache can hand two runs the same filename and different bytes. questions_sha256 covers what was asked rather than how many rows there were, so adding states to an unchanged question set leaves the fingerprint alone – dataset_sha256 still moves, and adding a row is a change to the data, not to the question.

It covers instructions too, because the instruction text is the prompt. build_sequence renders "<type> question: <instructions>" into the tokenized head, Agent refuses a question without one (“add the text the model should answer”), and Laya’s own question identity already counts it: Router._question_schema and this harness’s batch grouping both key on the whole questions dict, and tests/test_router_batch.py pins that rewording instructions alone moves a row into its own batch group. So does a reworded instruction still compare equal to a baseline? No – and that is the point. “Judge whether a refund is justified” and “Be conservative and only approve explicit refund requests” ask different questions, and the metric gate can only notice when the difference happens to move a number further than the tolerance you named. Naming a choice option is the same argument: criteria is the decision space, and the metamorphic checks in research/eval/metamorphic.py exist because renaming a label flips answers.

Nothing about the instruction text is normalized except the one step the engine itself applies: a non-string instructions is hashed as json.dumps(ins, ensure_ascii=False), matching Agent._to_internal. So whitespace and wording both count, and a rewording that a human considers a copy edit is treated as a new experiment. That is the honest default – the alternative is a similarity heuristic standing between a run and its baseline, and no evaluation system in common use has one.

Nothing time-bearing is recorded, so a report is still byte-reproducible for a fixed runner.

REPORT_SCHEMA, questions_fingerprint(dataset) and file_fingerprint(path) are public, so a caller driving laya.evals.evaluate directly gets the same identity a CLI run does.

Baseline and CI gate

• Keep the dataset, a baseline report (--json output you have reviewed), and the tolerances together, committed, so a change is a reviewable diff. --tolerance METRIC=VALUE is the maximum absolute drift allowed for that metric.
• laya-evals run ... --baseline baseline.json --tolerance ... exits non-zero on drift, so it drops into CI unchanged. laya.evals.EvalReport.compare and assert_regression expose the same logic for tests.

The metric gate answers “did the numbers move”. It cannot answer “were these the same numbers”, because compare reads overall and only overall – so a baseline recorded against one dataset would pass a candidate scored on another, with identical arithmetic. EvalReport.comparable_to closes that: it compares schema, dataset_sha256 and questions_sha256, and run --baseline and compare still print every delta, then fail with a non-zero exit naming the key and both values:

FAIL: baseline is not comparable: dataset_sha256 (dataset bytes): baseline is <sha>, this run is <sha>

A key missing on either side is unknown, not a conflict, so every report written before the identity existed keeps comparing exactly as it did. That includes the scheduled gate’s baseline below, which comes from research/eval/ and has no config.schema at all.

Two CI surfaces use this:

• a weight-free job in .github/workflows/ci.yml runs tests/test_evals.py and tests/test_evals_api.py, so metric math, dataset parsing and the CLI are covered on every PR without downloading a checkpoint;
• .github/workflows/evals.yml runs weekly, before a release and on demand: it evaluates the English checkpoint on the MASSIVE English suite and compares to research/results/eval_english_51_languages.json with the tolerances in research/evals/thresholds.json. It uploads the report as an artifact and does not block a PR.

The harness is deterministic for a fixed checkpoint revision, so a report is reproducible. run records the dataset, model and device, plus the timing facts of the run, in the report’s config block, and revisions: the commit each checkpoint that answered was actually loaded from. --revision <SHA> pins that commit for every checkpoint the run loads, and --revision english=<SHA> pins one checkpoint (repeatable) — which is the form an auto-routing run wants, since the three checkpoints are three repositories and one commit cannot exist in all of them. Left unpinned, the run takes the checkpoint’s default branch and the report still says which commit answered, so a baseline drift can be attributed to the weights or to the code. laya/revisions.py publishes reviewed commit SHAs in PINNED_REVISIONS for callers who want to opt in. With --onnx, only a bare --revision <SHA> applies, to the config and tokenizer download.

Adding the real labelled set

Drop a JSONL in research/evals/ and a reviewed baseline beside it, then point a workflow (or research/evals/check_regression.py) at both. The format is the same as the fixture; nothing in the harness knows about MASSIVE.