# oryxeval changelog Every release and what changed in it. Newest first; versions follow calver (`YY.M.D`). All notable changes to **oryxeval** are recorded here. This file is read by humans *and* by AI coding agents diagnosing regressions after an upgrade, so the format is load-bearing: - Newest first. One `## [version] - YYYY-MM-DD` heading per release; version is calver `YY.M.D` matching `pyproject.toml` / `oryxeval.__version__`. Unreleased work goes under `## [Unreleased]`. - Group bullets under `### Added` / `### Changed` / `### Deprecated` / `### Removed` / `### Fixed` / `### Security` (Keep a Changelog: https://keepachangelog.com/). - **Every breaking change is a bullet that STARTS with the literal token `BREAKING:`** and carries a same-bullet `Migration:` clause with the old-\>new fix. - **Name the actual symbol in backticks** (`` `sweep` ``, `` `EvalResult.verdict()` ``), never prose. Agents grep this file for the symbol in their traceback. ## \[26.11.6\] - 2026-11-06 ### Added - `oryxeval`: first release as a standalone package. The code is `oryxflow.evals` as of oryxflow 26.10.22, unchanged apart from the import name. Requires `oryxflow>=26.11.6`. Migration: change only the import lines, `import oryxflow.evals as ev` -\> `import oryxeval as ev` and `from oryxflow.evals. import X` -\> `from oryxeval. import X`; existing eval caches then stay valid (no cache key contains the module path). Keep `import oryxflow` and leave every function body as it is: `code_version()`, `case()` and their helpers are hashed into the cache key, so any edit there (even `oryxflow.hash_files` -\> `ev.hash_files`) re-runs every arm. The next run reuses the cache but prints StalenessWarnings; clear them once, from `evals/`: `python -c "import oryxflow; oryxflow.accept_code()"` (the `accept_code()` the warning names covers only the scoring stage). New evals use `ev.hash_files`. - `oryxeval` re-exports oryxflow's `Parameter`, `IntParameter`, `FloatParameter`, `BoolParameter`, `ChoiceParameter`, `ListParameter`, `DictParameter`, `DateParameter`, `EnumParameter` and `hash_files` (the same objects). ### Fixed - `pyyaml` is a declared dependency (`load_cases` reads `.yaml` files). - `oryxeval.cli` and `oryxeval.build_app` import and resolve on first use as documented. ## Before the split (shipped as oryxflow.evals) The entries below are from oryxflow's changelog, verbatim, under their original versions. ## \[26.10.5\] - 2026-10-05 ### Added - `oryxflow.evals`: **a cell is now two cached stages -- the model calls, then the scoring.** `TaskEval` generates an `Outputs` task (`TaskJson`, one record per case x repeat: output, inputs, timing, metrics, attributes) that the scoring task requires. Editing an evaluator, changing a judge rubric or swapping the judge model re-scores the stored outputs with ZERO calls to the model under test; editing what the arm reads re-runs both. Before, one `code_version()` keyed both, so a scorer change either re-billed every call or served stale scores. A subclass's `code_version()` keeps its meaning (what the ARM reads) and now keys the model-call stage only; the new `TaskEval.scorer_version()` keys the scoring stage and defaults to every evaluator's code and configuration (`as_spec()`). A typed output is rebuilt with `model_validate`, so `ctx.output.attr` scorers are unchanged. Evaluators that read the run's span tree (`HasMatchingSpan`, `ToolCorrectness`, `TrajectoryMatch`, `ArgumentCorrectness`, `MaxToolCalls`, `MaxModelRequests`, or any with `needs_trace = True`) score in the model-call stage, because spans are not stored. - `oryxflow.evals`: `sweep(rescore=True)` / `ev.cli --rescore` -- re-run only the scorers over the stored outputs (after a transient judge failure, say). The bill prints `re-scoring N arms from stored outputs: no model calls` and counts an LLM judge's calls separately. - `oryxflow.evals`: `baseline=` on `sweep()` (or a `baseline` attribute on a `TaskEval`; default: an arm named `baseline`). The verdict then reports every arm's paired difference from it with its interval, and names no winner -- mixed results need a written judgement, not a pass/fail. - `oryxflow.evals`: `EvalResult.side_by_side(path=None, all=False, template=None)` -- every arm's output per case as markdown, by default only the cases where the arms disagree on a column the verdict reads, plus failures. Rendered with Jinja from the stored outputs (no model call); `template=` takes a path or a string. - `oryxflow.evals`: `load_cases()` also takes a list of row dicts, routed exactly like a file's rows -- for a probe that keeps a handful of cases inline. - `oryxflow.evals`: `ev.cli --side-by-side` writes `EvalResult.side_by_side()` to the eval module's own `results/--side-by-side.md`, wherever the run was launched from. - `oryxflow.evals`: `ev.cli` warns when launched from a directory other than the eval script's own -- the cache and relative paths resolve against the working directory, so such a run reads and writes a different cache. - `oryxflow.evals` — an LLM-eval harness over [pydantic-evals](), installed with the new `evals` extra (`pip install oryxflow[evals]`; pydantic-evals, scipy, typer). Nothing in it is imported by `oryxflow/__init__.py`, so a user without the extra is unaffected; import it explicitly as `import oryxflow.evals as ev`. pydantic-evals keeps scoring — cases, evaluators, repeats, concurrency, retries, the success/failure split — and oryxflow adds the matrix and the cache: one parquet per arm, so a re-run of an unchanged arm makes zero calls and editing the target function or the case file re-runs only the cells that read it. - `ev.sweep(target, cases=..., metric=..., **arms)` runs the grid and returns an `ev.EvalResult` (`.df`, `.verdict()`, `.report()`, `.best()`). `target` may be a plain async function — no class and no scaffold for a first eval — or an `ev.TaskEval` subclass. Any keyword holding a list is an axis. The projected call count, the cached/new split and a cost estimate print **before** anything is billed, and a swept parameter the function cannot accept raises rather than running N identical cells labelled as branches. `watch=` (glob patterns, or a callable) folds files the target **reads** — a prompt template, a `.sql` file — into the cache key, so editing one re-runs the arms that read it; without it a prompt sits outside the key entirely and an edit re-runs nothing. - `ev.TaskEval` is the class form (a `TaskPqPandas` with `persists = ['cases']`): declare `dataset`, `metric`, `guardrail`, `slices` and one `case()` coroutine. `preflight()` makes one real call before the sweep, so a credential failure costs one call instead of a matrix; above `max_failure_rate` (default 20%) the run raises `ev.EvalFailureRateError` and **saves nothing**, because a parquet of exceptions is indistinguishable from a measurement. `concurrency` is `significant=False`, so it cannot fragment the cache. - `ev.load_cases(path, inputs=..., expected=...)` reads cases from `.csv` / `.yaml` / `.jsonl`, routing columns by name; a `@path/to/file.md` value is replaced by that file's text. `synthetic` cases are reported beside the headline and never in it, and `holdout` cases are withheld from every arm with the count stated. - `ev.Metric(label, column, where=, higher_is_better=, budget=, coverage=)` is the number a sweep is judged on. `ev.rate_ci` / `ev.delta_ci` (scipy, clustered by case) put a confidence interval on it. The verdict names **no winner** when the delta interval spans zero, never says "wins" unqualified when a guardrail budget broke, flags a metric that is identically 0% or 100% in every arm, and flags a denominator that moved between arms. - `ev.git_tree(ref, paths, dest=, expect_absent=)` materializes a baseline arm from a git ref (byte-exact, and the only mechanism that can restore a *deleted* file); `ev.git_sha(ref)` is the resolved SHA to use as that arm's `code_version()`. `ev.Variant(old, new)` is the anchored-replacement probe for a rewrite that has not shipped, and raises when its anchor is gone rather than silently becoming a no-op. - `ev.cli(TaskEvalSubclass)` builds a Typer command line whose flags are **derived** from the task's Parameters, so adding a parameter adds a repeatable flag and the two cannot drift. Built-ins: `--repeats --concurrency --reset --check --csv --yes`; `--check` runs `preflight()` only and exits. `ev.cli` / `ev.build_app` resolve on first use, so `import oryxflow.evals` does not pay for Typer. ### Changed - `oryxflow.evals`: the `evals` extra now requires `pydantic-evals>=2` (for the tool-call evaluators, `CaseLifecycle` and Windows-safe report output) and adds `jinja2` (for `side_by_side()`). - `oryxflow.evals`: `reset=True` / `--reset` discards BOTH stages, model calls included. Use `rescore` to re-run only the scorers. - `oryxflow.evals`: the function form's arm identity now also follows what the target CALLS (`codehash.callable_code_hash`), not only the target's own body -- editing a project-local helper or prompt loader re-runs the arms that use it. - `oryxflow.evals`: `rep` is read from pydantic-evals' own `" [i/N]"` naming when outputs are stored, instead of being re-derived from row order (`TaskEval._assign_reps` is now a no-op kept as an override point). - `oryxflow.evals`: **judge validation.** `ev.judge_alignment(df, judge, human)` reports a judge's TPR, TNR and Cohen's kappa against human labels, and `ev.corrected_rate(observed, alignment)` applies the Rogan-Gladen correction -- the standard estimator for a proportion measured with an imperfect test, which is what an LLM judge is. Declare it on the metric (`ev.Metric('pairing', 'pairing_holds', human='pairing_human')`) and the verdict prints the alignment and the corrected rate per arm. Kappa is reported *and labelled* as the one to read, because raw agreement inflates under class imbalance: on a 90%-pass set a judge that passes everything scores 90% agreement and has learned nothing (kappa 0). The correction is REFUSED when `TPR + TNR <= 1`, where the estimator's denominator collapses and the "correction" is amplified noise. This matters beyond tidiness: an imperfect judge pulls every rate toward its own error floor, so it shrinks the very difference a sweep exists to detect -- a true 85% vs 55% reads as 78% vs 60% through a judge that is 88% accurate. - `oryxflow.evals`: a **guardrail inside budget on the point estimate but not on its interval** now says so (`the interval for prod reaches 25% -- at this sample size the guardrail is not demonstrated, only not yet violated`). At n=12, `0% (0/12)` is entirely consistent with a true rate above a 5% budget, and the bound is the thing to act on. - `oryxflow.evals`: a **lone arm scoring 100%** is flagged as saturated rather than reported as a win -- a case set that never fails cannot show a regression or an improvement, so the response is harder cases. (Multi-arm saturation was already caught by the dead-metric check.) - `oryxflow.evals`: `ev.PromptArm` -- where one arm READS its prompt files and the cache key that MATCHES them, in one object (`.dir()` and `.code_version()`). Written by hand those are two places that must agree, and when they drift the failure is silent: a baseline arm reading a checked-out tree but keyed on a hash of the LIVE files re-runs when a live prompt is edited, and does *not* re-run when it is pointed at a different ref. A working-tree arm is keyed on the content of its `globs`, a `ref` arm on the resolved sha; both hand back the same layout via `subdir=`, so a loader never has to know which arm it got. An arm with no key at all is refused at construction. - `oryxflow.evals`: the cost confirmation now offers the one-call preflight **at the prompt** -- `Run 56 new calls for X? (y/n/c=check one call first)`. `c` spends one call, reports whether the wiring works, and asks again. A `--check` flag only helps someone who already knows it exists, and the person who needs it most is running the eval for the first time. Deliberately still *offered* rather than run automatically before the bill: a tool that bills money should not make its first call unasked, however cheap. - `oryxflow.evals`: `sweep()` accepts `evaluators=` and passes them to the dataset it generates. The target function is handed only `inputs`, so scoring against a case's `expected_output` — the shape every classifier eval has — previously required building a `Dataset` by hand and passing both `cases=` and `dataset=`. - `oryxflow.evals`: a sweep axis with a **single value** no longer appears in the arm name. It is identical in every arm, so it added width to every row of every table and distinguished nothing: `model_id=['gemini-3.5-flash'], prompt_version=['prod','preship']` now names the arms `prod` and `preship` rather than `model_id_gemini-3.5-flash_prompt_version_prod`. When *no* axis varies the full name is kept, since a lone cell called `default` says less. - `oryxflow.evals`: `verdict()` no longer reprints the call-count header that `sweep()` already emitted as the bill, and "Weakest slices" lists only slices actually **below** the arm's own rate — a 100% slice under that heading reads as a warning about something that is fine. ### Fixed - `oryxflow.evals`: a `TaskEval` subclass now re-runs when its case set changes. Only the function form folded the cases into its cache key, so editing `cases.csv` under the class form served the previous cases' results. - `oryxflow.evals`: under pandas 3 the `_preview` columns were never written, because text columns get a `str` dtype instead of `object`. - `oryxflow.evals`: the per-case frame was discarding most of what the report carried. `_to_frame()` read only assertions, scores, metadata and the output, so **six** things went on the floor. Now columns: `case.metrics` (what `increment_eval_metric` recorded -- token counts, retries, anything a task measures about its own call), `case.attributes` (`set_eval_attribute` -- a resolved model snapshot, say), `case.expected_output` (flattened to `expected` / `expected_*`), `case.labels`, `case.total_duration`, and `report.report_evaluator_failures`. The `expected_*` recovery is the one that changes how code reads: a confusion matrix or a per-label breakdown is now a crosstab of two frame columns instead of joining the answer key back on by case name. - `oryxflow.evals`: `EvalResult.verdict()` reports **measured usage** when the task recorded it -- `measured usage: input tokens 9,612, output tokens 2,720` -- read from the conventional `input_tokens` / `output_tokens` / `total_tokens` / `cost` columns. Before this, cost could only ever be the pre-run estimate, because the real numbers pydantic-evals collected were being thrown away. Reported by name rather than inferred from any numeric column: guessing which column is a token count eventually sums something that is not one. - `oryxflow.evals`: an evaluator that **raised** is no longer indistinguishable from one that declined to score. `ReportCase.evaluator_failures` was read by nothing, so a judge crashing on every case (a missing API key, say) left its column absent and the metric reported `n/a` with no hint that anything was wrong. The frame now carries `evaluator_errors` / `evaluator_error` and the verdict prints them above the rate. `case.labels` was likewise dropped and is now a column like assertions and scores. - `oryxflow.evals`: a sync `case()` (or a sync `sweep()` target) no longer silently serializes the run. `concurrency` is scheduled as coroutines by pydantic-evals, so a blocking case holds the event loop and the cases run one at a time while the setting still reads as honored -- same numbers, \~5x the wall clock (8 half-second cases at `concurrency=4`: 1.0s async, 4.9s sync). It now emits a `RuntimeWarning` naming the task and the ignored concurrency. Nothing about oryxflow itself is async; the requirement comes from the eval runner, and `concurrency=1` is the quiet way to opt out. - `oryxflow.evals`: the dead-metric flag no longer fires on a **guardrail sitting inside its budget in every arm**. `0%` false-positives on both sides is the outcome a guardrail exists to confirm, and reporting it as "this usually measures the harness" teaches the reader to ignore the one line that matters when it is real. A guardrail identically *over* budget is still flagged. - `oryxflow.evals`: declining the cost confirmation on the `ev.cli` path exits 0 with a one-line message instead of raising through to a traceback. Choosing not to spend is the outcome the confirmation exists to make easy. - `oryxflow.evals`: a `Metric` no longer counts an **unmeasured** row as a failed one. A row whose metric column is null (an LLM judge that grades only some cases, or one switched off with `--judge-model none`) is now excluded from the rate exactly as an errored row is, instead of being coerced to `False`. Previously that both deflated the numerator and padded the denominator: a prompt passing 2 of 2 judged rows out of 6 reported `33%`, and a metric with no verdicts at all reported a confident `0%` with a confidence interval. `Metric.compute()` now counts only measured rows, a wholly unmeasured metric returns `NaN` and renders `n/a`, and `Metric.unmeasured()` / the verdict state how many rows carried no verdict. New `Metric.eligible()` is the denominator before that filter. - `oryxflow.evals`: `sweep()` no longer silently overwrites a case metadata column named `arm` (or one named after a sweep axis) with its own cell label. `arm` is the natural word for a dataset that compares, say, an `outline` arm with a `guidelines` arm, and the tag destroyed it: any `slices=('arm',)` then reported the arm label back with no error and a plausible-looking table. The clash is knowable from the cases alone, so it now raises **before any call is made**, naming the key and the fix. `arm` joins `RESERVED_METADATA`.