# LLM eval quickstart You changed a prompt. Is it better? This page is the whole answer in one file: two versions of a prompt, run over a set of cases, scored by a judge, and reported with a confidence interval so you know whether the gap you're looking at is real. Nothing is hidden in a base class you have to go read — everything that decides the number is on this page. ``` pip install oryxeval ``` That pulls pydantic-evals 2.x, the case runner and scorers this is built on. `oryxeval` is its own package (Python 3.10+), so if you never run evals it costs you nothing. Import it on purpose (migrating an existing eval? See [the note]()): ``` import oryxeval as ev ``` ## The whole thing Two files. First the cases — one row per case, in a spreadsheet, because the person with the real examples usually doesn't edit Python: cases.csv ``` name,question capital_fr,What is the capital of France? capital_jp,What is the capital of Japan? largest_au,What is the largest city in Australia? oldest_eu,Which European capital is the oldest? ``` Then the eval: eval.py ``` import oryxflow import oryxeval as ev from pydantic import BaseModel from pydantic_ai import Agent from pydantic_evals.evaluators import LLMJudge MODEL = ... # the model under test — anything pydantic-ai accepts JUDGE = ... # the judge — a DIFFERENT model family from MODEL class Inputs(BaseModel): question: str async def answer(inputs, style): instruction = 'Answer in one word.' if style == 'terse' else 'Answer helpfully.' reply = await Agent(MODEL).run('{}\n\n{}'.format(instruction, inputs.question)) return {'reply': reply.output} if __name__ == '__main__': ev.sweep(answer, cases=ev.load_cases('cases.csv', inputs=Inputs), evaluators=(LLMJudge(rubric='The reply names one specific city.', model=JUDGE, include_input=True),), metric=ev.Metric('names a city', 'LLMJudge'), style=['terse', 'helpful']).verdict() ``` That's it. No class of your own, no output directory to configure, no `--reset` to remember. ## What each piece is | In the file | What it does | | --- | --- | | `class Inputs` | One field per input column in `cases.csv`. `load_cases` routes columns by **name**, so a misspelled column lands in metadata instead and pydantic raises naming the field it never got. That check is what makes keeping cases in a spreadsheet safe. | | `async def answer(inputs, style)` | **The thing under test.** It takes one case's inputs plus whatever you're varying, and returns a dict (or a pydantic model). Write it `async` — see below. | | `style=['terse', 'helpful']` | Any keyword holding a **list** becomes an axis. Two values, two arms. Add `model_id=['small', 'large']` and you have four. A keyword holding a single value is just a setting applied to every arm. | | `evaluators=(LLMJudge(...),)` | How a case is scored. `LLMJudge` ships with pydantic-evals and answers one yes/no question about each output. | | `ev.Metric('names a city', 'LLMJudge')` | The number the arms are compared on. The second argument is the **column** the score lands in — for an evaluator, its class name. | | `.verdict()` | Prints the comparison, with intervals. | Two rules about `answer` that are easy to get wrong and expensive to discover later: - **Write it `async`.** Cases are scheduled as coroutines, so a plain `def` returns the same numbers but blocks the loop — the concurrency you asked for silently does nothing. Eight half-second cases at `concurrency=8`: 1.0s async, 4.9s sync. You get a warning, but the numbers won't tell you. - **Let exceptions raise.** Catching one and returning it in a field makes the runner record a *successful* case: the failure list comes back empty, the failure rate reads zero, and the metric scores an error string as if it were an answer. Raising is what puts the case in the failure list and keeps it out of every rate. ## Run it ``` python eval.py ``` Before anything is billed you get the projected bill, and are asked: ``` answer · 4 cases × 2 arms × 1 rep = 8 calls cached 0 · new 8 (+1 preflight call per new arm) estimated cost: not estimated -- pass cost_per_call= for a crude calls x cost_per_call figure Run 8 new calls for answer? (y/n/c=check one call first) ``` `c` spends **one call** on the first uncached arm — enough to prove the credentials and the wiring work — prints the result, and asks again. Then: ``` NAMES A CITY (higher is better) terse 100% (4/4) 95% CI [51%, 100%] helpful 75% (3/4) 95% CI [30%, 95%] Δ terse − helpful = +25pp [-25pp, +75pp] inside noise VERDICT terse vs helpful = +25pp [-25pp, +75pp], inside noise at 1 rep -- raise repeats before calling this real. ``` **Read the last line first.** Four cases and one repeat cannot separate these two arms, and the verdict says so rather than handing you a winner. That is the point of the whole exercise: real harnesses have measured the *same* prompt at 69% and then 64% an hour later. The levers for a narrower band are more cases and `repeats=`. ## Run it again — you pay nothing ``` answer · 4 cases × 2 arms × 1 rep = 8 calls cached 8 · new 0 ``` Each arm is one cached cell on disk. Add a third arm and you pay for the third arm. Edit `answer` and its arms re-run; everything you have already evaluated stays put. Nothing to remember, and no stale cell reported as a fresh number. The model's outputs and their scores are cached **separately**. Rewrite the judge's rubric and only the scoring re-runs, over the outputs you already paid for: ``` answer · 4 cases × 2 arms × 1 rep = 8 calls cached 8 · new 0 re-scoring 2 arms from stored outputs: no model calls (but 1 LLM judge evaluator(s) still call their model per case) ``` Run from the directory the script lives in. The cache (`data/`) is created relative to where you launch, and running from elsewhere quietly starts a second, empty one; the command line form warns when that happens. ## What to add next, in this order The quickstart above is honest but thin. Four additions, each worth more than the last: 1. **More cases.** Fifteen to twenty-five, harvested from real material rather than invented, and worked across the axes that matter (hard/easy, long/short, in-scope/out-of-scope). This moves the interval more than anything else on the list. 2. **A guardrail.** The thing a prompt can win your headline metric by breaking. Without one, a prompt that answers *everything* scores 100% on "did it answer" and ships a regression: ``` guardrail=ev.Metric('makes something up', 'Hallucinated', higher_is_better=False, budget=0.05) ``` A winner that breaks it is reported as **not a clean win**. 3. **A coverage companion.** A rate over a filtered subset improves as the subset shrinks, so a prompt can score well on quality by staying silent on a third of the cases. `coverage=` is rendered first, above the quality number: ``` metric=ev.Metric('quality', 'Good', where='answered', coverage=ev.Metric('answered', 'answered')) ``` 4. **Check the judge.** A judge nobody checked is an opinion with a confidence interval printed around it. Hand-label a sample, put those labels in a column, and name it — the verdict then reports how well the judge agrees with you, and a corrected rate: ``` metric=ev.Metric('names a city', 'LLMJudge', human='human_label') ``` Then `slices=('kind', 'length')` breaks the best arm down by any metadata column, worst first — which is where your next batch of cases comes from. Two more once there is a version you are comparing *against*: - **Name the baseline.** `baseline='helpful'` (or an arm literally named `baseline`) reports every other arm as a difference from it, with its interval. No winner is named: one arm up on one label and down on another is a judgement for you to write down, not a pass/fail. - **Read the outputs.** `r.side_by_side()` writes every arm's output for each case where the arms disagree to `results/--side-by-side.md` — no model calls, just the stored outputs. A rate tells you how often; only the outputs tell you whether the scorer rewards the right thing. ## Three shapes, in order of how much you need | Reach for | When | | --- | --- | | **`ev.sweep(fn, ...)`** — this page | You want the answer today. A function, a grid, a metric. | | **`ev.TaskEval` + `ev.cli`** | You want the eval to **stay right**: caching keyed on the prompt files it actually reads, a baseline arm checked out of git, and a command line whose flags are derived from your arms. See [the guide](). | | **The Claude Code plugin** | You want it **built for you**: four commands that settle what you're measuring before any code exists, scaffold it, grow a real case set, and run it. See [Scaffold an LLM eval](). | They are the same machinery at three levels of ceremony — `sweep` builds a `TaskEval` for you, and the plugin writes one. Nothing you learn here stops applying at the next level up. ## Read next - **[LLM evals]()** — the full guide: case files, arms and baselines, what caching is keyed on, reading the verdict, and the rules about your own code that matter most. - **[Classes and functions]()** — everything `oryxeval` exports, in the order you'll meet it. - **[What an eval plan must contain]()** — the six sections a plan needs before anyone can run it, for when you are building an eval from someone else's. - **[Scaffold an LLM eval]()** — the plugin's four eval commands.