LLM eval quickstart¶

You changed a prompt. Is it better?

This page is the whole answer in one file: two versions of a prompt, run over a set of cases, scored by a judge, and reported with a confidence interval so you know whether the gap you're looking at is real. Nothing is hidden in a base class you have to go read — everything that decides the number is on this page.

pip install oryxeval

That pulls pydantic-evals 2.x, the case runner and scorers this is built on.

oryxeval is its own package (Python 3.10+), so if you never run evals it costs you nothing. Import it on purpose (migrating an existing eval? See the note):

import oryxeval as ev

The whole thing¶

Two files. First the cases — one row per case, in a spreadsheet, because the person with the real examples usually doesn't edit Python:

name,question
capital_fr,What is the capital of France?
capital_jp,What is the capital of Japan?
largest_au,What is the largest city in Australia?
oldest_eu,Which European capital is the oldest?

Then the eval:

import oryxflow
import oryxeval as ev
from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_evals.evaluators import LLMJudge

MODEL = ...   # the model under test — anything pydantic-ai accepts
JUDGE = ...   # the judge — a DIFFERENT model family from MODEL


class Inputs(BaseModel):
    question: str


async def answer(inputs, style):
    instruction = 'Answer in one word.' if style == 'terse' else 'Answer helpfully.'
    reply = await Agent(MODEL).run('{}\n\n{}'.format(instruction, inputs.question))
    return {'reply': reply.output}


if __name__ == '__main__':
    ev.sweep(answer,
             cases=ev.load_cases('cases.csv', inputs=Inputs),
             evaluators=(LLMJudge(rubric='The reply names one specific city.',
                                  model=JUDGE, include_input=True),),
             metric=ev.Metric('names a city', 'LLMJudge'),
             style=['terse', 'helpful']).verdict()

That's it. No class of your own, no output directory to configure, no --reset to remember.

What each piece is¶

Two rules about answer that are easy to get wrong and expensive to discover later:

• Write it async. Cases are scheduled as coroutines, so a plain def returns the same numbers but blocks the loop — the concurrency you asked for silently does nothing. Eight half-second cases at concurrency=8: 1.0s async, 4.9s sync. You get a warning, but the numbers won't tell you.
• Let exceptions raise. Catching one and returning it in a field makes the runner record a successful case: the failure list comes back empty, the failure rate reads zero, and the metric scores an error string as if it were an answer. Raising is what puts the case in the failure list and keeps it out of every rate.

Run it¶

python eval.py

Before anything is billed you get the projected bill, and are asked:

answer · 4 cases × 2 arms × 1 rep = 8 calls
  cached 0 · new 8   (+1 preflight call per new arm)
  estimated cost: not estimated -- pass cost_per_call= for a crude calls x cost_per_call figure

Run 8 new calls for answer? (y/n/c=check one call first)

c spends one call on the first uncached arm — enough to prove the credentials and the wiring work — prints the result, and asks again. Then:

NAMES A CITY (higher is better)
  terse      100%  (4/4)    95% CI [51%, 100%]
  helpful     75%  (3/4)    95% CI [30%, 95%]
  Δ  terse − helpful  = +25pp  [-25pp, +75pp]   inside noise

VERDICT  terse vs helpful = +25pp [-25pp, +75pp], inside noise at 1 rep -- raise repeats
         before calling this real.

Read the last line first. Four cases and one repeat cannot separate these two arms, and the verdict says so rather than handing you a winner. That is the point of the whole exercise: real harnesses have measured the same prompt at 69% and then 64% an hour later. The levers for a narrower band are more cases and repeats=.

Run it again — you pay nothing¶

answer · 4 cases × 2 arms × 1 rep = 8 calls
  cached 8 · new 0

Each arm is one cached cell on disk. Add a third arm and you pay for the third arm. Edit answer and its arms re-run; everything you have already evaluated stays put. Nothing to remember, and no stale cell reported as a fresh number.

The model's outputs and their scores are cached separately. Rewrite the judge's rubric and only the scoring re-runs, over the outputs you already paid for:

answer · 4 cases × 2 arms × 1 rep = 8 calls
  cached 8 · new 0
  re-scoring 2 arms from stored outputs: no model calls (but 1 LLM judge evaluator(s) still call their model per case)

Run from the directory the script lives in. The cache (data/) is created relative to where you launch, and running from elsewhere quietly starts a second, empty one; the command line form warns when that happens.

What to add next, in this order¶

The quickstart above is honest but thin. Four additions, each worth more than the last:

1. More cases. Fifteen to twenty-five, harvested from real material rather than invented, and worked across the axes that matter (hard/easy, long/short, in-scope/out-of-scope). This moves the interval more than anything else on the list.

2. A guardrail. The thing a prompt can win your headline metric by breaking. Without one, a prompt that answers everything scores 100% on "did it answer" and ships a regression:

   guardrail=ev.Metric('makes something up', 'Hallucinated',
                       higher_is_better=False, budget=0.05)
   

   A winner that breaks it is reported as not a clean win.

3. A coverage companion. A rate over a filtered subset improves as the subset shrinks, so a prompt can score well on quality by staying silent on a third of the cases. coverage= is rendered first, above the quality number:

   metric=ev.Metric('quality', 'Good', where='answered',
                    coverage=ev.Metric('answered', 'answered'))
   

4. Check the judge. A judge nobody checked is an opinion with a confidence interval printed around it. Hand-label a sample, put those labels in a column, and name it — the verdict then reports how well the judge agrees with you, and a corrected rate:

   metric=ev.Metric('names a city', 'LLMJudge', human='human_label')
   

Then slices=('kind', 'length') breaks the best arm down by any metadata column, worst first — which is where your next batch of cases comes from.

Two more once there is a version you are comparing against:

• Name the baseline. baseline='helpful' (or an arm literally named baseline) reports every other arm as a difference from it, with its interval. No winner is named: one arm up on one label and down on another is a judgement for you to write down, not a pass/fail.
• Read the outputs. r.side_by_side() writes every arm's output for each case where the arms disagree to results/<date>-<name>-side-by-side.md — no model calls, just the stored outputs. A rate tells you how often; only the outputs tell you whether the scorer rewards the right thing.

Three shapes, in order of how much you need¶

They are the same machinery at three levels of ceremony — sweep builds a TaskEval for you, and the plugin writes one. Nothing you learn here stops applying at the next level up.

Read next¶

• LLM evals — the full guide: case files, arms and baselines, what caching is keyed on, reading the verdict, and the rules about your own code that matter most.
• Classes and functions — everything oryxeval exports, in the order you'll meet it.
• What an eval plan must contain — the six sections a plan needs before anyone can run it, for when you are building an eval from someone else's.
• Scaffold an LLM eval — the plugin's four eval commands.