Vieval
By Moeru AI

Evaluate agents and models, with confidence.

Agents behave differently from run to run. Vieval runs your cases across models, parameters, and languages, so you can see what works, what breaks, and why, before you ship the next change.

vieval
$ pnpm vieval run --report-out .vieval/reports/candidateRUN  vieval✓ |desktop-agent| (412 cases)  cases 407 passed | 5 failed | 0 timeout$ pnpm vieval report cases .vieval/reports/candidate --where state=failed  ✗ chrome/open-settings  pass 0  wrong tab  ✗ gimp/resize-image  timeout  60m  ✗ vscode/rename-symbol  pass 0  old name kept  ✗ calc/sum-column  rubric 0.42  partial  ✗ chat/refund-policy  rubric 0.31  hallucinated
How it works

Write once. Evaluate everywhere.

One green run proves little. Run the same eval again, across every combination you care about, and keep a record of why each case passed or failed.

01 · Write

Write evals like tests

Evals are TypeScript files in your repo. Give casesFromInputs a list of inputs, and each item becomes a case that runs the same code.

evals/desktop.eval.ts
import { defineScore, describeTask, expect } from 'vieval'const inputs = await loadEpisodes('s3://evals')const solved = defineScore('desktop.solved', { primary: true })describeTask('desktop', ({ casesFromInputs }) => {  casesFromInputs('episode', inputs, async (ctx) => {    const { task, goal } = ctx.matrix.inputs    const run = await agent.play(task)    expect(run.state).toEqual(goal)    ctx.recordScore(solved, await judge(run))  })})
02 · Expand

Cover every combination

List the models and scenarios to compare, and Vieval runs every combination. Add rubrics, and each run is scored once per rubric. You know the total before anything starts.

vieval.config.ts
defineConfig({  projects: [{    runMatrix: { extend: {      model: ['gpt', 'claude', 'qwen', 'llama'],      scenario: ['cold', 'warm', 'noisy'],    } },    evalMatrix: { extend: {      rubric: ['strict', 'lenient'],    } },  }],})
03 · Observe

Find out why it failed

Every run, retry, and model call is recorded. Watch it live in the dashboard, or export it to OpenTelemetry.

gimp/resizescore 0.91
2 attempts · 1 retry
transcriptscreenshot
04 · Compare

Compare scores side by side

Reports are plain files. Pick two runs and compare scores by category or metric to see what improved and what regressed.

Used by

Capabilities

Everything you need to evaluate

Any data source

Load inputs from Hugging Face, S3, or local files, or generate them in code. Change the data without touching the eval, and keep stable case IDs to compare runs over time.

Learn more →
Open source

Open source, built together

Vieval is MIT licensed. Thanks to the projects and organizations we build on, the universities that evaluate with us, and every contributor.

Star on GitHub