Write evals like tests
Evals are TypeScript files in your repo. Give casesFromInputs a list of inputs, and each item becomes a case that runs the same code.
import { defineScore, describeTask, expect } from 'vieval'const inputs = await loadEpisodes('s3://evals')const solved = defineScore('desktop.solved', { primary: true })describeTask('desktop', ({ casesFromInputs }) => { casesFromInputs('episode', inputs, async (ctx) => { const { task, goal } = ctx.matrix.inputs const run = await agent.play(task) expect(run.state).toEqual(goal) ctx.recordScore(solved, await judge(run)) })})










