01 · 编写
像写测试一样写评测
评测就是放在项目里的 TypeScript 文件。准备好一组输入,casesFromInputs 会把每一条变成一个用例,用同一段代码跑完。
evals/desktop.eval.ts
import { defineScore, describeTask, expect } from 'vieval'const inputs = await loadEpisodes('s3://evals')const solved = defineScore('desktop.solved', { primary: true })describeTask('desktop', ({ casesFromInputs }) => { casesFromInputs('episode', inputs, async (ctx) => { const { task, goal } = ctx.matrix.inputs const run = await agent.play(task) expect(run.state).toEqual(goal) ctx.recordScore(solved, await judge(run)) })})inputs → cases
chrome/open-tab✓Open a new tab
gimp/resize✓Resize to 800px
vscode/rename✗Rename a symbol
calc/sum-cells✓Sum column B
+ 408 more










