Skip to content
Vieval

Core Concepts ​

Vieval organizes evaluation work into a small set of nested concepts. The following lifecycle is the map for the rest of the guide:

text
comparison -> workspace -> project

project
  -> task <-> run matrix + eval matrix
    -> case -> assertion / score / metric evidence

task -> attempt -> retry

run --json -> machine-readable stdout
run -> report artifacts (.vieval/reports by default) -> analyze / compare

A workspace is a config that lists projects, inline or collected from other configs such as packages/*. A comparison is a config that lists workspaces and runs each one in its own process, side by side. A project defines a discovery boundary and project-level configuration. A discovered eval definition may carry a task, while DSL definitions such as describeTask register tasks directly. Matrix rows expand the registered tasks into concrete scheduled work. A task contains one or more cases, where assertions, named scores, and custom metrics provide execution evidence. One score per task is its primary score; reports show it as the headline, next to the completion rate, the share of cases that produced a result.

Execution reliability has two scopes. An attempt reruns a full task and retains the evidence from every attempt. A retry lets a failed case run again within an attempt. Output is explicit: --json prints a machine-readable result to standard output, and every run writes the report artifacts consumed by analysis and comparison commands, by default to .vieval/reports next to the config. --report-out <directory> changes that root and --no-report turns reports off.

ConceptResponsibilityTypical scope
ComparisonLists workspaces, runs each in its own process, and prints a leaderboardSeveral implementations of one benchmark
WorkspaceLists projects, schedules them in one process, and provides their envOne repository or monorepo root
ProjectDefines discovery, configuration, executors, and matrix defaultsOne product area or evaluation suite
TaskDescribes evaluation behavior and connects run/eval matrix selectionsOne behavior evaluated across variants
CaseExecutes one input or scenario and emits assertion, score, and metric evidenceOne check within a task
Run matrixExpands execution-side choices such as models or scenariosProject, eval, or task definition
Eval matrixExpands evaluation-side choices such as rubricsProject, eval, or task definition
AttemptRepeats the complete task to measure reliabilityOne scheduled task
RetryRe-executes a failed case inside an attemptOne case execution
Report artifactsPreserve run summaries, events, and case evidence; written unless --no-report is usedOne project run and its report directory
Analysis and comparisonRead artifacts to inspect results or compare runs and methodsOne or more report artifact sets

Next, learn how to author Tasks, Cases, and Inputs.