Reports and Comparisons
The default CLI summary answers what happened in the active terminal. Report artifacts retain the evidence needed to inspect cases, analyze several runs, and compare a baseline with a candidate later.
Move from terminal output to retained artifacts
vieval run prints a human-readable summary and writes report artifacts by default, under .vieval/reports next to the config file. --report-out <path> changes the report root, and --no-report writes nothing (case artifacts then go to a temporary directory that is removed when the run ends). --json changes stdout to the machine-readable run output; it does not change which files are written.
vieval run \
--config ./vieval.config.ts \
--experiment baseline \
--attempt attempt-a \
--report-out .vieval/reportsThe command still prints its human-readable summary of the active run and its project, task, and case state. Add --json when a script needs the machine-readable run output on stdout: it has the invocationId and a projects list, and each project has its own runId and reportDirectory. With --no-report, each reportDirectory is null and the vieval report ... commands have no new artifacts from that run to read.
Know what a run writes
Every project is its own run. Vieval places each run under the report root as <workspace>/<project>/<experiment>/<attempt>/<run>/. The workspace segment is the name of the workspace config's directory (default-workspace is used only without a config file). The runs of one vieval run invocation share experimentId, attemptId, and invocationId; runs started by a comparison also carry comparisonId.
| Artifact | Confirmed contents |
|---|---|
run-summary.json | The run output of exactly one project, with schemaVersion: 2, runId, reportDirectory, identities, project status, case summaries and failures, matrix summaries, the score summaries of every task run, the scoreDefinitions the tasks announced, and the resolved headline (see below). vieval report commands reject other versions. |
events.jsonl | Ordered run, task, case, score (score.defined, score.recorded), metric, and custom event envelopes captured during execution. |
cases.jsonl | One normalized final record per observed task/case (schemaVersion: 2), with identities, state, timing, a derived retry count, scores, metrics, optional input/output, and, for tasks with automatic attempts, attemptIndex, autoAttempt, and every attempt in attempts. scores maps score names to the counted value, unscored maps score names to the reason the grader gave none, and scoreDetails keeps the rationale, grader, and metadata of each record. |
metrics-summary.json | Overall score counts, sums, and averages per score name, derived from normalized case records. |
otlp/traces.json, otlp/logs.json, otlp/metrics.json | Local OTLP-shaped projections derived from the case records. Score gauges are named vieval.score.<name>. Each resource and the vieval.run span carry vieval.workspace.id, vieval.experiment.id, vieval.attempt.id, and vieval.run.id. |
vieval report index creates another artifact, index/runs.jsonl by default. It is not written by vieval run.
The same workspaceId, experimentId, attemptId, and runId appear in run-summary.json, every event, every case record, and the OTLP attributes above, so one run can be traced across all files.
Automatic attempts require one reading caveat. run-summary.json contains the task score aggregated from all completed attempt evidence, while cases.jsonl keeps a final projection for each task/case: its top-level fields describe the last attempt, and attempts lists the state, scores, retries, and timing of each attempt. In events.jsonl, case.started and case.ended of a task with automatic attempts carry attemptIndex (0 for the first attempt) and autoAttempt.
The retryCount field counts retries within the final attempt. For tasks with automatic attempts it uses attemptIndex, so additional attempts are not counted as retries. Reports written before attemptIndex existed have no such field; for them, repeated starts from additional attempts may still increase retryCount.
Event lines have schemaVersion: 3 and case records have schemaVersion: 2. Version 3 of the events replaced the fixed exact and judge score kinds with named scores. New optional fields and new event names do not change the version, and readers ignore fields they do not know. A renamed, removed, or retyped field increments the version. Readers reject reports written with an older version ("This report was written by an older vieval. Run it again."), and the viewer lists such a run as unsupported; there is no migration command.
Treat report artifacts as potentially sensitive
Artifacts may contain case inputs and outputs, error messages, custom metrics, model or benchmark identifiers, and other event payloads. Store them under an appropriate retention policy, restrict access, and remove or redact fields that should not leave the evaluation environment.
Read the headline
The headline of each project and of the run is resolved once, when the run ends, and written into run-summary.json as { headline, completion }. The CLI, the viewer, and vieval report read it instead of deriving it again:
headline.kind | When | Shown as |
|---|---|---|
score | Every task that has scores shares one primary score name. | The pooled primary score with its label, for example Pass 87.5%. |
per-task | The tasks have different primary scores. | No run-level score. Each task's primary score is listed with its task name. |
passed-cases | No task has a primary score. | The share of passed cases, labeled "Passed cases". |
completion is the completion rate, (passed + failed) / (passed + failed + errored + timeout), and it is always shown next to the headline. The run summary line prints both, for example:
Pass 87.5% · completed 96.8% (4 errored)Every score summary has name, mean (or null when no case counted), n (cases that counted), total (settled cases that were not skipped), sum, and unscored (counts per reason). Summaries of a project or a run pool the cases of their tasks; they are not a mean of task means.
Index, inspect, and analyze reports
Every report command requires a report path. It can usually be a single run directory or a higher report root that contains multiple runs.
vieval report index <report-directory>index discovers run-summary.json files recursively and writes compact run rows to <report-directory>/index/runs.jsonl unless --output changes the path. --format table|json|jsonl controls stdout, not the index file format.
vieval report cases <report-directory>cases reads cases.jsonl. Repeat --where key=value for equality filters, add --group-by <key> for grouped score summaries, and choose --format table|json|jsonl. JSON and JSONL are stdout formats; this command does not write a new case artifact.
vieval report analyze <report-directory>analyze reads run summaries and events, filters runs, and rolls them up by workspace and experiment. Its filters include workspace, project, experiment, attempt, run, event or error text, and run/eval matrix selectors. --task-state and --case-state currently accept passed, failed, or skipped; timeout is not currently accepted by --case-state. --attempt-index <n> keeps runs where at least one case ran automatic attempt n. Each run row and experiment group in JSON, JSONL, and CSV output includes autoAttemptedCases (cases that ran in more than one attempt; an additional attempt reruns every case of the task) and maxAttemptIndex (null without automatic attempts). --format table|json|jsonl|csv controls stdout.
vieval report attempts --config vieval.config.tsattempts lists the attempts in a report root, newest first: per experiment and attempt, the workspaces, the passed, failed, and no-result case counts, and the completion rate. "←" marks the attempt that vieval run --resume continues without an id; notes show which attempt an attempt resumed, interrupts, and runs killed before they wrote a summary. Pass a report root, or --config to list the reports next to it; filter with --experiment and --workspace, and choose --format table|json.
Share reports
vieval report export .vieval/reports --output report-siteDeploy report-site to a static host, then share the URL.
Compare two existing report sets
The report comparison command requires a left baseline and a right candidate:
vieval report compare <left-report-directory> <right-report-directory>It reads normalized cases, aligns them by a case key, and reports matched deltas plus added and removed cases. Per-case and grouped deltas use matched records and are right - left; grouping uses the right-side record's selector value. A missing selected score on a matched case is treated as 0. The overall delta instead subtracts the two full record-set means, ignoring records that lack the selected score, so additions and removals affect it and it is not the mean matched delta. --score <name> selects the score and defaults to the primary score in run-summary.json. Output can be table or json.
The default alignment key is the benchmark.case.id metric when present, then vieval.case.id, then <taskId>::<caseName>. Use --case-key <key> when the benchmark has another stable identifier. An explicit key must exist on every record. Duplicate resolved keys on either side are an error rather than an arbitrary match.
Selectors used by --case-key, --group-by, and report cases look for an exact metric name first, then scores.<name> or a bare score name, then a direct case-record field. For example, benchmark.category can refer to a metric, while state refers to the normalized record field.
vieval report compare <left> <right> compares case artifacts that already exist. To run several implementations of one benchmark side by side, use a comparison config instead.
Run a comparison
A config that lists workspaces is a comparison. Each entry is a path or glob of directories that hold a vieval.config.*, '!<glob>' to exclude directories, or { root, id?, env? }. A workspace id defaults to its directory name, and duplicate ids are an error. A comparison has no models, plugins, or reporters: each workspace runs with its own config.
import { defineConfig, loadEnv } from 'vieval'
export default defineConfig({
cache: { namespace: 'locomo-cases-v1' },
env: loadEnv('test', import.meta.dirname, ''),
id: 'agent-memory-semantic-approaches',
workspaces: ['eval/test-objects/*'],
})vieval run --config evals/agent-memory/vieval.config.ts- Every workspace runs in its own process, with the
vievalthat the workspace resolves. It must be the same version, so the workspace's dependencies must be installed. - All workspaces start at once.
concurrency: { workspace }or--workspace-concurrencylimits them. - The comparison
env(and an entry'senv) wins over each workspace config'senv, and is applied before the workspace's plugins run. cache: { namespace }makes all workspaces share.vieval/cache/<comparison id>/<namespace>/next to the comparison config.- One live terminal window shows
Workspaces,Projects,Tasks,Cases,Concurrency, andTimerows for all workspaces. A failing workspace does not stop the others. - At the end, a leaderboard prints one row per workspace and project, ranked by the primary score, with the completion rate in its own column, then the failed workspaces with the last lines of their stderr. If the workspaces have different primary scores, the leaderboard says so and ranks by passed cases.
- The exit code is
1when any workspace failed or had failing cases.--jsonprints the comparison output. - Reports go to
.vieval/reportsnext to the comparison config. The experiment defaults to the comparison id, and all workspaces share one attempt, so every run of the comparison sits under<workspace>/<project>/<comparison id>/<attempt>/. - With
--with-viewer, one viewer serves the comparison's report root.
Selectors narrow a run. They are repeatable, and a value that contains * is a glob: --workspace <id> (comparison only; an error on a workspace config), --project <name>, --task <name> (the describeTask name or entry id), and --case <name> (the case name). A selector that matches nothing is an error that lists what exists; in a comparison, --project, --task, and --case must match in at least one workspace.
vieval run --config evals/agent-memory/vieval.config.ts --workspace mem9 --with-viewer --open
vieval run --task zh --case 'case #1*'Run one complete comparison flow
The following flow retains two executions, indexes the combined root, inspects candidate cases, analyzes all runs, and finally compares case scores. It does not depend on a fabricated run ID because report commands discover runs recursively.
vieval run \
--config ./vieval.config.ts \
--experiment baseline \
--attempt attempt-a \
--report-out .vieval/reports/baselinevieval run \
--config ./vieval.config.ts \
--experiment candidate \
--attempt attempt-a \
--report-out .vieval/reports/candidatevieval report index .vieval/reportsvieval report cases .vieval/reports/candidate \
--where state=failed \
--format jsonlvieval report analyze .vieval/reports --format jsonvieval report compare \
.vieval/reports/baseline \
.vieval/reports/candidate \
--case-key benchmark.case.id \
--score retrieval.recall \
--format tableThe final command requires every case to emit a unique benchmark.case.id. Omit --case-key if that guarantee does not exist and the built-in fallback identities are appropriate.
A common failure is pointing report commands at the wrong root: with --report-out, the reports are under that path, not under .vieval/reports, and a run with --no-report leaves nothing to read. Another is aligning on generated or duplicate identifiers; comparison is meaningful only when the chosen key denotes the same benchmark case on both sides.
See API for public entrypoints and Config for the current configuration surface.