Skip to content
Vieval

Config ​

Vieval configuration belongs to the repository or package that owns the evaluations. vieval run starts at the command's working directory and uses the nearest readable vieval.config.ts, .mts, .cts, .js, .mjs, .cjs, or .json, searching upward through parent directories. Pass --config <path> to select a file explicitly; a relative path is resolved from the command's working directory.

The config file's directory becomes the base for relative project roots. Each project's include and exclude patterns are then evaluated within that project root. This keeps discovery ownership explicit when one config describes more than one project.

Configuration domains ​

DomainWhat belongs hereLearn more
Loading and environmentConfig discovery or --config selection, defineConfig, and top-level env. Vieval does not load .env* files automatically; when those values are needed, call loadEnv explicitly from the config file.Models and Inference Executors
Workspaces and projectsA config that lists projects is a workspace. Each entry is an inline project (name, root, discovery patterns, matrices, models, executors, plugins, reporters, and scheduler) or a reference to other configs: a path or glob string such as 'packages/*', or { from, env?, scheduler? }. Collected projects load with their own config. Project names must be unique in a workspace.Core Concepts
ComparisonsA config that lists workspaces is a comparison: defineConfig({ id?, workspaces, cache?, env?, concurrency?: { workspace? } }). vieval run runs each workspace in its own process. A comparison has no models, plugins, or reporters. projects and workspaces cannot be used in one config.Reports and Comparisons
Plugins and modelsTop-level or project-local config plugins, model registrations, provider metadata, aliases, and project model overrides. A registration makes a model available to task code; it does not make an inference request by itself.Models and Inference Executors
Run and eval matricesProject runMatrix and evalMatrix definitions, plus how eval- and task-level layers extend, override, or disable axes. Matrix expansion schedules variants; task code decides what each selected value does.Matrices and Datasets
Execution policy and concurrencyscheduler.concurrency pools for projects, tasks, and cases, scheduler.middleware, and the task/case attempt, retry, and timeout controls. A pool is shared by the work where it is declared: top-level config and the vieval run concurrency flags bound the whole run, and projects, tasks, and case groups can only lower concurrency for their own work. --attempt labels report artifacts and does not configure automatic attempts.Reliable Execution
CacheThe task cache is injected through TaskRunContext, not declared in a workspace config. CLI runs place project cache data under .vieval/cache inside the project root. A comparison's cache: { namespace } makes all of its workspaces share .vieval/cache/<comparison id>/<namespace>/ next to the comparison config.Reliable Execution
Reporters and artifactsTop-level reporters are inherited by projects unless a project supplies its own list. vieval run writes local report artifacts by default to .vieval/reports next to the config; --report-out changes the root and --no-report turns them off. --json changes standard output only.Reports and Comparisons
Telemetryreporting.openTelemetry controls runtime OpenTelemetry spans and an end-of-run lifecycle callback. Local reporter events and the OTLP-shaped files in the report directory do not depend on that switch.Reports and Comparisons

Scope and precedence ​

Configuration only inherits where the owning types and runtime define inheritance. In a workspace, env, scheduler, and reporting apply to the whole workspace. Env is process-wide: the workspace env, the env of each referenced config, and the env of each reference are merged, and a key with two different values is an error. scheduler on a reference replaces the scheduler of the collected projects. In a comparison, the comparison env (and an entry's env) wins over each workspace config's env and is applied before the workspace's plugins run. Top-level scheduler is not inherited: scheduler.concurrency declares pools shared by the whole run, and a project's scheduler.concurrency declares separate task and case pools for that project, so work has to fit both. Top-level scheduler.middleware wraps project scheduler.middleware. Models and reporters behave differently: their top-level arrays feed project defaults, but a project-provided array replaces the complete inherited list. Matrix layers have their own project → eval → task resolution rules. Plugins may transform top-level config, while project-local plugins run against a project-scoped config view. These rules do not imply that every field is available at every scope.

The run command can select a config with --config; select workspaces (comparison only), projects, tasks, and cases with --workspace, --project, --task, and --case; add run-wide concurrency caps that never raise a configured value; label report organization with --experiment and --attempt; resume an earlier attempt (run only its cases without a result) with --resume; and choose output behavior with --json, --report-out, and --no-report. Treat these as command-specific runtime inputs rather than a generic override mechanism for arbitrary config fields.

Each documented field records Type → Default → Scope → CLI override → Behavior → Example → Interactions. When a step does not apply to that field, the reference states that explicitly instead of inferring a value or an override.

For the exported config types and helpers, see the API map.