Configuration
Create viteval.config.ts with Viteval's own settings. No worker setup file,
manual initialization, or Vitest configuration is required.
Project settings
Project root, relative to the config's directory. Defaults to the config
directory, or the CLI root without a config. An explicit --root
overrides this setting for both file discovery and worker persistence.
Available storage integrations. Omitted registers local JSON; an explicit array replaces that default. Duplicate identities are errors. Use separate configs for separate projects.
Unmodified AI SDK provider objects, not instantiated models. One instance per provider identity. No Viteval SDK wrappers are needed.
Default language model for judges. Tasks still choose their own application model.
Default embedding model for similarity scorers.
Default dataset-write destinations. Defaults to JSON when registered; otherwise no persistence.
TypeScript path aliases are read from the project's tsconfig.json; there is no
public Vite resolve block.
Evaluation defaults
Per-evaluation options override CLI flags; CLI flags override project defaults.
Use --timeout, --retries, and
--concurrency to override project defaults from the CLI.
Evaluation patterns. Defaults to **/*.eval.{ts,mts,js,mjs}.
Additional evaluation exclusions. Dependency and Git directories remain excluded by the runner.
60000Positive per-case timeout in milliseconds, including task execution and scoring.
5Positive case concurrency limit within each worker. Evaluations can lower their own limit; the worker limit remains the ceiling.
0Additional attempts after a failed case. This reruns the task and scorers, not result uploads.
1Finite per-case aggregation threshold.
meanCombines each case's scorer results.
Result-write destinations. Defaults to JSON when registered; otherwise no persistence. Registration alone never enables hosted uploads.
Destination inheritance
An evaluation's results and a dataset definition's providers follow the same
rules: omitted inherits the corresponding config array, an explicit array replaces
it, and [] disables persistence. There are no boolean or map variants.
Dataset sources and result destinations are independent. A dataset can be generated to Langfuse, read from Langfuse, and evaluated with results saved only to JSON.
Model resolution and validation
Model references use provider:model-id, resolved through AI SDK's provider
registry. Viteval determines the provider identity from locally constructed SDK
models; it does not guess import-variable names or contact the service during
configuration. Providers must support constructing a language or embedding model
without a network request.
TypeScript checks the namespaced string shape, not an exhaustive remote model catalog. Raw SDK imports do not register global Viteval model types. The registry checks active provider registration and delegates model-kind/ID validation to the SDK; some SDKs defer unknown-ID errors until the actual API call. No credentials or remote catalog lookup is required just to load config.
Direct SDK model instances remain supported on scorers. Deterministic scorers need no model configuration.
Storage integration imports contribute typed names and dataset handles. Type visibility does not prove an integration is active in the selected config; registration and capabilities are also checked at runtime.
Example
defineConfig is also exported from viteval/config. Config modules may run in
the coordinating process and again in workers. Do not make model calls, generate
datasets, or open resources at module scope. Viteval initializes selected storage
integrations lazily and closes them after evaluation persistence.