For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt, and this page is available as Markdown at https://viteval.dev/api/configuration.md.
V1 release candidate available.Migration guide →
GitHub

Configuration

Create viteval.config.ts with Viteval's own settings. No worker setup file, manual initialization, or Vitest configuration is required.

Project settings

Project
rootstringoptional

Project root, relative to the config's directory. Defaults to the config directory, or the CLI root without a config. An explicit --root overrides this setting for both file discovery and worker persistence.

providersProvider[]optional

Available storage integrations. Omitted registers local JSON; an explicit array replaces that default. Duplicate identities are errors. Use separate configs for separate projects.

models.providersModelProvider[]optional

Unmodified AI SDK provider objects, not instantiated models. One instance per provider identity. No Viteval SDK wrappers are needed.

models.defaults.languageprovider:model-idoptional

Default language model for judges. Tasks still choose their own application model.

models.defaults.embeddingprovider:model-idoptional

Default embedding model for similarity scorers.

dataset.providersDatasetProviderName[]optional

Default dataset-write destinations. Defaults to JSON when registered; otherwise no persistence.

TypeScript path aliases are read from the project's tsconfig.json; there is no public Vite resolve block.

Evaluation defaults

Per-evaluation options override CLI flags; CLI flags override project defaults. Use --timeout, --retries, and

--concurrency to override project defaults from the CLI.

eval
includestring[]optional

Evaluation patterns. Defaults to **/*.eval.{ts,mts,js,mjs}.

excludestring[]optional

Additional evaluation exclusions. Dependency and Git directories remain excluded by the runner.

timeoutnumberoptional
Default: 60000

Positive per-case timeout in milliseconds, including task execution and scoring.

concurrencynumberoptional
Default: 5

Positive case concurrency limit within each worker. Evaluations can lower their own limit; the worker limit remains the ceiling.

retriesnumberoptional
Default: 0

Additional attempts after a failed case. This reruns the task and scorers, not result uploads.

thresholdnumberoptional
Default: 1

Finite per-case aggregation threshold.

aggregation'mean' | 'median' | 'sum'optional
Default: mean

Combines each case's scorer results.

resultsResultProviderName[]optional

Result-write destinations. Defaults to JSON when registered; otherwise no persistence. Registration alone never enables hosted uploads.

Destination inheritance

An evaluation's results and a dataset definition's providers follow the same rules: omitted inherits the corresponding config array, an explicit array replaces it, and [] disables persistence. There are no boolean or map variants.

Dataset sources and result destinations are independent. A dataset can be generated to Langfuse, read from Langfuse, and evaluated with results saved only to JSON.

Model resolution and validation

Model references use provider:model-id, resolved through AI SDK's provider registry. Viteval determines the provider identity from locally constructed SDK models; it does not guess import-variable names or contact the service during configuration. Providers must support constructing a language or embedding model without a network request.

TypeScript checks the namespaced string shape, not an exhaustive remote model catalog. Raw SDK imports do not register global Viteval model types. The registry checks active provider registration and delegates model-kind/ID validation to the SDK; some SDKs defer unknown-ID errors until the actual API call. No credentials or remote catalog lookup is required just to load config.

Direct SDK model instances remain supported on scorers. Deterministic scorers need no model configuration.

Storage integration imports contribute typed names and dataset handles. Type visibility does not prove an integration is active in the selected config; registration and capabilities are also checked at runtime.

Example

import { openai } from '@ai-sdk/openai';
import { braintrust } from '@viteval/providers/braintrust';
import { createJsonProvider, defineConfig } from 'viteval';

export default defineConfig({
  eval: {
    include: ['src/**/*.eval.ts'],
    exclude: ['**/*.draft.eval.ts'],
    timeout: 60_000,
    concurrency: 5,
    retries: 2,
    threshold: 0.8,
    aggregation: 'mean',
    results: ['json', 'braintrust'],
  },
  dataset: { providers: ['json'] },
  models: {
    providers: [openai],
    defaults: {
      language: 'openai:gpt-4.1-mini',
      embedding: 'openai:text-embedding-3-small',
    },
  },
  providers: [
    createJsonProvider({ root: import.meta.dirname }),
    braintrust({ project: 'support', apiKey: process.env.BRAINTRUST_API_KEY }),
  ],
});

defineConfig is also exported from viteval/config. Config modules may run in the coordinating process and again in workers. Do not make model calls, generate datasets, or open resources at module scope. Viteval initializes selected storage integrations lazily and closes them after evaluation persistence.

References