---
url: https://viteval.dev/features.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

Why Viteval

# Test prompts and agents
against reference cases.

Run your prompts and agents against real cases. Score the answers and catch regressions before you ship.

[Explore features](#familiar-tooling)[See examples](/examples)

![Captured order extraction evaluation: three cases pass with ExactMatch scores of 1](/assets/viteval-hero-run.svg)

[Vitest-native
](#familiar-tooling)[Scoring
](#built-in-defaults)[Local results
](#local-first)[Your infrastructure
](#run-anywhere)

Familiar tooling


## Use your Vitest configuration

Keep native configuration, watch mode, retries, and CI exit codes. Import your
application functions into an evaluation, then score their answers.
[Write an eval →
](/guide/getting-started)

VITEST![](/assets/vitest-mark.svg)

VITEVAL![](/assets/viteval-flask.svg)

Same runner. A new kind of test.

Your criteria


## Score exact answers and free-form responses

Use deterministic checks for exact answers and structured output. Add embeddings
or LLM judges for free-form responses, or define a scorer for your own criteria.
[Choose a scorer →
](/api/scorers)

SCORER RESULTSexactMatchanswerSimilarityyourScorer

Local by default


## Inspect results locally

Start with JSON datasets and results in your project. Open the local dashboard
to inspect inputs, reference answers, outputs, and individual scores.
[Open the dashboard →
](/guide/cli#view-results)

~/my-agentviteval runresults saved locally

Run on your infrastructure


## Run evaluations in CI

Run the same evaluations on your laptop, in CI, or on a runner you control.
Return a nonzero exit code when a case fails its scoring threshold.

[Set up CI →
](/guide/advanced/ci)[Choose providers →
](/guide/providers)

LAPTOPCIPRIVATE

## Put your agent to the test.

Open source. No hosted account required.

[Write your first eval](/guide/getting-started)[See examples](/examples)


---
url: https://viteval.dev/guide/getting-started.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Getting started

Run a passing evaluation, save its JSON report, and inspect it locally. Requires Node >=26.

> \[!NOTE]
> V1 is currently a release candidate. Install `viteval@rc`; untagged installs
> still resolve to v0. V0 support ends when v1 launches.

## What is Viteval?

Viteval runs LLM evaluations as Vitest tests. Each evaluation combines test cases,
a task function, and scorers. You can:

- Run evaluations with native Vitest configuration and CLI options
- Store datasets as JSON or use an optional provider
- Use built-in scorers or write your own
- Fail a CI job when an evaluation fails

## Installation

Install the release candidate:

```bash [npm]
npm install --save-dev viteval@rc
```

```bash [pnpm]
pnpm add -D viteval@rc
```

```bash [yarn]
yarn add --dev viteval@rc
```

## Initialize a Viteval project

```bash [npm]
npx viteval init
```

```bash [pnpm]
pnpm exec viteval init
```

```bash [yarn]
yarn viteval init
```

This creates `vitest.config.ts` for ESM projects (`vitest.config.mts` for CommonJS),
`viteval.setup.ts`, and a passing `src/viteval.eval.ts` example. Existing
configuration is preserved; if you already have a Vitest config, follow the
printed instructions to register the setup file and reporter.

The setup file binds JSON dataset storage to the project root. In noninteractive
environments, initialization skips the env-file prompt; use `--create-env-file`
to create it explicitly, or `--no-env-file` to skip it locally.

## Your first evaluation

Init generates a small uppercase-string smoke test. Replace its contents with
this order-ID extraction check; it also needs no API key:

```ts
import { evaluate, scorers } from 'viteval';

evaluate('Order extraction', {
  data: [
    { input: 'Where is order ORD-1042?', expected: 'ORD-1042' },
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

### Running your evaluation

Run the evaluation:

```bash
pnpm exec viteval run
```

The command:

1. Discovers the files matched by your native Vitest configuration
2. Executes each evaluation
3. Displays Vitest test results
4. Exits with a non-zero code if any evaluation fails

### Understanding the results

Vitest reports three passing cases. The JSON reporter saves inputs, outputs, scores,
and execution diagnostics under `.viteval/results`. Inspect them with:

```bash
pnpm exec viteval ui
```

Remove the regular expression’s `i` flag and run again. The lowercase ID now
produces an empty string, scores 0, and fails the default threshold of 1. Restore
the flag before continuing. These cases check normalization and missing values;
they do not establish extraction accuracy on arbitrary support messages.

## Your first dataset

Create `src/order-extraction.dataset.ts`:

```ts
import { defineDataset } from 'viteval/dataset';

export default defineDataset({
  name: 'order-extraction',
  data: async () => [
    { input: 'Where is order ORD-1042?', expected: 'ORD-1042' },
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
});
```

### Adding the dataset to the evaluation

Import the dataset and pass it to `data`:

```ts
import { evaluate, scorers } from 'viteval';
import dataset from './order-extraction.dataset';

evaluate('Order extraction', {
  data: dataset,
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

### Running the evaluation with the dataset

Save the dataset, then run the evaluation:

```bash
pnpm exec viteval data
pnpm exec viteval run
```

With the generated JSON provider setup, local datasets live in `.viteval/datasets`
under the project root.

## Evaluate a model

Install an AI client and configure its credentials in your environment:

```bash
pnpm add ai @ai-sdk/openai
```

Replace the deterministic task with a model call in `src/viteval.eval.ts`.
Keep the same dataset so the two implementations face the same checks:

```ts
import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';
import { evaluate, scorers } from 'viteval';
import dataset from './order-extraction.dataset';

evaluate('Model order extraction', {
  data: dataset,
  task: async ({ input }) => {
    const result = await generateText({
      model: openai('gpt-4.1-mini'),
      system:
        'Extract the ORD- order ID. Return only the uppercase ID, or an empty string if none is present.',
      prompt: input,
    });
    return result.text.trim();
  },
  scorers: [scorers.exactMatch()],
});
```

This example makes three model requests and requires `OPENAI_API_KEY`. It is not
guaranteed to pass: extra explanation or an invented ID fails exact matching. Built-in
LLM judges also require `initializeModel()` in worker setup. See [Configuration](https://viteval.dev/api/configuration.md).

## Next steps

For more detail:

- [Core concepts](https://viteval.dev/guide/concepts.md): datasets, scorers, and tasks
- [CLI usage](https://viteval.dev/guide/cli.md): file filters, dataset generation, and the dashboard
- [Examples](https://viteval.dev/examples/index.md): deterministic and model-backed evaluations
- [CI integration](https://viteval.dev/guide/advanced/ci.md): automated runs and saved reports



---
url: https://viteval.dev/api/evaluate.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# `evaluate()`

`evaluate()` creates a Vitest suite with a test for each case. It runs your task,
scores the output, and checks the threshold.

## Import

```ts
import { evaluate } from 'viteval';
```

## Signature

```ts
function evaluate<DATA extends Data, TASK_OUTPUT = InferDataOutput<DATA>>(
  name: string,
  {
    data,
    aggregation = 'mean',
    task,
    scorers,
    threshold = 1.0,
    testOptions,
  }: Eval<DATA, TASK_OUTPUT>
);
```

## Parameters

### `name`

- **Type**: `string`
- **Required**: Yes
- **Description**: The suite name shown in results

### `options`

- **Type**: `Eval<DATA, TASK_OUTPUT>`
- **Required**: Yes
- **Description**: Configuration object for the evaluation

## Eval interface

```ts
interface Eval<DATA extends Data, TASK_OUTPUT = InferDataOutput<DATA>> {
  /**
   * The description of the evaluation.
   */
  description?: string;
  /**
   * The data to use for the evaluation.
   */
  data: DATA;
  /**
   * The task to evaluate.
   */
  task: Task<InferDataInput<DATA>, TASK_OUTPUT, InferDataExtra<DATA>>;
  /**
   * The scorers to use for the evaluation.
   */
  scorers: Scorer<TASK_OUTPUT, InferDataOutput<DATA>, InferDataExtra<DATA>>[];
  /**
   * The aggregation type for the evaluation.
   *
   * @default 'mean'
   */
  aggregation?: ScorerAggregationType;
  /**
   * The threshold for the evaluation.
   *
   * @default 1.0
   */
  threshold?: number;
  /** Native Vitest options applied to every generated test case. */
  testOptions?: TestOptions;
}
```

### `data`

Provide cases as an array, a generator function, or a dataset.

**Type**: `Data<DATA_ITEM>`

```ts
type Data<DATA_ITEM extends DataItem> =
  | DATA_ITEM[]
  | DataGenerator<DATA_ITEM>
  | Dataset<DataGenerator<DATA_ITEM>>;
```

```ts
// Inline data
data: [
  { input: 'What is 2+2?', expected: '4' },
  { input: 'What is 3+3?', expected: '6' },
];

// From function
data: async () => {
  const response = await fetch('https://example.com/api/test-cases');
  if (!response.ok)
    throw new Error(`Dataset request failed: ${response.status}`);
  return response.json();
};

// From dataset
data: mathDataset;
```

### `task`

The function that receives a case's input and returns the output to score.
`TASK_OUTPUT` is inferred from the task's return value and can differ from the
dataset's expected type. Additional dataset fields are available alongside
`input`; `expected` is reserved for scorers.

**Type**: `Task<InferDataInput<DATA>, TASK_OUTPUT, InferDataExtra<DATA>>`

```ts
type Task<INPUT, OUTPUT, EXTRA extends Extra> = (
  args: TaskArgs<INPUT, EXTRA>
) => Promise<OUTPUT> | OUTPUT;

type TaskArgs<INPUT, EXTRA extends Extra> = EXTRA & { input: INPUT };
```

```ts
import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';

// Simple text generation
task: async ({ input }) => {
  const result = await generateText({
    model: openai('gpt-4.1-mini'),
    prompt: input,
  });
  return result.text;
};

// With chat messages
task: async ({ input }) => {
  const result = await generateText({
    model: openai('gpt-4.1-mini'),
    messages: [
      { role: 'system', content: 'You are a math tutor.' },
      { role: 'user', content: input },
    ],
  });
  return result.text;
};

// Structured input
task: async ({ input, context }) => {
  return await answerQuestion(input.question, context);
};
```

### `scorers`

Scorer functions to run against each task output.

**Type**: `Scorer<TASK_OUTPUT, InferDataOutput<DATA>, InferDataExtra<DATA>>[]`

```ts
import { scorers, createScorer } from 'viteval';

// Built-in scorers
scorers: [scorers.exactMatch()];
scorers: [scorers.levenshtein(), scorers.factual()];

// Custom scorer
const customScorer = createScorer<string>({
  name: 'length-check',
  score: ({ output, expected }) => {
    return {
      score: output.length === expected.length ? 1 : 0,
      metadata: { method: 'length_comparison' },
    };
  },
});

scorers: [customScorer, scorers.answerSimilarity()];
```

### `threshold` (optional)

Minimum score for each case's configured mean, median, or sum aggregation.

**Type**: `number`\
**Default**: `1.0`\
**Range**: Any finite number. Sum thresholds may exceed `1.0`.

```ts
// Require 90% average score
threshold: 0.9;

// More lenient threshold
threshold: 0.6;

// Perfect scores only (default)
threshold: 1.0;
```

### `testOptions` (optional)

Native [Vitest test options](https://vitest.dev/api/test.html#test-options) applied
to every generated test case. Set timeouts, retries, repeats, concurrency, or
tags here.

**Type**: `TestOptions`

```ts
// Set a timeout and retry transient model failures.
testOptions: { timeout: 60_000, retry: 2 };

// Disable the timeout.
testOptions: { timeout: 0 };
```

### `aggregation` (optional)

How to combine a case's scorer results before checking its threshold.

**Type**: `ScorerAggregationType`\
**Default**: `'mean'`\
**Options**: `'mean' | 'median' | 'sum'`

```ts
// Use mean score (default)
aggregation: 'mean';

// Use median score
aggregation: 'median';

// Use sum of scores
aggregation: 'sum';
```

### `description` (optional)

An optional description of what the evaluation checks.

**Type**: `string`

```ts
description: 'Evaluates math problem solving capabilities';
```

## DataItem interface

Each case is a `DataItem`:

```ts
type DataItem<
  INPUT = unknown,
  OUTPUT = unknown,
  EXTRA extends Extra = Extra,
> = EXTRA & {
  name?: string;
  input: INPUT;
  expected: OUTPUT;
};
```

`Extra` holds additional case fields:

```ts
type Extra = Record<string, unknown>;
```

### Simple test cases

```ts
{ input: "Hello", expected: "Hi there!" }
```

### Complex input types

```ts
// Object input
{
  input: {
    question: "What's the weather?",
    location: "San Francisco"
  },
  expected: "Sunny, 72°F"
}

// Array input
{
  input: ["apple", "banana", "cherry"],
  expected: "apple, banana, cherry"
}
```

### Test case metadata

```ts
{
  input: "Translate: Hello",
  expected: "Hola",
  difficulty: "easy",
  language: "spanish"
}
```

### Named test cases

```ts
{
  name: "Basic addition",
  input: "What is 2+2?",
  expected: "4"
}
```



---
url: https://viteval.dev/examples.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Examples

These examples cover deterministic checks, generated datasets, and model-based
scoring. The first two run without an API key.

## Extract order IDs

Check a deterministic extractor against casing differences and a missing ID.
Use these same cases when replacing the task with a model call.

```ts
import { evaluate, scorers } from 'viteval';

evaluate('Order extraction', {
  data: [
    { input: 'Where is order ORD-1042?', expected: 'ORD-1042' },
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

## Reusable dataset

Generate valid IDs in both cases to check the normalization rule over more values.
This adds coverage of the ID format, not the variety of real customer messages.

```ts
import { defineDataset } from 'viteval/dataset';
import { evaluate, scorers } from 'viteval';

const orderMessages = defineDataset({
  name: 'order-id-casing',
  data: async () =>
    Array.from({ length: 10 }, (_, index) => {
      const id = `ORD-${1042 + index}`;
      return {
        input: `Please cancel ${index % 2 === 0 ? id : id.toLowerCase()}.`,
        expected: id,
      };
    }),
});

evaluate('Order ID casing', {
  data: orderMessages,
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

## Multiple scorers

Install `ai` and `@ai-sdk/openai`, and provide `OPENAI_API_KEY` in the environment.
Initialize the judge and embedding models in the setup file registered by your
Vitest configuration:

```ts [viteval.setup.ts]
import { openai } from '@ai-sdk/openai';
import { initializeModel } from 'viteval';

initializeModel({
  language: openai('gpt-4.1-mini'),
  embedding: openai.embedding('text-embedding-3-small'),
});
```

Keep any existing provider initialization in that setup file. Then add the eval:

```ts
import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';
import { evaluate, scorers } from 'viteval';

const model = openai('gpt-4.1-mini');

evaluate('Support policy answers', {
  data: [
    {
      input: 'Can I return an unopened item after 14 days?',
      expected: 'Unopened items can be returned within 30 days of delivery.',
    },
  ],
  task: async ({ input }) =>
    (
      await generateText({
        model,
        system:
          'Answer using this policy: unopened items can be returned within 30 days of delivery. Return shipping is paid by the customer.',
        prompt: input,
      })
    ).text,
  scorers: [
    scorers.factual(),
    scorers.answerSimilarity(),
    scorers.moderation(),
  ],
  threshold: 0.8,
});
```

The threshold applies to the mean of the three scores. This example performs
real generation, embedding, and judge requests. Judge scores vary by model and
are not proof that an answer follows your policy. For exact extraction checks,
use the [model-backed getting-started example](https://viteval.dev/guide/getting-started.md#evaluate-a-model).

## Runnable examples

The [repository examples](https://github.com/viteval/viteval/tree/main/examples)
include project configuration and setup files:

- [Basic Evaluation](https://github.com/viteval/viteval/tree/main/examples/basic): deterministic evaluations without API keys.
- [Vercel AI SDK](https://github.com/viteval/viteval/tree/main/examples/vercel-ai): model-backed evaluations with the AI SDK.
- [VoltAgent](https://github.com/viteval/viteval/tree/main/examples/voltagent): evaluations for agent workflows.



---
url: https://viteval.dev/api.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# API reference


[ConfigurationConfigure Viteval in `vitest.config.ts` using Vitest's `test` options. Initialize models and storage providers in a worker setup file.](/api/configuration)[createScorer()`createScorer()` gives a scoring function a name. Use the returned scorer in an evaluation's `scorers` array.](/api/create-scorer)[defineDataset()`defineDataset()` wraps a case generator in a reusable dataset. Pass the dataset to `evaluate()` or save it with `viteval data`.](/api/define-dataset)[evaluate()`evaluate()` creates a Vitest suite with a test for each case. It runs your task, scores the output, and checks the threshold.](/api/evaluate)[ScorersViteval includes deterministic scorers, embedding similarity scorers, and LLM judges. Call each factory to create a scorer. LLM and embedding scorers use the models initialized in Vitest setup. See [Configuration](/api/configuration).](/api/scorers)


---
url: https://viteval.dev/api/configuration.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Configuration

Configure Viteval in `vitest.config.ts` using Vitest's `test` options. Initialize
models and storage providers in a worker setup file.

## Import

```ts
import { defineConfig } from 'viteval/config';
import { JsonReporter } from 'viteval/reporters';
```

## Configuration file

Create a `vitest.config.ts` file in your project root:

```ts
import { defineConfig } from 'viteval/config';

import { JsonReporter } from 'viteval/reporters';

export default defineConfig({
  test: {
    include: ['src/**/*.eval.ts'],
    reporters: [
      'default',
      new JsonReporter({ outputFile: '.viteval/results/<run>.json' }),
    ],
    setupFiles: ['./viteval.setup.ts'],
    testTimeout: 100_000,
  },
});
```

`viteval/config` re-exports Vitest's `defineConfig`. Use the standard
[Vitest configuration options](https://vitest.dev/config/).

## Models and providers

Initialize non-serializable models and Viteval providers in a setup file. Vitest executes setup files in each worker before collecting evaluations.

```ts
import { fileURLToPath } from 'node:url';
import { openai } from '@ai-sdk/openai';
import { initializeModel } from 'viteval';
import { createJsonProvider, initializeProvider } from 'viteval/provider';

initializeModel({
  language: openai('gpt-4.1-mini'),
  embedding: openai.embedding('text-embedding-3-small'),
});
const providerResult = await initializeProvider({
  datasets: createJsonProvider({
    root: fileURLToPath(new URL('.', import.meta.url)),
  }),
});
if (!providerResult.ok) throw providerResult.error;
```

`initializeModel()` is synchronous and configures the model used by LLM-based
scorers. Deterministic scorers need no model. See [Providers](https://viteval.dev/guide/providers.md)
for SQLite and Braintrust setup.

## Evaluation files

Use Vitest's `test.include` and `test.exclude` options:

```ts
export default defineConfig({
  test: {
    include: ['src/**/*.eval.ts', 'evaluations/**/*.ts'],
    exclude: ['**/*.draft.eval.ts', 'src/experimental/**'],
  },
});
```

## Timeouts

Set the default with Vitest's `test.testTimeout`. Pass native `testOptions` to override it for one evaluation suite.

```ts
export default defineConfig({
  test: {
    testTimeout: 300_000,
  },
});
```

```ts
evaluate('slow evaluation', {
  data,
  scorers,
  task,
  testOptions: { timeout: 600_000 },
});
```

## Vite options

Set Vite options, including aliases and server settings, at the top level:

```ts
export default defineConfig({
  resolve: {
    alias: {
      '#': new URL('./src', import.meta.url).pathname,
    },
  },
});
```



---
url: https://viteval.dev/api/create-scorer.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# `createScorer()`

`createScorer()` gives a scoring function a name. Use the returned scorer in
an evaluation's `scorers` array.

## Import

```ts
import { createScorer } from 'viteval';
```

## Signature

```ts
function createScorer<OUTPUT, EXPECTED = OUTPUT, EXTRA extends Extra = Extra>(
  config: ScorerConfig<OUTPUT, EXPECTED, EXTRA>
): Scorer<OUTPUT, EXPECTED, EXTRA>;
```

## Parameters

### `config`

- **Type**: `ScorerConfig<OUTPUT, EXPECTED, EXTRA>`
- **Required**: Yes
- **Description**: Configuration object for the scorer

## ScorerConfig

```ts
interface ScorerConfig<OUTPUT, EXPECTED = OUTPUT, EXTRA extends Extra = Extra> {
  name: string;
  score: (
    args: ScorerArgs<OUTPUT, EXPECTED, EXTRA>
  ) => ScoreResult | Promise<ScoreResult>;
}

type ScoreResult = Omit<Score, 'name'> & { name?: string };
```

### `name`

Unique identifier for the scorer.

**Type**: `string`\
**Required**: Yes

```ts
createScorer<string>({
  name: 'exact-match',
  score: ({ output, expected }) => ({ score: output === expected ? 1 : 0 }),
});
```

### `score`

The function that scores a task output against the expected value.

**Type**: `(args: ScorerArgs<OUTPUT, EXPECTED, EXTRA>) => ScoreResult | Promise<ScoreResult>`

**Required**: Yes

The function receives `ScorerArgs` and returns a score with optional metadata.
Viteval adds the scorer name:

```ts
type ScorerArgs<OUTPUT, EXPECTED, EXTRA extends Extra> = EXTRA & {
  output: OUTPUT;
  expected: EXPECTED;
};

interface Score {
  name: string;
  score: number | null;
  metadata?: Record<string, unknown>;
}
```

Use normalized scores between 0 and 1 for the usual mean/median gates. V1 accepts
any finite score; null counts as zero. Non-finite scores and scorer exceptions
fail the case. Typical normalized scores mean:

- `1.0` = Perfect score
- `0.0` = Complete failure
- `0.5` = Partial success

## Return value

Returns a `Scorer<OUTPUT, EXPECTED, EXTRA>` function that can be used in evaluations:

```ts
type Scorer<OUTPUT, EXPECTED = OUTPUT, EXTRA extends Extra = Extra> = (
  args: ScorerArgs<OUTPUT, EXPECTED, EXTRA>
) => Score | Promise<Score>;
```

`createScorer()` supplies the configured name, overriding any name returned by
the callback. `evaluate()` passes additional case fields to each scorer. It also
passes `input` at runtime, but the current `ScorerArgs` type does not infer its
type separately. Declare `input` in `EXTRA` when using it in a typed custom
scorer. Task output and expected values may have different types:

```ts
const correctAnswer = createScorer<{ answer: string }, string>({
  name: 'correct-answer',
  score: ({ output, expected }) => ({
    score: output.answer === expected ? 1 : 0,
  }),
});
```



---
url: https://viteval.dev/api/define-dataset.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# `defineDataset()`

`defineDataset()` wraps a case generator in a reusable dataset. Pass the dataset
to `evaluate()` or save it with `viteval data`.

## Import

```ts
import { defineDataset } from 'viteval/dataset';
```

## Signature

```ts
function defineDataset<DATA_FUNC extends DataGenerator>(
  config: DatasetConfig<DATA_FUNC>
): Dataset<DATA_FUNC>;
```

## Parameters

### `config`

- **Type**: `DatasetConfig<DATA_FUNC>`
- **Required**: Yes
- **Description**: Configuration object for the dataset

## DatasetConfig

```ts
interface DatasetConfig<DATA extends DataGenerator = DataGenerator> {
  /**
   * The storage type of the dataset.
   *
   * @default 'local'
   */
  storage?: DatasetStorage;
  /**
   * The name of the dataset.
   */
  name: string;
  /**
   * The description of the dataset.
   */
  description?: string;
  /**
   * The data generator of the dataset.
   */
  data: DATA;
}
```

### `name`

Unique identifier for the dataset.

**Type**: `string`\
**Required**: Yes

```ts
defineDataset({
  name: 'math-problems',
  data: async () => [{ input: '2 + 2', expected: '4' }],
});
```

### `data`

Function that generates or loads the dataset.

**Type**: `() => Promise<DataItem<INPUT, OUTPUT, EXTRA>[]>`\
**Required**: Yes

```ts
// Static data
data: async () => [
  { input: '2 + 2', expected: '4' },
  { input: '3 + 3', expected: '6' },
];

// Async data loading
data: async () => {
  const response = await fetch('https://example.com/api/datasets/math');
  if (!response.ok)
    throw new Error(`Dataset request failed: ${response.status}`);
  return response.json();
};

// Generated data
data: async () => {
  const problems = [];
  for (let i = 0; i < 1000; i++) {
    const a = Math.floor(Math.random() * 100);
    const b = Math.floor(Math.random() * 100);
    problems.push({
      input: `${a} + ${b}`,
      expected: String(a + b),
    });
  }
  return problems;
};
```

### `storage` (optional)

Choose persisted storage or regenerate cases in memory on each load.

**Type**: `DatasetStorage`\
**Default**: `'local'`

```ts
// Use the configured dataset provider (JSON by default).
storage: 'local';

// Memory storage (not persisted)
storage: 'memory';
```

### `description` (optional)

An optional description of the cases in the dataset.

**Type**: `string`

```ts
description: 'Math problems for basic arithmetic evaluation';
```

## DataItem interface

The generator returns an array of `DataItem` objects:

```ts
type DataItem<
  INPUT = unknown,
  OUTPUT = unknown,
  EXTRA extends Extra = Extra,
> = EXTRA & {
  name?: string;
  input: INPUT;
  expected: OUTPUT;
};
```

`Extra` holds additional case fields:

```ts
type Extra = Record<string, unknown>;
```

## Return value

Returns a `Dataset<DATA_FUNC>` object:

```ts
interface Dataset<DATA_FUNC extends DataGenerator> {
  name: string;
  storage: 'local' | 'memory';
  description?: string;
  exists(): Promise<Result<boolean>>;
  load(options?: {
    create?: boolean;
  }): Promise<Result<Awaited<ReturnType<DATA_FUNC>> | null>>;
  save(options?: { overwrite?: boolean }): Promise<Result<void>>;
}
```

Use `load()` and `save()`; the returned dataset does not expose a `data()` method:

```ts
const saved = await dataset.save({ overwrite: true });
if (!saved.ok) throw saved.error;
const loaded = await dataset.load({ create: true });
if (!loaded.ok) throw loaded.error;
const cases = loaded.value;
```

Local storage uses the configured dataset provider, falling back to JSON under
the current working directory. Generated setup binds that directory to the
selected project. Memory datasets regenerate on load and do not persist. See
[Providers](https://viteval.dev/guide/providers.md) for optional backends and serialization rules.

`storage: 'local'` is the persisted-storage mode, not a guarantee of local files.
With an S3 dataset provider configured, the same dataset reads and writes S3.
`evaluate()` loads the saved dataset by name and generates it only when missing.
Use `viteval data --overwrite` to replace saved cases after changing the generator.

## CLI discovery

`viteval data` discovers datasets in `.dataset.*` and `.eval.*` modules. Default
exports, named exports, and non-exported `defineDataset()` calls are supported.
Multiple datasets can share a module. Re-exporting the same dataset does not
generate it twice; distinct objects must have unique names.

Evaluation tasks and scorers are not executed during generation. See
[CLI datasets](https://viteval.dev/guide/cli.md#datasets) for patterns, overwrite behavior, and module
import semantics.



---
url: https://viteval.dev/api/scorers.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# `scorers`

Viteval includes deterministic scorers, embedding similarity scorers, and LLM
judges. Call each factory to create a scorer. LLM and embedding scorers use the
models initialized in Vitest setup. See [Configuration](https://viteval.dev/api/configuration.md).

The scores in the examples illustrate how to read results. Model-based scores
vary with the configured model and are not guarantees of correctness or safety.

Judge scorers accept `{ model }` to override the initialized language model.
Model failures return `score: null` with an error in `metadata`; null scores
contribute 0 to evaluation aggregation.

## Import

```ts
import { scorers } from 'viteval';
```

## Text similarity

### `exactMatch`

Returns 1 if output exactly matches expected, 0 otherwise.

```ts
scorers.exactMatch();
```

**Use cases**: Exact answers, structured outputs, code generation

**Example**:

```ts
// Input: "What is 2+2?"
// Output: "4"
// Expected: "4"
// Score: 1.0

// Output: "Four"
// Expected: "4"
// Score: 0.0
```

### `levenshtein`

Scores text similarity using Levenshtein edit distance.

```ts
scorers.levenshtein();
```

**Range**: 0.0 - 1.0 (higher = more similar)\
**Use cases**: Approximate text matching, typo tolerance

**Example**:

```ts
// Output: "The sky is blue"
// Expected: "The sky is bleu"
// Score: ~0.87 (two edits across 15 characters)

// Output: "Blue sky"
// Expected: "The sky is blue"
// Score: ~0.27 (11 edits across 15 characters)
```

### `answerSimilarity`

Semantic similarity using embeddings.

```ts
scorers.answerSimilarity();
```

**Range**: 0.0 - 1.0\
**Use cases**: Meaning-based comparison, paraphrasing

**Example**:

```ts
// Output: "The capital of France is Paris"
// Expected: "Paris is France's capital city"
// Score: ~0.95 (same meaning, different wording)
```

### `embeddingSimilarity`

Measures semantic similarity using embeddings.

```ts
scorers.embeddingSimilarity();
```

**Range**: 0.0 - 1.0\
**Use cases**: Semantic search, content matching

**Model setup**:

- Uses the embedding model configured in worker setup
- Compares outputs and expected answers with cosine similarity
- Language support depends on the selected model

## Content quality

### `factual`

Asks an LLM judge to compare the output's factual accuracy with the expected answer.

```ts
scorers.factual();
```

**Range**: 0.0 - 1.0\
**Use cases**: Knowledge-based QA, educational content

The reported scorer name is `Factuality`. The judge returns one of five
categories: consistent subset (0.4), consistent superset (0.6), same details (1),
disagreement (0), or immaterial differences (1). This is not a continuous
confidence score.

**Example**:

```ts
// Question: "What is the capital of France?"
// Output: "Paris is the capital of France."
// Expected: "Paris is the capital of France."
// Score: 1.0 (factually correct)

// Output: "London is the capital of France"
// Expected: "Paris"
// Score: 0.0 (factually incorrect)
```

### `summary`

Asks an LLM judge to compare the submitted summary with an expert summary of
`input`. Returns 0 or 1.

```ts
scorers.summary();
```

**Range**: 0.0 - 1.0\
**Use cases**: Text summarization, content condensation

**Criteria**:

- Compares which summary better describes the original text
- Ignores style, grammar, and punctuation differences

### `translation`

Asks an LLM judge to assess a translation against the expected answer.

```ts
scorers.translation();
```

**Range**: 0.0 - 1.0\
**Use cases**: Machine translation, multilingual content

**Criteria**:

- Meaning preservation, including nouns and tenses
- Verbs or adjectives that change the meaning
- Ignores style and punctuation differences

Include a `language` field in each case to name the source language. The judge
compares translations into English and returns 0 or 1.

## Answer quality

### `answerCorrectness`

Asks an LLM judge to compare an answer with the expected answer.

```ts
scorers.answerCorrectness();
```

**Range**: 0.0 - 1.0\
**Use cases**: QA systems, educational assessments

### `answerRelevancy`

Asks an LLM judge how relevant the answer is to the question.

```ts
scorers.answerRelevancy();
```

**Range**: 0.0 - 1.0\
**Use cases**: Chat systems, search results

**Example**:

```ts
// Question: "What's the weather like?"
// Output: "It's sunny and 75°F"
// Score: 1.0 (highly relevant)

// Output: "I like pizza"
// Score: 0.0 (not relevant)
```

## Context evaluation

### `contextEntityRecall`

Uses an LLM judge to check whether the context mentions the expected entities.

```ts
scorers.contextEntityRecall();
```

**Range**: 0.0 - 1.0\
**Use cases**: Information extraction, entity recognition

### `contextPrecision`

Uses an LLM judge to assess how much of the retrieved context is relevant to the
question and expected answer. Pass the retrieved context as `output`.

```ts
scorers.contextPrecision();
```

**Range**: 0.0 - 1.0\
**Use cases**: RAG systems, context-aware generation

### `contextRecall`

Uses an LLM judge to assess whether the retrieved context supports all claims in
the expected answer. Pass the retrieved context as `output`.

```ts
scorers.contextRecall();
```

**Range**: 0.0 - 1.0\
**Use cases**: Reading comprehension, context utilization

### `contextRelevancy`

Uses an LLM judge to assess whether the retrieved context is relevant to the
question. Pass the retrieved context as `output`.

```ts
scorers.contextRelevancy();
```

**Range**: 0.0 - 1.0\
**Use cases**: Context filtering, relevance ranking

## Safety and moderation

### `moderation`

Asks an LLM judge to classify the output as safe or unsafe. This is a model-based
assessment, not a guarantee that an output is safe.

```ts
scorers.moderation();
```

**Range**: 0.0 - 1.0 (1.0 = safe, 0.0 = unsafe)\
**Use cases**: Content filtering, safety checks

**The judge checks for**:

- Hate speech
- Violence
- Self-harm
- Sexual content
- Harassment
- Dangerous or illegal activities

**Example**:

```ts
// Output: "Here's a helpful recipe for cookies"
// Score: 1.0 (safe content)

// Output: "Instructions for harmful activities"
// Score: 0.0 (unsafe content)
```

## Structured data

### `jsonDiff`

Compares JSON structures for differences.

```ts
scorers.jsonDiff();
```

**Range**: 0.0 - 1.0\
**Use cases**: API responses, structured output validation

**Example**:

```ts
// Output: '{"name": "John", "age": 30}'
// Expected: '{"name": "John", "age": 30}'
// Score: 1.0 (exact match)

// Output: '{"name": "John"}'
// Expected: '{"name": "John", "age": 30}'
// Score: 0.5 (partial match)
```

### `sql`

Asks an LLM judge whether the submitted SQL is equivalent to the expected query.
It does not execute queries or validate them against a database.

```ts
scorers.sql();
```

**Range**: 0.0 - 1.0\
**Use cases**: SQL generation, query validation

**Criteria**:

- Whether the queries would return equivalent results
- Whether the submitted query appears likely to fail
- Ignores whitespace, style, and output column naming differences

## Numeric and list comparison

### `numericDiff`

Calculates numerical difference between outputs.

```ts
scorers.numericDiff();
```

**Range**: 0.0 - 1.0\
**Use cases**: Mathematical calculations, numeric predictions

**Example**:

```ts
// Output: "42"
// Expected: "40"
// Score: ~0.95 (small difference)

// Output: "100"
// Expected: "40"
// Score: 0.4 (1 - 60 / 100)
```

### `listContains`

Verifies if a list contains expected items.

```ts
scorers.listContains();
```

**Range**: 0.0 - 1.0\
**Use cases**: List generation, item retrieval

**Example**:

```ts
// Output: ["apple", "banana", "cherry"]
// Expected: ["apple", "banana"]
// Score: 1.0 (contains all expected items)

// Output: ["apple"]
// Expected: ["apple", "banana"]
// Score: 0.5 (contains half of expected items)
```

## Specialized scorers

### `possible`

Asks an LLM judge whether a response offers a solution (1) or declares the task
impossible (0). It does not compare the response with `expected` or check whether
the proposed solution is correct.

```ts
scorers.possible();
```

**Range**: 0.0 - 1.0\
**Use cases**: Detecting declarations of impossibility versus solution attempts

**Example**:

```ts
// Input: "Write a haiku about rain."
// Output: "Rain taps on the roof / Rivers gather in the street / Clouds drift into dusk"
// Score: 1.0 (offers a solution)

// Output: "Writing a haiku about rain is impossible."
// Score: 0.0 (declares the task impossible)
```

### `humor`

Asks an LLM judge whether the output is funny: no (0), unsure (0.5), or yes (1).

```ts
scorers.humor();
```

**Range**: 0.0 - 1.0\
**Use cases**: Creative writing, entertainment content

This is a humor check, not a safety or appropriateness check. Use `moderation`
separately when needed.



---
url: https://viteval.dev/brand.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

Viteval / Brand & assets

# Brand assets

Download Viteval logos and icons for documentation, integrations, and projects.

[Logos](#logos)[Icons & motion](#icons)[Colors](#colors)[Typography](#typography)

## The wordmark

Custom lettering, a round flask, and tight parentheses. Use the complete lockup; don’t recreate it with a font.

![Viteval wordmark for dark surfaces](/assets/viteval-wordmark.svg)

### Dark surfaces

[Download SVG](/assets/viteval-wordmark.svg)

![Viteval wordmark for light surfaces](/assets/viteval-wordmark-light.svg)

### Light surfaces

[Download SVG](/assets/viteval-wordmark-light.svg)

Transparent backgrounds. Vector paths, no fonts or embedded images. Preserve the proportions and leave generous clear space.

## The flask, in motion

The standalone icon for app icons, favicons, and loading states. A stationary outline with two gently moving liquid layers. No floating bubbles.

![Flask at 16 pixels](/assets/viteval-flask.svg)

16px

![Flask at 24 pixels](/assets/viteval-flask.svg)

24px

![Flask at 32 pixels](/assets/viteval-flask.svg)

32px

![Flask at 64 pixels](/assets/viteval-flask.svg)

64px

### App icon

[SVG](/assets/viteval-flask.svg)[PNG](/assets/viteval.png)[ICO](/favicon.ico)

![Loading flask with gently moving cyan liquid](/assets/viteval-loading.svg)Pause preview

### Loading icon

[Download animated SVG](/assets/viteval-loading.svg)

The loading SVG loops on its own and respects reduced motion. When using it in an interface, pair it with an accessible loading status. The preview above is artwork, not a page loading state.

## Four foundations

Blue for the identity. Cyan for the liquid and supporting accents. Dark, simple surfaces with off-white text.

### Background

`#0B1016`

### Blue

`#168DFA`

### Cyan

`#12D9E9`

### Off-white

`#F8FAFC`

Use darker blue and cyan for text on light surfaces. Green, amber, and red communicate results; they aren’t additional brand accents.

## Type that gets out of the way

Bold sans-serif headings, restrained body copy, monospace for code. The site uses Geist and Geist Mono; the logo remains custom artwork.

Geist / Heading

Measure what
your agent does.

Geist / Body

Keep evaluations beside your application code. Run real inputs, compare answers, and inspect what changed.

Geist Mono / Code

````
pnpm exec viteval run
````

## Keep it recognizable

Keep the liquid wave visible, the neck short, and the outline round. Don’t stretch, recolor, add bubbles, or turn the wordmark into text. Use the light variant only on light surfaces.

These SVGs are manually reconstructed from the supplied artwork using the brand guide’s flat palette. The original raster texture is not part of the vector assets.




---
url: https://viteval.dev/guide.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Guide


[CLI usageUse the CLI to initialize projects, run evaluations, generate datasets, and open the dashboard. Evaluation commands pass through to Vitest.](/guide/cli)[Core conceptsAn evaluation runs a task against test cases and scores each output. Datasets hold the cases; thresholds determine whether they pass.](/guide/concepts)[Getting startedRun a passing evaluation, save its JSON report, and inspect it locally. Requires Node >=26.](/guide/getting-started)[Migrate to v1V1 requires Node >=26 and uses native Vitest 5 configuration.](/guide/migration)[ProvidersViteval is open source and uses local JSON storage by default. You do not need a hosted evaluation account. With the default JSON provider, datasets live under `.viteval/datasets` and runs under `.viteval/results`. The JSON reporter also writes local snapshots for dashboard inspection. Configured hosted providers such as Braintrust, Langfuse, and VoltOps persist runs through their own services; they do not move those hosted runs into the local directory.](/guide/providers)


---
url: https://viteval.dev/guide/advanced.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Advanced


[CI integrationRun `viteval run` in CI to check prompt and application changes against your evaluation suite. A failed case returns a nonzero exit code.](/guide/advanced/ci)[ReportersUse Vitest reporters for test results and Viteval's JSON reporter for evaluation inputs, outputs, and scores.](/guide/advanced/reporters)


---
url: https://viteval.dev/guide/advanced/ci.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# CI integration

Run `viteval run` in CI to check prompt and application changes against your
evaluation suite. A failed case returns a nonzero exit code.

## Overview

Run evaluations on pushes or pull requests and use the result as a deployment
check. Save JSON reports so you can compare runs later.

## GitHub Actions

The example below runs evaluations when the selected source paths change.

:::tip

Use `paths` to limit model requests on unrelated changes. Include evaluation
files, datasets, configuration, and dependencies as well as the code under test.
Omit the filter if you want evaluations to run on every change.

:::

Adapt the paths to your repository:

```yaml
name: LLM Evaluation

on:
  push:
    branches: [main, develop]
    # Include every path that can affect evaluation results.
    paths:
      - 'src/**'
      - 'evals/**'
      - 'datasets/**'
      - '**/*.eval.ts'
      - '**/*.dataset.ts'
      - 'vitest.config.*'
      - 'viteval.setup.*'
      - 'package.json'
      - 'package-lock.json'
      - '.github/workflows/evaluate.yml'
  pull_request:
    branches: [main]
    # Keep this filter in sync with the push filter.
    paths:
      - 'src/**'
      - 'evals/**'
      - 'datasets/**'
      - '**/*.eval.ts'
      - '**/*.dataset.ts'
      - 'vitest.config.*'
      - 'viteval.setup.*'
      - 'package.json'
      - 'package-lock.json'
      - '.github/workflows/evaluate.yml'
jobs:
  evaluate:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v7

      - name: Setup Node.js
        uses: actions/setup-node@v7
        with:
          node-version: '26'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run evaluations
        run: npx viteval run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

## Preserve reports

Configure `JsonReporter` to write `.viteval/results/<id>.json`, then upload the
files even when a score or execution failure makes the evaluation step fail:

```yaml
- name: Upload evaluation reports
  if: always()
  uses: actions/upload-artifact@v7
  with:
    name: evaluation-results
    path: .viteval/results/*.json
    if-no-files-found: warn
```

The same `viteval run` command works in GitLab, CircleCI, and other CI systems.
Install with a lockfile, provide model credentials through the CI environment,
and retain the JSON reports. A nonzero Vitest exit code fails the job.



---
url: https://viteval.dev/guide/advanced/reporters.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Reporters

Use Vitest reporters for test results and Viteval's JSON reporter for evaluation
inputs, outputs, and scores.

## Vitest reporters

### `default`

Vitest's default reporter prints results to the console. Viteval does not wrap or replace it.

### `json`

Vitest includes native JSON, JUnit, TAP, tree, verbose, dot, and other reporters. Pass them through Vitest's native CLI:

```sh
viteval run --reporter json
```

## Evaluation JSON

Viteval's JSON reporter saves evaluation scores and artifacts. Vitest's built-in
JSON reporter saves test results without those details. Configure Viteval's
reporter alongside the default console reporter to save evaluation details:

```ts
import { JsonReporter } from 'viteval/reporters';
import { defineConfig } from 'viteval/config';

export default defineConfig({
  test: {
    reporters: [
      'default',
      new JsonReporter({ outputFile: '.viteval/results/<run>.json' }),
    ],
  },
});
```

Viteval's reporter implements Vitest's `Reporter` interface. It reads evaluation
artifacts and records the final case states reported by Vitest.

## Run history and failures

`<id>`, `<run>`, and `<timestamp>` placeholders use the current run identity.
Watch reruns receive new identities. Static output filenames are
overwritten. JSON file replacement is atomic.

For a stable destination, configure the evaluation reporter in `vitest.config.ts`:

```ts
new JsonReporter({ outputFile: '.viteval/results/latest.json' });
```

The old `--outputPath` flag is not part of v1. Native Vitest's `--outputFile` applies
to its built-in reporters; use the reporter constructor above for Viteval's
evaluation JSON. With `new JsonReporter()` and no output file, the completed
evaluation report is written to stdout.

Reports include `schemaVersion: 1`, passed/failed/skipped counts, project names,
case IDs and final states, execution errors, and suite summaries. The suite
median is the median of per-case median scores. Native tests without evaluation
metadata are excluded from evaluation counts; the CLI still preserves Vitest's
overall exit code.

The dashboard uses those recorded outcomes instead of recalculating every case
from its mean score. This preserves median/sum gates, retries, skips, and
execution failures. Empty or malformed files are reported as warnings.



---
url: https://viteval.dev/guide/cli.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# CLI usage

Use the CLI to initialize projects, run evaluations, generate datasets, and open
the dashboard. Evaluation commands pass through to Vitest.

## Commands

### Initialize a project

```bash
viteval init
viteval init --no-env-file
viteval init --create-env-file --env-file-path .env.local
```

Initialization preserves existing configuration. In noninteractive environments,
it creates the configuration and setup module without prompting. Pass `--env-file`
to create an environment file explicitly.

Use its `--create-env-file` alias when selecting a custom path to avoid Node's
runtime `--env-file` option. Absolute and relative paths target the same file
when created and loaded by the generated setup module.

Init writes `vitest.config.ts` for ESM projects and `vitest.config.mts` for
CommonJS projects, so the generated configuration can import the ESM-only Viteval
entry points. Existing Vitest configuration files in either format are preserved.

### Run evaluations

```bash
# Run all evaluations
viteval

# Run specific files
viteval chat.eval.ts

# Run multiple path filters
viteval run chat.eval.ts summarization.eval.ts
```

Evaluation commands use Vitest's CLI directly. File arguments are path filters, not globs. Native options such as `--watch`, `--project`, `--retry`, `--testTimeout`, `--reporter`, and `--coverage` work unchanged.

### Datasets

Generate saved datasets from TypeScript modules:

```bash
# Generate datasets from dataset and eval files
viteval data

# Generate matching datasets
viteval data "src/user-questions.dataset.ts"

# Regenerate existing datasets
viteval data --overwrite

# Import datasets from a specific Vitest project
viteval data --root ./app --project evaluation
```

The default pattern is `**/*.{dataset,eval}.{js,ts,mts,mjs}`. Dataset modules can
use default exports, named exports, or multiple `defineDataset()` calls. Datasets
defined inside eval files are discovered even when they are not exported.

```ts
// src/questions.eval.ts
import { evaluate, scorers } from 'viteval';
import { defineDataset } from 'viteval/dataset';

const questions = defineDataset({
  name: 'questions',
  data: async () => [{ input: 'hello', expected: 'HELLO' }],
});

evaluate('uppercase', {
  data: questions,
  task: ({ input }) => input.toUpperCase(),
  scorers: [scorers.exactMatch()],
});
```

Generation imports these modules in the selected Vitest worker, with its setup
files and configured provider. It does not execute evaluation tasks or scorers.
Module-level code still runs on import; keep expensive calls inside dataset
generators or evaluation tasks. Native setup hooks remain active.

Aliases and re-exports of the same dataset generate it once. Distinct dataset
objects with the same name fail instead of overwriting each other. Existing
datasets are preserved unless `--overwrite` is passed. Memory datasets run their
generator without writing a persisted file. Modules that fail during import do
not publish their datasets.

### View results

```bash
viteval ui --root ./app
viteval ui --port 6274 --open
```

The UI reads datasets and evaluation results from the selected project's `.viteval`
directory. Configure the JSON reporter to save results there.

The installed package includes the dashboard server and browser assets; you do
not need a development server. You can open or reload dashboard, results,
datasets, suites, and settings URLs directly.

### Debugging

```bash
# Show all native Vitest options
viteval run --help --expand-help

# Show Viteval project commands
viteval --help
```

### Compiled CLI

The repository can compile Viteval into an executable using maltty's Bun bundler.
See the [build instructions](https://github.com/viteval/viteval/blob/main/contributing/commands.md#compiled-cli).

The executable runs help, version, and initialization directly. Evaluation,
dataset, and UI commands use the project's installed `viteval` package through
Node >=26. Install `viteval` in the project before using those commands. The
binary preserves native Vitest arguments and exit codes.



---
url: https://viteval.dev/guide/concepts.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Core concepts

An evaluation runs a task against test cases and scores each output. Datasets
hold the cases; thresholds determine whether they pass.

## Evaluations

An evaluation defines:

- `data`: inputs and expected outputs
- `task`: the function that produces an output for each input
- `scorers`: functions that score the output
- `threshold`: the minimum aggregated score for a case to pass

```ts
import { evaluate, scorers } from 'viteval';

evaluate('Order extraction', {
  data: [{ input: 'Where is order ORD-1042?', expected: 'ORD-1042' }],
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
  threshold: 1,
});
```

## Data and datasets

### Inline data

Define cases inline for a small evaluation:

```ts
evaluate('Order extraction', {
  data: async () => [
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

### Local datasets

For reusable datasets, use `defineDataset()`:

```ts
import { defineDataset } from 'viteval/dataset';

const orderMessages = defineDataset({
  name: 'order-extraction',
  data: async () => [
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
});

evaluate('Order extraction', {
  data: orderMessages,
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

### Remote datasets

Initialize a dataset provider in native Vitest setup to use SQLite or Braintrust.
S3, Google Cloud Storage, Langfuse, and VoltOps are also supported within their
documented capabilities. The `defineDataset()` API stays the same. See
[Providers](https://viteval.dev/guide/providers.md).

## Tasks

The task function receives a case's input and returns the output to score.
Call your model or application code inside this function:

```ts
import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';

const extractOrderId = async ({ input }: { input: string }) => {
  const result = await generateText({
    model: openai('gpt-4.1-mini'),
    system:
      'Extract the ORD- order ID. Return only the uppercase ID, or an empty string if none is present.',
    prompt: input,
  });
  return result.text.trim();
};
```

Pass `extractOrderId` as the evaluation’s `task`. This requires `ai`,
`@ai-sdk/openai`, and `OPENAI_API_KEY`. Expected values go to scorers, not to the
task. For structured output, return the object and use `scorers.jsonDiff()`;
see [Scorers](https://viteval.dev/api/scorers.md#jsondiff).

## Scorers

Scorers receive the task output and expected value. Each returns a score and
optional metadata.

### Built-in scorers

Built-in scorers cover text matching, semantic similarity, and model-based judgments:

```ts
import { scorers } from 'viteval';

// Text similarity
scorers.levenshtein(); // Edit distance
scorers.exactMatch(); // Exact match
scorers.answerSimilarity(); // Semantic similarity

// Content quality
scorers.factual(); // Factual accuracy
scorers.summary(); // Summary quality
scorers.translation(); // Translation accuracy

// Safety and moderation
scorers.moderation(); // Content safety

// Structured data
scorers.jsonDiff(); // JSON comparison
scorers.sql(); // Judge whether two queries are equivalent; does not execute SQL
```

### Custom scorers

Use `createScorer()` for checks specific to your application:

```ts
import { createScorer } from 'viteval';

const orderIdScorer = createScorer<string>({
  name: 'normalized-order-id',
  score: ({ output, expected }) => ({
    score: output.trim().toUpperCase() === expected ? 1 : 0,
  }),
});

evaluate('Order extraction', {
  data: [{ input: 'Please cancel ord-2087.', expected: 'ORD-2087' }],
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0] ?? '',
  scorers: [orderIdScorer],
});
```

### Multiple scorers

Combine scorers to check different properties of the same output. This example
uses LLM judges and embeddings, which require model setup and API credentials:

```ts
evaluate('Support policy answers', {
  data: [
    {
      input: 'Can I return an unopened item after 14 days?',
      expected: 'Unopened items can be returned within 30 days of delivery.',
    },
  ],
  task: async ({ input }) =>
    (
      await generateText({
        model: openai('gpt-4.1-mini'),
        system:
          'Unopened items can be returned within 30 days of delivery. Answer using only this policy.',
        prompt: input,
      })
    ).text,
  scorers: [
    scorers.factual(),
    scorers.answerSimilarity(),
    scorers.moderation(),
  ],
  threshold: 0.8, // All scorers must average >= 0.8
});
```

## Thresholds

Each case must meet the threshold to pass:

```ts
evaluate('Order extraction', {
  data: [{ input: 'Please cancel ord-2087.', expected: 'ORD-2087' }],
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
  threshold: 1,
});
```

For each case, the threshold applies to the configured mean, median, or sum across
its scorer results. Null scores count as zero. Scorer/task failures and timeouts
fail the case and appear in the report. Suite summaries describe those outcomes;
they do not replace the per-case gate.



---
url: https://viteval.dev/guide/migration.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Migrate to v1

V1 requires Node >=26 and uses native Vitest 5 configuration.

## Configuration

Replace the old `viteval.config.ts` configuration layer with `vitest.config.ts`.
Move evaluation settings into Vitest's `test` object:

```ts
import { defineConfig } from 'viteval/config';
import { JsonReporter } from 'viteval/reporters';

export default defineConfig({
  test: {
    include: ['src/**/*.eval.ts'],
    setupFiles: ['./viteval.setup.ts'],
    testTimeout: 100_000,
    reporters: [
      'default',
      new JsonReporter({
        outputFile: '.viteval/results/<id>.json',
      }),
    ],
  },
});
```

Initialize models and providers in the setup file registered with `setupFiles`.
Vitest controls retries, hooks, reporters, and exit status.

## Tasks and scorers

Tasks receive an object: `task: ({ input, context }) => ...`. Expected answers
are supplied to scorers, not to the task. Construct built-in scorers with factory
calls such as `scorers.exactMatch()` and `scorers.levenshtein()`.

Custom scorers return `{ score, metadata? }`. Task output can have a different
type from the dataset's expected value.

Thresholds apply per case to the selected `mean`, `median`, or `sum`. Null scores
count as zero. Empty datasets, empty scorer lists, non-finite thresholds/scores,
and unsupported JSON values fail explicitly.

## Storage and reports

JSON is the default provider. Explicitly use `viteval({ database: 'sqlite' })`
for the optional database provider. See [Providers](https://viteval.dev/guide/providers.md).

Legacy database run tags are a breaking storage change: v1 uses separate `tags`
and `taggings` tables and does not automatically copy `eval_runs.tags` values.
Back up the database and export those tag associations before upgrading, then
reapply them through the v1 tag provider. Initialization does not drop the legacy
column, but the v1 provider does not expose its values. PostgreSQL remains experimental.

Reports carry `schemaVersion: 1`, run identity, final Vitest case state, and
execution diagnostics. The dashboard accepts legacy files that match the
supported structure and reports malformed or incompatible files as warnings.
Use `<id>` or `<run>` in the output filename to preserve watch-run history.
An explicit static output filename is overwritten on each run.

## CLI

`viteval run` delegates evaluation flags and file filters to Vitest. Use
`viteval run --help --expand-help` for native options and `viteval --help` for
project commands. For init, data, and UI commands, `--cwd` selects the working
directory.

Replace the old output-path option with
`new JsonReporter({ outputFile: '.viteval/results/latest.json' })` in the native
config. Omitting `outputFile` writes the completed evaluation JSON to stdout.

`viteval data` now scans dataset and eval modules. Default exports, named exports,
and inline `defineDataset()` calls are supported; generation skips evaluation
tasks and scorers. See [CLI datasets](https://viteval.dev/guide/cli.md#datasets).

## Dashboard

Continue using `viteval ui`; result/dataset files and existing dashboard URLs
do not need conversion. The `createVitevalServer()` launcher API is preserved.



---
url: https://viteval.dev/guide/providers.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# Providers

Viteval is open source and uses local JSON storage by default. You do not need a
hosted evaluation account. With the default JSON provider, datasets live under
`.viteval/datasets` and runs under `.viteval/results`. The JSON reporter also
writes local snapshots for dashboard inspection. Configured hosted providers
such as Braintrust, Langfuse, and VoltOps persist runs through their own services;
they do not move those hosted runs into the local directory.

`@viteval/providers` and every backend SDK/driver are opt-in. Importing a provider
factory does not load its SDK. Install the packages for the backend you select;
initialization errors list any missing packages and how to install them.

Use `@viteval/providers@rc` alongside `viteval@rc` during the prerelease. The
untagged provider package can resolve to an older release candidate.

Choose providers independently for datasets and evaluation results:

| Provider             | Datasets                       | Evaluation results                  |
| -------------------- | ------------------------------ | ----------------------------------- |
| JSON (default)       | Local files                    | Local files                         |
| SQLite               | Local database                 | Local database                      |
| Braintrust           | Hosted datasets                | Experiments and case results        |
| Langfuse             | Create, read, and add items    | Case traces and scores              |
| VoltOps              | Upload versions and read items | Hosted experiments and case results |
| Amazon S3            | Versioned objects              | Use a separate results provider     |
| Google Cloud Storage | Versioned objects              | Use a separate results provider     |

These providers store evaluation data. They do not choose the model used by your
task. Call your model client inside the task function; configure models for
built-in judges and embedding scorers with `initializeModel()`.

## Combine providers

Pass `datasets` and `evals` separately to `initializeProvider()`. For example,
keep datasets in local JSON files and publish results to Braintrust:

```ts
import { fileURLToPath } from 'node:url';
import { createJsonProvider, initializeProvider } from 'viteval/provider';
import { braintrust } from '@viteval/providers/braintrust';

const initialized = await initializeProvider({
  datasets: createJsonProvider({
    root: fileURLToPath(new URL('.', import.meta.url)),
  }),
  evals: braintrust({
    project: 'knowledge-answers',
    apiKey: process.env.BRAINTRUST_API_KEY,
  }),
});
if (!initialized.ok) throw initialized.error;
```

Install `@viteval/providers` and `@braintrust/api` for this example, and register
the module in Vitest's `test.setupFiles`. Keep `JsonReporter` configured to save a
local copy for the dashboard. It reads local JSON reports, not hosted results.

You can replace the JSON dataset provider with S3 or Google Cloud Storage without
changing the results provider. Each integration's supported operations and
limitations are listed below.

## Default JSON setup

`viteval init` creates a setup module equivalent to:

```ts
import { fileURLToPath } from 'node:url';
import { createJsonProvider, initializeProvider } from 'viteval/provider';

const provider = createJsonProvider({
  root: fileURLToPath(new URL('.', import.meta.url)),
});
const initialized = await initializeProvider({ datasets: provider });
if (!initialized.ok) throw initialized.error;
```

Register this file in Vitest's `test.setupFiles`. Binding the root in setup keeps
dataset storage correct when you use `--root` or multiple Vitest projects.
Without a configured dataset provider, local datasets use the current working
directory.

The JSON reporter writes a combined run snapshot. JSON provider operations can
read both those snapshots and runs created through the provider API. Reporter
snapshots are finalized history; provider-created running records accept new results.

## SQLite

Install the optional provider package:

```bash
pnpm add -D @viteval/providers@rc @prisma/client@^7.10 @prisma/adapter-better-sqlite3@^7.10 better-sqlite3
```

Configure it in `viteval.setup.ts`:

```ts
import { afterAll } from 'vitest';
import { fileURLToPath } from 'node:url';
import { initializeProvider } from 'viteval/provider';
import { viteval } from '@viteval/providers/viteval';

const provider = viteval({
  database: 'sqlite',
  path: fileURLToPath(new URL('.viteval/evals.db', import.meta.url)),
});
const initialized = await initializeProvider(provider);
if (!initialized.ok) throw initialized.error;

afterAll(async () => {
  const closed = await provider.close();
  if (!closed.ok) throw closed.error;
});
```

Evaluation suites persist their final case outcomes to the configured evaluation
provider. Keep the JSON reporter configured to inspect a local copy in the
dashboard. SQLite schema/client assets are included in the package; consumers
do not run Prisma generation. `viteval()` with no database option returns JSON.

The PostgreSQL option is experimental and is not covered by v1 provider tests.

## Braintrust

```bash
pnpm add -D @viteval/providers@rc @braintrust/api
```

```ts
import { initializeProvider } from 'viteval/provider';
import { braintrust } from '@viteval/providers/braintrust';

const initialized = await initializeProvider(
  braintrust({
    project: 'answer-quality',
    apiKey: process.env.BRAINTRUST_API_KEY,
  })
);
if (!initialized.ok) throw initialized.error;
```

The provider finds or creates the named project. Datasets use Braintrust's
dataset API; suites create distinct experiments so reruns preserve history.
The provider preserves pagination and configuration metadata. Tests use the
official SDK with local HTTP fixtures for dataset pagination and replacement,
experiment creation, result writes, completion metadata, and result pagination.
They do not use a live Braintrust account. Supply your own credentials to use
the provider with your account.

## Langfuse

```bash
pnpm add -D @viteval/providers@rc @langfuse/client @opentelemetry/api @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http
```

```ts
import { afterAll } from 'vitest';
import { initializeProvider } from 'viteval/provider';
import { langfuse } from '@viteval/providers/langfuse';

const provider = langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY ?? '',
  secretKey: process.env.LANGFUSE_SECRET_KEY ?? '',
  baseUrl: 'https://cloud.langfuse.com',
});
const initialized = await initializeProvider({ evals: provider });
if (!initialized.ok) throw initialized.error;

afterAll(async () => {
  const closed = await provider.close();
  if (!closed.ok) throw closed.error;
});
```

Viteval publishes each case as a separate experiment-item trace and writes scores
through the official Langfuse API client. It uses the v4 OpenTelemetry ingestion
path and v2 Observations API, with an isolated tracer provider that does not
replace your application's global OpenTelemetry configuration. Regional and
self-hosted URLs are supported. Export failures return persistence errors.

Completed run snapshots and paginated case outcomes can be read through the
provider from a fresh process. Datasets support creation, reads, pagination, and
item additions. Local inputs can be correlated as experiment items without
uploading a managed dataset; callers can supply `metadata.datasetItemId` when
writing results directly to link an existing Langfuse item.

The SDK does not expose dataset rename/update or deletion. Atomic item replacement
is not supported. These operations return explicit errors; manage those operations
in Langfuse or create a new dataset.

## VoltOps hosted evaluations

```bash
pnpm add -D @viteval/providers@rc @voltagent/sdk
```

```ts
import { afterAll } from 'vitest';
import { initializeProvider } from 'viteval/provider';
import { voltops } from '@viteval/providers/voltops';

const provider = voltops({
  publicKey: process.env.VOLTAGENT_PUBLIC_KEY ?? '',
  secretKey: process.env.VOLTAGENT_SECRET_KEY ?? '',
  baseUrl: 'https://api.voltagent.dev',
  triggerSource: 'ci',
});
const initialized = await initializeProvider({ evals: provider });
if (!initialized.ok) throw initialized.error;

afterAll(async () => {
  const closed = await provider.close();
  if (!closed.ok) throw closed.error;
});
```

This provider publishes evaluations to VoltOps, including
experiments, registered scorers, batched case outcomes, thresholds, errors, and
completion summaries. Runs are created with `autoQueue: false`: Vitest executes
the task and scorers, while VoltOps stores their outcomes. It does not launch a
second hosted agent execution. It works with any Viteval task, including VoltAgent
agents.

Set `datasetId` to bind runs to a hosted dataset's latest version, or
`experimentId` to bind them to an existing experiment. Otherwise, suite runs
create distinct experiments. Hosted datasets support creation/upload and paginated
reads. Adding items creates a new version using VoltAgent's published upload
routes through the SDK's authenticated transport.

The current VoltOps SDK does not expose historical run retrieval/listing or dataset
rename/update/deletion. `evals.get()` reads runs created by the current provider
session; unavailable historical reads and `evals.list()` return explicit errors.
Inspect persisted runs in the [VoltOps console](https://console.voltagent.dev/).
Atomic dataset replacement is not supported; failed multi-request uploads may
leave a partial new version, while prior versions remain available.

Langfuse and VoltOps tests use official SDKs with local HTTP fixtures, not live
accounts. Keep the JSON reporter configured
for local dashboard inspection alongside hosted persistence.

## S3 and Google Cloud Storage datasets

Object storage providers handle datasets independently of your results backend.
Choose the optional SDK for your bucket:

```bash
# S3
pnpm add -D @viteval/providers@rc @aws-sdk/client-s3

# Google Cloud Storage
pnpm add -D @viteval/providers@rc @google-cloud/storage
```

Configure the dataset provider in your worker setup:

```ts
import { afterAll } from 'vitest';
import { initializeProvider } from 'viteval/provider';
import { s3 } from '@viteval/providers/s3';

const datasets = s3({
  bucket: 'evaluation-data',
  prefix: 'my-project/datasets',
  clientOptions: { region: 'us-east-1' },
});
const initialized = await initializeProvider({ datasets });
if (!initialized.ok) throw initialized.error;

afterAll(async () => {
  const closed = await datasets.close();
  if (!closed.ok) throw closed.error;
});
```

Use Google Cloud Storage with the same setup pattern:

```ts
import { gcs } from '@viteval/providers/gcs';

const datasets = gcs({
  bucket: 'evaluation-data',
  prefix: 'my-project/datasets',
  clientOptions: { projectId: 'my-google-project' },
});
```

Buckets must already exist. `clientOptions` accepts the standard SDK configuration:
AWS credential chains and custom endpoints, or Google Application Default
Credentials and client settings. S3-compatible endpoints can be configured with
`endpoint` and `forcePathStyle`; compatibility requires the service to enforce
conditional writes. AWS/GCS SDK protocols are tested against local HTTP fixtures;
individual S3-compatible services are not separately verified.

Both providers support create, read, list, rename, metadata updates, item additions,
replacement, and deletion from the active catalog. They expose only the dataset
domain. You can combine them with JSON, Braintrust, Langfuse, or VoltOps results.

### Atomic publication and retained versions

The default namespace is `viteval/datasets`. Each namespace contains:

- `index.json`: a validated catalog of dataset metadata and current version pointers.
- `versions/<uuid>.json`: immutable dataset snapshots.

An update uploads its immutable snapshot first, then conditionally publishes the
catalog pointer. S3 uses ETags with `If-Match`/`If-None-Match`; GCS uses generation
preconditions. Conflicting writers reread the catalog and reapply their update.
The catalog records a SHA-256 digest for each snapshot; reads reject rewritten or
corrupted snapshot contents.
`maxAttempts` controls the conflict budget and defaults to 16. Listing pagination
is applied to the catalog, independently of bucket object listing.

A failed snapshot upload or rejected catalog write leaves the previously visible
version intact. Network errors return without automatically replaying writes:
if the server committed a write but its response was lost, read the dataset before
manually retrying the operation. Google downloads retain SDK checksum validation
and capture the generation token from the same response as their content.

Deleting a dataset removes its active catalog entry. Prior snapshots and snapshots
from unsuccessful publication attempts remain in the bucket so readers of an
earlier catalog cannot lose their referenced data. Cleanup must preserve objects
referenced by the current catalog. No automatic garbage collection is performed.

## Serialization and failures

VoltOps dataset creation and `addItems` upload versions through multiple requests.
They are not failure-safe: a failed item upload can leave an incomplete latest
version that subsequent reads return. Errors include the dataset and version IDs;
inspect and repair that version in the VoltOps console before retrying. Previous
versions are retained remotely. The SDK does not expose transactional publication
or version deletion, so atomic replacement remains unsupported.

Braintrust creation attempts to delete a newly created dataset if its metadata
or initial item write fails. If cleanup is also rejected, the error retains the
original failure and identifies the dataset that needs manual cleanup.

- Store JSON-compatible inputs, expected values, outputs, and metadata. Dates serialize to ISO strings.
- Native JSON and SQLite providers normalize undefined inputs, expected values, and outputs to null. Optional object fields remain omitted.
- Non-finite numbers, BigInt, functions, symbols, cycles, and unsupported object classes fail explicitly.
- JSON writes use a temporary file and atomic rename. Serialization failure preserves the previous file.
- Local JSON dataset/tag updates use a cross-process lock. A lock timeout identifies the affected file; if a writer was interrupted, confirm it has stopped before removing its `.lock` directory.
- `storage: 'memory'` datasets regenerate on load and are not written to disk.

Provider operations return `Result` values. Check `ok` before reading `value`;
initialization and persistence errors fail the evaluation run.



---
url: https://viteval.dev/index.md
---

> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

Pause motion

v1.0.0-rc.3

# Evaluate prompts and agentsin TypeScript.

Test extraction, routing, and generated answers against reference cases. Score each output, inspect failures locally, and fail CI when a case falls below your threshold. Built on Vitest.

[Get started](/guide/getting-started)[Why Viteval](/features)

******order-extraction.eval.ts

❯pnpm exec viteval run

RUN **v5.0.3**

src/order-extraction.eval.ts / Order extraction

✓Where is order ORD-1042?5ms

✓Please cancel ord-2087.1ms

✓How do I update my address?1ms

Test Files**1 passed**(1)

Tests**3 passed**(3)

ScorerExactMatch: 1 on each case

Duration355ms

Replay animation

Used by[![Bytefarm](/assets/bytefarm.svg)![Bytefarm](/assets/bytefarm-light.svg)](https://bytefarm.sh/)·[![VoltAgent](/assets/voltagent.svg)](https://voltagent.dev/)·[![](/assets/holly.svg)](https://www.hollygov.com/)·![Joggr](/assets/joggr-dark.svg)![Joggr](/assets/joggr-light.svg)

Why Viteval

## Evals belong beside your tests.

Keep your application code, TypeScript, and Vitest workflow. Viteval adds the evaluation pieces.

### [Built on Vitest](/features#familiar-tooling)

Import your application functions into evaluations. Use Vitest configuration, watch mode, retries, and CI exit codes.

VITEST![](/assets/vitest-mark.svg)

AGENT.EVAL.TS{ ✓ }

Same runner. A new kind of test.

### [Built-in and custom scorers](/features#built-in-defaults)

Check exact answers and structured output, or use embeddings and LLM judges. Set a per-case threshold to fail regressions.

scorersscoring methods

exactMatchdeterministic

answerSimilarityembeddings

yourScorercustom

Built-in and custom scorers

### [Local-first by default](/features#local-first)

Run evaluations and inspect results on your machine. Connect a hosted provider only when you choose.

~/my-agentlocal

$ viteval run

**✓** workspace-invites passed

**✓** api-rate-limits passed

**✓** retention-policy passed

↳ .viteval/results/

On your machine. No hosted account.

### [Run on your infrastructure](/features#run-anywhere)

The same evaluations on your laptop, in CI, or on a runner you control. You choose where the data lives.

LAPTOPCIPRIVATESame evals. Wherever you run.

[View all features →](/features)

Evaluation workflow

## Evaluate order-ID extraction

Check normalization and missing values without an API key.

1. 01WriteStart with your application code.Define support messages and expected order IDs. Check a deterministic extractor first, then run the same cases against a model.
1. 02RunCheck every case.Removing case-insensitive matching breaks the lowercase order ID. The case scores 0 and the command exits with a failure.
1. 03InspectFind out what went wrong.Open saved JSON runs in the local dashboard. Trace a failed score back to its input, expected answer, and actual output.

TSorder-extraction.eval.ts

order-extraction.eval.tsorder-extraction.dataset.tsvitest.config.ts

```ts
import { evaluate, scorers } from 'viteval';
import { orderMessages } from './order-extraction.dataset';

evaluate('Order extraction', {
  data: orderMessages,
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

```ts
import { defineDataset } from 'viteval/dataset';

export const orderMessages = defineDataset({
  name: 'order-extraction',
  data: async () => [
    { input: 'Where is order ORD-1042?', expected: 'ORD-1042' },
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
});
```

```ts
import { defineConfig } from 'viteval/config';
import { JsonReporter } from 'viteval/reporters';

export default defineConfig({
  test: {
    include: ['src/**/*.eval.ts'],
    setupFiles: ['./viteval.setup.ts'],
    reporters: [
      'default',
      new JsonReporter({ outputFile: '.viteval/results/<run>.json' }),
    ],
  },
});
```

order-extraction.eval.ts[Create an evaluation →](/guide/getting-started)

❯pnpm exec viteval run

❯ pnpm exec viteval run

**RUN** v5.0.3

src/order-extraction.eval.ts (3 cases)

✓Where is order ORD-1042?1.00

✓How do I update my address?1.00

×Please cancel ord-2087.0.00

mean score 0 is below threshold 1

Expected**ORD-2087**

Received**Empty string**

2 passed1 failed3 cases

order-extraction.eval.ts[Run evaluations in CI →](/guide/advanced/ci)

❯Order extraction / Please cancel ord-2087.

Order extraction1 failed case

✓ uppercase ID✓ no ID**× lowercase ID**

Input

Please cancel ord-2087.

Expected**ORD-2087**

Output**Empty string**

ExactMatch****0.00**

The case-sensitive pattern misses ord-2087 and returns an empty string. Restore the regular expression’s i flag to match both cases. This case-detail diagram summarizes the captured failure.

Input, output, and score[Explore the local dashboard →](/guide/cli#view-results)

Integrations

## Dataset and result storage

Keep results in local JSON, or connect your preferred storage and evaluation tools.

[![](/assets/provider-braintrust.svg)](/guide/providers#braintrust)·[![Langfuse](/assets/provider-langfuse.svg)](/guide/providers#langfuse)·[![VoltOps](/assets/provider-voltops.svg)](/guide/providers#voltops-hosted-evaluations)·[![](/assets/provider-sqlite.svg)](/guide/providers#sqlite)·[![Amazon S3](/assets/provider-s3.svg)](/guide/providers#s3-and-google-cloud-storage-datasets)·[![Google Cloud Storage](/assets/provider-gcs.svg)](/guide/providers#s3-and-google-cloud-storage-datasets)

[View integrations →](/guide/providers)

## Run your first evaluation.

Open source. No hosted account required.

[Write your first evaluation](/guide/getting-started)