> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

Pause motion

v1.0.0-rc.3

# Evaluate prompts and agentsin TypeScript.

Test extraction, routing, and generated answers against reference cases. Score each output, inspect failures locally, and fail CI when a case falls below your threshold. Built on Vitest.

[Get started](/guide/getting-started)[Why Viteval](/features)

******order-extraction.eval.ts

❯pnpm exec viteval run

RUN **v5.0.3**

src/order-extraction.eval.ts / Order extraction

✓Where is order ORD-1042?5ms

✓Please cancel ord-2087.1ms

✓How do I update my address?1ms

Test Files**1 passed**(1)

Tests**3 passed**(3)

ScorerExactMatch: 1 on each case

Duration355ms

Replay animation

Used by[![Bytefarm](/assets/bytefarm.svg)![Bytefarm](/assets/bytefarm-light.svg)](https://bytefarm.sh/)·[![VoltAgent](/assets/voltagent.svg)](https://voltagent.dev/)·[![](/assets/holly.svg)](https://www.hollygov.com/)·![Joggr](/assets/joggr-dark.svg)![Joggr](/assets/joggr-light.svg)

Why Viteval

## Evals belong beside your tests.

Keep your application code, TypeScript, and Vitest workflow. Viteval adds the evaluation pieces.

### [Built on Vitest](/features#familiar-tooling)

Import your application functions into evaluations. Use Vitest configuration, watch mode, retries, and CI exit codes.

VITEST![](/assets/vitest-mark.svg)

AGENT.EVAL.TS{ ✓ }

Same runner. A new kind of test.

### [Built-in and custom scorers](/features#built-in-defaults)

Check exact answers and structured output, or use embeddings and LLM judges. Set a per-case threshold to fail regressions.

scorersscoring methods

exactMatchdeterministic

answerSimilarityembeddings

yourScorercustom

Built-in and custom scorers

### [Local-first by default](/features#local-first)

Run evaluations and inspect results on your machine. Connect a hosted provider only when you choose.

~/my-agentlocal

$ viteval run

**✓** workspace-invites passed

**✓** api-rate-limits passed

**✓** retention-policy passed

↳ .viteval/results/

On your machine. No hosted account.

### [Run on your infrastructure](/features#run-anywhere)

The same evaluations on your laptop, in CI, or on a runner you control. You choose where the data lives.

LAPTOPCIPRIVATESame evals. Wherever you run.

[View all features →](/features)

Evaluation workflow

## Evaluate order-ID extraction

Check normalization and missing values without an API key.

1. 01WriteStart with your application code.Define support messages and expected order IDs. Check a deterministic extractor first, then run the same cases against a model.
1. 02RunCheck every case.Removing case-insensitive matching breaks the lowercase order ID. The case scores 0 and the command exits with a failure.
1. 03InspectFind out what went wrong.Open saved JSON runs in the local dashboard. Trace a failed score back to its input, expected answer, and actual output.

TSorder-extraction.eval.ts

order-extraction.eval.tsorder-extraction.dataset.tsvitest.config.ts

```ts
import { evaluate, scorers } from 'viteval';
import { orderMessages } from './order-extraction.dataset';

evaluate('Order extraction', {
  data: orderMessages,
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
});
```

```ts
import { defineDataset } from 'viteval/dataset';

export const orderMessages = defineDataset({
  name: 'order-extraction',
  data: async () => [
    { input: 'Where is order ORD-1042?', expected: 'ORD-1042' },
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
});
```

```ts
import { defineConfig } from 'viteval/config';
import { JsonReporter } from 'viteval/reporters';

export default defineConfig({
  test: {
    include: ['src/**/*.eval.ts'],
    setupFiles: ['./viteval.setup.ts'],
    reporters: [
      'default',
      new JsonReporter({ outputFile: '.viteval/results/<run>.json' }),
    ],
  },
});
```

order-extraction.eval.ts[Create an evaluation →](/guide/getting-started)

❯pnpm exec viteval run

❯ pnpm exec viteval run

**RUN** v5.0.3

src/order-extraction.eval.ts (3 cases)

✓Where is order ORD-1042?1.00

✓How do I update my address?1.00

×Please cancel ord-2087.0.00

mean score 0 is below threshold 1

Expected**ORD-2087**

Received**Empty string**

2 passed1 failed3 cases

order-extraction.eval.ts[Run evaluations in CI →](/guide/advanced/ci)

❯Order extraction / Please cancel ord-2087.

Order extraction1 failed case

✓ uppercase ID✓ no ID**× lowercase ID**

Input

Please cancel ord-2087.

Expected**ORD-2087**

Output**Empty string**

ExactMatch****0.00**

The case-sensitive pattern misses ord-2087 and returns an empty string. Restore the regular expression’s i flag to match both cases. This case-detail diagram summarizes the captured failure.

Input, output, and score[Explore the local dashboard →](/guide/cli#view-results)

Integrations

## Dataset and result storage

Keep results in local JSON, or connect your preferred storage and evaluation tools.

[![](/assets/provider-braintrust.svg)](/guide/providers#braintrust)·[![Langfuse](/assets/provider-langfuse.svg)](/guide/providers#langfuse)·[![VoltOps](/assets/provider-voltops.svg)](/guide/providers#voltops-hosted-evaluations)·[![](/assets/provider-sqlite.svg)](/guide/providers#sqlite)·[![Amazon S3](/assets/provider-s3.svg)](/guide/providers#s3-and-google-cloud-storage-datasets)·[![Google Cloud Storage](/assets/provider-gcs.svg)](/guide/providers#s3-and-google-cloud-storage-datasets)

[View integrations →](/guide/providers)

## Run your first evaluation.

Open source. No hosted account required.

[Write your first evaluation](/guide/getting-started)