For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt, and this page is available as Markdown at https://viteval.dev/index.md.

V1 release candidate available.Migration guide →

v1.0.0-rc.5

Evaluate prompts and agentsin TypeScript.

Run evaluations against real datasets, score outputs, and inspect results. Use your own models and infrastructure. Powered by Vitest.

Get started Introduction

order-extraction.eval.ts

❯pnpm exec viteval run

RUN v5.0.3

src/order-extraction.eval.ts / Order extraction

✓Where is order ORD-1042?5ms

✓Please cancel ord-2087.1ms

✓How do I update my address?1ms

Test Files1 passed(1)

Tests3 passed(3)

ScorerExactMatch: 1 on each case

Duration355ms

Used by

Why Viteval

Your AI stack. Measurable results.

Keep your models and agent frameworks. Choose your scoring criteria, storage, and where evaluations run.

Powered by Vitest

Evaluate your existing AI code with Vitest handling execution, watch mode, and retries.

Measure what matters

Check exact answers and structured output, or use embeddings and LLM judges. Set a per-case threshold to fail regressions.

Store data where it fits

Start with local datasets and results, or connect supported storage and evaluation platforms.

datasets.json

results.json

Local files

Git repo

S3 / GCSDatasets only

Braintrust
Langfuse

Keep control of execution

Run the same evaluations on your laptop, in CI, or in your cloud. You choose the infrastructure.

Evaluation workflow

AI evals, end to end.

Write an eval, run your cases, and inspect failures.

support-routing.eval.ts

import { evaluate, scorers } from 'viteval';
import { classifyTicket } from './support-agent';

evaluate('Support routing', {
  data: [
    { input: 'I was charged twice.', expected: 'billing' },
    { input: 'Reset my password.', expected: 'account' },
    { input: 'Cancel my subscription.', expected: 'billing' },
  ],
  task: ({ input }) => classifyTicket(input),
  scorers: [scorers.exactMatch()],
});

localhost · Viteval

Support routingFailed

support-routing.eval.ts 3 evals

Passed2

Failed1

Total3

Mean Score0.67

Pass Rate66.7%

Results

Name	Output	Mean	Threshold	Result
	billing	1.00	1	Passed
Input: I was charged twice. Expected: billing Output: billing Scores: ExactMatch: 1
	account	1.00	1	Passed
Input: Reset my password. Expected: account Output: account Scores: ExactMatch: 1
	account	0.00	1	Failed
Input: Cancel my subscription. Expected: billing Output: account Scores: ExactMatch: 0

Integrations

Connect the tools you already use.

Load datasets and save results with local files, cloud storage, or supported evaluation platforms.

View integrations

Run your first evaluation.

Open source. No hosted account required.

Write your first evaluation