Viteval

VitevalNext Generation Eval Framework

Define, run, and debug LLM evaluations with a familiar API

Get Started
View Examples
Viteval

Powered by Vitest

Define and run evaluations the same way you run tests, with a familiar and intuitive interface

Fully Featured

Run evals, create datasets, and more with everything you need out of the box

CI/CD Ready

Integrate seamlessly into your CI pipeline and run evaluations alongside your tests

See it in action ​

ts
import { evaluate, scorers } from 'viteval';

evaluate('Color detection', {
  data: async () => [
    { input: 'What color is the sky?', expected: 'Blue' },
    { input: 'What color is grass?', expected: 'Green' },
  ],
  task: async (input) => {
    const result = await generateText(input);
    return result.text;
  },
  scorers: [scorers.levenshtein],
  threshold: 0.8,
});

Viteval is free and open source,
made possible by wonderful sponsors.

Special Sponsors

Joggr