---
url: /api/create-scorer.md
---
# `createScorer()`

The `createScorer()` function allows you to create custom scoring functions for evaluating LLM outputs.

## Import

```ts
import { createScorer } from 'viteval';
```

## Signature

```ts
function createScorer<OUTPUT, EXTRA extends Extra = Extra>(
  config: ScorerConfig<OUTPUT, EXTRA>
): Scorer<OUTPUT, EXTRA>;
```

## Parameters

### `config`

* **Type**: `ScorerConfig<OUTPUT, EXTRA>`
* **Required**: Yes
* **Description**: Configuration object for the scorer

## ScorerConfig

```ts
interface ScorerConfig<OUTPUT, EXTRA extends Extra = Extra> {
  name: string;
  score: (
    args: ScorerArgs<OUTPUT, EXTRA>
  ) => Omit<Score, 'name'> | Promise<Omit<Score, 'name'>>;
}
```

### `name`

Unique identifier for the scorer.

**Type**: `string`\
**Required**: Yes

```ts
createScorer({
  name: 'exact-match',
  // ...
});
```

### `score`

Function that calculates the score for an output.

**Type**: `(args: ScorerArgs<OUTPUT, EXTRA>) => Omit<Score, 'name'> | Promise<Omit<Score, 'name'>>`\
**Required**: Yes

The function receives `ScorerArgs` and should return a `Score` object (without the name, as it's automatically added):

```ts
type ScorerArgs<OUTPUT, EXTRA extends Extra> = TF.Merge<
  EXTRA,
  {
    output: OUTPUT;
    expected: OUTPUT;
  }
>;

interface Score {
  score: number | null;
  metadata?: Record<string, unknown>;
}
```

The function should return a score between 0 and 1, where:

* `1.0` = Perfect score
* `0.0` = Complete failure
* `0.5` = Partial success

## Return Value

Returns a `Scorer<OUTPUT, EXTRA>` function that can be used in evaluations:

```ts
type Scorer<OUTPUT, EXTRA extends Extra = Extra> = (
  args: ScorerArgs<OUTPUT, EXTRA>
) => Score | Promise<Score>;
```

---

---
url: /api/configuration.md
---
# `defineConfig()`

Viteval provides flexible configuration options to customize evaluation behavior, performance, and output.

## Import

```ts
import { defineConfig } from 'viteval/config';
```

## Configuration File

Create a `viteval.config.ts` file in your project root:

```ts
// viteval.config.ts
import { defineConfig } from 'viteval/config';

export default defineConfig({
  reporter: 'default',
  eval: {
    include: ['src/**/*.eval.ts'],
    setupFiles: ['./viteval.setup.ts'],
    timeout: 100000, // 100 seconds (default)
  },
});
```

## Configuration Options

### Provider Configuration

#### `provider`

Configure API providers for LLM access:

```ts
export default defineConfig({
  provider: {
    openai: {
      apiKey: process.env.OPENAI_API_KEY,
      project: 'my-project',
      organization: 'my-org',
    },
  },
});
```

Or provide a client directly:

```ts
import { OpenAI } from 'openai';

export default defineConfig({
  provider: {
    openai: {
      client: new OpenAI({ apiKey: process.env.OPENAI_API_KEY }),
    },
  },
});
```

### Evaluation Configuration

#### `eval.include`

* **Type:** `string[]`
* **Default:** `undefined`

Glob patterns to include evaluation files:

```ts
export default defineConfig({
  eval: {
    include: ['src/**/*.eval.ts', 'tests/**/*.eval.ts'],
  },
});
```

#### `eval.exclude`

* **Type:** `string[]`
* **Default:** `undefined`

Glob patterns to exclude:

```ts
export default defineConfig({
  eval: {
    exclude: ['**/*.draft.eval.ts', 'src/experimental/**'],
  },
});
```

#### `eval.setupFiles`

* **Type:** `string[]`
* **Default:** `undefined`

Setup files to run before evaluations:

```ts
export default defineConfig({
  eval: {
    setupFiles: ['./viteval.setup.ts'],
  },
});
```

#### `eval.timeout`

* **Type:** `number`
* **Default:** `100000` (100 seconds)

Timeout for individual evaluations (in milliseconds):

```ts
export default defineConfig({
  eval: {
    timeout: 300000, // 5 minutes
  },
});
```

### Server Configuration

#### `server.sourcemap`

* **Type:** `boolean | 'inline'`
* **Default:** `undefined`

Configure sourcemap generation:

```ts
export default defineConfig({
  server: {
    sourcemap: true,
  },
});
```

#### `server.deps`

Configure dependency handling:

```ts
export default defineConfig({
  server: {
    deps: {
      inline: ['some-package'],
      external: ['external-package'],
    },
  },
});
```

### Dependencies Configuration

#### `deps.interopDefault`

* **Type:** `boolean`
* **Default:** `true`

Whether to use the interopDefault plugin:

```ts
export default defineConfig({
  deps: {
    interopDefault: false,
  },
});
```

#### `deps.optimizer`

Configure dependency optimization:

```ts
export default defineConfig({
  deps: {
    optimizer: {
      include: ['some-package'],
    },
  },
});
```

---

---
url: /api/define-dataset.md
---
# `defineDataset()`

The `defineDataset()` function creates reusable datasets that can be shared across multiple evaluations.

## Import

```ts
import { defineDataset } from 'viteval/dataset';
```

## Signature

```ts
function defineDataset<DATA_FUNC extends DataGenerator>(
  config: DatasetConfig<DATA_FUNC>
): Dataset<DATA_FUNC>;
```

## Parameters

### `config`

* **Type**: `DatasetConfig<DATA_FUNC>`
* **Required**: Yes
* **Description**: Configuration object for the dataset

## DatasetConfig

```ts
interface DatasetConfig<DATA extends DataGenerator = DataGenerator> {
  /**
   * The storage type of the dataset.
   *
   * @default 'local'
   */
  storage?: DatasetStorage;
  /**
   * The name of the dataset.
   */
  name: string;
  /**
   * The description of the dataset.
   */
  description?: string;
  /**
   * The data generator of the dataset.
   */
  data: DATA;
}
```

### `name`

Unique identifier for the dataset.

**Type**: `string`\
**Required**: Yes

```ts
defineDataset({
  name: 'math-problems',
  // ...
});
```

### `data`

Function that generates or loads the dataset.

**Type**: `() => Promise<DataItem<INPUT, OUTPUT, EXTRA>[]>`\
**Required**: Yes

```ts
// Static data
data: () => [
  { input: '2 + 2', expected: '4' },
  { input: '3 + 3', expected: '6' },
];

// Async data loading
data: async () => {
  const response = await fetch('/api/datasets/math');
  return response.json();
};

// Generated data
data: async () => {
  const problems = [];
  for (let i = 0; i < 1000; i++) {
    const a = Math.floor(Math.random() * 100);
    const b = Math.floor(Math.random() * 100);
    problems.push({
      input: `${a} + ${b}`,
      expected: String(a + b),
    });
  }
  return problems;
};
```

### `storage` (optional)

The storage type for the dataset.

**Type**: `DatasetStorage`\
**Default**: `'local'`

```ts
// Local file storage (default)
storage: 'local';

// Memory storage (not persisted)
storage: 'memory';
```

### `description` (optional)

A description of the dataset.

**Type**: `string`

```ts
description: 'Math problems for basic arithmetic evaluation';
```

## DataItem Interface

The data function should return an array of `DataItem` objects:

```ts
type DataItem<
  INPUT = unknown,
  OUTPUT = unknown,
  EXTRA extends Extra = Extra,
> = TF.Merge<
  EXTRA,
  {
    name?: string;
    input: INPUT;
    expected?: OUTPUT;
  }
>;
```

Where `Extra` is:

```ts
type Extra = Record<string, unknown>;
```

## Return Value

Returns a `Dataset<DATA_FUNC>` object:

```ts
type Dataset<DATA_FUNC extends DataGenerator> = TF.SetRequired<
  DatasetConfig<DATA_FUNC>,
  'storage' | 'data'
> & {
  data: DatasetGenerator<DATA_FUNC>;
};
```

The `data` property is a function that can be called with optional configuration:

```ts
type DatasetGenerator<DATA_FUNC extends DataGenerator> = (
  config?: DatasetGeneratorConfig
) => Promise<Awaited<ReturnType<DATA_FUNC>>>;

type DatasetGeneratorConfig = {
  /**
   * Whether to overwrite the dataset if it already exists.
   *
   * @default false
   */
  overwrite?: boolean;
};
```

---

---
url: /api/evaluate.md
---
# `evaluate()`

The `evaluate()` function is the core API for defining and running LLM evaluations using Vitest.

## Import

```ts
import { evaluate } from 'viteval';
```

## Signature

```ts
function evaluate<DATA_ITEM extends DataItem, DATA extends Data<DATA_ITEM>>(
  name: string,
  {
    data,
    aggregation = 'mean',
    task,
    scorers,
    threshold = 1.0,
    timeout,
  }: Eval<DATA>
);
```

## Parameters

### `name`

* **Type**: `string`
* **Required**: Yes
* **Description**: A human-readable name for the evaluation

### `options`

* **Type**: `Eval<DATA>`
* **Required**: Yes
* **Description**: Configuration object for the evaluation

## Eval Interface

```ts
interface Eval<DATA extends Data> {
  /**
   * The description of the evaluation.
   */
  description?: string;
  /**
   * The data to use for the evaluation.
   */
  data: DATA;
  /**
   * The task to evaluate.
   */
  task: Task<InferDataInput<DATA>, InferDataOutput<DATA>, InferDataExtra<DATA>>;
  /**
   * The scorers to use for the evaluation.
   */
  scorers: Scorer<InferDataOutput<DATA>, InferDataExtra<DATA>>[];
  /**
   * The aggregation type for the evaluation.
   *
   * @default 'mean'
   */
  aggregation?: ScorerAggregationType;
  /**
   * The threshold for the evaluation.
   *
   * @default 1.0
   */
  threshold?: number;
  /**
   * The timeout for the evaluation.
   *
   * @default Value provided to `timeout`, or if not supplied, the runtime configuration value (`config.eval?.timeout`), or 25000ms if none provided.
   */
  timeout?: number;
}
```

### `data`

The data to use for the evaluation. Can be an array, function, or dataset.

**Type**: `Data<DATA_ITEM>`

```ts
type Data<DATA_ITEM extends DataItem> =
  | DATA_ITEM[]
  | DataGenerator<DATA_ITEM>
  | Dataset<DataGenerator<DATA_ITEM>>;
```

```ts
// Inline data
data: [
  { input: 'What is 2+2?', expected: '4' },
  { input: 'What is 3+3?', expected: '6' },
];

// From function
data: async () => {
  const response = await fetch('/api/test-cases');
  return response.json();
};

// From dataset
data: mathDataset;
```

### `task`

Function that processes input and returns the model's output.

**Type**: `Task<InferDataInput<DATA>, InferDataOutput<DATA>, InferDataExtra<DATA>>`

```ts
type Task<INPUT, OUTPUT, EXTRA extends Extra> = (
  args: TaskArgs<INPUT, EXTRA>
) => Promise<OUTPUT> | OUTPUT;

type TaskArgs<INPUT, EXTRA extends Extra> = TF.Merge<
  EXTRA,
  {
    input: INPUT;
  }
>;
```

```ts
// Simple text generation
task: async ({ input }) => {
  const result = await generateText({
    model: 'gpt-4',
    prompt: input,
  });
  return result.text;
};

// With chat messages
task: async ({ input }) => {
  const result = await generateText({
    model: 'gpt-4',
    messages: [
      { role: 'system', content: 'You are a math tutor.' },
      { role: 'user', content: input },
    ],
  });
  return result.text;
};

// Structured input
task: async ({ input, context }) => {
  return await answerQuestion(input.question, context);
};
```

### `scorers`

Array of scorer functions to evaluate the output quality.

**Type**: `Scorer<InferDataOutput<DATA>, InferDataExtra<DATA>>[]`

```ts
import { scorers, createScorer } from 'viteval';

// Built-in scorers
scorers: [scorers.exactMatch];
scorers: [scorers.levenshtein, scorers.factual];

// Custom scorer
const customScorer = createScorer({
  name: 'length-check',
  score: ({ output, expected }) => {
    return {
      score: output.length === expected?.length ? 1 : 0,
      metadata: { method: 'length_comparison' },
    };
  },
});

scorers: [customScorer, scorers.answerSimilarity];
```

### `threshold` (optional)

Minimum average score required for the evaluation to pass.

**Type**: `number`\
**Default**: `1.0`\
**Range**: `0.0` to `1.0`

```ts
// Require 90% average score
threshold: 0.9;

// More lenient threshold
threshold: 0.6;

// Perfect scores only (default)
threshold: 1.0;
```

### `timeout` (optional)

Maximum time (in milliseconds) for each test case.

**Type**: `number`\
**Default**: Value provided to `timeout`, or if not supplied, the runtime configuration value (`config.eval?.timeout`), or 25000ms if none provided.

```ts
// 1 minute timeout
timeout: 60000;

// Quick timeout for fast models
timeout: 5000;

// No timeout (not recommended)
timeout: 0;
```

### `aggregation` (optional)

How to aggregate scores across multiple scorers.

**Type**: `ScorerAggregationType`\
**Default**: `'mean'`\
**Options**: `'mean' | 'median' | 'sum'`

```ts
// Use mean score (default)
aggregation: 'mean';

// Use median score
aggregation: 'median';

// Use sum of scores
aggregation: 'sum';
```

### `description` (optional)

Human-readable description of the evaluation.

**Type**: `string`

```ts
description: 'Evaluates math problem solving capabilities';
```

## DataItem Interface

The data should contain `DataItem` objects:

```ts
type DataItem<
  INPUT = unknown,
  OUTPUT = unknown,
  EXTRA extends Extra = Extra,
> = TF.Merge<
  EXTRA,
  {
    name?: string;
    input: INPUT;
    expected?: OUTPUT;
  }
>;
```

Where `Extra` is:

```ts
type Extra = Record<string, unknown>;
```

### Simple Test Cases

```ts
{ input: "Hello", expected: "Hi there!" }
```

### Complex Input Types

```ts
// Object input
{
  input: {
    question: "What's the weather?",
    location: "San Francisco"
  },
  expected: "Sunny, 72°F"
}

// Array input
{
  input: ["apple", "banana", "cherry"],
  expected: "apple, banana, cherry"
}
```

### Test Case Metadata

```ts
{
  input: "Translate: Hello",
  expected: "Hola",
  difficulty: "easy",
  language: "spanish"
}
```

### Named Test Cases

```ts
{
  name: "Basic addition",
  input: "What is 2+2?",
  expected: "4"
}
```

---

---
url: /api/scorers.md
---
# `scorers`

Viteval provides a comprehensive set of pre-built scorers for common evaluation scenarios, powered by the `autoevals` library.

## Import

```ts
import { scorers } from 'viteval';
```

## Text Similarity

### `exactMatch`

Returns 1 if output exactly matches expected, 0 otherwise.

```ts
scorers.exactMatch;
```

**Use cases**: Exact answers, structured outputs, code generation

**Example**:

```ts
// Input: "What is 2+2?"
// Output: "4"
// Expected: "4"
// Score: 1.0

// Output: "Four"
// Expected: "4"
// Score: 0.0
```

### `levenshtein`

Measures text similarity using edit distance algorithm.

```ts
scorers.levenshtein;
```

**Range**: 0.0 - 1.0 (higher = more similar)\
**Use cases**: Approximate text matching, typo tolerance

**Example**:

```ts
// Output: "The sky is blue"
// Expected: "The sky is bleu"
// Score: ~0.92 (small typo)

// Output: "Blue sky"
// Expected: "The sky is blue"
// Score: ~0.66 (partial match)
```

### `answerSimilarity`

Semantic similarity using embeddings.

```ts
scorers.answerSimilarity;
```

**Range**: 0.0 - 1.0\
**Use cases**: Meaning-based comparison, paraphrasing

**Example**:

```ts
// Output: "The capital of France is Paris"
// Expected: "Paris is France's capital city"
// Score: ~0.95 (same meaning, different wording)
```

### `embeddingSimilarity`

Measures semantic similarity using embeddings.

```ts
scorers.embeddingSimilarity;
```

**Range**: 0.0 - 1.0\
**Use cases**: Semantic search, content matching

**Features**:

* Uses state-of-the-art embedding models
* Captures semantic meaning beyond keywords
* Language-agnostic comparison

## Content Quality

### `factual`

Evaluates factual accuracy against ground truth.

```ts
scorers.factual;
```

**Range**: 0.0 - 1.0\
**Use cases**: Knowledge-based QA, educational content

**Example**:

```ts
// Output: "Paris is the capital of France"
// Expected: "Paris"
// Score: 1.0 (factually correct)

// Output: "London is the capital of France"
// Expected: "Paris"
// Score: 0.0 (factually incorrect)
```

### `summary`

Evaluates summary quality and completeness.

```ts
scorers.summary;
```

**Range**: 0.0 - 1.0\
**Use cases**: Text summarization, content condensation

**Criteria**:

* Relevance to source material
* Completeness of key points
* Conciseness
* Accuracy

### `translation`

Assesses translation accuracy between languages.

```ts
scorers.translation;
```

**Range**: 0.0 - 1.0\
**Use cases**: Machine translation, multilingual content

**Criteria**:

* Meaning preservation
* Grammar correctness
* Cultural appropriateness
* Fluency

## Answer Quality

### `answerCorrectness`

Measures how correct an answer is compared to expected.

```ts
scorers.answerCorrectness;
```

**Range**: 0.0 - 1.0\
**Use cases**: QA systems, educational assessments

### `answerRelevancy`

Evaluates how relevant the answer is to the question.

```ts
scorers.answerRelevancy;
```

**Range**: 0.0 - 1.0\
**Use cases**: Chat systems, search results

**Example**:

```ts
// Question: "What's the weather like?"
// Output: "It's sunny and 75°F"
// Score: 1.0 (highly relevant)

// Output: "I like pizza"
// Score: 0.0 (not relevant)
```

## Context Evaluation

### `contextEntityRecall`

Checks if all expected entities are mentioned in context.

```ts
scorers.contextEntityRecall;
```

**Range**: 0.0 - 1.0\
**Use cases**: Information extraction, entity recognition

### `contextPrecision`

Measures precision of context usage in responses.

```ts
scorers.contextPrecision;
```

**Range**: 0.0 - 1.0\
**Use cases**: RAG systems, context-aware generation

### `contextRecall`

Evaluates how well context information is recalled.

```ts
scorers.contextRecall;
```

**Range**: 0.0 - 1.0\
**Use cases**: Reading comprehension, context utilization

### `contextRelevancy`

Assesses relevance of provided context to the task.

```ts
scorers.contextRelevancy;
```

**Range**: 0.0 - 1.0\
**Use cases**: Context filtering, relevance ranking

## Safety and Moderation

### `moderation`

Detects harmful, inappropriate, or unsafe content.

```ts
scorers.moderation;
```

**Range**: 0.0 - 1.0 (1.0 = safe, 0.0 = unsafe)\
**Use cases**: Content filtering, safety checks

**Categories detected**:

* Hate speech
* Violence
* Self-harm
* Sexual content
* Harassment

**Example**:

```ts
// Output: "Here's a helpful recipe for cookies"
// Score: 1.0 (safe content)

// Output: "Instructions for harmful activities"
// Score: 0.0 (unsafe content)
```

## Structured Data

### `jsonDiff`

Compares JSON structures for differences.

```ts
scorers.jsonDiff;
```

**Range**: 0.0 - 1.0\
**Use cases**: API responses, structured output validation

**Example**:

```ts
// Output: '{"name": "John", "age": 30}'
// Expected: '{"name": "John", "age": 30}'
// Score: 1.0 (exact match)

// Output: '{"name": "John"}'
// Expected: '{"name": "John", "age": 30}'
// Score: 0.5 (partial match)
```

### `sql`

Validates SQL query syntax and correctness.

```ts
scorers.sql;
```

**Range**: 0.0 - 1.0\
**Use cases**: SQL generation, query validation

**Criteria**:

* Syntax correctness
* Logical structure
* Expected results

## Numeric and List Comparison

### `numericDiff`

Calculates numerical difference between outputs.

```ts
scorers.numericDiff;
```

**Range**: 0.0 - 1.0\
**Use cases**: Mathematical calculations, numeric predictions

**Example**:

```ts
// Output: "42"
// Expected: "40"
// Score: ~0.95 (small difference)

// Output: "100"
// Expected: "40"
// Score: ~0.0 (large difference)
```

### `listContains`

Verifies if a list contains expected items.

```ts
scorers.listContains;
```

**Range**: 0.0 - 1.0\
**Use cases**: List generation, item retrieval

**Example**:

```ts
// Output: ["apple", "banana", "cherry"]
// Expected: ["apple", "banana"]
// Score: 1.0 (contains all expected items)

// Output: ["apple"]
// Expected: ["apple", "banana"]
// Score: 0.5 (contains half of expected items)
```

## Specialized Scorers

### `possible`

Checks if an answer is logically possible.

```ts
scorers.possible;
```

**Range**: 0.0 - 1.0\
**Use cases**: Logical reasoning, plausibility checking

**Example**:

```ts
// Question: "How many wheels does a bicycle have?"
// Output: "2"
// Score: 1.0 (possible)

// Output: "7"
// Score: 0.0 (not possible)
```

### `humor`

Evaluates humor quality and appropriateness.

```ts
scorers.humor;
```

**Range**: 0.0 - 1.0\
**Use cases**: Creative writing, entertainment content

**Criteria**:

* Humor presence
* Appropriateness
* Cleverness
* Timing

---

---
url: /guide/advanced/ci.md
---
# CI Integration

Learn how to integrate Viteval into your continuous integration and deployment pipelines to ensure consistent LLM performance.

## Overview

Viteval is designed to work seamlessly in CI/CD environments, allowing you to:

* Run evaluations automatically on code changes
* Gate deployments based on evaluation results
* Track performance regressions over time
* Generate evaluation reports

## GitHub Actions

You can use Viteval in your GitHub Actions workflows to run evaluations on code changes.

::: tip

We highly recommend restricting `paths` to only the files that changed (so you don't run evals on non-impactful changes).

:::

Here's an example workflow:

```yaml
name: LLM Evaluation

on:
  push:
    branches: [main, develop]
    # HIGHLY recommend restricting paths to only the files that changed (so you don't run evals on non-impactful changes)
    paths:
      - apps/api/src/ai/**/*.ts
      - apps/api/src/lib/ai/**/*.ts
  pull_request:
    branches: [main]
    # HIGHLY recommend restricting paths to only the files that changed (so you don't run evals on non-impactful changes)
    paths:
      - apps/api/src/ai/**/*.ts
      - apps/api/src/lib/ai/**/*.ts
jobs:
  evaluate:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run evaluations
        run: npx viteval run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

## GitLab CI/CD

Coming soon!

## CircleCI

Coming soon!

---

---
url: /guide/cli.md
---
# CLI Usage

The Viteval CLI provides powerful commands for running evaluations, managing datasets, and more.

## Commands

### Run Evaluations

```bash
# Run all evaluations
viteval

# Run specific files
viteval chat.eval.ts

# Run with glob patterns
viteval "**/*.eval.ts"
```

### Datasets

Easily create and manage datasets for your evaluations.

```bash
# Generate/update all datasets
viteval data

# Generate specific dataset
viteval data --name "user-questions"

# List available datasets
viteval data --list
```

### Debugging

```bash
# Verbose output with stack traces
viteval --verbose
```

---

---
url: /guide/concepts.md
---
# Core Concepts

Understanding these core concepts will help you make the most of Viteval.

## Evaluations

An evaluation is the fundamental unit in Viteval. It defines:

* **Data**: The input-expected output pairs to test
* **Task**: The function that generates output from input
* **Scorers**: How to measure the quality of outputs
* **Threshold**: The minimum score required to pass

```ts
import { evaluate, scorers } from 'viteval';

evaluate('My Evaluation', {
  data: async () => [
    /* test cases */
  ],
  task: async (input) => {
    /* your LLM call */
  },
  scorers: [scorers.exactMatch],
  threshold: 0.9,
});
```

## Data and Datasets

### Inline Data

For simple evaluations, define data directly:

```ts
evaluate('Simple eval', {
  data: async () => [
    { input: 'Hello', expected: 'Hi there!' },
    { input: 'Goodbye', expected: 'See you later!' },
  ],
  // ...
});
```

### Local Datasets

For reusable datasets, use `defineDataset()`:

```ts
import { defineDataset } from 'viteval/dataset';

const greetings = defineDataset({
  name: 'greetings',
  data: async () => {
    // Generate or load data
    return [
      { input: 'Hello', expected: 'Hi there!' },
      { input: 'Goodbye', expected: 'See you later!' },
    ];
  },
});

// Use in evaluations
evaluate('Greeting test', {
  data: () => greetings.data(),
  // ...
});
```

### Remote Datasets

Coming soon!

## Tasks

The task function is where your LLM logic lives. It receives input and should return the model's output:

```ts
// Simple text generation
task: async (input) => {
  const result = await generateText({
    model: 'gpt-4',
    prompt: input,
  });
  return result.text;
};

// With system prompts
task: async (input) => {
  const result = await generateText({
    model: 'gpt-4',
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: input },
    ],
  });
  return result.text;
};

// Structured output
task: async (input) => {
  const result = await generateObject({
    model: 'gpt-4',
    prompt: input,
    schema: z.object({
      answer: z.string(),
      confidence: z.number(),
    }),
  });
  return JSON.stringify(result.object);
};
```

## Scorers

Scorers measure the quality of your model's output against the expected result.

### Built-in Scorers

Viteval provides many pre-built scorers:

```ts
import { scorers } from 'viteval';

// Text similarity
scorers.levenshtein; // Edit distance
scorers.exactMatch; // Exact string match
scorers.answerSimilarity; // Semantic similarity

// Content quality
scorers.factual; // Factual accuracy
scorers.summary; // Summary quality
scorers.translation; // Translation accuracy

// Safety and moderation
scorers.moderation; // Content safety

// Structured data
scorers.jsonDiff; // JSON comparison
scorers.sql; // SQL validation
```

### Custom Scorers

Create custom scorers for specific needs:

```ts
import { createScorer } from 'viteval';

const lengthScorer = createScorer({
  name: 'length-check',
  score: ({ output, expected }) => {
    const outputLength = output.length;
    const expectedLength = expected.length;
    const diff = Math.abs(outputLength - expectedLength);
    return Math.max(0, 1 - diff / Math.max(outputLength, expectedLength));
  },
});

// Use in evaluations
evaluate('Length test', {
  // ...
  scorers: [lengthScorer],
});
```

### Multiple Scorers

Combine multiple scorers for comprehensive evaluation:

```ts
evaluate('Comprehensive test', {
  // ...
  scorers: [
    scorers.factual, // Is it factually correct?
    scorers.answerSimilarity, // Is it semantically similar?
    scorers.moderation, // Is it safe content?
    customScorer, // Custom business logic
  ],
  threshold: 0.8, // All scorers must average >= 0.8
});
```

## Thresholds

Thresholds determine when an evaluation passes or fails:

```ts
evaluate('My test', {
  // ...
  threshold: 0.8, // Require 80% average score across all scorers
});
```

For multiple scorers, the threshold applies to the average score across all scorers.

---

---
url: /examples.md
---
# Examples

This section provides practical examples of using Viteval for different scenarios.

## Basic Text Evaluation

```ts
import { evaluate, scorers } from 'viteval';

evaluate('Simple QA', {
  data: async () => [
    { input: 'What is the capital of France?', expected: 'Paris' },
    { input: 'What is 2+2?', expected: '4' },
  ],
  task: async (input) => {
    return await callYourLLM(input);
  },
  scorers: [scorers.levenshtein],
  threshold: 0.8,
});
```

## Custom Dataset

```ts
import { defineDataset } from 'viteval/dataset';

const mathDataset = defineDataset({
  name: 'basic-math',
  data: async () => {
    const problems = [];
    for (let i = 0; i < 100; i++) {
      const a = Math.floor(Math.random() * 10);
      const b = Math.floor(Math.random() * 10);
      problems.push({
        input: `What is ${a} + ${b}?`,
        expected: String(a + b),
      });
    }
    return problems;
  },
});

evaluate('Math solver', {
  data: () => mathDataset.data(),
  task: async (input) => await solveMath(input),
  scorers: [scorers.exactMatch],
  threshold: 0.9,
});
```

## Multiple Scorers

```ts
evaluate('Content quality', {
  data: async () => loadQuestions(),
  task: async (input) => await generateContent(input),
  scorers: [
    scorers.factual, // Must be factually correct
    scorers.answerSimilarity, // Must be semantically similar
    scorers.moderation, // Must be safe content
  ],
  threshold: 0.8, // an average of the scores is used to determine if threshold is met
});
```

## Runnable Examples

There are a number of examples available in the [Viteval GitHub repository](https://github.com/viteval/viteval/tree/main/examples).

[**Basic Evaluation**](https://github.com/viteval/viteval/tree/main/examples/basic)
Basic evaluation with Viteval

[**Vercel AI SDK**](https://github.com/viteval/viteval/tree/main/examples/vercel-ai)
Real-world evaluation with Vercel AI SDK

[**Voltagent**](https://github.com/viteval/viteval/tree/main/examples/voltagent)
Real-world evaluation with Voltagent

---

---
url: /guide/getting-started.md
---
# Getting Started

Welcome to Viteval! This guide will help you get up and running with the next generation LLM evaluation framework.

## What is Viteval?

Viteval is an LLM evaluation framework powered by Vitest that makes it easy to:

* Define and run evaluations with a familiar, Vitest-like API
* Create and manage datasets locally or remotely
* Use built-in scorers or create custom ones
* Integrate evaluations into your CI/CD pipeline

## Installation

Install Viteval using your preferred package manager:

::: code-group

```bash [npm]
npm install viteval
```

```bash [pnpm]
pnpm add viteval
```

```bash [yarn]
yarn add viteval
```

:::

## Initialize a Viteval Project

::: code-group

```bash [npm]
npx viteval init
```

```bash [pnpm]
pnpm viteval init
```

```bash [yarn]
yarn viteval init
```

:::

This will create a `viteval.config.ts` && `viteval.setup.ts` (so you can configure your environment variables) file in your project root.

## Your First Evaluation

Let's create a simple evaluation to test color detection. Create a file called `color.eval.ts`:

```ts
import { evaluate, scorers } from 'viteval';
import { generateText } from 'ai'; // or your preferred LLM library

evaluate('Color detection', {
  data: async () => [
    { input: 'What color is the sky?', expected: 'Blue' },
    { input: 'What color is grass?', expected: 'Green' },
    { input: 'What color is snow?', expected: 'White' },
  ],
  task: async (input) => {
    const result = await generateText({
      model: 'gpt-4', // Configure your model here
      prompt: input,
    });
    return result.text;
  },
  scorers: [scorers.levenshtein],
  threshold: 0.8,
});
```

### Running Your Evaluation

Run your evaluation using the Viteval CLI:

```bash
npx viteval
```

This will:

1. Discover all `*.eval.ts` files in your project
2. Execute each evaluation
3. Display results with scoring details
4. Exit with a non-zero code if any evaluation fails

### Understanding the Results

Viteval will output something like:

```
✓ Color detection (3/3 passed)
  ✓ What color is the sky? → Blue (score: 1.0)
  ✓ What color is grass? → Green (score: 1.0)
  ✓ What color is snow? → White (score: 1.0)

Evaluations: 1 passed, 0 failed
```

## Your First Dataset

Now that you've run your first evaluation, let's create a dataset, that you can use in the evaluation.

```ts
import { defineDataset } from 'viteval/dataset';

export default defineDataset({
  name: 'color-dataset',
  data: async () => [
    { input: 'What color is the sky?', expected: 'Blue' },
    { input: 'What color is grass?', expected: 'Green' },
    { input: 'What color is snow?', expected: 'White' },
  ],
});
```

### Adding the Dataset to the Evaluation

You can then use the dataset in the evaluation by importing it and passing it to the `data` option.

```ts
import dataset from './color.dataset';

evaluate('Color detection', {
  data: dataset,
  task: async (input) => {
    const result = await generateText({
      model: 'gpt-4', // Configure your model here
      prompt: input,
    });
    return result.text;
  },
  scorers: [scorers.levenshtein],
  threshold: 0.8,
});
```

### Running the Evaluation with the Dataset

Now you can run the evaluation again, and it will use the dataset.

```bash
npx viteval
```

If you configured the dataset to be stored locally, it will be stored in the `./viteval/datasets` directory relative to the config file.

## Next Steps

Now that you have your first evaluation running:

* [Learn about core concepts](/guide/concepts) like datasets, scorers, and tasks
* [Explore the CLI options](/guide/cli) for different use cases
* [Check out examples](/examples/) for more complex scenarios
* [Set up CI integration](/guide/advanced/ci) to run evaluations automatically

---

---
url: /guide/advanced/reporters.md
---
# Reporters

You can use provided reporters to get the results of your evals in a variety of formats.

## Built-in Reporters

### `default`

The default reporter will print the results to the console.

### `json`

The json reporter will print the results to a JSON file. You can pass this in the CLI or in your config file.

```ts
export default defineConfig({
  reporters: ['json'],
});
```

```sh
viteval run --reporters json
```

### `file`

The file reporter will print the results to a file. You can pass this in the CLI or in your config file.

```ts
export default defineConfig({
  reporters: ['file'],
});
```

```sh
viteval run --reporters file
```

## Multiple Reporters

You can pass multiple reporters to the CLI or in your config file. This allows you to combine the default reporter with the file reporter to get the results in both formats.

```ts
export default defineConfig({
  reporters: ['default', 'file'],
});
```

```sh
viteval run --reporters default file
```

## Custom Reporters

> \[!CAUTION]
> Custom reporters are not yet supported, if interested please let us know by [creating an issue](https://github.com/viteval/viteval/issues/new).