For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt, and this page is available as Markdown at https://viteval.dev/guide/models.md.
V1 release candidate available.Migration guide →
GitHub

Configure models

Configure scorer models in viteval.config.ts. Choose your task model separately inside the task function. No worker setup module is required.

Prerequisites

  • Complete Getting started and keep a passing evaluation.
  • Choose a model provider and have its API credentials available.
  • This walkthrough uses OpenAI through the AI SDK.

Steps

1

Install a model provider

This example uses OpenAI through the AI SDK:

pnpm add ai @ai-sdk/openai

Provide OPENAI_API_KEY in the process environment before running evaluations. Installing the SDK does not load credentials from an env file automatically.

2

Register scorer models

Add raw AI SDK providers and default model references to viteval.config.ts:

import { openai } from '@ai-sdk/openai';
import { defineConfig } from 'viteval/config';

export default defineConfig({
  models: {
    providers: [openai],
    defaults: {
      language: 'openai:gpt-4.1-mini',
      embedding: 'openai:text-embedding-3-small',
    },
  },
});

Use provider:model-id references for defaults. Deterministic scorers need no model configuration. Registration does not make network requests.

3

Choose the task model

The default language model is the judge, not the task model. Call the client you want to evaluate inside the task. Replace src/viteval.eval.ts with:

import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';
import { evaluate, scorers } from 'viteval';

evaluate('Support policy answers', {
  data: [
    {
      input: 'Can I return an unopened item after 14 days?',
      expected: 'Unopened items can be returned within 30 days of delivery.',
    },
  ],
  task: async ({ input }) => {
    const result = await generateText({
      model: openai('gpt-4.1-mini'),
      system: 'Unopened items can be returned within 30 days of delivery.',
      prompt: input,
    });
    return result.text;
  },
  scorers: [scorers.factual(), scorers.answerSimilarity()],
  threshold: 0.8,
});
4

Override a judge when needed

Pass model to an LLM judge factory to use a different judge for that scorer. Embedding factories do not support per-scorer model overrides.

const factual = scorers.factual({ model: openai('gpt-4.1') });

Use factual instead of scorers.factual() in the evaluation's scorers array.

Viteval resolves defaults in each worker from the selected project config.

Verification

pnpm exec viteval run

Confirm the task produces an answer and each scorer returns a numeric score. Model-based scores vary and the example is not guaranteed to pass its threshold. Use saved reports to inspect score metadata.

Troubleshooting

Missing model or authentication errors

Confirm models.providers and models.defaults are configured and OPENAI_API_KEY is available to the evaluation process. A null score with error metadata indicates a failed measurement, not a low-quality answer.

Use a different judge for one scorer

Use a factory's model override to change one judge. Embedding factories do not accept per-scorer model overrides.

References