For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt, and this page is available as Markdown at https://viteval.dev/api/evaluate.md.
V1 release candidate available.Migration guide →
GitHub

evaluate()

evaluate() registers an evaluation with a check for each case. It runs your task, scores the output, and checks the threshold.

Import

import { evaluate } from 'viteval';

Signature

function evaluate<DATA extends Data, TASK_OUTPUT = InferDataOutput<DATA>>(
  name: string,
  {
    data,
    aggregation = 'mean',
    task,
    scorers,
    threshold = 1.0,
    timeout,
    concurrency,
    retries,
    results,
    skip,
  }: Eval<DATA, TASK_OUTPUT>
): void;

Parameters

Parameters
namestringrequired

The suite name shown in Vitest and evaluation reports.

optionsEval<DATA, TASK_OUTPUT>required

Cases, task, scorers, and passing criteria.

dataData<DATA_ITEM>required

A case array, asynchronous generator, or reusable dataset. Empty datasets fail the evaluation.

inputINPUTrequired

Input passed to the task.

expectedEXPECTEDrequired

Reference value passed to scorers, not the task.

namestringoptional

Display name for the case.

...extraRecord<string, unknown>optional

Additional context or metadata passed to the task and scorers.

taskTask<INPUT, TASK_OUTPUT, EXTRA>required

Synchronous or asynchronous function receiving input and extra case fields. Expected values are not passed to it. The returned output type may differ from the expected-value type.

scorersScorer<TASK_OUTPUT, EXPECTED, EXTRA>[]required

At least one scorer. Each receives output, expected value, input, and additional case fields. See createScorer().

thresholdnumberoptional
Default: 1

A finite minimum for each case's aggregate score. The case passes when its aggregate is greater than or equal to this value. Sum thresholds may exceed 1.

aggregation'mean' | 'median' | 'sum'optional
Default: 'mean'

Combines each case's scorer values. Null contributes zero. Task or scorer exceptions and timeouts fail the case regardless of threshold.

timeoutnumberoptional

Positive per-case timeout in milliseconds, including task and scoring.

concurrencynumberoptional

Positive case concurrency limit, bounded by the worker limit.

retriesnumberoptional

Additional attempts after a failed case; reruns tasks and scorers.

resultsResultProviderName[]optional

Result destinations. Omitted inherits config; an explicit array replaces it; an empty array disables persistence.

skipbooleanoptional

Register the evaluation without executing its cases.

descriptionstringoptional

Descriptive text that does not affect scoring or case discovery.

type Data<DATA_ITEM extends DataItem> =
  DATA_ITEM[] | DataGenerator<DATA_ITEM> | Dataset<DataGenerator<DATA_ITEM>>;

type Task<INPUT, OUTPUT, EXTRA extends Extra> = (
  args: TaskArgs<INPUT, EXTRA>
) => Promise<OUTPUT> | OUTPUT;

type TaskArgs<INPUT, EXTRA extends Extra> = EXTRA & { input: INPUT };

Each case is a DataItem. Additional fields are preserved in EXTRA:

type DataItem<
  INPUT = unknown,
  OUTPUT = unknown,
  EXTRA extends Extra = Extra,
> = EXTRA & {
  name?: string;
  input: INPUT;
  expected: OUTPUT;
};

Extra holds additional case fields:

type Extra = Record<string, unknown>;

Return value

Returns void. Evaluation outcomes are recorded during execution, not returned as completed scores or runner objects by this call.

Worker-local dependencies

data can select a registered integration's dataset handle. task and scorers also accept factories that resolve dependencies from the worker-local context. Generation destinations do not select the evaluation's read source:

evaluate('Reviewed support answers', {
  data: ({ langfuse }) => langfuse.dataset('reviewed-support'),
  task: ({ input }) => answerSupportQuestion(input),
  scorers: [scorers.factual()],
  results: ['json'],
});

Import and register the Langfuse integration in config before using its handle. Storage integrations initialize lazily, fail the run on persistence errors, and close after persistence. A successful write is not rolled back if another destination fails.

Example

import { evaluate, scorers } from 'viteval';

evaluate('Order extraction', {
  data: [
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
  threshold: 1,
  timeout: 60_000,
  retries: 2,
});

References