`createScorer()`

createScorer() gives a scoring function a name. Use the returned scorer in an evaluation's scorers array.

Import

import { createScorer } from 'viteval';

Signature

function createScorer<OUTPUT, EXPECTED = OUTPUT, EXTRA extends Extra = Extra>(
  config:
    | ScorerConfig<OUTPUT, EXPECTED, EXTRA>
    | ((context: RuntimeContext) => ScorerConfig<OUTPUT, EXPECTED, EXTRA>)
): Scorer<OUTPUT, EXPECTED, EXTRA>;

Parameters

Scorer identity and scoring callback. EXPECTED defaults to OUTPUT.

Name attached to each score. The callback cannot override it.

Receives the task output, expected value, and additional case fields.

The value returned by the task.

The reference value from the dataset.

Additional case fields. Input is passed at runtime. Declare it in EXTRA when you need a typed input in the callback.

Any finite score. Use 0 to 1 for the usual mean or median gates. Null contributes zero. Non-finite scores and thrown errors fail the case.

Diagnostics saved alongside the score.

If returned, Viteval replaces it with the configured name.

Return value

Returns a Scorer<OUTPUT, EXPECTED, EXTRA> function that can be used in evaluations:

type Scorer<OUTPUT, EXPECTED = OUTPUT, EXTRA extends Extra = Extra> = (
  args: ScorerArgs<OUTPUT, EXPECTED, EXTRA>
) => Score | Promise<Score>;

type ScorerArgs<OUTPUT, EXPECTED, EXTRA extends Extra> = EXTRA & {
  output: OUTPUT;
  expected: EXPECTED;
};

interface Score {
  name: string;
  score: number | null;
  metadata?: Record<string, unknown>;
}

type ScoreResult = Omit<Score, 'name'> & { name?: string };

createScorer() supplies the configured name, overriding any name returned by the callback. evaluate() passes additional case fields to each scorer. It also passes input at runtime, but the current ScorerArgs type does not infer its type separately. Declare input in EXTRA when using it in a typed custom scorer.

A config factory receives worker-local model and integration dependencies. Use it to resolve dependencies, not to open external resources for each case.

Example

Task output and expected values may have different types:

const correctAnswer = createScorer<{ answer: string }, string>({
  name: 'correct-answer',
  score: ({ output, expected }) => ({
    score: output.answer === expected ? 1 : 0,
  }),
});

To adapt an existing scorer instead of writing a new one, use wrapScorer().

#createScorer()

#Import

#Signature

#Parameters