#scorers

Viteval includes deterministic scorers, embedding similarity scorers, and LLM judges. Call each factory to create a scorer. LLM and embedding scorers use the models configured in viteval.config.ts. See Configuration.

The scores in the examples illustrate how to read results. Model-based scores vary with the configured model and are not guarantees of correctness or safety.

Judge scorers accept { model } to override the configured language model. Model failures return score: null with an error in metadata. Null scores contribute 0 to evaluation aggregation.

#Import

import { scorers } from 'viteval';

#Factory contracts

All factories return a Scorer accepting { output, expected, ...extra } and returning a Score or Promise<Score>. The evaluator also passes the case's input. Scores include a name, number | null value, and optional metadata. See createScorer() for the full callback type and wrapScorer() for adapting structured values.

#Deterministic and embedding inputs

scorers.exactMatch({ caseSensitive: false, trim: true });
scorers.levenshtein({ threshold: 0.8 });
scorers.jsonDiff({ threshold: 0.9 });
scorers.numericDiff({ tolerance: 2 });
scorers.listContains({ threshold: 0.8 });

#Judge inputs and score values

These are the fields read by each judge's prompt. output is always the value being assessed. Context scorers expect retrieved context there, not the final answer. All judges return discrete rubric scores, not continuous confidence estimates.

DataItem still requires an expected field even when a judge does not read it. Set translation's language field to the source language. When omitted it defaults to 'Unknown'.

scorers.factual({ model: openai('gpt-4.1') });

#Metadata and failures

Null contributes zero to aggregation. It is not a separate automatic failure when the scorer returns normally. A permissive evaluation threshold can therefore still pass. Embedding scorers return 1 immediately for identical string values, without calling a model.

#Text similarity

#exactMatch

Returns 1 if output exactly matches expected, 0 otherwise.

scorers.exactMatch();

Use cases: Exact answers, structured outputs, code generation

Example:

// Input: "What is 2+2?"
// Output: "4"
// Expected: "4"
// Score: 1.0

// Output: "Four"
// Expected: "4"
// Score: 0.0

#levenshtein

Scores text similarity using Levenshtein edit distance.

scorers.levenshtein();

Range: 0.0 - 1.0 (higher = more similar)
Use cases: Approximate text matching, typo tolerance

Example:

// Output: "The sky is blue"
// Expected: "The sky is bleu"
// Score: ~0.87 (two edits across 15 characters)

// Output: "Blue sky"
// Expected: "The sky is blue"
// Score: ~0.27 (11 edits across 15 characters)

#answerSimilarity

Semantic similarity using embeddings.

scorers.answerSimilarity();

Range: 0.0 - 1.0
Use cases: Meaning-based comparison, paraphrasing

Example:

// Output: "The capital of France is Paris"
// Expected: "Paris is France's capital city"
// Score: ~0.95 (same meaning, different wording)

#embeddingSimilarity

Measures semantic similarity using embeddings.

scorers.embeddingSimilarity();

Range: 0.0 - 1.0
Use cases: Semantic search, content matching

Model setup:

• Uses the embedding model configured in models.defaults.embedding
• Compares outputs and expected answers with cosine similarity
• Language support depends on the selected model

#Content quality

#factual

Asks an LLM judge to compare the output's factual accuracy with the expected answer.

scorers.factual();

Range: 0.0 - 1.0
Use cases: Knowledge-based QA, educational content

The reported scorer name is Factuality. The judge returns one of five categories: consistent subset (0.4), consistent superset (0.6), same details (1), disagreement (0), or immaterial differences (1). This is not a continuous confidence score.

Example:

// Question: "What is the capital of France?"
// Output: "Paris is the capital of France."
// Expected: "Paris is the capital of France."
// Score: 1.0 (factually correct)

// Output: "London is the capital of France"
// Expected: "Paris"
// Score: 0.0 (factually incorrect)

#summary

Asks an LLM judge to compare the submitted summary with an expert summary of input. Returns 0 or 1.

scorers.summary();

Range: 0.0 - 1.0
Use cases: Text summarization, content condensation

Criteria:

• Compares which summary better describes the original text
• Ignores style, grammar, and punctuation differences

#translation

Asks an LLM judge to assess a translation against the expected answer.

scorers.translation();

Range: 0.0 - 1.0
Use cases: Machine translation, multilingual content

Criteria:

• Meaning preservation, including nouns and tenses
• Verbs or adjectives that change the meaning
• Ignores style and punctuation differences

Include a language field in each case to name the source language. The judge compares translations into English and returns 0 or 1.

#Answer quality

#answerCorrectness

Asks an LLM judge to compare an answer with the expected answer.

scorers.answerCorrectness();

Range: 0.0 - 1.0
Use cases: QA systems, educational assessments

#answerRelevancy

Asks an LLM judge how relevant the answer is to the question.

scorers.answerRelevancy();

Range: 0.0 - 1.0
Use cases: Chat systems, search results

Example:

// Question: "What's the weather like?"
// Output: "It's sunny and 75°F"
// Score: 1.0 (highly relevant)

// Output: "I like pizza"
// Score: 0.0 (not relevant)

#Context evaluation

#contextEntityRecall

Uses an LLM judge to check whether the context mentions the expected entities.

scorers.contextEntityRecall();

Range: 0.0 - 1.0
Use cases: Information extraction, entity recognition

#contextPrecision

Uses an LLM judge to assess how much of the retrieved context is relevant to the question and expected answer. Pass the retrieved context as output.

scorers.contextPrecision();

Range: 0.0 - 1.0
Use cases: RAG systems, context-aware generation

#contextRecall

Uses an LLM judge to assess whether the retrieved context supports all claims in the expected answer. Pass the retrieved context as output.

scorers.contextRecall();

Range: 0.0 - 1.0
Use cases: Reading comprehension, context utilization

#contextRelevancy

Uses an LLM judge to assess whether the retrieved context is relevant to the question. Pass the retrieved context as output.

scorers.contextRelevancy();

Range: 0.0 - 1.0
Use cases: Context filtering, relevance ranking

#Safety and moderation

#moderation

Asks an LLM judge to classify the output as safe or unsafe. This is a model-based assessment, not a guarantee that an output is safe.

scorers.moderation();

Range: 0.0 - 1.0 (1.0 = safe, 0.0 = unsafe)
Use cases: Content filtering, safety checks

The judge checks for:

• Hate speech
• Violence
• Self-harm
• Sexual content
• Harassment
• Dangerous or illegal activities

Example:

// Output: "Here's a helpful recipe for cookies"
// Score: 1.0 (safe content)

// Output: "Instructions for harmful activities"
// Score: 0.0 (unsafe content)

#Structured data

#jsonDiff

Compares JSON structures for differences.

scorers.jsonDiff();

Range: 0.0 - 1.0
Use cases: API responses, structured output validation

Example:

// Output: '{"name": "John", "age": 30}'
// Expected: '{"name": "John", "age": 30}'
// Score: 1.0 (exact match)

// Output: '{"name": "John"}'
// Expected: '{"name": "John", "age": 30}'
// Score: 0.5 (partial match)

#sql

Asks an LLM judge whether the submitted SQL is equivalent to the expected query. It does not execute queries or validate them against a database.

scorers.sql();

Range: 0.0 - 1.0
Use cases: SQL generation, query validation

Criteria:

• Whether the queries would return equivalent results
• Whether the submitted query appears likely to fail
• Ignores whitespace, style, and output column naming differences

#Numeric and list comparison

#numericDiff

Calculates numerical difference between outputs.

scorers.numericDiff();

Range: 0.0 - 1.0
Use cases: Mathematical calculations, numeric predictions

Example:

// Output: "42"
// Expected: "40"
// Score: ~0.95 (small difference)

// Output: "100"
// Expected: "40"
// Score: 0.4 (1 - 60 / 100)

#listContains

Verifies if a list contains expected items.

scorers.listContains();

Range: 0.0 - 1.0
Use cases: List generation, item retrieval

Example:

// Output: ["apple", "banana", "cherry"]
// Expected: ["apple", "banana"]
// Score: ~0.67 (two exact matches divided by the longer list's length)

// Output: ["apple"]
// Expected: ["apple", "banana"]
// Score: 0.5 (contains half of expected items)

Items are converted to strings and compared using Levenshtein similarity with optimal one-to-one matching. The total is divided by the longer list's length, so extra output items reduce the score. Two empty lists score 1. One empty list scores 0.

#Specialized scorers

#possible

Asks an LLM judge whether a response offers a solution (1) or declares the task impossible (0). It does not compare the response with expected or check whether the proposed solution is correct.

scorers.possible();

Range: 0.0 - 1.0
Use cases: Detecting declarations of impossibility versus solution attempts

Example:

// Input: "Write a haiku about rain."
// Output: "Rain taps on the roof / Rivers gather in the street / Clouds drift into dusk"
// Score: 1.0 (offers a solution)

// Output: "Writing a haiku about rain is impossible."
// Score: 0.0 (declares the task impossible)

#humor

Asks an LLM judge whether the output is funny: no (0), unsure (0.5), or yes (1).

scorers.humor();

Range: 0.0 - 1.0
Use cases: Creative writing, entertainment content

This is a humor check, not a safety or appropriateness check. Use moderation separately when needed.