> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt. # `scorers` Viteval includes deterministic scorers, embedding similarity scorers, and LLM judges. Call each factory to create a scorer. LLM and embedding scorers use the models configured in `viteval.config.ts`. See [Configuration](https://viteval.dev/api/configuration.md). The scores in the examples illustrate how to read results. Model-based scores vary with the configured model and are not guarantees of correctness or safety. Judge scorers accept `{ model }` to override the configured language model. Model failures return `score: null` with an error in `metadata`. Null scores contribute 0 to evaluation aggregation. ## Import ```ts import { scorers } from 'viteval'; ``` ## Factory contracts All factories return a `Scorer` accepting `{ output, expected, ...extra }` and returning a `Score` or `Promise`. The evaluator also passes the case's `input`. Scores include a name, `number | null` value, and optional metadata. See [createScorer()](https://viteval.dev/api/scoring/create-scorer.md) for the full callback type and [wrapScorer()](https://viteval.dev/api/scoring/wrap-scorer.md) for adapting structured values. ### Deterministic and embedding inputs | Factory | Values compared | Options | Model | | ----------------------- | -------------------------------------------------------------------------------- | ----------------------- | --------- | | `exactMatch()` | Output and expected. Objects use JSON serialization, other values use `String()` | `caseSensitive`, `trim` | None | | `levenshtein()` | Output and expected converted with `String()` | `threshold` | None | | `jsonDiff()` | JSON values or JSON strings, compared recursively | `threshold` | None | | `numericDiff()` | Output and expected converted with `Number()` | `tolerance` | None | | `listContains()` | Arrays, JSON array strings, or newline-separated strings | `threshold` | None | | `answerSimilarity()` | Output and expected converted with `String()` | None | Embedding | | `embeddingSimilarity()` | Output and expected converted with `String()` | None | Embedding | Deterministic scorer options caseSensitivebooleanoptional Default: `true` `exactMatch()` only. Set false to lowercase both normalized values before comparison. trimbooleanoptional Default: `false` `exactMatch()` only. Trim both normalized values before comparison. thresholdnumberoptional Default: `0` `levenshtein()`, `jsonDiff()`, and `listContains()` only. Similarities below this value return zero. Other scores are unchanged. This does not set the evaluation's pass threshold. tolerancenumberoptional Default: `0` `numericDiff()` only. A positive tolerance returns 1 when the absolute difference is within that tolerance. Otherwise, relative similarity is used. ```ts scorers.exactMatch({ caseSensitive: false, trim: true }); scorers.levenshtein({ threshold: 0.8 }); scorers.jsonDiff({ threshold: 0.9 }); scorers.numericDiff({ tolerance: 2 }); scorers.listContains({ threshold: 0.8 }); ``` ### Judge inputs and score values These are the fields read by each judge's prompt. `output` is always the value being assessed. Context scorers expect retrieved context there, not the final answer. All judges return discrete rubric scores, not continuous confidence estimates. | Factory | Prompt fields | Possible scores | | ----------------------- | ----------------------------------------- | --------------- | | `answerCorrectness()` | `input`, `output`, `expected` | 0, 0.5, 1 | | `answerRelevancy()` | `input`, `output` | 0, 0.5, 1 | | `contextEntityRecall()` | `input`, `output`, `expected` | 0, 0.3, 0.7, 1 | | `contextPrecision()` | `input`, `output`, `expected` | 0, 0.5, 1 | | `contextRecall()` | `input`, `output`, `expected` | 0, 0.5, 1 | | `contextRelevancy()` | `input`, `output` | 0, 0.5, 1 | | `factual()` | `input`, `output`, `expected` | 0, 0.4, 0.6, 1 | | `summary()` | `input`, `output`, `expected` | 0, 1 | | `translation()` | `input`, `output`, `expected`, `language` | 0, 1 | | `moderation()` | `output` | 0, 1 | | `sql()` | `input`, `output`, `expected` | 0, 1 | | `possible()` | `input`, `output` | 0, 1 | | `humor()` | `output` | 0, 0.5, 1 | `DataItem` still requires an `expected` field even when a judge does not read it. Set translation's `language` field to the source language. When omitted it defaults to `'Unknown'`. Judge options modelLanguageModeloptional Default: `configured language model` Override the judge model for one scorer. All judge factories accept this option. See [Configuration](https://viteval.dev/api/configuration.md). ```ts scorers.factual({ model: openai('gpt-4.1') }); ``` ### Metadata and failures | Scorer family | Successful metadata | Failure behavior | | --------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------- | | LLM judges | `choice`, `rationale` | Missing model, request, or structured-response failures return null with `metadata.error` | | Embeddings | `similarity` (raw cosine similarity) | Embedding failures return null with `metadata.error` and `similarity: 0` | | Levenshtein | `distance` | Exceptions propagate and fail the case | | Numeric difference | None normally | Non-finite numeric conversions return 0 with `metadata.error: 'non-numeric input'` | | Other deterministic scorers | None | Exceptions propagate and fail the case | Null contributes zero to aggregation. It is not a separate automatic failure when the scorer returns normally. A permissive evaluation threshold can therefore still pass. Embedding scorers return 1 immediately for identical string values, without calling a model. ## Text similarity ### `exactMatch` Returns 1 if output exactly matches expected, 0 otherwise. ```ts scorers.exactMatch(); ``` **Use cases**: Exact answers, structured outputs, code generation **Example**: ```ts // Input: "What is 2+2?" // Output: "4" // Expected: "4" // Score: 1.0 // Output: "Four" // Expected: "4" // Score: 0.0 ``` ### `levenshtein` Scores text similarity using Levenshtein edit distance. ```ts scorers.levenshtein(); ``` **Range**: 0.0 - 1.0 (higher = more similar)\ **Use cases**: Approximate text matching, typo tolerance **Example**: ```ts // Output: "The sky is blue" // Expected: "The sky is bleu" // Score: ~0.87 (two edits across 15 characters) // Output: "Blue sky" // Expected: "The sky is blue" // Score: ~0.27 (11 edits across 15 characters) ``` ### `answerSimilarity` Semantic similarity using embeddings. ```ts scorers.answerSimilarity(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Meaning-based comparison, paraphrasing **Example**: ```ts // Output: "The capital of France is Paris" // Expected: "Paris is France's capital city" // Score: ~0.95 (same meaning, different wording) ``` ### `embeddingSimilarity` Measures semantic similarity using embeddings. ```ts scorers.embeddingSimilarity(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Semantic search, content matching **Model setup**: - Uses the embedding model configured in `models.defaults.embedding` - Compares outputs and expected answers with cosine similarity - Language support depends on the selected model ## Content quality ### `factual` Asks an LLM judge to compare the output's factual accuracy with the expected answer. ```ts scorers.factual(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Knowledge-based QA, educational content The reported scorer name is `Factuality`. The judge returns one of five categories: consistent subset (0.4), consistent superset (0.6), same details (1), disagreement (0), or immaterial differences (1). This is not a continuous confidence score. **Example**: ```ts // Question: "What is the capital of France?" // Output: "Paris is the capital of France." // Expected: "Paris is the capital of France." // Score: 1.0 (factually correct) // Output: "London is the capital of France" // Expected: "Paris" // Score: 0.0 (factually incorrect) ``` ### `summary` Asks an LLM judge to compare the submitted summary with an expert summary of `input`. Returns 0 or 1. ```ts scorers.summary(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Text summarization, content condensation **Criteria**: - Compares which summary better describes the original text - Ignores style, grammar, and punctuation differences ### `translation` Asks an LLM judge to assess a translation against the expected answer. ```ts scorers.translation(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Machine translation, multilingual content **Criteria**: - Meaning preservation, including nouns and tenses - Verbs or adjectives that change the meaning - Ignores style and punctuation differences Include a `language` field in each case to name the source language. The judge compares translations into English and returns 0 or 1. ## Answer quality ### `answerCorrectness` Asks an LLM judge to compare an answer with the expected answer. ```ts scorers.answerCorrectness(); ``` **Range**: 0.0 - 1.0\ **Use cases**: QA systems, educational assessments ### `answerRelevancy` Asks an LLM judge how relevant the answer is to the question. ```ts scorers.answerRelevancy(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Chat systems, search results **Example**: ```ts // Question: "What's the weather like?" // Output: "It's sunny and 75°F" // Score: 1.0 (highly relevant) // Output: "I like pizza" // Score: 0.0 (not relevant) ``` ## Context evaluation ### `contextEntityRecall` Uses an LLM judge to check whether the context mentions the expected entities. ```ts scorers.contextEntityRecall(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Information extraction, entity recognition ### `contextPrecision` Uses an LLM judge to assess how much of the retrieved context is relevant to the question and expected answer. Pass the retrieved context as `output`. ```ts scorers.contextPrecision(); ``` **Range**: 0.0 - 1.0\ **Use cases**: RAG systems, context-aware generation ### `contextRecall` Uses an LLM judge to assess whether the retrieved context supports all claims in the expected answer. Pass the retrieved context as `output`. ```ts scorers.contextRecall(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Reading comprehension, context utilization ### `contextRelevancy` Uses an LLM judge to assess whether the retrieved context is relevant to the question. Pass the retrieved context as `output`. ```ts scorers.contextRelevancy(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Context filtering, relevance ranking ## Safety and moderation ### `moderation` Asks an LLM judge to classify the output as safe or unsafe. This is a model-based assessment, not a guarantee that an output is safe. ```ts scorers.moderation(); ``` **Range**: 0.0 - 1.0 (1.0 = safe, 0.0 = unsafe)\ **Use cases**: Content filtering, safety checks **The judge checks for**: - Hate speech - Violence - Self-harm - Sexual content - Harassment - Dangerous or illegal activities **Example**: ```ts // Output: "Here's a helpful recipe for cookies" // Score: 1.0 (safe content) // Output: "Instructions for harmful activities" // Score: 0.0 (unsafe content) ``` ## Structured data ### `jsonDiff` Compares JSON structures for differences. ```ts scorers.jsonDiff(); ``` **Range**: 0.0 - 1.0\ **Use cases**: API responses, structured output validation **Example**: ```ts // Output: '{"name": "John", "age": 30}' // Expected: '{"name": "John", "age": 30}' // Score: 1.0 (exact match) // Output: '{"name": "John"}' // Expected: '{"name": "John", "age": 30}' // Score: 0.5 (partial match) ``` ### `sql` Asks an LLM judge whether the submitted SQL is equivalent to the expected query. It does not execute queries or validate them against a database. ```ts scorers.sql(); ``` **Range**: 0.0 - 1.0\ **Use cases**: SQL generation, query validation **Criteria**: - Whether the queries would return equivalent results - Whether the submitted query appears likely to fail - Ignores whitespace, style, and output column naming differences ## Numeric and list comparison ### `numericDiff` Calculates numerical difference between outputs. ```ts scorers.numericDiff(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Mathematical calculations, numeric predictions **Example**: ```ts // Output: "42" // Expected: "40" // Score: ~0.95 (small difference) // Output: "100" // Expected: "40" // Score: 0.4 (1 - 60 / 100) ``` ### `listContains` Verifies if a list contains expected items. ```ts scorers.listContains(); ``` **Range**: 0.0 - 1.0\ **Use cases**: List generation, item retrieval **Example**: ```ts // Output: ["apple", "banana", "cherry"] // Expected: ["apple", "banana"] // Score: ~0.67 (two exact matches divided by the longer list's length) // Output: ["apple"] // Expected: ["apple", "banana"] // Score: 0.5 (contains half of expected items) ``` Items are converted to strings and compared using Levenshtein similarity with optimal one-to-one matching. The total is divided by the longer list's length, so extra output items reduce the score. Two empty lists score 1. One empty list scores 0. ## Specialized scorers ### `possible` Asks an LLM judge whether a response offers a solution (1) or declares the task impossible (0). It does not compare the response with `expected` or check whether the proposed solution is correct. ```ts scorers.possible(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Detecting declarations of impossibility versus solution attempts **Example**: ```ts // Input: "Write a haiku about rain." // Output: "Rain taps on the roof / Rivers gather in the street / Clouds drift into dusk" // Score: 1.0 (offers a solution) // Output: "Writing a haiku about rain is impossible." // Score: 0.0 (declares the task impossible) ``` ### `humor` Asks an LLM judge whether the output is funny: no (0), unsure (0.5), or yes (1). ```ts scorers.humor(); ``` **Range**: 0.0 - 1.0\ **Use cases**: Creative writing, entertainment content This is a humor check, not a safety or appropriateness check. Use `moderation` separately when needed.