For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt, and this page is available as Markdown at https://viteval.dev/api/define-dataset.md.
V1 release candidate available.Migration guide →
GitHub

defineDataset()

defineDataset() defines how cases are generated and where to save them, including hosted integrations such as Langfuse. Run generation with viteval data; importing a definition does not generate cases or write to providers.

Import

import { defineDataset } from 'viteval/dataset';

Signature

function defineDataset<DATA_FUNC extends DatasetData>(
  config: DatasetConfig<DATA_FUNC>
): Dataset<DATA_FUNC>;

Parameters

Configuration
namestringrequired

Unique dataset name.

descriptionstringoptional

Description saved with the dataset.

dataDataItem[] | (context) => DataItem[] | Promise<DataItem[]>required

Cases or generator using typed integration/model dependencies.

providersDatasetProviderName[]optional

Dataset-write destinations. Omitted inherits config; an explicit array replaces it; an empty array generates without saving.

Example

Save this as src/order-extraction.eval.ts. The two cases check a lowercase ID and a message without an ID. No model or API key is required:

import { evaluate, scorers } from 'viteval';
import { defineDataset } from 'viteval/dataset';

const orderMessages = defineDataset({
  name: 'order-extraction',
  description: 'Order ID normalization and missing IDs',
  data: () => [
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
});

evaluate('Order extraction', {
  data: orderMessages,
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
  threshold: 1,
});
pnpm exec viteval data "src/order-extraction.eval.ts"
pnpm exec viteval run order-extraction.eval.ts

Both cases pass. With the default JSON provider, generation saves the cases under .viteval/datasets; subsequent evaluations reuse them. Use viteval data --overwrite when you deliberately want to replace saved cases after changing the generator.

Generation and destinations

The callback can be synchronous or asynchronous and runs once per generation operation. It can access direct dataset handles and the model registry:

defineDataset({
  name: 'reviewed-support',
  providers: ['json', 'langfuse'],
  data: async ({ langfuse }) => {
    const cases = await langfuse.dataset('support-cases').load();
    return cases.filter((item) => item.reviewed === true);
  },
});

The generator runs once and the same cases are sent to every selected destination. Without --overwrite, existing destinations are preserved and only missing ones are populated. Overwrite requires replacement support; Langfuse and VoltOps do not support atomic replacement. Cross-provider saves are not transactional: successful writes remain if another provider fails.

DataItem interface

The generator returns an array of DataItem objects:

type DataItem<
  INPUT = unknown,
  OUTPUT = unknown,
  EXTRA extends Extra = Extra,
> = EXTRA & {
  name?: string;
  input: INPUT;
  expected: OUTPUT;
};

Extra holds additional case fields:

type Extra = Record<string, unknown>;

Return value

Returns a Dataset<DATA_FUNC> object:

interface Dataset<DATA_FUNC extends DatasetData, DATA_ITEM extends DataItem> {
  name: string;
  storage: 'local' | 'memory';
  description?: string;
  exists(): Promise<Result<boolean>>;
  load(options?: { create?: boolean }): Promise<Result<DATA_ITEM[] | null>>;
  save(options?: { overwrite?: boolean }): Promise<Result<void>>;
}

Use load() and save(); the returned dataset does not expose a data() method:

const saved = await dataset.save({ overwrite: true });
if (!saved.ok) throw saved.error;
const loaded = await dataset.load({ create: true });
if (!loaded.ok) throw loaded.error;
const cases = loaded.value;

With one destination, load({ create: true }) and passing the definition to evaluate() retain load-or-generate behavior. Multiple destinations are ambiguous for reading; select a source explicitly:

data: ({ langfuse }) => langfuse.dataset('reviewed-support');

With providers: [], load() generates in memory and save() generates without writing. The low-level storage property selects local or in-memory storage; use provider arrays for configured destinations. See Providers for serialization and integration capability limits.

CLI discovery

viteval data discovers datasets in .dataset.* and .eval.* modules. Default exports, named exports, and non-exported defineDataset() calls are supported. Multiple datasets can share a module. Re-exporting the same dataset does not generate it twice; distinct objects must have unique names.

Evaluation tasks and scorers are not executed during generation. See CLI datasets for patterns, overwrite behavior, and module import semantics.

DATASET_FILE_SCHEMA

The exported Zod schema validates V1 dataset files used by JSON storage, object storage snapshots, and the dashboard. It is available from @viteval/core/dataset, not re-exported by viteval/dataset. Install @viteval/core as a direct dependency before using this import (pnpm does not expose Viteval's transitive dependencies):

npm
pnpm
yarn
npm install --save-dev @viteval/core@rc

Keep @viteval/core at the same version as your installed viteval package. Then validate the saved dataset:

import { readFile } from 'node:fs/promises';
import { DATASET_FILE_SCHEMA } from '@viteval/core/dataset';

const text = await readFile('.viteval/datasets/order-extraction.json', 'utf8');
const parsed = DATASET_FILE_SCHEMA.safeParse(JSON.parse(text));
if (!parsed.success) throw parsed.error;
const cases = parsed.data.data;

safeParse() returns { success: true, data } or { success: false, error }, not a provider Result. Invalid JSON throws before schema validation.

FieldRequiredContract
schemaVersionYesLiteral 1
idYesNonempty string
nameYesNonempty string
descriptionNoString
versionYesPositive integer dataset revision, independent of schemaVersion
metadataYesString-keyed object; use {} when empty
createdAtYesISO datetime string in UTC, such as 2026-10-26T12:00:00.000Z
updatedAtYesISO datetime string in UTC
dataYesArray of objects with required input and expected; extra case fields are preserved

The top-level object is strict: unknown fields, missing required fields, and pre-V1 or unknown schema versions are rejected. Case values may be null. An empty data array is valid as a stored file, but cannot be evaluated. The schema does not validate your application's input or expected-value types; validate those separately. See Providers for serialization limits.

References