> For AI agents: the complete documentation index is available at https://viteval.dev/llms.txt, the full documentation bundle is available at https://viteval.dev/llms-full.txt.

# `defineDataset()`

`defineDataset()` defines how cases are generated and where to save them, including
hosted integrations such as Langfuse. Run generation with `viteval data`; importing
a definition does not generate cases or write to providers.

## Import

```ts
import { defineDataset } from 'viteval/dataset';
```

## Signature

```ts
function defineDataset<DATA_FUNC extends DatasetData>(
  config: DatasetConfig<DATA_FUNC>
): Dataset<DATA_FUNC>;
```

## Parameters


Configuration

namestringrequired

Unique dataset name.

descriptionstringoptional

Description saved with the dataset.

dataDataItem[] | (context) => DataItem[] | Promise<DataItem[]>required

Cases or generator using typed integration/model dependencies.

providersDatasetProviderName[]optional

Dataset-write destinations. Omitted inherits config; an explicit array
replaces it; an empty array generates without saving.

## Example

Save this as `src/order-extraction.eval.ts`. The two cases check a lowercase ID
and a message without an ID. No model or API key is required:

```ts
import { evaluate, scorers } from 'viteval';
import { defineDataset } from 'viteval/dataset';

const orderMessages = defineDataset({
  name: 'order-extraction',
  description: 'Order ID normalization and missing IDs',
  data: () => [
    { input: 'Please cancel ord-2087.', expected: 'ORD-2087' },
    { input: 'How do I update my address?', expected: '' },
  ],
});

evaluate('Order extraction', {
  data: orderMessages,
  task: ({ input }) => input.match(/\bORD-\d+\b/i)?.[0].toUpperCase() ?? '',
  scorers: [scorers.exactMatch()],
  threshold: 1,
});
```

```bash
pnpm exec viteval data "src/order-extraction.eval.ts"
pnpm exec viteval run order-extraction.eval.ts
```

Both cases pass. With the default JSON provider, generation saves the cases under
`.viteval/datasets`; subsequent evaluations reuse them. Use `viteval data --overwrite`
when you deliberately want to replace saved cases after changing the generator.

## Generation and destinations

The callback can be synchronous or asynchronous and runs once per generation
operation. It can access direct dataset handles and the model registry:

```ts
defineDataset({
  name: 'reviewed-support',
  providers: ['json', 'langfuse'],
  data: async ({ langfuse }) => {
    const cases = await langfuse.dataset('support-cases').load();
    return cases.filter((item) => item.reviewed === true);
  },
});
```

The generator runs once and the same cases are sent to every selected destination.
Without `--overwrite`, existing destinations are preserved and only missing ones
are populated. Overwrite requires replacement support; Langfuse and VoltOps do not
support atomic replacement. Cross-provider saves are not transactional: successful
writes remain if another provider fails.

## DataItem interface

The generator returns an array of `DataItem` objects:

```ts
type DataItem<
  INPUT = unknown,
  OUTPUT = unknown,
  EXTRA extends Extra = Extra,
> = EXTRA & {
  name?: string;
  input: INPUT;
  expected: OUTPUT;
};
```

`Extra` holds additional case fields:

```ts
type Extra = Record<string, unknown>;
```

## Return value

Returns a `Dataset<DATA_FUNC>` object:

```ts
interface Dataset<DATA_FUNC extends DatasetData, DATA_ITEM extends DataItem> {
  name: string;
  storage: 'local' | 'memory';
  description?: string;
  exists(): Promise<Result<boolean>>;
  load(options?: { create?: boolean }): Promise<Result<DATA_ITEM[] | null>>;
  save(options?: { overwrite?: boolean }): Promise<Result<void>>;
}
```

Use `load()` and `save()`; the returned dataset does not expose a `data()` method:

```ts
const saved = await dataset.save({ overwrite: true });
if (!saved.ok) throw saved.error;
const loaded = await dataset.load({ create: true });
if (!loaded.ok) throw loaded.error;
const cases = loaded.value;
```

With one destination, `load({ create: true })` and passing the definition to
`evaluate()` retain load-or-generate behavior. Multiple destinations are ambiguous
for reading; select a source explicitly:

```ts
data: ({ langfuse }) => langfuse.dataset('reviewed-support');
```

With `providers: []`, `load()` generates in memory and `save()` generates without
writing. The low-level `storage` property selects local or in-memory storage;
use provider arrays for configured destinations. See [Providers](https://viteval.dev/guide/providers.md) for serialization
and integration capability limits.

## CLI discovery

`viteval data` discovers datasets in `.dataset.*` and `.eval.*` modules. Default
exports, named exports, and non-exported `defineDataset()` calls are supported.
Multiple datasets can share a module. Re-exporting the same dataset does not
generate it twice; distinct objects must have unique names.

Evaluation tasks and scorers are not executed during generation. See
[CLI datasets](https://viteval.dev/cli/data.md) for patterns, overwrite behavior, and module
import semantics.

## `DATASET_FILE_SCHEMA`

The exported Zod schema validates V1 dataset files used by JSON storage, object
storage snapshots, and the dashboard. It is available from `@viteval/core/dataset`,
not re-exported by `viteval/dataset`. Install `@viteval/core` as a direct dependency
before using this import (pnpm does not expose Viteval's transitive dependencies):


```sh [npm]
npm install --save-dev @viteval/core@rc
```

```sh [pnpm]
pnpm add -D @viteval/core@rc
```

```sh [yarn]
yarn add --dev @viteval/core@rc
```

Keep `@viteval/core` at the same version as your installed `viteval` package.
Then validate the saved dataset:

```ts
import { readFile } from 'node:fs/promises';
import { DATASET_FILE_SCHEMA } from '@viteval/core/dataset';

const text = await readFile('.viteval/datasets/order-extraction.json', 'utf8');
const parsed = DATASET_FILE_SCHEMA.safeParse(JSON.parse(text));
if (!parsed.success) throw parsed.error;
const cases = parsed.data.data;
```

`safeParse()` returns `{ success: true, data }` or `{ success: false, error }`,
not a provider `Result`. Invalid JSON throws before schema validation.

| Field           | Required | Contract                                                                               |
| --------------- | -------- | -------------------------------------------------------------------------------------- |
| `schemaVersion` | Yes      | Literal `1`                                                                            |
| `id`            | Yes      | Nonempty string                                                                        |
| `name`          | Yes      | Nonempty string                                                                        |
| `description`   | No       | String                                                                                 |
| `version`       | Yes      | Positive integer dataset revision, independent of `schemaVersion`                      |
| `metadata`      | Yes      | String-keyed object; use `{}` when empty                                               |
| `createdAt`     | Yes      | ISO datetime string in UTC, such as `2026-10-26T12:00:00.000Z`                         |
| `updatedAt`     | Yes      | ISO datetime string in UTC                                                             |
| `data`          | Yes      | Array of objects with required `input` and `expected`; extra case fields are preserved |

The top-level object is strict: unknown fields, missing required fields, and
pre-V1 or unknown schema versions are rejected. Case values may be `null`. An
empty `data` array is valid as a stored file, but cannot be evaluated. The schema
does not validate your application's input or expected-value types; validate those
separately. See [Providers](https://viteval.dev/guide/providers.md#serialization-and-failures) for
serialization limits.

## References

- [Datasets](https://viteval.dev/concepts/datasets.md)
- [`evaluate()`](https://viteval.dev/api/evaluate.md)
- [`sampleItems()`](https://viteval.dev/api/sample-items.md) for fixed-count generators
- [Dataset generation](https://viteval.dev/cli/data.md)
- [Set up storage providers](https://viteval.dev/guide/providers.md)
