defineDataset()
defineDataset() defines how cases are generated and where to save them, including
hosted integrations such as Langfuse. Run generation with viteval data; importing
a definition does not generate cases or write to providers.
Import
Signature
Parameters
Unique dataset name.
Description saved with the dataset.
Cases or generator using typed integration/model dependencies.
Dataset-write destinations. Omitted inherits config; an explicit array replaces it; an empty array generates without saving.
Example
Save this as src/order-extraction.eval.ts. The two cases check a lowercase ID
and a message without an ID. No model or API key is required:
Both cases pass. With the default JSON provider, generation saves the cases under
.viteval/datasets; subsequent evaluations reuse them. Use viteval data --overwrite
when you deliberately want to replace saved cases after changing the generator.
Generation and destinations
The callback can be synchronous or asynchronous and runs once per generation operation. It can access direct dataset handles and the model registry:
The generator runs once and the same cases are sent to every selected destination.
Without --overwrite, existing destinations are preserved and only missing ones
are populated. Overwrite requires replacement support; Langfuse and VoltOps do not
support atomic replacement. Cross-provider saves are not transactional: successful
writes remain if another provider fails.
DataItem interface
The generator returns an array of DataItem objects:
Extra holds additional case fields:
Return value
Returns a Dataset<DATA_FUNC> object:
Use load() and save(); the returned dataset does not expose a data() method:
With one destination, load({ create: true }) and passing the definition to
evaluate() retain load-or-generate behavior. Multiple destinations are ambiguous
for reading; select a source explicitly:
With providers: [], load() generates in memory and save() generates without
writing. The low-level storage property selects local or in-memory storage;
use provider arrays for configured destinations. See Providers for serialization
and integration capability limits.
CLI discovery
viteval data discovers datasets in .dataset.* and .eval.* modules. Default
exports, named exports, and non-exported defineDataset() calls are supported.
Multiple datasets can share a module. Re-exporting the same dataset does not
generate it twice; distinct objects must have unique names.
Evaluation tasks and scorers are not executed during generation. See CLI datasets for patterns, overwrite behavior, and module import semantics.
DATASET_FILE_SCHEMA
The exported Zod schema validates V1 dataset files used by JSON storage, object
storage snapshots, and the dashboard. It is available from @viteval/core/dataset,
not re-exported by viteval/dataset. Install @viteval/core as a direct dependency
before using this import (pnpm does not expose Viteval's transitive dependencies):
Keep @viteval/core at the same version as your installed viteval package.
Then validate the saved dataset:
safeParse() returns { success: true, data } or { success: false, error },
not a provider Result. Invalid JSON throws before schema validation.
The top-level object is strict: unknown fields, missing required fields, and
pre-V1 or unknown schema versions are rejected. Case values may be null. An
empty data array is valid as a stored file, but cannot be evaluated. The schema
does not validate your application's input or expected-value types; validate those
separately. See Providers for
serialization limits.
References
- Datasets
evaluate()sampleItems()for fixed-count generators- Dataset generation
- Set up storage providers