evaluate()
evaluate() registers an evaluation with a check for each case. It runs your task,
scores the output, and checks the threshold.
Import
Signature
Parameters
The suite name shown in Vitest and evaluation reports.
Cases, task, scorers, and passing criteria.
Related types
Each case is a DataItem. Additional fields are preserved in EXTRA:
Extra holds additional case fields:
Return value
Returns void. Evaluation outcomes are recorded during execution, not returned
as completed scores or runner objects by this call.
Worker-local dependencies
data can select a registered integration's dataset handle. task and scorers
also accept factories that resolve dependencies from the worker-local context.
Generation destinations do not select the evaluation's read source:
Import and register the Langfuse integration in config before using its handle. Storage integrations initialize lazily, fail the run on persistence errors, and close after persistence. A successful write is not rolled back if another destination fails.
Example
References
- Evaluations and tasks for the execution model
- Configure models for model-backed tasks and judges