CI Integration

Learn how to integrate Viteval into your continuous integration and deployment pipelines to ensure consistent LLM performance.

Overview

Viteval is designed to work seamlessly in CI/CD environments, allowing you to:

• Run evaluations automatically on code changes
• Gate deployments based on evaluation results
• Track performance regressions over time
• Generate evaluation reports

GitHub Actions

You can use Viteval in your GitHub Actions workflows to run evaluations on code changes.

TIP

We highly recommend restricting paths to only the files that changed (so you don't run evals on non-impactful changes).

Here's an example workflow:

name: LLM Evaluation

on:
  push:
    branches: [ main, develop ]
    # HIGHLY recommend restricting paths to only the files that changed (so you don't run evals on non-impactful changes)
    paths:
      - apps/api/src/ai/**/*.ts
      - apps/api/src/lib/ai/**/*.ts
  pull_request:
    branches: [ main ]
    # HIGHLY recommend restricting paths to only the files that changed (so you don't run evals on non-impactful changes)
    paths:
      - apps/api/src/ai/**/*.ts
      - apps/api/src/lib/ai/**/*.ts
jobs:
  evaluate:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: '20'
        cache: 'npm'
    
    - name: Install dependencies
      run: npm ci
    
    - name: Run evaluations
      run: npx viteval run
      env:
        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

GitLab CI/CD

Coming soon!

CircleCI

Coming soon!