Evaluation Engine
The Evaluation Engine measures AI workflow quality, safety and compliance using provider-independent metrics, datasets, experiments and drift analysis.
Evaluation Datasets
A dataset is a versioned collection of cases: inputs, optional reference outputs and metadata. Datasets are immutable once published, so an evaluation can always be reproduced against the exact data it originally ran on.
Evaluation Providers
Providers implement the actual scoring. The engine is provider-independent: a provider receives cases and returns metric scores against a stable interface. Built-in heuristic providers ship with KAVACH, and external providers can be registered.
Metrics
Metrics are named, typed scores such as answer relevance, groundedness, toxicity or policy compliance. Each metric declares its range and direction so results are comparable across runs and providers.
POST /api/v1/evaluations
{
"dataset_id": "support-agent",
"provider": "trulens",
"metrics": ["answer_relevance", "groundedness", "toxicity"]
}
// => { "evaluation_id": "eval-result-981", "status": "RUNNING" }Experiments
An experiment groups related evaluation runs — for example comparing two prompt versions or two models on the same dataset. Experiments give each run a shared context so results can be interpreted together.
Historical Comparison
Because datasets and results are versioned, the engine can compare any run against prior runs on the same dataset. This surfaces regressions before they reach production.
Drift Analysis
Drift analysis tracks metric distributions over time. When a metric moves beyond a configured threshold relative to its baseline, KAVACH raises a drift signal that can feed into governance decisions.
Drift as evidence
Drift signals are first-class evidence. A decision can require that no active drift alert exists for a workflow before approval.TruLens Integration
TruLens is supported as an example external provider. It plugs into the provider interface, contributing feedback-function scores such as groundedness and relevance that KAVACH records as standard metrics and can attach to decisions as evidence.