Replay Management

Overview

A Replay is not a retry of the original workflow. Kavach re-resolves the authorized historical source, freezes its configuration and references, then produces a distinct execution with explicit source-to-replay lineage. The historical source is never changed.

Lifecycle

Creation validates immutable evidence and freezes a Replay configuration. Submission produces the replay execution; evaluation then creates governance evidence without rerunning that workflow.

DRAFT → READY → QUEUED → RUNNING → EXECUTION_COMPLETED → EVALUATING → COMPARING → COMPLETED

Failed and cancelled replays retain the evidence that was successfully recorded before the terminal transition. Replays may be archived, but are not deleted through the normal lifecycle.

Baseline: the original scorecard

A baseline evaluation is the saved scorecard for the original historical execution. It is the before side of a replay comparison. The evaluation produced for the new replay execution is theafter side. Kavach compares those two scorecards to show what changed and to calculate drift.

Original historical execution
  → saved evaluation = BASELINE (before)

New replay execution
  → new evaluation = REPLAY EVALUATION (after)

BASELINE + REPLAY EVALUATION
  → metric comparison + drift summary + immutable Replay result

The baseline is not a default model, prompt, or configuration. It is a specific, already-persisted evaluation result belonging to the source execution. Replay Management does not rerun the source to manufacture one, because that would no longer be historical evidence.

Kavach can select the baseline in three ways. The default is LATEST_COMPATIBLE.

A baseline is compatible only when it has:

• the same source execution and tenant scope;
• the same evaluator type and evaluator version; and
• at least one metric shared with the replay evaluation.

These checks keep the comparison meaningful. For example, ananswer_relevance score from evaluator version 1 cannot be treated as directly comparable with a similarly named score from a different evaluator version or rubric.

Lineage - What Happened

Lineage is the durable chain of references that answers, “what produced this result?” It is not a visual guess or an ontology dependency. Kavach records these references on the Replay as work progresses, and Studio renders the graph directly from them even when an ontology projection is delayed or unavailable.

Source execution
  └─ REPLAY_OF → Replay request
                     ├─ SUBMITTED_AS → Replay execution job
                     ├─ PRODUCED → New replay execution
                     ├─ EVALUATED_BY → Replay evaluation job
                     ├─ COMPARED_WITH → Source baseline evaluation
                     └─ PRODUCED → Comparison, drift, and immutable result

In the Lineage tab, select any node to inspect its identity and metadata. The graph distinguishes the original source from the produced execution, and the baseline evaluation from the replay evaluation. This matters when reviewing a failure: an execution job, an evaluation job, a comparison, and a final result are related evidence, not interchangeable statuses.

You can move nodes to inspect a complex graph and select Resetto restore the generated layout. These positions are a local viewing preference only; they do not modify lineage or any persisted evidence.

Drift Metrics - What Changed

Drift is the structured answer to “how did the replay evaluation differ from the baseline?” It is a comparison signal for review, not by itself a release approval or a root-cause explanation.

The built-in governance-drift-v1 analyzer assigns severity from the largest absolute changed-metric delta and structural changes to the metric set: NONE when nothing changed, LOW below 0.10, MEDIUM at 0.10 or above—or when a metric is new or removed—and HIGH at 0.20 or above. The result also retains the effective drift-threshold policy as governance context for review.

Worker Execution

The shared worker runtime claims one leased job at a time and dispatchesREPLAY_EXECUTION, REPLAY_EVALUATION, EVALUATION, and EXPERIMENT jobs. It commits each durable boundary before acknowledging the job, which allows safe recovery after an interruption.

On successful replay execution, the worker idempotently submits the separate replay-evaluation job. Cancellation reaches queued and running handlers at safe boundaries. API and worker processes can run together or separately, provided they share the same durable stores.

API & Studio

Source selection uses GET /api/v1/replay-executions/search, which returns a small, cursor-paginated execution projection. The Replay API exposes creation, listing, submission, evaluation, cancellation, archival, and immutable result retrieval under /api/v1/replays.

Studio provides /replays, /replays/new, and /replays/[replayId]. The detail view renders persisted lineage and evidence references; it does not reconstruct workflows, select baselines, or calculate drift in the browser.

Run Locally

# Terminal 1: API (demo data is seeded in local/dev mode)
uv run uvicorn kavach.api.app:app --reload

# Terminal 2: durable worker
uv run python -m kavach.workers.replay_worker_runtime

# Optional end-to-end smoke test
sh scripts/replay/run-smoke.sh

Docker Compose starts kavach-replay-worker separately and shares the kavach_sqlite_data volume with the API. For local demo data, 27 replayable historical sources include compatible primary mock evaluations, and the Replay inventory includes completed and active lifecycle examples.

Production Search

Replay source discovery is an eventually consistent search projection, not the authority for replay creation. It should be backed by a projected execution index and searched with bounded filters and opaque keyset cursors. Kavach always re-resolves the complete source execution before validating or creating a Replay, so a browser selection cannot bypass authorization or frozen-evidence checks.