DEV Community

Agent Replay Lab: a local-first VCR for TypeScript agent tool I/O

Agent Replay Lab: a local-first VCR for TypeScript agent tool I/O I open-sourced Agent Replay Lab, a local-first TypeScript toolkit that records, mock-replays, diffs, and CI-tests the tool I/O of tool-using agents. Project page: https://jarvisosorio.com/agent-replay-lab/ Source: https://github.com/JarvisOsorio/agent-replay-lab It answers a narrower question than observability products, which usually re-run live: does this agent still take the same tool path (name / order / args / responses) against the same recorded tool world, offline, in CI? That is behavioral regression for tool trajectories — not proof that the LLM would make the same decisions again. Local-first. SQLite plus portable JSON. It is not an observability dashboard, eval leaderboard, agent framework, or a full-trajectory “deterministic agent” engine. Honest modes in v0.1 Mode What it does In v0.1? offline / mock Stub tools from the cassette; fail closed if a mock is missing Yes (default) structural compare Diff tool name / order / args / response Yes live-tools Call real tools No (0.2) live-model Call a real model No (0.2) Offline mock stubs tools only. It does not re-execute or pin LLM requests, responses, or tool-selection decisions; agent state machines, retries, or parallel scheduling; or clock, randomness, or other nondeterministic inputs. You can optionally record model.requested / model.responded events for inspection, but agent-replay replay does not play them back as a live model. The cassette category is crowded (Cassette, Agentic Stash, Agent VCR, StoneTape, TracePact, agent-retrace, AgentTape, and others). Fail-closed offline replay, redaction, and CI diffs are not unique inventions — the bar for this repo is a small, honest TypeScript/SQLite knife. Install (not on npm yet) v0.1.0, Apache-2.0. Requires Node.js 22+ and pnpm (the repo pins [email protected]). Clone the monorepo: git clone https://github.com/JarvisOsorio/agent-replay-lab.git
cd agent-replay-lab
pnpm install
pnpm build
pnpm test

5-minute proof

pnpm --filter @agent-replay-lab/example-basic-agent demo
Packages: @agent-replay-lab/core and @agent-replay-lab/cli (agent-replay commands: init, list, inspect, replay, diff, fixture, test, export, import, doctor). fixture create pins tool sequence, args, and responses. test golden --run fails with exit 1 on structural drift. Replay tests use in-memory or local cassette data and make no network calls. Capture seam is wrapTool from @agent-replay-lab/core. Known secret key names and common token shapes are redacted before persist — best-effort, not a PII product. Pair with agentscrub for after-the-fact store remediation. Not in v0.1: MCP adapter, OpenTelemetry adapter, flexible matching, semantic compare, dashboard, live auto-replay, telemetry, or behavioral-policy enforcement. I'm Jarvis Osorio; I built this. Happy to answer questions about the fail-closed mock path or the structural CI gate.

Powered by Readybot.io