Track What Claude Code Actually Tests

DEV Community

Claude Code Said It Ran the Tests. Its Telemetry Shows What Ran

You ask Claude Code to fix a failing checkout test. A few minutes later it replies: "Fixed. The tests pass now." You merge. Then CI goes red, and when you scroll back through the session you can't find the moment a test actually ran. It's a common enough experience that a recent DEV post title sums it up: I Got Tired of Claude Code Saying 'It's Fixed' When Nothing Had Run. The summary at the end of a session is the model's account of what happened. It isn't a record. Claude Code can give you that record. It has built-in OpenTelemetry support, and with it turned on, every tool call becomes an event: which tool, whether it succeeded, how long it took and, if you opt in, the exact shell command. In an earlier article I covered what to log for an AI agent you run in production. This one points the same idea at the agent that writes your code: send its events to a file on your machine, then check what actually ran after the last edit. Everything about Claude Code's telemetry below comes from its monitoring documentation. What Claude Code can record Telemetry is off until you set CLAUDE_CODE_ENABLE_TELEMETRY=1. Claude Code then exports metrics (token and cost counters, sessions, commits) if you configure a metrics exporter, and events if you configure a logs exporter. The events are what matter here: claude_code.user_prompt is logged when you submit a prompt. The prompt text is redacted unless you set OTEL_LOG_USER_PROMPTS=1. claude_code.tool_result is logged when a tool finishes. It carries tool_name, success ("true" or "false"), duration_ms, and an error_type such as ShellError when the tool failed. claude_code.tool_decision is logged when a permission decision is made. A call you reject produces this event with decision: "reject" and no tool_result, because the tool never ran. claude_code.api_error and claude_code.api_retries_exhausted tell you when requests to the model failed after retries, which explains some sessions that simply stall. Two attributes tie it together. Every event produced while Claude Code works on one prompt shares a prompt.id, so you can pull up everything that happened for "fix the checkout test" in one filter. event.sequence is a counter that orders events within a Claude Code process. By default, the tool_result event tells you that Bash ran, not what it ran. Commands are part of the detail you have to opt into with OTEL_LOG_TOOL_DETAILS=1. With it set, a Bash event gets a tool_parameters JSON string containing full_command, and every tool's arguments are added as tool_input. Step 1: Run a collector that writes to a file You don't need a dashboard to start. The OpenTelemetry Collector can receive Claude Code's events over OTLP and write them to a local file with its file exporter. This config is all it takes (I used otelcol-contrib 0.165.0): receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317
http:
endpoint: localhost:4318

exporters:
file:
path: ./claude-events.jsonl

service:
pipelines:
logs:
receivers: [otlp]
exporters: [file]
Start it with otelcol-contrib --config collector.yaml. With the default format: json, the exporter writes newline-delimited OTLP JSON, one line per batch. Binding to localhost keeps the receiver off your network. Note that the file exporter is marked alpha for logs, and its README says exact field names aren't guaranteed to stay stable. Step 2: Point Claude Code at it In the shell where you start Claude Code: export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_LOG_TOOL_DETAILS=1
claude
This is the docs' logs-only configuration plus the tool-details flag. To check it works, submit a prompt and look for a user_prompt event in the file. If nothing arrives, the docs suggest starting Claude Code with claude --debug-file and looking for [3P telemetry] errors in that log. A word on what...

Powered by Readybot.io