Vibe Coding Observability Before Production

DEV Community

Vibe coding observability: what to check before production

On a Monday morning a team merges an order export written almost entirely by a coding assistant. Tests are green, the review finds nothing odd, the endpoint returns 200. Three weeks later a customer reports missing rows in their file. That is the real question behind vibe coding observability: not « is the AI any good at coding? » but « what proves, in production, that this code does what I think it does? » I start from that operating problem rather than from a feature list. There are three decisions to make: where to put spans, what to look at in Dynatrace SaaS, and what to demand before a merge. For the vendor’s own take, read its post on runtime; the structure and examples below are mine. Method: I ran the Python code in the screenshots with opentelemetry-sdk 1.45.2 and show the real output. I did not run the DQL queries or send any data to a Dynatrace environment, so they are marked « not verified ». Everything applies to Dynatrace SaaS; Managed is not covered. Why vibe coding observability is not a separate discipline An assistant writes plausible code. It compiles, it passes the tests it wrote itself, it handles the happy path. What it lacks is knowledge of your production: real volumes, dirty data, slow dependencies. That is no moral failing, just a blind spot: nobody gave it that context. I quote no figure on the quality of generated code, because I have not read a study I could check. The argument holds without one. The more code you accept without writing it line by line, the more the proof must come from execution rather than reading. It is plain SRE hygiene: a measured SLI beats declared confidence. Do not mix up two jobs. Monitoring the assistant itself (cost, usage, sessions) is covered in monitoring a coding agent. This article is about the code the assistant delivers once it runs in production. For getting started with an AI-assisted IDE, the ThinkRail piece is a different subject. A textbook case: 200 OK and a quarter of the rows gone I rebuilt the scenario in twenty lines. The function does what assistants often do: loop over orders, run one query per order to find the customer, and wrap everything in a try/except that swallows any error so the export « never crashes ». Two classic flaws hide in there. First, one query per order: 1 + 40 queries for 40 rows. Second, the exception is ignored, so the response stays green. Here is the actual run. The response claims success, yet 10 orders out of 40 are missing: one customer is absent from the table. One HTTP request produced 41 spans: the signature of an N+1. The root span stays UNSET, so an SLI computed on it alone would see nothing. The 10 child spans carry ERROR and the message: that is where the truth settled. First decision: instrument the low layer, not the handler The lesson is architectural. The span recorded the error because the exception crossed the with block in query_one before the calling code silenced it. Had I wrapped only the handler, I would have got a green span. A practical rule: put spans as close as possible to outgoing calls (database, HTTP, message queue), where generated code cannot hide them by catching everything. OpenTelemetry instrumentation libraries often do this for you; I did not check which ones cover your driver. To know what a span holds, read the anatomy of an OpenTelemetry span. Second rule: ban except Exception: pass in review unless it logs or counts. That is not purism. It is the line that manufactures invisible incidents. Sending the telemetry to Dynatrace SaaS According to the OTLP page of the documentation, the SaaS endpoint is https://{environment-id}.live.dynatrace.com/api/v2/otlp, with /v1/traces for traces. Authentication uses an Authorization: Bearer header and a token with the openpipeline:traces:ingest scope. Transport is HTTP with binary Protobuf: neither gRPC nor JSON. So set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, or go through a Collector that converts. If you copy the ID from the browser address bar, ...

Powered by Readybot.io