OpenTelemetry Java Agent 3.0: Kafka Migration Guide
ملخص
Pixel shares a practical guide to migrating Spring Boot Kafka observability to OpenTelemetry Java agent 3.0, which can rename Kafka spans and silently break dashboards and alerts. The announcement recommends auditing queries, emitting old and new telemetry side by side, remapping names in the Collector, canarying a consumer group, and linting alert queries in CI.
DEV Community
OpenTelemetry Java Agent 3.0 Migration: Kafka Dashboards
Short answer: upgrading to the OpenTelemetry Java agent 3.0 can silently break Spring Boot Kafka dashboards and alerts, because Kafka spans move to new verb-style names such as poll orders. To migrate safely, inventory every query that depends on span names, emit old and new telemetry side by side, remap names in the Collector, canary one consumer group and lint alert queries in CI. The incidents that cost me the most sleep made no noise at all. No crash, no page. One panel on a wall dashboard drew a flat line, and the first person to notice was a customer. Swapping an observability agent version can produce exactly that. The OpenTelemetry Java agent 3.0 changes several telemetry defaults, including how Kafka work is named. I treat a major agent bump as a production change with its own inventory, canary and diff, the same way a schema migration would be handled, and not as a line in pom.xml. Start here: what the agent actually does Some vocabulary, kept short, since the rest of the post leans on it. OpenTelemetry (OTel) is the CNCF project that standardises how applications emit traces, metrics and logs. A span is one timed unit of work, say a Kafka poll or an HTTP handler. A trace strings together the spans of one request as it crosses services. In Kafka, producers write to a named topic and consumers in a consumer group read from it. The Java agent is a JAR passed with -javaagent at JVM start. It rewrites library classes as they load, so Spring MVC, Kafka clients and JDBC drivers emit spans with zero source changes. The convenience has a price: you never chose those span names, so you may not know how many dashboards quietly depend on them. Every span carries a name and attributes (key-value facts such as topic or consumer group). Dashboards, alerts and sampling rules filter on those. Rename one and the filter matches nothing, with no error anywhere. What changes in 3.0, and why Kafka is the loudest case The agent follows the OpenTelemetry semantic conventions, the project's naming spec for spans, metrics and attributes. As the conventions evolve, the agent's output follows. In the 3.0 line, Kafka spans move to verb-first names: what used to appear as orders receive or orders process is now poll orders or process orders. Attribute keys move too. The kind of change to expect, based on the messaging conventions, looks like this: Old shape (example) New shape (example) span orders receive span poll orders span orders process span process orders messaging.kafka.consumer.group messaging.consumer.group.name messaging.kafka.destination.partition messaging.destination.partition.id Treat that table as an illustration of the pattern, not the authoritative list. I have not verified every row against the 3.0 build you will run, and the exact set depends on your instrumentation versions, so diff real spans (Step 4) and read the release notes before you write anything into a runbook. Why Kafka hurts most, in practice: Event-driven systems are watched through message-path spans, which are the ones renamed. A Grafana Tempo query like { name = "orders process" && resource.service.name = "orders-consumer" } simply returns an empty set after the upgrade. Lag and processing-time alerts are frequently built from span-derived metrics (a spanmetrics connector, for instance), such as duration_milliseconds_bucket{span_name="orders process"}. The series stops being written under that label, and the alert expression evaluates to nothing. A consumer has no user waiting on it, so nobody complains when its alert goes blind. The silence can last days. Kafka share groups will reshape consumer topology again; I covered that in Kafka share groups in Spring Boot payment pipelines. Whatever contract file you build below should survive that change as well. On an airline platform handling roughly 130 million requests a day, I once watched a latency chart that had not broken at all. A refactor had changed what its query matched, so it was now averaging a smal...

Powered by Readybot.io