Safe HTTP Telemetry for Rust Services
ملخص
Pixel shares an open-source Rust crate for adding safer HTTP telemetry to Axum and Tower applications. It redacts sensitive fields before queueing, limits body capture, and exports events asynchronously to Grafana Loki, helping teams investigate incidents without blocking requests.
DEV Community
Building a Safe-by-Design HTTP Telemetry Crate in Rust
Incidents are easier to investigate when you can see what happened inside your HTTP service: which route failed, how long it took, and what the request looked like. But logging request data can create new problems. Passwords and tokens may end up in logs. Large payloads can consume memory. And if telemetry export waits on a slow or unavailable backend, it can affect the request path. To address these tradeoffs, I built safe-http-telemetry, an open-source Rust crate for Axum and Tower applications. Its design goal is to make HTTP telemetry useful while keeping capture, resource use, and export behavior under the application’s control. Three design priorities Redact sensitive data before queueing When an application captures request data, it needs to consider credentials such as passwords, bearer tokens, and API keys. safe-http-telemetry applies capture limits and redaction before events enter the export queue. It supports configured sensitive-field matching in query parameters, headers, and supported structured body formats. JSON objects are traversed recursively, with limits on depth, visited nodes, and string size. Body capture is optional. It requires the body-capture feature and a route configured for the Detailed profile. Applications can also add field names specific to their own domain. Redaction is a safeguard, not a guarantee that every secret will be identified. I recommend starting with the Minimal profile, enabling richer capture only on reviewed routes, and limiting access to the telemetry backend. Keep the request path independent from Loki The middleware submits events to a bounded queue without waiting for a network export. Background workers can consume and batch events for asynchronous delivery to Grafana Loki. If the queue fills up, the default DropNewest policy rejects the incoming event and updates operational counters. This keeps queue memory bounded and avoids waiting for Loki to become available. The application can monitor these counters and configure limits to suit its workload. The event flow looks like this: HTTP request
│
▼
Axum / Tower middleware
│
▼
Route policy → bounded capture → redaction
│
▼
Bounded event queue ── full? ──► drop newest + update counters
│
▼
Background exporter workers
│
▼
Grafana Loki
The queue and exporter have separate responsibilities: the middleware prepares events, while the consumer handles delivery and shutdown. Make geolocation an explicit local option For applications that need it, the optional geolocation feature can look up country information using a local MMDB database. The database is supplied separately, loaded during setup, and is not downloaded by the crate. Lookups do not call an external geolocation API. The cache has configurable capacity and time-to-live limits. By default, the event includes country code and country name; the application can choose other supported fields. Geolocation depends on resolving the client IP correctly, so forwarded headers should only be trusted from configured proxy networks. A small Axum integration surface After creating a bounded queue and route policies, attach the layer to your router: let telemetry_layer =
axum_telemetry_layer(queue.clone(), policies, MiddlewareConfig::default())?;
let app = Router::new()
.route("/hello/{name}", get(handler))
.layer(telemetry_layer);
With Axum, register routes before applying the layer so the middleware can record normalized route templates. The crate leaves server setup, queue consumption, and shutdown sequencing to the application. What the benchmarks show I used Criterion to measure middleware overhead with tower::ServiceExt::oneshot. The recorded run compared a service without telemetry to three capture profiles: Baseline: 2.38 µs Minimal: 27.31 µs Standard: 60.16 µs Detailed: 71.58 µs These are estimates from a documented run on one machine, not universal performance guarantees. The benchmark does not measure networ...

Powered by Readybot.io