Queue or Stream? Rate-Limit Webhook Processing
Summary
Pixel’s announcement compares durable queues and fast streams for rate-limiting HTTP-based logistics webhooks. It explains when to prioritize restart-safe delivery, how worker-side token buckets protect destination quotas, and why jobs should store their intended delivery time.
DEV Community
Configure a Durable Queue or Fast Streams for Rate-Limiting HTTP-Based Logistics Webhooks
Configure a Durable Queue or Fast Streams for Rate-Limiting HTTP-Based Logistics Webhooks To configure a queue for rate-limiting HTTP-based webhook processing in a logistics renewal workflow, choose a durable queue when missing a business deadline is worse than adding a few milliseconds of handoff time. Choose a fast stream only when the downstream contract can tolerate replay, retention, and consumer-management work. The queue is not the rate limiter. It is the shock absorber; the worker-side limiter is the clock that protects the webhook destination. Here is the decision matrix I use before writing a producer: Constraint Durable queue Fast stream Reminder must survive a worker restart Strong fit Possible, with explicit retention and offsets Burst arrival versus fixed webhook quota Pair with a token bucket Pair with a token bucket One consumer should own each job Natural model Requires consumer-group rules Operators need a dead-letter path Built-in pattern in many systems Application convention Lowest possible handoff latency matters Usually not the first choice Better candidate My default is the durable option for renewal reminders. The decision changes when the deadline is soft, the destination is idempotent, and the team already operates stream offsets confidently. That is the whole fork. How should I configure a queue for rate-limiting HTTP-based webhook processing? A logistics renewal reminder has at least three times attached to it: when the carrier account became eligible, when the business wants a reminder sent, and when the webhook receiver will reject a late request. Confusing those timestamps creates a queue that looks healthy while delivering useless work. Store the intended delivery time with the job. Do not use queue arrival as a substitute. A cron task can poll for eligible records every minute, publish them, and exit. That task should not perform the HTTP call itself. A short cron run that discovers 10,000 renewals can otherwise create a burst against a receiver that permits 20 requests per second. The worker needs two independent gates. First, it waits until the job is eligible. Second, it acquires a rate-limit token for the receiver and route. The first gate protects business timing; the second protects an external system. A retry belongs behind both gates, or a failed request can immediately consume the whole retry budget in a tight loop. This is the small contract I want between the scheduler and the worker: type RenewalReminder = {
id: string;
accountId: string;
destination: string;
dueAt: string;
attempt: number;
idempotencyKey: string;
};
function isEligible(job: RenewalReminder, now: Date): boolean {
return Date.parse(job.dueAt) <= now.getTime();
}
The payload is deliberately boring. An immutable job id, an explicit due time, an attempt count, and an idempotency key give the worker enough information to make a decision after a restart. A URL alone does not. The trade-off is intentional: more fields make the contract larger, but they make delayed processing inspectable. No shortcuts. Where should rate limiting live? Put the limit at the narrowest boundary that owns the quota. If a receiver allows 20 requests per second per account, a process-wide limiter is wrong as soon as two accounts share a worker. If the quota is per destination, key the limiter by destination. If the receiver documents a global quota, use a shared limiter or partition work so each worker sees the same budget. A token bucket is a useful mental model: tokens arrive at a fixed rate up to a capacity, and each HTTP attempt spends one. Capacity controls the burst. Refill rate controls the sustained pace. Do not let the queue's concurrency setting pretend to be a rate limit. Ten concurrent requests can still violate a per-second quota, especially when response times vary. The limitation is operational: a shared limiter needs shared state, while a local limiter is easier to run but cannot enforce a global quota. The wo...

Powered by Readybot.io