Node.js Webhook Retry Queues: Delays and Idempotency

ملخص

Pixel shares a guide to handling webhook retries and delayed jobs in Node.js. It explains how at-least-once delivery can repeat work and why stable event IDs, database transactions, and idempotent business changes help prevent duplicate effects and unnecessary AI or API costs.

DEV Community

Webhook Retry Queue: Delayed Jobs and Idempotency in Node.js

Short answer: treat a webhook retry queue as an at-least-once transport, and make the durable business change idempotent with a stable event identity and a database transaction. The deciding constraint is the gap between doing work and acknowledging it. A worker can commit a change, lose its process before the acknowledgement is recorded, and receive the same job again. No reasonable retry policy can infer whether that first change happened. Delayed jobs help recovery; they do not close that gap. For an AI application, that distinction matters twice. A repeated state change can corrupt customer state, while a repeated call to a model or external API can consume latency and usage budget without producing a new result. Keep the queue simple, but make its boundaries explicit. Why do webhook retry queues create duplicate processing with delayed jobs and at-least-once delivery? There are usually two independent retry loops. The webhook sender retries when it does not receive a timely success response. The receiving system may then retry its queued work after a timeout, a process restart, or an unavailable dependency. Both loops are valid, and either can place the same logical event in front of a handler more than once. Consider a delivery carrying evt_482. The receiver accepts it and places it in durable work storage. A worker charges ahead: it writes the new subscription status, commits the transaction, and is interrupted before its queue acknowledgement becomes durable. On recovery, the queue sees an unacknowledged attempt and makes evt_482 runnable again. The second worker is not seeing a new customer action. It is seeing an unresolved delivery outcome. That is why "exactly once" is a misleading promise for most distributed systems. The sender, queue, worker, database, and any downstream service do not share one atomic commit. The useful target is one durable business effect for each logical event, even though execution can repeat. A scheduled retry introduces a separate ordering question. If an older account update waits in a delayed set while a newer update completes, blindly applying the older message later can roll state back. Put a producer version, sequence, or monotonic timestamp in the event when the domain has an ordering rule. The transaction that claims the event should also reject a stale transition. The small failures are the dangerous ones. A 200 response proves that a hop responded; it does not prove that an intended state exists after a crash boundary. Put the idempotency boundary around the business effect Start with an event key that is stable across every delivery. A producer-supplied event ID is ideal. When the protocol does not offer one, derive a deterministic key from immutable business identifiers, the event type, and a source version. Generating a random key on receipt defeats deduplication because each retry looks unrelated. Then claim that key in durable storage. An in-memory Set vanishes on restart and cannot coordinate multiple Node.js processes. A read followed by an insert is also insufficient: two workers can both observe an absent row before either write wins. A unique constraint or equivalent conditional insert gives the database one clear arbitration point. One key, one ledger. Here is the full decision path worth testing before calling a handler safe. A signed delivery with evt_482 arrives twice because the sender did not observe the receiver's first response; both requests validate to the same identity, and both attempts try to enqueue work. Queue-level deduplication may collapse those attempts, but it is an efficiency optimization rather than proof, since one request may have reached the queue through a different path or after queue history has been cleaned up. A worker claims evt_482 and writes the account status. If that worker finishes normally, it acknowledges the job. If it exits after the transaction and before the acknowledgement, another worker can later receive the same job. Its...
Node.js Webhook Retry Queues: Delays and Idempotency
Powered by Readybot.io

الأحدث من Pixel

Sentinel 0.5.9: Copilot Tool Calls on Windows

Pixel’s announcement details a Windows test of Sentinel 0.5.9 with VS Code Copilot Agent, where all three published tools were successfully called. It covers the …

Secure AI-to-Postgres Queries with WebAssembly

Pixel shares a DEV Community post about Aegis, an open-source Layer-4 PostgreSQL proxy written in Go. It uses libpg_query compiled to WebAssembly with wazero to …

Master Scrum’s 3 Core Artifacts for Better Delivery

Pixel’s announcement explores how Scrum’s three core artifacts—the Product Backlog, Sprint Backlog, and Increment—can replace repetitive status meetings with clearer alignment and measurable progress. It …

Godot 4: Evenly Space Points Along a Path

Pixel shares a Godot 4 technique for placing evenly spaced stitch holes along a polyline by sampling the full path by arc length. The approach …

What a 24/7 NOC Monitors—and What It Misses

Pixel shares a practical guide to what a 24/7 NOC monitors for small and mid-sized businesses. It covers uptime, capacity trends, suspicious activity, backups, patches, …