Make Cron Jobs Idempotent: Prevent Duplicate Sends
Resumo
Pixel shares a practical guide to handling duplicate work in scheduled jobs, explaining why cron triggers and retries create at-least-once delivery. It demonstrates using a PostgreSQL unique constraint and stable event IDs to atomically claim events and prevent duplicate side effects.
DEV Community
Your cron job will fire twice. Design for it.
Every tutorial for scheduled work has the same shape. A cron expression, a route handler, a loop over users, a call to an email API. It runs on the author's machine for two days and then it ships. Then production happens. The platform retries a 502 that actually succeeded. A deploy lands mid-run and two workers process the same batch. The retry after a rate limit sends the digest a second time. A customer forwards the duplicate to support, and someone writes a ticket titled "we got this twice," and that is the first time anybody thinks about delivery semantics. Cron is a trigger, not a scheduler. The moment your work leaves the process, you are in at-least-once territory, and at-least-once means duplicates are not an edge case. They are the contract. Make the duplicate impossible, not unlikely The fix is not a sent_at column you check before sending, because check-then-send has a race and you will find it at 3 a.m. with two workers that both read null. The fix is a unique constraint that decides the winner atomically, before any side effect: import pg from "pg";
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
async function claim(eventId) {
const { rowCount } = await pool.query(
insert into processed_events (event_id, claimed_at)
values ($1, now())
on conflict (event_id) do nothing,
[eventId]
);
return rowCount === 1; // true only for the first caller
}
if (!(await claim(event.id))) return; // someone else has it, or we already did
Two details matter more than the code. The key must come from the event, not from a timestamp or a random UUID you generate at send time, and it must be stable across retries. If your queue redelivers with a fresh ID, the constraint does nothing for you. And on conflict do nothing returning rowCount = 0 tells you the event was already claimed, which is exactly the information you need to skip quietly instead of erroring. For the recurring job itself, the same idea applies at the job level. A single-flight lock beats a if (running) return flag in memory, which does nothing when you run three replicas: const { rows } = await pool.query(
select pg_try_advisory_lock(hashtext('nightly-digest')) as locked
);
if (!rows[0].locked) return;
try {
await runDigest();
} finally {
await pool.query(select pg_advisory_unlock(hashtext('nightly-digest')));
}
Bound the concurrency or drown The other half of this problem is what happens when your provider pushes back. A loop over ten thousand users with await inside is slow but polite. A loop with Promise.all is fast and gets you a 429 within seconds, and then the retry storm makes it worse. Bound the in-flight requests, and treat 429 as an instruction rather than an error: import pLimit from "p-limit";
const limit = pLimit(4); // provider allows ~5 concurrent, leave headroom
async function sendWithRetry(payload, attempt = 0) {
try {
return await limit(() => post("/v1/messages", payload));
} catch (err) {
if (err.status === 429 && attempt < 5) {
const retryAfter = Number(err.headers.get("retry-after") ?? 0);
const backoff = retryAfter * 1000 || 2 ** attempt * 1000;
await sleep(backoff + Math.random() * 250); // jitter
return sendWithRetry(payload, attempt + 1);
}
throw err;
}
}
Three things in there that get skipped in tutorials. Retry-After is honored when the server sends it, because the server knows better than your exponential curve. Jitter is added, because identical backoff across workers synchronizes them into a thundering herd. And there is an attempt cap, because a permanent failure should land in a dead letter queue where a human sees it, not in an infinite loop that burns your quota. Cap it, then route it. A row in a failed_jobs table with the last error and the payload hash is worth more than any amount of clever retry logic, because it makes the failure visible instead of ambient. Don't enqueue from inside the transaction you can still roll...

Powered by Readybot.io