Temporal Schedules vs. a Database Sweeper

Resumo

Pixel shares an article comparing one Temporal Schedule per customer with a single sweeper for recurring tasks in Go. It explains how storing recurrence rules and next-run dates in a database avoids conflicting sources of truth, while a sweeper processes due tasks in batches of 500 and catches up after skipped runs.

DEV Community

One Temporal Schedule per customer, or one sweeper?

Your users schedule a task themselves, one that comes back every week, every month or every year. You use Temporal. The first reflex is to create one Temporal Schedule per customer. I considered it, then dropped it. As soon as the date lives both in your database and in Temporal, you have two sources of truth. And nothing keeps them in agreement. TL;DR: one Schedule per customer copies the recurrence rule into two systems, with no shared transaction. Keep the recurrence in the database, with its next date. One Schedule starts a sweeper that processes everything due, in batches of 500. If a run is skipped, the next one catches up, because the database knows what is late. In this project, I keep native Schedules for jobs defined in code. The acceptable delay guides that choice. This article is for Go developers who schedule recurring tasks with Temporal, or who are thinking about it. You do not need to know Temporal in depth: I define every term. The context: recurrences created while the system runs In early October, I designed task scheduling for about twenty microservices. The project is in Go, and the chosen engine is Temporal. There were two kinds of tasks. First, system jobs with a fixed frequency, like a reminder sent every day at 8 am. Then, recurrences chosen by the user. A recurrence comes back every week, every month or every year. It has a start date and sometimes an end date. It is created and changed through an API, while the system runs. I ruled out a home-made loop that wakes up every minute to look for work. Temporal does the triggering. Temporal offers Schedules for that. A Schedule starts a workflow at planned times. It resembles cron, but managed by Temporal. A workflow is a Go function that Temporal records step by step, so it can resume it after a failure. I built a prototype in Go, with Postgres and Temporal, to compare two options. The obvious option: one Schedule per customer The idea is simple. When the API creates a recurrence, it also creates a Schedule just for it. Temporal fires it on every due date, and its UI shows each customer. The number of Schedules is not a problem in itself. On Temporal's official forum, a 2024 answer says there is no limit. The Schedules page gives a single load hint. An action usually starts within one second of its planned time. Unless many Schedules fire at the same moment. So volume was not my problem. The problem was where the date lives. Two systems, two truths The recurrence has to live in your database. The API shows it, changes it and knows when it ends. With one Schedule per customer, the same rule also lives in Temporal. Every change then becomes two writes, in two systems. They share no transaction. If the second one fails, the first one stays. Take a simple case. A user switches a recurrence from monthly to weekly. The database saves the change. Then the call that updates the Schedule fails on a network cut. Your database says "every week". Temporal still fires every month. Nothing warns you. To guard against it, you need a job that compares both sides and fixes the gaps. But Temporal serves the list of Schedules through its search system. The documentation warns that this list is eventually consistent. A Schedule that was just created or deleted may not show up right away. So your check job compares your database with a list that may lag behind. A Schedule's calendar cannot say "the last day of the month" either. Temporal's API definition says it plainly. Such relative expressions are not representable, and you must list the days you want. Yet a recurrence created on the 31st must land on February 28, then return to March 31. I covered that calculation in the AddDate article. With one Schedule per customer, that calculation would stay in your code anyway. The sweeper: the database is the only truth The other option keeps everything in the database. Each recurrence carries its next date, next_run_at. A single Schedule starts a sweeper at a fixed interval. T...
Temporal Schedules vs. a Database Sweeper
Powered by Readybot.io

As últimas novidades de Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …