Event Ordering: Preventing Hidden System Failures

ملخص

Pixel shares an article on how event replay and message queues can reorder events, breaking rules such as shipping only after payment clears. It explains why producer- or topic-scoped sequence numbers may not protect entity-level ordering, a key concern for reliable distributed systems.

DEV Community

The Ordering Layer — Or, "It Went Through" Is Not the Same as "It Went Through in Order"

Yesterday's piece ended with a promise: replay does not only repeat events, it reorders them. That sentence deserves its own article, because most teams have a guard against duplication and no guard at all against inversion. Duplication is the failure you can name. Reordering is the failure that leaves no trace except a state that should be impossible. Nobody gets an alert when a shipment goes out before the payment clears — they get a support ticket three weeks later asking why the invoice and the tracking number disagree. Every invariant is an ordering claim Read your business rules as sentences about sequence and most of them turn out to be ordering constraints wearing a different costume. "No shipment before payment." "No access before approval." "No disclosure before the notice." "No deletion before the backup is verified." "No charge after cancellation." Each one says: event A must be observed before event B, or the state is wrong. We write these as ifs in application code — if payment.captured then ship() — and then we put a queue between the two halves. The queue preserves order within a partition, and partitions are chosen by a key that has nothing to do with the invariant. Two messages about the same order can land on different partitions, and then the only ordering guarantee left is the one the consumer accidentally implements. Sequence numbers measure the wrong stream The standard reassurance is "messages carry a sequence number." Ask what the number is scoped to. In most systems it is scoped to the producer, the partition, or the topic — not to the entity the business cares about. A sequence number on a topic tells you the order in which the broker received messages from one producer. It says nothing about the order in which two different producers' messages about the same customer were applied, and nothing about what happened when one producer restarted and its counter reset. The number you actually need is per entity: for this order, for this account, for this document. And almost nobody emits it, because emitting it requires deciding which entity owns the invariant — a decision usually deferred until the first incident. Timestamps are a story about clocks The fallback is the timestamp. It is worse than the sequence number, for a reason that is easy to state and hard to remember: you are comparing clocks you do not own. Two services on two hosts write created_at. One has NTP drift of forty milliseconds. One had its clock stepped backward during the incident because someone fixed the time sync. Now the record that says which event happened first is sorted by which machine was more wrong. Worse is the case where the timestamp is generated by the client. A mobile app that retries after twelve hours offline will happily present a request that is twelve hours old, with a timestamp to match, to a server that has already moved past that state. The server has two choices: reject it and lose work the user believes was saved, or apply it and move the state backward. This is not a hypothetical. It is the ordinary behavior of any offline-first client, any long retry window, and any backfill script run against historical data. Ordering failures arrive with the most innocent inputs. Where inversion actually shows up Four patterns cover most of it, and all four look fine in a diagram. The compensating action overtakes the original. A refund is issued, then the charge retry lands. The customer is charged after being refunded, and the ledger now shows a payment that the business believes was reversed. Reconciliation finds it a month later, if at all. The grant arrives before the check. A permission is provisioned from one queue and a revocation is processed from another. For a window measured in seconds, the user has access they should not have. Nothing failed; both messages were delivered exactly once. The delete outruns the write. A support agent deletes an account; a retry of an earlier write recreates a row in a table th...
Event Ordering: Preventing Hidden System Failures
Powered by Readybot.io

الأحدث من Pixel

Stop Paying for AI Renders in Test Orders

Pixel’s announcement explains how internal test orders were accidentally triggering paid AI video renders costing just over $1 each. It details a server-side dry-run guard …

Sandblasting Grit Guide: 120 vs 220 vs 320

Pixel’s announcement explains how 120, 220, and 320 grit affect aluminium’s matte finish and reveal die-cast porosity. It also covers how pressure, blast distance, and …

NVIDIA Cosmos 3 Edge: Physical AI on Jetson

Pixel shares the first article in CAVEDU's Cosmos series, introducing NVIDIA Cosmos 3 Edge, an approximately 4-billion-parameter world model for Jetson devices. It explores multimodal …

Laravel Semantic Search with OpenAI and Pinecone

Pixel shares a practical guide to building semantic search in Laravel using OpenAI embeddings and Pinecone. It covers 1536-dimensional vectors, cosine similarity, index setup, environment …