Crash the workers behind a queue
Three workers take the bakery’s orders from a queue. Crash one, choose when they acknowledge, and watch every order end up printed once, twice, or not at all.
Virtual time 4.000 s, seed 28Every run crashes worker 2 at 2.400 s.
Order 18 was printed twice: worker 2 crashed after printing its ticket and before its ack left, so the broker sent it again and worker 1 printed it a second time. Nothing was lost; 25 orders are done and 13 are queued or with a worker.
Orders waiting in the queue11 waiting, the oldest for 1.0 s
- No order was lostHolds
- No order was printed twiceBroken: 1 order printed twice
- The queue drainsHolds; checked once orders stop
No order was lost, but 1 order was printed twice. Whether the queue drains is checked once orders stop.
What happened
- 2.400 s Worker 2 crashes holding order 18, after its ticket printed and before its ack left. The broker has no ack for it, so it will send it again
- 2.404 s The broker sees worker 2's connection close and puts order 18 back at the head of the queue
- 2.444 s The broker sends order 18 again, to worker 1, marked redelivered
- 2.873 s Worker 1 prints order 18's ticket again: order 18 is printed twice
What a queue promises
A broker holds the bakery’s online orders until a worker takes one, prints its oven ticket and says it is done. That last step, the acknowledgement, is the whole contract. Until the broker hears it, the order is still the broker’s, and if the worker’s connection drops, the broker gives the order to another worker. So a queue delivers at least once. It cannot promise exactly once: a worker can crash after it printed the ticket and before its acknowledgement left, and then the order is printed again.
Exactly once is something the worker builds on top. It keeps a record of every order it has printed, keyed by the order’s id, and writes the record in the same step as the ticket. A second delivery of the same order finds its id and prints nothing. That record is an idempotency key.
Three things to try
Crash a worker between the ticket and the acknowledgement. The first run does it: worker 2 crashes at 2.4 s, order 18 goes back to the head of the queue, and worker 1 prints it a second time. Turn on the idempotency key and start over. The same crash at the same moment prints order 18 once; the second delivery is skipped.
Acknowledge before the work. Switch to acknowledging on delivery, then crash a worker. The orders it held are gone, never printed. Acknowledging first trades duplicates for losses: at most once instead of at least once.
Publish a poison order, then stop taking orders. It fails every time. With requeue, it comes straight back and keeps a worker busy forever, and the queue never drains. With dead-lettering, it is retried after 200, 400 and 800 ms, then set aside after the fourth try, and the queue drains.
Then slow one worker down with ten orders of prefetch, and again with one: with ten it sits on nine orders it cannot reach. And try a rush: three workers handle about 9.7 orders a second, so at 14 a second the queue grows, and each order’s wait with it.
What the lines and lamps mean
Orders enter on the left and wait in the tank. Each worker’s track holds the orders it has taken, up to its prefetch; the lit lamp is the one it is working on. A ticket at the right is filled once printed, and an order printed twice carries a second ring. The siding that ends in buffer stops holds the dead letters. The three properties under the picture are checked after every event.
Credits
The model is the acknowledgement, prefetch and dead-lettering of the AMQP 0-9-1 protocol, described here without any one broker’s details. Idempotency keys as a worker’s own record: Brandur Leach, Implementing Stripe-like Idempotency Keys in Postgres (2017). The queue’s growth under load is Little’s law: the number of orders waiting is the rate they arrive times how long each waits (John D. C. Little, 1961).
What it costs
Nothing to run. Every file is static, and your browser runs the simulation.
How it is built
The broker, the workers and the orders are one TypeScript module, run in virtual time. A message takes 2 to 6 ms each way, a worker spends 10 to 40 ms between printing a ticket and sending its acknowledgement, and with dead-lettering a failing order waits 200, 400 and 800 ms between tries. Every random choice comes from one seeded stream, and your presses are recorded at the moment they happened, so the same seed and the same presses give the same run. That is how the page opens with a picture drawn in advance, 143 events into the first run, and how your browser then takes over at that exact moment.
After every event, a checker finds every order in exactly one place: the queue, the wire, a worker, done, the dead letters or lost. Every 250 ms it also checks each order against its own record. The site's tests make 2,000 seeded runs of crashes, restarts and slow spells, acknowledging after the work with the key on. In every one no order is lost or printed twice, and once orders stop and the pool grows to 5 workers, the queue drains within 30 s. The same runs with the key off, or acknowledging first, break a property in some of them, and so do a broker and workers broken on purpose.