How odin stays up

odin runs code as workflows: ordinary Python functions whose every step is written down, so a crash anywhere, in the engine, in a worker or in the network between them, costs a retry and never an order. This page is how it does that, and what it costs.

The shape of it

Three engines own the workflows. They decide what happens next, write it to Postgres, and hand the work to RabbitMQ. Three workers take the work, run the actual code, and report back. The mock services stand in for payments, shipping and drivers. Everything you see on the live view is this loop, a few hundred times a minute.

Postgres is the only source of truth. RabbitMQ is a delivery mechanism, Valkey only carries telemetry and rate limits, and the engines hold nothing a restart can lose.

Every step is written down

A workflow's history is an append-only list of events: started, activity scheduled, activity completed, timer fired, signal received, completed. Workflow code never talks to the outside world itself. It asks for an activity (charge the card, ship a parcel) or a timer, and the engine records that request in the history.

When a worker picks a workflow up, it replays the history through the code: each await that already has a recorded result gets it back at once, and the code arrives at the first step that hasn't happened yet. That's why workflow code has to be deterministic, and why a worker can die at any line without losing anything.

64 shards, one writer each

Every workflow belongs to one of 64 shards, and every shard has exactly one engine that may write to it. Engines agree on who owns what with rendezvous hashing: each shard goes to the live node that scores highest for it, so when a node joins or leaves, only its share moves.

Ownership is a lease in Postgres: 10 seconds, renewed every 3. A node that can't renew stops serving its shards after 8 seconds by its own clock, two seconds before the lease could expire. Each acquisition bumps the shard's fencing number (range_id), and every write checks it inside the same transaction. A node that froze and woke up still believing it owns a shard gets its write refused, which the pause-engine-node scenario tests each time the rotation comes round to it.

A node that leaves on purpose hands its shards over first: it stops taking new work, drains its writers, and releases each lease, so a deploy moves shards in about a second instead of waiting for leases to expire.

Side effects happen once

Two things can go wrong between "decide" and "do": the engine can crash after deciding, or the message can get lost.

• The outbox. An engine writes the task to send in the same transaction as the event that created it. A relay publishes it to RabbitMQ afterwards, and a sweeper republishes anything left behind. A task is never decided without being sent, or sent without being decided.
• At least once, then idempotent. RabbitMQ's quorum queues deliver at least once. Before running a task, a worker asks the engine whether it's still wanted, and every activity carries an idempotency key that the services honour, so a payment retried after a lost response returns the original charge, not a second one.

Refusing work instead of drowning in it

Each engine admits new runs through admission control: a quota per origin (visitors, synthetic load, benchmarks), the p99 time tasks wait in the queue, and caps on queue depth and on the outbox. Past any of them, new work gets RESOURCE_EXHAUSTED with a retry-after, and nothing already running slows down. That's also why a start can be refused for a few seconds after a deploy: the queue delay is settling.

Proving it, every ten minutes

The chaos runner breaks something on a schedule: kills or freezes a node, cuts a node off from Postgres or RabbitMQ, takes the broker away, or makes payments slow, failing or forgetful. Seven minutes later, once every run the fault could have touched is past its deadline, a checker reads Postgres and confirms that each run finished, in order, charged at most once. The chaos page shows every result.

What it doesn't do

• One machine. Three engines on one host survive process and network failures, not the host dying. A second region is out of scope.
• Python replay is the first bottleneck. Each workflow task replays the history from the start. Histories here are short, but a sticky cache of workflow state is the obvious next step.
• A fixed shard count. 64 shards is a guess that suits three nodes; changing it needs a migration and a drain.

The numbers behind all of this are on the benchmarks page.