kill-engine-node
Kills an engine outright (SIGKILL), and starts it again after 5 s.
Its shards have no owner until their leases expire; then the other engines take them over and carry on every run.
Chaos
Every ten minutes something breaks on purpose: a node is killed or frozen, a network link is cut, or a dependency starts failing. Seven minutes later a checker reads the history of every run the fault touched and checks that each one finished, charged at most once, by its deadline.
The last 20 runs
Asking the cluster…
The ten scenarios
pause-engine-node
Freezes an engine for 15 s, longer than its leases last.
The frozen node wakes up believing it still owns its shards. Fencing makes sure its late writes are refused.
kill-worker
Kills a worker in the middle of whatever it's running, and starts it after 5 s.
Its tasks come back from RabbitMQ to another worker, which replays the history and continues.
broker-outage
Cuts every engine and worker off from RabbitMQ for 10 s.
Tasks wait in the outbox, which is in Postgres, and go out once the broker is back.
partition-node-postgres
Cuts one engine off from Postgres for 20 s.
It can't renew its leases, so it fences itself and the others take its shards.
partition-node-rabbitmq
Cuts one engine off from RabbitMQ for 20 s.
It keeps its shards; its outbox fills up and drains when the link is back.
database-outage
Cuts every engine off from Postgres for 20 s.
Nothing can commit, so nothing is lost either: runs pause and resume where they were.
slow-payment
Makes the payment service take 2 s to answer, for 60 s.
Activities run longer, and the queueing limit keeps new work from piling up behind them.
payment-errors
Makes 30% of payment calls fail, for 60 s.
Charges are retried with backoff, under the same idempotency key, so a retry never charges twice.
payment-lost-responses
Charges the card but loses 30% of the answers, for 60 s.
The retry asks again with the same key and gets the original charge back, not a second one.