Split-Brain Is a Math Problem: Quorums, Circuit Breakers and Chaos Engineering, Simulated

Split-brain sounds like a networking failure. It's actually an arithmetic failure — a system that let two groups of nodes both believe they had "enough" votes to lead, because "enough" was never defined as a strict majority in the first place. Every distributed database that survives partitions correctly is running the same one-line safety argument underneath very different marketing.

The quorum is the whole proof

Raft's leader election works because of one constraint: a candidate needs votes from a strict majority of the cluster, more than half of replicaCount, not just "a lot" of it. That constraint is what makes split-brain structurally impossible during a network partition — with a true majority requirement, at most one side of any partition can ever contain a majority, so at most one leader can ever be elected. Configure quorumSize as anything less than a strict majority — half, or a round number that happens to be smaller — and that guarantee silently disappears: both halves of a partition can now independently believe they have quorum, both elect a leader, and you have two nodes accepting writes that will conflict the moment the network heals. The failure mode isn't exotic; it's a configuration value that looked reasonable and wasn't checked against the one inequality that matters.

A circuit breaker is a state machine, not a switch

"Circuit breaker" sounds binary, but the pattern popularized by Hystrix and carried forward by Resilience4j has three states for a reason. Closed is normal — requests flow, failures are counted. Once consecutive failures cross a threshold, the breaker trips to open: it stops forwarding requests to the failing dependency entirely and fails fast, which is the actual point — a service that's already struggling doesn't need a thundering herd of retries piling on top of it. After a cooldown, the breaker moves to half-open and lets exactly one trial request through; succeed and it closes again, fail and it reopens the cooldown. That three-state cycle is what stops a single slow dependency from cascading into an outage of everything that calls it.

Latency, packet loss and a kill switch

Real infrastructure failures aren't binary "up or down" either, which is why a useful simulation needs more failure modes than "kill the node." Packet loss in the 5-50% range degrades a connection without severing it. A latency spike stresses timeout configuration specifically, the class of bug that only shows up under load, never in a local dev environment. And killing or partitioning a node outright is the blunt instrument, useful for confirming the system's failure detection actually fires. Watching per-node status, latency and error-rate metrics react to each of these separately — rather than just "does it work or doesn't it" — is what teaches you which failure mode your architecture actually handles gracefully.

Round robin isn't the only answer

Load balancing strategy is a design decision with real consequences, not a solved problem you set once. Round robin is fair when every backend and every request is roughly equal cost, and wrong the moment they aren't. Weighted routing accounts for backends with different capacity. Least-connections routing adapts to requests that take wildly different amounts of time to serve — the strategy that actually matters once your traffic stops looking like a synthetic benchmark.

Scaffolding, not a deploy button

The payoff for modeling a topology correctly is that the shape translates. Export the same graph of load balancers, gateways, services, queues, caches and databases as a Docker Compose file, Kubernetes manifests, or a Mermaid or PlantUML diagram, and you've turned a design sketch into a starting point — placeholder images and no resource limits or secrets yet, deliberately, because the point is the shape of the system, not a finished deployment.

Load the Raft preset, partition the database node, and check whether quorumSize was actually configured as a majority before you trust the result. Then trip a circuit breaker on purpose and watch it protect everything upstream. The lesson that took an outage to learn in production takes about ninety seconds here.