Imagine three servers, each holding a copy of the same database. A client writes x = 5 to the first server. A different client writes x = 10 to the third server at the same moment. Both writes succeed locally. Both clients get a confirmation. And now all three servers silently disagree on what x actually is with no error, no alert, no indication that anything went wrong.
This is the core problem of distributed systems. Not latency. Not throughput. Agreement. How do you get multiple machines to maintain a consistent view of the world when they can only communicate by passing messages over an unreliable network?
The simple answer is to have every server check with the others before accepting a write. But what if one server is slow to respond? You can't tell whether it's overloaded or crashed. If you wait for it, you've made every write in your system hostage to the slowest node. If you ignore it and proceed, you risk exactly the split state you were trying to prevent. Every solution you reach for reveals a deeper problem underneath.
The insight that breaks this deadlock is that you don't need everyone to agree, you need enough of them to agree that any two successful groups must overlap. In a three-node cluster, a majority is two. In a five-node cluster, it's three. Formally, a cluster needs 2f + 1 nodes to tolerate f failures. For example three nodes to survive one failure, five nodes to survive two. The overlap between any two majorities guarantees at least one shared witness, a node that was present for both decisions and can resolve any conflict. You get fault tolerance without requiring full agreement.
Note: This is why managed Kubernetes services like EKS, GKE, and AKS run three control plane nodes by default, fully abstracted from the user, but following exactly this formula
2f+1. Three nodes, one tolerated failure, minimum viable fault tolerance.
This is not a new observation. Engineers have been wrestling with it since distributed databases became practical. And the consequences of getting it wrong are not theoretical. They show up in the systems that modern infrastructure depends on. When your Kubernetes cluster starts behaving strangely, pods evicting for no apparent reason, API calls timing out intermittently, etcd re-electing its leader every few minutes. what you are watching is a consensus system under stress. The cluster isn't broken. It is navigating exactly this problem in real time. And the algorithm doing that navigation, the one deciding who leads, what gets written, and what gets discarded when things go wrong is called Raft.
Understanding it doesn't just satisfy intellectual curiosity. it changes how you read production systems when they go wrong.
That is what this article is about.
Origin of Raft
Leslie Lamport published a solution in 1989 called Paxos. It worked. The problem was that almost nobody could implement it correctly. Google's engineers building Chubby, their distributed lock service, essentially re-derived the algorithm from scratch because the original paper left too many details unspecified. Apache's team built something different and called it "Paxos-like." A survey of real-world Paxos implementations found that virtually every production system had invented its own variant and given it the same name.
In 2014, a Stanford PhD student named Diego Ongaro made a simple argument: understandability should be a first-class design goal. If an algorithm is too difficult to reason about, engineers will implement it incorrectly, and correctness is the only thing that matters in a consensus protocol. He built a new algorithm around that constraint.
He called it Raft. It powers etcd, which powers Kubernetes. It powers HashiCorp Vault's integrated storage and Consul's service catalog.
Raft breaks this problem into three parts: leader election, log replication, and failure handling. Each mostly independent, each solvable on its own terms. Start with the first, and the rest follows.
At any moment, every node in a Raft cluster is exactly one of three things: a leader, a follower, or a candidate. There is always one leader. All writes go through it. Everything else flows from that.
⠀
Leader Election
Every Raft cluster has exactly one leader at any given time. The leader is the only node that accepts writes, the only node that replicates log entries, and the only node whose word is law on the current state of the system. Every other node is a follower passive, deferential, and entirely dependent on the leader for direction.
This asymmetry is a deliberate design choice. Raft concentrates authority in a single node not because that's the only way to build a consensus system, but because it's the simplest way to reason about one. When there is one authoritative source of truth, you eliminate an entire class of coordination problems. The cost is that you need a reliable way to replace that authority when it fails.
That mechanism is leader election.
How a follower becomes a candidate
Followers know the leader is alive through heartbeats. The leader sends them periodically, even when there are no writes, just to suppress timeouts. Each follower maintains an election timer that resets every time a heartbeat arrives. When the timer expires without a reset, the follower concludes the leader is gone and transitions to candidate state.
The candidate immediately increments its term ( Raft's logical clock ) and sends a RequestVote RPC to every other node. The term increment is not ceremonial. It signals that a new election epoch has begun. Any node that receives a message from a higher term automatically updates its own term and steps down to follower if it was doing anything else. This is how stale leaders are neutralized the moment a new election starts.
Why the vote is not just a popularity contest
A follower grants its vote only if two conditions are both satisfied. First, it has not already voted in this term. Second, and this is the safety critical part, the candidate's log must be at least as up-to-date as the follower's own log.
Up-to-date has a precise definition. If the last entries in two logs have different term numbers, the one with the higher term is more up-to-date. If the term numbers are equal, the longer log wins. A follower that has seen more recent committed entries will refuse to vote for a candidate that hasn't. This single rule is what guarantees that whoever wins an election always has the most complete view of committed history in the cluster. No committed entry can be lost in a leadership transition because no node missing that entry can ever collect enough votes to win.
The winner needs a simple majority, more than half the cluster. In a five-node cluster, three votes win. The moment a candidate crosses that threshold, it declares itself leader, starts sending heartbeats immediately, and begins accepting writes.
The split vote problem and why randomization solves it
If two followers notice the leader's silence simultaneously, both become candidates and split the available votes. Neither reaches the majority. Neither becomes a leader.
Raft resolves this with randomized election timeouts. Each follower waits for a random interval, typically 150 to 300 milliseconds before becoming a candidate. In practice, one node's timer almost always fires before the others, giving it enough of a head start to collect votes before competition begins. When a split does occur, the same randomization resolves it in the next round. No coordination mechanism, no special tiebreaker, just statistical separation doing the work.
What election behavior tells you in production
A healthy cluster elects a leader once, at startup, and never again unless something changes. If you are watching etcd_server_leader_changes_seen_total climb on a running cluster, that is worth investigating immediately.
The most common cause is not a crashed node. It is latency, a disk too slow to flush write-ahead log entries within the heartbeat interval, or a network with enough jitter to occasionally drop heartbeats. The leader is not dying. It is failing to prove it is alive fast enough, which from the cluster's perspective is the same thing.
etcd's default election timeout is 1000ms with a heartbeat interval of 100ms. That ten-to-one ratio gives the leader enough runway to miss occasional heartbeats without triggering a false election. Tightening it makes failure detection faster but makes the cluster sensitive to any transient latency spike. It is not a performance knob. It is a stability knob, and it should only be changed when you understand exactly what your infrastructure's latency floor looks like under load.
⠀
Log Replication
Knowing who the leader is solves only half the problem. The other half is what the leader actually does with that authority. specifically, how it takes a client write and makes sure that write survives node failures, network drops, and leadership changes without ever being lost or applied out of order.
That mechanism is log replication, and it is the core of what Raft actually does day to day.
The log is not a backup. It is the truth.
Every node in a Raft cluster maintains a log, an ordered sequence of entries, each one representing a state change. The state machine, whatever it is a key-value store, a lock service, a secrets manager derives its entire state by replaying this log from the beginning. The log is not a record of what happened. It is what happened. Two nodes with identical logs are guaranteed to have identical state, regardless of how they got there.
This is why replication is about preserving order, not just copying data. A write that arrives in the wrong position in the log produces a different state machine outcome than the same write in the correct position. Raft does not just move bytes between nodes. It enforces a single authoritative sequence.
What happens when a write arrives
A client sends a write to the leader. The leader does not apply it to its state machine immediately. Instead it appends the entry to its own log and sends an AppendEntries RPC to every follower in parallel, the same RPC it uses for heartbeats, just now carrying actual content.
Each follower that receives the entry appends it to its own log and sends an acknowledgment back to the leader. The leader waits. Once a majority of nodes, including itself have acknowledged the entry, the leader considers it committed. It applies the entry to its state machine, returns a response to the client, and includes the updated commit index in its next AppendEntries so followers know they can apply it too.
The client never hears success until the entry is committed. Committed means a majority of the cluster has it. That guarantee is what makes the write durable even if the leader crashes immediately after responding, the entry exists on enough nodes that the next leader will inherit it.
How followers stay consistent
Every AppendEntries message carries two pieces of context: the index and term of the entry immediately preceding the new one. Before a follower appends anything, it checks whether its own log matches at that position. If it does, the follower appends and acknowledges. If it doesn't, if there's a gap, or a conflicting entry from an old term, the follower rejects the message.
When a follower rejects, the leader steps back one entry and tries again, walking backward through the log until it finds a point of agreement. From that point, it resends everything forward. This process called the consistency check ensures that no follower ever has a gap in its log, and that every committed entry appears in exactly the same position on every node that has it.
Raft does not just ensure that every node eventually gets every entry. It ensures every node gets every entry in the same order, with no holes.
Where it gets complicated
Consider a leader that appends an entry to its own log and sends AppendEntries to its followers, then crashes before any acknowledgment comes back. The entry is on the leader's disk. The followers never got it. The new leader that gets elected has no knowledge of this entry. It is simply gone.
This is acceptable. The original leader never committed the entry, it never reached majority. so no client ever received a success response for it. The write is lost from the cluster's perspective, but from the client's perspective the write simply failed. The client can retry.
The harder case is a leader that replicates an entry to a majority, commits it, responds to the client and then crashes before sending the updated commit index to its followers. Those followers have the entry in their logs but don't know it's been committed. The new leader, however, will have that entry. it was on a majority, so the election safety rule guarantees the winner has it. The new leader will re-replicate it as part of normal operation, and followers will eventually learn it is committed. Nothing is lost. The math holds.
What log replication looks like in etcd
In etcd, every write to the Kubernetes API server creating a pod, updating a configmap, scaling a deployment goes through this exact path. The API server writes to etcd, etcd's leader appends to its Raft log, replicates to followers, waits for majority acknowledgment, commits, and only then confirms the write back up the chain.
This is why etcd write latency is sensitive to disk speed. The bottleneck is not CPU or network it is the time it takes to flush a log entry to disk on a majority of nodes. A slow disk on any single node in a three-node cluster can become the bottleneck for every write in your entire Kubernetes cluster. This is why etcd's documentation recommends SSDs almost as a requirement, not a suggestion. The sequential write performance of your storage directly determines the write throughput of your control plane.
⠀
Failure Scenarios
Everything described so far assumes the happy path. A stable leader, cooperative followers, a network that delivers messages. Real clusters don't stay that way. Understanding Raft deeply means understanding what happens when things break, because that's when the protocol's guarantees are actually being earned.
The leader crashes mid-replication
The most common failure. The leader has accepted a write, appended it to its log, sent AppendEntries to followers then dies. There are two versions of this depending on how far the write got.
If the entry never reached a majority, it was never committed. No client received a success response. The new leader that gets elected simply doesn't have the entry, and it never surfaces again. The client retries and the world moves on.
If the entry reached a majority before the crash, it is committed by definition even if the leader died before telling anyone. The election safety rule guarantees that the next leader will have been on that majority, which means it has the entry. It will replicate it forward as part of normal operation. The cluster converges. The entry survives.
The write either made it to majority or it didn't. There is no middle ground where a committed entry disappears.
The leader crashes after committing but before responding to the client
The entry is committed. The cluster has it. But the client never got its acknowledgment and has no way to know whether the write succeeded. From the client's perspective the request just timed out.
This is not a Raft problem. It is a distributed systems reality. Raft guarantees that committed writes survive. It does not guarantee exactly-once delivery to clients. The standard solution is idempotent writes, the client retries with the same operation, and the system is designed to recognize and discard duplicates. etcd handles this through revision numbers. Kubernetes controllers handle it through reconciliation loops that re-apply desired state regardless of whether the previous attempt succeeded.
Network partition
A partition splits the cluster into two groups that cannot communicate. What happens depends entirely on the sizes.
If the partition creates a minority and a majority. say, two nodes on one side and three on the other in a five-node cluster. The majority side continues operating normally. The minority side cannot commit any writes because it cannot reach quorum. If the old leader happens to be on the minority side, it keeps receiving client writes, appending them to its log, sending AppendEntries into the void and nothing commits. The majority side elects a new leader and moves forward.
When the partition heals, the old leader receives a message from the new leader carrying a higher term. It immediately steps down. Its uncommitted entries are overwritten by the new leader's log. Any client that was routed to the old leader during the partition and received no acknowledgment simply retries. The minority side was never able to commit anything. The majority side's history is the only history that counts.
The split-brain that can't happen
Engineers new to Raft sometimes worry about a scenario where both sides of a partition elect a leader and both start committing writes, producing two diverging histories. This cannot happen. Committing requires majority acknowledgment. A minority partition cannot form a majority. By definition, only one side of any partition can have a majority, and therefore only one side can ever commit. Two leaders can exist simultaneously, Raft doesn't prevent that but only one of them can do anything meaningful. The other is shouting into a void.
| Scenario | Outcome |
|---|---|
| Entry never reaches majority | Lost, client retries |
| Entry reaches majority, leader crashes | Survives, next leader inherits it |
| Leader on minority partition | Cannot commit, overwritten on heal |
| Two leaders during partition | Only majority side can commit |
| Slow leader misses heartbeat window | Replaced by election |
| Returning node with stale log | Rebuilt from leader, read-only until current |
A slow leader
A leader that is alive but slow due to disk latency, GC pauses, CPU contention stops sending heartbeats fast enough. Followers time out and start an election. The slow leader loses its position not because it crashed but because it failed the responsiveness test.
This is correct behavior. A leader that cannot respond in time is operationally indistinguishable from a dead one. What it produces in practice is leadership churn, frequent re-elections on clusters under resource pressure. If your etcd metrics show leader changes correlated with high disk I/O or memory pressure on the leader node, this is exactly what's happening. The fix is not to tune the election timeout. The fix is to address the resource pressure.
A returning node with a stale log
A node that was offline for minutes or hours comes back with a log that is behind the current cluster state. It rejoins as a follower, and the leader begins sending it AppendEntries from the point of divergence forward. The consistency check walks back to where their logs agree, then the leader ships everything forward from that point.
If the node was offline long enough that log compaction has already removed the entries it needs, the leader sends it a snapshot instead, a complete serialized state of the system at a specific point and the node rebuilds from there. The returning node never influences cluster state during catch-up. It is read-only until it is current. It can never overwrite newer data with its stale copy. Its log is behind, its term is lower, and it cannot win an election. The direction of log repair is always one way: from the leader outward.
⠀
Raft in Production
Understanding Raft in the abstract is one thing. Seeing where it lives in your actual infrastructure is another. Most engineers working with Kubernetes, secrets management, or service discovery are already running Raft in production. they just don't always know it by name.
etcd
etcd is the most visible Raft deployment in the modern infrastructure stack. It is the only persistent store for Kubernetes cluster state. Every pod definition, every secret, every configmap, every service, every node registration lives in etcd. When etcd is unhealthy, Kubernetes is unhealthy. The API server can't schedule pods. Controllers can't reconcile state. The cluster doesn't crash immediately, but it stops being able to make decisions.
etcd runs a three or five node Raft cluster depending on the deployment. Every write from the Kubernetes API server and there are many, even on a quiet cluster goes through the full Raft commit path. Append to leader log, replicate to majority, commit, respond. This is why etcd is so sensitive to disk latency. The write-ahead log needs to be flushed to disk on a majority of nodes before any write is acknowledged. A single slow node in a three-node cluster becomes the bottleneck for the entire Kubernetes control plane.
The operational implication is concrete: etcd should run on dedicated nodes with fast local SSDs, isolated from workloads that compete for I/O. Running etcd on shared nodes with application workloads is one of the most common causes of subtle, hard-to-diagnose Kubernetes instability not crashes, just intermittent slowness and occasional leader churn that seems to have no cause.
HashiCorp Vault
Vault added integrated Raft storage in version 1.4 as an alternative to Consul backend. Before that, running Vault in high availability required operating a separate Consul cluster just to give Vault somewhere to store its state. Integrated storage collapsed that dependency. Vault now manages its own Raft cluster internally.
The practical difference this makes is significant. A Vault cluster with integrated storage is self-contained. There is no external system to operate, no separate failure domain to monitor, no Consul outage that takes down your secrets management. What integrated storage means operationally is that Vault's availability is now governed by the same quorum math as everything else. A three-node Vault cluster tolerates one node failure. A five-node cluster tolerates two. If you are running a single Vault node which many smaller deployments do there is no Raft, no replication, and no fault tolerance. A single node failure takes down secrets management for everything that depends on it.
Consul
Consul's use of Raft is slightly different because the data it manages has different characteristics. etcd stores relatively small, high-value configuration objects. Vault stores secrets with strict access controls. Consul stores service registrations, health check results, and key-value configuration that can change at high frequency as services come and go.
Consul partitions its data into distinct areas, the default datacenter catalog, ACL tokens, and others each managed by its own Raft log. This keeps log sizes manageable and allows different types of data to have different consistency characteristics. The architecture is more complex than etcd's, but the underlying consensus mechanism is identical.
The operational lesson Consul surfaces clearly is around datacenter topology. Consul supports multi-datacenter deployments, but Raft consensus is strictly per-datacenter. Cross-datacenter replication is eventual, not consistent. Engineers who assume they are getting strong consistency guarantees across datacenters because Consul "uses Raft" are mistaken. Raft's guarantees are scoped to a single Raft group, and Consul's cross-datacenter replication is outside that boundary.
⠀
Conclusion
Raft is not an academic curiosity. It is running right now inside the infrastructure most engineers interact with every day, making thousands of small decisions about what is true and what should be committed.
What makes it worth understanding deeply is not the algorithm itself though the algorithm is elegant, it is what understanding it changes about how you read systems. Leader churn in etcd stops being a mysterious metric and becomes a legible signal about disk latency. A Vault node failing stops being an availability incident and becomes a quorum calculation. A Kubernetes API server timing out stops being a networking problem and starts being a question about what happened to the write-ahead log on the etcd leader.
Distributed systems fail in ways that are subtle and frequently misread. The engineers who diagnose them fastest are not the ones with the most monitoring. They are the ones who understand the protocol well enough to work backward from symptoms to causes, to look at a misbehaving cluster and know, before opening any dashboard, what the system is actually trying to do.
Raft gives you that. A small, understandable algorithm that, once internalized, makes an entire class of production behavior suddenly readable.
⠀
