Distributed systems: software across connected computers

Understand how services coordinate over networks, why failures are unavoidable, and how replication, consistency, and careful design help systems remain useful at scale.

What makes a system distributed

A distributed system is a collection of independent computers that communicate over a network and cooperate to provide a service. A user may see one application, even though its work is handled by many processes on different machines.

Distribution can improve capacity, availability, and geographic reach. It also introduces new problems: machines do not share one memory, communication takes time, networks can fail, and the system cannot always know whether a remote component is slow, disconnected, or stopped.

• Nodes: computers or processes that participate in the system.
• Messages: requests and responses exchanged between nodes.
• State: data that components must read, update, or replicate.
• Failures: unavailable nodes, lost messages, delays, or incorrect behavior.

A program split into modules on one computer is not necessarily a distributed system. Distribution begins when separate execution components communicate across a boundary where timing and failure are no longer fully under one program's control.

Communication and partial failure

A network request has a cost: serialization, transmission, queuing, processing, and return travel all take time. A response timeout means the caller stopped waiting; it does not prove that the server did not receive or complete the request.

This creates uncertainty. If a client submits an order and the response is lost, retrying could create a second order. The server may have completed the first attempt, or the request may never have arrived. Applications need operation identifiers, idempotency rules, or status checks where duplicate effects matter.

Timeouts and retries

Every remote call should have an appropriate timeout so a failure does not consume resources forever. A retry can recover from a temporary issue, but repeated immediate retries can overload a struggling service. Exponential backoff with jitter spaces attempts and reduces synchronized retry spikes.

Retries should be bounded and used only when repeating the operation is safe or protected against duplication. A timeout policy also needs a user-facing outcome: report uncertainty honestly when the system cannot confirm whether an operation finished.

Failure domains

A failure domain is a group of components that can fail together, such as a machine, rack, region, or shared network link. Replicas on the same machine do not protect against that machine failing. Resilience depends on understanding shared dependencies such as power, credentials, DNS, and software deployments.

Clocks, ordering, and causality

Computers have clocks, but independent clocks drift and can disagree. Timestamps from different machines do not always provide a reliable total ordering of events. Network delays further complicate knowing what happened first.

Some systems use synchronized wall clocks for approximate time, while logical clocks track causal relationships. If event A causes event B, then A happened before B in the causal sense. Concurrent events may have no single meaningful order unless the application introduces coordination.

For example, two users may edit the same document while temporarily disconnected. Both changes can be valid independently, but the system needs a rule to merge them or resolve a conflict. “Use the latest timestamp” is one possible policy, but clock skew can cause it to select the change that was not actually later.

Replication and availability

Replication keeps copies of data or services on multiple nodes. Replicas can share traffic, improve read capacity, and survive some failures. They also need to coordinate updates and recover when a replica falls behind.

A primary-replica design routes writes to one primary and copies changes to replicas. Synchronous replication waits for selected replicas before confirming a write, which can improve durability or consistency but increase latency. Asynchronous replication responds sooner but may lose recent changes if the primary fails before replication completes.

Failover is a correctness decision

Failover moves work from an unhealthy component to another. Automatically promoting a replica can restore availability, but if the old primary is still accepting writes, the system may create two conflicting leaders—a split-brain situation. A safe design uses a clear authority or fencing mechanism so only one component can act as primary at a time.

Replication is not a backup. Replicas often copy accidental deletion and corrupted writes quickly. Backups preserve recovery points; replicas primarily help serve data and recover from component failures.

Consistency models

Consistency describes what users can observe when data is read and written across replicas. Strong consistency may require reads to reflect the latest completed write, which can require coordination and waiting. Eventual consistency allows replicas to temporarily disagree but promises they will converge if updates stop and communication recovers.

Eventual consistency can be a good fit for a like counter or activity feed where a brief delay is acceptable. It may be inappropriate for a bank balance or inventory allocation without additional rules that prevent overspending or overselling.

Consistency is chosen per requirement

One application can use different guarantees for different features. A profile display may tolerate stale data, while changing account permissions should take effect reliably before another protected action is allowed. Decide what users must observe, then select storage and coordination patterns that support it.

Some databases offer tunable consistency: a client chooses how many replicas must confirm a read or write. These settings affect latency, availability, and the chance of stale results. Read the system's actual guarantees; product labels alone do not describe every failure scenario.

CAP is about a network partition

The CAP result is often summarized as a trade-off among consistency, availability, and partition tolerance. More precisely, when a network partition prevents nodes from communicating, a system that continues responding on both sides cannot always guarantee that every read reflects the latest write. A design may reject or delay some operations to preserve stronger consistency, or return results while allowing temporary disagreement.

Networks can partition in real deployments, so systems must define their behavior during that condition. CAP does not mean every database simply chooses two properties forever, nor does it describe all latency and consistency trade-offs. It highlights one impossibility boundary under a particular failure condition.

Coordination, leaders, and consensus

Distributed components sometimes need to agree on a decision, such as which node is the leader or which updates belong in a single ordered log. Consensus protocols help a group of nodes agree despite certain failures, under specific assumptions about communication and the number of faulty nodes.

Consensus is not a magic way to eliminate network problems. It typically requires a quorum of participants to communicate, and it may stop making progress when too many are unreachable. A system may choose consistency over availability for a particular operation during a partition, or allow limited operations and reconcile later.

Leader election and leases

A leader can simplify coordination by making ordering decisions. If the leader becomes unreachable, the system may elect a new one. But delayed messages can make the old leader believe it is still active. Epoch numbers, leases with carefully defined assumptions, or fencing tokens help prevent an outdated leader from changing shared state.

When using a database or coordination service, rely on its documented guarantees rather than implementing a homegrown election protocol. Coordination mechanisms are subtle and errors can appear only during rare timing failures.

Partitioning and scaling

Scaling means supporting more users, data, or work. Vertical scaling gives a machine more resources; horizontal scaling adds machines. Horizontal scaling can increase capacity but requires traffic distribution, coordination, monitoring, and failure handling.

Partitioning (or sharding) divides data among nodes, often using a key such as user ID. A good partition key distributes work evenly and keeps related queries manageable. A poor key can create a hot spot where one node receives much more traffic than others.

Rebalancing partitions as capacity changes can move large amounts of data and disrupt traffic. Cross-partition queries and transactions may be more expensive than operations confined to one partition. Design partitioning around measured access patterns and expected growth rather than guessing too early.

Queues and asynchronous work

A message queue lets one component accept work and another process it later. Queues can absorb bursts and isolate temporary slowdowns. They also introduce delivery semantics: a message might be delivered more than once, delayed, or placed in a dead-letter queue after repeated failures.

Consumers should be idempotent when duplicate delivery is possible. Track processing status, make retries bounded, and monitor queue age as well as length. A growing queue can indicate that workers are not keeping up even when request endpoints appear healthy.

Delivery guarantees and duplicate work

Messaging systems commonly describe delivery as at-most-once, at-least-once, or exactly-once within a defined scope. At-most-once avoids duplicates but can lose work; at-least-once retries until acknowledged but may deliver duplicates. “Exactly once” usually depends on coordinated state changes within the messaging system or a carefully limited transaction boundary. It does not automatically guarantee that an external payment, email, or third-party API happens exactly once.

A practical consumer records a stable operation ID and makes repeated processing safe. For example, before granting a reward, check whether that event ID has already been applied. This is an application-level guarantee that must be designed together with storage, acknowledgment, and retry behavior.

Design for failure

Reliable distributed systems expect some component to be unavailable at some point. Use explicit timeouts, bounded retries, health checks, and clear degraded behavior. Avoid cascading failure: when one service slows down, unlimited requests from dependent services can make the slowdown spread.

1. Define a service's important user-visible behaviors and reliability goals.
2. Identify dependencies and shared failure domains.
3. Set timeouts and retry policies that fit each operation.
4. Protect critical dependencies with queues, limits, or circuit breakers where appropriate.
5. Monitor latency, errors, saturation, and data freshness.
6. Test recovery procedures and practice failure scenarios safely.

A circuit breaker temporarily stops calls to a failing dependency so resources are not exhausted by repeated requests. It can later allow a limited probe to see whether the dependency has recovered. It is not a substitute for fixing the underlying problem or returning a meaningful result.

Compact glossary

Partial failure

A failure in which some components continue working while others do not.

Replication

Maintaining copies of data or service state on multiple nodes.

Consistency model

A description of which values reads may observe after updates.

Quorum

A required subset of participants whose agreement supports an operation.

Idempotent operation

An operation that can be repeated without changing its intended effect.

Partition

A division of data or nodes, or a network split that prevents communication.

References and further reading

1. Martin Kleppmann. Designing Data-Intensive Applications. O'Reilly Media, 2017. Book information.
2. Maarten van Steen and Andrew S. Tanenbaum. Distributed Systems, 3rd edition. Free book resources.
3. Google. Site Reliability Engineering. Free online book.
4. Jepsen. Distributed Systems Safety Research. Testing and analysis resources.

Try it yourself

1. A client times out after creating an order. Explain why it cannot know from the timeout alone whether the order exists.
2. Choose a data type that can tolerate eventual consistency and one that probably cannot.
3. Why are replicas on one physical machine not enough to survive a machine failure?
4. Describe how a retry policy could worsen an overloaded service.
5. Design a partition key for a simple messaging app and discuss one way it could become a hot spot.

Remember: networks create uncertainty. Make failure behavior explicit, match consistency to user needs, and test recovery as part of the design.