Sharding
Sharding is a database scaling technique that splits a large dataset horizontally across multiple independent database instances (shards), each holding a subset of the total data, so that no single machine needs to store or serve the…
Definition
Sharding is a database scaling technique that splits a large dataset horizontally across multiple independent database instances (shards), each holding a subset of the total data, so that no single machine needs to store or serve the entire dataset.
Overview
Where vertical scaling adds more CPU, memory, or storage to a single database server, sharding scales horizontally by distributing rows across multiple servers based on a shard key — a column or set of columns (like a user ID or geographic region) whose value determines which shard a given row lives on. A query touching only one shard's data can be served quickly by that shard alone, but queries spanning multiple shards (like an aggregate across all users) become more complex, often requiring a coordinating layer to fan out and merge results. Common sharding strategies include range-based sharding (dividing data by value ranges, e.g., user IDs 1–1,000,000 on shard A), hash-based sharding (distributing rows evenly using a hash of the shard key to avoid hotspots), and directory-based sharding (a lookup service maps keys to shards, allowing more flexible rebalancing). Choosing a good shard key is one of the hardest parts of sharding: a poor choice can create "hot shards" that receive disproportionate traffic, undermining the whole point of distributing load. Sharding is distinct from, but often combined with, database replication — replication copies the same data across multiple nodes for redundancy and read scaling, while sharding partitions different data across nodes for write scaling and storage capacity. Distributed SQL databases like CockroachDB, YugabyteDB, and TiDB automate sharding internally, while systems like MongoDB and Vitess for MySQL expose sharding as a more explicit, configurable feature. Because resharding an already-live, unevenly distributed dataset is operationally difficult — it typically requires carefully migrating data between shards with minimal downtime — teams are generally advised to delay sharding until simpler techniques like read replicas, caching, and query optimization are no longer sufficient, a topic explored in depth in courses like PostgreSQL Mastery.
Key Concepts
- Horizontal partitioning of data across multiple independent database instances
- Shard key determines which shard a given row is stored on
- Range-based, hash-based, and directory-based sharding strategies
- Enables write scaling and storage growth beyond a single server's limits
- Cross-shard queries require coordination and result merging
- Distinct from but often paired with database replication
- Automated in distributed SQL databases like CockroachDB and YugabyteDB
- Resharding a live, unevenly distributed dataset is operationally challenging