Spanner: Google’s Globally-Distributed Database
📜 Abstract
Spanner is Google’s scalable, multi-version, globally-distributed, and synchronously-replicated database. It is the first system to distribute data at global scale and support externally-consistent distributed transactions. This paper describes how Spanner is structured, its feature set, the rationale underlying various design decisions, and a novel time API that exposes clock uncertainty. This API and its implementation are critical to supporting external consistency and a variety of powerful features: non-blocking reads in the past, lock-free read-only transactions, and atomic schema changes, across all of Spanner.
✨ Summary
Summary
Spanner presents a globally distributed database that combines automatic sharding, synchronous replication through Paxos, multi-version storage, a semi-relational schema, SQL-like queries, and general-purpose distributed transactions. Data is organized into tablets and Paxos groups, while directories provide a unit for data placement, replication policy, locality control, and online movement across servers and datacenters. The schema’s interleaving mechanism lets applications colocate related rows and thereby influence locality and transaction performance.
The paper’s central contribution is TrueTime, an API that returns a bounded interval rather than an apparently exact clock reading. Spanner uses this uncertainty bound to assign globally meaningful commit timestamps and to enforce external consistency through commit-wait. This enables serializable distributed transactions whose timestamp order agrees with real-time ordering, lock-free read-only transactions, globally consistent historical reads, and largely non-blocking atomic schema changes. Replicas track safe timestamps so that reads can be served at appropriate historical versions without acquiring locks.
The evaluation shows that replication can preserve roughly stable write latency in the tested local-datacenter configuration, while snapshot-read throughput increases with the number of replicas. Two-phase commit remains practical for moderate participant counts, with noticeable latency growth at 100 or more participants in the reported experiments. The F1 advertising backend demonstrates the operational motivation: Spanner removed manual resharding, supplied synchronous replication and failover, and supported transactions across arbitrarily related data.
Influence on Research and Industry
The paper directly motivated subsequent work on Spanner’s database layer. A later Google paper described its evolution toward a full SQL system, including distributed query execution, query routing, resharding-aware execution, and improved storage formats. (research.google) Google also reports that Firestore was implemented over Spanner because of its unrestricted transactions and strong consistency guarantees; a later migration paper states that more than one million Datastore databases had been migrated to Firestore by the time of publication. (research.google)
The architecture influenced independent distributed-SQL systems. CockroachDB’s developers explicitly describe its design as inspired by Spanner and discuss adapting the external-consistency problem to commodity hardware without TrueTime’s specialized clock infrastructure. (cockroachlabs.com) Google Cloud’s current Spanner documentation continues to expose the paper’s core ideas—TrueTime-based timestamps, MVCC, snapshot reads, and external consistency—as fundamental service semantics. (docs.cloud.google.com)