RADOS: A Scalable, Reliable Storage Service for Petabyte-scale Storage Clusters
📜 Abstract
Brick and object-based storage architectures have emerged as a means of improving the scalability of storage clusters. However, existing systems continue to treat storage nodes as passive devices, despite their ability to exhibit significant intelligence and autonomy. We present the design and implementation of RADOS, a reliable object storage service that can scales to many thousands of devices by leveraging the intelligence present in individual storage nodes. RADOS preserves consistent data access and strong safety semantics while allowing nodes to act semi-autonomously to self-manage replication, failure detection, and failure recovery through the use of a small cluster map. Our implementation offers excellent performance, reliability, and scalability while providing clients with the illusion of a single logical object store.
✨ Summary
Summary
RADOS is a distributed object-storage service designed for clusters ranging from dozens to many thousands of storage devices. Its central design principle is to make storage nodes active participants in cluster management rather than passive responders to client requests.
The system maintains a versioned cluster map replicated by clients, monitors, and storage nodes. The map records membership, device state, placement-group configuration, and CRUSH placement rules. Objects are first assigned to placement groups and then mapped to ordered sets of OSDs using CRUSH. This two-stage placement scheme avoids a large per-object metadata directory, supports heterogeneous devices and failure domains, and limits data movement when the cluster changes.
RADOS distributes replication, failure detection, peering, migration, and recovery across OSDs. It supports primary-copy, chain, and splay replication while preserving serializable read/write semantics. Map epochs attached to messages prevent stale clients or OSDs from operating incorrectly after topology changes. Heartbeats and coordination among replicas address the consistency risks created by partial failures and delayed failure detection.
Failure recovery is organized independently for each placement group. During peering, OSDs exchange metadata and log information to determine the authoritative state of a placement group. Recovery then proceeds in parallel across many OSDs; the primary coordinates missing-object transfers and attempts to read each object only once during re-replication. A small monitor cluster uses a Paxos-based service to maintain the authoritative cluster map, while incremental map updates are propagated lazily through OSD communication. The paper’s evaluation focuses primarily on map propagation and reports that update-duplication overhead approaches a bounded level as the cluster grows, although monitor-cluster performance was not experimentally evaluated.
The paper directly established architectural foundations later incorporated into Ceph. Current Ceph documentation retains the RADOS concepts of object-to-placement-group mapping, CRUSH-based placement, OSD primaries, peering, and parallel recovery. These mechanisms remain part of Ceph’s documented architecture for petabyte-scale storage clusters. (ssrc.us)
The paper also helped define a research direction centered on decentralized placement and autonomous storage-node management. Its proposed extensions—including key-value objects, snapshots, load balancing, quality of service, and parity-based redundancy—anticipated functionality and research themes subsequently developed around Ceph and related distributed-storage systems. The paper’s publication metadata is confirmed by the authors’ research-group record and bibliographic listings. (ssrc.us)