Ceph: A Scalable, High-Performance Distributed File System
📜 Abstract
We have developed Ceph, a distributed file system that provides excellent performance, reliability, and scalability. Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable object storage devices (OSDs). We leverage device intelligence by distributing data replication, failure detection and recovery to semi-autonomous OSDs running a specialized local object file system. A dynamic distributed metadata cluster provides extremely efficient metadata management and seamlessly adapts to a wide range of general purpose and scientific computing file system workloads. Performance measurements under a variety of workloads show that Ceph has excellent I/O performance and scalable metadata management, supporting more than 250,000 metadata operations per second.
✨ Summary
Summary
The paper presents Ceph as a distributed, near-POSIX file system designed for petabyte-scale and highly dynamic storage clusters. Its architecture separates namespace metadata from file-data I/O: metadata operations are handled by a dynamically partitioned metadata-server cluster, while clients communicate directly with object storage devices for data access.
The central technical contribution is the replacement of explicit allocation tables with CRUSH, a deterministic pseudo-random placement function. Clients, metadata servers, and storage devices can independently calculate object locations from object identifiers and the cluster map. This reduces metadata overhead, limits data movement when devices are added or removed, supports weighted placement, and enables replica placement across failure domains.
The object store, RADOS, delegates replication, failure detection, recovery, and cluster adaptation to semi-autonomous storage devices. The prototype separates update visibility from durable disk commitment: writes are acknowledged after replication into OSD memory and receive a later commit notification after safe persistence. Ceph also introduces Dynamic Subtree Partitioning, which moves metadata subtrees among metadata servers according to recent workload and selectively replicates or hashes hot directories.
The reported prototype results include nearly linear OSD throughput scaling until network saturation and more than 250,000 metadata operations per second with a 128-node metadata-server cluster. The evaluation also identifies limitations, including imperfect metadata scaling under heavily shared read workloads, prototype-level client and FUSE issues, and unimplemented metadata-server recovery and security mechanisms.
Influence on research and industry
The work became the foundational research publication for the Ceph storage system. The Ceph project’s historical account states that the paper appeared at OSDI in 2006, that Ceph code was released as open source under the LGPL, and that the design subsequently evolved into a production storage platform used in cloud, research, and enterprise environments. (ceph.com)
Its architectural ideas were carried into later Ceph development, including the RADOS object-storage model and CRUSH-based placement. Subsequent systems research continued to analyze Ceph’s design and operational experience; for example, a later SOSP paper explicitly studies lessons from a decade of Ceph evolution and the suitability of conventional file systems as distributed-storage backends. (pdl.cmu.edu)
In industry, Ceph became integrated with OpenStack as a backend for block and image storage, with official OpenStack documentation describing Ceph RBD use for services including Glance, Cinder, and Nova. (docs.openstack.org)