Flat Datacenter Storage
📜 Abstract
Flat Datacenter Storage (FDS) is a high-performance, fault-tolerant, large-scale, locality-oblivious blob store. Using a novel combination of full bisection bandwidth networks, data and metadata striping, and flow control, FDS multiplexes an application’s large-scale I/O across the available throughput and latency budget of every disk in a cluster. FDS therefore makes many optimizations around data locality unnecessary. Disks also communicate with each other at their full bandwidth, making recovery from disk failures extremely fast. FDS is designed for datacenter scale, fully distributing metadata operations that might otherwise become a bottleneck. FDS applications achieve single-process read and write performance of more than 2GB/s. We measure recovery of 92GB data lost to disk failure in 6.2 s and recovery from a total machine failure with 655GB of data in 33.7 s. Application performance is also high: we describe our FDS-based sort application which set the 2012 world record for disk-to-disk sorting.
✨ Summary
Summary
Flat Datacenter Storage presents a locality-oblivious blob store designed for datacenters with full-bisection-bandwidth networks. It divides blobs into fixed-size tracts, uses tractservers that access raw disks directly, and places tracts deterministically through a cached tract locator table. This avoids putting individual data locations on the critical path of reads and writes while distributing both data and per-blob metadata across the storage cluster.
The design combines uniform striping, asynchronous I/O, deep read-ahead/write-ahead, and application-level RTS/CTS flow control. Because computation is independent of storage location, applications can dynamically assign small units of work to workers, reducing the effect of heterogeneous hardware and stragglers. Replication is integrated into the locator table, and failures trigger table-version changes and parallel re-replication across the remaining disks. The paper also discusses failure domains, cluster growth, variable replication, and the trade-off between high performance and weak consistency.
In a testbed of approximately 1,000 disks, FDS achieved aggregate remote read and write throughput of up to 67 GB/s and single-process throughput above 2 GB/s. It recovered 92 GB after a disk failure in 6.2 seconds and approximately 655 GB after a seven-disk machine failure in 33.7 seconds. An FDS-based sorting system processed 1,401 GB in 59 seconds in the general-purpose Daytona category and 1,470 GB in 59.4 seconds in the Indy category, establishing the 2012 MinuteSort records. The evaluated applications also included stock-market-data analysis and web-index serving.
Influence
The paper’s architectural ideas were used directly in Blizzard, a later Microsoft Research system for cloud-scale block storage. Blizzard selected FDS as its low-level storage substrate and adopted the use of full-bisection-bandwidth networking and related congestion-control techniques to provide high-performance remote block storage to unmodified POSIX and Win32 applications. (microsoft.com)
Subsequent distributed-file-system literature treats FDS as a representative example of shared-disk-like storage and computed data placement, particularly for systems operating over flat datacenter networks. A later survey identifies FDS’s deterministic placement and load-balancing techniques as notable extensions of this design approach. (doi.org) The work also had a documented industry-facing impact through its 2012 MinuteSort records, which demonstrated that remote, general-purpose blob storage could achieve competitive large-scale data-processing performance without relying on storage locality. (microsoft.com)