A Solution to the Network Challenges of Data Recovery in Erasure-coded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster
📜 Abstract
Erasure codes, such as Reed-Solomon (RS) codes, are being increasingly employed in data centers to combat the cost of reliably storing large amounts of data. Although these codes provide optimal storage efficiency, they require significantly high network and disk usage during recovery of missing data. In this paper, we first present a study on the impact of recovery operations of erasure-coded data on the data-center network, based on measurements from Facebook’s warehouse cluster in production. To the best of our knowledge, this is the first study of its kind available in the literature. Our study reveals that recovery of RS-coded data results in a significant increase in network traffic, more than a hundred terabytes per day, in a cluster storing multiple petabytes of RS-coded data. To address this issue, we present a new storage code using our recently proposed Piggybacking framework, that reduces the network and disk usage during recovery by 30% in theory, while also being storage optimal and supporting arbitrary design parameters. The implementation of the proposed code in the Hadoop Distributed File System (HDFS) is underway. We use the measurements from the warehouse cluster to show that the proposed code would lead to a reduction of close to fifty terabytes of cross-rack traffic per day.
✨ Summary
The paper documents the network cost of repairing Reed–Solomon-coded data at Facebook’s warehouse cluster and proposes Piggybacked-RS codes. The production measurements reported a median of more than 95,500 recovered blocks and more than 180 TB of cross-rack traffic per day. Piggybacked-RS was designed to preserve MDS storage efficiency and failure tolerance while reducing recovery disk reads and network downloads by approximately 30% in theory; the paper estimated a reduction exceeding 50 TB of cross-rack traffic per day.
Subsequent work treated the paper as an early systems study motivating repair-bandwidth-aware erasure coding. Later research cited Piggybacked-RS in discussions of recovery-efficient codes, including work on fast reconstruction in erasure-coded data centers and studies of erasure coding in clustered file systems. (doi.org) A recent survey of erasure coding identifies this line of work as part of the research shift toward reducing recovery bandwidth while retaining storage efficiency. (doi.org) The work was presented at HotStorage ’13, and the authors’ proposed HDFS implementation and production-scale evaluation were identified as future work at the time. (usenix.org)