f4: Facebook’s Warm BLOB Storage System
📜 Abstract
Facebook’s corpus of photos, videos, and other Binary Large OBjects (BLOBs) that need to be reliably stored and quickly accessible is massive and continues to grow. As the footprint of BLOBs increases, storing them in our traditional storage system, Haystack, is becoming increasingly inefficient. To increase our storage efficiency, measured in the effective-replication-factor of BLOBs, we examine the underlying access patterns of BLOBs and identify temperature zones that include hot BLOBs that are accessed frequently and warm BLOBs that are accessed far less often. Our overall BLOB storage system is designed to isolate warm BLOBs and enable us to use a specialized warm BLOB storage system, f4. f4 is a new system that lowers the effective-replication-factor of warm BLOBs while remaining fault tolerant and able to support the lower throughput demands. f4 currently stores over 65PBs of logical BLOBs and reduces their effective-replication-factor from 3.6 to either 2.8 or 2.1. f4 provides low latency; is resilient to disk, host, rack, and datacenter failures; and provides sufficient throughput for warm BLOBs.
✨ Summary
The paper presents f4, Facebook’s storage system for “warm” immutable BLOBs whose access rates have declined but which must remain quickly retrievable. It separates warm data from newly created, high-throughput data handled by Haystack, using a router, caching, transformation, and migration architecture to make the storage tiers interchangeable.
f4 stores locked volumes with Reed–Solomon(10,4) erasure coding across racks within a datacenter and geo-replicated XOR coding across datacenters. Deletes are implemented by removing per-BLOB encryption keys, avoiding write-heavy journal maintenance and prompt physical reclamation. The production evaluation reports lower peak workload than Haystack, median local read latency of 17 ms, tolerance of disk, host, rack, and datacenter failures, and a reduction in effective replication from 3.6 to 2.8 or 2.1. At the reported deployment scale, f4 stored more than 65 PB of logical data and saved more than 53 PB of physical storage.
A web search found the paper subsequently cited as a production example in surveys of distributed file-system design and in later work on rack-aware placement and erasure-coded storage recovery. These citations indicate that f4 has served primarily as a reference architecture demonstrating the practical trade-offs between replication, erasure coding, failure-domain placement, and warm-data tiering; the sources located do not establish a specific direct lineage of implementations derived from f4. (doi.org)