f4: Facebook’s Warm BLOB Storage System
📜 Abstract
Facebook’s corpus of photos, videos, and other Binary Large OBjects (BLOBs) that need to be reliably stored and quickly accessible is massive and continues to grow. As the footprint of BLOBs increases, storing them in our traditional storage system, Haystack, is becoming increasingly inefficient. To increase our storage efficiency, measured in the effective-replication-factor of BLOBs, we examine the underlying access patterns of BLOBs and identify temperature zones that include hot BLOBs that are accessed frequently and warm BLOBs that are accessed far less often. Our overall BLOB storage system is designed to isolate warm BLOBs and enable us to use a specialized warm BLOB storage system, f4. f4 is a new system that lowers the effective-replication-factor of warm BLOBs while remaining fault tolerant and able to support the lower throughput demands. f4 currently stores over 65PBs of logical BLOBs and reduces their effective-replication-factor from 3.6 to either 2.8 or 2.1. f4 provides low latency; is resilient to disk, host, rack, and datacenter failures; and provides sufficient throughput for warm BLOBs.
✨ Summary
The paper presents f4, a production warm-storage system for Facebook’s immutable binary objects. It observes that BLOB access rates decline sharply with age, making triple replication inefficient for older content. Facebook therefore separates hot and warm data: Haystack handles newly created, heavily accessed content, while f4 stores older, less frequently accessed volumes.
f4 uses Reed–Solomon (10,4) erasure coding within a datacenter, places stripe blocks across racks, and uses geo-replicated XOR coding to tolerate datacenter failures. It supports online reconstruction of individual requested BLOBs, background block rebuilding, transparent migration through a router tier, and logical deletion by removing per-BLOB encryption keys. In production, the system stored more than 65 PB of logical data, reduced the effective replication factor from 3.6 to 2.8 or 2.1, maintained low read latency, and saved tens of petabytes of storage.
The paper’s documented influence is primarily in subsequent erasure-coded warm-storage research. A 2025 FAST paper, Revisiting Network Coding for Warm Blob Storage, explicitly uses Facebook’s f4 as its motivating production example and targets the same workload characteristics—warm data with many small BLOBs—while proposing NCBlob to improve repair performance and reduce I/O amplification. Its evaluation reports up to 45.0% lower single-block repair time and up to 38.4% lower full-node repair time relative to systematic MSR-code approaches. (usenix.org) The paper is also cited in later work on hybrid replication and erasure-coded storage as a concrete example of using erasure coding for warm data. (onlinelibrary.wiley.com)