paper

The Google File System

  • Authors:
Read the paper

📜 Abstract

We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points. The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients. In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.

✨ Summary

Summary

The paper presents a distributed file system designed for Google’s large-scale, data-intensive workloads. Its central design choices are a single metadata master, replicated fixed-size chunks managed by chunkservers, direct client-to-chunkserver data transfer, leases for ordering mutations, relaxed consistency semantics, atomic record append, copy-on-write snapshots, checksumming, and automatic re-replication and recovery. The system prioritizes sustained aggregate throughput and operational resilience over POSIX compatibility, low latency, and efficient small-file or random-write workloads.

The paper had concrete influence beyond the original GFS deployment. Google later identified GFS as the basis for the Hadoop Distributed File System (HDFS), extending the paper’s architectural ideas into a widely used open-source big-data platform. (cloudplatform.googleblog.com) Google’s Bigtable system also used GFS to store its logs and data files, demonstrating GFS’s role as infrastructure for subsequent distributed data systems. (research.google.com) Within Google, Colossus became the successor to GFS and addressed scaling limitations of the original centralized metadata architecture while retaining its role as a foundational cluster-level storage system. (cloud.google.com)