Large-scale cluster management at Google with Borg
📜 Abstract
Google’s Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.
✨ Summary
The paper describes Borg, Google’s production cluster-management system for running long-lived services and batch workloads across cells containing thousands to tens of thousands of heterogeneous machines. Its architecture combines a replicated Borgmaster backed by Paxos, a separate scheduler, and Borglet agents on worker machines. Borg uses declarative job specifications, priorities and quota-based admission control, constraint-aware scheduling, preemption, fine-grained resource requests, package caching, failure-domain spreading, and cgroup-based isolation. It also provides stable naming, health monitoring, logging, checkpointing, and the Fauxmaster simulator.
The evaluation argues that high utilization depends on pooling heterogeneous workloads rather than separating production and batch jobs, using large cells to reduce fragmentation, supporting fine-grained resource requests, and reclaiming unused reserved capacity for lower-priority work. In the evaluated workloads, separating production and non-production tasks would require approximately 20–30% more machines, while resource reclamation enabled about 20% of the workload to run using reclaimed resources. The paper also reports scheduler optimizations—score caching, equivalence classes, and relaxed randomization—that reduced full-workload scheduling from more than three days without them to typically a few hundred seconds.
The paper’s documented industry influence is particularly direct: Google later described Borg, Omega, and Kubernetes together, using Borg’s operational experience to explain Kubernetes design choices such as pods, labels, services, introspection, and an API-server-centered architecture. (research.google) Subsequent research also used the paper and its associated Google cluster traces as a baseline; the 2020 study “Borg: the Next Generation” performed a longitudinal comparison of newer 2019 Borg traces with the 2011 trace discussed in this work. (github.com) Google’s publication record confirms the paper’s EuroSys 2015 venue, author list, and abstract. (research.google)