paper

Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

  • Authors:
Read the paper

đź“ś Abstract

We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today’s frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our results show that Mesos can achieve near-optimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) nodes, and is resilient to failures.

✨ Summary

Impact

  • The paper established the resource-offer model and the separation between a cluster manager and framework-level schedulers. These concepts were carried into Apache Mesos, whose architecture documentation describes resource offers, framework-controlled task placement, pluggable allocation policies, and offer rejection for satisfying framework constraints. (mesos.apache.org)
  • The paper introduced Spark as a specialized Mesos framework for iterative workloads. Apache Spark subsequently documented Mesos as a supported cluster manager and described running Spark alongside Hadoop through Mesos, demonstrating the paper’s proposed multi-framework operating model. Mesos support is now deprecated in recent Spark documentation. (spark.apache.org)
  • The Mesos project’s historical materials record adoption in industry, including use at Twitter, and its later “Powered by Mesos” list names organizations such as Twitter, Uber, Yelp, and Verizon Labs. These sources document deployment and ecosystem adoption rather than proving that every deployment was directly caused by the paper. (svn.apache.org)
  • Subsequent research continued to examine resource allocation and fairness in Mesos environments, including empirical studies of offer behavior and second-level scheduling policies. (arxiv.org)