Sinfonia: A New Paradigm for Building Scalable Distributed Systems
📜 Abstract
We propose a new paradigm for building scalable distributed systems. Our approach does not require dealing with message-passing protocols—a major complication in existing distributed systems. Instead, developers just design and manipulate data structures within our service called Sinfonia. Sinfonia keeps data for applications on a set of memory nodes, each exporting a linear address space. At the core of Sinfonia is a novel minitransaction primitive that enables efficient and consistent access to data, while hiding the complexities that arise from concurrency and failures. Using Sinfonia, we implemented two very different and complex applications in a few months: a cluster file system and a group communication service. Our implementations perform well and scale to hundreds of machines.
✨ Summary
Overview
Sinfonia proposes a data-sharing service intended to simplify the construction of fault-tolerant, scalable infrastructure applications. Rather than requiring application developers to implement distributed message-passing protocols for replication, consistency, cache coordination, and recovery, Sinfonia exposes multiple independent linear address spaces hosted by memory nodes. Applications are responsible for organizing bytes into higher-level data structures and for choosing data placement.
The central abstraction is the minitransaction, which contains compare, read, and write items. A minitransaction conditionally performs reads and writes across multiple memory nodes only when all comparisons succeed, providing atomicity, isolation, serializability, and optional durability. Its restricted interface allows execution to be integrated into the two-phase commit protocol, reducing communication overhead. Single-participant operations can use a one-phase protocol. Lock acquisition is nonblocking; conflicts cause an abort and randomized retry, avoiding distributed deadlocks under the intended low-contention workload.
Sinfonia provides configurable fault tolerance through disk images, redo logging, replication, and transactionally consistent backups. Its recovery design avoids dependence on application-node coordinators by allowing a management node to recover uncertain transactions, while failures of memory nodes may cause temporary blocking unless replicas are available. Applications control caching, load balancing, and data placement, which enables locality but also leaves significant systems-design responsibility outside Sinfonia.
The authors demonstrate the approach with two implementations. SinfoniaFS is a shared cluster file system exporting NFSv2. It uses minitransactions to maintain consistency among inodes, directories, allocation metadata, chaining lists, and file blocks, while validating application-side caches. SinfoniaGCS is a total-order group communication service that stores per-member message queues and uses minitransactions to thread messages into a global ordered list. Both applications were implemented relatively quickly and with substantially less code than comparable systems.
Evaluation on as many as 246 machines found that Sinfonia’s throughput generally increased with system size when each minitransaction touched only a small number of memory nodes. The most important scalability rule was to distribute load across transactions while keeping individual transactions localized. In a one-memory-node configuration, the system achieved more than 7,500 minitransactions per second with NVRAM-like storage and approximately 2,400 transactions per second with synchronous disk logging. SinfoniaFS performed comparably to or better than the tested Linux NFS server for most workloads and scaled much more effectively as clients were added. SinfoniaGCS achieved substantially higher throughput than the tested Spread configuration, although the comparison excluded IP multicast and broadcast, technologies for which Spread was optimized.
The principal limitations are equally important: the address-space interface is low level; applications must design their own data layouts, locality strategies, cache policies, and load-balancing mechanisms; and workloads requiring operations not directly supported by minitransactions can perform poorly under high contention because they rely on retries. The design is consequently best suited to cooperative data-center environments rather than wide-area, adversarial, or highly asynchronous systems.
Influence on subsequent work
A direct follow-on research use appeared in A Practical Scalable Distributed B-Tree. That system was implemented on top of Sinfonia and used its minitransactions to atomically update B-tree nodes distributed across servers, including operations such as node splits; the paper explicitly presents Sinfonia as the underlying fault-tolerant distributed data-sharing and atomic-commit layer. (vldb.org)
The paper also continued as a substantially expanded journal article in ACM Transactions on Computer Systems in November 2009, preserving the minitransaction-based paradigm and its evaluation. (researchgate.net) Publicly available sources identify the SOSP 2007 paper as a best-paper recipient and list Sinfonia as a continuing research contribution by the authors. (mkaguilera.kawazoe.org) The available evidence supports concrete influence on subsequent distributed-data-structure research; it does not establish broad industrial deployment of Sinfonia itself.