Life beyond Distributed Transactions: an Apostate’s Opinion
📜 Abstract
Many decades of work have been invested in the area of distributed transactions including protocols such as 2PC, Paxos, and various approaches to quorum. These protocols provide the application programmer a façade of global serializability. Personally, I have invested a non-trivial portion of my career as a strong advocate for the implementation and use of platforms providing guarantees of global serializability. My experience over the last decade has led me to liken these platforms to the Maginot Line. In general, application developers simply do not implement large scalable applications assuming distributed transactions. When they attempt to use distributed transactions, the projects founder because the performance costs and fragility make them impractical. Natural selection kicks in… Instead, applications are built using different techniques which do not provide the same transactional guarantees but still meet the needs of their businesses. This paper explores and names some of the practical approaches used in the implementations of large-scale mission-critical applications in a world which rejects distributed transactions. We discuss the management of fine-grained pieces of application data which may be repartitioned over time as the application grows. We also discuss the design patterns used in sending messages between these repartitionable pieces of data. The reason for starting this discussion is to raise awareness of new patterns for two reasons. First, it is my belief that this awareness can ease the challenges of people hand-crafting very large scalable applications. Second, by observing the patterns, hopefully the industry can work towards the creation of platforms that make it easier to build these very large applications.
✨ Summary
The paper argues that applications intended to scale across very large numbers of machines should avoid relying on global distributed transactions. Instead, data should be divided into uniquely identified entities, each confined to one transactional serializability scope. Transactions may update data within an entity, but not across entities; communication between entities occurs through asynchronously enqueued messages.
Because practical messaging systems commonly provide at-least-once delivery and may reorder messages during repartitioning, applications must make message handling idempotent. The paper introduces activities as per-partner state maintained by an entity to remember processed messages, track replies, and represent the evolving state of relationships with other entities.
For operations spanning entities, the paper recommends application-level workflow rather than atomic commit. Tentative operations, confirmations, cancellations, reservations, and compensating business actions allow systems to manage uncertainty explicitly. It also observes that alternate indices cannot generally be updated atomically with their primary entities at extreme scale, so they may be temporarily inconsistent and must be maintained through asynchronous workflows.
The paper’s main contribution is conceptual: it names and organizes design patterns already emerging in large-scale systems—entities, entity-addressed messaging, activities, idempotence, and fine-grained workflow. A concrete downstream connection is Google’s Megastore, whose paper cites Helland’s work while describing serializable transactions within fine-grained partitions rather than across the entire distributed database. (research.google.com) Later work on edge computing explicitly presents itself as extending and evaluating Helland’s entity-and-activity model for edge environments. (explore.openaire.eu) The paper has also continued to appear in research bibliographies and surveys concerning coordination avoidance, distributed transactions, and application-level transaction design. (chuzhe.me)