Commodifying Replicated State Machines with OpenReplica

Abstract

📜 Abstract

This paper describes OpenReplica, an open service that provides replication and synchronization support for large-scale distributed systems. OpenReplica is designed to commodify Paxos replicated state machines by providing infrastructure for their construction, deployment and maintenance. OpenReplica is based on a novel Paxos replicated state machine implementation that employs an object-oriented approach in which the system actively creates and maintains live replicas for user-provided objects. Clients access these replicated objects transparently as if they are local objects. OpenReplica supports complex distributed synchronization constructs through a multi-return mechanism that enables the replicated objects to control the execution flow of their clients, in essence providing blocking and non-blocking method invocations that can be used to implement richer synchronization constructs. Further, it supports elasticity requirements of cloud deployments by enabling any number of servers to be replaced dynamically. A rack-aware placement manager places replicas on nodes that are unlikely to fail together. Experiments with the system show that the latencies associated with replication are comparable to ZooKeeper, and that the system scales well.

Description

✨ Summary

OpenReplica presents an object-oriented coordination service for constructing, deploying, and maintaining Paxos-based replicated state machines. Users provide ordinary Python objects, while the system generates client proxies and server-side replication logic so that clients can invoke replicated objects as though they were local. Replicas maintain live object instances and agree on a totally ordered command history rather than repeatedly serializing and restoring complete object state.

The design supports both ordinary synchronous operations and rendezvous objects, whose methods can suspend and later resume client execution. This multi-return mechanism allows distributed locks, semaphores, barriers, and condition variables to be implemented using programming patterns close to their centralized counterparts. The system also supports dynamic changes to replica, acceptor, and name-server membership; DNS-based discovery of current replicas; rack-aware placement to reduce correlated failures; handling of selected nondeterministic operations; optional inconsistent reads; and garbage collection through snapshots.

The evaluation, conducted on an eleven-server cluster, reported latency comparable to ZooKeeper, lower latency variance in the tested workload, performance largely insensitive to replicated-state size, and scaling to larger replica and acceptor groups. OpenReplica achieved 327 operations per second for consistent operations with five replicas and five acceptors, compared with 1,872 operations per second for ZooKeeper in the same experiment. Leader-failure recovery averaged 2.75 seconds for OpenReplica versus 2 seconds for ZooKeeper. The authors attributed the throughput gap primarily to the prototype’s Python implementation and ZooKeeper’s batching optimizations.

The paper was published as a Cornell technical report on June 5, 2012. (ecommons.cornell.edu) A concrete later use is the AR2C2 resilient software-defined-network control-plane work, which implemented active replication using OpenReplica and evaluated controller recovery and migration. (eurekamag.com) Later distributed-systems literature also cites OpenReplica as an example of transparent object replication, Paxos-based implementation, binary-rewritten proxies, and runtime view changes. (researchgate.net) The OpenReplica project website describes the system as an educational coordination service and states that it is not actively maintained, so the available evidence indicates research and educational reuse rather than sustained mainstream industry adoption. (openreplica.org)