SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol
📜 Abstract
Several distributed peer-to-peer applications require weakly-consistent knowledge of process group membership information at all participating processes. SWIM is a generic software module that offers this service for large-scale process groups. The SWIM effort is motivated by the unscalability of traditional heart-beating protocols, which either impose network loads that grow quadratically with group size, or compromise response times or false positive frequency w.r.t. detecting process crashes. This paper reports on the design, implementation and performance of the SWIM sub-system on a large cluster of commodity PCs. Unlike traditional heartbeating protocols, SWIM separates the failure detection and membership update dissemination functionalities of the membership protocol. Processes are monitored through an efficient peer-to-peer periodic randomized probing protocol. Both the expected time to first detection of each process failure, and the expected message load per member, do not vary with group size. Information about membership changes, such as process joins, drop-outs and failures, is propagated via piggybacking on ping messages and acknowledgments. This results in a robust and fast infection style (also epidemic or gossip-style) of dissemination. The rate of false failure detections in the SWIM system is reduced by modifying the protocol to allow group members to suspect a process before declaring it as failed - this allows the system to discover and rectify false failure detections. Finally, the protocol guarantees a deterministic time bound to detect failures. Experimental results from the SWIM prototype are presented. We discuss the extensibility of the design to a WAN-wide scale.
✨ Summary
Summary
SWIM separates failure detection from membership-update dissemination. Members periodically probe peers, use indirect probes when direct probes fail, disseminate updates through piggybacked gossip, and apply suspicion and incarnation-number mechanisms to reduce false positives. The design targets constant expected message load and failure-detection time with respect to group size, while providing logarithmic-scale dissemination latency and an optional deterministic detection bound.
Influence
The protocol became a foundation for subsequent systems and implementations. HashiCorp’s memberlist explicitly identifies itself as based on SWIM and extends it to improve propagation, convergence, and robustness; HashiCorp’s Consul uses a modified SWIM-based Serf gossip protocol for cluster membership and failure detection. (github.com) Uber’s Ringpop also adopted SWIM as a scalable building block for production infrastructure. (research.google) Later research introduced Lifeguard, a set of SWIM extensions that uses local health awareness to improve failure-detection accuracy when the monitoring process itself is slow or resource-starved. (arxiv.org)