Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-intensive Systems

Abstract

📜 Abstract

Large, production quality distributed systems still fail periodically, and do so sometimes catastrophically, where most or all users experience an outage or data loss. We present the result of a comprehensive study investigating 198 randomly selected, user-reported failures that occurred on Cassandra, HBase, Hadoop Distributed File System (HDFS), Hadoop MapReduce, and Redis, with the goal of understanding how one or multiple faults eventually evolve into a user-visible failure. We found that from a testing point of view, almost all failures require only 3 or fewer nodes to reproduce, which is good news considering that these services typically run on a very large number of nodes. However, multiple inputs are needed to trigger the failures with the order between them being important. Finally, we found the error logs of these systems typically contain sufficient data on both the errors and the input events that triggered the failure, enabling the diagnose and the reproduction of the production failures. We found the majority of catastrophic failures could easily have been prevented by performing simple testing on error handling code – the last line of defense – even without an understanding of the software design. We extracted three simple rules from the bugs that have lead to some of the catastrophic failures, and developed a static checker, Aspirator, capable of locating these bugs. Over 30% of the catastrophic failures would have been prevented had Aspirator been used and the identified bugs fixed. Running Aspirator on the code of 9 distributed systems located 143 bugs and bad practices that have been fixed or confirmed by the developers.

Description

✨ Summary

The paper analyzes 198 randomly selected, user-reported failures from Cassandra, HBase, HDFS, Hadoop MapReduce, and Redis. The authors manually examined issue reports, discussions, logs, source code, and patches, and reproduced 73 failures. The study finds that failures are often triggered by short but order-sensitive sequences of events: 77% require multiple input events, while 90% require no more than three. Nearly all failures can be manifested on three or fewer nodes, 74% are deterministic given the appropriate event sequence, and 77% can be reproduced with a unit test. Logs contain explicit failure messages in 76% of cases and all triggering events in 84%, although the median failure produces 824 log messages, making diagnosis labor-intensive.

The central result concerns catastrophic failures. Among the 48 catastrophic failures studied, 92% resulted from incorrect handling of non-fatal errors that were explicitly signaled by the software. Thirty-five percent involved simple patterns detectable without detailed system knowledge: ignored errors or handlers that only log, aborts caused by overly broad exception catches, and unfinished handlers marked with “TODO” or “FIXME.” A further 23% were system-specific but would have been exposed by complete statement coverage of the relevant error-handling code. The remaining cases included more complex error-handling logic and latent errors.

The authors implement Aspirator, a rule-based static checker for Java bytecode. Applied to nine distributed systems, it reported 121 bugs, 379 bad practices, and 115 false positives; 143 reported issues were fixed or confirmed by developers. The proposed engineering strategy combines static checking for simple handler defects, focused code review of error-handling paths, and bottom-up test generation that deliberately constructs executions reaching those paths, rather than relying only on top-down workload generation and fault injection.

The paper’s accompanying project page makes the analyzed failure reports and Aspirator available and records discussion among HBase developers following the work. (eecg.utoronto.ca) Apache HBase subsequently created an issue to review the paper’s findings and investigate adding related checks to its testing process; the issue also records a later effort to add Aspirator-derived rules to Google Error Prone. (issues.apache.org) The empty-catch-block rule was subsequently reported by the authors as included in Error Prone version 2.4.0. (issues.apache.org) Later research on diagnosing and fixing distributed bugs cites this paper as prior empirical work on production failures, logs, and failure reproduction. (sciencedirect.com)