paper

Comparing the Robustness of POSIX Operating Systems

  • Authors:
Read the paper

📜 Abstract

Critical system designers are turning to off-the-shelf operating system (OS) software to reduce costs and time-to-market. Unfortunately, general-purpose OSes do not always respond to exceptional conditions robustly, either accepting exceptional values without complaint, or suffering abnormal task termination. Even though direct measurement is impractical, this paper uses a multi-version comparison technique to reveal a 6% to 19% normalized rate at which exceptional parameter values cause no error report in commercial POSIX OS implementations. Additionally, 168 functions across 13 OSes are compared to reveal common mode robustness failures. While the best single OS has a 12.6% robustness failure rate for system calls, 3.8% of failures are common across all 13 OSes examined. However, combining C library calls with system calls increases these rates to 29.5% for the best single OS and 17.0% for common mode failures. These results suggest that OS implementations are not completely diverse, and that C library functions are both less diverse and less robust than system calls.

✨ Summary

Impact

  • The paper established a large-scale empirical application of Ballista robustness testing and multi-version comparison to commercial POSIX implementations, documenting silent failures, abnormal terminations, catastrophic failures, and common-mode behavior across operating systems. (users.ece.cmu.edu)
  • Its results were subsequently expanded into the peer-reviewed article The Exception Handling Effectiveness of POSIX Operating Systems in IEEE Transactions on Software Engineering, which analyzed up to 233 POSIX functions across 15 operating-system implementations and further quantified exception-handling failures. (users.ece.cmu.edu)
  • The Ballista project made the POSIX robustness test suite available for comparing operating systems and different versions of the same system, supporting reproduction and practical testing beyond the original study. (users.ece.cmu.edu)
  • Later Ballista-related work extended the approach to other APIs, including Microsoft Win32, and subsequent robustness-testing research cited the project as an antecedent for scalable interface testing and for testing autonomy software. (scribd.com)