Making reliable distributed systems in the presence of software errors
đź“ś Abstract
The work described in this thesis is the result of a research program started in 1981 to find better ways of programming Telecom applications. These applications are large programs which despite careful testing will probably contain many errors when the program is put into service. We assume that such programs do contain errors, and investigate methods for building reliable systems despite such errors. The research has resulted in the development of a new programming language (called Erlang), together with a design methodology, and set of libraries for building robust systems (called OTP). At the time of writing the technology described here is used in a number of major Ericsson, and Nortel products. A number of small companies have also been formed which exploit the technology. The central problem addressed by this thesis is the problem of constructing reliable systems from programs which may themselves contain errors. Constructing such systems imposes a number of requirements on any programming language that is to be used for the construction. I discuss these language requirements, and show how they are satisfied by Erlang. Problems can be solved in a programming language, or in the standard libraries which accompany the language. I argue how certain of the requirements necessary to build a fault-tolerant system are solved in the language, and others are solved in the standard libraries. Together these form a basis for building fault-tolerant software systems. No theory is complete without proof that the ideas work in practice. To demonstrate that these ideas work in practice I present a number of case studies of large commercially successful products which use this technology. At the time of writing the largest of these projects is a major Ericsson product, having over a million lines of Erlang code. This product (the AXD301) is thought to be one of the most reliable products ever made by Ericsson. Finally, I ask if the goal of finding better ways to program Telecom applications was fulfilled—I also point to areas where I think the system could be improved.
✨ Summary
- The thesis presents Erlang, the OTP libraries, and a programming methodology for constructing reliable systems from components that may contain software errors. Its central architectural principles are lightweight isolated processes, asynchronous message passing, fail-fast behavior, non-local error handling, supervision hierarchies, generic behaviours, and dynamic code replacement.
- Its industrial evidence includes analyses of Ericsson’s AXD301 ATM switch and products developed by Bluetail, Alteon, and Nortel. The thesis reports large Erlang codebases, extensive use of OTP behaviours, runtime recovery from software failures, and support for updating systems without stopping service. It also identifies limitations, including incomplete operational evidence and cases where the ideal one-process-per-concurrent-activity model was impractical.
- The thesis formalized concepts that remain part of Erlang/OTP’s documented design model. Current Erlang documentation continues to define supervision trees around workers and supervisors, with behaviours separating generic process infrastructure from application-specific callback code. (erlang.org)
- The work also influenced subsequent distributed-systems research as a cited description of Erlang’s reliability and scalability model; for example, later research on scalable reliable distributed systems cites the dissertation when discussing Erlang-based systems and actor-oriented designs. (kar.kent.ac.uk)
- The same supervision-oriented design has propagated through other BEAM technologies. Official Elixir documentation describes child specifications that allow components such as agents, tasks, and servers to participate directly in supervision trees, extending the OTP model into a separate language ecosystem. (elixir-lang.org)
Overall, the paper’s concrete legacy is the continued use and documentation of Erlang/OTP’s process-isolation, behaviour, and supervision-tree architecture in industrial software and later BEAM-based research and systems.