The Impact of Operating System Structure on Memory System Performance
📜 Abstract
In this paper we evaluate the memory system behavior of two distinctly different implementations of the UNIX operating system: DEC’s Ultrix, a monolithic system, and Mach 3.0 with CMU’s UNIX server, a microkernel-based system. In our evaluation we use combined system and user memory reference traces of thirteen industry-standard workloads. We show that the microkernel-based system executes substantially more non-idle system instructions for an equivalent workload than the monolithic system. Furthermore, the average instruction for programs running on Mach has a higher cost, in terms of memory cycles per instruction, than on Ultrix. In the context of our traces, we explore a number of popular assertions about the memory system behavior of modern operating systems, paying special attention to the effect that Mach’s microkernel architecture has on system performance. Our results indicate that many, but not all of the assertions are true, and that a few, while true, have only negligible impact on real system performance.
✨ Summary
The paper experimentally compares DEC Ultrix, a monolithic UNIX system, with Mach 3.0 and its user-level UNIX server, a microkernel-based system. The authors collect combined user- and system-level memory-reference traces for thirteen representative workloads and simulate the memory hierarchy of a DECstation 5000/200. The central result is that Mach generally executes more non-idle instructions and incurs a higher memory-cycle cost per instruction than Ultrix, particularly for workloads that rely heavily on operating-system services.
The study finds that system code has poorer locality than user code, with Mach exhibiting poorer system locality than Ultrix. System self-interference, block memory operations, streaming writes, and virtual-to-physical page-placement strategies can all materially affect performance. By contrast, competition between user and system references is measurable but has little performance impact for the tested workloads. The results also show that operating-system execution is more sensitive to memory-system latency than application execution because system code and data have relatively poor locality.
The paper influenced subsequent microkernel research by helping shift attention from the direct cost of interprocess communication to broader memory-system effects, including cache footprint, cache capacity misses, and instruction locality. Later analyses of microkernel performance explicitly identify the large working set of Mach as a major bottleneck and use these observations to motivate smaller, more aggressively optimized microkernels such as L4. (trustworthy.systems) Subsequent L4 research revisited the comparison experimentally; Härtig and colleagues state that their work repeats related experiments to assess the effect of a second-generation microkernel on application performance, supporting the broader conclusion that the implementation and memory footprint of the underlying microkernel are decisive performance factors. (os.inf.tu-dresden.de) The paper therefore provided an empirical foundation for later efforts to reduce microkernel working sets, optimize IPC paths, and evaluate operating-system structure using complete workload traces rather than isolated microbenchmarks.