A Case for High Performance Computing with Virtual Machines
📜 Abstract
Virtual machine (VM) technologies are experiencing a resurgence in both industry and research communities. VMs offer many desirable features such as security, ease of management, OS customization, performance isolation, checkpointing, and migration, which can be very beneficial to the performance and the manageability of high performance computing (HPC) applications. However, very few HPC applications are currently running in a virtualized environment due to the performance overhead of virtualization. Further, using VMs for HPC also introduces additional challenges such as management and distribution of OS images. In this paper we present a case for HPC with virtual machines by introducing a framework which addresses the performance and management overhead associated with VM-based computing. Two key ideas in our design are: Virtual Machine Monitor (VMM) bypass I/O and scalable VM image management. VMM-bypass I/O achieves high communication performance for VMs by exploiting the OS-bypass feature of modern high speed interconnects such as InfiniBand. Scalable VM image management significantly reduces the overhead of distributing and managing VMs in large scale clusters. Our current implementation is based on the Xen VM environment and InfiniBand. However, many of our ideas are readily applicable to other VM environments and high speed interconnects. We carry out detailed analysis on the performance and management overhead of our VM-based HPC framework. Our evaluation shows that HPC applications can achieve almost the same performance as those running in a native, non-virtualized environment. Therefore, our approach holds promise to bring the benefits of VMs to HPC applications with very little degradation in performance.
✨ Summary
The paper established an early systems case for using virtual machines in tightly coupled HPC clusters. Its central technical contribution was VMM-bypass I/O, implemented in Xen-IB, which allowed time-critical InfiniBand operations to proceed from guest domains without placing the hypervisor or privileged domain on the critical communication path. The accompanying management design addressed small customized VM images, scalable binomial-tree distribution, and image caching. In the reported eight-node evaluation, MPI latency and bandwidth were nearly native with registration caching enabled; NAS and HPL results were also close to native performance. The prototype reduced guest-VM memory overhead and image-distribution time compared with full operating-system images.
Subsequent work cited the paper as an early demonstration of Xen-based HPC with InfiniBand and used it as a baseline while examining parallel-I/O scalability and migration limitations. Later research also extended the broader idea of accelerating virtualized HPC workloads to other devices, including GPUs, and continued investigating high-performance hypervisor architectures. The Ohio State project associated with the paper further developed VMM-bypass I/O and Xen-based HPC infrastructure. (usenix.org)
The paper’s influence is therefore primarily visible in follow-on research on low-overhead device virtualization, virtualized HPC communication, parallel I/O, accelerator virtualization, and VM management. The available evidence supports research influence; it does not establish broad commercial industry adoption of this specific Xen-IB implementation.