TitleMessage Passing on Data-Parallel Architectures (In Proceedings)
inProceedings of the 23rd IEEE International Parallel and Distributed Processing Symposium
Author(s) Jeff A. Stuart, John D. Owens
Keyword(s)multi-GPU, CUDA, MPI, DCGN
Year May 2009
LocationRome, Italy
DateMay 25–29, 2009
Abstract This paper explores the challenges in implementing a message passing interface usable on systems with data-parallel processors. As a case study, we design and implement the "DCGN" API on NVIDIA GPUs that is similar to MPI and allows full access to the underlying architecture. We introduce the notion of data-parallel thread-groups as a way to map resources to MPI ranks. We use a method that also allows the data-parallel processors to run autonomously from user-written CPU code. In order to facilitate communication, we use a sleep-based polling system to store and retrieve messages. Unlike previous systems, our method provides both performance and flexibility. By running a test suite of applications with different communication requirements, we find that a tolerable amount of overhead is incurred, somewhere between one and five percent depending on the application, and indicate the locations where this overhead accumulates. We conclude that with innovations in chipsets and drivers, this overhead will be mitigated and provide similar performance to typical CPU-based MPI implementations while providing fully-dynamic communication.
Note Research support for the authors is gratefully acknowledged: the SciDAC Institute for Ultrascale Visualization, NSF Award 0541448, and DOE Early Career Principal Award DE-FG02-04ER25609.