Analyzing MPI performance over 10-Gigabit ethernet

Recent work with 10-Gigabit (10 GbE) network adapters has demonstrated good performance in TCP/IP-based local- and wide-area networks (LANs and WANs). In the present work we present an evaluation of host-based 10 GbE adapters in a system-area network (SAN) in support of a cluster. This evaluation focuses on the performance of the message-passing interface (MPI) when running over a 10 GbE interconnect. We find that MPI over 10 GbE provides communications performance comparable to that of TCP alone and fairly competitive with more exotic technologies such as MPI over Quadrics. The optimization of MPI and MPI-based applications to make use of this performance, however, is a non-trivial task. Consequently, it is difficult for MPI-based applications to realize this performance when running current-generation 10 GbE hardware.