Indirect cube: A power-efficient topology for compute clusters

Interconnection networks arranged as k-ary n-trees or spines are widely used to build high-performance computing clusters. Current blade-based technology allows the integration of the first level of the network together with the compute elements. The remaining network stages require dedicated rack space. In most systems one or several racks house the upper network stages, separated from the compute elements. This incurs significant additional costs, especially if a rack containing only a few switches has to be added.In this paper we propose and evaluate the performance and power-consumption of an alternative arrangement that connects elements in a cube-like topology. Building an indirect cube topology requires only the use of the switches that are integrated within the compute elements and also simplifies deployment. We explore a wide variety of system scales, ranging from 120 to 7680 compute nodes, in order to find out to which size the proposed topology can scale while keeping adequate performance levels and low power demands. An additional advantage of our proposal is that the same equipment can be reused to form a tree-based topology if a performance boost is needed.