Energy/performance modeling for collective communication in 3-D torus cluster networks

As supercomputers scale ever larger, energy consumption in interconnection networks is an emerging problem. In this work, we analyze the energy consumption and traffic patterns in a 3-D torus network in order to locate and exploit opportunities to save energy by disabling network links dynamically. Using a custom-built simulator, TorusSim, we show that, for common scientific computing codes that utilize collective communications, regularities in the network and algorithmic data flow result in many unused and under-utilized links that can be disabled to save energy at no performance cost. In the case of a reduce operation, we see that at least 56% of the links in a 4x4x4 torus network can be disabled during communication, with significant other opportunity to save energy on under-utilized links that could lead to over 80% overall link energy savings.