One of the hot areas in supercomputing is hybrid compute: balancing the computational load between one or more CPUs and GPUs. Along these lines Nolan Davis and Daniel Redig at <a href="http://www.saic.com/">SAIC</a> recently presented work on <a href="http://nvidia.fullviewmedia.com/gtc2010/0923-a3-2100.html">Hybrid GPU/Multicore Solutions for Large Linear Algebra Problems</a> where they developed a novel algorithm for LU decomposition, one of the most important routines in linear algebra.

Here's a snapshot view of their setup:
<table width="650px" border="1" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td valign="top" width="148">System Specs:</td>
<td valign="top" width="132"></td>
<td valign="top" width="163"></td>
</tr>
<tr>
<td valign="top" width="148">GPU</td>
<td valign="top" width="132">Nvidia® Tesla™ 2050</td>
<td valign="top" width="163">448 processing cores3 GB dedicated memory</td>
</tr>
<tr>
<td valign="top" width="148">Multicore Host</td>
<td valign="top" width="132">24 cores64 GB system memory

Red Hat® Enterprise Linux 5</td>
<td valign="top" width="163">Two AMD Opteron™ 6172 12-core processors</td>
</tr>
<tr>
<td valign="top" width="148">Host-to-GPU Communications</td>
<td valign="top" width="132">PCIE 2.0</td>
<td valign="top" width="163">16 channels at 500 MB/sec/laneTheoretical peak bandwidth of 8 GB/sec</td>
</tr>
</tbody>
</table>
&nbsp;

Their initial results are very promising. For matrix sizes N up to around 11,000 (single precision) the GPU running Jacket computed at about 3.5x faster than the 24-core host running MATLAB. The GPU with Jacket reached a performance peak of 287 Gflops/sec with N=8,100.

<a href="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid1.png"><img class="aligncenter size-full wp-image-1073" title="Nolan-Hybrid1" src="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid1.png" alt="" width="600" height="450" /></a><a href="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid2.png"><img class="aligncenter size-full wp-image-1073" title="Nolan-Hybrid2" src="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid2.png" alt="" width="600" height="450" /></a>

For double precision, the GPU-native LU decomposition performance peaks around 110 Gflop/sec for size N=6400. This is still about 3.5x faster than the same multi-threaded computation performed on the 24-core host. The system delivers an LU decomposition of a N=6400 double-precision matrix in 1.6 seconds, and N=8100 in 3.7 seconds.

<a href="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid3.png"><img class="aligncenter size-full wp-image-1073" title="Nolan-Hybrid3" src="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid3.png" alt="" width="600" height="450" /></a>

<a href="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid4.png"><img class="aligncenter size-full wp-image-1074" title="Nolan-Hybrid4" src="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid4.png" alt="" width="600" height="450" /></a>

When problem sizes exceed GPU memory, the researchers show that a Hybrid GPU + CPU computing approach effectively provides better performance than dropping the GPU altogether. For LU decomposition, they break the larger problem into smaller manageable sub-problems, which fit into GPU memory and can be solved one-at-a-time.

<a href="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid5.png"><img class="aligncenter size-full wp-image-1075" title="Nolan-Hybrid5" src="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid5.png" alt="" width="600" height="450" /></a>

<a href="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid6.png"><img class="aligncenter size-full wp-image-1070" title="Nolan-Hybrid6" src="http://blog.accelereyes.com/blog/wp-content/uploads/2011/02/Nolan-Hybrid6.png" alt="" width="600" height="450" /></a>

Most of SAIC research efforts are pretty quiet, so it was cool to see these guys publicly describe some of their work. We'll keep an eye out for more. Special thanks to Nolan Davis and Daniel Redig for sharing their numbers.