Re: Infiniband and commodity hardware

From: Axel Kohlmeyer (
Date: Sun Sep 18 2011 - 02:38:48 CDT

On Sat, 2011-09-17 at 15:59 -0400, Thomas Albers wrote:
> Hello!

hi thomas,

> Has anyone on this list perhaps experience with Infiniband and commodity
> PC motherboards?
> Recently SDR Infiniband cards have appeared on the market at very good
> prices. The Mellanox manual states that a MHEA28-XTC card requires a
> PCI-E x8 slot, however the average motherboard with two PCI-E x16 slots
> tends to have only one slot with full x16 bandwith, the other one is
> usually x4.
> Does the Infiniband card fall back to x4 bandwith when plugged into the
> x4 slot as your usual graphics card does?

yes. at half the bandwidth, of course. however, you have to
pay particular attention to the fact whether the card requires
PCI-e gen v2.x or gen 1.x. if the latter, it may force down
other devices to gen 1.x (and much lower throughput).

> If it doesn't do that, and the CUDA graphics card has to go into the x4
> slot, will bus bandwith be a limiting issue for performance?

even with 16x PCI-e gen 2.x the bus bandwidth is a limiting
factor for most CUDA applications. sticking the GPU into a
4x slot would be "very bad(TM)".


> Regards,
> Thomas

This archive was generated by hypermail 2.1.6 : Wed Feb 29 2012 - 15:57:45 CST