NAMD on Infiniband - problems and successes

From: Martin Cuma (mcuma_at_chpc.utah.edu)
Date: Sun Mar 25 2007 - 12:30:57 CDT

Hello everybody,

I figured I'll share my experiences building NAMD on our new cluster, and,
ask few questions in the process.

Our cluster is Opteron 2.4 GHz dual-core dual with Cisco (Topspin) IB
cards (SDR), running RHEL 4.0 and OpenIB 1.1.

I got both the 2.6 NAMD and 5.9 charm++ release, and also the latest CVS
of both.

I found the NAMD Wiki post in Infiniband and followed that to build it.
FIRST with MVAPICH 0.9.8. It built and ran, but, the load balancing seems
to be messed up due (seems like) wrong wall time reporting, e.g.:
....
OPENING EXTENDED SYSTEM TRAJECTORY FILE
Info: Initial time: 12 CPUs 0.773921 s/step 4.47871 days/ns 89471 kB
memory
LDB: LOAD: AVG 307114 MAX 1.78672e+06 MSGS: TOTAL 257 MAXC 25 MAXP 5
None
Info: Adjusted background load on 5 nodes.
Warning: overload assign to 0
LDB: LOAD: AVG 307117 MAX 446680 MSGS: TOTAL 298 MAXC 30 MAXP 6 Alg7
LDB: LOAD: AVG 307117 MAX 446680 MSGS: TOTAL 298 MAXC 30 MAXP 6 Alg7
Info: Initial time: 12 CPUs 1117.6 s/step 6467.59 days/ns 107876 kB memory
LDB: LOAD: AVG 23974 MAX 335022 MSGS: TOTAL 298 MAXC 30 MAXP 6 None
Warning: 3 processors are overloaded due to high background load.
Info: ERROR: Could not refine at max overload
LDB: LOAD: AVG 23974 MAX 558337 MSGS: TOTAL 298 MAXC 30 MAXP 6 Refine
Info: Initial time: 12 CPUs -1115.93 s/step -6457.94 days/ns 107811 kB
memory
LDB: LOAD: AVG 27954.8 MAX 223368 MSGS: TOTAL 298 MAXC 30 MAXP 6 None
Warning: 3 processors are overloaded due to high background load.
....

Notice the negative time per step in 2nd report, and, exceedingly large
time in 1st.

First suspicion was cpuspeed. I turned it off, problem persists. Then I
ran the MPICH Ethernet build on the same machine - that ran fine.

So, my conclusion on this one is that there's some trouble with MVAPICH
that skewes the timing. Few questions on that. First, did anybody else see
this? Second - what timers does NAMD use?

My SECOND attempt was to build charm++ using the net-ibvers interface.
Just changed the path to the OpenIB libraries.
That also built fine, but, NAMD in parallel gets stuck here:
...
Info: *****************************
Info: Entering startup phase 0 with 63320 kB of memory in use.

The charm++ test programs seem to be stuck on communication, too. It
almost looks like a deadlock. Any pointers what to do here? Anybody else
running net-ibvers on similar setup (IB drivers) like us? My experience
with charm++ is rather limited, and, I can't find any documentation on
the ibvers part. It'd be good to get the ibvers version working in hopes
for better performance.

The THIRD attempt is a success story - charm++ and NAMD built fine with
OpenMPI 1.2, and, it runs fine too, with good scaling (compared to the
posted benchmarks on NAMD website).

To conclude, I'd like to ask one more question - can those who run on
Infiniband reply and say what type of MPI and IB drivers they use? That'd
be useful to get a good picture on the NAMD status on IB.

Thanks,
MC

-- 
Martin Cuma
Center for High Performance Computing
University of Utah

This archive was generated by hypermail 2.1.6 : Wed Feb 29 2012 - 15:46:07 CST