Re: mpirun compilation for NAMD

From: Axel Kohlmeyer (akohlmey_at_cmm.chem.upenn.edu)
Date: Fri Dec 05 2008 - 03:10:24 CST

On 12/5/08, Neelanjana Sengupta <senguptan_at_gmail.com> wrote:
> We do have mpirun in our cluster. In fact, all programs other than NAMD are
> being run using mpirun.
>
> For some reason, NAMD is extremely slow in our cluster, using only ~20% of
> the CPU. The clocktimes are far slower than my previous experience on
> another machine.
>
> Since other programs seem to run fine with mpirun, we think it may be
> because charm++ is not suited for this machine.

no. your machine is either badly set up, the other codes are of the
embarrassingly parallel type, or you are being misled by "benchmarking"
with the 'top' program.

with today's hardware, there are a number of issues to consider. if you have
many cpu cores per node, you first suffer from memory bandwidth limitations,
but also communication contention (e.g. if you have 8 cores/node they all
have to communicate via one network card). if you run with the default binaries,
you run over tcp/ip and thus also the backplane capacity of your switch and
the setup of your hardware (e.g. is TCP hardware congestion control enabled
(it should not, since you _want_ to saturate the bandwidth).

finally, with tcp/ip you always have limited scaling compared to networks
with lower latency. it is not the use of MPI in charm++ that makes the
difference,
but the fact, that MPI supports using that communication hardware wherease
the precompiled binaries don't. so one may get the impression that MPI is
faster, but this is only because with a faster network you _have_ to use a
selfcompiled charm++ over MPI (well, not always, but that is beyond the
scope of this discussion) while in the case of GigE it usually isn't worth
the effort to compile an MPI version.

as far as the 20% cpu goes. please note that many MPI libraries
use "busy-waiting", i.e. the cpu is actively polling to see whether a
new message has arrived, so the CPU load is always 100% even
if there is no computation going on. charm++ and some other MPI
implementations use the select(2) system call to have processes
been woken up by the kernel when data arrives. the latter method
may incur a small overhead (it is negligible on linux for example),
but leaves the cpu empty for other uses (which is very useful when
running on large SMP machines, as it frees the cpus for other tasks).

as for scaling tests, with multi-core nodes you always have to compare
the scaling per node when using the same number of cores when comparing
different machines. the more cores per node you have, the worse the
scaling in most cases for reasons outlined above.

cheers,
     axel.

> Please let me know what you think.
> Regards,
> Neelanjana
>
> On Thu, Dec 4, 2008 at 11:43 PM, Giacomo Fiorin
> <gfiorin_at_seas.upenn.edu>wrote:
>
>> NAMD always uses charm++. You wouldn't be able to compile it otherwise.
>>
>> Using NAMD with MPI means that the charm++ communication runs over the
>> MPI protocol, instead of the "regular" TCP/IP Internet protocol. In
>> that case, charmrun becomes interchangeable with mpirun, but the
>> charm++ functions (with load balancing and all the other goodies) are
>> always inside the executable.
>>
>> Giacomo
>>
>>
>> ---- -----
>> Giacomo Fiorin
>> Center for Molecular Modeling at
>> University of Pennsylvania
>> 231 S 34th Street, Philadelphia, PA 19104-6323
>> phone: (+1)-215-573-4773
>> fax: (+1)-215-573-6233
>> mobile: (+1)-267-324-7676
>> mail: giacomo.fiorin_<at>_gmail.com
>> web: http://www.cmm.upenn.edu/
>> ---- ----
>>
>>
>>
>> On Thu, Dec 4, 2008 at 1:08 PM, Warner Yuen <wyuen_at_apple.com> wrote:
>> > Is MPI the best way to use NAMD? I thought that the use of charm++ was
>> the
>> > more efficient way to go. I haven't run NAMD in quite a while, but I
>> recall
>> > building it with Myrinet support with charm++
>> >
>> >
>> > Warner Yuen
>> >
>> > Scientific Computing
>> >
>> > Consulting Engineer
>> >
>> > Apple Computer
>> >
>> > email: wyuen_at_apple.com
>> >
>> > Tel: 408.718.2859
>> >
>> > Fax: 408.715.0133
>> >
>> >
>> > On Dec 3, 2008, at 8:42 AM, Giacomo Fiorin wrote:
>> >
>> > Hi Neelanjana, the instructions on the NAMD release notes are generic,
>> > but on the charm++ documentation it is written quite clearly that
>> > "mpi-linux" is one of the modes of compilation for charm++, in which
>> > the library (charm++) uses MPI instead of the "basic" Internet
>> > protocol to transfer data across processors.
>> >
>> > But beware: if you want to compile charm++ (and later, namd2) to use
>> > MPI it means that you already have mpirun, and the rest of one of the
>> > many MPI libraries, already installed. Needless to say, you must make
>> > sure that you compile charm++ to use that one, and none other. You
>> > may have special network hardware, so you could need to add extra
>> > flags for that.
>> >
>> > In summary, once you've gathered info on how the machine you intend to
>> > run is organized, try to find instructions about similar machines on
>> > the pertinent webpage (charm.cs.uiuc.edu), in the Charm++ Wiki, and in
>> > the NAMD Wiki.
>> >
>> > There's just a tiny bit more information than in notes.txt.
>> >
>> > Giacomo
>> >
>> >
>> > ---- -----
>> > Giacomo Fiorin
>> > Center for Molecular Modeling at
>> > University of Pennsylvania
>> > 231 S 34th Street, Philadelphia, PA 19104-6323
>> > phone: (+1)-215-573-4773
>> > fax: (+1)-215-573-6233
>> > mobile: (+1)-267-324-7676
>> > mail: giacomo.fiorin_<at>_gmail.com
>> > web: http://www.cmm.upenn.edu/
>> > ---- ----
>> >
>> >
>> >
>> > On Wed, Dec 3, 2008 at 8:46 AM, Neelanjana Sengupta <senguptan_at_gmail.com
>> >
>> > wrote:
>> >
>> > Hello all,
>> >
>> > I am trying to compile NAMD (from the source) for mpirun usage. Now, the
>> >
>> > instructions given in "notes.txt" is the foll:
>> >
>> > ---------
>> >
>> > If you're building an MPI version you will probably need to edit
>> >
>> > compiler flags or commands in the Charm++ src/arch directory. The
>> >
>> > file charm/src/arch/mpi-linux/conv-mach.sh contains the definitions
>> >
>> > that select the mpiCC compiler for mpi-linux, while other compiler
>> >
>> > choices are defined by files in charm/src/arch/common/.
>> >
>> > ----------
>> >
>> > I edited the conv-mach.sh file. Since there is no specific instruction
>> for
>> >
>> > mpirun compilation, I did the foll:
>> >
>> > ./build charm++ net-linux --no-shared -O -DCMK_OPTIMIZE=1
>> >
>> > This built charm++, and not mpirun.
>> >
>> > This did not get me the mpirun compilation either:
>> >
>> > ./build mpirun net-linux --no-shared -O -DCMK_OPTIMIZE=1
>> >
>> > Can someone point out what I should be doing here?
>> >
>> > Thanks,
>> >
>> > Neela
>> >
>> > --
>> >
>> > ~~~~~~~~~~~~~~~~~~~~
>> >
>> > Neelanjana Sengupta, PhD
>> >
>> > Physical and Materials Chemistry Division
>> >
>> > National Chemical Laboratory
>> >
>> > Dr. Homi Bhaba Road
>> >
>> > Pune 411008, India
>> >
>> > Phone: +91-20-2590 2087
>> >
>> > ~~~~~~~~~~~~~~~~~~~~
>> >
>> >
>> >
>> >
>>
>>
>

-- 
=======================================================================
Axel Kohlmeyer   akohlmey_at_cmm.chem.upenn.edu   http://www.cmm.upenn.edu
  Center for Molecular Modeling   --   University of Pennsylvania
Department of Chemistry, 231 S.34th Street, Philadelphia, PA 19104-6323
tel: 1-215-898-1582,  fax: 1-215-573-6233,  office-tel: 1-215-898-5425
=======================================================================
If you make something idiot-proof, the universe creates a better idiot.

This archive was generated by hypermail 2.1.6 : Wed Feb 29 2012 - 15:50:12 CST