Re: namd ibverbs

From: Gianluca Interlandi (gianluca_at_u.washington.edu)
Date: Thu Jul 16 2015 - 16:20:34 CDT

Also, apropos documentation. In the NAMD folder that you downloaded there
is a file called "notes.txt". That file contains information how to run
NAMD (and also compile it in case you download the source code). There,
they have a section on "Linux Clusters with InfiniBand".

I copy it here:

-- Linux Clusters with InfiniBand or Other High-Performance Networks --

Charm++ provides a special ibverbs network layer that uses InfiniBand
networks directly through the OpenFabrics OFED ibverbs library. This
avoids efficiency and portability issues associated with MPI. Look for
pre-built ibverbs NAMD binaries or specify ibverbs when building Charm++.

Writing batch job scripts to run charmrun in a queueing system can be
challenging. Since most clusters provide directions for using mpiexec
to launch MPI jobs, charmrun provides a ++mpiexec option to use mpiexec
to launch non-MPI binaries. If "mpiexec -np <procs> ..." is not
sufficient to launch jobs on your cluster you will need to write an
executable mympiexec script like the following from TACC:

   #!/bin/csh
   shift; shift; exec ibrun $*

The job is then launched (with full paths where needed) as:

   charmrun +p<procs> ++mpiexec ++remote-shell mympiexec namd2 <configfile>

Gianluca

On Thu, 16 Jul 2015, Gianluca Interlandi wrote:

> You might want to try different ways. You can try with
>
> ibrun $BINDIR/namd2 config.namd
>
> and see whether that is faster or slower than the method I first indicated. I
> would first try on a single node.
>
> Gianluca
>
> On Thu, 16 Jul 2015, Maxime Boissonneault wrote:
>
>> Hi,
>> Yes, the goal is to see how that user can scale its NAMD computation on
>> multiple GPU nodes. He already gets a good scaling on a single node. Each
>> of our nodes have 8 x K20s.
>>
>> Thanks,
>>
>> Maxime
>>
>> Le 2015-07-16 16:56, Gianluca Interlandi a écrit :
>>> Exactly, mpiexec is just a launcher. You will be using ibverbs and not
>>> MPI. This is at least my understanding. I tried out different ways of
>>> running NAMD with ibverbs and found that the method from Jim Philipps (one
>>> of the NAMD developers) is fastest.
>>>
>>> Are you running on multiple nodes?
>>>
>>> Gianluca
>>>
>>> On Thu, 16 Jul 2015, Maxime Boissonneault wrote:
>>>
>>>> Hi Gianluca,
>>>>
>>>> So, the NAMD ibverbs package requires MPI ? I thought the whole point of
>>>> doing an ibverbs was to get rid of MPI and run directly through the
>>>> Infiniband verbs.
>>>>
>>>> Or is mpiexec only used as a launcher (and could be done through pdsh or
>>>> something like that ?)
>>>>
>>>> Thanks,
>>>>
>>>> Maxime
>>>>
>>>> Le 2015-07-16 16:31, Gianluca Interlandi a écrit :
>>>>> Hi Maxime,
>>>>>
>>>>> I usually use the option "++mpiexec" to run NAMD with ibverbs (the
>>>>> following is a single line):
>>>>>
>>>>> $BINDIR/charmrun +p${NPROCS} ++mpiexec ++remote-shell mpiexec_wrapper
>>>>> $BINDIR/namd2 config.namd
>>>>>
>>>>> $BINDIR is the path to your NAMD executable.
>>>>>
>>>>> "mpiexec_wrapper" is a script that I got a while ago from Jim Philipps.
>>>>> You can copy the following lines to a file called "mpiexec_wrapper" and
>>>>> make it executable:
>>>>>
>>>>> #!/bin/csh
>>>>>
>>>>> exec mpiexec -machinefile $PBS_NODEFILE $*
>>>>>
>>>>>
>>>>> If you run on GPU as well, you should be able to simply add to the line
>>>>> above:
>>>>>
>>>>> +devices 0,1,2 +idlepoll
>>>>>
>>>>> However, I need to say that if you are running on a single node and you
>>>>> do not need to use the network, the multicore version might be slightly
>>>>> faster than ibverbs. This is because you will not have the overhead of
>>>>> the network libraries. I usually use ibverbs only if running on more
>>>>> than one node.
>>>>>
>>>>> Gianluca
>>>>>
>>>>> On Thu, 16 Jul 2015, Maxime Boissonneault wrote:
>>>>>
>>>>>> Hi,
>>>>>> One of our user would like to use namd 2.10 ibverbs with GPU on our GPU
>>>>>> cluster. I cannot find documentation on how to use it. Is there such
>>>>>> documentation ? We're used to either use the multicore-cuda version or
>>>>>> the Cuda+MPI version, but since you now provide an ibverbs+GPU
>>>>>> binaries, we downloaded them and wanted to try them.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ---------------------------------
>>>>>> Maxime Boissonneault
>>>>>> Analyste de calcul - Calcul Québec, Université Laval
>>>>>> Président - Comité de coordination du soutien à la recherche de Calcul
>>>>>> Québec
>>>>>> Ph. D. en physique
>>>>>>
>>>>>>
>>>>>
>>>>> -----------------------------------------------------
>>>>> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
>>>>> +1 (206) 685 4435
>>>>> http://artemide.bioeng.washington.edu/
>>>>>
>>>>> Research Assistant Professor at the Department of Bioengineering
>>>>> at the University of Washington, Seattle WA U.S.A.
>>>>> -----------------------------------------------------
>>>>
>>>>
>>>> --
>>>> ---------------------------------
>>>> Maxime Boissonneault
>>>> Analyste de calcul - Calcul Québec, Université Laval
>>>> Président - Comité de coordination du soutien à la recherche de Calcul
>>>> Québec
>>>> Ph. D. en physique
>>>>
>>>>
>>>
>>> -----------------------------------------------------
>>> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
>>> +1 (206) 685 4435
>>> http://artemide.bioeng.washington.edu/
>>>
>>> Research Assistant Professor at the Department of Bioengineering
>>> at the University of Washington, Seattle WA U.S.A.
>>> -----------------------------------------------------
>>
>>
>> --
>> ---------------------------------
>> Maxime Boissonneault
>> Analyste de calcul - Calcul Québec, Université Laval
>> Président - Comité de coordination du soutien à la recherche de Calcul
>> Québec
>> Ph. D. en physique
>>
>>
>
> -----------------------------------------------------
> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
> +1 (206) 685 4435
> http://artemide.bioeng.washington.edu/
>
> Research Assistant Professor at the Department of Bioengineering
> at the University of Washington, Seattle WA U.S.A.
> -----------------------------------------------------

-----------------------------------------------------
Gianluca Interlandi, PhD gianluca_at_u.washington.edu
                     +1 (206) 685 4435
                     http://artemide.bioeng.washington.edu/

Research Assistant Professor at the Department of Bioengineering
at the University of Washington, Seattle WA U.S.A.
-----------------------------------------------------

This archive was generated by hypermail 2.1.6 : Thu Dec 31 2015 - 23:21:58 CST