Re: namd ibverbs

From: Maxime Boissonneault (maxime.boissonneault_at_calculquebec.ca)
Date: Thu Jul 16 2015 - 16:43:06 CDT

Thanks. I would never have found this file.

I searched for things like README, or USAGE, but not "notes.txt", which
looks more like an internal document meant for the developper team itself ;)

Maxime

Le 2015-07-16 17:20, Gianluca Interlandi a écrit :
> Also, apropos documentation. In the NAMD folder that you downloaded
> there is a file called "notes.txt". That file contains information how
> to run NAMD (and also compile it in case you download the source
> code). There, they have a section on "Linux Clusters with InfiniBand".
>
> I copy it here:
>
> -- Linux Clusters with InfiniBand or Other High-Performance Networks --
>
> Charm++ provides a special ibverbs network layer that uses InfiniBand
> networks directly through the OpenFabrics OFED ibverbs library. This
> avoids efficiency and portability issues associated with MPI. Look for
> pre-built ibverbs NAMD binaries or specify ibverbs when building Charm++.
>
> Writing batch job scripts to run charmrun in a queueing system can be
> challenging. Since most clusters provide directions for using mpiexec
> to launch MPI jobs, charmrun provides a ++mpiexec option to use mpiexec
> to launch non-MPI binaries. If "mpiexec -np <procs> ..." is not
> sufficient to launch jobs on your cluster you will need to write an
> executable mympiexec script like the following from TACC:
>
> #!/bin/csh
> shift; shift; exec ibrun $*
>
> The job is then launched (with full paths where needed) as:
>
> charmrun +p<procs> ++mpiexec ++remote-shell mympiexec namd2
> <configfile>
>
>
> Gianluca
>
> On Thu, 16 Jul 2015, Gianluca Interlandi wrote:
>
>> You might want to try different ways. You can try with
>>
>> ibrun $BINDIR/namd2 config.namd
>>
>> and see whether that is faster or slower than the method I first
>> indicated. I would first try on a single node.
>>
>> Gianluca
>>
>> On Thu, 16 Jul 2015, Maxime Boissonneault wrote:
>>
>>> Hi,
>>> Yes, the goal is to see how that user can scale its NAMD computation
>>> on multiple GPU nodes. He already gets a good scaling on a single
>>> node. Each of our nodes have 8 x K20s.
>>>
>>> Thanks,
>>>
>>> Maxime
>>>
>>> Le 2015-07-16 16:56, Gianluca Interlandi a écrit :
>>>> Exactly, mpiexec is just a launcher. You will be using ibverbs and
>>>> not MPI. This is at least my understanding. I tried out different
>>>> ways of running NAMD with ibverbs and found that the method from
>>>> Jim Philipps (one of the NAMD developers) is fastest.
>>>>
>>>> Are you running on multiple nodes?
>>>>
>>>> Gianluca
>>>>
>>>> On Thu, 16 Jul 2015, Maxime Boissonneault wrote:
>>>>
>>>>> Hi Gianluca,
>>>>>
>>>>> So, the NAMD ibverbs package requires MPI ? I thought the whole
>>>>> point of doing an ibverbs was to get rid of MPI and run directly
>>>>> through the Infiniband verbs.
>>>>>
>>>>> Or is mpiexec only used as a launcher (and could be done through
>>>>> pdsh or something like that ?)
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Maxime
>>>>>
>>>>> Le 2015-07-16 16:31, Gianluca Interlandi a écrit :
>>>>>> Hi Maxime,
>>>>>>
>>>>>> I usually use the option "++mpiexec" to run NAMD with ibverbs
>>>>>> (the following is a single line):
>>>>>>
>>>>>> $BINDIR/charmrun +p${NPROCS} ++mpiexec ++remote-shell
>>>>>> mpiexec_wrapper
>>>>>> $BINDIR/namd2 config.namd
>>>>>>
>>>>>> $BINDIR is the path to your NAMD executable.
>>>>>>
>>>>>> "mpiexec_wrapper" is a script that I got a while ago from Jim
>>>>>> Philipps. You can copy the following lines to a file called
>>>>>> "mpiexec_wrapper" and make it executable:
>>>>>>
>>>>>> #!/bin/csh
>>>>>>
>>>>>> exec mpiexec -machinefile $PBS_NODEFILE $*
>>>>>>
>>>>>>
>>>>>> If you run on GPU as well, you should be able to simply add to
>>>>>> the line above:
>>>>>>
>>>>>> +devices 0,1,2 +idlepoll
>>>>>>
>>>>>> However, I need to say that if you are running on a single node
>>>>>> and you do not need to use the network, the multicore version
>>>>>> might be slightly faster than ibverbs. This is because you will
>>>>>> not have the overhead of the network libraries. I usually use
>>>>>> ibverbs only if running on more than one node.
>>>>>>
>>>>>> Gianluca
>>>>>>
>>>>>> On Thu, 16 Jul 2015, Maxime Boissonneault wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> One of our user would like to use namd 2.10 ibverbs with GPU on
>>>>>>> our GPU cluster. I cannot find documentation on how to use it.
>>>>>>> Is there such documentation ? We're used to either use the
>>>>>>> multicore-cuda version or the Cuda+MPI version, but since you
>>>>>>> now provide an ibverbs+GPU binaries, we downloaded them and
>>>>>>> wanted to try them.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ---------------------------------
>>>>>>> Maxime Boissonneault
>>>>>>> Analyste de calcul - Calcul Québec, Université Laval
>>>>>>> Président - Comité de coordination du soutien à la recherche de
>>>>>>> Calcul Québec
>>>>>>> Ph. D. en physique
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------
>>>>>> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
>>>>>> +1 (206) 685 4435
>>>>>> http://artemide.bioeng.washington.edu/
>>>>>>
>>>>>> Research Assistant Professor at the Department of Bioengineering
>>>>>> at the University of Washington, Seattle WA U.S.A.
>>>>>> -----------------------------------------------------
>>>>>
>>>>>
>>>>> --
>>>>> ---------------------------------
>>>>> Maxime Boissonneault
>>>>> Analyste de calcul - Calcul Québec, Université Laval
>>>>> Président - Comité de coordination du soutien à la recherche de
>>>>> Calcul Québec
>>>>> Ph. D. en physique
>>>>>
>>>>>
>>>>
>>>> -----------------------------------------------------
>>>> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
>>>> +1 (206) 685 4435
>>>> http://artemide.bioeng.washington.edu/
>>>>
>>>> Research Assistant Professor at the Department of Bioengineering
>>>> at the University of Washington, Seattle WA U.S.A.
>>>> -----------------------------------------------------
>>>
>>>
>>> --
>>> ---------------------------------
>>> Maxime Boissonneault
>>> Analyste de calcul - Calcul Québec, Université Laval
>>> Président - Comité de coordination du soutien à la recherche de
>>> Calcul Québec
>>> Ph. D. en physique
>>>
>>>
>>
>> -----------------------------------------------------
>> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
>> +1 (206) 685 4435
>> http://artemide.bioeng.washington.edu/
>>
>> Research Assistant Professor at the Department of Bioengineering
>> at the University of Washington, Seattle WA U.S.A.
>> -----------------------------------------------------
>
> -----------------------------------------------------
> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
> +1 (206) 685 4435
> http://artemide.bioeng.washington.edu/
>
> Research Assistant Professor at the Department of Bioengineering
> at the University of Washington, Seattle WA U.S.A.
> -----------------------------------------------------

-- 
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Ph. D. en physique

This archive was generated by hypermail 2.1.6 : Thu Dec 31 2015 - 23:21:58 CST