From: Norman Geist (norman.geist_at_uni-greifswald.de)
Date: Fri Jul 13 2012 - 01:28:49 CDT
Hi!
What value do you use for fullelectfequency??
How many GPUs are there per node in this cluster?
What kind of interconnect?
Norman Geist.
> -----Ursprüngliche Nachricht-----
> Von: owner-namd-l_at_ks.uiuc.edu [mailto:owner-namd-l_at_ks.uiuc.edu] Im
> Auftrag von Gianluca Interlandi
> Gesendet: Freitag, 13. Juli 2012 00:26
> An: Aron Broom
> Cc: NAMD list
> Betreff: Re: namd-l: Running NAMD on Forge (CUDA)
>
> Yes, I was totally surprised, too. I also ran a non-CUDA job on Forge
> using 16 CPUs. I got 0.122076 s/step, which is 16% slower than using
> the 6
> GPUs (0.1 s/step) and a bit slower than 0.10932 s/step that I get on
> Trestles using 16 cores. This difference might be statistical
> fluctuations
> though (or configuration setup) since Forge and Trestles have the exact
> same CPU, i.e., eight-core 2.4 GHz Magny-Cours.
>
> Yes, Forge also uses NVIDIA M2070.
>
> I keep thinking of this guy here in Seattle who works for NVIDIA
> downtown
> and a few years ago he asked me: "How come you don't use CUDA?" Maybe
> the
> code still needs some optimization, and CPU manufacturers have been
> doing
> everything to catch up.
>
> Gianluca
>
> On Thu, 12 Jul 2012, Aron Broom wrote:
>
> > So your speed for 1 or 2 GPUs (based on what your sent) is about 1.7
> ns/day, which
> > seems decent given the system size. I was getting 2.0 and 2.6 ns/day
> for a 100k atom
> > system with roughly those same parameters (and also 6-cpu cores), so
> given a scaling
> > of ~nlogn, I would expect to see ~1.5 to 2.0 ns/day for you. So in
> my mind, the
> > speed you are getting with the GPUs isn't so surprising, it's that
> you get such a
> > good speed with only the CPUs that shocks me. In my case I didn't
> see speeds
> > matching my 1 GPU until 48 CPU cores alone. Seems like those Magny
> Cours are pretty
> > awesome.
> >
> > Which GPUs are you using? I was using mainly the M2070s.
> >
> > Also, one thing that might be useful, if you are able to get roughly
> the same speed
> > with 6 cores and 2 GPUs and you get with 16 cores alone, is to test
> running 3 jobs at
> > once, with 5 cores and 2 GPUs assigned to each and see how much
> slowdown there is.
> > You might be able to benefit from various replica techniques more
> than just hitting a
> > single job with more power.
> >
> > Still, the overall conclusion from what you've got seems to be that
> it makes more
> > sense to go with more of those CPUs rather than putting GPUs in
> there.
> >
> > ~Aron
> >
> > On Thu, Jul 12, 2012 at 4:58 PM, Gianluca Interlandi
> <gianluca_at_u.washington.edu>
> > wrote:
> > What are your simulation parameters:
> >
> > timestep (and also any multistepping values)
> >
> > 2 fs, SHAKE, no multistepping
> >
> > cutoff (and also the pairlist and PME grid spacing)
> >
> > 8-10-12 PME grid spacing ~ 1 A
> >
> > Have you tried giving it just 1 or 2 GPUs alone (using the
> > +devices)?
> >
> >
> > Yes, this is the benchmark time:
> >
> > np 1: 0.48615 s/step
> > np 2: 0.26105 s/step
> > np 4: 0.14542 s/step
> > np 6: 0.10167 s/step
> >
> > I post here also part of the log running on 6 devices (in case it is
> helpful to
> > localize the problem):
> >
> > Pe 4 has 57 local and 64 remote patches and 1066 local and 473 remote
> computes.
> > Pe 1 has 57 local and 65 remote patches and 1057 local and 482 remote
> computes.
> > Pe 5 has 57 local and 56 remote patches and 1150 local and 389 remote
> computes.
> > Pe 2 has 57 local and 57 remote patches and 1052 local and 487 remote
> computes.
> > Pe 3 has 58 local and 57 remote patches and 1079 local and 487 remote
> computes.
> > Pe 0 has 57 local and 57 remote patches and 1144 local and 395 remote
> computes.
> >
> > Gianluca
> >
> > Gianluca
> >
> > On Thu, 12 Jul 2012, Aron Broom wrote:
> >
> > have you tried the multicore build? I wonder if
> the
> > prebuilt
> > smp one is just not
> > working for you.
> >
> > On Thu, Jul 12, 2012 at 3:21 PM, Gianluca
> Interlandi
> > <gianluca_at_u.washington.edu>
> > wrote:
> > are other people also using those GPUs?
> >
> >
> > I don't think so since I reserved the entire node.
> >
> > What are the benchmark timings that you are
> given
> > after
> > ~1000
> > steps?
> >
> >
> > The benchmark time with 6 processes is 101 sec for
> 1000
> > steps. This is only
> > slightly faster than Trestles where I get 109 sec
> for
> > 1000
> > steps running on 16
> > CPUs. So, yes 6 GPUs on Forge are much faster than
> 6
> > cores on
> > Trestles, but in
> > terms of SUs it makes no difference, since on Forge
> I
> > still
> > have to reserve the
> > entire node (16 cores).
> >
> > Gianluca
> >
> > is some setup time.
> >
> > I often run a system of ~100,000 atoms, and I
> > generally
> > see an
> > order of magnitude
> > improvement in speed compared to the same
> number
> > of
> > cores without
> > the GPUs. I would
> > test the non-CUDA precompiled cude on your
> Forge
> > system
> > and see how
> > that compares, it
> > might be the fault of something other than
> CUDA.
> >
> > ~Aron
> >
> > On Thu, Jul 12, 2012 at 2:41 PM, Gianluca
> > Interlandi
> > <gianluca_at_u.washington.edu>
> > wrote:
> > Hi Aron,
> >
> > Thanks for the explanations. I don't
> know
> > whether
> > I'm doing
> > everything
> > right. I don't see any speed advantage
> > running on
> > the CUDA
> > cluster
> > (Forge) versus running on a non-CUDA
> > cluster.
> >
> > I did the following benchmarks on Forge
> > (the
> > system has
> > 127,000 atoms and
> > ran for 1000 steps):
> >
> > np 1: 506 sec
> > np 2: 281 sec
> > np 4: 163 sec
> > np 6: 136 sec
> > np 12: 218 sec
> >
> > On the other hand, running the same
> system
> > on 16
> > cores of
> > Trestles (AMD
> > Magny Cours) takes 129 sec. It seems
> that
> > I'm not
> > really
> > making good use
> > of SUs by running on the CUDA cluster.
> Or,
> > maybe
> > I'm doing
> > something
> > wrong? I'm using the ibverbs-smp-CUDA
> > pre-compiled version of
> > NAMD 2.9.
> >
> > Thanks,
> >
> > Gianluca
> >
> > On Tue, 10 Jul 2012, Aron Broom wrote:
> >
> > if it is truly just one node, you
> can
> > use
> > the
> > multicore-CUDA
> > version and avoid the
> > MPI charmrun stuff. Still, it
> boils
> > down
> > to much the
> > same
> > thing I think. If you do
> > what you've done below, you are
> > running one
> > job with 12
> > CPU
> > cores and all GPUs. If
> > you don't specify the +devices,
> NAMD
> > will
> > automatically
> > find
> > the available GPUs, so I
> > think the main benefit of
> specifying
> > them
> > is when you
> > are
> > running more than one job
> > and don't want the jobs sharing
> GPUs.
> >
> > I'm not sure you'll see great
> scaling
> > across 6 GPUs for
> > a
> > single job, but that would
> > be great if you did.
> >
> > ~Aron
> >
> > On Tue, Jul 10, 2012 at 1:14 PM,
> > Gianluca
> > Interlandi
> > <gianluca_at_u.washington.edu>
> > wrote:
> > Hi,
> >
> > I have a question
> concerning
> > running
> > NAMD on a
> > CUDA
> > cluster.
> >
> > NCSA Forge has for example
> 6
> > CUDA
> > devices and 16
> > CPU
> > cores per node. If I
> > want to use all 6 CUDA
> devices
> > in a
> > node, how
> > many
> > processes is it
> > recommended to spawn? Do I
> need
> > to
> > specify
> > "+devices"?
> >
> > So, if for example I want
> to
> > spawn 12
> > processes,
> > do I
> > need to specify:
> >
> > charmrun +p12 -machinefile
> > $PBS_NODEFILE +devices
> > 0,1,2,3,4,5 namd2
> > +idlepoll
> >
> > Thanks,
> >
> > Gianluca
> >
> >
> >
> > -----------------------------------------------------
> > Gianluca Interlandi, PhD
> > gianluca_at_u.washington.edu
> > +1
> (206)
> > 685 4435
> >
> >
> > http://artemide.bioeng.washington.edu/
> >
> > Research Scientist at the
> > Department
> > of
> > Bioengineering
> > at the University of
> > Washington,
> > Seattle WA
> > U.S.A.
> >
> >
> > -----------------------------------------------------
> >
> >
> >
> >
> > --
> > Aron Broom M.Sc
> > PhD Student
> > Department of Chemistry
> > University of Waterloo
> >
> >
> >
> >
> >
> > ---------------------------------------------------
> --
> > Gianluca Interlandi, PhD
> > gianluca_at_u.washington.edu
> > +1 (206) 685 4435
> >
> > http://artemide.bioeng.washington.edu/
> >
> > Research Scientist at the Department of
> > Bioengineering
> > at the University of Washington,
> Seattle WA
> > U.S.A.
> >
> > ---------------------------------------------------
> --
> >
> >
> >
> >
> > --
> > Aron Broom M.Sc
> > PhD Student
> > Department of Chemistry
> > University of Waterloo
> >
> >
> >
> >
> > ---------------------------------------------------
> --
> > Gianluca Interlandi, PhD gianluca_at_u.washington.edu
> > +1 (206) 685 4435
> >
> > http://artemide.bioeng.washington.edu/
> >
> > Research Scientist at the Department of
> Bioengineering
> > at the University of Washington, Seattle WA U.S.A.
> > ---------------------------------------------------
> --
> >
> >
> >
> >
> > --
> > Aron Broom M.Sc
> > PhD Student
> > Department of Chemistry
> > University of Waterloo
> >
> >
> >
> >
> > -----------------------------------------------------
> > Gianluca Interlandi, PhD gianluca_at_u.washington.edu
> > +1 (206) 685 4435
> >
> http://artemide.bioeng.washington.edu/
> >
> > Research Scientist at the Department of Bioengineering
> > at the University of Washington, Seattle WA U.S.A.
> > -----------------------------------------------------
> >
> >
> >
> >
> > --
> > Aron Broom M.Sc
> > PhD Student
> > Department of Chemistry
> > University of Waterloo
> >
> >
> >
> >
> > -----------------------------------------------------
> > Gianluca Interlandi, PhD gianluca_at_u.washington.edu
> > +1 (206) 685 4435
> > http://artemide.bioeng.washington.edu/
> >
> > Research Scientist at the Department of Bioengineering
> > at the University of Washington, Seattle WA U.S.A.
> > -----------------------------------------------------
> >
> >
> >
> >
> > --
> > Aron Broom M.Sc
> > PhD Student
> > Department of Chemistry
> > University of Waterloo
> >
> >
> >
>
> -----------------------------------------------------
> Gianluca Interlandi, PhD gianluca_at_u.washington.edu
> +1 (206) 685 4435
> http://artemide.bioeng.washington.edu/
>
> Research Scientist at the Department of Bioengineering
> at the University of Washington, Seattle WA U.S.A.
> -----------------------------------------------------
This archive was generated by hypermail 2.1.6 : Mon Dec 31 2012 - 23:21:47 CST