Re: LDB questions/problems

From: Brian Bennion (brian_at_youkai.llnl.gov)
Date: Thu Dec 18 2003 - 10:23:25 CST

Hello Jim,

Thank you for your help. Setting the maxselfpart 1 an dmaxpairpart 1
seemed to work. The job was able to run through two loadbalancing events
and finish the desired number of minimization steps. The actual number of
atoms is 3M. I am curious about the reasoning for creating such multiple
objects when they can't be distributed across actual cpus? Am I missing
something basic?
I did search through the SimParameters.C code, it is quite the treasure
trove of neat undocumented commands. It was here that I learned I could
turn LDB off.

Again, thank you for your help.
Brian

On Tue, 16 Dec 2003, Jim Phillips wrote:

>
> I don't understand the psfgen reference, but system size shouldn't be a
> problem for psfgen. I'm guessing the load balancer is either running out
> of memory or choking in the face of too many options (2M atoms is a lot).
> I can definitely say it's getting stuck in Rebalancer::multirefine().
>
> Try setting "maxSelfPart 1" and "maxPairPart 1", which eliminates the
> partitional of intra- and inter-patch interactions into multiple objects.
> This should reduce the number of choices the load balancer has to make.
>
> Here's what these look like in SimParameters.C:
>
> // Maximum Partition options
> opts.optional("main", "maxSelfPart",
> "maximum number of self partitions in one patch", &maxSelfPart, 50);
> opts.range("maxSelfPart",POSITIVE);
> opts.optional("main", "maxPairPart",
> "maximum number of pair partitions in one patch", &maxPairPart, 20);
> opts.range("maxPairPart",POSITIVE);
> opts.optional("main", "numAtomsSelf",
> "maximum number of atoms in one self compute distribution",
> &numAtomsSelf, 125);
> opts.range("numAtomsSelf",NOT_NEGATIVE);
> opts.optional("main", "numAtomsPair",
> "maximum number of atoms in one pair compute distribution",
> &numAtomsPair, 200);
> opts.range("numAtomsPair",NOT_NEGATIVE);
> opts.optional("main", "numAtomsPair2",
> "maximum number of atoms in one pair compute distribution",
> &numAtomsPair2, 400);
> opts.range("numAtomsPair2",NOT_NEGATIVE);
>
> -Jim
>
>
> On Tue, 16 Dec 2003, Brian Bennion wrote:
>
> > I am running on a linux cluster, using an in house rh 7.3 kernel with elan
> > quadrics switches. So I don't think the solaris info applies. It may be
> > just to big for psfgen to do correctly.
> > I have also had several elan buffering errors which I worked around by
> > increasing the mpi buffers from 200MB to 400MB
> > Brian
> >
> >
> > On Tue, 16 Dec 2003, Gengbin Zheng
> > wrote:
> >
> > >
> > > Hi,
> > >
> > > I think I'v heard this before - there is some weird memory usage
> > > behavior with the Solaris-Sparc-SMP-CC version of Charm++
> > > built upon net-sol-smp version of Charm++.
> > > You may try the latest version of NAMD, but you have to build every thing
> > > from source out of the CVS - Charm++ and NAMD (they have separate CVS
> > > servers).
> > >
> > > Gengbin
> > >
> > > On Mon, 15 Dec 2003, Brian Bennion wrote:
> > >
> > > >
> > > > Interesting...so I assume you meant reduce the number of processors I was
> > > > using. I went from 64 to 32 cpus and it made it through the load
> > > > balancing step before the queue timedout. However, one stray pme error did
> > > > persist.
> > > > Now i am more perplexed. my system has >2 million atoms so I don't know
> > > > how to get this system minimized effeciently with any peace of mind...
> > > >
> > > > Brian
> > > >
> > > >
> > > >
> > > > On Tue, 16 Dec 2003, himanshu chandola
> > > > wrote:
> > > >
> > > > > I was getting the same kinda errors when i used to run it on a solaris box
> > > > > .Though in my case the namd would give ldb messages - stay there for a
> > > > > few minutes and then crash showing memory problems. Reducing the number of
> > > > > processes helped and removed this problem from reappearing.
> > > > >
> > > > > You can just try this perhaps!
> > > > >
> > > > >
> > > > > cheers
> > > > >
> > > > > himanshu
> > > > >
> > > > > ----------------------------------------
> > > > > Morpheus: Do you believe in fate, Neo?
> > > > > Neo: No.
> > > > > Morpheus: Why Not?
> > > > > Neo: Because I don't like the idea that I'm not in control of my life.
> > > > >
> > > > >
> > > > > On Mon, 15 Dec 2003, Brian Bennion wrote:
> > > > >
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I am attempting to minimize a large system. I call for 100 steps of min,
> > > > > > however, at the 10th step the load distribution system kicks in, gives the
> > > > > > following message in the log and stays there for at least 20 minutes at
> > > > > > which point I get kicked out of the queue. I turn the balancer off, with
> > > > > > ldbstrategy none and things progress, but not without many stray pme grid
> > > > > > errors.
> > > > > >
> > > > > > Any insight?
> > > > > > Brian
> > > > > >
> > > > > > log snippet....
> > > > > >
> > > > > > ENERGY: 9 4446565.9422 815057.1233 0.0000
> > > > > > 0.0000 -8238197.7208 17559560.4307 0.0000 0.0000
> > > > > > 0.0000 14582985.7753 0.0000 14582985.7753 14582985.7753
> > > > > > 0.0000 228178.0317 226466.3996 35898114.1381 228178.0317
> > > > > > 226466.3996
> > > > > >
> > > > > > LDB: LOAD: AVG 17.6773 MAX 22.956 MSGS: TOTAL 17196 MAXC 336 MAXP 5
> > > > > > None
> > > > > > LDB: LOAD: AVG 17.6773 MAX 21.2128 MSGS: TOTAL 20029 MAXC 633 MAXP 6
> > > > > > Alg7
> > > > > >
> > > > > > --
> > > > > > *****************************************************************
> > > > > > **Brian Bennion, Ph.D. **
> > > > > > **Computational and Systems Biology Division **
> > > > > > **Biology and Biotechnology Research Program **
> > > > > > **Lawrence Livermore National Laboratory **
> > > > > > **P.O. Box 808, L-448 bennion1_at_llnl.gov **
> > > > > > **7000 East Avenue phone: (925) 422-5722 **
> > > > > > **Livermore, CA 94550 fax: (925) 424-6605 **
> > > > > > *****************************************************************
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > > *****************************************************************
> > > > **Brian Bennion, Ph.D. **
> > > > **Computational and Systems Biology Division **
> > > > **Biology and Biotechnology Research Program **
> > > > **Lawrence Livermore National Laboratory **
> > > > **P.O. Box 808, L-448 bennion1_at_llnl.gov **
> > > > **7000 East Avenue phone: (925) 422-5722 **
> > > > **Livermore, CA 94550 fax: (925) 424-6605 **
> > > > *****************************************************************
> > > >
> > > >
> > >
> > >
> >
> > --
> > *****************************************************************
> > **Brian Bennion, Ph.D. **
> > **Computational and Systems Biology Division **
> > **Biology and Biotechnology Research Program **
> > **Lawrence Livermore National Laboratory **
> > **P.O. Box 808, L-448 bennion1_at_llnl.gov **
> > **7000 East Avenue phone: (925) 422-5722 **
> > **Livermore, CA 94550 fax: (925) 424-6605 **
> > *****************************************************************
> >
> >
>

-- 
*****************************************************************
**Brian Bennion, Ph.D.                                         **
**Computational and Systems Biology Division                   **
**Biology and Biotechnology Research Program                   **
**Lawrence Livermore National Laboratory                       **
**P.O. Box 808, L-448    bennion1_at_llnl.gov                     **
**7000 East Avenue       phone: (925) 422-5722                 **
**Livermore, CA  94550   fax:   (925) 424-6605                 **
*****************************************************************

This archive was generated by hypermail 2.1.6 : Wed Feb 29 2012 - 05:17:59 CST