AW: performance variation during run time

From: Norman Geist (norman.geist_at_uni-greifswald.de)
Date: Thu Nov 29 2012 - 01:26:31 CST

Dear NAMD_2.9_Linux-x86_64-ibverbs? O_o

Well I think such a heavy increase in step time can only occur due
interfering tasks running on the cluster or the nodes you are using.
Something is eating up the CPU or the memory on the nodes. Possibly a
machine is swapping.
Another options might be the network. Possibly someone started another
simulation, eating a lot of switching capacity, but that’s more unlikely.

You should check this 1st. if you are 100% sure nothing is interfering with
your simulation, we can check other options regarding NAMD, imho.

Norman Geist.

> -----Ursprüngliche Nachricht-----
> Von: owner-namd-l_at_ks.uiuc.edu [mailto:owner-namd-l_at_ks.uiuc.edu] Im
> Auftrag von Thomas C. Bishop
> Gesendet: Donnerstag, 29. November 2012 01:53
> An: namd-l_at_ks.uiuc.edu
> Betreff: namd-l: performance variation during run time
>
> Dear NAMD_2.9_Linux-x86_64-ibverbs,
>
> Can you help me figure out why I get the following variations in
> performance when running on a 160,000 atom simulation on
> 180 cores of queenbee.loni.org? On all other systems I use the "hours
> remaining" decreases monotonically... as one might expect.
> But really jumping from 6hrs remaining to 19hrs remaining is just odd.
>
> Ideas?
>
> Thanks in advance.
> Tom
>
> [bishop_at_qb3 log]$ grep -i benchmark ~/work/yeast/chr14/0280419/dyn3.out
> Info: Benchmark time: 180 CPUs 0.0484492 s/step 0.280378 days/ns
> 315.965
> MB memory
> Info: Benchmark time: 180 CPUs 0.0489626 s/step 0.283348 days/ns 321.41
> MB memory
> Info: Benchmark time: 180 CPUs 0.0485597 s/step 0.281017 days/ns 321.41
> MB memory
> [bishop_at_qb3 log]$ grep -i remain ~/work/yeast/chr14/0280419/dyn3.out
> TIMING: 5000 CPU: 242.99, 0.048576/step Wall: 243.198,
> 0.0486176/step,
> 6.68492 hours remaining, 321.410156 MB of memory in use.
> TIMING: 10000 CPU: 486.51, 0.048704/step Wall: 486.844,
> 0.0487293/step, 6.63261 hours remaining, 321.410156 MB of memory in
> use.
> TIMING: 15000 CPU: 730.33, 0.048764/step Wall: 730.76,
> 0.0487831/step,
> 6.57217 hours remaining, 321.410156 MB of memory in use.
> TIMING: 20000 CPU: 974.6, 0.048854/step Wall: 975.126,
> 0.0488732/step,
> 6.51642 hours remaining, 321.410156 MB of memory in use.
> TIMING: 25000 CPU: 1570, 0.11908/step Wall: 1570.64, 0.119104/step,
> 15.715 hours remaining, 321.410156 MB of memory in use.
> TIMING: 30000 CPU: 2091.09, 0.104218/step Wall: 2091.9,
> 0.104251/step,
> 13.6105 hours remaining, 321.410156 MB of memory in use.
> TIMING: 35000 CPU: 2335.54, 0.04889/step Wall: 2336.47,
> 0.0489152/step, 6.31821 hours remaining, 321.410156 MB of memory in
> use.
> TIMING: 40000 CPU: 3105.87, 0.154066/step Wall: 3106.93,
> 0.154091/step, 19.6893 hours remaining, 321.410156 MB of memory in use.
> [bishop_at_qb3 log]$
>
> --
> *******************************
> Thomas C. Bishop
> Tel: 318-257-5209
> Fax: 318-257-3823
> www.latech.edu/~bishop
> ********************************

This archive was generated by hypermail 2.1.6 : Tue Dec 31 2013 - 23:22:48 CST