Re: CUDA error vs clashes

From: Francesco Pietra (chiendarret_at_gmail.com)
Date: Fri Nov 09 2012 - 01:52:32 CST

I can confirm that Linux NAMD-CUDA 2.9 Sept 26 2012 nightly build
fails to both minimize and NVT heat with my heme protein in a water
box,

Recourse to Linux multicore NAMD 2012-11-08 nightly build did both
above tasks correctly.

However, Linux NAMD-CUDA 2.9 Sept 26 2012 nightly build is now NPT
equilibrating the system minimized-heated with the non-cuda version
(it is at step 250,000 out of due 500,000).

I don't see anything special in this new heme protein with respect to
previous heme proteins that were dealt with correctly, in toto, from
minimizing to production, with the above cuda version of namd, on the
same machine. The new heme protein was elucidaded by X0ray diffraction
at high-res (1.8 A) at 100K, so that it was already somewhat "heated".
Disappointingly, heating the non-cuda minimized system with the cuda
version crashed at the step of heating to 1K.

francesco pietra

On Thu, Nov 8, 2012 at 6:42 PM, Francesco Pietra <chiendarret_at_gmail.com> wrote:
> No, there is something wrong with CUDA. I could minimize with the same
> files with the non-cuda version (gradient 0.5). I though that it
> would be a probmem of minimization only, because it is with
> minimization that energies are most involved. However, the non-cuda
> minimized system crashed with the cuda version, again for cuda error
> in raising the temp to 1K. Now, the same files are going with the
> non-cuda: it is at 190K at the moment. We will see if it will bring
> the system to 311K. If so, I'll try equilibration with the cuda
> version.
>
> The installed cuda version works fine with a similar system (similar
> in being heme protein, but different protein and different origin of
> the X-ray diffr data. Present ones are at 1.8A resolution.
>
> chiendarret
>
> On Thu, Nov 8, 2012 at 5:16 PM, Aron Broom <broomsday_at_gmail.com> wrote:
>> Can you just simulate the heme on it's own? It seems like there must be
>> something formatted wrong with your stream file and it's getting weird
>> parameters from it.
>>
>> ~Aron
>>
>>
>> On Thu, Nov 8, 2012 at 1:21 AM, Francesco Pietra <chiendarret_at_gmail.com>
>> wrote:
>>>
>>> On Wed, Nov 7, 2012 at 8:08 PM, Aron Broom <broomsday_at_gmail.com> wrote:
>>> > So in your attempts to detect clashes, did you output frames and
>>> > energies
>>> > every step? Also, can you visualize which atoms are moving too fast
>>> > before
>>> > they get horribly far away (e.g. with a very low timestep, like 0.01fs)?
>>>
>>> I was using ts 0.01fs. I have to check all that carefully.
>>> >
>>> > The part about the CUDA error being replaced with this when you
>>> > re-minimize
>>> > and such is really odd, I've got no suggestions there.
>>>
>>> I was not clear. On repeatedly launching the heating MD simulation,
>>> the error is mostly "CUDA error". less frequently, high energy, with
>>> fifferent atoms flying away in different runs.
>>>
>>> >
>>> > So you've used that same stream file with other systems run with NAMD
>>> > 2.9?
>>> > It sounds like maybe it's not getting the parameters from the file
>>> > properly?
>>>
>>>
>>> I was also wondering about that, as I used the streamfile from older
>>> wor. I have now repeated the heating MD with the streamfile used a few
>>> weeks ago for another heme protein. As you can see, the same cuda
>>> error came out:
>>>
>>>
>>> francesco_at_gig64:~/MD$ charmrun namd2 heat-01.conf +p6 +idlepoll 2>&1 |
>>> tee heat-01.log
>>> Running command: namd2 heat-01.conf +p6 +idlepoll
>>>
>>> Charm++: standalone mode (not using charmrun)
>>> Converse/Charm++ Commit ID: v6.4.0-beta1-0-g5776d21
>>> CharmLB> Load balancer assumes all CPUs are same.
>>> Charm++> Running on 1 unique compute nodes (12-way SMP).
>>> Charm++> cpu topology info is gathered in 0.001 seconds.
>>> Info: NAMD CVS-2012-09-26 for Linux-x86_64-multicore-CUDA
>>> Info:
>>> Info: Please visit http://www.ks.uiuc.edu/Research/namd/
>>> Info: for updates, documentation, and support information.
>>> Info:
>>> Info: Please cite Phillips et al., J. Comp. Chem. 26:1781-1802 (2005)
>>> Info: in all publications reporting results obtained with NAMD.
>>> Info:
>>> Info: Based on Charm++/Converse 60400 for multicore-linux64-iccstatic
>>> Info: Built Wed Sep 26 02:25:08 CDT 2012 by jim on lisboa.ks.uiuc.edu
>>> Info: 1 NAMD CVS-2012-09-26 Linux-x86_64-multicore-CUDA 6 gig64
>>> francesco
>>> Info: Running on 6 processors, 1 nodes, 1 physical nodes.
>>> Info: CPU topology information available.
>>> Info: Charm++/Converse parallel runtime startup completed at 0.010195 s
>>> Pe 1 physical rank 1 will use CUDA device of pe 2
>>> Pe 5 physical rank 5 will use CUDA device of pe 4
>>> Pe 3 physical rank 3 will use CUDA device of pe 4
>>> Pe 2 physical rank 2 binding to CUDA device 0 on gig64: 'GeForce GTX
>>> 680' Mem: 2047MB Rev: 3.0
>>> Pe 4 physical rank 4 binding to CUDA device 1 on gig64: 'GeForce GTX
>>> 680' Mem: 2047MB Rev: 3.0
>>> Did not find +devices i,j,k,... argument, using all
>>> Pe 0 physical rank 0 will use CUDA device of pe 2
>>> Info: 8.40234 MB of memory in use based on /proc/self/stat
>>> Info: Configuration file is heat-01.conf
>>> Info: Working in the current directory
>>> /home/francesco/work_heme-oxygenase/MD
>>> TCL: Suspending until startup complete.
>>> Info: EXTENDED SYSTEM FILE ./min-04.restart.xsc
>>> Info: SIMULATION PARAMETERS:
>>> Info: TIMESTEP 0.01
>>> Info: NUMBER OF STEPS 0
>>> Info: STEPS PER CYCLE 10
>>> Info: PERIODIC CELL BASIS 1 81.8 0 0
>>> Info: PERIODIC CELL BASIS 2 0 78.66 0
>>> Info: PERIODIC CELL BASIS 3 0 0 82.31
>>> Info: PERIODIC CELL CENTER -100.627 -13.9064 -83.667
>>> Info: WRAPPING ALL CLUSTERS AROUND PERIODIC BOUNDARIES ON OUTPUT.
>>> Info: WRAPPING TO IMAGE NEAREST TO PERIODIC CELL CENTER.
>>> Info: LOAD BALANCER Centralized
>>> Info: LOAD BALANCING STRATEGY New Load Balancers -- DEFAULT
>>> Info: LDB PERIOD 2000 steps
>>> Info: FIRST LDB TIMESTEP 50
>>> Info: LAST LDB TIMESTEP -1
>>> Info: LDB BACKGROUND SCALING 1
>>> Info: HOM BACKGROUND SCALING 1
>>> Info: PME BACKGROUND SCALING 1
>>> Info: MIN ATOMS PER PATCH 40
>>> Info: INITIAL TEMPERATURE 0.5
>>> Info: CENTER OF MASS MOVING INITIALLY? NO
>>> Info: DIELECTRIC 1
>>> Info: EXCLUDE SCALED ONE-FOUR
>>> Info: 1-4 ELECTROSTATICS SCALED BY 1
>>> Info: MODIFIED 1-4 VDW PARAMETERS WILL BE USED
>>> Info: NO DCD TRAJECTORY OUTPUT
>>> Info: NO EXTENDED SYSTEM TRAJECTORY OUTPUT
>>> Info: NO VELOCITY DCD OUTPUT
>>> Info: NO FORCE DCD OUTPUT
>>> Info: OUTPUT FILENAME ./heat-01
>>> Info: RESTART FILENAME ./heat-01.restart
>>> Info: RESTART FREQUENCY 100
>>> Info: BINARY RESTART FILES WILL BE USED
>>> Info: SWITCHING ACTIVE
>>> Info: SWITCHING ON 10
>>> Info: SWITCHING OFF 12
>>> Info: PAIRLIST DISTANCE 13.5
>>> Info: PAIRLIST SHRINK RATE 0.01
>>> Info: PAIRLIST GROW RATE 0.01
>>> Info: PAIRLIST TRIGGER 0.3
>>> Info: PAIRLISTS PER CYCLE 2
>>> Info: PAIRLISTS ENABLED
>>> Info: MARGIN 100
>>> Info: HYDROGEN GROUP CUTOFF 2.5
>>> Info: PATCH DIMENSION 116
>>> Info: ENERGY OUTPUT STEPS 1000
>>> Info: CROSSTERM ENERGY INCLUDED IN DIHEDRAL
>>> Info: TIMING OUTPUT STEPS 10000
>>> Info: LANGEVIN DYNAMICS ACTIVE
>>> Info: LANGEVIN TEMPERATURE 0
>>> Info: LANGEVIN USING BBK INTEGRATOR
>>> Info: LANGEVIN DAMPING COEFFICIENT IS 1 INVERSE PS
>>> Info: LANGEVIN DYNAMICS NOT APPLIED TO HYDROGENS
>>> Info: VELOCITY REASSIGNMENT FREQ 50
>>> Info: VELOCITY REASSIGNMENT TEMP 1
>>> Info: VELOCITY REASSIGNMENT INCR 1
>>> Info: VELOCITY REASSIGNMENT HOLD 311
>>> Info: PARTICLE MESH EWALD (PME) ACTIVE
>>> Info: PME TOLERANCE 1e-06
>>> Info: PME EWALD COEFFICIENT 0.257952
>>> Info: PME INTERPOLATION ORDER 4
>>> Info: PME GRID DIMENSIONS 90 81 90
>>> Info: PME MAXIMUM GRID SPACING 1
>>> Info: Attempting to read FFTW data from
>>> FFTW_NAMD_CVS-2012-09-26_Linux-x86_64-multicore-CUDA.txt
>>> Info: Optimizing 6 FFT steps. 1... 2... 3... 4... 5... 6... Done.
>>> Info: Writing FFTW data to
>>> FFTW_NAMD_CVS-2012-09-26_Linux-x86_64-multicore-CUDA.txt
>>> Info: FULL ELECTROSTATIC EVALUATION FREQUENCY 5
>>> Info: USING VERLET I (r-RESPA) MTS SCHEME.
>>> Info: C1 SPLITTING OF LONG RANGE ELECTROSTATICS
>>> Info: PLACING ATOMS IN PATCHES BY HYDROGEN GROUPS
>>> Info: RIGID BONDS TO HYDROGEN : WATER
>>> Info: ERROR TOLERANCE : 1e-06
>>> Info: MAX ITERATIONS : 100
>>> Info: RIGID WATER USING SETTLE ALGORITHM
>>> Info: RANDOM NUMBER SEED 12343
>>> Info: USE HYDROGEN BONDS? NO
>>> Info: COORDINATE PDB ./4G7T_heme_O2free_WTS_WTBOX_ION.pdb
>>> Info: STRUCTURE FILE ./4G7T_heme_O2free_WTS_WTBOX_ION.psf
>>> Info: PARAMETER file: CHARMM format!
>>> Info: PARAMETERS ./par_all27_prot_lipid.prm
>>> Info: PARAMETERS ./toppar_all22_prot_heme.str
>>> Info: USING ARITHMETIC MEAN TO COMBINE L-J SIGMA PARAMETERS
>>> Info: BINARY COORDINATES ./min-04.restart.coor
>>> Info: SKIPPING rtf SECTION IN STREAM FILE
>>> Info: SUMMARY OF PARAMETERS:
>>> Info: 185 BONDS
>>> Info: 467 ANGLES
>>> Info: 601 DIHEDRAL
>>> Info: 47 IMPROPER
>>> Info: 6 CROSSTERM
>>> Info: 121 VDW
>>> Info: 0 VDW_PAIRS
>>> Info: 0 NBTHOLE_PAIRS
>>> Info: TIME FOR READING PSF FILE: 0.231728
>>> Info: TIME FOR READING PDB FILE: 0.065865
>>> Info:
>>> Info: ****************************
>>> Info: STRUCTURE SUMMARY:
>>> Info: 49686 ATOMS
>>> Info: 34287 BONDS
>>> Info: 21815 ANGLES
>>> Info: 9486 DIHEDRALS
>>> Info: 641 IMPROPERS
>>> Info: 211 CROSSTERMS
>>> Info: 0 EXCLUSIONS
>>> Info: 46071 RIGID BONDS
>>> Info: 102987 DEGREES OF FREEDOM
>>> Info: 17232 HYDROGEN GROUPS
>>> Info: 4 ATOMS IN LARGEST HYDROGEN GROUP
>>> Info: 17232 MIGRATION GROUPS
>>> Info: 4 ATOMS IN LARGEST MIGRATION GROUP
>>> Info: TOTAL MASS = 304563 amu
>>> Info: TOTAL CHARGE = 2.58163e-06 e
>>> Info: MASS DENSITY = 0.954941 g/cm^3
>>> Info: ATOM DENSITY = 0.0938154 atoms/A^3
>>> Info: *****************************
>>> Info: Reading from binary file ./min-04.restart.coor
>>> Info:
>>> Info: Entering startup at 0.368485 s, 28.2344 MB of memory in use
>>> Info: Startup phase 0 took 9.799e-05 s, 28.2461 MB of memory in use
>>> Info: ADDED 65398 IMPLICIT EXCLUSIONS
>>> Info: Startup phase 1 took 0.0204082 s, 36.7617 MB of memory in use
>>> Info: Startup phase 2 took 8.70228e-05 s, 36.8555 MB of memory in use
>>> Info: Startup phase 3 took 5.88894e-05 s, 36.8555 MB of memory in use
>>> Info: Startup phase 4 took 0.000846148 s, 39.2852 MB of memory in use
>>> Info: Startup phase 5 took 5.57899e-05 s, 39.4688 MB of memory in use
>>> Info: PATCH GRID IS 2 (PERIODIC) BY 2 (PERIODIC) BY 2 (PERIODIC)
>>> Info: PATCH GRID IS 2-AWAY BY 2-AWAY BY 2-AWAY
>>> Info: REMOVING COM VELOCITY 0.00139852 -0.000902423 0.000836289
>>> Info: LARGEST PATCH (4) HAS 6331 ATOMS
>>> Info: Startup phase 6 took 0.011281 s, 48.9883 MB of memory in use
>>> Info: PME using 6 and 6 processors for FFT and reciprocal sum.
>>> Info: PME USING 1 GRID NODES AND 1 TRANS NODES
>>> Info: PME GRID LOCATIONS: 0 1 2 3 4 5
>>> Info: PME TRANS LOCATIONS: 0 1 2 3 4 5
>>> Info: Optimizing 4 FFT steps. 1... 2... 3... 4... Done.
>>> Info: Startup phase 7 took 0.00118399 s, 50.9766 MB of memory in use
>>> Info: Startup phase 8 took 0.000118017 s, 51.2266 MB of memory in use
>>> LDB: Central LB being created...
>>> Info: Startup phase 9 took 0.000226974 s, 51.4883 MB of memory in use
>>> Info: CREATING 602 COMPUTE OBJECTS
>>> Info: NONBONDED TABLE R-SQUARED SPACING: 0.0625
>>> Info: NONBONDED TABLE SIZE: 769 POINTS
>>> Info: INCONSISTENCY IN FAST TABLE ENERGY VS FORCE: 0.000325096 AT 11.9556
>>> Info: INCONSISTENCY IN SCOR TABLE ENERGY VS FORCE: 0.000324844 AT 11.9556
>>> Info: INCONSISTENCY IN VDWA TABLE ENERGY VS FORCE: 0.0040507 AT 0.251946
>>> Info: INCONSISTENCY IN VDWB TABLE ENERGY VS FORCE: 0.00150189 AT 0.251946
>>> Pe 4 hosts 0 local and 1 remote patches for pe 4
>>> Pe 5 hosts 1 local and 0 remote patches for pe 4
>>> Pe 0 hosts 0 local and 1 remote patches for pe 4
>>> Pe 1 hosts 1 local and 0 remote patches for pe 4
>>> Pe 3 hosts 1 local and 1 remote patches for pe 4
>>> Pe 0 hosts 0 local and 1 remote patches for pe 2
>>> Pe 1 hosts 1 local and 0 remote patches for pe 2
>>> Pe 3 hosts 1 local and 1 remote patches for pe 2
>>> Pe 2 hosts 1 local and 1 remote patches for pe 2
>>> Pe 5 hosts 1 local and 0 remote patches for pe 2
>>> Pe 4 hosts 0 local and 1 remote patches for pe 2
>>> Pe 2 hosts 1 local and 1 remote patches for pe 4
>>> Info: useSync: 1 useProxySync: 0
>>> Info: Startup phase 10 took 0.102988 s, 92.8633 MB of memory in use
>>> Info: Startup phase 11 took 7.82013e-05 s, 93.1055 MB of memory in use
>>> Info: Startup phase 12 took 6.58035e-05 s, 93.1055 MB of memory in use
>>> Info: Finished startup at 0.505981 s, 93.1172 MB of memory in use
>>>
>>> TCL: Running for 20000 steps
>>> REASSIGNING VELOCITIES AT STEP 0 TO 1 KELVIN.
>>> Pe 4 has 4 local and 4 remote patches and 250 local and 250 remote
>>> computes.
>>> Pe 2 has 4 local and 4 remote patches and 250 local and 250 remote
>>> computes.
>>> ETITLE: TS BOND ANGLE DIHED
>>> IMPRP ELECT VDW BOUNDARY MISC
>>> KINETIC TOTAL TEMP POTENTIAL
>>> TOTAL3 TEMPAVG PRESSURE GPRESSURE
>>> VOLUME PRESSAVG GPRESSAVG
>>>
>>> ENERGY: 0 219.4599 632.6613 597.7338
>>> 27.3461 -231987.2055 28848.8427 0.0000
>>> 0.0000 101.7393 -201559.4222 0.9943
>>> -201661.1616 -201559.4204 0.9943 104.1592
>>> 297.1824 529614.4763 104.1592 297.1824
>>>
>>> FATAL ERROR: CUDA error in cuda_check_remote_progress on Pe 2 (gig64
>>> device 0): unspecified launch failure
>>> ------------- Processor 2 Exiting: Called CmiAbort ------------
>>> Reason: FATAL ERROR: CUDA error in cuda_check_remote_progress on Pe 2
>>> (gig64 device 0): unspecified launch failure
>>>
>>> Charm++ fatal error:
>>> FATAL ERROR: CUDA error in cuda_check_remote_progress on Pe 2 (gig64
>>> device 0): unspecified launch failure
>>> *****
>>>
>>>
>>>
>>>
>>>
>>> >
>>> > Overall those aren't very useful suggestions, good luck with it!
>>> >
>>> > ~Aron
>>> >
>>> >
>>> > On Wed, Nov 7, 2012 at 1:57 PM, Francesco Pietra <chiendarret_at_gmail.com>
>>> > wrote:
>>> >>
>>> >> Here the forwarding
>>> >>
>>> >>
>>> >> ---------- Forwarded message ----------
>>> >> From: Francesco Pietra <chiendarret_at_gmail.com>
>>> >> Date: Wed, Nov 7, 2012 at 4:26 PM
>>> >> Subject: Fwd: namd-l: CUDA error vs clashes
>>> >> To: NAMD <namd-l_at_ks.uiuc.edu>
>>> >>
>>> >>
>>> >> Hello:
>>> >> After i posted as below for misleading cuda error, I noticed, by
>>> >> re-minimizing and re-running the stepwise heating procedure, that
>>> >> alternatively to "cuda error" problems of high energy come out, as
>>> >> reported below. As I said, I was unable to fix the problem. May be
>>> >> that the output below suggests what to do. Thanks fp
>>> >>
>>> >> Below, it is not clear to me what "Info: SKIPPING rtf SECTION IN
>>> >> STREAM FILE" means, for this pentacoordinated heme protein. The stream
>>> >> is "toppar_all22_prot_heme.str, which proved OK with other
>>> >> penta-coordinated heme proteins.
>>> >>
>>> >> Atom 1982 is HIS134 HE1, which has no contacts. In previous cases, the
>>> >> flying-out atoms where H-atoms of heme. Also them had no contacts. Why
>>> >> heating from 0 to 1K products that crash?
>>> >>
>>> >> Running command: namd2 heat-01.conf +p6 +idlepoll
>>> >>
>>> >> Charm++: standalone mode (not using charmrun)
>>> >> Converse/Charm++ Commit ID: v6.4.0-beta1-0-g5776d21
>>> >> CharmLB> Load balancer assumes all CPUs are same.
>>> >> Charm++> Running on 1 unique compute nodes (12-way SMP).
>>> >> Charm++> cpu topology info is gathered in 0.001 seconds.
>>> >> Info: NAMD CVS-2012-09-26 for Linux-x86_64-multicore-CUDA
>>> >> Info:
>>> >> Info: Please visit http://www.ks.uiuc.edu/Research/namd/
>>> >> Info: for updates, documentation, and support information.
>>> >> Info:
>>> >> Info: Please cite Phillips et al., J. Comp. Chem. 26:1781-1802 (2005)
>>> >> Info: in all publications reporting results obtained with NAMD.
>>> >> Info:
>>> >> Info: Based on Charm++/Converse 60400 for multicore-linux64-iccstatic
>>> >> Info: Built Wed Sep 26 02:25:08 CDT 2012 by jim on lisboa.ks.uiuc.edu
>>> >> Info: 1 NAMD CVS-2012-09-26 Linux-x86_64-multicore-CUDA 6 gig64
>>> >> francesco
>>> >> Info: Running on 6 processors, 1 nodes, 1 physical nodes.
>>> >> Info: CPU topology information available.
>>> >> Info: Charm++/Converse parallel runtime startup completed at 0.010654 s
>>> >> Pe 1 physical rank 1 will use CUDA device of pe 2
>>> >> Pe 5 physical rank 5 will use CUDA device of pe 4
>>> >> Pe 3 physical rank 3 will use CUDA device of pe 4
>>> >> Pe 2 physical rank 2 binding to CUDA device 0 on gig64: 'GeForce GTX
>>> >> 680' Mem: 2047MB Rev: 3.0
>>> >> Pe 4 physical rank 4 binding to CUDA device 1 on gig64: 'GeForce GTX
>>> >> 680' Mem: 2047MB Rev: 3.0
>>> >> Did not find +devices i,j,k,... argument, using all
>>> >> Pe 0 physical rank 0 will use CUDA device of pe 2
>>> >> Info: 8.20703 MB of memory in use based on /proc/self/stat
>>> >> Info: Configuration file is heat-01.conf
>>> >> Info: Working in the current directory
>>> >> /home/francesco/work_heme-oxygenase/MD
>>> >> TCL: Suspending until startup complete.
>>> >> Info: EXTENDED SYSTEM FILE ./min-03.restart.xsc
>>> >> Info: SIMULATION PARAMETERS:
>>> >> Info: TIMESTEP 0.01
>>> >> Info: NUMBER OF STEPS 0
>>> >> Info: STEPS PER CYCLE 10
>>> >> Info: PERIODIC CELL BASIS 1 81.8 0 0
>>> >> Info: PERIODIC CELL BASIS 2 0 78.66 0
>>> >> Info: PERIODIC CELL BASIS 3 0 0 82.31
>>> >> Info: PERIODIC CELL CENTER -100.627 -13.9064 -83.667
>>> >> Info: WRAPPING ALL CLUSTERS AROUND PERIODIC BOUNDARIES ON OUTPUT.
>>> >> Info: WRAPPING TO IMAGE NEAREST TO PERIODIC CELL CENTER.
>>> >> Info: LOAD BALANCER Centralized
>>> >> Info: LOAD BALANCING STRATEGY New Load Balancers -- DEFAULT
>>> >> Info: LDB PERIOD 2000 steps
>>> >> Info: FIRST LDB TIMESTEP 50
>>> >> Info: LAST LDB TIMESTEP -1
>>> >> Info: LDB BACKGROUND SCALING 1
>>> >> Info: HOM BACKGROUND SCALING 1
>>> >> Info: PME BACKGROUND SCALING 1
>>> >> Info: MIN ATOMS PER PATCH 40
>>> >> Info: INITIAL TEMPERATURE 0.5
>>> >> Info: CENTER OF MASS MOVING INITIALLY? NO
>>> >> Info: DIELECTRIC 1
>>> >> Info: EXCLUDE SCALED ONE-FOUR
>>> >> Info: 1-4 ELECTROSTATICS SCALED BY 1
>>> >> Info: MODIFIED 1-4 VDW PARAMETERS WILL BE USED
>>> >> Info: NO DCD TRAJECTORY OUTPUT
>>> >> Info: NO EXTENDED SYSTEM TRAJECTORY OUTPUT
>>> >> Info: NO VELOCITY DCD OUTPUT
>>> >> Info: NO FORCE DCD OUTPUT
>>> >> Info: OUTPUT FILENAME ./heat-01
>>> >> Info: RESTART FILENAME ./heat-01.restart
>>> >> Info: RESTART FREQUENCY 100
>>> >> Info: BINARY RESTART FILES WILL BE USED
>>> >> Info: SWITCHING ACTIVE
>>> >> Info: SWITCHING ON 10
>>> >> Info: SWITCHING OFF 12
>>> >> Info: PAIRLIST DISTANCE 13.5
>>> >> Info: PAIRLIST SHRINK RATE 0.01
>>> >> Info: PAIRLIST GROW RATE 0.01
>>> >> Info: PAIRLIST TRIGGER 0.3
>>> >> Info: PAIRLISTS PER CYCLE 2
>>> >> Info: PAIRLISTS ENABLED
>>> >> Info: MARGIN 100
>>> >> Info: HYDROGEN GROUP CUTOFF 2.5
>>> >> Info: PATCH DIMENSION 116
>>> >> Info: ENERGY OUTPUT STEPS 1000
>>> >> Info: CROSSTERM ENERGY INCLUDED IN DIHEDRAL
>>> >> Info: TIMING OUTPUT STEPS 10000
>>> >> Info: LANGEVIN DYNAMICS ACTIVE
>>> >> Info: LANGEVIN TEMPERATURE 0
>>> >> Info: LANGEVIN USING BBK INTEGRATOR
>>> >> Info: LANGEVIN DAMPING COEFFICIENT IS 1 INVERSE PS
>>> >> Info: LANGEVIN DYNAMICS NOT APPLIED TO HYDROGENS
>>> >> Info: VELOCITY REASSIGNMENT FREQ 50
>>> >> Info: VELOCITY REASSIGNMENT TEMP 1
>>> >> Info: VELOCITY REASSIGNMENT INCR 1
>>> >> Info: VELOCITY REASSIGNMENT HOLD 311
>>> >> Info: PARTICLE MESH EWALD (PME) ACTIVE
>>> >> Info: PME TOLERANCE 1e-06
>>> >> Info: PME EWALD COEFFICIENT 0.257952
>>> >> Info: PME INTERPOLATION ORDER 4
>>> >> Info: PME GRID DIMENSIONS 90 81 90
>>> >> Info: PME MAXIMUM GRID SPACING 1
>>> >> Info: Attempting to read FFTW data from
>>> >> FFTW_NAMD_CVS-2012-09-26_Linux-x86_64-multicore-CUDA.txt
>>> >> Info: Optimizing 6 FFT steps. 1... 2... 3... 4... 5... 6... Done.
>>> >> Info: Writing FFTW data to
>>> >> FFTW_NAMD_CVS-2012-09-26_Linux-x86_64-multicore-CUDA.txt
>>> >> Info: FULL ELECTROSTATIC EVALUATION FREQUENCY 5
>>> >> Info: USING VERLET I (r-RESPA) MTS SCHEME.
>>> >> Info: C1 SPLITTING OF LONG RANGE ELECTROSTATICS
>>> >> Info: PLACING ATOMS IN PATCHES BY HYDROGEN GROUPS
>>> >> Info: RIGID BONDS TO HYDROGEN : WATER
>>> >> Info: ERROR TOLERANCE : 1e-06
>>> >> Info: MAX ITERATIONS : 100
>>> >> Info: RIGID WATER USING SETTLE ALGORITHM
>>> >> Info: RANDOM NUMBER SEED 12347
>>> >> Info: USE HYDROGEN BONDS? NO
>>> >> Info: COORDINATE PDB ./4G7T_heme_O2free_WTS_WTBOX_ION.pdb
>>> >> Info: STRUCTURE FILE ./4G7T_heme_O2free_WTS_WTBOX_ION.psf
>>> >> Info: PARAMETER file: CHARMM format!
>>> >> Info: PARAMETERS ./par_all27_prot_lipid.prm
>>> >> Info: PARAMETERS ./toppar_all22_prot_heme.str
>>> >> Info: USING ARITHMETIC MEAN TO COMBINE L-J SIGMA PARAMETERS
>>> >> Info: BINARY COORDINATES ./min-03.restart.coor
>>> >> Info: SKIPPING rtf SECTION IN STREAM FILE
>>> >> Info: SUMMARY OF PARAMETERS:
>>> >> Info: 185 BONDS
>>> >> Info: 467 ANGLES
>>> >> Info: 601 DIHEDRAL
>>> >> Info: 47 IMPROPER
>>> >> Info: 6 CROSSTERM
>>> >> Info: 121 VDW
>>> >> Info: 0 VDW_PAIRS
>>> >> Info: 0 NBTHOLE_PAIRS
>>> >> Info: TIME FOR READING PSF FILE: 0.228889
>>> >> Info: TIME FOR READING PDB FILE: 0.0652549
>>> >> Info:
>>> >> Info: ****************************
>>> >> Info: STRUCTURE SUMMARY:
>>> >> Info: 49686 ATOMS
>>> >> Info: 34287 BONDS
>>> >> Info: 21815 ANGLES
>>> >> Info: 9486 DIHEDRALS
>>> >> Info: 641 IMPROPERS
>>> >> Info: 211 CROSSTERMS
>>> >> Info: 0 EXCLUSIONS
>>> >> Info: 46071 RIGID BONDS
>>> >> Info: 102987 DEGREES OF FREEDOM
>>> >> Info: 17232 HYDROGEN GROUPS
>>> >> Info: 4 ATOMS IN LARGEST HYDROGEN GROUP
>>> >> Info: 17232 MIGRATION GROUPS
>>> >> Info: 4 ATOMS IN LARGEST MIGRATION GROUP
>>> >> Info: TOTAL MASS = 304563 amu
>>> >> Info: TOTAL CHARGE = 2.58163e-06 e
>>> >> Info: MASS DENSITY = 0.954941 g/cm^3
>>> >> Info: ATOM DENSITY = 0.0938154 atoms/A^3
>>> >> Info: *****************************
>>> >> Info: Reading from binary file ./min-03.restart.coor
>>> >> Info:
>>> >> Info: Entering startup at 0.359897 s, 28.1562 MB of memory in use
>>> >> Info: Startup phase 0 took 0.000100851 s, 28.168 MB of memory in use
>>> >> Info: ADDED 65398 IMPLICIT EXCLUSIONS
>>> >> Info: Startup phase 1 took 0.0204542 s, 36.6836 MB of memory in use
>>> >> Info: Startup phase 2 took 8.70228e-05 s, 36.7773 MB of memory in use
>>> >> Info: Startup phase 3 took 5.19753e-05 s, 36.7773 MB of memory in use
>>> >> Info: Startup phase 4 took 0.000445843 s, 39.1758 MB of memory in use
>>> >> Info: Startup phase 5 took 5.4121e-05 s, 39.3203 MB of memory in use
>>> >> Info: PATCH GRID IS 2 (PERIODIC) BY 2 (PERIODIC) BY 2 (PERIODIC)
>>> >> Info: PATCH GRID IS 2-AWAY BY 2-AWAY BY 2-AWAY
>>> >> Info: REMOVING COM VELOCITY 0.000933667 0.00123997 -0.000505543
>>> >> Info: LARGEST PATCH (4) HAS 6331 ATOMS
>>> >> Info: Startup phase 6 took 0.0114 s, 48.8281 MB of memory in use
>>> >> Info: PME using 6 and 6 processors for FFT and reciprocal sum.
>>> >> Info: PME USING 1 GRID NODES AND 1 TRANS NODES
>>> >> Info: PME GRID LOCATIONS: 0 1 2 3 4 5
>>> >> Info: PME TRANS LOCATIONS: 0 1 2 3 4 5
>>> >> Info: Optimizing 4 FFT steps. 1... 2... 3... 4... Done.
>>> >> Info: Startup phase 7 took 0.000933886 s, 51.0664 MB of memory in use
>>> >> Info: Startup phase 8 took 0.000126123 s, 51.0664 MB of memory in use
>>> >> LDB: Central LB being created...
>>> >> Info: Startup phase 9 took 0.000140905 s, 51.3281 MB of memory in use
>>> >> Info: CREATING 602 COMPUTE OBJECTS
>>> >> Info: NONBONDED TABLE R-SQUARED SPACING: 0.0625
>>> >> Info: NONBONDED TABLE SIZE: 769 POINTS
>>> >> Info: INCONSISTENCY IN FAST TABLE ENERGY VS FORCE: 0.000325096 AT
>>> >> 11.9556
>>> >> Info: INCONSISTENCY IN SCOR TABLE ENERGY VS FORCE: 0.000324844 AT
>>> >> 11.9556
>>> >> Info: INCONSISTENCY IN VDWA TABLE ENERGY VS FORCE: 0.0040507 AT
>>> >> 0.251946
>>> >> Info: INCONSISTENCY IN VDWB TABLE ENERGY VS FORCE: 0.00150189 AT
>>> >> 0.251946
>>> >> Pe 4 hosts 0 local and 1 remote patches for pe 4
>>> >> Pe 0 hosts 0 local and 1 remote patches for pe 4
>>> >> Pe 5 hosts 1 local and 0 remote patches for pe 4
>>> >> Pe 3 hosts 1 local and 1 remote patches for pe 4
>>> >> Pe 1 hosts 1 local and 0 remote patches for pe 4
>>> >> Pe 3 hosts 1 local and 1 remote patches for pe 2
>>> >> Pe 5 hosts 1 local and 0 remote patches for pe 2
>>> >> Pe 4 hosts 0 local and 1 remote patches for pe 2
>>> >> Pe 0 hosts 0 local and 1 remote patches for pe 2
>>> >> Pe 1 hosts 1 local and 0 remote patches for pe 2
>>> >> Pe 2 hosts 1 local and 1 remote patches for pe 2
>>> >> Pe 2 hosts 1 local and 1 remote patches for pe 4
>>> >> Info: useSync: 1 useProxySync: 0
>>> >> Info: Startup phase 10 took 0.102759 s, 91.7188 MB of memory in use
>>> >> Info: Startup phase 11 took 7.60555e-05 s, 91.8516 MB of memory in use
>>> >> Info: Startup phase 12 took 0.000114918 s, 91.8516 MB of memory in use
>>> >> Info: Finished startup at 0.496642 s, 91.8711 MB of memory in use
>>> >>
>>> >> TCL: Running for 20000 steps
>>> >> REASSIGNING VELOCITIES AT STEP 0 TO 1 KELVIN.
>>> >> Pe 2 has 4 local and 4 remote patches and 250 local and 250 remote
>>> >> computes.
>>> >> Pe 4 has 4 local and 4 remote patches and 250 local and 250 remote
>>> >> computes.
>>> >> ERROR: Atom 1982 velocity is -4.47844e+17 -2.29595e+18 -2.74304e+18
>>> >> (limit is 1.2e+06, atom 81 of 6230 on patch 5 pe 1)
>>> >> ERROR: Atoms moving too fast; simulation has become unstable (1 atoms
>>> >> on patch 5 pe 1).
>>> >> ETITLE: TS BOND ANGLE DIHED
>>> >> IMPRP ELECT VDW BOUNDARY MISC
>>> >> KINETIC TOTAL TEMP POTENTIAL
>>> >> TOTAL3 TEMPAVG PRESSURE GPRESSURE
>>> >> VOLUME PRESSAVG GPRESSAVG
>>> >>
>>> >> ENERGY: 0 219.4599 632.6613 597.7338
>>> >> 27.3461 -231987.2055 -9999999999.9999 0.0000
>>> >> 0.0000 102.3055 -9999999999.9999 0.9998
>>> >> -9999999999.9999 9999999999.9999 0.9998 9999999999.9999
>>> >> 9999999999.9999 529614.4763 9999999999.9999 9999999999.9999
>>> >>
>>> >> ERROR: Exiting prematurely; see error messages above.
>>> >> ====================================================
>>> >>
>>> >> WallClock: 1.033846 CPUTime: 1.033846 Memory: 195.863281 MB
>>> >> Program finished.
>>> >>
>>> >>
>>> >> ---------- Forwarded message ----------
>>> >> From: Francesco Pietra <chiendarret_at_gmail.com>
>>> >> Date: Wed, Nov 7, 2012 at 12:33 PM
>>> >> Subject: namd-l: CUDA error (as a misleding error message)
>>> >> To: NAMD <namd-l_at_ks.uiuc.edu>
>>> >>
>>> >>
>>> >> Hello:
>>> >> System: protein in a water box.
>>> >>
>>> >> Minimization (namd 2.9 cuda linux) went on regularly up to very low
>>> >> gradient.
>>> >>
>>> >> Gentle heating crashed because of atoms moving too fast. I was unable
>>> >> to detect clashes. various methods revealed none. Reducing ts,
>>> >> increasing outputenergy, increasing margin, diminishing heating per
>>> >> step, did not help.
>>> >>
>>> >> Further minimization. Now, CUDA error (which could be reproduced;
>>> >> check with another system: CUDA OK)
>>> >>
>>> >> francesco_at_gig64:~/MD$ charmrun namd2 heat-01.conf +p6 +idlepoll 2>&1 |
>>> >> tee heat-01.log
>>> >> Running command: namd2 heat-01.conf +p6 +idlepoll
>>> >>
>>> >> Charm++: standalone mode (not using charmrun)
>>> >> Converse/Charm++ Commit ID: v6.4.0-beta1-0-g5776d21
>>> >> CharmLB> Load balancer assumes all CPUs are same.
>>> >> Charm++> Running on 1 unique compute nodes (12-way SMP).
>>> >> Charm++> cpu topology info is gathered in 0.001 seconds.
>>> >> Info: NAMD CVS-2012-09-26 for Linux-x86_64-multicore-CUDA
>>> >> Info:
>>> >> Info: Please visit http://www.ks.uiuc.edu/Research/namd/
>>> >> Info: for updates, documentation, and support information.
>>> >> Info:
>>> >> Info: Please cite Phillips et al., J. Comp. Chem. 26:1781-1802 (2005)
>>> >> Info: in all publications reporting results obtained with NAMD.
>>> >> Info:
>>> >> Info: Based on Charm++/Converse 60400 for multicore-linux64-iccstatic
>>> >> Info: Built Wed Sep 26 02:25:08 CDT 2012 by jim on lisboa.ks.uiuc.edu
>>> >> Info: 1 NAMD CVS-2012-09-26 Linux-x86_64-multicore-CUDA 6 gig64
>>> >> francesco
>>> >> Info: Running on 6 processors, 1 nodes, 1 physical nodes.
>>> >> Info: CPU topology information available.
>>> >> Info: Charm++/Converse parallel runtime startup completed at 0.0102639
>>> >> s
>>> >> Pe 3 physical rank 3 will use CUDA device of pe 4
>>> >> Pe 5 physical rank 5 will use CUDA device of pe 4
>>> >> Pe 1 physical rank 1 will use CUDA device of pe 2
>>> >> Pe 2 physical rank 2 binding to CUDA device 0 on gig64: 'GeForce GTX
>>> >> 680' Mem: 2047MB Rev: 3.0
>>> >> Pe 4 physical rank 4 binding to CUDA device 1 on gig64: 'GeForce GTX
>>> >> 680' Mem: 2047MB Rev: 3.0
>>> >> Did not find +devices i,j,k,... argument, using all
>>> >> Pe 0 physical rank 0 will use CUDA device of pe 2
>>> >> Info: 8.22656 MB of memory in use based on /proc/self/stat
>>> >> Info: Configuration file is heat-01.conf
>>> >> Info: Working in the current directory
>>> >> /home/francesco/work_heme-oxygenase/MD
>>> >> TCL: Suspending until startup complete.
>>> >> Info: EXTENDED SYSTEM FILE ./min-02.restart.xsc
>>> >> Info: SIMULATION PARAMETERS:
>>> >> Info: TIMESTEP 0.01
>>> >> Info: NUMBER OF STEPS 0
>>> >> Info: STEPS PER CYCLE 10
>>> >> Info: PERIODIC CELL BASIS 1 81.8 0 0
>>> >> Info: PERIODIC CELL BASIS 2 0 78.66 0
>>> >> Info: PERIODIC CELL BASIS 3 0 0 82.31
>>> >> Info: PERIODIC CELL CENTER -100.627 -13.9064 -83.667
>>> >> Info: WRAPPING ALL CLUSTERS AROUND PERIODIC BOUNDARIES ON OUTPUT.
>>> >> Info: WRAPPING TO IMAGE NEAREST TO PERIODIC CELL CENTER.
>>> >> Info: LOAD BALANCER Centralized
>>> >> Info: LOAD BALANCING STRATEGY New Load Balancers -- DEFAULT
>>> >> Info: LDB PERIOD 2000 steps
>>> >> Info: FIRST LDB TIMESTEP 50
>>> >> Info: LAST LDB TIMESTEP -1
>>> >> Info: LDB BACKGROUND SCALING 1
>>> >> Info: HOM BACKGROUND SCALING 1
>>> >> Info: PME BACKGROUND SCALING 1
>>> >> Info: MIN ATOMS PER PATCH 40
>>> >> Info: INITIAL TEMPERATURE 0.5
>>> >> Info: CENTER OF MASS MOVING INITIALLY? NO
>>> >> Info: DIELECTRIC 1
>>> >> Info: EXCLUDE SCALED ONE-FOUR
>>> >> Info: 1-4 ELECTROSTATICS SCALED BY 1
>>> >> Info: MODIFIED 1-4 VDW PARAMETERS WILL BE USED
>>> >> Info: NO DCD TRAJECTORY OUTPUT
>>> >> Info: NO EXTENDED SYSTEM TRAJECTORY OUTPUT
>>> >> Info: NO VELOCITY DCD OUTPUT
>>> >> Info: NO FORCE DCD OUTPUT
>>> >> Info: OUTPUT FILENAME ./heat-01
>>> >> Info: RESTART FILENAME ./heat-01.restart
>>> >> Info: RESTART FREQUENCY 100
>>> >> Info: BINARY RESTART FILES WILL BE USED
>>> >> Info: SWITCHING ACTIVE
>>> >> Info: SWITCHING ON 10
>>> >> Info: SWITCHING OFF 12
>>> >> Info: PAIRLIST DISTANCE 13.5
>>> >> Info: PAIRLIST SHRINK RATE 0.01
>>> >> Info: PAIRLIST GROW RATE 0.01
>>> >> Info: PAIRLIST TRIGGER 0.3
>>> >> Info: PAIRLISTS PER CYCLE 2
>>> >> Info: PAIRLISTS ENABLED
>>> >> Info: MARGIN 100
>>> >> Info: HYDROGEN GROUP CUTOFF 2.5
>>> >> Info: PATCH DIMENSION 116
>>> >> Info: ENERGY OUTPUT STEPS 1000
>>> >> Info: CROSSTERM ENERGY INCLUDED IN DIHEDRAL
>>> >> Info: TIMING OUTPUT STEPS 10000
>>> >> Info: LANGEVIN DYNAMICS ACTIVE
>>> >> Info: LANGEVIN TEMPERATURE 0
>>> >> Info: LANGEVIN USING BBK INTEGRATOR
>>> >> Info: LANGEVIN DAMPING COEFFICIENT IS 1 INVERSE PS
>>> >> Info: LANGEVIN DYNAMICS NOT APPLIED TO HYDROGENS
>>> >> Info: VELOCITY REASSIGNMENT FREQ 50
>>> >> Info: VELOCITY REASSIGNMENT TEMP 1
>>> >> Info: VELOCITY REASSIGNMENT INCR 1
>>> >> Info: VELOCITY REASSIGNMENT HOLD 311
>>> >> Info: PARTICLE MESH EWALD (PME) ACTIVE
>>> >> Info: PME TOLERANCE 1e-06
>>> >> Info: PME EWALD COEFFICIENT 0.257952
>>> >> Info: PME INTERPOLATION ORDER 4
>>> >> Info: PME GRID DIMENSIONS 90 81 90
>>> >> Info: PME MAXIMUM GRID SPACING 1
>>> >> Info: Attempting to read FFTW data from
>>> >> FFTW_NAMD_CVS-2012-09-26_Linux-x86_64-multicore-CUDA.txt
>>> >> Info: Optimizing 6 FFT steps. 1... 2... 3... 4... 5... 6... Done.
>>> >> Info: Writing FFTW data to
>>> >> FFTW_NAMD_CVS-2012-09-26_Linux-x86_64-multicore-CUDA.txt
>>> >> Info: FULL ELECTROSTATIC EVALUATION FREQUENCY 5
>>> >> Info: USING VERLET I (r-RESPA) MTS SCHEME.
>>> >> Info: C1 SPLITTING OF LONG RANGE ELECTROSTATICS
>>> >> Info: PLACING ATOMS IN PATCHES BY HYDROGEN GROUPS
>>> >> Info: RIGID BONDS TO HYDROGEN : WATER
>>> >> Info: ERROR TOLERANCE : 1e-06
>>> >> Info: MAX ITERATIONS : 100
>>> >> Info: RIGID WATER USING SETTLE ALGORITHM
>>> >> Info: RANDOM NUMBER SEED 12347
>>> >> Info: USE HYDROGEN BONDS? NO
>>> >> Info: COORDINATE PDB ./WTS_WTBOX_ION.pdb
>>> >> Info: STRUCTURE FILE ./WTS_WTBOX_ION.psf
>>> >> Info: PARAMETER file: CHARMM format!
>>> >> Info: PARAMETERS ./par_all27_prot_lipid.prm
>>> >> Info: PARAMETERS ./toppar_all22_prot_heme.str
>>> >> Info: USING ARITHMETIC MEAN TO COMBINE L-J SIGMA PARAMETERS
>>> >> Info: BINARY COORDINATES ./min-02.restart.coor
>>> >> Info: SKIPPING rtf SECTION IN STREAM FILE
>>> >> Info: SUMMARY OF PARAMETERS:
>>> >> Info: 185 BONDS
>>> >> Info: 467 ANGLES
>>> >> Info: 601 DIHEDRAL
>>> >> Info: 47 IMPROPER
>>> >> Info: 6 CROSSTERM
>>> >> Info: 121 VDW
>>> >> Info: 0 VDW_PAIRS
>>> >> Info: 0 NBTHOLE_PAIRS
>>> >> Info: TIME FOR READING PSF FILE: 0.239432
>>> >> Info: TIME FOR READING PDB FILE: 0.0656481
>>> >> Info:
>>> >> Info: ****************************
>>> >> Info: STRUCTURE SUMMARY:
>>> >> Info: 49686 ATOMS
>>> >> Info: 34287 BONDS
>>> >> Info: 21815 ANGLES
>>> >> Info: 9486 DIHEDRALS
>>> >> Info: 641 IMPROPERS
>>> >> Info: 211 CROSSTERMS
>>> >> Info: 0 EXCLUSIONS
>>> >> Info: 46071 RIGID BONDS
>>> >> Info: 102987 DEGREES OF FREEDOM
>>> >> Info: 17232 HYDROGEN GROUPS
>>> >> Info: 4 ATOMS IN LARGEST HYDROGEN GROUP
>>> >> Info: 17232 MIGRATION GROUPS
>>> >> Info: 4 ATOMS IN LARGEST MIGRATION GROUP
>>> >> Info: TOTAL MASS = 304563 amu
>>> >> Info: TOTAL CHARGE = 2.58163e-06 e
>>> >> Info: MASS DENSITY = 0.954941 g/cm^3
>>> >> Info: ATOM DENSITY = 0.0938154 atoms/A^3
>>> >> Info: *****************************
>>> >> Info: Reading from binary file ./min-02.restart.coor
>>> >> Info:
>>> >> Info: Entering startup at 0.392675 s, 28.2539 MB of memory in use
>>> >> Info: Startup phase 0 took 9.60827e-05 s, 28.2656 MB of memory in use
>>> >> Info: ADDED 65398 IMPLICIT EXCLUSIONS
>>> >> Info: Startup phase 1 took 0.02037 s, 36.7812 MB of memory in use
>>> >> Info: Startup phase 2 took 8.70228e-05 s, 36.8711 MB of memory in use
>>> >> Info: Startup phase 3 took 3.98159e-05 s, 36.8711 MB of memory in use
>>> >> Info: Startup phase 4 took 0.000478029 s, 39.2539 MB of memory in use
>>> >> Info: Startup phase 5 took 4.69685e-05 s, 39.4023 MB of memory in use
>>> >> Info: PATCH GRID IS 2 (PERIODIC) BY 2 (PERIODIC) BY 2 (PERIODIC)
>>> >> Info: PATCH GRID IS 2-AWAY BY 2-AWAY BY 2-AWAY
>>> >> Info: REMOVING COM VELOCITY 0.000933667 0.00123997 -0.000505543
>>> >> Info: LARGEST PATCH (4) HAS 6331 ATOMS
>>> >> Info: Startup phase 6 took 0.011183 s, 48.918 MB of memory in use
>>> >> Info: PME using 6 and 6 processors for FFT and reciprocal sum.
>>> >> Info: PME USING 1 GRID NODES AND 1 TRANS NODES
>>> >> Info: PME GRID LOCATIONS: 0 1 2 3 4 5
>>> >> Info: PME TRANS LOCATIONS: 0 1 2 3 4 5
>>> >> Info: Optimizing 4 FFT steps. 1... 2... 3... 4... Done.
>>> >> Info: Startup phase 7 took 0.000873089 s, 51.1914 MB of memory in use
>>> >> Info: Startup phase 8 took 0.000119925 s, 51.1914 MB of memory in use
>>> >> LDB: Central LB being created...
>>> >> Info: Startup phase 9 took 0.000138044 s, 51.4766 MB of memory in use
>>> >> Info: CREATING 602 COMPUTE OBJECTS
>>> >> Info: NONBONDED TABLE R-SQUARED SPACING: 0.0625
>>> >> Info: NONBONDED TABLE SIZE: 769 POINTS
>>> >> Info: INCONSISTENCY IN FAST TABLE ENERGY VS FORCE: 0.000325096 AT
>>> >> 11.9556
>>> >> Info: INCONSISTENCY IN SCOR TABLE ENERGY VS FORCE: 0.000324844 AT
>>> >> 11.9556
>>> >> Info: INCONSISTENCY IN VDWA TABLE ENERGY VS FORCE: 0.0040507 AT
>>> >> 0.251946
>>> >> Info: INCONSISTENCY IN VDWB TABLE ENERGY VS FORCE: 0.00150189 AT
>>> >> 0.251946
>>> >> Pe 4 hosts 0 local and 1 remote patches for pe 4
>>> >> Pe 5 hosts 1 local and 0 remote patches for pe 4
>>> >> Pe 0 hosts 0 local and 1 remote patches for pe 4
>>> >> Pe 3 hosts 1 local and 1 remote patches for pe 4
>>> >> Pe 1 hosts 1 local and 0 remote patches for pe 4
>>> >> Pe 0 hosts 0 local and 1 remote patches for pe 2
>>> >> Pe 3 hosts 1 local and 1 remote patches for pe 2
>>> >> Pe 4 hosts 0 local and 1 remote patches for pe 2
>>> >> Pe 1 hosts 1 local and 0 remote patches for pe 2
>>> >> Pe 2 hosts 1 local and 1 remote patches for pe 2
>>> >> Pe 5 hosts 1 local and 0 remote patches for pe 2
>>> >> Pe 2 hosts 1 local and 1 remote patches for pe 4
>>> >> Info: useSync: 1 useProxySync: 0
>>> >> Info: Startup phase 10 took 0.107934 s, 93.6289 MB of memory in use
>>> >> Info: Startup phase 11 took 7.60555e-05 s, 93.6797 MB of memory in use
>>> >> Info: Startup phase 12 took 5.88894e-05 s, 93.6797 MB of memory in use
>>> >> Info: Finished startup at 0.534176 s, 93.6914 MB of memory in use
>>> >>
>>> >> TCL: Running for 20000 steps
>>> >> REASSIGNING VELOCITIES AT STEP 0 TO 1 KELVIN.
>>> >> Pe 2 has 4 local and 4 remote patches and 250 local and 250 remote
>>> >> computes.
>>> >> Pe 4 has 4 local and 4 remote patches and 250 local and 250 remote
>>> >> computes.
>>> >> ETITLE: TS BOND ANGLE DIHED
>>> >> IMPRP ELECT VDW BOUNDARY MISC
>>> >> KINETIC TOTAL TEMP POTENTIAL
>>> >> TOTAL3 TEMPAVG PRESSURE GPRESSURE
>>> >> VOLUME PRESSAVG GPRESSAVG
>>> >>
>>> >> ENERGY: 0 219.4599 632.6613 597.7338
>>> >> 27.3461 -231987.2055 28848.8514 0.0000
>>> >> 0.0000 102.3071 -201558.8457 0.9998
>>> >> -201661.1529 -201558.8439 0.9998 104.1807
>>> >> 297.1944 529614.4763 104.1807 297.1944
>>> >>
>>> >> LDB: ============= START OF LOAD BALANCING ============== 2.65752
>>> >> LDB: ============== END OF LOAD BALANCING =============== 2.65758
>>> >> Info: useSync: 1 useProxySync: 0
>>> >> LDB: =============== DONE WITH MIGRATION ================ 2.65797
>>> >> FATAL ERROR: CUDA error in cuda_check_remote_progress on Pe 2 (gig64
>>> >> device 0): unspecified launch failure
>>> >> ------------- Processor 2 Exiting: Called CmiAbort ------------
>>> >> Reason: FATAL ERROR: CUDA error in cuda_check_remote_progress on Pe 2
>>> >> (gig64 device 0): unspecified launch failure
>>> >>
>>> >> Charm++ fatal error:
>>> >> FATAL ERROR: CUDA error in cuda_check_remote_progress on Pe 2 (gig64
>>> >> device 0): unspecified launch failure.
>>> >> ***************
>>> >>
>>> >> Being short of ideas at this point, hope someone can think better.
>>> >>
>>> >> Thanks
>>> >>
>>> >> francesco pietra
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Aron Broom M.Sc
>>> > PhD Student
>>> > Department of Chemistry
>>> > University of Waterloo
>>> >
>>
>>
>>
>>
>> --
>> Aron Broom M.Sc
>> PhD Student
>> Department of Chemistry
>> University of Waterloo
>>

This archive was generated by hypermail 2.1.6 : Tue Dec 31 2013 - 23:22:44 CST