Re: about the memery lock in openMPI on infiniband

From: 刘鹏 (liupeng012_at_mail.nankai.edu.cn)
Date: Mon Mar 09 2009 - 04:27:11 CDT

Dear Axel
  Thanks!
  Thanks a lot !!
   I fix it!
   
  new problem pop up !
  when i compiling the namd!
   useing the binary file tcl-linux-ia64.tar.gz,
   
 ./confige tcl fftw plugins Linux-amd-MPI
  cd ./Linux-amd-MPI

  make

  the error message as follow

  
charmc> ----------- Module dependency search complete ------------
Adding registration for module NeighborLB
Adding registration for module commlib
charmc: Executing mpicxx -m64 -fPIC -D_REENTRANT -D__CHARMC__=1 -c moduleinit20276.C -o moduleinit20276.o
charmc: Executing mpicxx -m64 -fPIC -D_REENTRANT -rdynamic -L/hptc_cluster/NAMD/namd2/tcl/linux-ia64/lib -L/hptc_cluster/NAMD/namd2/fftw/linux-ia64/lib -I.rootdir/charm-6.0//include -DCMK_OPTIMIZE=1 -Iinc -Isrc -O3 -m64 -fexpensive-optimizations -ffast-math -o namd2 -L.rootdir/charm-6.0//bin/../lib -I.rootdir/charm-6.0//bin/../include .rootdir/charm-6.0//bin/../lib/libldb-rand.o obj/buildinfo.o obj/common.o obj/dcdlib.o obj/erf.o obj/fitrms.o obj/main.o obj/mainfunc.o obj/memusage.o obj/strlib.o obj/AlgSeven.o obj/AlgRecBisection.o obj/AlgNbor.o obj/AtomMap.o obj/BackEnd.o obj/BroadcastMgr.o obj/BroadcastClient.o obj/CollectionMaster.o obj/CollectionMgr.o obj/Communicate.o obj/Compute.o obj/ComputeAngles.o obj/ComputeBonds.o obj/ComputeConsForce.o obj/ComputeConsForceMsgs.o obj/ComputeCrossterms.o obj/ComputeCylindricalBC.o obj/ComputeDihedrals.o obj/ComputeDPME.o obj/ComputeDPMEMsgs.o obj/ComputeDPMTA.o obj/ComputeEField.o obj/ComputeEwald.o obj/ComputeExt.o obj/ComputeFullDirect.o obj/ComputeHomePatch.o ob
j/ComputeHomePatches.o obj/ComputeImpropers.o obj/ComputeGlobal.o obj/ComputeGlobalMsgs.o obj/ComputeGridForce.o obj/ComputeMap.o obj/ComputeMgr.o obj/ComputeNonbondedSelf.o obj/ComputeNonbondedPair.o obj/ComputeNonbondedUtil.o obj/ComputeNonbondedStd.o obj/ComputeNonbondedFEP.o obj/ComputeNonbondedLES.o obj/ComputeNonbondedPProf.o obj/ComputePatch.o obj/ComputePatchPair.o obj/ComputePme.o obj/ComputeRestraints.o obj/ComputeSphericalBC.o obj/ComputeStir.o obj/ComputeTclBC.o obj/ConfigList.o obj/Controller.o obj/ccsinterface.o obj/DataStream.o obj/DumpBench.o obj/FreeEnergyAssert.o obj/FreeEnergyGroup.o obj/FreeEnergyLambda.o obj/FreeEnergyLambdMgr.o obj/FreeEnergyParse.o obj/FreeEnergyRestrain.o obj/FreeEnergyRMgr.o obj/FreeEnergyVector.o obj/GlobalMaster.o obj/GlobalMasterServer.o obj/GlobalMasterTest.o obj/GlobalMasterIMD.o obj/GlobalMasterTcl.o obj/GlobalMasterSMD.o obj/GlobalMasterTMD.o obj/GlobalMasterFreeEnergy.o obj/GlobalMasterEasy.o obj/GlobalMasterMisc.o obj/colvarmodule.o obj/colvarparse.o !
 obj/colv
a
r.o obj/colvarvalue.o obj/colvarbias.o obj/colvarbias_abf.o obj/colvarbias_meta.o obj/colvaratoms.o obj/colvarcomp.o obj/colvarcomp_angles.o obj/colvarcomp_coordnums.o obj/colvarcomp_distances.o obj/colvarcomp_protein.o obj/colvarcomp_rotations.o obj/colvarproxy_namd.o obj/GridForceGrid.o obj/GromacsTopFile.o obj/heap.o obj/HomePatch.o obj/IMDOutput.o obj/InfoStream.o obj/LdbCoordinator.o obj/LJTable.o obj/Measure.o obj/MGridforceParams.o obj/MStream.o obj/MigrateAtomsMsg.o obj/Molecule.o obj/NamdCentLB.o obj/NamdNborLB.o obj/NamdState.o obj/NamdOneTools.o obj/Node.o obj/Output.o obj/Parameters.o obj/ParseOptions.o obj/Patch.o obj/PatchMgr.o obj/PatchMap.o obj/PDB.o obj/PDBData.o obj/PmeBase.o obj/PmeKSpace.o obj/PmeRealSpace.o obj/ProcessorPrivate.o obj/ProxyMgr.o obj/ProxyPatch.o obj/Rebalancer.o obj/RecBisection.o obj/ReductionMgr.o obj/RefineOnly.o obj/RefineTorusLB.o obj/ScriptTcl.o obj/Sequencer.o obj/Set.o obj/Settle.o obj/SimParameters.o obj/Sync.o obj/TclCommands.o obj/TorusLB.o obj/WorkDistrib.o ob
j/pub3dfft.o obj/vmdsock.o obj/parm.o obj/imd.o obj/CompressPsf.o obj/AtomsDisInfo.o obj/FileIO.o obj/dcdplugin.o moduleinit20276.o -lmoduleNeighborLB -lmodulecommlib .rootdir/charm-6.0//bin/../lib/libmemory-default.o .rootdir/charm-6.0//bin/../lib/libthreads-default.o -lck -lconv-cplus-y -lconv-core -lconv-util -lpthread -lckqt -ldl -ltcl8.3 -ldl -lsrfftw -lsfftw -lm -lmoduleNeighborLB -lmodulecommlib -lm
/usr/bin/ld: skipping incompatible /hptc_cluster/NAMD/namd2/tcl/linux-ia64/lib/libtcl8.3.so when searching for -ltcl8.3
/usr/bin/ld: cannot find -ltcl8.3
collect2: ld returned 1 exit status
Fatal Error by charmc in directory /hptc_cluster/NAMD/namd2/Linux-amd64-MPI
   Command mpicxx -m64 -fPIC -D_REENTRANT -rdynamic -L/hptc_cluster/NAMD/namd2/tcl/linux-ia64/lib -L/hptc_cluster/NAMD/namd2/fftw/linux-ia64/lib -I.rootdir/charm-6.0//include -DCMK_OPTIMIZE=1 -Iinc -Isrc -O3 -m64 -fexpensive-optimizations -ffast-math -o namd2 -L.rootdir/charm-6.0//bin/../lib -I.rootdir/charm-6.0//bin/../include .rootdir/charm-6.0//bin/../lib/libldb-rand.o obj/buildinfo.o obj/common.o obj/dcdlib.o obj/erf.o obj/fitrms.o obj/main.o obj/mainfunc.o obj/memusage.o obj/strlib.o obj/AlgSeven.o obj/AlgRecBisection.o obj/AlgNbor.o obj/AtomMap.o obj/BackEnd.o obj/BroadcastMgr.o obj/BroadcastClient.o obj/CollectionMaster.o obj/CollectionMgr.o obj/Communicate.o obj/Compute.o obj/ComputeAngles.o obj/ComputeBonds.o obj/ComputeConsForce.o obj/ComputeConsForceMsgs.o obj/ComputeCrossterms.o obj/ComputeCylindricalBC.o obj/ComputeDihedrals.o obj/ComputeDPME.o obj/ComputeDPMEMsgs.o obj/ComputeDPMTA.o obj/ComputeEField.o obj/ComputeEwald.o obj/ComputeExt.o obj/ComputeFullDirect.o obj/ComputeHomePatch.o obj/Compu
teHomePatches.o obj/ComputeImpropers.o obj/ComputeGlobal.o obj/ComputeGlobalMsgs.o obj/ComputeGridForce.o obj/ComputeMap.o obj/ComputeMgr.o obj/ComputeNonbondedSelf.o obj/ComputeNonbondedPair.o obj/ComputeNonbondedUtil.o obj/ComputeNonbondedStd.o obj/ComputeNonbondedFEP.o obj/ComputeNonbondedLES.o obj/ComputeNonbondedPProf.o obj/ComputePatch.o obj/ComputePatchPair.o obj/ComputePme.o obj/ComputeRestraints.o obj/ComputeSphericalBC.o obj/ComputeStir.o obj/ComputeTclBC.o obj/ConfigList.o obj/Controller.o obj/ccsinterface.o obj/DataStream.o obj/DumpBench.o obj/FreeEnergyAssert.o obj/FreeEnergyGroup.o obj/FreeEnergyLambda.o obj/FreeEnergyLambdMgr.o obj/FreeEnergyParse.o obj/FreeEnergyRestrain.o obj/FreeEnergyRMgr.o obj/FreeEnergyVector.o obj/GlobalMaster.o obj/GlobalMasterServer.o obj/GlobalMasterTest.o obj/GlobalMasterIMD.o obj/GlobalMasterTcl.o obj/GlobalMasterSMD.o obj/GlobalMasterTMD.o obj/GlobalMasterFreeEnergy.o obj/GlobalMasterEasy.o obj/GlobalMasterMisc.o obj/colvarmodule.o obj/colvarparse.o obj/col!
 var.o ob
j
/colvarvalue.o obj/colvarbias.o obj/colvarbias_abf.o obj/colvarbias_meta.o obj/colvaratoms.o obj/colvarcomp.o obj/colvarcomp_angles.o obj/colvarcomp_coordnums.o obj/colvarcomp_distances.o obj/colvarcomp_protein.o obj/colvarcomp_rotations.o obj/colvarproxy_namd.o obj/GridForceGrid.o obj/GromacsTopFile.o obj/heap.o obj/HomePatch.o obj/IMDOutput.o obj/InfoStream.o obj/LdbCoordinator.o obj/LJTable.o obj/Measure.o obj/MGridforceParams.o obj/MStream.o obj/MigrateAtomsMsg.o obj/Molecule.o obj/NamdCentLB.o obj/NamdNborLB.o obj/NamdState.o obj/NamdOneTools.o obj/Node.o obj/Output.o obj/Parameters.o obj/ParseOptions.o obj/Patch.o obj/PatchMgr.o obj/PatchMap.o obj/PDB.o obj/PDBData.o obj/PmeBase.o obj/PmeKSpace.o obj/PmeRealSpace.o obj/ProcessorPrivate.o obj/ProxyMgr.o obj/ProxyPatch.o obj/Rebalancer.o obj/RecBisection.o obj/ReductionMgr.o obj/RefineOnly.o obj/RefineTorusLB.o obj/ScriptTcl.o obj/Sequencer.o obj/Set.o obj/Settle.o obj/SimParameters.o obj/Sync.o obj/TclCommands.o obj/TorusLB.o obj/WorkDistrib.o obj/pub3d
fft.o obj/vmdsock.o obj/parm.o obj/imd.o obj/CompressPsf.o obj/AtomsDisInfo.o obj/FileIO.o obj/dcdplugin.o moduleinit20276.o -lmoduleNeighborLB -lmodulecommlib .rootdir/charm-6.0//bin/../lib/libmemory-default.o .rootdir/charm-6.0//bin/../lib/libthreads-default.o -lck -lconv-cplus-y -lconv-core -lconv-util -lpthread -lckqt -ldl -ltcl8.3 -ldl -lsrfftw -lsfftw -lm -lmoduleNeighborLB -lmodulecommlib -lm returned error code 1
charmc exiting...
rm -f moduleinit20276.C moduleinit20276.o
make: *** [namd2] Error 1

 may be !
 the problem is that tcl is incompatible with the namd!

  how to fix this ??
  
 compile tcl for the new platform??

   thanks !

2009-03-09

--------------------------------------------------------------------------------

liupeng012

--------------------------------------------------------------------------------

发件人: Axel Kohlmeyer
发送时间: 2009-03-09 13:39:30
收件人: 刘鹏
抄送:
主题: Re: namd-l: about the memery lock in openMPI on infiniband
On Sun, 2009-03-08 at 16:55 +0800, 刘鹏 wrote:
> Hi!

>
> Everyone!
> iIcompile NAMD/charm++ on x86_64 cluster,the network is infiniband
> the compiler is OpenMPI/mpicxx.
>
> when compiling the charmm++(charmm-6.0)., I get the message that
>
> "charm++ built successfully.
> Next, try out a sample program like tests/charm++/simplearrayhello "
>
> next ,i test the ./tests/charm++/simplearrayhello
>
> it pop up
>
> #ibibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> # This will severely limit memory registrations.
> #--------------------------------------------------------------------------
> #The OpenIB BTL failed to initialize while trying to allocate some
> #locked memory. This typically can indicate that the memlock limits
> #are set too low. For most HPC installations, the memlock limits
> #should be set to "unlimited". The failure occured here:
>
> I type the "ulimit -l unlimited" in the command line.

that is really more of a unix/infiniband question:
have you checked whether the locked memory area was
actually increased from your command? my guess is not,
since on most machine the 32k limit is a "hard" limit,
i.e. you have to have the sysadmin change the hard/soft
limits appropriately (and on the _whole_ machine, it doesn't
help to just do it on the node where you launch mpirun).
this is something a sysadmin _should_ know about. how old
is that machine? did anybody ever run a proper MPI job on it?
cheers,
   axel.
>
> but the problem is the same !
>
> what should i do next??
>
>
> thanks!
> peng liu
>

-- 
=======================================================================
Axel Kohlmeyer   akohlmey_at_cmm.chem.upenn.edu   http://www.cmm.upenn.edu
   Center for Molecular Modeling   --   University of Pennsylvania
Department of Chemistry, 231 S.34th Street, Philadelphia, PA 19104-6323
tel: 1-215-898-1582,  fax: 1-215-573-6233,  office-tel: 1-215-898-5425
=======================================================================
If you make something idiot-proof, the universe creates a better idiot.
__________ Information from ESET NOD32 Antivirus, version of virus signature database 3917 (20090307) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com

This archive was generated by hypermail 2.1.6 : Wed Feb 29 2012 - 15:52:27 CST