From: Vermaas, Josh (vermaasj_at_msu.edu)
Date: Thu Jan 06 2022 - 15:17:37 CST
Ok! I just wasn’t convinced by the file name!
In that case, multicore builds are dead simple. The single core case is just:
namd3 +p1 +devices 0 script.conf > log.log
In a queuing system, you can get a little fancier:
#SBATCH --gres=gpu:1
#SBATCH --gpu-bind=closest
#SBATCH --gres-flags=enforce-binding #Make sure these GPUs are on the same socket. The QPI links across sockets can be slow.
#SBATCH --nodes=1 #1 node please
#SBATCH --ntasks-per-node=1 #Start 1 process on that node
#SBATCH --cpus-per-task=1 #Each process will use 1 CPUs
#SBATCH --mem=4G #Ask for some memory. On at least my system, you can get occasional segfaults since the default memory allocation wasn’t generous enough (<1GB)
namd3 +p1 script.conf > log.log
In my experience, SLURM will hide the other devices if you only ask for one (nvidia-smi can check this pretty easily). If you want to add more GPUs, you just need one CPU per GPU, and adjust the sbatch options accordingly. You should see linearish scaling so long as each GPU has enough work to do, can communicate quickly with other GPUs working together and CUDASOAIntegrate is enabled. (https://urldefense.com/v3/__https://developer.nvidia.com/blog/delivering-up-to-9x-throughput-with-namd-v3-and-a100-gpu/__;!!DZ3fjg!qlVNCHLDmdJmdMNSUpf3iEbCjbLQxKC7suYcUnSGsrtHDkH35djY1F7Grq12gAtn9Q$ )
-Josh
From: "Sokovic, Ana Marija" <anamaria_at_uic.edu>
Date: Thursday, January 6, 2022 at 3:38 PM
To: "Vermaas, Josh" <vermaasj_at_msu.edu>
Cc: "namd-l_at_ks.uiuc.edu" <namd-l_at_ks.uiuc.edu>
Subject: Re: Building NAMD3
Hi Josh,
In my recipe you will see that I am secreting NAMD 3 with:
git checkout release-3-0-alpha-9
And during my compilation I do get indeed NAMD3:
Pe 0 physical rank 0 binding to CUDA device 0 on gpu-a-0.extreme: 'Tesla P100-PCIE-16GB' Mem: 16280MB Rev: 6.0 PCI: 0:4:0
Info: NAMD 3.0alpha9 for Linux-x86_64-multicore-CUDA
Info:
So I did build NAMD3. The question is how to run it?
Would you guys recommend this 1GPU, 1CPU scenario:
namd3 +p1 +setcpuaffinity +devices 0 +idlepoll sim2-smd-woharm_160ns_2A.step5.inp > out_sim2_smd_woharm_2A.sim2.step5.out
Or something else might be better for NAMD 3.0alpha9.
Thanks
Ana
On Jan 6, 2022, at 2:33 PM, Vermaas, Josh <vermaasj_at_msu.edu<mailto:vermaasj_at_msu.edu>> wrote:
Hi Ana,
I’m going to forward this to NAMD-L so others can benefit as well (and so David can correct anything I get overly wrong). Are you sure that is NAMD3? If you download the source code from the nightly build, I think you might be on the 2.15 codepath, rather than the 3.0 one. In git, the 3.0 code path is the “devel” branch, while the 2.15 code path is the “master” branch. If you download a tgz directly from the NAMD website (NAMD_Git-2022-01-06_Source.tar.gz), I think you are pulling the master branch rather than the devel branch. The way you could check this is to put “CUDASOAIntegrate on” in your NAMD configuration file, which will fail if you are still on the 2.15 codepath that doesn’t have this feature.
-Josh
From: "Sokovic, Ana Marija" <anamaria_at_uic.edu<mailto:anamaria_at_uic.edu>>
Date: Thursday, January 6, 2022 at 3:07 PM
To: "Vermaas, Josh" <vermaasj_at_msu.edu<mailto:vermaasj_at_msu.edu>>
Subject: Re: Building NAMD3
Hi Josh,
Thank you so much for getting back to me. I managed to build NAMD3. My recipe is in attach, I dod use a lot from your GitHub page :)
Can you please tell me what would be the best way to run it in order to get max performance?
My GPU node (with 4 GPU devices) specification is in attach.
Should I be running my codes with something like this:
namd3 +ppn 1 +setcpuaffinity +devices 0 +idlepoll sim2-smd-woharm_160ns_2A.step5.inp > out_sim2_smd_woharm_2A.sim2.step5.out
If not what what parameters should I use?
It is recommended by NAMD developers to use low CPU core counts for NAMD3.
“The single-node version of NAMD 3.0 has almost everything offloaded to the GPU, so large CPU core counts are NOT necessary to get good performance. We recommend running NAMD with a low +p count, maybe 2-4 depending on system size, especially if the user plans on running multiple replica simulations within a node.”
Can you please tell me what is your experience, which parameters could give me the best performance?
Cheers,
Ana
On Jan 6, 2022, at 11:49 AM, Vermaas, Josh <vermaasj_at_msu.edu<mailto:vermaasj_at_msu.edu>> wrote:
Hi Ana,
The first thing you need is access to the gitlab repository. If you go here: https://urldefense.com/v3/__https://gitlab.com/tcbgUIUC/namd__;!!DZ3fjg!qlVNCHLDmdJmdMNSUpf3iEbCjbLQxKC7suYcUnSGsrtHDkH35djY1F7Grq1RnGX1HA$ <https://urldefense.com/v3/__https:/nam04.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2Fnam04.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Fgitlab.com*2FtcbgUIUC*2Fnamd*26data*3D04*7C01*7Canamaria*40uic.edu*7Cd91d668fa9114def0fc608d9d13cf1a7*7Ce202cd477a564baa99e3e3b71a7c77dd*7C0*7C0*7C637770881952848469*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*26sdata*3DKeYXRKdmshcThG15YpZ8nOTPIaz5sby8GyLIMsojU2I*3D*26reserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUl!!HXCxUKc!lBtq0z3V5lzWnRj2y4K6-4tejf4QOLulPl-czMOiffXogKYSboqefNisnQ3iilY*24&data=04*7C01*7Canamaria*40uic.edu*7C18b516388b30418ca1eb08d9d153c549*7Ce202cd477a564baa99e3e3b71a7c77dd*7C0*7C0*7C637770979987978323*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=hShmVrawDvH8df3PMUsoUZBY8EzlfmndE5R3ixpTgE0*3D&reserved=0__;JSUlJSUlJSUlJSoqKioqJSUqKioqKioqKioqKiolJSolJSUlJSUlJSUlJSUlJSUlJQ!!HXCxUKc!krhkAzwaVkiHkNzS-zypeS67O8OrAp-05s5i56TMB3TszM1_gXoTBACsgrEMkx8$>, you should see a page, but if you read on, you’ll see that you need to fill out an access request form. Once you’ve done that and set your git settings appropriately, you should be able to download it.
-Josh
From: "Sokovic, Ana Marija" <anamaria_at_uic.edu<mailto:anamaria_at_uic.edu>>
Date: Thursday, January 6, 2022 at 12:06 PM
To: "Vermaas, Josh" <vermaasj_at_msu.edu<mailto:vermaasj_at_msu.edu>>
Subject: Building NAMD3
Hi Josh,
I found your instructions for building NAMD on Github. Can you please tell me from where I can download source codes for NAMD3?
I am trying to do:
fatal: HTTP request failed
Please advise,
This archive was generated by hypermail 2.1.6
: Tue Dec 13 2022 - 14:32:44 CST
git clone https://urldefense.com/v3/__https://gitlab.com/tcbgUIUC/namd.git__;!!DZ3fjg!qlVNCHLDmdJmdMNSUpf3iEbCjbLQxKC7suYcUnSGsrtHDkH35djY1F7Grq34WnupcA$ <https://urldefense.com/v3/__https:/nam04.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2Fnam04.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2Fgitlab.com*2FtcbgUIUC*2Fnamd.git__*3B!!HXCxUKc!nd84WW66uqLd1_WtdQXH8AccN84NLo7O5I-VEYKyQoad6gyotc1D6OGjSsGIIys*24*26data*3D04*7C01*7Canamaria*40uic.edu*7Cd91d668fa9114def0fc608d9d13cf1a7*7Ce202cd477a564baa99e3e3b71a7c77dd*7C0*7C0*7C637770881952848469*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*26sdata*3DkQhEdstjMXL8hn1VMphT*2FqZBXB4aRUNzpSiIb*2B8jV5g*3D*26reserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!HXCxUKc!lBtq0z3V5lzWnRj2y4K6-4tejf4QOLulPl-czMOiffXogKYSboqefNis7ISP7bs*24&data=04*7C01*7Canamaria*40uic.edu*7C18b516388b30418ca1eb08d9d153c549*7Ce202cd477a564baa99e3e3b71a7c77dd*7C0*7C0*7C637770979987978323*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=bVmYc9zQIrUqPgJit5suIp2kg3asENR2OPEGRQ7rJ7Y*3D&reserved=0__;JSUlJSUlJSUlJSoqKioqKioqKioqJSUqKioqKioqKioqKiolJSoqKiUlJSUlJSUlJSUlJSUlJSUl!!HXCxUKc!krhkAzwaVkiHkNzS-zypeS67O8OrAp-05s5i56TMB3TszM1_gXoTBACspnKt4Mo$>
Initialized empty Git repository in /software/linux-el7-nvidia/apps/namd/.git/
error: The requested URL returned error: 401 Unauthorized while accessinghttps://gitlab.com/tcbgUIUC/namd.git/info/refs
Ana