From: Vlad Cojocaru (vlad.cojocaru_at_mpi-muenster.mpg.de)
Date: Fri Sep 17 2021 - 08:05:40 CDT
Hi Josh
Thanks a lot for sharing this. I don't have experience with running the
GPU NAMD, this is the first time I actually decided to test it thoroughly.
I think the NAMD build was not a pure SMP built. Maybe this is where the
problems come from in the first place .... I share below the build
procedure recommended by the HPC site. If there is anything that you
immediately spot to be problematic, I could give it a try to build NAMD
again.
I will also share your email with the support team at the HPC site.
Best wishes
Vlad
#### NAMD build procedure ####
module load Intel ParaStationMPI FFTW Tcl
tar -xf NAMD_2.14_Source.tar.gz
cd NAMD_2.14_Source
tar -xf charm-6.10.2.tar
cd charm-6.10.2
./build charm++ ucx-linux-x86_64 icc smp --with-production
cd ..
./config Linux-x86_64-icc --charm-arch ucx-linux-x86_64-smp-icc --with-tcl --tcl-prefix $EBROOTTCL --cc "mpicc" --cc-opts "-O3 -march=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC -std=c++11" --cxx "mpicxx" --cxx-opts "-O3 -march=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC -std=c++11 --std=c++11" --with-tcl --tcl-prefix $EBROOTTCL --with-fftw3 --fftw-prefix $EBROOTFFTW --with-cuda
cd Linux-x86_64-icc
echo "TCLLIB=-L\$(EBROOTTCL)/lib -ltcl8.6 -ldl -lpthread" >> Make.config
echo "COPTS+=-DNAMD_DISABLE_SSE" >> Make.config
echo "CXXOPTS+=-DNAMD_DISABLE_SSE" >> Make.config
make
On 9/17/21 14:31, Vermaas, Josh wrote:
> Hi Vlad,
>
> I've run NAMD on 4000 nodes, and it'll scale just fine (although the system was much larger than 500k atoms!). There are a few gotchas involved with multinode GPU NAMD. In no particular order:
>
> 1. This is an SMP build, yeah? Straight MPI builds with CUDA support are *possible* to build, but perform terribly relative to their SMP bretheren
> 2. I've found that each node performs best when a GPU gets its own rank/task that command dedicated CPUs. On the local resources here at MSU, that looks something like this:
>
> #!/bin/bash
> #SBATCH --gres=gpu:4 #4 GPUs per node
> #SBATCH --nodes=2
> #SBATCH --ntasks-per-node=4 #Number of tasks per node should match the number of GPUs
> #SBATCH --cpus-per-task=12 #48 CPUs total, means each task gets 12
> #SBATCH --gpu-bind=single:1 #Bind the GPU to a single task. Prevents a CPU from trying to distribute work over multiple GPUs, and lowers PCIE contention
>
> cd $SLURM_SUBMIT_DIR
> module use /mnt/home/vermaasj/modules
> module load NAMD/2.14-gpu
> #other modules are loaded automatically by the NAMD module.
> srun namd2 +ppn 11 +ignoresharing configfile.namd > logfile.log
> #With this setup, NAMD sees 8 logical nodes, 4 from each physical node.
>
> 3. Set expectations appropriately. 10 nodes with 4 GPUs each = 40 GPUs. If the only thing you were doing is simulating 500k atoms (no replicas or anything), each GPU is responsible for ~10k atoms. There are two layers of communication for NAMD 2.14 on GPUs, transfers across the PCIe bus between GPU and CPU every timestep, and communication between logical nodes whenever pairlists get recomputed. If there isn't enough work for each GPU to do, those extra communication steps are going to murder performance. TLDR, at some point the scaling will break down, and for a system that small, it might happen before you think it will.
>
> 4. If the simulations you are planning are going to be regular equilibrium simulations, NAMD3 will likely be faster on modern hardware, as it eliminates CPU-GPU communication at most timesteps.
>
> -Josh
>
> On 9/17/21, 6:44 AM,"owner-namd-l_at_ks.uiuc.edu on behalf of Vlad Cojocaru" <owner-namd-l_at_ks.uiuc.edu on behalf of vlad.cojocaru_at_mpi-muenster.mpg.de> wrote:
>
> Dear all,
>
> We have been doing some tests with the CUDA (11 I believe) accelerated
> version of NAMD 2.14 on a remote supercomputer. On 1 node (96 threads, 4
> GPUs), we see a 10 fold acceleration compared to a non-CUDA NAMD 2.14.
> There is a decent scaling between 1 and 2 GPUs but from 2 to 4 GPUs
> almost no scaling. The simulation (classical MD) time per day for 500K
> atoms is similar to what expected (comparable to what is published on
> the NAMD website).
>
> However, for a large scale project, the supercomputer site requires
> scaling up to at least 10 nodes. And we are not able to get any scaling
> to more than 1 node. In fact, as soon as running on 2 nodes (with 4 GPUs
> each), the performance is getting worse than on a single node.
>
> I know that lots of details are needed to actually pinpoint the
> issue(s), many of these are architecture dependent and we do not have
> all these details.
>
> However, I would still like to ask in general if any of you has
> routinely managed to scale up the performance of the CUDA accelerated
> NAMD 2.14 with the number of nodes. And if yes, are there any general
> tips and tricks that could be tried ?
>
> Thank you for any insights !
> Vlad
>
> --
> Vlad Cojocaru, PD (Habil.), Ph.D.
> -----------------------------------------------
> Project Group Leader
> Department of Cell and Developmental Biology
> Max Planck Institute for Molecular Biomedicine
> Röntgenstrasse 20, 48149 Münster, Germany
> -----------------------------------------------
> Tel: +49-251-70365-324; Fax: +49-251-70365-399
> Email: vlad.cojocaru[at]mpi-muenster.mpg.de
> https://urldefense.com/v3/__http://www.mpi-muenster.mpg.de/43241/cojocaru__;!!DZ3fjg!rk6Iphoo5rM4f2l2YT1vs5SwsRjzDgBZatmHwqU4VwajBWBxMQ4BM2F0_C7PXf-jew$
>
>
>
-- Vlad Cojocaru, PD (Habil.), Ph.D. ----------------------------------------------- Project Group Leader Department of Cell and Developmental Biology Max Planck Institute for Molecular Biomedicine Röntgenstrasse 20, 48149 Münster, Germany ----------------------------------------------- Tel: +49-251-70365-324; Fax: +49-251-70365-399 Email: vlad.cojocaru[at]mpi-muenster.mpg.de https://urldefense.com/v3/__http://www.mpi-muenster.mpg.de/43241/cojocaru__;!!DZ3fjg!r1ccVWYnFhS8SM_XTIXz4Bvted68phFds9smvuTnBJXhCZq4CXh01dvJQFPjIgeDSQ$
This archive was generated by hypermail 2.1.6 : Fri Dec 31 2021 - 23:17:11 CST