AW: Why CPU Usage is low when I run ibverbs-smp-cuda version NAMD

From: Norman Geist (norman.geist_at_uni-greifswald.de)
Date: Tue Nov 11 2014 - 02:48:52 CST

Ok, you actually DON’T have a problem! You compare apples with oranges. To compare the performance of different binaries, you SHOULD use the same hardware. So you would want to test the ibverbs version on the machine with 4gpus+16 cores or vice versa the multicore binary on one of the 2GPU+12cores nodes.

 

Away from that, using multiple nodes introduce a new bottleneck which is network bandwidth and latency. So you always will have losses due the additional overhead and you cpus spending time in waiting for communication rather than working. This varies for different system sizes (Amdahl’s law). BUT actually your scaling isn’t that bad. From 2 to 4 nodes It scales by 46% instead of ideal 50% (u miss the 1 node case btw.).

 

So don’t care about CPU usage, only about the actual timings. Also try to namd2 “+idlepoll” which can improve parallel scaling across network.

Also for CUDA and small systems try in config:

 

twoawayx yes

 

only if that brings improvement try

 

twoawayx yes

twoawayy yes

 

only if that brings improvement try

 

twoawayx yes

twoawayy yes

twoawayz yes

 

Most of cases twoawayx is enough or already too much.

 

Norman Geist.

 

Von: owner-namd-l_at_ks.uiuc.edu [mailto:owner-namd-l_at_ks.uiuc.edu] Im Auftrag von Bin He
Gesendet: Montag, 10. November 2014 20:51
An: Norman Geist
Cc: namd-l_at_ks.uiuc.edu
Betreff: Re: namd-l: Why CPU Usage is low when I run ibverbs-smp-cuda version NAMD

 

1. Using the servers mentioned above, I got the result:

 

multicores-CUDA

GPU

CORE

TIME

        
        
4

16

64s

        

 

ibverbs-smp-cuda

GPU

CORE

NODE

TIME

        
2 per node

12 per node

2

57

        
2 per node

12 per node

4

37

 

when running ibverbs-smp-cuda, the cpu usr usage is less than 50%, and sys usage is about 30%.

 

The cpu usage is too ugly. What I want to do is to find the reason why the cpu usage is so strange.

 

2. If I want to get the best performance with cuda, what parameters in the config file I can modify?

 

 

 

------------------------
Best Regards!
Bin He
Member of IT
Unique Studio
Room 811,Building LiangSheng,1037 Luoyu Road, Wuhan 430074,P.R. China
☎:(+86) 13163260252
Weibo:何斌_HUST
Email:binhe_at_hustunique.com <mailto:Email%3Abinhe_at_hustunique.com>
Email:binhe22_at_gmail.com <mailto:Email%3Abinhe22_at_gmail.com>

 

 

2014-11-10 14:53 GMT+08:00 Norman Geist <norman.geist_at_uni-greifswald.de>:

What you observe might be expectable as the CUDA code of NAMD is officially tuned for the multicore version. BUT, do you actually notice any performance difference regarding time/step?

 

Norman Geist.

 

Von: owner-namd-l_at_ks.uiuc.edu [mailto:owner-namd-l_at_ks.uiuc.edu] Im Auftrag von Bin He
Gesendet: Samstag, 8. November 2014 08:25
An: namd-l_at_ks.uiuc.edu
Betreff: namd-l: Why CPU Usage is low when I run ibverbs-smp-cuda version NAMD

 

Hi everyone,

 

I am a fresh man to NAMD.

 

The Desc of our clusters:

cpu:E5-2670(8cores)

memory:32GB

socket:2

network:IB

GPU:k20m*2

CUDA:6.5

workload: f1atpase(numsteps2000)

 

When I run the multicores-namd version, the cpu usage is about 100% and GPU usage is about 50%;

CMD:./namd2 +p16 +devices 0,1 ../workload/f1atpase/f1atpase.namd

cpu time is about 88s.

When I run the ibverbs-smp-cuda version, the cpu usage is about just 40%us and 30 % sy. GPU usage is about 50%.

CMD:/home/gpuusr/binhe/namd/NAMD_2.10b1_Linux-x86_64-ibverbs-smp-CUDA/charmrun ++p 60 ++ppn 15 ++nodelist nodelist ++scalable-start ++verbose /home/gpuusr/binhe/namd/NAMD_2.10b1_Linux-x86_64-ibverbs-smp-CUDA/namd2 +devices 0,1 /home/gpuusr/binhe/namd/workload/f1atpase/f1atpase.namd

cpu time is about 37s.

 

when I try to use setcpuaffinity, the result is worst.

So what is wrong with my operation?

Thanks

------------------------
Best Regards!
Bin He
Member of IT
Unique Studio

 

  _____

 <http://www.avast.com/>

Diese E-Mail ist frei von Viren und Malware, denn der avast! Antivirus <http://www.avast.com/> Schutz ist aktiv.

 

 

---
Diese E-Mail ist frei von Viren und Malware, denn der avast! Antivirus Schutz ist aktiv.
http://www.avast.com

This archive was generated by hypermail 2.1.6 : Thu Dec 31 2015 - 23:21:21 CST