From: Jo, Sunhwan (sjo_at_anl.gov)
Date: Tue Jul 22 2014 - 13:06:12 CDT
RMSD colvar is particularly problematic if you use large number of CPUs. I also had about 10x slow down in performance when I had to include RMSD colvar in BG/Q. In addition, the scaling performance saturate pretty quickly. At the time, I had ~160K atoms and the scaling saturated after 32 cores with colvar turned on. I had total of 9 clovers (4 RMSDs).
At the time, I have found a bottleneck in the colvar implementation and I’d like to share it here. Below is the modification that I made in colvaratoms.C file and it almost speed up the simulation about 9x. The scaling also improved up to 128 cores. Find the lines ends with the comment “MOD”.
The bottleneck was gradient calculation of the best fit rotation matrix of the reference atoms. Knowing CHARMM skips gradient of rotation matrix, so I decided to skip it. It is possible this subroutine is required for ABF but I wasn’t using it so I didn’t check. I also didn’t have RMSD colvar slowing the simulation in other cluster (i.e., Kraken) where I only had to use 1 or 2 nodes.
> void cvm::atom_group::calc_fit_gradients()
> {
> if (b_dummy) return;
>
> if ((!b_center) && (!b_rotate)) return; // no fit
>
> if (cvm::debug())
> cvm::log ("Calculating fit gradients.\n");
>
> atom_group *group_for_fit = ref_pos_group ? ref_pos_group : this;
> group_for_fit->fit_gradients.assign (group_for_fit->size(), cvm::rvector (0.0, 0.0, 0.0));
>
> if (0 && b_center) { // MOD
> // add the center of geometry contribution to the gradients
> for (size_t i = 0; i < this->size(); i++) {
> // need to bring the gradients in original frame first
> cvm::rvector const atom_grad = b_rotate ?
> (rot.inverse()).rotate ((*this)[i].grad) :
> (*this)[i].grad;
> for (size_t j = 0; j < group_for_fit->size(); j++) {
> group_for_fit->fit_gradients[j] +=
> (-1.0)/(cvm::real (group_for_fit->size())) *
> atom_grad;
> }
> }
> }
>
> if (0 && b_rotate) { // MOD
>
> // add the rotation matrix contribution to the gradients
> cvm::rotation const rot_inv = rot.inverse();
> cvm::atom_pos const cog = this->center_of_geometry();
>
> for (size_t i = 0; i < this->size(); i++) {
>
> cvm::atom_pos const pos_orig = rot_inv.rotate ((b_center ? ((*this)[i].pos - cog) : ((*this)[i].pos)));
>
> for (size_t j = 0; j < group_for_fit->size(); j++) {
> // calculate \partial(R(q) \vec{x}_i)/\partial q) \cdot \partial\xi/\partial\vec{x}_i
> cvm::quaternion const dxdq =
> rot.q.position_derivative_inner (pos_orig, (*this)[i].grad);
> // multiply by \cdot {\partial q}/\partial\vec{x}_j and add it to the fit gradients
> for (size_t iq = 0; iq < 4; iq++) {
> group_for_fit->fit_gradients[j] += dxdq[iq] * rot.dQ0_1[j][iq];
> }
> }
> }
> }
> if (cvm::debug())
> cvm::log ("Done calculating fit gradients.\n");
> }
Thanks,
Sunhwan
On Jul 22, 2014, at 6:27 AM, George Patargias <gpat_at_bioacademy.gr> wrote:
> Hello,
>
> I am comparing the Benchmark times for a given system (~258,000 atoms) using
> NAMD_CVS-2014-06-02_Source version with and without the colvar module on.
>
> When the colvar module is *off* (equilibrium MD), I get something like
>
> Benchmark time: 256 CPUs 0.0268481 s/step 0.155371 days/ns 540.59 MB memory
>
> When the colvar module is *on* (moving harmonic restraint on an RMSD
> colvar calculated from 1981 C-alpha atoms), I get
>
> Benchmark time: 256 CPUs 0.320515 s/step 1.85483 days/ns 569.555 MB memory
>
> which is 11-12 times slower.
>
> Is this the actual computational cost of a colvar calculation?
>
> Also the scaling of this particular colvar calculation is not good:
>
> Benchmark time: 48 CPUs 0.331787 s/step 1.92006 days/ns
> Benchmark time: 128 CPUs 0.322472 s/step 1.86616 days/ns
> Benchmark time: 256 CPUs 0.320515 s/step 1.85483 days/ns
>
> Thanks in advance!
>
> George
>
> Dr. George Patargias
> Postdoctoral Research Fellow
> Biomedical Research Foundation
> Academy of Athens
> 4, Soranou Ephessiou
> 115 27
> Athens
> Greece
>
> Office: +302106597568
>
This archive was generated by hypermail 2.1.6 : Wed Dec 31 2014 - 23:22:38 CST