From: John Stone (johns_at_ks.uiuc.edu)
Date: Fri Nov 18 2011 - 11:18:44 CST

Lam,
  Okay, I see one thing that may explain the memory use issue you're
having:

On Thu, Nov 17, 2011 at 12:20:41PM -0800, lam nguyen wrote:
[...]
> Info) OpenGL renderer: Quadro FX 580/PCI/SSE2
> Info) Features: STENCIL MSAA(16) MDE CVA MTX NPOT PP PS GLSL(OVF)

The MSAA(16) indicates that your display driver is using 16 antialiasing
samples per pixel in the VMD OpenGL window. This is 4 times higher than is
normally used, and for each sample more GPU memory is used...

I would suggest that you try force-disabling multisample antialiasing
on your GPU and see if the memory usage is significantly decreased.
You should be able to do this using a command like the ones at the
top of this NVIDIA documentation page:
  http://us.download.nvidia.com/XFree86/Linux-x86_64/285.05.09/README/openglenvvariables.html

Try this:
  nvidia-settings --assign FSAA=0 --assign FSAAAppControlled=0 --assign FSAAAppEnhanced=0

If you set that and then re-run VMD, how does it affect your GPU memory use?

To revert to your previous antialiasing setting, do this:
  nvidia-settings --assign FSAA=5 --assign FSAAAppControlled=1 --assign FSAAAppEnhanced=0

To set it for something that's a potential compromise, try this:
  nvidia-settings --assign FSAA=5 --assign FSAAAppControlled=0 --assign FSAAAppEnhanced=1

Cheers,
  John Stone
  vmd_at_ks.uiuc.edu

-- 
NIH Resource for Macromolecular Modeling and Bioinformatics
Beckman Institute for Advanced Science and Technology
University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
http://www.ks.uiuc.edu/~johns/           Phone: 217-244-3349
http://www.ks.uiuc.edu/Research/vmd/       Fax: 217-244-6078