From: John Stone (johns_at_ks.uiuc.edu)
Date: Fri Sep 26 2014 - 13:40:48 CDT

Hi,
  If 0xf works, then cards 0,1,2,3 all work, and it is card 4
that does not work for some reason. I'm not familiar with the
Quadro 2000D, but that would seem to be the card that is problematic.

It is just as well that you avoid using the Quadro 2000D, since the
four Tesla K20Xm boards are all Kepler boards and they will greatly
outperform the Quadro 2000D on basically all tasks.

I'll work on adding code to list the OptiX-capable devices
so fiddling with the device mask like this is easier in the future.

The next beta version of VMD will include a newer version of OptiX
which is required for support of the latest Maxwell GPUs like the
GeForce 980s that just came out.

Cheers,
  John Stone
  vmd_at_ks.uiuc.edu

On Fri, Sep 26, 2014 at 01:12:14PM -0400, Stober, Spencer T wrote:
> John,
>
> Thanks again... here are the results. 0xf allows all 4 cards to work. So which card does not work, 3 or 4?
>
> MASK DEV. RESULT
> 0x1 0 works
> 0x3 0,1 works
> 0x5 0,2 works
> 0x1d 0,2,3,4 does not work
> 0x7 0,1,2 works
> 0xf 0,1,2,? works
> 0x17 0,1,2,? does not work
>
> Best regards, Spence
>
> Spencer T. Stober, Ph.D.
> ExxonMobil Research and Engineering
> 600 Billingsport Rd
> Paulsboro, NJ 08066
> Phone: 856-224-2638
> Email: spencer.t.stober_at_exxonmobil.com
>
>
> -----Original Message-----
> From: John Stone [mailto:johns_at_ks.uiuc.edu]
> Sent: Friday, September 26, 2014 11:43 AM
> To: Stober, Spencer T
> Cc: vmd-l_at_ks.uiuc.edu
> Subject: Re: vmd-l: VMD 1.9.2 b1 OptiX Tachyon Error
>
> Spencer,
> If devices 0, 1, and 2 work, then try this mask for those three:
> setenv VMDOPTIXDEVICEMASK 0x7
>
> Then, to add the last one, you would try either:
> setenv VMDOPTIXDEVICEMASK 0xf
> or
> setenv VMDOPTIXDEVICEMASK 0x17
>
> Let me know if either 0xf or 0x17 cures your problem and allows you
> to have all four of your K20Xm cards usable.
>
> Cheers,
> John Stone
> vmd_at_ks.uiuc.edu
>
>
> On Fri, Sep 26, 2014 at 11:35:07AM -0400, Stober, Spencer T wrote:
> > Hi John,
> >
> >
> >
> > Thanks for the fast reply.
> >
> >
> >
> > I think that I almost have it fixed, here is a test I ran:
> >
> >
> >
> > MASK DEV. RESULT
> >
> > 0x1 0 works
> >
> > 0x3 0,1 works
> >
> > 0x5 0,2 works
> >
> > 0x1d 0,2,3,4 does not work
> >
> >
> >
> > So, devices 0,1, and 2 work. Device 3 or 4 must not be compatible.
> > Therefore, I think that my (four) K20X's work, but my (single) Quadro
> > 2000D does not. Presumably, the Quadro is either device 3 or 4.
> >
> >
> >
> > I'm not sure how the mask hex number translates to which devices will be
> > used, if you could possibly send the following masks, one of them must be
> > OK:
> >
> >
> >
> > Devices 0,1,2,3
> >
> > Devices 0,1,2,4
> >
> >
> >
> > Also, below is the output from nividia-smi (if you still need it). Device
> > 0 (the Quadro) is missing from the startup information in VMD because I
> > set export VMDCUDANODISPLAYGPUS=1.
> >
> >
> >
> > Thanks very much for your help; I'm happy to run a few tests to help
> > support the beta release.
> >
> >
> >
> > Best regards, Spence
> >
> >
> >
> > ~~~~~~~~~~~~~~~ nvidia-smi ~~~~~~~~~~~~~~~~~~
> >
> >
> >
> > Fri Sep 26 11:19:49 2014
> >
> > +------------------------------------------------------+
> >
> > | NVIDIA-SMI 331.20 Driver Version: 331.20
> > |
> >
> > |-------------------------------+----------------------+----------------------+
> >
> > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
> > ECC |
> >
> > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute
> > M. |
> >
> > |===============================+======================+======================|
> >
> > | 0 Quadro 2000D Off | 0000:01:00.0 On |
> > N/A |
> >
> > | 30% 39C P0 N/A / N/A | 198MiB / 1023MiB | 6%
> > Default |
> >
> > +-------------------------------+----------------------+----------------------+
> >
> > | 1 Tesla K20Xm Off | 0000:02:00.0 Off
> > | 0 |
> >
> > | N/A 36C P0 56W / 235W | 82MiB / 5759MiB | 0%
> > Default |
> >
> > +-------------------------------+----------------------+----------------------+
> >
> > | 2 Tesla K20Xm Off | 0000:03:00.0 Off
> > | 0 |
> >
> > | N/A 36C P0 55W / 235W | 82MiB / 5759MiB | 0%
> > Default |
> >
> > +-------------------------------+----------------------+----------------------+
> >
> > | 3 Tesla K20Xm Off | 0000:83:00.0 Off
> > | 0 |
> >
> > | N/A 36C P0 57W / 235W | 82MiB / 5759MiB | 0%
> > Default |
> >
> > +-------------------------------+----------------------+----------------------+
> >
> > | 4 Tesla K20Xm Off | 0000:84:00.0 Off
> > | 0 |
> >
> > | N/A 36C P0 55W / 235W | 82MiB / 5759MiB | 0%
> > Default |
> >
> > +-------------------------------+----------------------+----------------------+
> >
> >
> >
> > +-----------------------------------------------------------------------------+
> >
> > | Compute processes: GPU
> > Memory |
> >
> > | GPU PID Process name
> > Usage |
> >
> > |=============================================================================|
> >
> > | 1 5951 /data/lib/vmd/vmd_LINUXAMD64
> > 369MiB |
> >
> > | 2 5951 /data/lib/vmd/vmd_LINUXAMD64
> > 369MiB |
> >
> > | 3 5951 /data/lib/vmd/vmd_LINUXAMD64
> > 369MiB |
> >
> > | 4 5951 /data/lib/vmd/vmd_LINUXAMD64
> > 369MiB |
> >
> > +-----------------------------------------------------------------------------+
> >
> >
> >
> >
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Spencer T. Stober, Ph.D.
> >
> > ExxonMobil Research and Engineering
> >
> > 600 Billingsport Rd
> >
> > Paulsboro, NJ 08066
> >
> > Phone: 856-224-2638
> >
> > Email: spencer.t.stober_at_exxonmobil.com
> >
> >
> >
> > -----Original Message-----
> > From: John Stone [mailto:johns_at_ks.uiuc.edu]
> > Sent: Friday, September 26, 2014 10:56 AM
> > To: Stober, Spencer T
> > Cc: vmd-l_at_ks.uiuc.edu
> > Subject: Re: vmd-l: VMD 1.9.2 b1 OptiX Tachyon Error
> >
> >
> >
> > Spencer,
> >
> > The CUDA device list shown by VMD seems to be missing device [1],
> >
> > what kind of GPU is device [1]? Can you email the output of "nvidia-smi"?
> >
> >
> >
> > Most likely what's going on here is that you have a mix of GPU hardware
> >
> > generations, and OptiX is trying to use all 5 of them, although [1] is
> >
> > perhaps not usable for some reason. If so, you'll need to tell VMD not
> >
> > to let OptiX try to use the problematic GPU, by setting the
> > VMDOPTIXDEVICEMASK
> >
> > environment variablle appropriately.
> >
> >
> >
> > In C-shell you would do:
> >
> > setenv VMDOPTIXDEVICEMASK 0x1
> >
> > in bourne/bash shell you would do
> >
> > VMDOPTIXDEVICEMASK=0x1
> >
> > export VMDOPTIXDEVICEMASK
> >
> >
> >
> > The device mask represents binary bits associated with each GPU.
> >
> > Here are some simple examples:
> >
> >
> >
> > Use only device 0:
> >
> > setenv VMDOPTIXDEVICEMASK 0x1
> >
> > Use devices 0, 1:
> >
> > setenv VMDOPTIXDEVICEMASK 0x3
> >
> > Use devices 0, 2:
> >
> > setenv VMDOPTIXDEVICEMASK 0x5
> >
> > Use devices 0, 2, 3, 4
> >
> > setenv VMDOPTIXDEVICEMASK 0x1d
> >
> >
> >
> > The tricky part is that OptiX uses a different device numbering than
> >
> > CUDA does, so you may have to fiddle with the mask to get it right.
> >
> > I plan to add more VMD startup output to emit the OptiX device numbering
> >
> > much like the CUDA code does, to make this process simpler.
> >
> >
> >
> > Cheers,
> >
> > John Stone
> >
> > [1]vmd_at_ks.uiuc.edu
> >
> >
> >
> > On Fri, Sep 26, 2014 at 10:40:07AM -0400, Stober, Spencer T wrote:
> >
> > > Hello,
> >
> > >
> >
> > >
> >
> > >
> >
> > > I am having issues using the new CUDA accelerated Tachyon renderer.
> > Any
> >
> > > help is much appreciated.
> >
> > >
> >
> > >
> >
> > >
> >
> > > I receive the following error when I attempt to render a scene using
> > OptiX
> >
> > > Tachyon:
> >
> > >
> >
> > >
> >
> > >
> >
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > >
> >
> > > Info) Rendering current scene to 'vmdscene.ppm' ...
> >
> > >
> >
> > > ERROR) OptiX error: Invalid value (Details: Function "RTresult
> >
> > > _rtContextSetDevices(RTcontext, unsigned int, const int*)" caught
> >
> > > exception: The list of devices is incompatible., [1575404])
> >
> > > (OptiXRenderer.C:505
> >
> > >
> >
> > > OptiX: An error occured validating the context. Rendering is aborted.
> >
> > >
> >
> > > Total OptiX rendering time: 0.2 sec
> >
> > >
> >
> > > Info) Executing post-render cmd 'display vmdscene.ppm' ...
> >
> > >
> >
> > > display: Improper image header `vmdscene.ppm' @
> > pnm.c/ReadPNMImage/297.
> >
> > >
> >
> > > Info) Rendering complete.
> >
> > >
> >
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > >
> >
> > >
> >
> > >
> >
> > > I believe my CUDA devices are recognized and configured properly
> > (they
> >
> > > work for NAMD), and this is the startup message I get when I start
> > VMD:
> >
> > >
> >
> > >
> >
> > >
> >
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > >
> >
> > > Info) VMD for LINUXAMD64, version 1.9.2beta1 (September 12, 2014)
> >
> > >
> >
> > > Info) [2]http://www.ks.uiuc.edu/Research/vmd/
> >
> > >
> >
> > > Info) Email questions and bug reports to [3]vmd_at_ks.uiuc.edu
> >
> > >
> >
> > > Info) Please include this reference in published work using VMD:
> >
> > >
> >
> > > Info) Humphrey, W., Dalke, A. and Schulten, K., `VMD - Visual
> >
> > >
> >
> > > Info) Molecular Dynamics', J. Molec. Graphics 1996, 14.1, 33-38.
> >
> > >
> >
> > > Info) -------------------------------------------------------------
> >
> > >
> >
> > > Info) Multithreading available, 16 CPUs detected.
> >
> > >
> >
> > > Info) Free system memory: 254232MB (98%)
> >
> > >
> >
> > > Info) Creating CUDA device pool and initializing hardware...
> >
> > >
> >
> > > Info) Ignoring CUDA-capable GPUs used for display
> >
> > >
> >
> > > Info) Detected 4 available CUDA accelerators:
> >
> > >
> >
> > > Info) [0] Tesla K20Xm 14 SM_3.5 @ 0.73 GHz, 5.6GB RAM, AE2,
> > ZCP
> >
> > >
> >
> > > Info) [2] Tesla K20Xm 14 SM_3.5 @ 0.73 GHz, 5.6GB RAM, AE2,
> > ZCP
> >
> > >
> >
> > > Info) [3] Tesla K20Xm 14 SM_3.5 @ 0.73 GHz, 5.6GB RAM, AE2,
> > ZCP
> >
> > >
> >
> > > Info) [4] Tesla K20Xm 14 SM_3.5 @ 0.73 GHz, 5.6GB RAM, AE2,
> > ZCP
> >
> > >
> >
> > > Info) Detected 5 available TachyonL/OptiX ray tracing accelerators
> >
> > >
> >
> > > Warning) Detected X11 'Composite' extension: if incorrect display
> > occurs
> >
> > >
> >
> > > Warning) try disabling this X server option. Most OpenGL drivers
> >
> > >
> >
> > > Warning) disable stereoscopic display when 'Composite' is enabled.
> >
> > >
> >
> > > Info) OpenGL renderer: Quadro 2000D/PCIe/SSE2
> >
> > >
> >
> > > Info) Features: STENCIL MSAA(4) MDE CVA MTX NPOT PP PS GLSL(OVFGS)
> >
> > >
> >
> > > Info) Full GLSL rendering mode is available.
> >
> > >
> >
> > > Info) Textures: 2-D (16384x16384), 3-D (2048x2048x2048),
> > Multitexture
> >
> > > (4)
> >
> > >
> >
> > > Info) Dynamically loaded 2 plugins in directory:
> >
> > >
> >
> > > Info) /data/lib/vmd/plugins/LINUXAMD64/molfile
> >
> > >
> >
> > > after#0
> >
> > >
> >
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > Spencer T. Stober, Ph.D.
> >
> > >
> >
> > > ExxonMobil Research and Engineering
> >
> > >
> >
> > > 600 Billingsport Rd
> >
> > >
> >
> > > Paulsboro, NJ 08066
> >
> > >
> >
> > > Phone: 856-224-2638
> >
> > >
> >
> > > Email: [1][4]spencer.t.stober_at_exxonmobil.com
> >
> > >
> >
> > >
> >
> > >
> >
> > > References
> >
> > >
> >
> > > Visible links
> >
> > > 1. [5]mailto:spencer.t.stober_at_exxonmobil.com
> >
> >
> >
> > --
> >
> > NIH Center for Macromolecular Modeling and Bioinformatics
> >
> > Beckman Institute for Advanced Science and Technology
> >
> > University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
> >
> > [6]http://www.ks.uiuc.edu/~johns/ Phone: 217-244-3349
> >
> > [7]http://www.ks.uiuc.edu/Research/vmd/
> >
> > References
> >
> > Visible links
> > 1. mailto:vmd_at_ks.uiuc.edu
> > 2. http://www.ks.uiuc.edu/Research/vmd/
> > 3. mailto:vmd_at_ks.uiuc.edu
> > 4. mailto:spencer.t.stober_at_exxonmobil.com
> > 5. mailto:spencer.t.stober_at_exxonmobil.com
> > 6. http://www.ks.uiuc.edu/~johns/
> > 7. http://www.ks.uiuc.edu/Research/vmd/
>
> --
> NIH Center for Macromolecular Modeling and Bioinformatics
> Beckman Institute for Advanced Science and Technology
> University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
> http://www.ks.uiuc.edu/~johns/ Phone: 217-244-3349
> http://www.ks.uiuc.edu/Research/vmd/

-- 
NIH Center for Macromolecular Modeling and Bioinformatics
Beckman Institute for Advanced Science and Technology
University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
http://www.ks.uiuc.edu/~johns/           Phone: 217-244-3349
http://www.ks.uiuc.edu/Research/vmd/