From: John Stone (johns_at_ks.uiuc.edu)
Date: Fri Apr 16 2021 - 13:01:03 CDT

Jason,
  The failure messages from the NVIDIA Thrust library are almost
certainly the result of an out-of-memory condition happening on
the GPU during the QuickSurf calculation. I've done a lot of work
on VMD to try and protect it from GPU memory shortage problems, but
when there are other apps running, particularly GPU-memory-hungry
apps like Google's chrome web browser (just one such example)
these issue can still crop up, and even on GPUs that have much
more memory than yours does.

QuickSurf allocates all of the memory it needs up-front to ensure that it
can't fail later on, but there's one "gap in the armor" and that is because
the Thrust library does its own memory allocations. There's not a great
way for VMD to hold memory in advance, and hand it to Thrust later.
I've been looking at replacing my use of Thrust with CUB, which does
allow me to pre-allocate arrays in advance, and implements many of the
same low-level sorting and prefix sum algorithms VMD needs, so going
forward this may improve.

Given that QuickSurf already ran out of memory prior to the
launch of hte OptiX renderer, I think that "the damage was already done".
To get the OptiX renderer to complete without errors it may be necessary
to close other apps that are competing with VMD for GPU memory, and/or
to force QuickSurf to run on the CPU instead.

Let me know more about what you typically have running on your machine
when you've got VMD up, and we can come up with a proposed workaround.
I can suggest a few things for you to try that may get you past these
difficulties.

Best regards,
  John Stone

On Thu, Apr 15, 2021 at 05:51:55PM +0000, Jason Smith wrote:
> John,
>
> Please see below for the VMD output. The crash only seems to happen with
> one of my custom materials on a quicksurf (Other drawing methods not
> tested). I tested materials that use either a transparency or mirror, as
> well as a material that uses both. It only seems to crash for the material
> that uses both. I can provide the pdb and vis state files privately if you
> like.
>
> Driver version 461.09
>
> If you are able to estimate that perhaps it does require more than the 2
> GB of dedicated memory, I can test sometime in the next week on a 4 GB 970
> desktop. The 960M system does currently have relatively low avail system
> memory, but I can't free it up just yet.
>
> Apologies to Nikhil for taking over your thread a little.
>
> ===================================================================
>
> Info) VMD for WIN64, version 1.9.4a51 (December 21, 2020)
> Info) http://www.ks.uiuc.edu/Research/vmd/
> Info) Email questions and bug reports to vmd_at_ks.uiuc.edu
> Info) Please include this reference in published work using VMD:
> Info) Humphrey, W., Dalke, A. and Schulten, K., `VMD - Visual
> Info) Molecular Dynamics', J. Molec. Graphics 1996, 14.1, 33-38.
> Info) -------------------------------------------------------------
> Info) Multithreading available, 8 CPUs detected.
> Info) CPU features: SSE2 SSE4.1 AVX AVX2 FMA F16 HT
> Info) Free system memory: 6349MB (38%)
> Info) Creating CUDA device pool and initializing hardware...
> CUDA error: invalid device symbol,
> C:/Users/johns/Desktop/build/vmd/src/CUDAClearDevice.cu line 54
> Info) Unable to load NVML library, GPU-CPU affinity unavailable.
> Info) Detected 1 available CUDA accelerator::
> Info) [0] GeForce GTX 960M 5 SM_5.0 1.2 GHz, 2.0GB RAM SP32 KT AE4 ZC
> Info) OpenGL renderer: GeForce GTX 960M/PCIe/SSE2
> Info) Features: STENCIL MDE CVA MTX NPOT PP PS GLSL(OVFGS)
> Info) Full GLSL rendering mode is available.
> Info) Textures: 2-D (16384x16384), 3-D (2048x2048x2048), Multitexture
> (4)
> Info) No joysticks found. Joystick interface disabled.
> Info) Detected 1 available TachyonL/OptiX ray tracing accelerator
> Info) Compiling OptiX shaders on 1 target GPU...
> Info) Dynamically loaded 76 plugins in directory:
> Info) C:/Program Files/VMD/plugins/WIN64/molfile
> vmd > Info) Using plugin pdb for structure file C:/Users/Jason
> Smith/OneDrive/COVID-19/Notes on Communication (papers in production)/5RFA
> 1187.pdb
> Info) Using plugin pdb for coordinates from file C:/Users/Jason
> Smith/OneDrive/COVID-19/Notes on Communication (papers in production)/5RFA
> 1187.pdb
> Info) Determining bond structure from distance search ...
> Info) Eliminating bonds duplicated from existing structure...
> Info) Finished with coordinate file C:/Users/Jason
> Smith/OneDrive/COVID-19/Notes on Communication (papers in production)/5RFA
> 1187.pdb.
> Info) Analyzing structure ...
> Info) Atoms: 5388
> Info) Bonds: 4807
> Info) Angles: 0 Dihedrals: 0 Impropers: 0 Cross-terms: 0
> Info) Bondtypes: 0 Angletypes: 0 Dihedraltypes: 0 Impropertypes: 0
> Info) Residues: 1294
> Info) Waters: 683
> Info) Segments: 1
> Info) Fragments: 694 Protein: 2 Nucleic: 0
> 0
> Info) In any publication of scientific results based in part or
> Info) completely on the use of the program STRIDE, please reference:
> Info) Frishman,D & Argos,P. (1995) Knowledge-based secondary structure
> Info) assignment. Proteins: structure, function and genetics, 23,
> 566-579.
> CUDA Thrust sort_by_key() failed:
> C:/Users/johns/Desktop/build/vmd/src/CUDASort.cu line 190
> dev_radix_sort_by_key() failed:
> C:/Users/johns/Desktop/build/vmd/src/CUDASpatialSearch.cu line 259
> CUDA Thrust sort_by_key() failed:
> C:/Users/johns/Desktop/build/vmd/src/CUDASort.cu line 190
> dev_radix_sort_by_key() failed:
> C:/Users/johns/Desktop/build/vmd/src/CUDASpatialSearch.cu line 259
> {{1 0 0 3.883} {0 1 0 2.328} {0 0 1 -9.474} {0 0 0 1}} {{0.465909
> 0.0444488 0.88373 0} {-0.848048 0.307467 0.431632 0} {-0.252536 -0.950541
> 0.18095 0} {0 0 0 1}} {{0.0692238 0 0 0} {0 0.0692238 0 0} {0 0 0.0692238
> 0} {0 0 0 1}} {{1 0 0 0.017413} {0 1 0 -0.049682} {0 0 1 0.485529} {0 0 0
> 1}}
> 0
> 0
> CUDA Thrust sort_by_key() failed:
> C:/Users/johns/Desktop/build/vmd/src/CUDASort.cu line 190
> dev_radix_sort_by_key() failed:
> C:/Users/johns/Desktop/build/vmd/src/CUDASpatialSearch.cu line 259
> Info) Rendering current scene to 'vmdscene.ppm' ...
> OptiXDisplayDevice) Total rendering time: 0.72 sec
> Info) Executing post-render cmd 'explorer vmdscene.ppm' ...
> Info) Rendering complete.
> Info) Rendering current scene to 'vmdscene.ppm' ...
> OptiXDisplayDevice) Total rendering time: 0.34 sec
> Info) Executing post-render cmd 'explorer vmdscene.ppm' ...
> Info) Rendering complete.
> Info) Rendering current scene to 'vmdscene.ppm' ...
> ERROR) OptiXRenderer) ERROR: Unknown error (Details: Function
> "_rtContextLaunch2D" caught exception: Encountered a CUDA error:
> cudaDriver().CuEventSynchronize( m_event ) returned (702): Launch timeout,
> file: <internal>, line: 0)
> (c:\users\johns\desktop\build\vmd\src\optixrenderer.c:4969
> ERROR) OptiXRenderer) ERROR: Unknown error (Details: Function
> "_rtContextLaunch2D" caught exception: Assertion failed: "!m_launching :
> Memory manager launch is already active", file: <internal>, line: 1103)
> (c:\users\johns\desktop\build\vmd\src\optixrenderer.c:4975
> OptiXRenderer) Error during rendering. Rendering aborted.
> ERROR) OptiXRenderer) ERROR: Unknown error (Details: Function
> "_rtAccelerationDestroy" caught exception: Assertion failed:
> "!m_launching", file: <internal>, line: 211)
> (c:\users\johns\desktop\build\vmd\src\optixrenderer.c:1426
> ERROR) OptiXRenderer) ERROR: Unknown error (Details: Function
> "_rtAccelerationDestroy" caught exception: Assertion failed:
> "!m_launching", file: <internal>, line: 211)
> (c:\users\johns\desktop\build\vmd\src\optixrenderer.c:1429
> ERROR) OptiXRenderer) ERROR: Unknown error (Details: Function
> "_rtAccelerationDestroy" caught exception: Assertion failed:
> "!m_launching", file: <internal>, line: 211)
> (c:\users\johns\desktop\build\vmd\src\optixrenderer.c:1432
> ERROR) OptiXRenderer) ERROR: Unknown error (Details: Function
> "_rtGroupDestroy" caught exception: Assertion failed: "!m_launching",
> file: <internal>, line: 211)
> (c:\users\johns\desktop\build\vmd\src\optixrenderer.c:1433
>
> Cheers,
>
> -
> Jason Smith
> Post-Doctoral Fellow
> Robert N. Young Lab, Simon Fraser University
> 8888 University Drive, Burnaby, BC, V5A-1S6
> Canada
>
> --------------------------------------------------------------------------
>
> From: John Stone <johns_at_ks.uiuc.edu>
> Sent: Thursday, 15 April 2021 9:38:02 AM
> To: Jason Smith
> Cc: Geist, Norman; Nikhil Maroli; Vmd L
> Subject: Re: vmd-l: Rendering using GPU
>
> Jason,
> There shouldn't be crashes on the older GPUs, even with things like
> mirror reflection being used. It's possible that on a 960M one might
> run short on GPU memory, but I'm still surprised you're getting a crash.
> What driver version are you running on that machine? Does VMD print any
> errors when this crash you've described occurs?
>
> Best,
> John Stone
>
> On Thu, Apr 15, 2021 at 04:26:05PM +0000, Jason Smith wrote:
> > My Apologies,
> >
> > I just tested 1.9.4a51v6 on a 960M (while John was writing his reply)
> and
> > both in-memory and interactive tachyon-optix works fine, so long as
> you
> > aren't attempting to use RT-based effects like mirror (which leads to
> a
> > crash-to-desktop).
> >
> > Cheers,
> >
> > -
> > Jason Smith
> > Post-Doctoral Fellow
> > Robert N. Young Lab, Simon Fraser University
> > 8888 University Drive, Burnaby, BC, V5A-1S6
> > Canada
> >
> >
> --------------------------------------------------------------------------
> >
> > From: Jason Smith
> > Sent: Thursday, 15 April 2021 9:08:29 AM
> > To: John Stone; Geist, Norman
> > Cc: Nikhil Maroli; Vmd L
> > Subject: Re: vmd-l: Rendering using GPU
> >
> >
> > Correct me if I'm wrong, but ray tracing with Optix requires the ray
> > tracing cores present on 2000 and later series NVidia GPUs.
> >
> > Cheers,
> >
> > -
> > Jason Smith
> > Post-Doctoral Fellow
> > Robert N. Young Lab, Simon Fraser University
> > 8888 University Drive, Burnaby, BC, V5A-1S6
> > Canada
> >
> >
> --------------------------------------------------------------------------
> >
> > From: owner-vmd-l_at_ks.uiuc.edu <owner-vmd-l_at_ks.uiuc.edu> on behalf of
> John
> > Stone <johns_at_ks.uiuc.edu>
> > Sent: Thursday, 15 April 2021 8:20:42 AM
> > To: Geist, Norman
> > Cc: Nikhil Maroli; Vmd L
> > Subject: Re: vmd-l: Rendering using GPU
> >
> > Hi,
> > When VMD starts up, it automatically detects NVIDIA GPUs
> > that support CUDA and OptiX (OptiX is the ray tracing development
> > framework I used to write the existing lightegith GPU version of
> Tachyon).
> > If you've got up-to-date NVIDIA drivers, you should see a VMD startup
> > message
> > that looks similar to this:
> >
> > Info) Detected 3 available TachyonL/OptiX ray tracing accelerators
> > Info) Compiling OptiX shaders on 3 target GPUs...
> >
> > If no OptiX-capable GPUs are seen by VMD, then you won't see the
> > various "TachyonL-OptiX" renderers listed in in the File->Render
> menu.
> >
> > Best regards,
> > John Stone
> >
> > On Thu, Apr 15, 2021 at 10:45:48AM +0200, Geist, Norman wrote:
> > > Use renderers with optix in their names.
> > >
> > > Am Donnerstag, den 15-04-2021 um 09:31 schrieb Nikhil Maroli:
> > >
> > > Dear All,
> > > I am having 3 x 1070 with a 2 x 12 core processor. I have
> > installed VMD
> > > for CUDA versions. When I load VMD I can see it uses 700Mb
> from the
> > > card, however rendering always running on the processor.
> > > Can anyone tell me how I use cards for rendering purposes
> > > Nikhil
> > > --
> >
> > --
> > NIH Center for Macromolecular Modeling and Bioinformatics
> > Beckman Institute for Advanced Science and Technology
> > University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
> > [1]http://www.ks.uiuc.edu/~johns/ Phone: 217-244-3349
> > [2]http://www.ks.uiuc.edu/Research/vmd/
> >
> > References
> >
> > Visible links
> > 1. [1]http://www.ks.uiuc.edu/~johns/
> > 2. [2]http://www.ks.uiuc.edu/Research/vmd/
>
> --
> NIH Center for Macromolecular Modeling and Bioinformatics
> Beckman Institute for Advanced Science and Technology
> University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
> [3]http://www.ks.uiuc.edu/~johns/ Phone: 217-244-3349
> [4]http://www.ks.uiuc.edu/Research/vmd/
>
> References
>
> Visible links
> 1. http://www.ks.uiuc.edu/~johns/
> 2. http://www.ks.uiuc.edu/Research/vmd/
> 3. http://www.ks.uiuc.edu/~johns/
> 4. http://www.ks.uiuc.edu/Research/vmd/

-- 
NIH Center for Macromolecular Modeling and Bioinformatics
Beckman Institute for Advanced Science and Technology
University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
http://www.ks.uiuc.edu/~johns/           Phone: 217-244-3349
http://www.ks.uiuc.edu/Research/vmd/