From: John Stone (johns_at_ks.uiuc.edu)
Date: Tue Jul 23 2019 - 23:28:39 CDT

Hi,
  I've reproduced the problem with the NORTX mode of VMD.
I'm personally convinced this is a bug within either
the specific video driver versions where it occurs, as
it has previously worked fine with other driver versions
with no other changes to the code. The error messages are from deep
inside the just-in-time (JIT) compilation phase of the OptiX runtime,
and they arise from code that isn't in the PTX resulting from compilation
of the VMD OptiX shaders, as there are no shuffle instructions in
the VMD code. I expect to meet up with members of the OptiX team
next week at Siggraph, and I may be able to demonstrate the issue
for them while I'm there.

In the mean time, I'm working on another fix for the issues that
still remain in the RTX-enabled rendering path due to what I believe
is recursion in excess of what OptiX permits.
I know about the leak issue, and that should be cured once I've
got the recursion issue solved.

Best,
  John Stone
  vmd_at_ks.uiuc.edu

On Fri, Jul 19, 2019 at 11:11:33AM +0200, Michael Gecht wrote:
> Dear John,
>
> I am also experiencing problems with the current alpha on RTX 2070
> cards (Driver Version: 430.34, CUDA Version: 10.1). Initial
> rendering works, but starts to fail after a few (dozen) frames. With
> the VMDOPTIXNORTX environment variable set before starting VMD, I am
> unable to render even one single frame with TachyonL-OptiX. VMD
> startup log + crash message below.
>
> Best,
> Michael
>
> migecht_at_workstation:frame3802$ export VMDOPTIXNORTX=1
> migecht_at_workstation:frame3802$ vmdar -e viewmaster-session.vmd
> Info) VMD for LINUXAMD64, version 1.9.4a35 (July 10, 2019)
> Info) http://www.ks.uiuc.edu/Research/vmd/
> Info) Email questions and bug reports to vmd_at_ks.uiuc.edu
> Info) Please include this reference in published work using VMD:
> Info) Humphrey, W., Dalke, A. and Schulten, K., `VMD - Visual
> Info) Molecular Dynamics', J. Molec. Graphics 1996, 14.1, 33-38.
> Info) -------------------------------------------------------------
> Info) Multithreading available, 8 CPUs detected.
> Info) CPU features: SSE2 AVX AVX2 FMA AVX512F AVX512CD
> Info) Free system memory: 56GB (89%)
> Info) Creating CUDA device pool and initializing hardware...
> Info) Detected 1 available CUDA accelerator::
> Info) [0] GeForce RTX 2070 36 SM_7.5 1.6 GHz, 7.8GB RAM SP32 KT AE3 ZC
> Warning) Detected X11 'Composite' extension: if incorrect display occurs
> Warning) try disabling this X server option. Most OpenGL drivers
> Warning) disable stereoscopic display when 'Composite' is enabled.
> Info) OpenGL renderer: GeForce RTX 2070/PCIe/SSE2
> Info) Features: STENCIL MSAA(4) MDE CVA MTX NPOT PP PS GLSL(OVFGS)
> Info) Full GLSL rendering mode is available.
> Info) Textures: 2-D (32768x32768), 3-D (16384x16384x16384),
> Multitexture (4)
> Info) Detected 1 available TachyonL/OptiX ray tracing accelerator
> Info) Compiling 1 OptiX shaders on 1 target GPU...
> Info) Dynamically loaded 3 plugins in directory:
> Info) /home/tb/migecht/.lib/vmdar/plugins/LINUXAMD64/molfile
> after#0
> 2.6
> Info) Using plugin pdb for structure file /<redacted>/migecht/DATA/projects/<redacted>/render/frame3802/frame3802.pdb
> Info) Using plugin pdb for coordinates from file /STORAGE/migecht/DATA/projects/<redacted>/render/frame3802/frame3802.pdb
> Info) Determining bond structure from distance search ...
> Info) Finished with coordinate file /<redacted>/migecht/DATA/projects/<redacted>/render/frame3802/frame3802.pdb.
> Info) Analyzing structure ...
> Info) Atoms: 98926
> Info) Bonds: 78052
> Info) Angles: 0 Dihedrals: 0 Impropers: 0 Cross-terms: 0
> Info) Bondtypes: 0 Angletypes: 0 Dihedraltypes: 0 Impropertypes: 0
> Info) Residues: 21066
> Info) Waters: 20502
> Info) Segments: 1
> Info) Fragments: 20916 Protein: 3 Nucleic: 0
> Info) Using plugin xtc for coordinates from file /<redacted>/migecht/DATA/projects/<redacted>/render/frame3802/traj.xtc
> Info) Finished with coordinate file /<redacted>/migecht/DATA/projects/<redacted>/render/frame3802/traj.xtc.
> vmd > Info) In any publication of scientific results based in part or
> Info) completely on the use of the program STRIDE, please reference:
> Info) Frishman,D & Argos,P. (1995) Knowledge-based secondary structure
> Info) assignment. Proteins: structure, function and genetics, 23, 566-579.
> Info) Rendering current scene to 'vmdscene.ppm' ...
> Info) Ambient occlusion enabled.
> Info) Shadow rendering enabled.
> ERROR) OptiXRenderer) ERROR: Unknown error (Details: Function
> "RTresult _rtContextLaunch2D(RTcontext, unsigned int, RTsize,
> RTsize)" caught exception: Encountered a CUDA error: ptxas
> application ptx input, line 59; error : Feature 'activemask'
> requires PTX ISA .version 6.2 or later
> ERROR) ptxas application ptx input, line 62; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 65; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 68; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 71; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 74; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 1209; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 1471; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 1473; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 1475; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 1477; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> ERROR) ptxas application ptx input, line 1479; error : Feature
> 'shfl.sync' requires PTX ISA .version 6.0 or later
> OptiXRenderer) Creating OptiX window: 1762 x 1341...
> OptiXRenderer) VMD TachyonL-OptiX Interactive Ray Tracer help:
> OptiXRenderer) ===============================================
> OptiXRenderer) Using 1 device:
> OptiXRenderer) [0] GeForce RTX 2070 CUDA[0], 7.8GB RAM, KTO
> OptiXRenderer)
> OptiXRenderer) Spaceball/SpaceNavigator/Magellan: Not available
> OptiXRenderer) Stereoscopic display: Not available
> OptiXRenderer) Vert retrace sync: On
> OptiXRenderer)
> OptiXRenderer) General controls:
> OptiXRenderer) space: save numbered snapshot image
> OptiXRenderer) =: reset to initial view
> OptiXRenderer) h: print this help info
> OptiXRenderer) p: print current rendering parameters
> OptiXRenderer) ESC,q: quit viewer
> OptiXRenderer)
> OptiXRenderer) Display controls
> OptiXRenderer) F1: override shadows on/off (off=AO off too)
> OptiXRenderer) F2: override AO on/off
> OptiXRenderer) F3: override DoF on/off
> OptiXRenderer) F4: override Depth cueing on/off
> OptiXRenderer) F12: toggle full-screen display on/off
> OptiXRenderer) 1-9,0: override samples per update auto-FPS off
> OptiXRenderer) Up: increase DoF focal distance
> OptiXRenderer) Down: decrease DoF focal distance
> OptiXRenderer) Left: decrease DoF f/stop
> OptiXRenderer) Right: increase DoF f/stop
> OptiXRenderer) S: toggle stereoscopic display on/off (if avail)
> OptiXRenderer) a: toggle AA/AO auto-FPS tuning on/off (on)
> OptiXRenderer) g: toggle gradient sky xforms on/off (on)
> OptiXRenderer) l: toggle light xforms on/off (on)
> OptiXRenderer)
> OptiXRenderer) Mouse controls:
> OptiXRenderer) f: mouse depth-of-field mode
> OptiXRenderer) r: mouse rotation mode
> OptiXRenderer) s: mouse scaling mode
> OptiXRenderer) t: mouse translation mode
> OptiXRenderer) An error occured in AS generation. Rendering is aborted.
>
> OptiXDisplayDevice) Total rendering time: 0.59 sec
> Info) Executing post-render cmd 'display vmdscene.ppm' ...
> display: improper image header `vmdscene.ppm' @
> error/pnm.c/ReadPNMImage/293.
> Info) Rendering complete.
>
>
> On 17/07/19 00:28, John Stone wrote:
> >Dallas,
> > I'm aware of the issue, for the moment try setting this environment
> >variable before running VMD as a workaround until the next beta.
> >I'm still chasing this bug and another related issue.
> >bash
> > export VMDOPTIXNORTX=1
> >csh:
> > setenv VMDOPTIXNORTX 1
> >
> >Best,
> > John
> >
> >On Wed, Jul 17, 2019 at 08:18:16AM +1000, Dallas Warren wrote:
> >> Note I think it is actually only CUDA 10.1, only place can find 10.2
> >> mentioned is from the nvidia-smi call, and can't find any mention of 10.2
> >> existing online.
> >> Catch ya,
> >>
> >> Dr. Dallas Warren
> >> Drug Delivery, Disposition and Dynamics
> >> Monash Institute of Pharmaceutical Sciences, Monash University
> >> 381 Royal Parade, Parkville VIC 3052
> >> [1]dallas.warren_at_monash.edu
> >> ---------------------------------
> >> When the only tool you own is a hammer, every problem begins to resemble a
> >> nail.
> >> On Mon, 15 Jul 2019 at 11:10, Dallas Warren <[2]dallas.warren_at_monash.edu>
> >> wrote:
> >>
> >> John,
> >> I'm still getting a memory leak with 1.9.4a35 and image rendering
> >> (TachyonL-OptiX) using an RTX2080Ti card, driver 430.26, CUDA 10.2
> >> I obtained the same issue with a31, but does not occur with a8 which is
> >> what I am having to use currently.
> >> Below I have output from nvidia-smi before and after a rendering step
> >> and post failure, vmd output when error occurs, and the start up output
> >> from vmd.
> >> Let me know if there is any further information I can provide to assist.
> >> ######################################
> >> Here are two nvidia-smi calls before and after a rendering using the
> >> movie extension, memory used creeps up with each rendered frame/image.
> >> dallas_at_morph:~> nvidia-smi
> >> Mon Jul 15 10:51:45 2019
> >> +-----------------------------------------------------------------------------+
> >> | NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA
> >> Version: 10.2 |
> >> |-------------------------------+----------------------+----------------------+
> >> | GPU Name Persistence-M| Bus-Id Disp.A |
> >> Volatile Uncorr. ECC |
> >> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
> >> Compute M. |
> >> |===============================+======================+======================|
> >> | 0 GeForce RTX 208... Off | 00000000:02:00.0 On |
> >> N/A |
> >> | 28% 55C P2 266W / 260W | 5525MiB / 11011MiB | 10%
> >> Default |
> >> +-------------------------------+----------------------+----------------------+
> >>
> >>
> >> +-----------------------------------------------------------------------------+
> >> | Processes:
> >> GPU Memory |
> >> | GPU PID Type Process name
> >> Usage |
> >> |=============================================================================|
> >> | 0 2413 G /usr/bin/X
> >> 183MiB |
> >> | 0 2922 G kwin_x11
> >> 29MiB |
> >> | 0 2926 G /usr/bin/krunner
> >> 6MiB |
> >> | 0 2931 G /usr/bin/plasmashell
> >> 73MiB |
> >> | 0 3564 C+G
> >> /usr/local/lib/vmd_1.9.4a35/vmd_LINUXAMD64 5227MiB |
> >> +-----------------------------------------------------------------------------+
> >> dallas_at_morph:~> nvidia-smi
> >> Mon Jul 15 10:51:49 2019
> >> +-----------------------------------------------------------------------------+
> >> | NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA
> >> Version: 10.2 |
> >> |-------------------------------+----------------------+----------------------+
> >> | GPU Name Persistence-M| Bus-Id Disp.A |
> >> Volatile Uncorr. ECC |
> >> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
> >> Compute M. |
> >> |===============================+======================+======================|
> >> | 0 GeForce RTX 208... Off | 00000000:02:00.0 On |
> >> N/A |
> >> | 27% 57C P2 250W / 260W | 5645MiB / 11011MiB | 37%
> >> Default |
> >> +-------------------------------+----------------------+----------------------+
> >>
> >>
> >> +-----------------------------------------------------------------------------+
> >> | Processes:
> >> GPU Memory |
> >> | GPU PID Type Process name
> >> Usage |
> >> |=============================================================================|
> >> | 0 2413 G /usr/bin/X
> >> 183MiB |
> >> | 0 2922 G kwin_x11
> >> 29MiB |
> >> | 0 2926 G /usr/bin/krunner
> >> 6MiB |
> >> | 0 2931 G /usr/bin/plasmashell
> >> 73MiB |
> >> | 0 3564 C+G
> >> /usr/local/lib/vmd_1.9.4a35/vmd_LINUXAMD64 5347MiB |
> >> +-----------------------------------------------------------------------------+
> >> ######################################
> >> Post failure:
> >> dallas_at_morph:~> nvidia-smi
> >> Mon Jul 15 10:56:30 2019
> >> +-----------------------------------------------------------------------------+
> >> | NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA
> >> Version: 10.2 |
> >> |-------------------------------+----------------------+----------------------+
> >> | GPU Name Persistence-M| Bus-Id Disp.A |
> >> Volatile Uncorr. ECC |
> >> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
> >> Compute M. |
> >> |===============================+======================+======================|
> >> | 0 GeForce RTX 208... Off | 00000000:02:00.0 On |
> >> N/A |
> >> | 24% 39C P8 29W / 260W | 10936MiB / 11011MiB |
> >> 2% Default |
> >> +-------------------------------+----------------------+----------------------+
> >>
> >>
> >> +-----------------------------------------------------------------------------+
> >> | Processes:
> >> GPU Memory |
> >> | GPU PID Type Process name
> >> Usage |
> >> |=============================================================================|
> >> | 0 2413 G /usr/bin/X
> >> 214MiB |
> >> | 0 2922 G kwin_x11
> >> 31MiB |
> >> | 0 2926 G /usr/bin/krunner
> >> 6MiB |
> >> | 0 2931 G /usr/bin/plasmashell
> >> 73MiB |
> >> | 0 3564 C+G
> >> /usr/local/lib/vmd_1.9.4a35/vmd_LINUXAMD64 10605MiB |
> >> +-----------------------------------------------------------------------------+
> >> ######################################
> >> vmd output at the point at which it fails:
> >> Info) Rendering current scene to
> >> '/home/dallas/Pictures/vmd/5_md_001_rotation.00067.ppm' ...
> >> Info) Ambient occlusion enabled.
> >> Info) Shadow rendering enabled.
> >> OptiXDisplayDevice) Total rendering time: 2.86 sec
> >> Info) Rendering complete.
> >> Info) Rendering current scene to
> >> '/home/dallas/Pictures/vmd/5_md_001_rotation.00068.ppm' ...
> >> Info) Ambient occlusion enabled.
> >> Info) Shadow rendering enabled.
> >> ERROR) OptiXRenderer) ERROR: Memory allocation failed (Details: Function
> >> "RTresult _rtContextLaunch2D(RTcontext, unsigned int, RTsize, RTsize)"
> >> caught exception: Out of memory) (OptiXRenderer.C:2621
> >> OptiXRenderer) Error during AS generation. Rendering aborted.
> >> OptiXDisplayDevice) Total rendering time: 0.32 sec
> >> Info) Rendering complete.
> >> ######################################
> >> vmd start up output:
> >> dallas_at_morph:~/Pictures/vmd> vmd_1.9.4a35
> >> /usr/local/lib/vmd_1.9.4a35/vmd_LINUXAMD64: /usr/lib64/libGL.so.1: no
> >> version information available (required by
> >> /usr/local/lib/vmd_1.9.4a35/vmd_LINUXAMD64)
> >> Info) VMD for LINUXAMD64, version 1.9.4a35 (July 10, 2019)
> >> Info) [3]http://www.ks.uiuc.edu/Research/vmd/
> >>
> >> Info) Email questions and bug reports to [4]vmd_at_ks.uiuc.edu
> >>
> >> Info) Please include this reference in published work using VMD:
> >> Info) Humphrey, W., Dalke, A. and Schulten, K., `VMD - Visual
> >> Info) Molecular Dynamics', J. Molec. Graphics 1996, 14.1, 33-38.
> >> Info) -------------------------------------------------------------
> >> Info) Multithreading available, 12 CPUs detected.
> >> Info) CPU features: SSE2 AVX
> >> Info) Free system memory: 33GB (93%)
> >> Info) Creating CUDA device pool and initializing hardware...
> >> Info) Detected 1 available CUDA accelerator::
> >> Info) [0] GeForce RTX 2080 Ti 68 SM_7.5 1.6 GHz, 11GB RAM SP32 KT AE3
> >> ZC
> >> Warning) Detected X11 'Composite' extension: if incorrect display occurs
> >> Warning) try disabling this X server option. Most OpenGL drivers
> >> Warning) disable stereoscopic display when 'Composite' is enabled.
> >> Info) OpenGL renderer: GeForce RTX 2080 Ti/PCIe/SSE2
> >> Info) Features: STENCIL MSAA(4) MDE CVA MTX NPOT PP PS GLSL(OVFGS)
> >> Info) Full GLSL rendering mode is available.
> >> Info) Textures: 2-D (32768x32768), 3-D (16384x16384x16384),
> >> Multitexture (4)
> >> Info) Detected 1 available TachyonL/OptiX ray tracing accelerator
> >> Info) Compiling 1 OptiX shaders on 1 target GPU...
> >> Info) Dynamically loaded 3 plugins in directory:
> >> Info) /usr/local/lib/vmd_1.9.4a35/plugins/LINUXAMD64/molfile
> >> after#0
> >> Catch ya,
> >>
> >> Dr. Dallas Warren
> >> Drug Delivery, Disposition and Dynamics
> >> Monash Institute of Pharmaceutical Sciences, Monash University
> >> 381 Royal Parade, Parkville VIC 3052
> >> [5]dallas.warren_at_monash.edu
> >> ---------------------------------
> >> When the only tool you own is a hammer, every problem begins to resemble
> >> a nail.
> >> On Thu, 11 Jul 2019 at 07:29, John Stone <[6]johns_at_ks.uiuc.edu> wrote:
> >>
> >> Hi,
> >> I've just posted two new Linux and MacOS X test binaries
> >> of VMD 1.9.4 alpha 35. The new version includes new structure
> >> building plugins, a new colvars dashboard tool, updates to the
> >> RTX-accelerated ray tracing engines to quash one of the
> >> outstanding bugs that people had run into with previous builds.
> >>
> >> I expect that we will begin a more rapid update cycle soon because
> >> we have also made some massive updates to VMD's Python support with
> >> contributions from Robin Betz and others, and I'm looking for a
> >> good way to redistribute pre-built VMD binaries that are linked
> >> against the latest Python 3.x versions.
> >>
> >> We will have alpha builds for other platforms (various Windows and
> >> 64-bit MacOS X) starting to become available as bugs get reported
> >> and quashed in the new plugins.
> >>
> >> Some of the contributors to the new tools in this beta will
> >> followup on this email with specifics of the new features/tools
> >> they've added for those eager to try them out.
> >>
> >> Best regards,
> >> John Stone
> >> [7]vmd_at_ks.uiuc.edu
> >>
> >> --
> >> NIH Center for Macromolecular Modeling and Bioinformatics
> >> Beckman Institute for Advanced Science and Technology
> >> University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
> >> [8]http://www.ks.uiuc.edu/~johns/ Phone: 217-244-3349
> >> [9]http://www.ks.uiuc.edu/Research/vmd/
> >>
> >>References
> >>
> >> Visible links
> >> 1. mailto:dallas.warren_at_monash.edu
> >> 2. mailto:dallas.warren_at_monash.edu
> >> 3. http://www.ks.uiuc.edu/Research/vmd/
> >> 4. mailto:vmd_at_ks.uiuc.edu
> >> 5. mailto:dallas.warren_at_monash.edu
> >> 6. mailto:johns_at_ks.uiuc.edu
> >> 7. mailto:vmd_at_ks.uiuc.edu
> >> 8. http://www.ks.uiuc.edu/~johns/
> >> 9. http://www.ks.uiuc.edu/Research/vmd/

-- 
NIH Center for Macromolecular Modeling and Bioinformatics
Beckman Institute for Advanced Science and Technology
University of Illinois, 405 N. Mathews Ave, Urbana, IL 61801
http://www.ks.uiuc.edu/~johns/           Phone: 217-244-3349
http://www.ks.uiuc.edu/Research/vmd/