In the VMD installation directory ($VMDDIR/scripts/vmd/) there is a perl script called chemical2vmd which will create a VMD command file and execute VMD. Since VMD does not block the calling process, Netscape and other web browsers cannot directly call VMD, as they do not know when to delete the temporary file containing the molecule or other data. The chemical2vmd script starts VMD with the -e command line option which runs the saved VMD script or molecule file.
It is also possible to install the previous script in the global .mailcap file to make it accessible to everyone. You will have to consult the documentation for your web browser(s) to find out how.