Re: Unable to read corrupted binary DCD trajectory file

From: Kenno Vanommeslaeghe (kvanomme_at_rx.umaryland.edu)
Date: Mon Sep 08 2014 - 11:56:01 CDT

On 09/08/2014 11:38 AM, Viswanath Pasumarthi wrote:
> I am using 64-bit Matlab R2013b version, and the file system is NTFS,
> which can handle files as large as 16 TB. I agree it is safe to follow
> your technique, but I am curious to know the reason for the size limit

Another thing to check is which *NAMD* version you were running, and
whether it's a 32-bit build. Since the final file size is exactly (?) 4GB,
it is likely the damage was done when running the simulation.

> and
> more importantly, extract the data from this DCD file for now.

Assuming NAMD simply stopped writing to the disk, catdcd might be worth a
try, with an optional command-line limit on the number of frames to
extract. However, there are indications this assumption may not be valid.
Specifically, it is entirely possible that the beginning of the trajectory
got overwritten with the data that was meant to go beyond 4GB, which would
make it a bit difficult to extract useful information from the file.

Lessons learned (purely in my opinion):
- There are reasons why good practices dictate splitting up long MD runs
into shorter legs (incidentally, a big part of the reason for the
existence of catdcd and the "velocities" keyword in NAMD is to make this
easier to do). There are many ways trajectory files can get lost or
corrupted during the course of a simulation...
- Windows is not an operating system for serious computational science.
For starters, a lot of important softwares in the field are not natively
supported, and one needs to install compatibility layers like Cygwin to
access them as well as to tap the powers of the UNIX shell, which are
almost indispensable in this line of work. And as shown by this incident,
even when you do come across an application that is natively supported
(such as NAMD), it might not be as thoroughly tested under all possible
usage scenarios as its Linux counterpart. Indeed, it is not impossible
that you've hit an issue in the windows binaries, and given the
aforementioned "best practice", it seems entirely possible that nobody
tried this before.

This archive was generated by hypermail 2.1.6 : Wed Dec 31 2014 - 23:22:50 CST