SOLVED: Failure to run namd-cuda with gtx-470

From: Francesco Pietra (chiendarret_at_gmail.com)
Date: Fri Jul 22 2011 - 09:56:34 CDT

With the new nvidia-kernel-dkms 275.09.07-5 and nvidia tools in Debian
GNU-Linux amd64
wheezy, all described problems with NAMD and the CUDA-accelerated viewer VMD
vanished. Thus, my suspicion that problems (absent with driver
270.41.19-1) were introduced by nvidia-kernel-dkms 275.09.07-1 was correct.

Surprisingly, now NAMD - on the same system of 250,000 atoms - is
running notably faster (just from observing the monitor, no benchmark
carried out) than with previous 270.41.19-1 driver.
Now NAMD is launched, without calling the X server, after

# nvidia-smi -L

# nvidia-smi -pm 1

fp

---------- Forwarded message ----------
From: Francesco Pietra <chiendarret_at_gmail.com>
Date: Wed, Jul 6, 2011 at 5:44 PM
Subject: namd-l: Failure to run namd-cuda with gtx-470
To: NAMD <namd-l_at_ks.uiuc.edu>

While I could run namd-cuda md simulations without problems after
running (as root)

nvidia-smi -L

the situation has gradually worsened. After a two-days successl
simulation, this afternoon a similar simulation does not start at all
and the computer does no more sense the keyboard, as if it were
hanged. Actually, looking at /var/log/messages of the gig64 machine
from a ssh-linked desktop, kernel problems are indicated, as reported
in part below.

******************
Jul  6 17:26:42 gig64 kernel: [  230.179942] NVRM: GPU at 0000:01:00.0
has fallen off the bus.
Jul  6 17:28:04 gig64 kernel: [  312.424094] Modules linked in:
powernow_k8 mperf cpufreq_conservative cpufreq_powersave cpufreq_stats
cpufreq_userspace fuse nfsd exportfs nfs lockd fscache nfs_acl
auth_rpcgss sunrpc ext2 it87 hwmon_vid loop firewire_sbp2
snd_hda_codec_hdmi nvidia(P) snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_pcm snd_seq snd_timer snd_seq_device evdev
pcspkr k10temp snd i2c_piix4 soundcore edac_core edac_mce_amd i2c_core
parport_pc snd_page_alloc parport wmi button processor thermal_sys
ext3 jbd mbcache dm_mod raid1 md_mod usbhid hid sg sr_mod sd_mod cdrom
crc_t10dif ata_generic ohci_hcd xhci_hcd pata_atiixp pata_jmicron ahci
libahci libata ehci_hcd firewire_ohci usbcore firewire_core scsi_mod
crc_itu_t floppy nls_base r8169 mii [last unloaded: scsi_wait_scan]
Jul  6 17:28:04 gig64 kernel: [  312.424204] CPU 4
Jul  6 17:28:04 gig64 kernel: [  312.424208] Modules linked in:
powernow_k8 mperf cpufreq_conservative cpufreq_powersave cpufreq_stats
cpufreq_userspace fuse nfsd exportfs nfs lockd fscache nfs_acl
auth_rpcgss sunrpc ext2 it87 hwmon_vid loop firewire_sbp2
snd_hda_codec_hdmi nvidia(P) snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_pcm snd_seq snd_timer snd_seq_device evdev
pcspkr k10temp snd i2c_piix4 soundcore edac_core edac_mce_amd i2c_core
parport_pc snd_page_alloc parport wmi button processor thermal_sys
ext3 jbd mbcache dm_mod raid1 md_mod usbhid hid sg sr_mod sd_mod cdrom
crc_t10dif ata_generic ohci_hcd xhci_hcd pata_atiixp pata_jmicron ahci
libahci libata ehci_hcd firewire_ohci usbcore firewire_core scsi_mod
crc_itu_t floppy nls_base r8169 mii [last unloaded: scsi_wait_scan]
Jul  6 17:28:04 gig64 kernel: [  312.424302]
Jul  6 17:28:04 gig64 kernel: [  312.424309] Pid: 2916, comm: namd2
Tainted: P           O 2.6.38-2-amd64 #1 Gigabyte Technology Co., Ltd.
GA-890FXA-UD5/GA-890FXA-UD5
Jul  6 17:28:04 gig64 kernel: [  312.424322] RIP:
0010:[<ffffffffa07d4f74>]  [<ffffffffa07d4f74>]
_nv015265rm+0x252/0x260 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.424951] RSP:
0018:ffff88042075fc88  EFLAGS: 00000297
Jul  6 17:28:04 gig64 kernel: [  312.424958] RAX: 00000000ffffffff
RBX: ffff88042b522000 RCX: 0000000000000019
Jul  6 17:28:04 gig64 kernel: [  312.424964] RDX: 00000000ffffffff
RSI: 0000000000005499 RDI: ffff88042b522028
Jul  6 17:28:04 gig64 kernel: [  312.424971] RBP: ffff8804217f2c88
R08: ffff8804217f2c98 R09: 0000000000000000
Jul  6 17:28:04 gig64 kernel: [  312.424977] R10: 0000000000000246
R11: 0000000000000028 R12: ffffffff8100a30e
Jul  6 17:28:04 gig64 kernel: [  312.424983] R13: 0000000000000000
R14: 0000000000000246 R15: 0000000000000028
Jul  6 17:28:04 gig64 kernel: [  312.424991] FS:
00007f2f355a1720(0000) GS:ffff8800bfb00000(0000)
knlGS:0000000000000000
Jul  6 17:28:04 gig64 kernel: [  312.424998] CS:  0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Jul  6 17:28:04 gig64 kernel: [  312.425004] CR2: 00007f2f33947000
CR3: 000000041ef84000 CR4: 00000000000006e0
Jul  6 17:28:04 gig64 kernel: [  312.425010] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Jul  6 17:28:04 gig64 kernel: [  312.425016] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul  6 17:28:04 gig64 kernel: [  312.425024] Process namd2 (pid: 2916,
threadinfo ffff88042075e000, task ffff88041e5cd7c0)
Jul  6 17:28:04 gig64 kernel: [  312.425065]  ffff88042b522000
0000000000000045 0000000000000000 0000000000000003
Jul  6 17:28:04 gig64 kernel: [  312.425077]  0000000000000000
ffffffffa045b0dc ffff88042b522000 ffff88042b057000
Jul  6 17:28:04 gig64 kernel: [  312.425088]  ffff88042fb30000
ffffffffa045b2e1 0000000000000002 0000000000000003
Jul  6 17:28:04 gig64 kernel: [  312.425494]  [<ffffffffa045b0dc>] ?
_nv002890rm+0x8b/0x9c [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.425853]  [<ffffffffa045b2e1>] ?
_nv005068rm+0x1f4/0x20a [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.426368]  [<ffffffffa069a5d7>] ?
_nv010159rm+0x97b/0x9a9 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.426881]  [<ffffffffa069303a>] ?
_nv010153rm+0x314/0x3eb [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.427194]  [<ffffffffa03dc728>] ?
_nv002567rm+0x8c6/0x9a8 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.427511]  [<ffffffffa03ebe35>] ?
_nv002030rm+0xac/0xf0 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.427827]  [<ffffffffa03ebe00>] ?
_nv002030rm+0x77/0xf0 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a20265>] ?
_nv002424rm+0x5b5/0x751 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a1a8b1>] ?
rm_ioctl+0x30/0x10a [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a390e9>] ?
nv_kern_ioctl+0x31a/0x381 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a3918d>] ?
nv_kern_unlocked_ioctl+0x1c/0x20 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff81104b0b>] ?
do_vfs_ioctl+0x467/0x4b4
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff810d60fd>] ?
do_brk+0x2ca/0x326
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff81104ba3>] ?
sys_ioctl+0x4b/0x70
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff81009952>] ?
system_call_fastpath+0x16/0x1b
Jul  6 17:28:04 gig64 kernel: [  312.428001] Call Trace:
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa045b0dc>] ?
_nv002890rm+0x8b/0x9c [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa045b2e1>] ?
_nv005068rm+0x1f4/0x20a [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa069a5d7>] ?
_nv010159rm+0x97b/0x9a9 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa069303a>] ?
_nv010153rm+0x314/0x3eb [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa03dc728>] ?
_nv002567rm+0x8c6/0x9a8 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa03ebe35>] ?
_nv002030rm+0xac/0xf0 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa03ebe00>] ?
_nv002030rm+0x77/0xf0 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a20265>] ?
_nv002424rm+0x5b5/0x751 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a1a8b1>] ?
rm_ioctl+0x30/0x10a [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a390e9>] ?
nv_kern_ioctl+0x31a/0x381 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffffa0a3918d>] ?
nv_kern_unlocked_ioctl+0x1c/0x20 [nvidia]
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff81104b0b>] ?
do_vfs_ioctl+0x467/0x4b4
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff810d60fd>] ?
do_brk+0x2ca/0x326
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff81104ba3>] ?
sys_ioctl+0x4b/0x70
Jul  6 17:28:04 gig64 kernel: [  312.428001]  [<ffffffff81009952>] ?
system_call_fastpath+0x16/0x1b
root_at_gig64:/var/log#
Message from syslogd_at_gig64 at Jul  6 17:29:28 ...
 kernel:[  396.424694] Stack:

Message from syslogd_at_gig64 at Jul  6 17:29:28 ...
 kernel:[  396.424762] Call Trace:

Message from syslogd_at_gig64 at Jul  6 17:29:28 ...
 kernel:[  396.428001] Code: be a0 e8 85 65 60 00 b8 00 00 00 00 e8 9d
65 60 00 85 c0 74 10 e8 84 28 63 00 0f 1f 00 eb 06 89 77 68 89 4f 6c
48 83 c5 10 5b c3 <41> 54 53 48 83 ec 08 48 83 ed 08 41 89 f4 39 77 68
73 17 39 77
******************

such messages from syslog_at_gig64 continue at slow pace.

Thanks for advice

francesco pietra

This archive was generated by hypermail 2.1.6 : Wed Feb 29 2012 - 15:57:29 CST