A few workarounds for very large systems on large linux clusters using elan interconnects and intel cc compilers. The job will hang or die with very cryptic core dumps if these steps are not followed. Increase the send/receive buffer from default before running the job.

setenv LIBELAN_ALLOC_SIZE 419430400

Increase the value for margin, for 3 million atoms on 1400 cpus a margin of 10 worked but could go lower.