Using NAMD in the RC Environment


PLEASE NOTE - these instructions are only appropriate when running NAMD on Janus. If you would like to run NAMD on Summit, please email rc-help@colorado.edu for assistance.


NAMD (http://www.ks.uiuc.edu/Research/namd/) is a parallel molecular dynamics code with particular application to large biomolecular systems.  


It’s acceptable to use the UIUC-provided precompiled distributions of NAMD in the CU-Boulder Research Computing environment.  Two of the prebuilt platforms are especially relevant here.  


Single-node jobs

For NAMD calculations that can run on a single node, download the “Linux-x86_64-multicore” platform and unpack it in your /projects directory.  Submitting a batch job through Slurm is then straightforward using a batch script like this:


#!/bin/bash

#SBATCH --job-name systemNr.txt

#SBATCH --qos janus

#SBATCH --nodes 1

#SBATCH --ntasks-per-node 12

#SBATCH --time 00:20:00

#SBATCH --output systemNr.${SLURM_JOBID}.out

cd /lustre/janus_scratch/username/NAMD

/projects/username/NAMD/NAMD_2.11_Linux-x86_64-multicore/namd2 +p$SLURM_NTASKS_PER_NODE +isomalloc_sync systemNr.conf


In the +p parameter, specify the number of CPU cores on the node that you’ve requested.  That would be 12 for Janus, but could be up to 64 for himem.  The SLURM_NTASKS_PER_NODE variable takes care of this for you automatically.  +isomalloc doesn’t affect runtime performance on Janus, but including it prevents a warning message in your output file.  Make sure your output is going to a scratch filesystem for best performance!  Also, do not use srun to execute namd2 because srun will try to start multiple namd2 processes; you just want one namd2 process using its built-in multi-threaded parallelism.


Multi-node jobs

If your NAMD jobs are big enough to benefit from spanning more than one node, download the

“Linux-x86_64-ibverbs-smp” platform and unpack it in your /projects directory.  Running a multi-node job involves a few extra subtleties.  


First, you’ll have to provide NAMD with a list of nodes that the job will run on.  Since you don’t know which nodes Slurm will assign to your job until it starts running, it’s necessary to generate the node list for NAMD from within your batch script.


Next, NAMD expects to reserve one CPU core per node for inter-node communication.  This means that you should tell NAMD to use one fewer than the physically available cores per node.  On Janus, that would be 11 cores.  Consequently, you’ll also tell NAMD that the total number of processes to start across all N nodes assigned to the job is N*11.


Finally, use charmrun to execute namd2; charmrun will take care of distributing processes across all of your nodes.


It’s straightforward to include this additional setup in your batch script.  For example:



#!/bin/bash

#SBATCH --job-name systemNr.txt

#SBATCH --qos janus

#SBATCH --nodes 4

#SBATCH --ntasks-per-node 12

#SBATCH --time 00:10:00

#SBATCH --output systemNr.${SLURM_JOBID}.out


# choose version of NAMD to use

export NAMD_DIR=/projects/username/NAMD/NAMD_2.11_Linux-x86_64-ibverbs-smp

export PATH=$PATH:$NAMD_DIR


cd /lustre/janus_scratch/username/NAMD


# generate NAMD nodelist

for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do

 echo "host $n ++cpus 12" >> nodelist.$SLURM_JOBID

done


# calculate total processes (P) and procs per node (PPN)

PPN=`expr $SLURM_NTASKS_PER_NODE - 1`

P="$(($PPN * $SLURM_NNODES))"


charmrun ${NAMD_DIR}/namd2 ++p $P ++ppn $PPN ++nodelist nodelist.$SLURM_JOBID +setcpuaffinity systemNr.conf


rm nodelist.$SLURM_JOBID


Please be sure to check how well your jobs scale with increasing nodes.  Beyond a certain number of nodes you may see little or no additional improvement in run time.


NAMD on GPU

 

Some NAMD calculations can be offloaded to a GPU coprocessor for improved performance.  CU Research Computing currently offers two fairly small GPU-enabled compute nodes, but will be greatly expanding our GPU capability on the new Summit supercomputer.  Email rc-help@colorado.edu if you are interested in trying NAMD on GPU.