Execution of Programs

Notice that “.” (dot) representing the current working directory is not added to your default search path (PATH). In order to run executables located in the current working directory add “./” in front of the executable name

$ ./myprog

or alternatively specify the absolute path to the executable.

OpenMP Applications

Before running multi-threaded code set the number of threads using the  OMP_NUM_THREADS environment variable. E.g.

$ export OMP_NUM_THREADS=4  
$ ./myprog

See below for a sample Openmp Job script for running batch jobs.

MPI Applications
Use the mpirun command to start MPI programs. E.g. running 8 MPI instances of  myprog

$ mpirun -np 8 ./myprog

For a complete specification of the option list, see the   the mpirun page.

Hybrid MPI/OpenMP Applications

Intel MPI  librarry provides support for mixed MPI/OpenMP applications. To make sure your hybrid code runs correctly, follow these steps:

Use the thread safe version of the Intel MPI Library by using the -mt_mpi compiler driver option.

Set the I_MPI_PIN_DOMAIN environment variable to select the desired process pinning scheme. The recommended setting is omp.

$ export I_MPI_PIN_DOMAIN=omp

A hybrid MPI/OpenMP job to be placed on unique CPUs. For example, running a 2-process MPI job with 3 threads per process

$ export OMP_NUM_THREADS=3 
$ mpirun -np 2 ./myprog

The threads would be placed as follows

rank 0 thread 0 on CPU 0 
rank 0 thread 1 on CPU 1 
rank 0 thread 2 on CPU 2 
rank 1 thread 0 on CPU 3 
rank 1 thread 1 on CPU 4 
rank 1 thread 2 on CPU 5

For a full list of the MPI environment settings, see the ‘ENVIRONMENT VARIABLES’ available by running in the mpivars  command.

SLURM (Simple Linux Utility for Resource Management)

Job Scheduling Policy

The job scheduling priority works on a complex algorithm which includes the time your job is in queue, size of your job and your previous history of job submission. Backfilling is enabled to maximize resource utilization, i.e. use available resources as long as the estimated start times of earlier jobs in the list is not pushed into the future. The scheduler will traverse the list of jobs in order to priority. A user can use a maximum of 320 cores simultaneously at any given time.

SLURM Commands

For example, submit a job using a job script called myjob.slurm:

$ sbatch myjob.slurm 
Submitted batch job 1234567

Check the status of my job:

$ squeue -j 1234567

Job Submission Options

Below is a list of some of the options for the sbatch  command. For a complete list of all options see the ‘sbatch –help ‘ .

More Examples

A  chunk is the smallest set of resources that will be allocated to a job. Since jobs are accounted for entire nodes, i.e. 16 (physical cores) times the number of nodes, the chunk size should be equal to one node. This means that  multiple jobs should not be run on the same node. Requesting resources at node-level is done using several options mentioned above.

E.g. requesting 10 nodes with 10 mpi processes per node (i.e total number of tasks  = Requested Node Count  X Requested CPUs per node):

#SBATCH -n 100 -N 10

SLURM support OpenMP applications by setting the  OMP_NUM_THREADS variable automatically based on the resource request of a job. If  ompthreads is not used,  OMP_NUM_THREADS is set to the value of  ncpus. E.g. running 60 threads on one node:

#SBATCH -n 60 -N 1 
export OMP_NUM_THREADS=60 
./myprogram

 The  walltime resource is specified at job-wide level, e.g. asking for 24 hours:

#SBATCH -t 24:00:00

 All jobs on the system will be run in the  batch queue. This is the default queue.

Job Arrays

SLURM allows you to submit a number of “near identical” jobs simultaneously in the form of a job array. To take advantage of this, you will need a set of jobs that differ only by an “index” of some kind.

For example, say that you would like to run tophat, a splice-aware transcript-to-genome mapping tool, on 30 separate transcript files named trans1.fq, trans2.fq, trans3.fq, etc. First, construct a SLURM batch script, called tophat.sh, using special SLURM job array variables:

#!/bin/bash  
#SBATCH -J tophat # A single job name for the array 
#SBATCH -n 1 # Number of cores 
#SBATCH -N 1 # All cores on one machine 
#SBATCH -p serial_requeue # Partition 
#SBATCH --mem 4000 # Memory request 
#SBATCH -t 0-2:00 # 2 hours (D-HH:MM) 
#SBATCH -o tophat%A%a.out # Standard output 
#SBATCH -e tophat%A%a.err # Standard error    

module load tophat 
tophat $PATH_BOWTIE_INDEX_Folder/BowtieIndex trans"${SLURM_ARRAY_TASK_ID}".fq

Then launch the batch process using the –array option to specify the indexes.

$ sbatch --array=1-30 tophat.sh

In the script, two types of substitution variables are available when running job arrays. The first, %A and %a, represent the job id and the job array index, respectively. These can be used in the sbatch parameters to generate unique names. The second, SLURM_ARRAY_TASK_ID, is a bash environment variable that contains the current array index and can be used in the script itself.

In this example, 30 jobs will be submitted each with a different input file and different standard error and standard out files. More detail can be found on the  SLURM job array documentation page.

Interactive Batch Job

To run an interacive batch job  with its own pseudo terminal, add the –pty  /bin/bash -l option to srun. When the job is scheduled, the standard input, output and error are sent to the terminal session in which srun is running.

$ srun -n 20 -t 2:00:00 --pty /bin/bash -l

In order to enable X11 forwarding, add the –x11=first option to srun.

$ srun -n 20 -t 2:00:00 --x11=first --pty /bin/bash -l

Further Information

See the  SLURM job array documentation page.

Sample Batch Scripts

MPI Job

Sample job running  MPI processes

#!/bin/bash 
#SBATCH -n 80 # Number of cores 
#SBATCH -N 4 # Ensure that all cores are on one machine 
#SBATCH -t 0-00:05 # Runtime in D-HH:MM 
#SBATCH --mem=100 # Memory pool for all cores (see also --mem-per-cpu) 
#SBATCH -o hostname.out # File to which STDOUT will be written 
#SBATCH -e hostname.err # File to which STDERR will be written 
#SBATCH --mail-type=END # Type of email notification- BEGIN,END,FAIL,ALL 
#SBATCH --mail-user=ajk@123.com # Email to which notifications will be sent   
module load intelmpi 
mpirun ./myprog

Large Memory OpenMP Job

Running 40 threads on one large memory  node

#!/bin/bash 
#SBATCH -J my_openmp_job 
#SBATCH -o my_openmp_job.o%j 
#SBATCH -t 24:00:00 #SBATCH -n 40 -N 1   

module load intel   
export OMP_NUM_THREADS=$SLURM_NTASKS   
./my_large_mem_App

Hybrid MPI/OpenMP Job

Sample job running 8 MPI processes, 4 on each node, with 10  OpenMP threads per MPI process

#!/bin/bash 
#SBATCH -J my_openmp_job  
#SBATCH -o my_openmp_job.o%j  
#SBATCH -t 24:00:00  #SBATCH -n 8 -N 2 -c 10  

module load intelmpi  
export I_MPI_PIN_DOMAIN=omp 
export OMP_NUM_THREAD=$SLURM_CPUS_PER_TASK  
mpirun ./myprog

Application Specific Sample Job Scripts

GROMACS Job

Gromacs is a parallel molecular dynamics code designed for high-performance simulation of biomolecular systems. Below is a sample slurm script to run Gromacs on 2 nodes and 20 cpu cores per node using MPI:

#!/bin/bash
#SBATCH -J gromacs_job
#SBATCH -o gromacs_job.o%j
#SBATCH -t 24:00:00
#SBATCH -n 40 -N 2   

module load gromacs/5.0.4

mpirun  mdrun_mpi -v -deffnm mytpr_file

Accelerating MD simulation with a single GPU

Gromacs jobs can also take advantage of GPUs for acceleration.

Below is a sample slurm script to run Gromacs on 1 node, 1 GPU and 20 cpu cores.

The example uses OpenMP for the 20 cpus:

#!/bin/bash 
#SBATCH -J gromacs_job  
#SBATCH -o gromacs_job.o%j  
#SBATCH -t 24:00:00
#SBATCH -p gpu  
#SBATCH -n 20 -N 1    

module load gromacs/5.0.4-gpu

mdrun_gpu  -ntomp  20  -v  -deffnm  mytpr_file

Using multiple GPUs and nodes

Below is a sample slurm script to run Gromacs on 2 nodes. Each node having 1 GPU
and 20 cpu cores. Example uses MPI+Openmp+GPU. This resolves to 1 MPI rank per
GPU and each MPI rank managing 20 threads (1 per cpu core)

#!/bin/bash 
#SBATCH -J gromacs_job  
#SBATCH -o gromacs_job.o%j  
#SBATCH -t 24:00:00
#SBATCH -p gpu  
#SBATCH -n 40 -N 2    

module load gromacs/5.0.4-gpu

mpirun  mdrun_mpi_gpu  -ntomp  20   -v  -deffnm mytpr_file

NAMD Job

NAMD is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems. See the  NAMD Home page.

To run NAMD on 2 nodes and 20 cpus per node using MPI, where apoa1 is the NAMD configuration file:

#!/bin/bash  
#SBATCH -J namd_job  
#SBATCH -o namd_job.o%j  
#SBATCH -t 24:00:00  
#SBATCH -n 40 -N 2    

module load namd2  
mpirun namd2 apoa1.namd

MATLAB Job

MATLAB® is a high-level language and interactive environment for numerical computation, visualization, and programming. Using a combination of parpool and parfor commands in Matlab, the can run loop iterations in parallel See the Matlab parpool Home page http://www.mathworks.com/help/distcomp/parpool.html.

The following SLURM batch script example shows how to run Matlab on  20 cpus within a single node in processing the M code – compute.m.  The contents of M code is shown further below. Note how parpool  is used to initialize or launch the workers before parfor is called.

#!/bin/bash  
#SBATCH -J matlab_job  
#SBATCH -o matlab_job.o%j  
#SBATCH -t 24:00:00  
#SBATCH -n 20 -N 1    

module load matlab  
matlab   < compute.m > compute.out

#compute.m contains 
$ cat compute.m 
%Example, perform three large eigenvalue computations using three 
% workers or cores with Parallel Computing Toolbox software:: 
mypool=parpool('local',str2num(getenv('SLURM_NPROCS'))) 
c=[] 
tic; parfor i=1:20; c(:,i)=eig(rand(1000)); end; toc   
tic; for i=1:20; c(:,i) = eig(rand(1000)); end; toc   
delete(mypool)

To use Matlab in GUI mode, launch an interactive job (X11 forwarding enabled) and then use the matlab-gui command to open the Matlab window.

Matlab Multithreading Support

Most Matlab functions are shipped with multithreading support. Unfortunately maximum number of allowed threads is set to 1 by default, so we need to enable  it with the maxNumCompThreads(Nthreads) function. Note it is not recommended to use multithreading and parallel workers in same code block, either turnoff multithreading to use parfor, or just use the serial version of for.  Below is an example of turning on multithreading support for processing your M code within the SLURM batch script. Here the maximum number of threads is provided by the SLURM environmental variable SLURM_NPROCS.

#!/bin/bash  
#SBATCH -J matlab_job  
#SBATCH -o matlab_job.o%j  
#SBATCH -t 24:00:00  
#SBATCH -n 20 -N 1    

module load matlab  
matlab  -r  "maxNumCompThreads(${SLURM_NPROCS})" < compute_multithreaded.m