Software compilation

If the software package or version you need is not available as an environment module, you may compile it yourself. The recommended location for user-installed software is the project directory, which is snapshotted and can easily be shared with members of a research group.

It is extremely important that software be built on the same type of node as it is intended to run on. This means that compiling on the login nodes is highly discouraged because the login nodes have a different hardware architecture and operating system packages than any compute nodes. Applications compiled on login nodes are unlikely to perform well and may not even run on compute nodes. Software that was built on Janus needs to be recompiled in order to run on Summit.

Compile nodes

Dedicated nodes are provided for compiling software meant to run on the Summit compute cluster. Access to these nodes is provided from any login node via SSH.

$ ssh scompile

The Summit compile nodes are otherwise identical to Summit general compute nodes and are most likely to produce problem-free builds, especially for applications that use MPI or are heavily hardware-optimized. Applications that are intended to run on GPU nodes must be compiled directly on a GPU node since the standard Summit compile nodes do not include the necessary GPU drivers. Start an interactive session to log into a GPU node for compiling:

 $ sinteractive --partition=sgpu --qos=debug --time=1:00:00 --exclusive

Compilers

RC provides three compiler suites, each of which can be enabled by loading the appropriate environment module. We recommend using the Intel Compiler Suite when compiling applications to be run on Research Computing resources, e.g.:

$ module load intel/17.0.0
Compiler Suite Module Family Language Command
Intel Compiler Suite intel C icc
Fortran ifort
C++ icpc
The Portland Group pgi C pgcc
Fortran pgfortran
C++ pgcc++
GNU Compiler Collection gcc C gcc
Fortran gfortran
C++ g++
Java gcj

MPI

Two MPI implementations are available as modules for use with Summit: OpenMPI and Intel MPI. We recommend Intel MPI.

All MPI implementations available in the Research Computing Environment provide standard wrapper commands.

Language Wrapper command
C mpicc
Fortran mpifc, mpif77, mpif90
C++ mpicxx

The MPI module you load is dependent on the compiler suite you have loaded. For example if you have the Intel compiler loaded you will only be able to load an MPI implementation built with that compiler. A compiler module must thus be loaded before an MPI module.

Optimization

Please choose appropriate compiler flags to produce an optimized executable. Application performance on modern computer architectures such as Summit often depends critically on compile-time optimization. It's not unusual to see a factor of two speedup when choosing correct optimization options. Keep in mind that that is equivalent to doubling the size of your compute allocation!

The general optimization flag for most compilers is -OX where X is an integer specifying the optimization level. Level 2 is appropriate for most applications and 3 produces additional speedups in many cases but may also reduce numerical accuracy. Try both to see which works better for your program.

Processor-specific optimization can also lead to noticeably improved performance. For the CPUs on Summit nodes, -march=core-avx2 or -xcore-avx2 (Intel compiler) or -mtune=haswell (GNU compiler) turn on the appropriate CPU-specific optimization.

The Intel compilers can provide additional information about how well your executable is vectorized, that is, whether multiple floating point operations per CPU cycle are enabled. Run man icc or man ifort and search for vec-report or qopt-report for more details.

Many optimized precompiled numerical functions are available in packages such as Intel's Math Kernel Library or the GNU Scientific Library. By linking these into your program, rather than writing your own versions, you can be sure that you are getting excellent optimization and vectorization without additional work on your part. MKL and GSL are both provided as RC modules.