Building HWRF on Janus

Building HWRF on Janus.

HWRF consists of 9 components:

  • WRF Atmospheric Model
  • WPS
  • UPP
  • GSI
  • HWRF Utilities
  • MPIPOM-TC
  • GFDL Vortex Tracker
  • NCEP Atmosphere-Ocean Coupler
  • HWRF Run Component

We are only going to build the WRF-NMM component of HWRF.

 

On Janus we recommend building software on a compile node and in your /projects directory as it has snap-shots and is replicated.

Lets login to a compile node and create a HWRF directory and change to it.

 login01 ~$ ssh janus-compile1 janus-compile1 ~$ mkdir /projects/${USER}/HWRF janus-compile1 ~$ cd /projects/${USER}/HWRF  

 

Obtain the latest version of HWRF from the DTC download page. Untar it and change into the directory.

 janus-compile1 $ tar zxf HWRF_v3.7a_WRFV3.tar.gz janus-compile1 $ cd WRFV3  

 

Before we can build WRF we need to load a few modules to satisfy it's dependencies. These dependencies are:

  • A compiler
  • A MPI implementation
  • A NetCDF IO library
  • A JPEG library (JasPer)
  • The Perl programming interpreter

We recommend using the Intel compiler and MPI library as these have been tuned for Janus. When you build the ocean component (POM-TC) of HWRF you will also need a parallel NetCDF library, we will load that now too.

 janus-compile1 $ ml intel janus-compile1 $ ml impi janus-compile1 $ ml pnetcdf janus-compile1 $ ml netcdf janus-compile1 $ ml jasper janus-compile1 $ ml perl  

Lets check the state of our modules

 janus-compile1 $ ml Currently Loaded Modules: 1) intel/15.0.2 (m) 4) szip/2.1 7) netcdf/4.3.3.1 2) slurm/slurm (S) 5) hdf5/1.8.15 8) jasper/1.900.1 3) impi/5.0.3.048 6) pnetcdf/1.6.1 9) perl/5.22.0 Where: S: Module is Sticky, requires --force to unload or purge m: built for host and native MIC  

Once all the modules are loaded we can save them as a group, for easier loading in the future. Lets call this group hwrf

 janus-compile1 $ ml save hwrf  

 

All package modules on Janus set an environment variable indicating thier base install directory, this variable is CURC_MODULE_ROOT. If the package has an include and/or a lib directory for it's header and library files, then the environment variables CURC_MODULE_INC and CURC_MODULE_LIB will also be set.

We need to set the following environment variables for building WRF.

 janus-compile1 $ HWRF=1 janus-compile1 $ WRF_NMM_NEST=1 janus-compile1 $ WRF_NMM_CORE=1 janus-compile1 $ WRFIO_NCD_LARGE_FILE_SUPPORT=1 janus-compile1 $ FORT_BUFFERED=true janus-compile1 $ PNETCDF=${CURC_PNETCDF_ROOT} janus-compile1 $ PNETCDF_QUILT=1 janus-compile1 $ NETCDF=${CURC_NETCDF_ROOT} janus-compile1 $ NETCDF4=1 janus-compile1 $ JASPER=${CURC_JASPER_ROOT} janus-compile1 $ JASPERLIB=${CURC_JASPER_LIB} janus-compile1 $ JASPERINC=${CURC_JASPER_INC} janus-compile1 $ export HWRF WRF_NMM_NEST WRF_NMM_CORE \ WRFIO_NCD_LARGE_FILE_SUPPORT \ FORT_BUFFERED \ PNETCDF PNETCDF_QUILT \ NETCDF NETCDF4 \ JASPER JASPERINC JASPERLIB  

 

The configuration of WRF prompts you for information as to how you want to build it. HWRF only supports the distributed memory (dmpar) option. Since we are using the Intel compiler, this means it is option 15.

 janus-compile1 $ ./configure checking for perl5... no checking for perl... found /curc/tools/x86_64/rh6/software/perl/5.22.0/bin/perl (perl) Will use NETCDF in dir: /curc/tools/x86_64/rh6/software/netcdf/4.3.3.1/impi/5.0.3.048/intel/15.0.2 Will use PNETCDF in dir: /curc/tools/x86_64/rh6/software/pnetcdf/1.6.1/impi/5.0.3.048/intel/15.0.2 PHDF5 not set in environment. Will configure WRF for use without. building WRF with HWRF option Will use 'time' to report timing information Configuring to use jasper library to build Grib2 I/O... $JASPERLIB = /curc/tools/x86_64/rh6/software/jasper/1.900.1/intel/15.0.2/lib $JASPERINC = /curc/tools/x86_64/rh6/software/jasper/1.900.1/intel/15.0.2/include ------------------------------------------------------------------------ Please select from among the following Linux x86_64 options: 1. (serial) 2. (smpar) 3. (dmpar) 4. (dm+sm) PGI (pgf90/gcc) 5. (serial) 6. (smpar) 7. (dmpar) 8. (dm+sm) PGI (pgf90/pgcc): SGI MPT 9. (serial) 10. (smpar) 11. (dmpar) 12. (dm+sm) PGI (pgf90/gcc): PGI accelerator 13. (serial) 14. (smpar) 15. (dmpar) 16. (dm+sm) INTEL (ifort/icc) 17. (dm+sm) INTEL (ifort/icc): Xeon Phi (MIC architecture) 18. (serial) 19. (smpar) 20. (dmpar) 21. (dm+sm) INTEL (ifort/icc): Xeon (SNB with AVX mods) 22. (serial) 23. (smpar) 24. (dmpar) 25. (dm+sm) INTEL (ifort/icc): SGI MPT 26. (serial) 27. (smpar) 28. (dmpar) 29. (dm+sm) INTEL (ifort/icc): IBM POE 30. (serial) 31. (dmpar) PATHSCALE (pathf90/pathcc) 32. (serial) 33. (smpar) 34. (dmpar) 35. (dm+sm) GNU (gfortran/gcc) 36. (serial) 37. (smpar) 38. (dmpar) 39. (dm+sm) IBM (xlf90_r/cc_r) 40. (serial) 41. (smpar) 42. (dmpar) 43. (dm+sm) PGI (ftn/gcc): Cray XC CLE 44. (serial) 45. (smpar) 46. (dmpar) 47. (dm+sm) CRAY CCE (ftn/gcc): Cray XE and XC 48. (serial) 49. (smpar) 50. (dmpar) 51. (dm+sm) INTEL (ftn/icc): Cray XC 52. (serial) 53. (smpar) 54. (dmpar) 55. (dm+sm) PGI (pgf90/pgcc) 56. (serial) 57. (smpar) 58. (dmpar) 59. (dm+sm) PGI (pgf90/gcc): -f90=pgf90 60. (serial) 61. (smpar) 62. (dmpar) 63. (dm+sm) PGI (pgf90/pgcc): -f90=pgf90 Enter selection [1-63] :  

A file called configure.wrf should have been created with correct compiler listed.

 

To build WRF we now run compile, in doing so we need to tell it what options we want and we should capture the output to a file in case anything goes wrong, so we can review the error later.

 janus-compile1 $ ./compile nmm_real 2>&1 | tee compile.log  

 

This will take awhile, when it is complete there should be two executables under the main directory:

  • wrf.exe
  • real_nmm.exe

If those executables are not there, then there was a problem with the build. It is best to read the compile.log file to see what the problem is. If a recompilation is necessary, a clean to remove all object files (except those in external) should be completed first.

 janus-compile1 $ ./clean  

A complete clean is strongly recommended if the compilation failed, the Registry has been changed, or the configuration file is changed. To conduct a complete clean that removes all built files in all directories, as well as the configure.wrf use the -a option.

 janus-compile1 $ ./clean -a