Skip to content

Compiling GEOS-Chem High Performance

GEOS-Chem High Performance (GCHP) uses Cmake to build. Consult the official documentation which contains full build instructions.[1]

Prerequisites

This is an MPI parallel code.

Modules to Load

Some of these will also load other prerequisites:

  • intel/composerxe/2020u4
  • picotte-openmpi/intel/2020
  • netcdf-fortran/intel/2020
  • esmf/intel/2020/8.2.0
  • cmake (this is only for configuring the build)

Download Sources

Download from Git, and make sure to check out the desired version.

[juser@picotte001 ~]$ cd /ifs/groups/myrsrchGrp
[juser@picotte001 myrsrchGrp]$ mkdir software
[juser@picotte001 myrsrchGrp]$ cd software
[juser@picotte001 software]$ git clone https://github.com/geoschem/GCHP
[juser@picotte001 software]$ cd GCHP
[juser@picotte001 GCHP]$ git checkout 13.4.0
HEAD is now at .....
[juser@picotte001 GCHP]$ git submodule update --init --recursive

Configure and Build

Configure Compilers

Use Cmake’s text user interface (TUI).

[juser@picotte001 GCHP]$ mkdir BUILD
[juser@picotte001 GCHP]$ cd BUILD
[juser@picotte001 BUILD]$ ccmake ..

You should see a blank screen, with “EMPTY CACHE” near the top, and a short guide to the command keys at the bottom:

                                                     Page 0 of 1
 EMPTY CACHE


...


EMPTY CACHE:
Keys: [enter] Edit an entry [d] Delete an entry                                     CMake Version 3.22.2
      [l] Show log output   [c] Configure
      [h] Help              [q] Quit without generating
      [t] Toggle advanced mode (currently off)

Type “c” to configure. The discovered compilers will not be correct, because Cmake will pick up the non-MPI compilers. They will have to be manually reset. The configure step should fail, and there should be this line

Could NOT find MPI_C (missing: MPI_C_WORKS)

amongst the output.

Type “e” to exit the error messages screen. Then, type “t” to toggle ON the advanced mode. Use the arrow keys to go down line by line. These three variables need to be changed:

  • CMAKE_CXX_COMPILER
  • CMAKE_C_COMPILER
  • CMAKE_Fortran_COMPILER

When you reach the appropriate lines, type “Enter” to edit the value. Use these:

  • CMAKE_CXX_COMPILER = /ifs/opt/openmpi/intel/2020/4.1.2/bin/mpic++
  • CMAKE_C_COMPILER = /ifs/opt/openmpi/intel/2020/4.1.2/bin/mpicc
  • CMAKE_Fortran_COMPILER = /ifs/opt/openmpi/intel/2020/4.1.2/bin/mpifort

After each change, type “Enter” to save the value.

Type “c” to configure again. The same error as before may show up. Type “e” to exit the error screen.

Changing the values of the compiler settings will trigger Cmake to start the configuration process from scratch. A minimal configuration showing only 3 lines should be seen:

                                                     Page 1 of 1
 CMAKE_CXX_COMPILER               /ifs/opt/openmpi/intel/2020/4.1.2/bin/mpic++
 CMAKE_C_COMPILER                 /ifs/opt/openmpi/intel/2020/4.1.2/bin/mpicc
 CMAKE_Fortran_COMPILER           /ifs/opt/openmpi/intel/2020/4.1.2/bin/mpifort

Type “c” again, to configure. The configure step will fail again.

Configure ESMF

This time, the configure fails because it cannot find ESMF[2].

 CMake Error at /ifs/opt/cmake/3.22.2/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
   Couldn't find one or more files in your ESMF installation! Set
   CMAKE_PREFIX_PATH to the directory/ies containing the missing files to
   finish locating ESMF.  (missing: ESMF_HEADERS_DIR ESMF_MOD_DIR
   ESMF_LIBRARY) (found version "NOTFOUND")
 Call Stack (most recent call first):
   /ifs/opt/cmake/3.22.2/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
   ESMA_cmake/FindESMF.cmake:123 (find_package_handle_standard_args)
   ESMA_cmake/FindBaselibs.cmake:117 (find_package)
   ESMA_cmake/esma.cmake:109 (include)
   CMakeLists.txt:84 (include)

 Configuring incomplete, errors occurred!
 See also "/ifs/opt/src/GCHP/BUILD/CMakeFiles/CMakeOutput.log".

ESMF settings will have to be set manually. The process is similar to the one used for setting the compilers, as above. Type “e” to exit the error messages screen.

Back at the configuration screen, make sure “advanced mode” is ON. Use the arrow keys to go down in the list of configuration values. You can also search for a string: type “/” (slash) and enter “ESMF” and type Enter. That should bring you to the first occurrence of “ESMF”. (Hitting “n” will bring you to subsequent occurrences.)

These three variables need to be modified:

  • ESMF_HEADERS_DIR
  • ESMF_LIBRARY
  • ESMF_MOD_DIR

The appropriate values are set by the esmf/intel/2020 modulefile. See them by doing this in a shell (separate from the one working with Cmake):

[juser@picotte001 GCHP]$ env | grep ESMF | sort
ESMF_COMM=openmpi
ESMF_COMPILER=intel
ESMF_HEADERS_DIR=/ifs/opt/esmf/intel/2020/8.2.0/include
ESMF_INSTALL_PREFIX=/ifs/opt/esmf/intel/2020/8.2.0
ESMF_LAPACK=mkl
ESMF_LIBRARY=/ifs/opt/esmf/intel/2020/8.2.0/lib/libO/Linux.intel.64.openmpi.default/libesmf.so
ESMF_MOD_DIR=/ifs/opt/esmf/intel/2020/8.2.0/mod/modO/Linux.intel.64.openmpi.default
ESMF_MPIRUN=/ifs/opt/openmpi/intel/2020/4.1.2/bin/mpirun -x LD_LIBRARY_PATH -x BASH_ENV
ESMF_XERCES=standard

So:

  • ESMF_HEADERS_DIR = /ifs/opt/esmf/intel/2020/8.2.0/include
  • ESMF_LIBRARY = /ifs/opt/esmf/intel/2020/8.2.0/lib/libO/Linux.intel.64.openmpi.default/libesmf.so
  • ESMF_MOD_DIR = /ifs/opt/esmf/intel/2020/8.2.0/mod/modO/Linux.intel.64.openmpi.default

The variable ESMF_MK_FILEPATH can be left unset.

Type “c” again, to configure. Then “e” to exit the messages screen.

Configure use of MKL (Math Kernel Library)

The installed ESMF is linked with Intel’s MKL[3] but GCHP’s Cmake configuration does not take this into account. Without setting appropriate library linking for the final executable, the build process will not produce the gchp program, and generates errors when linking gchp:

[100%] Linking Fortran executable ../bin/gchp
/ifs/opt/esmf/intel/2020/8.2.0/lib/libO/Linux.intel.64.openmpi.default/libesmf.so: undefined reference to `dgelsd_'
/ifs/opt/esmf/intel/2020/8.2.0/lib/libO/Linux.intel.64.openmpi.default/libesmf.so: undefined reference to `dsyevd_'
/ifs/opt/esmf/intel/2020/8.2.0/lib/libO/Linux.intel.64.openmpi.default/libesmf.so: undefined reference to `dgeev_'
/ifs/opt/esmf/intel/2020/8.2.0/lib/libO/Linux.intel.64.openmpi.default/libesmf.so: undefined reference to `ilaenv_'
/ifs/opt/esmf/intel/2020/8.2.0/lib/libO/Linux.intel.64.openmpi.default/libesmf.so: undefined reference to `dgelsy_'
make[2]: *** [src/CMakeFiles/gchp.dir/build.make:146: bin/gchp] Error 1
make[1]: *** [CMakeFiles/Makefile2:3102: src/CMakeFiles/gchp.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

The undefined references (dgelsd_, dsyevd, dgeev_, ilaenv_, dgelsy_) are all names of linear algebra functions defined in MKL.

Modify the following variable in Cmake. This uses the value of the MKLROOT environment variable:

  • MPI_Fortran_LINK_FLAGS = -L/ifs/opt/intel/2020/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin -Wl,-rpath,/ifs/opt/intel/2020/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin -lmkl_rt

Configure build of shared libraries

Next, configure the following variable to “ON”. Hit the spacebar to toggle the value between “OFF” and “ON”:

  • BUILD_SHARED_LIBS = ON

Configure use of OpenMP (multithreading parallel)

Set this variable to “ON”:

  • OMP = ON

Configure

Type “c” to configure. It should no longer show any failure messages, and instead show “Configuring done”:

 GEOS-Chem 13.4.0 (science codebase)
 Current status: 13.4.0
 =================================================================
 Settings:
   * OMP:  [32mON[m  OFF
   * USE_REAL8:  [32mON[m  OFF
   * APM:  ON  [32mOFF[m
   * RRTMG:  ON  [32mOFF[m
   * GTMM:  ON  [32mOFF[m
   * LUO_WETDEP:  ON  [32mOFF[m
 Configuring done

Type “e” to exit the messages.

Generate the makefile

Type “g” to generate the makefile. It may take a few minutes, showing a “Generating…” status, and exit automatically returning you to your shell:

[juser@picotte001 GCHP]$

Build a.k.a. compile a.k.a. make

Build the code, redirecting output to a file so that all output (including errors) are captured for later view:

[juser@picotte001 GCHP]$ make -j 24 | tee Make.out 2>&1
[ 1%] Building ...
...

The “-j 24” tells make to run up to 24 compilations simultaneously. The “tee” command splits the input it receives into the terminal and also the file Make.out. The “2>&1” at the end of the line joins stderr (i.e. error output) to stdout (i.e. normal output).

This may take a few hours.

A successful build should end with:

[100%] Linking Fortran executable ../bin/gchp
[100%] Built target gchp

And the ghcp program (executable) will be in the bin directory:

[juser@picotte001 BUILD]$ ls -l bin
total 39472
-rwxrwxr-x 1 juser myrsrchGrp 33429632 May 12 16:02 gchp

See Also

References

[1] GEOS-Chem High Performance official documentation

[2] Earth System Modeling Framework

[3] Intel® oneAPI Math Kernel Library