Skip to content

Compiling OpenFOAM 6

NOTE: This is a work in progress

Official Documentation

Requirements

  • gcc >= 4.8.5
    • default GCC on the system is 4.4.7
    • GCC provided by modulefiles is 4.8.1
    • GCC provided by the Software Collection devtoolset-3 is 4.9.2
  • but we will try using icc

module load intel/composerxe/2015.1.133

  • some issues using icc
    • use gcc-4.9 instead
  • cmake >= 3.3
  • boost >= 1.48
  • fftw >= 3.3.7 (opt)
  • Qt >= 5.9.3 (opt)

Edit Configs

Edit

/mnt/HA/opt/OpenFOAM6/OpenFOAM-6/etc/bashrc /mnt/HA/opt/OpenFOAM6/OpenFOAM-6/etc/cshrc

Should not have to modify anything in (yes, despite the names, these are really directories):

/mnt/HA/opt/OpenFOAM6/OpenFOAM-6/etc/config.sh/ /mnt/HA/opt/OpenFOAM6/OpenFOAM-6/etc/config.csh/

but it's worth taking a look.

Selecting Compiler

GCC 4.9 via Red Hat Software Collections

GCC 4.9 is provided by the devtoolset-3 software collection. To enable it, enable the collection in a new shell:

scl enable devtoolset-3 /bin/bash

Then, make sure the gcc/4.8.1 module is unloaded:

module unload gcc/4.8.1

Load cmake:

module load cmake/3.12.2

Rename Directories

When expanded, the downloaded source code tarballs do not produce the proper directory names. Rename them:

  • OpenFOAM-6
  • ThirdParty-6

i.e. remove "-version-6" from the directory names.

Configure

  • have to use cmake directly

Open MPI

Technically, this is a "third party" software. However, the version distributed with OpenFOAM 6 is old, and is not compatible with the hardware-specific libraries we have installed (MXM, FCA). While we already have a gcc48-compiled version installed on Proteus, we will compile an OpenFOAM6-private version.

Open MPI 1.8.1

Later releases in the Open MPI 1.8.x series are not compatible with the Mellanox drivers we have installed. 1.8.1 is known to work, so this is what we will use.

./configure CC=`which gcc` CXX=`which g++` FC=`which gfortran` --prefix=/mnt/HA/opt/OpenFOAM6 --with-sge \ --with-mxm=/opt/mellanox/mxm --with-fca=/opt/mellanox/fca

OpenBLAS

  • 0.3.3

Perl

  • 5.28.0

Python

FFTW

  • 3.3.8

Qt

Configure

./configure -prefix /mnt/HA/opt/OpenFOAM6  -opensource -release -rpath -no-evdev -qt-sqlite -qt-pcre -qt-zlib -nomake tests

If a make fails, just delete the entire directory, and untar the source tarball again. "make clean" and "make distclean" and "configure ... -recheck-all" don't seem to clear out some cached stuff.

NOTES:

  • XCB seems to be needed (so, can't pass "-no-xcb" to configure)

Third Party

The third party code should be in $FOAM_INST_DIR/ThirdParty-6 aka $WM_THIRD_PARTY_DIR

Download

Download all these packages, and expand the archives, in $WM_THIRD_PARTY_DIR:

CGAL depends on the other three packages. The makeCGAL script will build all of them in the correct order.

Order of Prereqs

  • Python
  • Open MPI
  • Qt
  • ParaView

ParaView

  • The makeParaView script downloads the source

./makeParaView -python -qmake `which qmake` -gcc -qt -qt-5.9.2

  • Or, download from:

https://github.com/Kitware/ParaView/releases

CGAL

  • https://www.cgal.org/
  • the makeCGAL script option for specifying boost version should take an underscore rather than hyphen, i.e.

./makeCGAL CGAL-4.1.0 boost_1_55_0 ...

MAY BE CHANGED

  • Edit the file ThirdParty-6/makeCGAL -- the option for setting the toolset to build Boost has changed for Boost 1.67.0; it used to be "intel", and should now be "intel-linux". Lines 152-156 should be:
    toolset="$WM_CC"
    if [ $WM_COMPILER == "Icc" ]; then
        toolset=intel-linux
    fi

And the line to bootstrap the Boost build needs to be changed: the toolset should be specified to bootstrap.sh:

        ./bootstrap.sh \
            --prefix=$BOOST_ARCH_PATH \
            --with-libraries=thread \
            --with-libraries=system \
            --with-toolset=$toolset \

To speed up the build, set:

    [juser@proteusi01 ThirdParty-6]$ export WM_NCOMPPROCS=16

Boost

Looks like Boost 1_67_0 may be too new.

GMP & MPFR

NOTE The makeCGAL script does not seem to build GMP and MPFR.

libquadmath -- for some reason, the gcc/4.8.1 module provided by Bright has a 32-bit libquadmath, which MPFR complains about when running the checks after building.

boost -- and if you unload gcc/4.8.1 you can't use the Proteus cmake/3.9.4 because that requires libraries from gcc/4.8.1

So,make sure to unload the gcc module to pick up the 64-bit library in /usr/lib64/libquadmath.so

export GMP_ARCH_PATH=${WM_THIRD_PARTY_DIR}/platforms/linux64Icc/gmp-6.1.2
export MPFR_ARCH_PATH=${WM_THIRD_PARTY_DIR}/platforms/linux64Icc/mpfr-4.0.1

Build and install GMP and MPFR using the ARCH_PATH directories above as the respective prefix directories.

For GMP:

[juser@proteusi01 gmp-6.1.2]$ ./configure --prefix=$GMP_ARCH_PATH

For MPFR:

[juser@proteusi01 mpfr-4.0.1]$ ./configure --prefix=$MPFR_ARCH_PATH --with-gmp=${GMP_ARCH_PATH}

Build CGAL

Then, build:

./makeCGAL CGAL-4.11.2 boost_1_67_0 gmp-6.1.2 mpfr-4.0.1 >& Make_CGAL.out &

See Also