Compiling foam-extend

foam-extend[1] is a fork of OpenFOAM. These are instructions for foam-extend 4.0.

Build Documentation♯

Build instructions are hosted at OpenFOAM's site.[2]

General Notes Before Starting♯

Documentation is practically nonexistent.

Environment♯

foam-extend requires the compiler GCC >= 4.9

This is provided by the Red Hat Software Collection devtoolset-3.[3]

You will need to enable to software collection, and then unload all conflicting modules:

[juser@proteusi01 ~]$ scl enable devtoolset-3 /bin/bash
[juser@proteusi01 ~]$ module purge
[juser@proteusi01 ~]$ module load shared
[juser@proteusi01 ~]$ module load proteus
[juser@proteusi01 ~]$ module load git/2.20.1

I.e. there should only be three modules loaded in your environment. Check that gcc is the proper version:

[juser@proteusi01 ~]$ gcc --version
gcc (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6)
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compilation Options♯

Look in:

wmake/rules/platform_name

Select Installation Location♯

The installation location will also be where the source code will be downloaded. Select a location, e.g.

/mnt/HA/groups/myrsrchGrp/foam

Download Source Repository♯

[juser@proteusi01 ~]$ cd /mnt/HA/groups/myrsrchGrp/foam
[juser@proteusi01 foam]$ git clone git://git.code.sf.net/p/foam-extend/foam-extend-4.0 foam-extend-4.0

Environment Setup♯

Edit foam-extend-4.0/etc/bashrc and foam-extend-4.0/etc/cshrc to set the install location:

For bashrc:

foamInstall=/mnt/HA/opt/$WM_PROJECT

For cshrc:

foamInstall=/mnt/HA/opt/$WM_PROJECT

Other Prerequisites♯

Installation prefix:

$WM_PROJECT_DIR

Download source into

$WM_PROJECT_DIR/proteus

Open MPI♯

The Open MPI 1.8.8 bundled with foam-extend 4.0 does not work with the Mellanox MXM installed on Proteus. So, build a separate one, and fix the preferences/setup scripts in foam-extend to use that.

./configure --prefix=$WM_PROJECT_DIR --with-verbs --with-sge --with-hwloc=$WM_PROJECT_DIR --enable-mpi-cxx

hwloc♯

hwloc is used by Open MPI. Build a separate one, as well.

./configure --prefix=$WM_PROJECT_DIR --enable-pci --enable-libnuma --enable-libudev

Qt 4.8.7♯

git clonegit://code.qt.io/qt/qt.git

Build Preferences♯

Need to set some build preferences so that some third party software is built with options suitable for Proteus. In particular:

OpenMPI needs to be built with Grid Engine integration

In the directory foam-extend-4.0/etc/ make copies of the example preferences files:

cp prefs.sh-EXAMPLE prefs.sh
cp prefs.csh-EXAMPLE prefs.csh

And edit them.

Build Preferences for Open MPI♯

The bundled Open MPI 1.8.8 requires Mellanox MXM 2.x, but Proteus has MXM 1.5 installed.

So, disable using the bundled version, and use a "system" version.

In prefs.sh:

# Specify system openmpi
# ~~~~~~~~~~~~~~~~~~~~~~
#
# Normally, you don't need to set more than these 3 env. variables
# The other openmpi related variables will be initialized using
# the command mpicc --showme:
#
export WM_MPLIB=SYSTEMOPENMPI
export OPENMPI_DIR=$WM_PROJECT_DIR
export OPENMPI_BIN_DIR=$OPENMPI_DIR/bin
#
export OPENMPI_LIB_DIR="`$OPENMPI_BIN_DIR/mpicc --showme:libdirs`"
export OPENMPI_INCLUDE_DIR="`$OPENMPI_BIN_DIR/mpicc --showme:incdirs`"
export OPENMPI_COMPILE_FLAGS="`$OPENMPI_BIN_DIR/mpicc --showme:compile`"
export OPENMPI_LINK_FLAGS="`$OPENMPI_BIN_DIR/mpicc --showme:link`"

And equivalently prefs.csh:

# Specify system openmpi
# ~~~~~~~~~~~~~~~~~~~~~~
#
# Normally, you don't need to set more than these 3 env. variables
# The other openmpi related variables will be initialized using
# the command mpicc --showme:
#
setenv WM_MPLIB SYSTEMOPENMPI
setenv OPENMPI_DIR $WM_PROJECT_DIR
setenv OPENMPI_BIN_DIR $OPENMPI_DIR/bin
#
setenv OPENMPI_LIB_DIR "`$OPENMPI_BIN_DIR/mpicc --showme:libdirs`"
setenv OPENMPI_INCLUDE_DIR "`$OPENMPI_BIN_DIR/mpicc --showme:incdirs`"
setenv OPENMPI_COMPILE_FLAGS "`$OPENMPI_BIN_DIR/mpicc --showme:compile`"
setenv OPENMPI_LINK_FLAGS "`$OPENMPI_BIN_DIR/mpicc --showme:link`"

Build preferences for hwloc♯

In prefs.sh:

# System installed hwloc
export HWLOC_SYSTEM=1
export HWLOC_DIR=$WM_PROJECT_DIR
export HWLOC_BIN_DIR=$HWLOC_DIR/bin

In prefs.csh:

# System installed hwloc
setenv HWLOC_SYSTEM 1
setenv HWLOC_DIR $WM_PROJECT_DIR
setenv HWLOC_BIN_DIR $HWLOC_DIR/bin

Build Preferences for Qt 4♯

For prefs.sh:

export QT_DIR=$WM_PROJECT_DIR export QT_BIN_DIR=$QT_DIR/bin

For prefs.csh:

setenv QT_DIR $WM_PROJECT_DIR setenv QT_BIN_DIR $QT_DIR/bin

Using Intel Composer XE♯

The Intel compiler suite generates code that runs faster than code generated with GCC. However, FOAM does not take advantage of the Math Kernel Library (MKL) which provides optimized math (primarily linear algebra) functions.

Modules♯

gcc/4.8.1 intel/composerxe/2015.1.133 proteus-openmpi/intel/2015/1.8.1-mlnx-ofed cmake/gcc/3.9.4

gcc/4.8.1 is needed because the Intel C++ compiler relies on GCC's libstdc++

Modify Compilation Options♯

Compilation options are in:

$WM_PROJECT_DIR/wmake/rules

Pick the directory for your OS/compiler combination. In our case, it is:

linux64Icc

Edit it to be:

c++DBUG     = -Wall -g -O0
c++OPT      = -ansi-alias -O3 -xHost -fno-alias -fp-model precise -fargument-noalias-global -unroll0

Modify etc Files♯

Using the above, modify:

etc/bashrc etc/prefs.sh etc/settings.sh

Since we are building the following ourselves or we are using versions already installed on Proteus, set the etc files above to say that we are using "system" versions:

Open MPI
Python
Qt
ParaView
cmake

Order of Build♯

Python 2.7.x - PyFOAM needs these, and their dependencies, too; versions listed are last versions which work with Python 2
- numpy 1.16.4
- scipy 1.2.2
- pandas
Qt 4
- Builds just fine with Intel C++ (icpc) 2015.1.133
Paraview 4.4.0
- git clone --branch v4.4.0 https://gitlab.kitware.com/paraview/paraview
- git submodule init
- git submodule update
- ccmake
  - enable Python 2
- make

Additional Toolkits♯

Fluid-Solid Interaction♯

From: https://openfoamwiki.net/index.php/Extend-bazaar/Toolkits/Fluid-structure_interaction

Need to fix the compilation options, which are all set as "c++WARN". Modify all these files:

./FluidSolidInteraction/run/ampFsiFoam/axisymmetricTube/writeFsiData/Make/options
./FluidSolidInteraction/run/fsiFoam/HronTurekFsi3/hronTurekReport/Make/options
./FluidSolidInteraction/src/fluidSolidInteraction/fluidSolvers/finiteVolume/RBFMeshMotionSolver/Make/options
./FluidSolidInteraction/src/fluidSolidInteraction/Make/options
./FluidSolidInteraction/src/solvers/ampFsiFoam/Make/options
./FluidSolidInteraction/src/solvers/fsiFoam/Make/options
./FluidSolidInteraction/src/solvers/weakFsiFoam/Make/options
./FluidSolidInteraction/src/utilities/functionObjects/centrifugalBodyForce/Make/options
./FluidSolidInteraction/src/utilities/functionObjects/pointHistory/Make/options

Replace the "c++WARN" line with:

c++WARN = -Wall

Edit the file:

FluidSolidInteraction/src/fluidSolidInteraction/fluidSolvers/pisoFluid/pisoFluid.C

At line 270, add a new line such that the section around it looks like:

    // Prepare for the pressure solution
    label pRefCell = 0;
    scalar pRefValue = 0.0;
    setRefCell(p_, fluidProperties(), pRefCell, pRefValue);
    mesh.schemesDict().setFluxRequired(p_.name());  // THIS IS THE NEW LINE

References♯

[1] foam-extend website at SourceForge

[2] Instructions for compiling/installing foam-extend 4.0 on Linux

[3] How to use Red Hat Software Collections (RHSCL) or Red Hat Developer Toolset (DTS)?