Compiling OpenFOAM 3.0.x
NOTE: THIS IS A WORK IN PROGRESS
NOTE 2: THIS IS ONLY FOR VERSION 3.0.x
OpenFOAM is free, open-source software for computational fluid dynamics.[1]
Installed Version
For information on using the pre-installed version on Proteus, see OpenFOAM.
Preliminaries
- Prerequisites:
- https://www.openfoam.com/documentation/system-requirements.php
- They recommend GCC >= 4.8.5, we have 4.8.1
- Follow build instructions for the development version:
- http://www.openfoam.com/code/build-guide.php
- http://www.openfoam.org/download/git.php
- However, do not use the development version to build -- it seems to use a newer OpenMPI, which fails to compile with the installed OpenMPI 1.8.1
- There are, roughly, two components to OpenFOAM:
- the actual CFD code FOAM -- definitely to be compiled with ICC
- the GUI analysis tools, based on ParaView (using the Qt GUI toolkit) -- only compiles properly with GCC; has option to use MPI for the compute-intensive analysis tasks, but we will drop that
- The documented build puts things in the user's home directory, i.e.
${HOME}/OpenFOAM/OpenFOAM-3.0.x/
- As compiler, we will use Intel Composer XE 2015 with OpenMPI 1.8.1
- Some of the ThirdParty packages need to be built first, e.g CGAL.
Download
Source packages are available from the download section of the OpenFOAM website:
http://www.openfoam.org/download/source.php
Expand and Rename
tar xf OpenFOAM-3.0.1.tar.gz
mv OpenFOAM-3.0.x-version-3.0.1 OpenFOAM-3.0.x
tar xf OpenFOAM-ThirdParty-3.0.1.tgz
mv ThirdParty-3.0.1 ThirdParty-3.0.x
Download Additional Third Party Sources
These third party sources are also required, but are not bundled:
- boost - 1.60.0 works
- gmp - 6.1.0 works
- mpfr - 3.1.3 works
- metis -5.1.0 is expected[2]
The source tarballs should be expanded in the ThirdParty-3.0.x
directory.
Prerequisites
Needs a DOS emulator, according to the docs:
dosemu
Or maybe not.
Setup
Modules
Load some modules:
module load intel/composerxe/2015.1.133
module load proteus-openmpi/intel/2015/1.8.1-mlnx-ofed
module load python/2.7.10
Unpack Source into Final Location
The source code must be unpacked into the top-level directory of the final installation, e.g. into /usr/local/OpenFOAM, or /mnt/HA/opt/OpenFOAM, or ${HOME}/OpenFOAM.
The default location for the source and the resulting executables are:
${HOME}/OpenFOAM/OpenFOAM-3.0.x/
This will be assigned to the environment variable WM_PROJECT_DIR, as
in the next section. From here on, we will refer to this unpacked source
directory as WM_PROJECT_DIR.
The source bundle, as distributed, will expand into a directory named:
OpenFoam-3.0.x-version-3.0.1/
so this must be moved and/or renamed appropriately.
Edit Shell Environment
Maybe, maybe not. The environment must be set up according to where the source code directories are located. Set these before modifying the env files:
export WM_CC=icc
export WM_CXX=icpc
export WM_CFLAGS="-m64 -fPIC -O3 -xHost -fopenmp -fp-model precise"
export WM_CXXFLAGS="-m64 -fPIC -O3 -xHost -fopenmp -fp-model precise"
export WM_COMPILER_ARCH=IccDPInt32Opt
This is accomplished by editing the init file:
${WM_PROJECT_DIR}/etc/bashrc
and cshrc, if you use a csh variant.
These changes to be made:
foamInstall=/mnt/HA/opt/$WM_PROJECT
...
export WM_COMPILER=Icc
Then, execute the file to set the environment variables:
. ${WM_PROJECT_DIR}/etc/bashrc
TODO: may need to add ThirdParty dirs to LD_LIBRARY_PATH etc.
wmake
Set environment variable
export WM_DIR=${WM_PROJECT_DIR}/wmake
Set Up Compilation Options a.k.a. rules
In the directory ${WM_DIR}/rules/linux64Icc/, modify these files:
cOpt, and c++Opt
### cOpt
cDBUG =
cOPT = -no-prec-div -O3 -xHost -fp-model precise
### c++Opt
c++DBUG =
c++OPT = -O2 -xHost -fp-model precise
Then, build wmake:
cd $WM_DIR
./makeWmake
Third Party
OpenFOAM has configs which set where it expects to find its "third
party" packages. There are in $FOAM_ETC/config/ as various shell
scripts, e.g. scotch.sh or scotch.csh.
These typically set the root of the installations as something like:
export SCOTCH_VERSION=scotch_6.0.3
export SCOTCH_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER$WM_LABEL_OPTION/$SCOTCH_VERSION
but some do not care about the WM_LABEL_OPTION:
### XXX we will modify this for boost in our installation
boost_version=boost-system
cgal_version=CGAL-4.7
export BOOST_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$boost_version
export CGAL_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$cgal_version
The top-level OpenFOAM-3.0.x/Allwmake will build Scotch, but not CGAL,
boost, gmp, and mpfr. So, those four packages need to be built
separately, and installed into the appropriate directory as above.
Prerequisites
These are for CGAL, which is required for OpenFOAM:
- Boost
- GMP
- MPFR
Also Metis, which is not distributed along with OpenFOAM source.
NB The OpenFOAM interface to Metis seems to be broken. Will not install Metis.
Source code should be downloaded and expanded into
$WM_THIRD_PARTY_DIR. The makeCGAL script should do the
compilation.
And to use parallel builds:
export WM_NCOMPPROCS=8
It seems like the makeCGAL script is expecting a different layout than the Allwmake script.
GMP
./configure CC=icc CXX=icpc CFLAGS="-xHost -O3 -fPIC -fp-model precise" CXXFLAGS="-xHost -O3 -fPIC -fp-model precise" --prefix=$WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/gmp-6.1.0 --enable-cxx
make -j 8
make check
make install
MPFR
NB Thread-safety (TLS) seems to be broken, so disable in configure.
./configure CC=icc CXX=icpc CFLAGS="-xHost -O3 -fPIC -fp-model precise" CXXFLAGS="-xHost -O3 -fPIC -fp-model precise" --prefix=$WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/mpfr-3.1.3 --with-gmp=$WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/gmp-6.1.0 --disable-thread-safe
make -j 8
make check
make install
Boost
No matter what the actual Boost version is, OpenFOAM sets boost_version=boost-system in OpenFOAM-3.0.x/etc/config/CGAL.sh
scotch
Edit the wmakefile template:
${WM_THIRD_PARTY_DIR}/etc/wmakefiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
--- Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32.orig 2016-01-27 20:33:17.812369912 -0500
+++ Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32 2016-01-27 20:32:44.074585298 -0500
@@ -3,13 +3,13 @@
OBJ = .o
MAKE = make
-AR = gcc
+AR = icc
ARFLAGS = -shared -o
CAT = cat
-CCS = gcc
+CCS = icc
CCP = mpicc
CCD = mpicc
-CFLAGS = -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
+CFLAGS = -O3 -xHost -fp-model precise -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
CLIBFLAGS = -shared -fPIC
LDFLAGS = -Xlinker --no-as-needed -lz -lm -lrt
CP = cp
Manually Munge the Platform Directory
cd $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER}
ln -s $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/lib .
ln -s $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/gmp-6.1.0 .
ln -s $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/mpfr-3.1.3 .
ln -s boost_1_60_0 boost-system
CGAL
Requires the following, which need to be downloaded separately into the appropriate directory. Using unoptimized system versions is the default behavior:
- Boost
- GMP
- MPFR
The makeCGAL script will build Boost, but not GMP or MPFR. Those
two need to be built and installed in the appropriate platforms
subdirectory, first. See above.
First, edit the makeCGAL script so that Boost is built with the
same compiler, i.e. Intel. The diff is below:
--- /mnt/HA/opt/src/OpenFOAM/ThirdParty-3.0.1/makeCGAL 2015-12-14 03:36:57.264335875 -0500
+++ ./makeCGAL 2016-01-25 18:11:09.743207164 -0500
@@ -131,6 +131,13 @@
BOOST_ARCH_PATH=$installBASE/$boostPACKAGE
BOOST_SOURCE_DIR=$WM_THIRD_PARTY_DIR/$boostPACKAGE
+if [ "$WM_COMPILER" = "Icc" ]
+then
+ BOOST_TOOLSET=intel
+else
+ BOOST_TOOLSET=gcc
+fi
+
if [ -d "$BOOST_ARCH_PATH" ]
then
boostInc="$BOOST_ARCH_PATH/include"
@@ -150,7 +157,7 @@
--prefix=$BOOST_ARCH_PATH \
--with-libraries=thread \
--with-libraries=system \
- && ./bjam toolset=$WM_CC -j $WM_NCOMPPROCS install \
+ && ./bjam toolset=$BOOST_TOOLSET -j $WM_NCOMPPROCS install \
&& echo "Built: boost"
) || {
echo "Error building: boost"
GMP and MPFR need to be dealt with differently.
Then, compile:
./makeCGAL boost_1_60_0 gmp-6.1.0 mpfr-3.1.3
This puts boost in:
${WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER}/boost_1_60_0
All Remaining ThirdParty Packages (except Qt & ParaView)
./Allwmake
Qt
The makeQt script seems to fail. Run it once till it fails. Then,
cd to the qt build directory and modify config.status so that gcc and
g++ are used. Also add CFLAGS and CXXFLAGS as needed.
Build and install:
./config.status
make -j 8
make install
ParaView
- Must be built from source because OpenFOAM has ParaView modules that need to be built, requiring ParaView headers.
-
Read https://github.com/OpenFOAM/ThirdParty-dev/blob/master/README.org
-
In particular: > The ParaView-4.3.¼.4.0 source-pack provided by KitWare will > not build and install correctly without a minor correction: > lines 653-656 must be removed or commented from file > ThirdParty-dev/ParaView-4.?.?/Qt/Components/CMakeLists.txt > after unpacking and renaming the ParaView-4.?.? directory. > This is because the file ui_pqExportStateWizard.h referred to > in the lines 653-656 … does not exist and cannot be installed > causing the build and installation to fail.
-
And the fix to
VTK/IO/Geometry/vtkSTLReader.cxx
-
Build OpenFOAM
Finally, build OpenFOAM itself:
cd $WM_PROJECT_DIR
./Allwmake
Adding New Utility Code
Errors due to "missing" .H file
If there are errors about a missing .H file, add a line to the EXE_INC
definition in the utility's Make/options file:[3]
-I$(LIB_SRC)/meshTools/lnInclude
This include should be the first, e.g. (Note the backslash "\" at the end of the line)
EXE_INC = \
-I$(LIB_SRC)/meshTools/lnInclude \
-I$(LIB_SRC)/postProcessing/postCalc \
-I$(LIB_SRC)/finiteVolume/lnInclude