Compiling OpenFOAM 3.0.x
NOTE: THIS IS A WORK IN PROGRESS
NOTE 2: THIS IS ONLY FOR VERSION 3.0.x
OpenFOAM is free, open-source software for computational fluid dynamics.[1]
Installed Version♯
For information on using the pre-installed version on Proteus, see OpenFOAM.
Preliminaries♯
- Prerequisites:
- https://www.openfoam.com/documentation/system-requirements.php
- They recommend GCC >= 4.8.5, we have 4.8.1
- Follow build instructions for the development version:
- http://www.openfoam.com/code/build-guide.php
- http://www.openfoam.org/download/git.php
- However, do not use the development version to build -- it seems to use a newer OpenMPI, which fails to compile with the installed OpenMPI 1.8.1
- There are, roughly, two components to OpenFOAM:
- the actual CFD code FOAM -- definitely to be compiled with ICC
- the GUI analysis tools, based on ParaView (using the Qt GUI toolkit) -- only compiles properly with GCC; has option to use MPI for the compute-intensive analysis tasks, but we will drop that
- The documented build puts things in the user's home directory, i.e.
${HOME}/OpenFOAM/OpenFOAM-3.0.x/
- As compiler, we will use Intel Composer XE 2015 with OpenMPI 1.8.1
- Some of the ThirdParty packages need to be built first, e.g CGAL.
Download♯
Source packages are available from the download section of the OpenFOAM website:
http://www.openfoam.org/download/source.php
Expand and Rename♯
tar xf OpenFOAM-3.0.1.tar.gz
mv OpenFOAM-3.0.x-version-3.0.1 OpenFOAM-3.0.x
tar xf OpenFOAM-ThirdParty-3.0.1.tgz
mv ThirdParty-3.0.1 ThirdParty-3.0.x
Download Additional Third Party Sources♯
These third party sources are also required, but are not bundled:
- boost - 1.60.0 works
- gmp - 6.1.0 works
- mpfr - 3.1.3 works
- metis -5.1.0 is expected[2]
The source tarballs should be expanded in the ThirdParty-3.0.x
directory.
Prerequisites♯
Needs a DOS emulator, according to the docs:
dosemu
Or maybe not.
Setup♯
Modules♯
Load some modules:
module load intel/composerxe/2015.1.133
module load proteus-openmpi/intel/2015/1.8.1-mlnx-ofed
module load python/2.7.10
Unpack Source into Final Location♯
The source code must be unpacked into the top-level directory of the final installation, e.g. into /usr/local/OpenFOAM, or /mnt/HA/opt/OpenFOAM, or ${HOME}/OpenFOAM.
The default location for the source and the resulting executables are:
${HOME}/OpenFOAM/OpenFOAM-3.0.x/
This will be assigned to the environment variable WM_PROJECT_DIR, as
in the next section. From here on, we will refer to this unpacked source
directory as WM_PROJECT_DIR.
The source bundle, as distributed, will expand into a directory named:
OpenFoam-3.0.x-version-3.0.1/
so this must be moved and/or renamed appropriately.
Edit Shell Environment♯
Maybe, maybe not. The environment must be set up according to where the source code directories are located. Set these before modifying the env files:
export WM_CC=icc
export WM_CXX=icpc
export WM_CFLAGS="-m64 -fPIC -O3 -xHost -fopenmp -fp-model precise"
export WM_CXXFLAGS="-m64 -fPIC -O3 -xHost -fopenmp -fp-model precise"
export WM_COMPILER_ARCH=IccDPInt32Opt
This is accomplished by editing the init file:
${WM_PROJECT_DIR}/etc/bashrc
and cshrc, if you use a csh variant.
These changes to be made:
foamInstall=/mnt/HA/opt/$WM_PROJECT
...
export WM_COMPILER=Icc
Then, execute the file to set the environment variables:
. ${WM_PROJECT_DIR}/etc/bashrc
TODO: may need to add ThirdParty dirs to LD_LIBRARY_PATH etc.
wmake♯
Set environment variable
export WM_DIR=${WM_PROJECT_DIR}/wmake
Set Up Compilation Options a.k.a. rules♯
In the directory ${WM_DIR}/rules/linux64Icc/, modify these files:
cOpt, and c++Opt
### cOpt
cDBUG =
cOPT = -no-prec-div -O3 -xHost -fp-model precise
### c++Opt
c++DBUG =
c++OPT = -O2 -xHost -fp-model precise
Then, build wmake:
cd $WM_DIR
./makeWmake
Third Party♯
OpenFOAM has configs which set where it expects to find its "third
party" packages. There are in $FOAM_ETC/config/ as various shell
scripts, e.g. scotch.sh or scotch.csh.
These typically set the root of the installations as something like:
export SCOTCH_VERSION=scotch_6.0.3
export SCOTCH_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER$WM_LABEL_OPTION/$SCOTCH_VERSION
but some do not care about the WM_LABEL_OPTION:
### XXX we will modify this for boost in our installation
boost_version=boost-system
cgal_version=CGAL-4.7
export BOOST_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$boost_version
export CGAL_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$cgal_version
The top-level OpenFOAM-3.0.x/Allwmake will build Scotch, but not CGAL,
boost, gmp, and mpfr. So, those four packages need to be built
separately, and installed into the appropriate directory as above.
Prerequisites♯
These are for CGAL, which is required for OpenFOAM:
- Boost
- GMP
- MPFR
Also Metis, which is not distributed along with OpenFOAM source.
NB The OpenFOAM interface to Metis seems to be broken. Will not install Metis.
Source code should be downloaded and expanded into
$WM_THIRD_PARTY_DIR. The makeCGAL script should do the
compilation.
And to use parallel builds:
export WM_NCOMPPROCS=8
It seems like the makeCGAL script is expecting a different layout than the Allwmake script.
GMP♯
./configure CC=icc CXX=icpc CFLAGS="-xHost -O3 -fPIC -fp-model precise" CXXFLAGS="-xHost -O3 -fPIC -fp-model precise" --prefix=$WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/gmp-6.1.0 --enable-cxx
make -j 8
make check
make install
MPFR♯
NB Thread-safety (TLS) seems to be broken, so disable in configure.
./configure CC=icc CXX=icpc CFLAGS="-xHost -O3 -fPIC -fp-model precise" CXXFLAGS="-xHost -O3 -fPIC -fp-model precise" --prefix=$WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/mpfr-3.1.3 --with-gmp=$WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/gmp-6.1.0 --disable-thread-safe
make -j 8
make check
make install
Boost♯
No matter what the actual Boost version is, OpenFOAM sets boost_version=boost-system in OpenFOAM-3.0.x/etc/config/CGAL.sh
scotch♯
Edit the wmakefile template:
${WM_THIRD_PARTY_DIR}/etc/wmakefiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32
--- Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32.orig 2016-01-27 20:33:17.812369912 -0500
+++ Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-64Int32 2016-01-27 20:32:44.074585298 -0500
@@ -3,13 +3,13 @@
OBJ = .o
MAKE = make
-AR = gcc
+AR = icc
ARFLAGS = -shared -o
CAT = cat
-CCS = gcc
+CCS = icc
CCP = mpicc
CCD = mpicc
-CFLAGS = -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
+CFLAGS = -O3 -xHost -fp-model precise -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
CLIBFLAGS = -shared -fPIC
LDFLAGS = -Xlinker --no-as-needed -lz -lm -lrt
CP = cp
Manually Munge the Platform Directory♯
cd $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER}
ln -s $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/lib .
ln -s $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/gmp-6.1.0 .
ln -s $WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER_ARCH}/mpfr-3.1.3 .
ln -s boost_1_60_0 boost-system
CGAL♯
Requires the following, which need to be downloaded separately into the appropriate directory. Using unoptimized system versions is the default behavior:
- Boost
- GMP
- MPFR
The makeCGAL script will build Boost, but not GMP or MPFR. Those
two need to be built and installed in the appropriate platforms
subdirectory, first. See above.
First, edit the makeCGAL script so that Boost is built with the
same compiler, i.e. Intel. The diff is below:
--- /mnt/HA/opt/src/OpenFOAM/ThirdParty-3.0.1/makeCGAL 2015-12-14 03:36:57.264335875 -0500
+++ ./makeCGAL 2016-01-25 18:11:09.743207164 -0500
@@ -131,6 +131,13 @@
BOOST_ARCH_PATH=$installBASE/$boostPACKAGE
BOOST_SOURCE_DIR=$WM_THIRD_PARTY_DIR/$boostPACKAGE
+if [ "$WM_COMPILER" = "Icc" ]
+then
+ BOOST_TOOLSET=intel
+else
+ BOOST_TOOLSET=gcc
+fi
+
if [ -d "$BOOST_ARCH_PATH" ]
then
boostInc="$BOOST_ARCH_PATH/include"
@@ -150,7 +157,7 @@
--prefix=$BOOST_ARCH_PATH \
--with-libraries=thread \
--with-libraries=system \
- && ./bjam toolset=$WM_CC -j $WM_NCOMPPROCS install \
+ && ./bjam toolset=$BOOST_TOOLSET -j $WM_NCOMPPROCS install \
&& echo "Built: boost"
) || {
echo "Error building: boost"
GMP and MPFR need to be dealt with differently.
Then, compile:
./makeCGAL boost_1_60_0 gmp-6.1.0 mpfr-3.1.3
This puts boost in:
${WM_THIRD_PARTY_DIR/platforms/${WM_ARCH}${WM_COMPILER}/boost_1_60_0
All Remaining ThirdParty Packages (except Qt & ParaView)♯
./Allwmake
Qt♯
The makeQt script seems to fail. Run it once till it fails. Then,
cd to the qt build directory and modify config.status so that gcc and
g++ are used. Also add CFLAGS and CXXFLAGS as needed.
Build and install:
./config.status
make -j 8
make install
ParaView♯
- Must be built from source because OpenFOAM has ParaView modules that need to be built, requiring ParaView headers.
-
Read https://github.com/OpenFOAM/ThirdParty-dev/blob/master/README.org
-
In particular: > The ParaView-4.3.¼.4.0 source-pack provided by KitWare will > not build and install correctly without a minor correction: > lines 653-656 must be removed or commented from file > ThirdParty-dev/ParaView-4.?.?/Qt/Components/CMakeLists.txt > after unpacking and renaming the ParaView-4.?.? directory. > This is because the file ui_pqExportStateWizard.h referred to > in the lines 653-656 … does not exist and cannot be installed > causing the build and installation to fail.
-
And the fix to
VTK/IO/Geometry/vtkSTLReader.cxx
-
Build OpenFOAM♯
Finally, build OpenFOAM itself:
cd $WM_PROJECT_DIR
./Allwmake
Adding New Utility Code♯
Errors due to "missing" .H file♯
If there are errors about a missing .H file, add a line to the EXE_INC
definition in the utility's Make/options file:[3]
-I$(LIB_SRC)/meshTools/lnInclude
This include should be the first, e.g. (Note the backslash "\" at the end of the line)
EXE_INC = \
-I$(LIB_SRC)/meshTools/lnInclude \
-I$(LIB_SRC)/postProcessing/postCalc \
-I$(LIB_SRC)/finiteVolume/lnInclude