Skip to content

Compiling Python

Python 2.7.6

These instructions will also work with any Python 2.7.x version, with appropriate modifications.

Assumptions

  • Group: myresearchGrp
  • Architecture: generic x86_64, executable on both AMD and Intel nodes

Environment

Modules:

Currently Loaded Modulefiles: 1) shared            2) proteus           3) sge/univa     4) gcc/4.8.1

Environment variables:

[juser@proteusa01 ~]$ export CFLAGS="-O3 -march=corei7-avx -mfpmath=sse -mavx -malign-double"

Configure

./configure --prefix=/mnt/HA/groups/myresearchGrp/opt --enable-shared --with-threads --with-fpectl --with-ensurepip

Test

[juser@proteusa01 Python-2.7.6]$ make test | tee Make.test.out

This will put a lot of output to the terminal.

Once that is done, look in the file Make.test.out for test results. If you have followed the instructions here, there should be no unexpected failures.

Install

[juser@proteusa01 Python-2.7.6]$ make install >& Make.install.out &

Python 3.7

numpy 1.17.2

numpy 1.17.2 (and possibly some earlier versions) makes use of the C99 standard. Set the following environment variable to enable C99 features:

  • for icc: export CFLAGS="-std=c99"
  • for gcc: export CFLAGS="-std=c99"

Create Module File

See Environment Modules#Creating Your Own Module Files for instructions on creating a modulefile that can be used by all members of your group. So, you can set up something for your group like:

[juser@proteusa01 ~]$ module load myresearch-python/2.7.6

Example

#%Module -*- tcl -*-
##
## modulefile
##
proc ModulesHelp { } {
    global version
    puts stderr "\tPersonal Python 2.7.6"
}

set version "2.7.6"

module-whatis "Personal Python $version"

set              root                 /mnt/HA/groups/myresearchGrp/opt
prepend-path     PATH                 $root/bin
prepend-path     LD_LIBRARY_PATH      $root/lib

Third-party Python Modules/Packages

First, install pip, and then, use it to install almost any Python module.

pip

pip is a Python module installer.

Installation

Follow instructions here: http://www.pip-installer.org/en/latest/installing.html

[juser@proteusa01 src]$ wgethttps://bootstrap.pypa.io/get-pip.py [juser@proteusa01 src]$ python2.7 get-pip.py

pip can successfully install many packages, including:

  • numpy
  • scipy
  • ipython
  • matplotlib
  • qiime -- this requires an older version of numpy than is current; the pip installation will automatically downgrade numpy, which may break scipy and matplotlib. Best to install the exact version of numpy that qiime requires first.
  • scikit-learn

Installing Modules in Your Own Account

Use pip, e.g.

[juser@proteusa01 ~]$ pip install --user qiime

numpy

numpy is a numerical analysis and linear algebra package for Python. It may be installed with pip.

NOTE

  • Installing numpy with pip may mean that certain features are not supported. In particular, BLAS support will not be compiled in. For BLAS support (required for scipy), you will need to download numpy source, modify numpy/distutils/site.cfg, and build/install numpy using its setup.py file. See instructions here: http://www.scipy.org/scipylib/building/linux.html
  • Despite reference to FFTW3 in the site.cfg file, neither numpy nor scipy make use of FFTW/FFTW3.[1]

site.cfg

[ALL]
library_dirs = /usr/local/lib
include_dirs = /usr/local/include

[mkl]
library_dirs = /mnt/HA/opt/intel/composer_xe_2019u1/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64
include_dirs = /mnt/HA/opt/intel/composer_xe_2019u1/compilers_and_libraries_2019.1.144/linux/mkl/include
mkl_libs = mkl_rt
lapack_libs =  mkl_lapack95_lp64

[fftw]
library_dirs = /mnt/HA/opt/intel/composer_xe_2019u1/mkl/lib/intel64
libraries = fftw3xc_intel

Specific versions

Some software, such as QIIME requires a specific version of numpy, older than the current version. To request a specific version:

[juser@proteusa01 ~]$ module load myresearch-python [juser@proteusa01 ~]$ pip install numpy

numpy Built-in Tests

To run numpy's built-in tests:

[juser@proteusa01 ~]$ python -c 'import numpy ; numpy.test()'

scipy

site.cfg

While scipy does not make use of FFTW3, for some reason, the build fails if the "[fftw]" section is left out.

[mkl]
library_dirs = /mnt/HA/opt/intel/composer_xe_2019u1/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64
include_dirs = /mnt/HA/opt/intel/composer_xe_2019u1/compilers_and_libraries_2019.1.144/linux/mkl/include
mkl_libs = mkl_rt
lapack_libs =  mkl_lapack95_lp64

[fftw]
library_dirs = /mnt/HA/opt/intel/composer_xe_2019u1/mkl/lib/intel64
libraries = fftw3xc_intel

QIIME

[juser@proteusa01 ~]$ pip install qiime

This requires an older numpy than is current. Please see the QIIME article for details on installing.

biopython

[juser@proteusa01 ~]$ pip install biopython

scikit-learn

[juser@proteusa01 ~]$ pip install scikit-learn

References

[1] numpy-discussion list: fftw supported?