This guide will explain the basics of how to load software when working on our HPC facilities.
On our HPC facilities we have many hundreds of software applications; far more than you would normally have installed on a typical single computer.
Often with the installation of many applications on a single computer we will have problems that arise because of conflicts between applications; different versions of libraries, one application needing a different settings in a configuration file to another application needing the same file!
The Linux Environment Modules software allows us to keep each application we install in its own directory, with its own configuration and not conflict or trample all over another application which may share the same files or data. This allows us to have, for example, multiple versions of Python or Matlab installed at the same time, and not have them conflict.
However, it does mean that not all of those applications are immediately available to us, and we do need to learn a few short commands to make use of them.
First of all, we need to be able to see what software modules are available for us to use. We can do this in one of two ways;
pyth)
The first method, listing everything available is easy; we can use module avail:
$ module avail
ABINIT/8.2.2-intel-2017.03-GCC-6.3
ABINIT/8.4.4-foss-2017b
ABINIT/8.4.4-intel-2017.03-GCC-6.3
ABINIT/8.10.1-intel-2018b
ABINIT/9.4.2-foss-2021b (D)
ABRicate/1.0.0-gompi-2021a
AFNI/20160329-intel-2017.03-GCC-6.3-Python-2.7.12
ALE/1.0.0-foss-2021a
ANSYS/17.0
ANSYS/18.1
ANSYS/19.4
ANSYS/2020-R2
ANSYS/2021
ANSYS/2022-R1
ANSYS/2024R1 (D)
ANTLR/2.7.7-GCCcore-11.2.0-Java-11.0.2
ANTLR/2.7.7-GCCcore-11.3.0-Java-11.0.2 (D)
ANTs/2.3.1-foss-2018b-Python-3.6.6
ANTs/2.3.2-foss-2019b-Python-3.7.4
ANTs/2.3.5-foss-2021a (D)
ATK/2.22.0-intel-2017.03-GCC-6.3
ATK/2.27.1-foss-2017b
ATK/2.28.1-foss-2018b
ATK/2.32.0-GCCcore-8.2.0
ATK/2.36.0-GCCcore-9.3.0
ATK/2.36.0-GCCcore-10.2.0
ATK/2.36.0-GCCcore-10.3.0
ATK/2.36.0-GCCcore-11.2.0
ATK/2.38.0-GCCcore-11.3.0
ATK/2.38.0-GCCcore-12.2.0 (D)
AUGUSTUS/3.3-foss-2017b
AUGUSTUS/3.4.0-foss-2020b
AUGUSTUS/3.4.0-foss-2021b (D)
Advisor/2017_update4
Amber/14-goolf-2017a_parallel
AmberTools/19-intel-2019a
Anaconda2/5.0.1
Anaconda2/2019.10 (D)
Anaconda3/5.0.1
Anaconda3/5.1.0
Anaconda3/5.3.0
Anaconda3/2018.12
Anaconda3/2019.10
Anaconda3/2020.02
Anaconda3/2021.11 (D)
Archive-Zip/1.68-GCCcore-11.2.0
Armadillo/11.4.3-foss-2022b
Armadillo/12.6.2-foss-2023a (D)
Arrow/6.0.0-foss-2021b
...
...
Oh, wait… that's a lot of software packages (we did say that we had hundreds). Whilst that does give us a list of everything, it is going to be hard work finding what we want from that list.
Let's search for the Python package Numpy by name instead by adding that name to the end of our module avail command:
$ module avail numpy
numpy/1.8.2-goolf-2017a-Python-2.7.11 numpy/1.11.1-goolf-2017a-Python-2.7.13
numpy/1.9.2-intel-2017.03-GCC-6.3-Python-2.7.12 numpy/1.12.1-goolf-2017a-Python-3.6.1
numpy/1.9.2-intel-2017.03-GCC-6.3-Python-2.7.13 numpy/1.14.0-foss-2017b-Python-2.7.14
numpy/1.10.1-goolf-1.7.20gcc493-Python-2.7.11 numpy/1.14.0-foss-2017b-Python-3.6.3 (D)
numpy/1.11.1-goolf-1.7.20gcc493-Python-2.7.11
Where:
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$
This shows us all of the versions of Numpy which are installed, as well as which version will be loaded ((D)) if we ask for Numpy without a specific version. Normally this will be the latest version of an application, but you should always check to ensure which version you are going to get.
Sometimes a search will still bring back too many results to be useful. Try the following, again using module avail:
$ module avail python
AFNI/20160329-intel-2017.03-GCC-6.3-Python-2.7.12
ANTs/2.3.1-foss-2018b-Python-3.6.6
ANTs/2.3.2-foss-2019b-Python-3.7.4
Biopython/1.72-foss-2017b-Python-2.7.14
Biopython/1.72-foss-2018b-Python-2.7.15
Biopython/1.78-foss-2020b
Biopython/1.79-foss-2021b
Biopython/1.79-foss-2022a (D)
Blender/2.79b-foss-2018b-Python-3.6.6-CUDA-9.2.88
Blender/2.81-foss-2019b-Python-3.7.4 (D)
Boost.Python/1.67.0-foss-2018b-Python-2.7.15
Boost.Python/1.67.0-foss-2018b-Python-3.6.6
Boost.Python/1.70.0-gompi-2019a
Boost.Python/1.79.0-GCC-11.3.0 (D)
Boost/1.59.0-goolf-2017a-Python-2.7.11
Boost/1.59.0-intel-2017.03-GCC-6.3-Python-2.7.10
Boost/1.59.0-intel-2017.03-GCC-6.3-Python-2.7.11
Boost/1.60.0-intel-2017.03-GCC-6.3-Python-2.7.11
Boost/1.61.0-goolf-2017a-Python-2.7.11
Boost/1.61.0-intel-2017.03-GCC-6.3-Python-2.7.12
Boost/1.63.0-goolf-2017a-Python-2.7.13
Boost/1.63.0-intel-2017.03-GCC-6.3-Python-2.7.13
Boost/1.65.1-foss-2017b-Python-2.7.14
Bottleneck/1.3.2-foss-2020a-Python-3.8.2
CGAL/4.9-intel-2017.03-GCC-6.3-Python-2.7.13
CGAL/4.11-foss-2017b-Python-2.7.14
CGAL/4.11.1-foss-2018b-Python-2.7.15
Cufflinks/2.2.1-intel-2017.03-GCC-6.3-Python-2.7.10-Boost-1.59.0
DLCpar/1.0-foss-2017b-Python-2.7.14
FIAT/1.6.0-goolf-2017a-Python-2.7.11
FIAT/1.6.0-intel-2017.03-GCC-6.3-Python-2.7.11
FIAT/2016.2.0-goolf-2017a-Python-2.7.11
FIAT/2016.2.0-intel-2017.03-GCC-6.3-Python-2.7.12 (D)
FSL/6.0.1-foss-2018b-Python-3.6.6
GATK/4.1.0.0-foss-2018b-Python-3.6.6
GDAL/2.1.3-intel-2017.03-GCC-6.3-Python-2.7.11
GDAL/2.1.3-intel-2017.03-GCC-6.3-Python-2.7.13
GDAL/2.2.3-foss-2017b-Python-2.7.14
GDAL/2.2.3-foss-2017b-Python-3.6.3
GDAL/2.2.3-intel-2018b-Python-3.6.6
GDAL/3.0.0-foss-2019a-Python-3.7.2
GDAL/3.0.2-foss-2019b-Python-3.7.4
GDAL/3.0.4-foss-2020a-Python-3.8.2
GEOS/3.6.1-intel-2017.03-GCC-6.3-Python-2.7.11
GEOS/3.6.1-intel-2017.03-GCC-6.3-Python-2.7.13
GEOS/3.6.2-foss-2017b-Python-2.7.14
GEOS/3.6.2-foss-2017b-Python-3.6.3
GEOS/3.6.2-intel-2018b-Python-3.6.6
GEOS/3.7.2-foss-2019a-Python-3.7.2
GEOS/3.8.0-GCC-8.3.0-Python-3.7.4
GEOS/3.8.1-GCC-9.3.0-Python-3.8.2
GObject-Introspection/1.53.5-foss-2017b-Python-2.7.14
GObject-Introspection/1.54.1-foss-2018b-Python-2.7.15
GObject-Introspection/1.60.1-GCCcore-8.2.0-Python-3.7.2
GObject-Introspection/1.63.1-GCCcore-9.3.0-Python-3.8.2
GObject-Introspection/1.64.0-GCCcore-9.3.0-Python-3.8.2
GRASS/7.8.3-foss-2020a-Python-3.8.2
GROMACS/2020.5-foss-2020b-Python-3.8.6
...
...
We still have too many results to find what we want. In this case you can pass regular expressions to the module command, by adding the -r flag. Here's an example searching for all modules that end in the string 3.8.2 using module -r avail:
$ module -r avail 3.8.2$
Bottleneck/1.3.2-foss-2020a-Python-3.8.2
GDAL/3.0.4-foss-2020a-Python-3.8.2
GEOS/3.8.1-GCC-9.3.0-Python-3.8.2
GObject-Introspection/1.63.1-GCCcore-9.3.0-Python-3.8.2
GObject-Introspection/1.64.0-GCCcore-9.3.0-Python-3.8.2
GRASS/7.8.3-foss-2020a-Python-3.8.2
Meson/0.53.2-GCCcore-9.3.0-Python-3.8.2
Meson/0.55.1-GCCcore-9.3.0-Python-3.8.2
MultiQC/1.9-foss-2020a-Python-3.8.2 (D)
OpenCV/4.2.0-foss-2020a-Python-3.8.2
PETSc/3.12.4-foss-2020a-Python-3.8.2
PLUMED/2.6.0-foss-2020a-Python-3.8.2 (D)
Pillow/7.0.0-GCCcore-9.3.0-Python-3.8.2
QUAST/5.0.2-foss-2020a-Python-3.8.2
SLEPc/3.12.2-foss-2020a-Python-3.8.2
SciPy-bundle/2020.03-foss-2020a-Python-3.8.2
TensorFlow/2.3.1-foss-2020a-Python-3.8.2
VTK/8.2.0-foss-2020a-Python-3.8.2
archspec/0.1.0-GCCcore-9.3.0-Python-3.8.2
h5py/2.10.0-foss-2020a-Python-3.8.2
libspatialite/4.3.0a-foss-2020a-Python-3.8.2
libspatialite/4.3.0a-foss-2020b-Python-3.8.2
matplotlib/3.2.1-foss-2020a-Python-3.8.2
molmod/1.4.5-foss-2020a-Python-3.8.2
networkx/2.4-foss-2020a-Python-3.8.2
numexpr/2.7.1-foss-2020a-Python-3.8.2
pandas/1.1.2-foss-2020a-Python-3.8.2 (D)
pkgconfig/1.5.1-GCCcore-9.3.0-Python-3.8.2
protobuf-python/3.13.0-foss-2020a-Python-3.8.2
pyEGA3/3.1.0-GCCcore-9.3.0-Python-3.8.2
pyEGA3/3.4.0-GCCcore-9.3.0-Python-3.8.2
pybind11/2.4.3-GCCcore-9.3.0-Python-3.8.2
statsmodels/0.11.1-foss-2020a-Python-3.8.2
wxPython/4.0.7.post2-GCC-9.3.0-Python-3.8.2
yaff/1.6.0-foss-2020a-Python-3.8.2
$
Once you have found the software module you need to use, you use the module load command to load it.
Let us assume we need to have Python available. Before we start, check which version of Python is available, if any:
$ which python
/usr/bin/python
$ python -V
Python 2.7.5
$
So it looks like without any modules loaded, the system provides Python 2.7.5 for us.
Warning!
We don't want to use the version of Python which came with the operating system! In fact it is dangerous to assume that this version (or any other system-provided application) will be available on every node in the HPC facility; we may very well have different nodes with different operating system versions and different software loaded. Wherever possible, load software from modules instead, this will mean your versions will always be consistent across every node.
To load the Python module, and (temporarily) replace the system provided version, we use the module load command:
$ module load Python
$ which python
/mnt/storage/apps/eb/software/Python/3.7.0-foss-2018b/bin/python
$ python -V
Python 3.7.0
$
So it looks like the default version of Python which loads from the modules collection is Python 3.7.0. What if we don't want that version? Well we can refer back to the output of module avail Python and load a different version instead:
$ module avail Python
...
...
Python/2.7.10-intel-2017.03-GCC-6.3
Python/2.7.11-goolf-1.7.20gcc493
Python/2.7.11-goolf-2017a
Python/2.7.11-intel-2017.03-GCC-6.3
Python/2.7.12-goolf-2017a
Python/2.7.12-intel-2017.03-GCC-6.3-bare
Python/2.7.12-intel-2017.03-GCC-6.3
Python/2.7.13-goolf-2017a
Python/2.7.13-intel-2017.03-GCC-6.3
Python/2.7.14-foss-2017b-bare
Python/2.7.14-foss-2017b
Python/2.7.15-foss-2018b
Python/2.7.15-GCCcore-7.3.0-bare
Python/2.7.15-GCCcore-8.2.0
Python/2.7.15-intel-2018b
Python/2.7.16-GCCcore-8.3.0
Python/2.7.18-GCCcore-9.3.0
Python/2.7.18-GCCcore-10.2.0
Python/2.7.18-GCCcore-10.3.0-bare
Python/2.7.18-GCCcore-11.2.0-bare
Python/2.7.18-GCCcore-11.3.0-bare
Python/3.6.1-goolf-2017a
Python/3.6.1-intel-2017.03-GCC-6.3
Python/3.6.3-foss-2017b
Python/3.6.6-foss-2018b
Python/3.6.6-intel-2018b
Python/3.7.0-foss-2018b (D)
Python/3.7.0-intel-2018b
Python/3.7.2-GCCcore-8.2.0
Python/3.7.4-GCCcore-8.3.0
Python/3.7.5-GCCcore-8.3.0
Python/3.8.2-GCCcore-9.3.0
Python/3.8.6-GCCcore-10.2.0
Python/3.9.5-GCCcore-10.3.0-bare
Python/3.9.5-GCCcore-10.3.0
Python/3.9.6-GCCcore-11.2.0-bare
Python/3.9.6-GCCcore-11.2.0
Python/3.10.4-GCCcore-11.3.0-bare
Python/3.10.4-GCCcore-11.3.0
Python/3.10.8-GCCcore-12.2.0-bare
Python/3.10.8-GCCcore-12.2.0
Python/3.11.3-GCCcore-12.3.0
...
...
Note: Output from module avail Python truncated for brevity.
We can see that version Python/3.7.0-foss-2018b is the default, and that matches what we got before. But there are other versions, let's try loading a more up to date version, again using module load, but now also adding the version number:
$ module load Python/3.11.3-GCCcore-12.3.0
$ which python
/mnt/storage/apps/eb/software/Python/3.11.3-GCCcore-12.3.0/bin/python
$ Python -V
Python 3.11.3
$
That's much better. We can add the optional version number with the / between the package name and we get that specific version loaded instead.
Assuming we are finished with the version of Python we loaded before, or we need to change to a different one, we can do that easily with module unload:
$ which python
/mnt/storage/apps/eb/software/Python/3.11.3-GCCcore-12.3.0/bin/python
$ module unload Python
$ which python
/usr/bin/python
$
This removes the Python module, and all references to it from our active session. We are back to the normal system Python in this case.
Note: You do not need to specify the version number on the end of the module - in 99.99% of cases you can only have one module of the same name loaded at any time.
If you have many different modules loaded you can unload all of them with a single module purge command:
$ module purge
$
| Command | Purpose | Example |
|---|---|---|
| module avail | Lists available modules | module avail |
| module avail STRING | Lists available modules that contain the case-insensitive string STRING | module avail python |
| module -r avail REGEX | Lists available modules that match the regular expression REGEX | module -r avail 3.8.2$ |
| module spider STRING | Lists module information, description and versions for all modules or text descriptions which contain the name STRING | module spider gcc |
| module load STRING | Load a single module with the exact name STRING | module load Python |
| module load STRING/VERSION | Load a single module with the exact name STRING and version VERSION | module load Python/3.12.8 |
| module list | Lists all modules which are currently loaded | |
| module unload STRING | Unload a single loaded module with the name STRING | module unload Python |
| module purge | Unload all loaded modules |