Introduction to Software Modules

This guide will explain the basics of how to load software when working on our HPC facilities.

Linux Environment Modules

On our HPC facilities we have many hundreds of software applications; far more than you would normally have installed on a typical single computer.

Often with the installation of many applications on a single computer we will have problems that arise because of conflicts between applications; different versions of libraries, one application needing a different settings in a configuration file to another application needing the same file!

The Linux Environment Modules software allows us to keep each application we install in its own directory, with its own configuration and not conflict or trample all over another application which may share the same files or data. This allows us to have, for example, multiple versions of Python or Matlab installed at the same time, and not have them conflict.

However, it does mean that not all of those applications are immediately available to us, and we do need to learn a few short commands to make use of them.

Searching For Software Modules

First of all, we need to be able to see what software modules are available for us to use. We can do this in one of two ways;

  • Simply list everything that is available
  • Search for an application by name, or by partial text string (e.g. finding all versions of Python by searching just for pyth)

The first method, listing everything available is easy; we can use module avail: 

$ module avail
   ABINIT/8.2.2-intel-2017.03-GCC-6.3
   ABINIT/8.4.4-foss-2017b
   ABINIT/8.4.4-intel-2017.03-GCC-6.3
   ABINIT/8.10.1-intel-2018b
   ABINIT/9.4.2-foss-2021b                                          (D)
   ABRicate/1.0.0-gompi-2021a
   AFNI/20160329-intel-2017.03-GCC-6.3-Python-2.7.12
   ALE/1.0.0-foss-2021a
   ANSYS/17.0
   ANSYS/18.1
   ANSYS/19.4
   ANSYS/2020-R2
   ANSYS/2021
   ANSYS/2022-R1
   ANSYS/2024R1                                                     (D)
   ANTLR/2.7.7-GCCcore-11.2.0-Java-11.0.2
   ANTLR/2.7.7-GCCcore-11.3.0-Java-11.0.2                           (D)
   ANTs/2.3.1-foss-2018b-Python-3.6.6
   ANTs/2.3.2-foss-2019b-Python-3.7.4
   ANTs/2.3.5-foss-2021a                                            (D)
   ATK/2.22.0-intel-2017.03-GCC-6.3
   ATK/2.27.1-foss-2017b
   ATK/2.28.1-foss-2018b
   ATK/2.32.0-GCCcore-8.2.0
   ATK/2.36.0-GCCcore-9.3.0
   ATK/2.36.0-GCCcore-10.2.0
   ATK/2.36.0-GCCcore-10.3.0
   ATK/2.36.0-GCCcore-11.2.0
   ATK/2.38.0-GCCcore-11.3.0
   ATK/2.38.0-GCCcore-12.2.0                                        (D)
   AUGUSTUS/3.3-foss-2017b
   AUGUSTUS/3.4.0-foss-2020b
   AUGUSTUS/3.4.0-foss-2021b                                        (D)
   Advisor/2017_update4
   Amber/14-goolf-2017a_parallel
   AmberTools/19-intel-2019a
   Anaconda2/5.0.1
   Anaconda2/2019.10                                                (D)
   Anaconda3/5.0.1
   Anaconda3/5.1.0
   Anaconda3/5.3.0
   Anaconda3/2018.12
   Anaconda3/2019.10
   Anaconda3/2020.02
   Anaconda3/2021.11                                                (D)
   Archive-Zip/1.68-GCCcore-11.2.0
   Armadillo/11.4.3-foss-2022b
   Armadillo/12.6.2-foss-2023a                                      (D)
   Arrow/6.0.0-foss-2021b
...
...

Oh, wait… that's a lot of software packages (we did say that we had hundreds). Whilst that does give us a list of everything, it is going to be hard work finding what we want from that list. Let's search for the Python package Numpy by name instead by adding that name to the end of our module avail command: 

$ module avail numpy
   numpy/1.8.2-goolf-2017a-Python-2.7.11              numpy/1.11.1-goolf-2017a-Python-2.7.13
   numpy/1.9.2-intel-2017.03-GCC-6.3-Python-2.7.12    numpy/1.12.1-goolf-2017a-Python-3.6.1
   numpy/1.9.2-intel-2017.03-GCC-6.3-Python-2.7.13    numpy/1.14.0-foss-2017b-Python-2.7.14
   numpy/1.10.1-goolf-1.7.20gcc493-Python-2.7.11      numpy/1.14.0-foss-2017b-Python-3.6.3   (D)
   numpy/1.11.1-goolf-1.7.20gcc493-Python-2.7.11

  Where:
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$

This shows us all of the versions of Numpy which are installed, as well as which version will be loaded ((D)) if we ask for Numpy without a specific version. Normally this will be the latest version of an application, but you should always check to ensure which version you are going to get.

Sometimes a search will still bring back too many results to be useful. Try the following, again using module avail: 

$ module avail python
   AFNI/20160329-intel-2017.03-GCC-6.3-Python-2.7.12
   ANTs/2.3.1-foss-2018b-Python-3.6.6
   ANTs/2.3.2-foss-2019b-Python-3.7.4
   Biopython/1.72-foss-2017b-Python-2.7.14
   Biopython/1.72-foss-2018b-Python-2.7.15
   Biopython/1.78-foss-2020b
   Biopython/1.79-foss-2021b
   Biopython/1.79-foss-2022a                                        (D)
   Blender/2.79b-foss-2018b-Python-3.6.6-CUDA-9.2.88
   Blender/2.81-foss-2019b-Python-3.7.4                             (D)
   Boost.Python/1.67.0-foss-2018b-Python-2.7.15
   Boost.Python/1.67.0-foss-2018b-Python-3.6.6
   Boost.Python/1.70.0-gompi-2019a
   Boost.Python/1.79.0-GCC-11.3.0                                   (D)
   Boost/1.59.0-goolf-2017a-Python-2.7.11
   Boost/1.59.0-intel-2017.03-GCC-6.3-Python-2.7.10
   Boost/1.59.0-intel-2017.03-GCC-6.3-Python-2.7.11
   Boost/1.60.0-intel-2017.03-GCC-6.3-Python-2.7.11
   Boost/1.61.0-goolf-2017a-Python-2.7.11
   Boost/1.61.0-intel-2017.03-GCC-6.3-Python-2.7.12
   Boost/1.63.0-goolf-2017a-Python-2.7.13
   Boost/1.63.0-intel-2017.03-GCC-6.3-Python-2.7.13
   Boost/1.65.1-foss-2017b-Python-2.7.14
   Bottleneck/1.3.2-foss-2020a-Python-3.8.2
   CGAL/4.9-intel-2017.03-GCC-6.3-Python-2.7.13
   CGAL/4.11-foss-2017b-Python-2.7.14
   CGAL/4.11.1-foss-2018b-Python-2.7.15
   Cufflinks/2.2.1-intel-2017.03-GCC-6.3-Python-2.7.10-Boost-1.59.0
   DLCpar/1.0-foss-2017b-Python-2.7.14
   FIAT/1.6.0-goolf-2017a-Python-2.7.11
   FIAT/1.6.0-intel-2017.03-GCC-6.3-Python-2.7.11
   FIAT/2016.2.0-goolf-2017a-Python-2.7.11
   FIAT/2016.2.0-intel-2017.03-GCC-6.3-Python-2.7.12                (D)
   FSL/6.0.1-foss-2018b-Python-3.6.6
   GATK/4.1.0.0-foss-2018b-Python-3.6.6
   GDAL/2.1.3-intel-2017.03-GCC-6.3-Python-2.7.11
   GDAL/2.1.3-intel-2017.03-GCC-6.3-Python-2.7.13
   GDAL/2.2.3-foss-2017b-Python-2.7.14
   GDAL/2.2.3-foss-2017b-Python-3.6.3
   GDAL/2.2.3-intel-2018b-Python-3.6.6
   GDAL/3.0.0-foss-2019a-Python-3.7.2
   GDAL/3.0.2-foss-2019b-Python-3.7.4
   GDAL/3.0.4-foss-2020a-Python-3.8.2
   GEOS/3.6.1-intel-2017.03-GCC-6.3-Python-2.7.11
   GEOS/3.6.1-intel-2017.03-GCC-6.3-Python-2.7.13
   GEOS/3.6.2-foss-2017b-Python-2.7.14
   GEOS/3.6.2-foss-2017b-Python-3.6.3
   GEOS/3.6.2-intel-2018b-Python-3.6.6
   GEOS/3.7.2-foss-2019a-Python-3.7.2
   GEOS/3.8.0-GCC-8.3.0-Python-3.7.4
   GEOS/3.8.1-GCC-9.3.0-Python-3.8.2
   GObject-Introspection/1.53.5-foss-2017b-Python-2.7.14
   GObject-Introspection/1.54.1-foss-2018b-Python-2.7.15
   GObject-Introspection/1.60.1-GCCcore-8.2.0-Python-3.7.2
   GObject-Introspection/1.63.1-GCCcore-9.3.0-Python-3.8.2
   GObject-Introspection/1.64.0-GCCcore-9.3.0-Python-3.8.2
   GRASS/7.8.3-foss-2020a-Python-3.8.2
   GROMACS/2020.5-foss-2020b-Python-3.8.6
   ...
   ...

We still have too many results to find what we want. In this case you can pass regular expressions to the module command, by adding the -r flag. Here's an example searching for all modules that end in the string 3.8.2 using module -r avail: 

$ module -r avail 3.8.2$
   Bottleneck/1.3.2-foss-2020a-Python-3.8.2
   GDAL/3.0.4-foss-2020a-Python-3.8.2
   GEOS/3.8.1-GCC-9.3.0-Python-3.8.2
   GObject-Introspection/1.63.1-GCCcore-9.3.0-Python-3.8.2
   GObject-Introspection/1.64.0-GCCcore-9.3.0-Python-3.8.2
   GRASS/7.8.3-foss-2020a-Python-3.8.2
   Meson/0.53.2-GCCcore-9.3.0-Python-3.8.2
   Meson/0.55.1-GCCcore-9.3.0-Python-3.8.2
   MultiQC/1.9-foss-2020a-Python-3.8.2                     (D)
   OpenCV/4.2.0-foss-2020a-Python-3.8.2
   PETSc/3.12.4-foss-2020a-Python-3.8.2
   PLUMED/2.6.0-foss-2020a-Python-3.8.2                    (D)
   Pillow/7.0.0-GCCcore-9.3.0-Python-3.8.2
   QUAST/5.0.2-foss-2020a-Python-3.8.2
   SLEPc/3.12.2-foss-2020a-Python-3.8.2
   SciPy-bundle/2020.03-foss-2020a-Python-3.8.2
   TensorFlow/2.3.1-foss-2020a-Python-3.8.2
   VTK/8.2.0-foss-2020a-Python-3.8.2
   archspec/0.1.0-GCCcore-9.3.0-Python-3.8.2
   h5py/2.10.0-foss-2020a-Python-3.8.2
   libspatialite/4.3.0a-foss-2020a-Python-3.8.2
   libspatialite/4.3.0a-foss-2020b-Python-3.8.2
   matplotlib/3.2.1-foss-2020a-Python-3.8.2
   molmod/1.4.5-foss-2020a-Python-3.8.2
   networkx/2.4-foss-2020a-Python-3.8.2
   numexpr/2.7.1-foss-2020a-Python-3.8.2
   pandas/1.1.2-foss-2020a-Python-3.8.2                    (D)
   pkgconfig/1.5.1-GCCcore-9.3.0-Python-3.8.2
   protobuf-python/3.13.0-foss-2020a-Python-3.8.2
   pyEGA3/3.1.0-GCCcore-9.3.0-Python-3.8.2
   pyEGA3/3.4.0-GCCcore-9.3.0-Python-3.8.2
   pybind11/2.4.3-GCCcore-9.3.0-Python-3.8.2
   statsmodels/0.11.1-foss-2020a-Python-3.8.2
   wxPython/4.0.7.post2-GCC-9.3.0-Python-3.8.2
   yaff/1.6.0-foss-2020a-Python-3.8.2
$

Loading Software Modules

Once you have found the software module you need to use, you use the module load command to load it.

Let us assume we need to have Python available. Before we start, check which version of Python is available, if any: 

$ which python
/usr/bin/python
$ python -V
Python 2.7.5
$

So it looks like without any modules loaded, the system provides Python 2.7.5 for us.

Warning!

We don't want to use the version of Python which came with the operating system! In fact it is dangerous to assume that this version (or any other system-provided application) will be available on every node in the HPC facility; we may very well have different nodes with different operating system versions and different software loaded. Wherever possible, load software from modules instead, this will mean your versions will always be consistent across every node.

To load the Python module, and (temporarily) replace the system provided version, we use the module load command: 

$ module load Python
$ which python
/mnt/storage/apps/eb/software/Python/3.7.0-foss-2018b/bin/python
$ python -V
Python 3.7.0
$

So it looks like the default version of Python which loads from the modules collection is Python 3.7.0. What if we don't want that version? Well we can refer back to the output of module avail Python and load a different version instead: 

$ module avail Python
...
...
   Python/2.7.10-intel-2017.03-GCC-6.3
   Python/2.7.11-goolf-1.7.20gcc493
   Python/2.7.11-goolf-2017a
   Python/2.7.11-intel-2017.03-GCC-6.3
   Python/2.7.12-goolf-2017a
   Python/2.7.12-intel-2017.03-GCC-6.3-bare
   Python/2.7.12-intel-2017.03-GCC-6.3
   Python/2.7.13-goolf-2017a
   Python/2.7.13-intel-2017.03-GCC-6.3
   Python/2.7.14-foss-2017b-bare
   Python/2.7.14-foss-2017b
   Python/2.7.15-foss-2018b
   Python/2.7.15-GCCcore-7.3.0-bare
   Python/2.7.15-GCCcore-8.2.0
   Python/2.7.15-intel-2018b
   Python/2.7.16-GCCcore-8.3.0
   Python/2.7.18-GCCcore-9.3.0
   Python/2.7.18-GCCcore-10.2.0
   Python/2.7.18-GCCcore-10.3.0-bare
   Python/2.7.18-GCCcore-11.2.0-bare
   Python/2.7.18-GCCcore-11.3.0-bare
   Python/3.6.1-goolf-2017a
   Python/3.6.1-intel-2017.03-GCC-6.3
   Python/3.6.3-foss-2017b
   Python/3.6.6-foss-2018b
   Python/3.6.6-intel-2018b
   Python/3.7.0-foss-2018b                                          (D)
   Python/3.7.0-intel-2018b
   Python/3.7.2-GCCcore-8.2.0
   Python/3.7.4-GCCcore-8.3.0
   Python/3.7.5-GCCcore-8.3.0
   Python/3.8.2-GCCcore-9.3.0
   Python/3.8.6-GCCcore-10.2.0
   Python/3.9.5-GCCcore-10.3.0-bare
   Python/3.9.5-GCCcore-10.3.0
   Python/3.9.6-GCCcore-11.2.0-bare
   Python/3.9.6-GCCcore-11.2.0
   Python/3.10.4-GCCcore-11.3.0-bare
   Python/3.10.4-GCCcore-11.3.0
   Python/3.10.8-GCCcore-12.2.0-bare
   Python/3.10.8-GCCcore-12.2.0
   Python/3.11.3-GCCcore-12.3.0
...
...

Note: Output from module avail Python truncated for brevity.

We can see that version Python/3.7.0-foss-2018b is the default, and that matches what we got before. But there are other versions, let's try loading a more up to date version, again using module load, but now also adding the version number: 

$ module load Python/3.11.3-GCCcore-12.3.0
$ which python
/mnt/storage/apps/eb/software/Python/3.11.3-GCCcore-12.3.0/bin/python
$ Python -V
Python 3.11.3
$

That's much better. We can add the optional version number with the / between the package name and we get that specific version loaded instead.

Unloading Software Modules

Assuming we are finished with the version of Python we loaded before, or we need to change to a different one, we can do that easily with module unload: 

$ which python
/mnt/storage/apps/eb/software/Python/3.11.3-GCCcore-12.3.0/bin/python
$ module unload Python
$ which python
/usr/bin/python
$

This removes the Python module, and all references to it from our active session. We are back to the normal system Python in this case.

Note: You do not need to specify the version number on the end of the module - in 99.99% of cases you can only have one module of the same name loaded at any time.

If you have many different modules loaded you can unload all of them with a single module purge command: 

$ module purge
$

Quick Reference

Command Purpose Example
module avail Lists available modules module avail
module avail STRING Lists available modules that contain the case-insensitive string STRING module avail python
module -r avail REGEX Lists available modules that match the regular expression REGEX module -r avail 3.8.2$
module spider STRING Lists module information, description and versions for all modules or text descriptions which contain the name STRING module spider gcc
module load STRING Load a single module with the exact name STRING module load Python
module load STRING/VERSION Load a single module with the exact name STRING and version VERSION module load Python/3.12.8
module list Lists all modules which are currently loaded
module unload STRING Unload a single loaded module with the name STRING module unload Python
module purge Unload all loaded modules

Back to Getting Started