====== Introduction to Software Modules ======
This guide will explain the basics of how to load software when working on our HPC facilities.
* [[#Linux Environment Modules]]
* [[#Searching For Software Modules]]
* [[#Loading Software Modules]]
* [[#Unloading Software Modules]]
* [[#Quick Reference]]
----
===== Linux Environment Modules =====
On our HPC facilities we have many hundreds of software applications; //far// more than you would normally have installed on a typical single computer.
Often with the installation of many applications on a single computer we will have problems that arise because of conflicts between applications; different versions of libraries, one application needing a different settings in a configuration file to another application needing the same file!
The [[https://modules.sourceforge.net/|Linux Environment Modules]] software allows us to keep each application we install in its own directory, with its own configuration and not conflict or trample all over another application which may share the same files or data. This allows us to have, for example, multiple versions of Python or Matlab installed at the same time, and not have them conflict.
However, it does mean that not all of those applications are immediately available to us, and we do need to learn a few short commands to make use of them.
----
==== Searching For Software Modules ====
First of all, we need to be able to see what software modules are available for us to use. We can do this in one of two ways;
* Simply list everything that is available
* Search for an application by name, or by partial text string (e.g. finding all versions of Python by searching just for ''pyth'')
The first method, //listing everything available// is easy; we can use ''module avail'':
$ module avail
ABINIT/8.2.2-intel-2017.03-GCC-6.3
ABINIT/8.4.4-foss-2017b
ABINIT/8.4.4-intel-2017.03-GCC-6.3
ABINIT/8.10.1-intel-2018b
ABINIT/9.4.2-foss-2021b (D)
ABRicate/1.0.0-gompi-2021a
AFNI/20160329-intel-2017.03-GCC-6.3-Python-2.7.12
ALE/1.0.0-foss-2021a
ANSYS/17.0
ANSYS/18.1
ANSYS/19.4
ANSYS/2020-R2
ANSYS/2021
ANSYS/2022-R1
ANSYS/2024R1 (D)
ANTLR/2.7.7-GCCcore-11.2.0-Java-11.0.2
ANTLR/2.7.7-GCCcore-11.3.0-Java-11.0.2 (D)
ANTs/2.3.1-foss-2018b-Python-3.6.6
ANTs/2.3.2-foss-2019b-Python-3.7.4
ANTs/2.3.5-foss-2021a (D)
ATK/2.22.0-intel-2017.03-GCC-6.3
ATK/2.27.1-foss-2017b
ATK/2.28.1-foss-2018b
ATK/2.32.0-GCCcore-8.2.0
ATK/2.36.0-GCCcore-9.3.0
ATK/2.36.0-GCCcore-10.2.0
ATK/2.36.0-GCCcore-10.3.0
ATK/2.36.0-GCCcore-11.2.0
ATK/2.38.0-GCCcore-11.3.0
ATK/2.38.0-GCCcore-12.2.0 (D)
AUGUSTUS/3.3-foss-2017b
AUGUSTUS/3.4.0-foss-2020b
AUGUSTUS/3.4.0-foss-2021b (D)
Advisor/2017_update4
Amber/14-goolf-2017a_parallel
AmberTools/19-intel-2019a
Anaconda2/5.0.1
Anaconda2/2019.10 (D)
Anaconda3/5.0.1
Anaconda3/5.1.0
Anaconda3/5.3.0
Anaconda3/2018.12
Anaconda3/2019.10
Anaconda3/2020.02
Anaconda3/2021.11 (D)
Archive-Zip/1.68-GCCcore-11.2.0
Armadillo/11.4.3-foss-2022b
Armadillo/12.6.2-foss-2023a (D)
Arrow/6.0.0-foss-2021b
...
...
Oh, wait... that's a __lot__ of software packages (we did say that we had //hundreds//). Whilst that does give us a list of everything, it is going to be hard work finding what we want from that list.
Let's search for the Python package **Numpy** by name instead by adding that name to the end of our ''module avail'' command:
$ module avail numpy
numpy/1.8.2-goolf-2017a-Python-2.7.11 numpy/1.11.1-goolf-2017a-Python-2.7.13
numpy/1.9.2-intel-2017.03-GCC-6.3-Python-2.7.12 numpy/1.12.1-goolf-2017a-Python-3.6.1
numpy/1.9.2-intel-2017.03-GCC-6.3-Python-2.7.13 numpy/1.14.0-foss-2017b-Python-2.7.14
numpy/1.10.1-goolf-1.7.20gcc493-Python-2.7.11 numpy/1.14.0-foss-2017b-Python-3.6.3 (D)
numpy/1.11.1-goolf-1.7.20gcc493-Python-2.7.11
Where:
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$
This shows us all of the versions of **Numpy** which are installed, as well as which version will be loaded (**(D)**) if we ask for **Numpy** without a specific version. Normally this will be the //latest// version of an application, but you should always check to ensure which version you are going to get.
Sometimes a search will //still// bring back too many results to be useful. Try the following, again using ''module avail'':
$ module avail python
AFNI/20160329-intel-2017.03-GCC-6.3-Python-2.7.12
ANTs/2.3.1-foss-2018b-Python-3.6.6
ANTs/2.3.2-foss-2019b-Python-3.7.4
Biopython/1.72-foss-2017b-Python-2.7.14
Biopython/1.72-foss-2018b-Python-2.7.15
Biopython/1.78-foss-2020b
Biopython/1.79-foss-2021b
Biopython/1.79-foss-2022a (D)
Blender/2.79b-foss-2018b-Python-3.6.6-CUDA-9.2.88
Blender/2.81-foss-2019b-Python-3.7.4 (D)
Boost.Python/1.67.0-foss-2018b-Python-2.7.15
Boost.Python/1.67.0-foss-2018b-Python-3.6.6
Boost.Python/1.70.0-gompi-2019a
Boost.Python/1.79.0-GCC-11.3.0 (D)
Boost/1.59.0-goolf-2017a-Python-2.7.11
Boost/1.59.0-intel-2017.03-GCC-6.3-Python-2.7.10
Boost/1.59.0-intel-2017.03-GCC-6.3-Python-2.7.11
Boost/1.60.0-intel-2017.03-GCC-6.3-Python-2.7.11
Boost/1.61.0-goolf-2017a-Python-2.7.11
Boost/1.61.0-intel-2017.03-GCC-6.3-Python-2.7.12
Boost/1.63.0-goolf-2017a-Python-2.7.13
Boost/1.63.0-intel-2017.03-GCC-6.3-Python-2.7.13
Boost/1.65.1-foss-2017b-Python-2.7.14
Bottleneck/1.3.2-foss-2020a-Python-3.8.2
CGAL/4.9-intel-2017.03-GCC-6.3-Python-2.7.13
CGAL/4.11-foss-2017b-Python-2.7.14
CGAL/4.11.1-foss-2018b-Python-2.7.15
Cufflinks/2.2.1-intel-2017.03-GCC-6.3-Python-2.7.10-Boost-1.59.0
DLCpar/1.0-foss-2017b-Python-2.7.14
FIAT/1.6.0-goolf-2017a-Python-2.7.11
FIAT/1.6.0-intel-2017.03-GCC-6.3-Python-2.7.11
FIAT/2016.2.0-goolf-2017a-Python-2.7.11
FIAT/2016.2.0-intel-2017.03-GCC-6.3-Python-2.7.12 (D)
FSL/6.0.1-foss-2018b-Python-3.6.6
GATK/4.1.0.0-foss-2018b-Python-3.6.6
GDAL/2.1.3-intel-2017.03-GCC-6.3-Python-2.7.11
GDAL/2.1.3-intel-2017.03-GCC-6.3-Python-2.7.13
GDAL/2.2.3-foss-2017b-Python-2.7.14
GDAL/2.2.3-foss-2017b-Python-3.6.3
GDAL/2.2.3-intel-2018b-Python-3.6.6
GDAL/3.0.0-foss-2019a-Python-3.7.2
GDAL/3.0.2-foss-2019b-Python-3.7.4
GDAL/3.0.4-foss-2020a-Python-3.8.2
GEOS/3.6.1-intel-2017.03-GCC-6.3-Python-2.7.11
GEOS/3.6.1-intel-2017.03-GCC-6.3-Python-2.7.13
GEOS/3.6.2-foss-2017b-Python-2.7.14
GEOS/3.6.2-foss-2017b-Python-3.6.3
GEOS/3.6.2-intel-2018b-Python-3.6.6
GEOS/3.7.2-foss-2019a-Python-3.7.2
GEOS/3.8.0-GCC-8.3.0-Python-3.7.4
GEOS/3.8.1-GCC-9.3.0-Python-3.8.2
GObject-Introspection/1.53.5-foss-2017b-Python-2.7.14
GObject-Introspection/1.54.1-foss-2018b-Python-2.7.15
GObject-Introspection/1.60.1-GCCcore-8.2.0-Python-3.7.2
GObject-Introspection/1.63.1-GCCcore-9.3.0-Python-3.8.2
GObject-Introspection/1.64.0-GCCcore-9.3.0-Python-3.8.2
GRASS/7.8.3-foss-2020a-Python-3.8.2
GROMACS/2020.5-foss-2020b-Python-3.8.6
...
...
We //still// have too many results to find what we want. In this case you can pass regular expressions to the module command, by adding the ''-r'' flag. Here's an example searching for all modules that //end// in the string **3.8.2** using ''module -r avail'':
$ module -r avail 3.8.2$
Bottleneck/1.3.2-foss-2020a-Python-3.8.2
GDAL/3.0.4-foss-2020a-Python-3.8.2
GEOS/3.8.1-GCC-9.3.0-Python-3.8.2
GObject-Introspection/1.63.1-GCCcore-9.3.0-Python-3.8.2
GObject-Introspection/1.64.0-GCCcore-9.3.0-Python-3.8.2
GRASS/7.8.3-foss-2020a-Python-3.8.2
Meson/0.53.2-GCCcore-9.3.0-Python-3.8.2
Meson/0.55.1-GCCcore-9.3.0-Python-3.8.2
MultiQC/1.9-foss-2020a-Python-3.8.2 (D)
OpenCV/4.2.0-foss-2020a-Python-3.8.2
PETSc/3.12.4-foss-2020a-Python-3.8.2
PLUMED/2.6.0-foss-2020a-Python-3.8.2 (D)
Pillow/7.0.0-GCCcore-9.3.0-Python-3.8.2
QUAST/5.0.2-foss-2020a-Python-3.8.2
SLEPc/3.12.2-foss-2020a-Python-3.8.2
SciPy-bundle/2020.03-foss-2020a-Python-3.8.2
TensorFlow/2.3.1-foss-2020a-Python-3.8.2
VTK/8.2.0-foss-2020a-Python-3.8.2
archspec/0.1.0-GCCcore-9.3.0-Python-3.8.2
h5py/2.10.0-foss-2020a-Python-3.8.2
libspatialite/4.3.0a-foss-2020a-Python-3.8.2
libspatialite/4.3.0a-foss-2020b-Python-3.8.2
matplotlib/3.2.1-foss-2020a-Python-3.8.2
molmod/1.4.5-foss-2020a-Python-3.8.2
networkx/2.4-foss-2020a-Python-3.8.2
numexpr/2.7.1-foss-2020a-Python-3.8.2
pandas/1.1.2-foss-2020a-Python-3.8.2 (D)
pkgconfig/1.5.1-GCCcore-9.3.0-Python-3.8.2
protobuf-python/3.13.0-foss-2020a-Python-3.8.2
pyEGA3/3.1.0-GCCcore-9.3.0-Python-3.8.2
pyEGA3/3.4.0-GCCcore-9.3.0-Python-3.8.2
pybind11/2.4.3-GCCcore-9.3.0-Python-3.8.2
statsmodels/0.11.1-foss-2020a-Python-3.8.2
wxPython/4.0.7.post2-GCC-9.3.0-Python-3.8.2
yaff/1.6.0-foss-2020a-Python-3.8.2
$
----
==== Loading Software Modules ====
Once you have found the software module you need to use, you use the ''module load'' command to load it.
Let us assume we need to have Python available. Before we start, check which version of Python is available, if any:
$ which python
/usr/bin/python
$ python -V
Python 2.7.5
$
So it looks like without any modules loaded, the system provides Python 2.7.5 for us.
**Warning!**
We //don't// want to use the version of Python which came with the operating system! In fact __it is dangerous__ to assume that this version (or any other system-provided application) will be available on every node in the HPC facility; we may very well have __different__ nodes with __different operating__ system versions and __different software__ loaded. **Wherever possible, load software from modules** instead, this will mean your versions will always be consistent across every node.
To load the Python module, and (temporarily) replace the system provided version, we use the ''module load'' command:
$ module load Python
$ which python
/mnt/storage/apps/eb/software/Python/3.7.0-foss-2018b/bin/python
$ python -V
Python 3.7.0
$
So it looks like the default version of Python which loads from the modules collection is Python 3.7.0. What if we don't want that version? Well we can refer back to the output of //module avail Python// and load a different version instead:
$ module avail Python
...
...
Python/2.7.10-intel-2017.03-GCC-6.3
Python/2.7.11-goolf-1.7.20gcc493
Python/2.7.11-goolf-2017a
Python/2.7.11-intel-2017.03-GCC-6.3
Python/2.7.12-goolf-2017a
Python/2.7.12-intel-2017.03-GCC-6.3-bare
Python/2.7.12-intel-2017.03-GCC-6.3
Python/2.7.13-goolf-2017a
Python/2.7.13-intel-2017.03-GCC-6.3
Python/2.7.14-foss-2017b-bare
Python/2.7.14-foss-2017b
Python/2.7.15-foss-2018b
Python/2.7.15-GCCcore-7.3.0-bare
Python/2.7.15-GCCcore-8.2.0
Python/2.7.15-intel-2018b
Python/2.7.16-GCCcore-8.3.0
Python/2.7.18-GCCcore-9.3.0
Python/2.7.18-GCCcore-10.2.0
Python/2.7.18-GCCcore-10.3.0-bare
Python/2.7.18-GCCcore-11.2.0-bare
Python/2.7.18-GCCcore-11.3.0-bare
Python/3.6.1-goolf-2017a
Python/3.6.1-intel-2017.03-GCC-6.3
Python/3.6.3-foss-2017b
Python/3.6.6-foss-2018b
Python/3.6.6-intel-2018b
Python/3.7.0-foss-2018b (D)
Python/3.7.0-intel-2018b
Python/3.7.2-GCCcore-8.2.0
Python/3.7.4-GCCcore-8.3.0
Python/3.7.5-GCCcore-8.3.0
Python/3.8.2-GCCcore-9.3.0
Python/3.8.6-GCCcore-10.2.0
Python/3.9.5-GCCcore-10.3.0-bare
Python/3.9.5-GCCcore-10.3.0
Python/3.9.6-GCCcore-11.2.0-bare
Python/3.9.6-GCCcore-11.2.0
Python/3.10.4-GCCcore-11.3.0-bare
Python/3.10.4-GCCcore-11.3.0
Python/3.10.8-GCCcore-12.2.0-bare
Python/3.10.8-GCCcore-12.2.0
Python/3.11.3-GCCcore-12.3.0
...
...
//Note: Output from ''module avail Python'' truncated for brevity.//
We can see that version **Python/3.7.0-foss-2018b** is the default, and that matches what we got before. But there are other versions, let's try loading a more up to date version, again using ''module load'', but now also adding the version number:
$ module load Python/3.11.3-GCCcore-12.3.0
$ which python
/mnt/storage/apps/eb/software/Python/3.11.3-GCCcore-12.3.0/bin/python
$ Python -V
Python 3.11.3
$
That's //much// better. We can add the optional version number with the **/** between the package name and we get that specific version loaded instead.
----
==== Unloading Software Modules ====
Assuming we are finished with the version of Python we loaded before, or we need to change to a different one, we can do that easily with ''module unload'':
$ which python
/mnt/storage/apps/eb/software/Python/3.11.3-GCCcore-12.3.0/bin/python
$ module unload Python
$ which python
/usr/bin/python
$
This removes the Python module, and all references to it from our active session. We are back to the normal system Python in this case.
//Note: You do not need to specify the version number on the end of the module - in 99.99% of cases you can only have one module of the same name loaded at any time.//
If you have //many// different modules loaded you can unload __all__ of them with a single ''module purge'' command:
$ module purge
$
----
===== Quick Reference =====
^ Command ^ Purpose ^ Example ^
| module avail | Lists available modules | module avail |
| module avail //STRING// | Lists available modules that contain the case-insensitive string //STRING// | module avail python |
| module -r avail //REGEX// | Lists available modules that match the regular expression //REGEX// | module -r avail 3.8.2$ |
| module spider //STRING// | Lists module information, description and versions for all modules or text descriptions which contain the name //STRING// | module spider gcc |
| module load //STRING// | Load a single module with the exact name //STRING// | module load Python |
| module load //STRING/////VERSION// | Load a single module with the exact name //STRING// and version //VERSION// | module load Python/3.12.8 |
| module list | Lists all modules which are currently loaded | |
| module unload //STRING// | Unload a single loaded module with the name //STRING// | module unload Python |
| module purge | Unload all loaded modules | |
----
[[:started:index|Back to Getting Started]]