====== Container Technology ====== Building more complex processing and analysis pipelines within a container can offer several advantages: * The container fully encapsulates the job and all software dependencies * Software dependencies do not need to be installed on the HPC facility * Your workflow can easily be replicated or moved between other systems * You can run //pre-built// job containers which application developers publish * You can build //your own// containers to move between systems, or share with your group * [[#apptainer|Apptainer]] should be your first choice. It is designed for HPC and almost perfectly compatible with dockerfiles. The **Comet** HPC facility supports the following container tools: * [[#apptainer|Apptainer]] * [[#podman|Podman]] - RedHat Linux specific container toolset, largely compatible with Docker * [[#apptainer|Singularity]] - Now //deprecated// and replaced by [[#apptainer|Apptainer]] Unlike Rocket, Comet is design to be __fully native container aware__ - all tools are configured to allow the running of container jobs under Slurm from the above technologies, enabling the community to take advantage of the increased number of ready-built container images available to them from various software developers. Please note that we __do not__ have support for Kubernetes at this time. **This section is incomplete** This documentation section on //Container Technology// is still being written. Please check back soon for updates ===== Docker ===== Whilst Docker does //technically// work on Comet, and //can// be run in rootless mode, there are a number of __serious__ drawbacks with its use that make it a sub-optimal choice compared to Podman or Apptainer: * Your Docker image library is not //easily// configurable to run from a network filesystem location (such as ''$HOME'' or ''/nobackup'') * The running Docker container //cannot// be located on a network filesystem (such as ''$HOME'' or ''/nobackup'') Since running a Docker container therefore requires the container image to be on a local, physical filesystem, this makes it impractical to use across the distributed compute nodes (and their local drives) which make up Comet. Apptainer and Podman __do not share these limitations__. As such, we do not intend to develop the Docker implementation on Comet any further. All users who wish to use containers are strongly urged to use Apptainer in the first instance, and Podman if Apptainer is unsuitable. ---- ===== Apptainer ===== If you have used Singularity previously, then [[https://apptainer.org/|Apptainer]] is a 100%, like-for-like replacement. You can create container images which exist in a single file and are easy to transport between different systems and provide a consistent environment for your software, reducing the need for the installation of additional libraries and other dependencies. You can use Apptainer to **build** and **run** container images on the Comet login nodes, as well as **run** container images on any of the available compute nodes. Please read our full guide on using Apptainer on Comet, which is linked below, or can be accessed from the [[:advanced:software|Advanced Software Topics]] section of the wiki: * [[:advanced:apptainer|Apptainer help guide]] ---- ===== Singularity ===== Singularity is now deprecated and the community have largely moved to Apptainer instead - an alternative version of Singularity now exists named SingularityCE; we __do not__ support SingularityCE. == Singularity CE == SingularityCE is the 'free' version maintained by a commercial entity (Sylabs). This software __is not installed on Comet__ and we __do not__ support it. == Apptainer == Apptainer is a like-for-like, drop-in replacement for Singularity, is open-source and supported by the Linux foundation. We have this tool installed on Comet and we //provide support// for it. If you have previously used Singularity, then you can start using Apptainer by simply changing any call to ''singularity'' to ''apptainer'' instead. Historically we //did// have Singularity on our previous **Rocket** HPC facility, but it was not advertised and was very much a "use at your own risk" capability. You are encouraged to read our [[:advanced:apptainer|Apptainer help guide]] to be aware of any specification configuration or limitations with the use of the technology on Comet. ---- ===== Podman ===== [[https://podman.io/|Podman]] is a container technology tool somewhat similar to Docker, but has a slightly different implementation making it easier to integrate into existing systems; it is often preferred to Docker due to being easier to run by a non-admin user. From a user perspective it has much of the same rich functionality as Docker and is more suited to expert users than something like Apptainer. Using Podman to **build** or **run** containers on the Comet login nodes as well as **run** containers on the Comet compute nodes is fully supported. Please read our Podman help guide for a walk-through of how to use Podman on Comet. * [[:advanced:podman|Podman help guide]] ---- [[advanced:index|Back to Advanced Topics]]