====== Podman Help ======
Links to the official Podman documentation material:
* [[https://podman.io/docs|Podman getting started guide]]
* [[https://docs.podman.io/en/latest/|Podman full documentation and API guide]]
We recommend Podman is used by more experienced users. If you are just starting out with containers then [[:advanced:apptainer|Apptainer]] has a lower level of complexity for first time users.
Unlike [[:advanced:apptainer|Apptainer]], workflow with Podman is much more similar to [[:advanced:docker|Docker]]:
* **(A)** You create a local repository of **images**
* **(B)** You //download// **images** or create them from [[https://docs.podman.io/en/stable/markdown/podman-build.1.html|container description files]], potentially layering them on top of each other - e.g. a basic Ubuntu image, then an image which has a specific compiler or version of Python
* **(C)** You then create one or more **containers** from those **images**
* **(D)** **Containers** can be //started//, //stopped// or //removed//, but the underlying **images** are not affected - you can create several running copies of an Ubuntu container from one Ubuntu image, for example. You then (finally) run commands and software inside those containers.
**Comet Workflow Restrictions**
With Comet, you can achieve **(A)** and **(B)** on **login** nodes __only__, but **(C)** and **(D)** is possible on //both// **login** nodes //and// **compute** nodes.
Do note that this help guide is //not// intended to be a complete introduction to Podman - this is just a starter for how Podman can be used on Comet, as well as some of the limitations and common queries. Please consider signing up to a future RSE container technology workshop.
===== Before You Start =====
To work correctly on Comet without requiring any additional permissions there is a small amount of configuration each user needs to make before using Podman.
Log in to Comet and run the following:
$ mkdir -p $HOME/.config/containers
$ nano $HOME/.config/containers/storage.conf
In the text editor, paste the configuration shown below:
[storage]
driver = "overlay"
runroot = "/tmp/podman_n1234"
graphroot = "/mnt/nfs/home/n1234/podman_images"
[storage.options.overlay]
force_mask = "private"
Make sure that you change **n1234** to your //real// University username. This config file tells Podman to store any **downloaded** or **created** images in ''$HOME/podman_images'', and also instructs Podman to **run** your container from ''/tmp''.
This is a **mandatory** configuration - if you do not set it then you will __not__ be able to use Podman to create or run containers.
===== Podman - A Quick Start On Comet =====
Load the Podman module:
$ module load Podman
$ podman version
Client: Podman Engine
Version: 5.5.3-dev
API Version: 5.5.3-dev
Go Version: go1.23.6
Git Commit: 87c980c6e2a3e2cb3b8ede4152d94ca204bbe483
Built: Fri Jun 27 16:17:49 2025
OS/Arch: linux/amd64$
Download a new image:
$ podman image pull ubuntu
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob 32f112e3802c done |
Copying config 65ae7a6f35 done |
Writing manifest to image destination
65ae7a6f3544bd2d2b6d19b13bfc64752d776bc92c510f874188bfd404d205a3
$
Check that the image is in your image store:
$ podman image list
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/ubuntu latest 65ae7a6f3544 3 weeks ago 80.6 MB
$
We will refer to image ID ''65ae7a6f3544'' as ''MY_IMAGE_ID'' for the rest of this guide.
Create a container from the image, using the **IMAGE_ID** shown in ''podman image list'':
$ podman create --tty --name ubuntu_test_container MY_IMAGE_ID
8334ec4230a039c5c4eaf601e9988276511038e685033aff9d10b590099850f9
$
Check that your container is listed and available:
$ podman ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8334ec4230a0 docker.io/library/ubuntu:latest /bin/bash 51 seconds ago Created ubuntu_test_container
$
Start your container:
$ podman container start ubuntu_test_container
ubuntu_test_container
$
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8334ec4230a0 docker.io/library/ubuntu:latest /bin/bash 3 minutes ago Up 3 seconds ubuntu_test_container
$
Stop your container:
$ podman container stop ubuntu_test_container
ubuntu_test_container
$
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$
You can also run a single command without needing to start the entire container as a running process. This will likely be the most common use case of your Podman containers on Comet:
$ podman container run --interactive --tty --name a_new_test_container MY_IMAGE_ID /bin/bash
root@b114109ae278:/# whoami
root
root@b114109ae278:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
root@b114109ae278:/# exit
exit
$
In this example we started a new container instance called "//a_new_test_container//" from the Ubuntu **IMAGE_ID** ''MY_IMAGE_ID'' and got an interactive BASH prompt inside the image. We then checked which user we were (//root//!) and got the version of Linux we were running in (//Ubuntu//, not Redhat which Comet uses).
In most cases you probably won't be working interactively in the container itself, instead just running a single command (or script of commands), you can achieve that easily by removing the ''--interactive'' and ''--tty'' flags, e.g.:
$ podman container run --replace --name a_new_test_container MY_IMAGE_ID whoami
root
$
Note the use of the ''--replace'' option; this replaces an existing container of the same name with a new one each time you call ''create'', ''run'' or ''start''. Since you may need to run the same container over and over, this is convenient over creating a unique container each time, or using ''podman container rm'' on the old one.
Check the documentation for Podman to be sure of the meaning behind this: [[https://docs.podman.io/en/v4.4/markdown/podman-create.1.html|podman-create]], [[https://docs.podman.io/en/v4.4/markdown/podman-run.1.html|podman-run]], and [[https://docs.podman.io/en/v4.4/markdown/podman-start.1.html|podman-start]].
If you are finished with the container, you can remove it:
$ podman container rm a_new_test_container
a_new_test_container
$
$ podman ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$
----
==== Run Podman container under Slurm ====
Podman will run happily under Slurm on a compute node (including GPU nodes), but is subject to the following restrictions:
* You __cannot__ **pull** images while on a compute node
* You __cannot__ **build** images while on a compute node
Compute nodes are therefore treated as a **run-only** environment for containers - download or build any containers you need whilst on a login node __first__.
#!/bin/bash
#SBATCH --partition=default_free
#SBATCH -c 1
#SBATCH -t 00:05:00
module load Podman
###########################################
echo "Starting Podman image on $HOSTNAME..."
# This simple job just prints the version of Python installed in the container image
podman container run --replace --name my_test_container MY_IMAGE_ID python -V
echo "Job completed!"
###########################################
Submit as:
$ sbatch podman.sh
Submitted batch job 11123455
$
Check the output:
$ cat slurm-11123455.out
Starting Podman image on compute029...
Python v3.12.4
Job completed!
$
----
===== Accessing $HOME, /nobackup & Other Directories =====
Use the ''--mount'' parameter to add a Comet directory or filesystem within the container.
== $HOME ==
To make your own ''$HOME'' directory available as ''/my_home'' within the container:
$ podman run --mount=type=bind,source=$HOME,destination=/my_home --name a_new_test_container 65ae7a6f3544 ls -l /
total 100
lrwxrwxrwx 1 root root 7 Apr 22 2024 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Apr 22 2024 boot
drwxr-xr-x 5 root root 340 Aug 7 21:07 dev
drwxr-xr-x 32 root root 4096 Aug 7 21:07 etc
drwxr-xr-x 3 root root 4096 Jul 14 14:14 home
lrwxrwxrwx 1 root root 7 Apr 22 2024 lib -> usr/lib
lrwxrwxrwx 1 root root 9 Apr 22 2024 lib64 -> usr/lib64
drwxr-xr-x 2 root root 4096 Jul 14 14:08 media
drwxr-xr-x 2 root root 4096 Jul 14 14:08 mnt
drwx------ 31 root root 4096 Aug 7 20:17 my_home
drwxr-xr-x 2 root root 4096 Jul 14 14:08 opt
dr-xr-xr-x 1015 nobody nogroup 0 Aug 7 21:07 proc
drwx------ 2 root root 4096 Jul 14 14:14 root
drwxr-xr-x 5 root root 4096 Aug 7 21:07 run
lrwxrwxrwx 1 root root 8 Apr 22 2024 sbin -> usr/sbin
drwxr-xr-x 2 root root 4096 Jul 14 14:08 srv
dr-xr-xr-x 13 nobody nogroup 0 Aug 7 21:07 sys
drwxrwxrwt 2 root root 4096 Jul 14 14:14 tmp
drwxr-xr-x 12 root root 4096 Jul 14 14:08 usr
drwxr-xr-x 11 root root 4096 Jul 14 14:14 var
Any files created in the mounted ''/my_home'' directory during the run of the container will be owned by you:
$ podman run --mount=type=bind,source=$HOME,destination=/my_home --name a_new_test_container 65ae7a6f3544 touch /my_home/hello_podman
$ ls -l $HOME/hello_podman
-rw-r--r-- 1 n1234 cometloginaccess 0 Aug 7 22:12 /mnt/nfs/home/n1234/hello_podman
$
== /nobackup ==
Access to ''/nobackup'' is added in the same way:
$ podman run --mount=type=bind,source=/nobackup,destination=/nobackup --name a_new_test_container 65ae7a6f3544 ls -l /nobackup/proj/mygroup
-rw-r--r-- 1 n1234 cometloginaccess 1001 Aug 7 22:12 file1.txt
-rw-r--r-- 1 n1234 cometloginaccess 2300 Aug 7 22:13 file2.txt
-rw-r--r-- 1 n1234 cometloginaccess 594 Aug 7 22:14 file3.txt
You can read and write to any ''/nobackup'' directories that your normal user account has permissions to.
----
===== Image Storage Locations =====
== Personal Image Storage Location ==
You need to set a few key options to get Podman to store and work with images correctly on Comet.
As mentioned in [[#Before You Start]] you must always have a ''storage.conf'' file created. If you want to work with downloaded images in your own personal ''$HOME'' directory, then save the file below as ''$HOME/.config/containers/storage.conf''.
The ''graphroot'' directive is where Podman will store any **images** you download or create; it is safe to set this to your ''$HOME'' directory as suggested, or even an area in a project folder under ''/nobackup''.
Note that the ''runroot'' directive **must** always point to the ''/tmp'' directory - this is where Podman will run your container, and it is **only** supported from a local filesystem; **do not** change this to ''$HOME'' or ''/nobackup''.
[storage]
driver = "overlay"
runroot = "/tmp/podman_n1234"
graphroot = "/mnt/nfs/home/n1234/podman_images"
[storage.options.overlay]
force_mask = "0700"
Change the username **n1234** to your real University username.
== Shared / Group Image Storage Location ==
**This section is incomplete**
This documentation section on //Container Technologies// is still being written and will not be complete until the **Comet** HPC facility is fully commissioned.
----
===== Errors & Restrictions of Podman on Comet =====
==== Common Errors ====
== I can't build (or pull) my images! ==
Please download and build your Podman images on the login nodes __only__; **it is not supported** on compute nodes.
To be clear; any ''podman image pull'' or ''podman image build'' or //similar// image creation process is only supported on login nodes. **Do not try to run these commands on compute nodes**
== Changing runroot ==
The ''runroot'' option must always point to a local, physical filesystem. You **must not** change this to ''$HOME'' or ''/nobackup''.
== Podman run or exec errors ==
The most common error ''Error: creating container storage: the container name "a_new_test_container" is already in use by ... '' means that you have tried to run or exec a new container with the same name as one which already exists. Try running ''podman ps --all'' to get a full list of all your existing containers; each new container must have a //unique// name.
== Where do my files go? ==
If you have not added the ''$HOME'' mount to your container, then any files you create inside the container will be part of the container //overlay// filesystem, the complexity of using overlay filesystems is beyond this simple guide. It is **highly** recommended that you mount ''$HOME'' and/or ''/nobackup'' to your container and explicitly save any files you create during running the container to a sub-directory of those paths.
** Restrictions of Podman on Comet **
Running network services is not supported; you **cannot run a web server, database server or anything similar** - there is no supported means to expose these services and any found running will be stopped and removed.
----
[[advanced:software|Back to Advanced Software Index]]