Table of Contents

Podman Help

This section is incomplete

This documentation section on Container Technologies is still being written and will not be complete until the Comet HPC facility is fully commissioned.

Links to the official Podman documentation material:

Unlike Apptainer, workflow with Podman is much more similar to Docker:

We recommend Podman is used by more experienced users. If you are just starting out with containers then Apptainer has a lower level of complexity for first time users.

Note that this help guide is not intended to be a complete introduction to Podman - this is just a starter for how Podman can be used on Comet. Please consider signing up to a future RSE container technology workshop.

Links TBC

Before You Start

To work correctly on Comet without requiring any additional permissions there is a small amount of configuration each user needs to make before using Podman.

Log in to Comet and run the following:

$ mkdir -p $HOME/.config/containers
$ nano $HOME/.config/containers/storage.conf

In the text editor, paste the configuration shown below:

[storage]
driver = "overlay"
runroot = "/tmp/podman_n1234"
graphroot = "/mnt/nfs/home/n1234/podman_images"

[storage.options.overlay]
force_mask = "0700"

Make sure that you change n1234 to your real University username. This config file tells Podman to store any downloaded or created images in $HOME/podman_images, and also instructs Podman to run your container from /tmp.

This is a mandatory configuration - if you do not set it then you will not be able to use Podman to create or run containers.

Podman - A Quick Start On Comet

Load the Podman module:

$ module load Podman
$ podman version
Client:       Podman Engine
Version:      5.5.3-dev
API Version:  5.5.3-dev
Go Version:   go1.23.6
Git Commit:   87c980c6e2a3e2cb3b8ede4152d94ca204bbe483
Built:        Fri Jun 27 16:17:49 2025
OS/Arch:      linux/amd64$

Download a new image:

$ podman image pull ubuntu
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob 32f112e3802c done   | 
Copying config 65ae7a6f35 done   | 
Writing manifest to image destination
65ae7a6f3544bd2d2b6d19b13bfc64752d776bc92c510f874188bfd404d205a3
$

Check that the image is in your image store:

$ podman image list
REPOSITORY                TAG         IMAGE ID      CREATED      SIZE
docker.io/library/ubuntu  latest      65ae7a6f3544  3 weeks ago  80.6 MB
$

Create a container from the image, using the IMAGE_ID shown in podman image list:

$ podman create --network slirp4netns --tty --name ubuntu_test_container 65ae7a6f3544
8334ec4230a039c5c4eaf601e9988276511038e685033aff9d10b590099850f9
$

Check that your container is listed and available:

$ podman ps --all
CONTAINER ID  IMAGE                            COMMAND     CREATED         STATUS      PORTS       NAMES
8334ec4230a0  docker.io/library/ubuntu:latest  /bin/bash   51 seconds ago  Created                 ubuntu_test_container
$

Start your container:

$ podman container start ubuntu_test_container
ubuntu_test_container
$
$ podman ps
CONTAINER ID  IMAGE                            COMMAND     CREATED        STATUS        PORTS       NAMES
8334ec4230a0  docker.io/library/ubuntu:latest  /bin/bash   3 minutes ago  Up 3 seconds              ubuntu_test_container
$

Stop your container:

$ podman container stop ubuntu_test_container
ubuntu_test_container
$
$ podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
$

You can also run a single command without needing to start the entire container as a running process. This will likely be the most common use case of your Podman containers on Comet:

$ podman container run --network slirp4netns  --interactive --tty --name a_new_test_container 65ae7a6f3544 /bin/bash
root@b114109ae278:/# whoami
root
root@b114109ae278:/# cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
root@b114109ae278:/# exit
exit
$

In this example we started a new container instance called “a_new_test_container” from the Ubuntu IMAGE_ID 65ae7a6f3544 and got an interactive BASH prompt inside the image. We then checked which user we were (root!) and got the version of Linux we were running in (Ubuntu, not Redhat which Comet uses).

In most cases you probably won't be working interactively in the container itself, instead just running a single command (or script of commands), you can achieve that easily by removing the –interactive and –tty flags, e.g.:

$ podman container run --replace --network slirp4netns --name a_new_test_container 65ae7a6f3544 whoami
root
$

Note the use of the –replace option; this replaces an existing container of the same name with a new one each time you call create, run or start. Since you may need to run the same container over and over, this is convenient over creating a unique container each time, or using podman container rm on the old one.

Check the documentation for Podman to be sure of the meaning behind this: podman-create, podman-run, and podman-start.

If you are finished with the container, you can remove it:

$ podman container rm a_new_test_container
a_new_test_container
$
$ podman ps --all
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
$


Run Podman container under Slurm

TBD

#!/bin/bash

#SBATCH --partition=default_free
#SBATCH -c 1
#SBATCH -t 00:05:00

module load Podman

###########################################
echo "Starting Podman image on $HOSTNAME..."

echo "Job completed!"
###########################################

Submit as:

$ sbatch podman.sh
Submitted batch job 11123455
$

Check the output:

$ cat slurm-11123455.out
Starting Podman image on compute029...

Job completed!
$


Accessing $HOME, /nobackup & Other Directories

Use the –mount parameter to add a Comet directory or filesystem within the container.

$HOME

To make your own $HOME directory available as /my_home within the container:

$ podman run --network slirp4netns --mount=type=bind,source=$HOME,destination=/my_home --name a_new_test_container 65ae7a6f3544 ls -l /
total 100
lrwxrwxrwx    1 root   root       7 Apr 22  2024 bin -> usr/bin
drwxr-xr-x    2 root   root    4096 Apr 22  2024 boot
drwxr-xr-x    5 root   root     340 Aug  7 21:07 dev
drwxr-xr-x   32 root   root    4096 Aug  7 21:07 etc
drwxr-xr-x    3 root   root    4096 Jul 14 14:14 home
lrwxrwxrwx    1 root   root       7 Apr 22  2024 lib -> usr/lib
lrwxrwxrwx    1 root   root       9 Apr 22  2024 lib64 -> usr/lib64
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 media
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 mnt
drwx------   31 root   root    4096 Aug  7 20:17 my_home
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 opt
dr-xr-xr-x 1015 nobody nogroup    0 Aug  7 21:07 proc
drwx------    2 root   root    4096 Jul 14 14:14 root
drwxr-xr-x    5 root   root    4096 Aug  7 21:07 run
lrwxrwxrwx    1 root   root       8 Apr 22  2024 sbin -> usr/sbin
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 srv
dr-xr-xr-x   13 nobody nogroup    0 Aug  7 21:07 sys
drwxrwxrwt    2 root   root    4096 Jul 14 14:14 tmp
drwxr-xr-x   12 root   root    4096 Jul 14 14:08 usr
drwxr-xr-x   11 root   root    4096 Jul 14 14:14 var

Any files created in the mounted /my_home directory during the run of the container will be owned by you:

$ podman run --network slirp4netns --mount=type=bind,source=$HOME,destination=/my_home --name a_new_test_container 65ae7a6f3544 touch /my_home/hello_podman
$ ls -l $HOME/hello_podman 
-rw-r--r-- 1 n1234 cometloginaccess 0 Aug  7 22:12 /mnt/nfs/home/n1234/hello_podman
$

/nobackup

TBC


Image Storage Locations

Personal Image Storage Location

You need to set a few key options to get Podman to store and work with images correctly on Comet.

As mentioned in Before You Start you must always have a storage.conf file created. If you want to work with downloaded images in your own personal $HOME directory, then save the file below as $HOME/.config/containers/storage.conf.

The graphroot directive is where Podman will store any images you download or create; it is safe to set this to your $HOME directory as suggested, or even an area in a project folder under /nobackup.

Note that the runroot directive must always point to the /tmp directory - this is where Podman will run your container, and it is only supported from a local filesystem; do not change this to $HOME or /nobackup.

[storage]
driver = "overlay"
runroot = "/tmp/podman_n1234"
graphroot = "/mnt/nfs/home/n1234/podman_images"

[storage.options.overlay]
force_mask = "0700"

Change the username n1234 to your real University username.

Shared / Group Image Storage Location

TBD


Errors & Restrictions of Podman on Comet

Common Errors

Changing runroot

The runroot option must always point to a local, physical filesystem. You must not change this to $HOME or /nobackup.

Podman run or exec errors

The most common error Error: creating container storage: the container name “a_new_test_container” is already in use by … means that you have tried to run or exec a new container with the same name as one which already exists. Try running podman ps –all to get a full list of all your existing containers; each new container must have a unique name.

Where do my files go?

If you have not added the $HOME mount to your container, then any files you create inside the container will be part of the container overlay filesystem, and instead spread out over the directory you set for runroot in $HOME/.config/containers/storage.conf.

It is highly recommended that you mount $HOME to your container and explicitly save any files you create during running the container to a sub-directory of that path.

Restrictions of Podman on Comet


Back to Advanced Software Index