====== Podman Help ======

<WRAP todo round>
**This section is incomplete**

This documentation section on //Container Technologies// is still being written and will not be complete until the **Comet** HPC facility is fully commissioned.
</WRAP>

Links to the official Podman documentation material:

   * [[https://podman.io/docs|Podman getting started guide]]
   * [[https://docs.podman.io/en/latest/|Podman full documentation and API guide]]

Unlike [[:advanced:apptainer|Apptainer]], workflow with Podman is much more similar to [[:advanced:docker|Docker]]:

   * You create a local repository of **images**
   * You //download// **images**, potentially layering them on top of each other - e.g. a basic Ubuntu image, then an image which has a specific compiler or version of Python
   * You then create one or more **containers** from those **images**
   * **Containers** can be //started//, //stopped// or //removed//, but the underlying **images** are not affected - you can create several running copies of an Ubuntu container from one Ubuntu image, for example

<WRAP tip round>
We recommend Podman is used by more experienced users. If you are just starting out with containers then [[:advanced:apptainer|Apptainer]] has a lower level of complexity for first time users.
</WRAP>

<WRAP info round>
Note that this help guide is //not// intended to be a complete introduction to Podman - this is just a starter for how Podman can be used on Comet. Please consider signing up to a future RSE container technology workshop.

Links TBC
</WRAP>
===== Before You Start =====

To work correctly on Comet without requiring any additional permissions there is a small amount of configuration each user needs to make before using Podman.

Log in to Comet and run the following:

<code lang=bash>
$ mkdir -p $HOME/.config/containers
$ nano $HOME/.config/containers/storage.conf
</code>

In the text editor, paste the configuration shown below:

<code title=storage.conf>
[storage]
driver = "overlay"
runroot = "/tmp/podman_n1234"
graphroot = "/mnt/nfs/home/n1234/podman_images"

[storage.options.overlay]
force_mask = "0700"
</code>

Make sure that you change **n1234** to your //real// University username. This config file tells Podman to store any **downloaded** or **created** images in ''$HOME/podman_images'', and also instructs Podman to **run** your container from ''/tmp''. 

<WRAP important round>
This is a **mandatory** configuration - if you do not set it then you will __not__ be able to use Podman to create or run containers.
</WRAP>
===== Podman - A Quick Start On Comet =====

Load the Podman module:

<code lang=bash>
$ module load Podman
$ podman version
Client:       Podman Engine
Version:      5.5.3-dev
API Version:  5.5.3-dev
Go Version:   go1.23.6
Git Commit:   87c980c6e2a3e2cb3b8ede4152d94ca204bbe483
Built:        Fri Jun 27 16:17:49 2025
OS/Arch:      linux/amd64$
</code>

Download a new image:

<code lang=bash>
$ podman image pull ubuntu
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob 32f112e3802c done   | 
Copying config 65ae7a6f35 done   | 
Writing manifest to image destination
65ae7a6f3544bd2d2b6d19b13bfc64752d776bc92c510f874188bfd404d205a3
$
</code>

Check that the image is in your image store:

<code lang=bash>
$ podman image list
REPOSITORY                TAG         IMAGE ID      CREATED      SIZE
docker.io/library/ubuntu  latest      65ae7a6f3544  3 weeks ago  80.6 MB
$
</code>

Create a container from the image, using the **IMAGE_ID** shown in ''podman image list'':

<code lang=bash>
$ podman create --network slirp4netns --tty --name ubuntu_test_container 65ae7a6f3544
8334ec4230a039c5c4eaf601e9988276511038e685033aff9d10b590099850f9
$
</code>

Check that your container is listed and available:

<code lang=bash>
$ podman ps --all
CONTAINER ID  IMAGE                            COMMAND     CREATED         STATUS      PORTS       NAMES
8334ec4230a0  docker.io/library/ubuntu:latest  /bin/bash   51 seconds ago  Created                 ubuntu_test_container
$
</code>

Start your container:

<code lang=bash>
$ podman container start ubuntu_test_container
ubuntu_test_container
$
$ podman ps
CONTAINER ID  IMAGE                            COMMAND     CREATED        STATUS        PORTS       NAMES
8334ec4230a0  docker.io/library/ubuntu:latest  /bin/bash   3 minutes ago  Up 3 seconds              ubuntu_test_container
$
</code>

Stop your container:

<code lang=bash>
$ podman container stop ubuntu_test_container
ubuntu_test_container
$
$ podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
$
</code>

You can also run a single command without needing to start the entire container as a running process. This will likely be the most common use case of your Podman containers on Comet:

<code lang=bash>
$ podman container run --network slirp4netns  --interactive --tty --name a_new_test_container 65ae7a6f3544 /bin/bash
root@b114109ae278:/# whoami
root
root@b114109ae278:/# cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
root@b114109ae278:/# exit
exit
$
</code>

In this example we started a new container instance called "//a_new_test_container//" from the Ubuntu **IMAGE_ID** 65ae7a6f3544 and got an interactive BASH prompt inside the image. We then checked which user we were (//root//!) and got the version of Linux we were running in (//Ubuntu//, not Redhat which Comet uses).

In most cases you probably won't be working interactively in the container itself, instead just running a single command (or script of commands), you can achieve that easily by removing the ''--interactive'' and ''--tty'' flags, e.g.:

<code lang=bash>
$ podman container run --replace --network slirp4netns --name a_new_test_container 65ae7a6f3544 whoami
root
$
</code>

<WRAP tip round>
Note the use of the ''--replace'' option; this replaces an existing container of the same name with a new one each time you call ''create'', ''run'' or ''start''. Since you may need to run the same container over and over, this is convenient over creating a unique container each time, or using ''podman container rm'' on the old one.

Check the documentation for Podman to be sure of the meaning behind this: [[https://docs.podman.io/en/v4.4/markdown/podman-create.1.html|podman-create]], [[https://docs.podman.io/en/v4.4/markdown/podman-run.1.html|podman-run]], and [[https://docs.podman.io/en/v4.4/markdown/podman-start.1.html|podman-start]].
</WRAP>

If you are finished with the container, you can remove it:

<code lang=bash>
$ podman container rm a_new_test_container
a_new_test_container
$
$ podman ps --all
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
$
</code>

----
==== Run Podman container under Slurm ====

TBD

<code lang=bash>
#!/bin/bash

#SBATCH --partition=default_free
#SBATCH -c 1
#SBATCH -t 00:05:00

module load Podman

###########################################
echo "Starting Podman image on $HOSTNAME..."

echo "Job completed!"
###########################################
</code>

Submit as:

<code lang=bash>
$ sbatch podman.sh
Submitted batch job 11123455
$
</code>

Check the output:

<code lang=bash>
$ cat slurm-11123455.out
Starting Podman image on compute029...

Job completed!
$
</code>

----
===== Accessing $HOME, /nobackup & Other Directories =====

Use the ''--mount'' parameter to add a Comet directory or filesystem within the container.

== $HOME ==

To make your own ''$HOME'' directory available as ''/my_home'' within the container:

<code lang=bash>
$ podman run --network slirp4netns --mount=type=bind,source=$HOME,destination=/my_home --name a_new_test_container 65ae7a6f3544 ls -l /
total 100
lrwxrwxrwx    1 root   root       7 Apr 22  2024 bin -> usr/bin
drwxr-xr-x    2 root   root    4096 Apr 22  2024 boot
drwxr-xr-x    5 root   root     340 Aug  7 21:07 dev
drwxr-xr-x   32 root   root    4096 Aug  7 21:07 etc
drwxr-xr-x    3 root   root    4096 Jul 14 14:14 home
lrwxrwxrwx    1 root   root       7 Apr 22  2024 lib -> usr/lib
lrwxrwxrwx    1 root   root       9 Apr 22  2024 lib64 -> usr/lib64
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 media
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 mnt
drwx------   31 root   root    4096 Aug  7 20:17 my_home
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 opt
dr-xr-xr-x 1015 nobody nogroup    0 Aug  7 21:07 proc
drwx------    2 root   root    4096 Jul 14 14:14 root
drwxr-xr-x    5 root   root    4096 Aug  7 21:07 run
lrwxrwxrwx    1 root   root       8 Apr 22  2024 sbin -> usr/sbin
drwxr-xr-x    2 root   root    4096 Jul 14 14:08 srv
dr-xr-xr-x   13 nobody nogroup    0 Aug  7 21:07 sys
drwxrwxrwt    2 root   root    4096 Jul 14 14:14 tmp
drwxr-xr-x   12 root   root    4096 Jul 14 14:08 usr
drwxr-xr-x   11 root   root    4096 Jul 14 14:14 var
</code>

Any files created in the mounted ''/my_home'' directory during the run of the container will be owned by you:

<code lang=bash>
$ podman run --network slirp4netns --mount=type=bind,source=$HOME,destination=/my_home --name a_new_test_container 65ae7a6f3544 touch /my_home/hello_podman
$ ls -l $HOME/hello_podman 
-rw-r--r-- 1 n1234 cometloginaccess 0 Aug  7 22:12 /mnt/nfs/home/n1234/hello_podman
$
</code>

== /nobackup ==

TBC

----
===== Image Storage Locations =====

== Personal Image Storage Location ==

You need to set a few key options to get Podman to store and work with images correctly on Comet. 

As mentioned in [[#Before You Start]] you must always have a ''storage.conf'' file created. If you want to work with downloaded images in your own personal ''$HOME'' directory, then save the file below as ''$HOME/.config/containers/storage.conf''. 

The ''graphroot'' directive is where Podman will store any **images** you download or create; it is safe to set this to your ''$HOME'' directory as suggested, or even an area in a project folder under ''/nobackup''.

Note that the ''runroot'' directive **must** always point to the ''/tmp'' directory - this is where Podman will run your container, and it is **only** supported from a local filesystem; **do not** change this to ''$HOME'' or ''/nobackup''. 

<code name=storage.conf>
[storage]
driver = "overlay"
runroot = "/tmp/podman_n1234"
graphroot = "/mnt/nfs/home/n1234/podman_images"

[storage.options.overlay]
force_mask = "0700"
</code>

Change the username **n1234** to your real University username.

== Shared / Group Image Storage Location ==

TBD

----
===== Errors & Restrictions of Podman on Comet =====

==== Common Errors ====

== Changing runroot ==

The ''runroot'' option must always point to a local, physical filesystem. You must not change this to ''$HOME'' or ''/nobackup''.

== Podman run or exec errors ==

The most common error ''Error: creating container storage: the container name "a_new_test_container" is already in use by ... '' means that you have tried to run or exec a new container with the same name as one which already exists. Try running ''podman ps --all'' to get a full list of all your existing containers; each new container must have a //unique// name.

== Where do my files go? ==

If you have not added the ''$HOME'' mount to your container, then any files you create inside the container will be part of the container //overlay// filesystem, and instead spread out over the directory you set for ''runroot'' in ''$HOME/.config/containers/storage.conf''.

It is **highly** recommended that you mount ''$HOME'' to your container and explicitly save any files you create during running the container to a sub-directory of that path.

==== Restrictions of Podman on Comet ====

   * Running network services is not supported; you cannot run a web server, database server or anything similar - there is no supported means to expose these services and any found running will be stopped and removed.

----

[[advanced:software|Back to Advanced Software Index]]