====== MADflow ======

> Originally developed for academic research, MADflow is a light-weight yet highly accurate numerical simulation tool for mobility analysis of gravity-driven flows of tailings, soil, rock and/or water mixtures. It is particularly suited for studying phenomena such as debris flows, tailings flow, mudflows, mud floods, flowslides and avalanches.
> 
> Being able to integrate complex 3D terrain or simplified 2D sections, MADflow stands out as a robust, strictly volume-conservative, depth-averaged dynamical model. It incorporates widely used rheological constitutive relationships tailored for non-Newtonian flows (including granular flows), ranging from frictional-based, visco-plastic, to their combination, to capture various flow behaviour from low to high solids concentrations. The model ensures realistic stress response to strain during movement, especially in flows with high solids concentration.
> 
> MADflow supports both Windows and Linux environments and incorporates multi-level local grid refinement within a fully parallelized framework. The code achieves near-optimal scalability through OpenMP on multicore CPUs and CUDA on NVIDIA GPUs, delivering substantial speedups that reduce wall-clock times from days to hours, hours to minutes, or minutes to seconds relative to conventional, non-accelerated solvers..

   * From: https://madflow.ca/

**Please note:** The software discussed above, from madflow.ca is __not__ the same as https://madflow.readthedocs.io/en/latest/index.html - these are two __different__ projects.

----
===== Licensing =====

Whilst MADflow.ca software is //freely available//, it is distributed on request directly from the author with a license key. If you wish to use a copy of MADflow on our HPC systems, please use the contact information provided on their website to request a version for your own use:

   * https://madflow.ca/contact.html

MADflow is __not__ installed globally on Comet, but the information provided below will assist you in configuring it for your use based on our HPC architecture.

----
===== Configuring madflow script =====

The MADflow.ca software requires a correctly configured Ubuntu operating system to run; whilst the Comet HPC is RedHat based, we have provided a minimal Ubuntu 24 LTS container image to run MADflow. You can find the image here:

   * ''/nobackup/shared/containers/madflow.sif''

You __do not__ need to make a copy or move the Ubuntu container - the instructions //below// configure your MADflow software install to use it correctly.

Once you have downloaded your MADflow.ca software bundle you will need to copy it to Comet and then follow these steps:

   * Copy the MADflow.ca zip file to your Comet ''$HOME'' directory
   * Either create a new folder in your home (if for your personal use only) or project directory (if it will be shared with other team members)
   * Unzip the MADflow.ca zip file
   * Change the ''Linux-cluster/madflow'' folder which has been created
   * Open the ''madflow'' shell script with a text editor

Assuming you downloaded the file to your ''$HOME'' directory, and you want to create the MADflow.ca software in your project folder named ''comet_myproject'', the following steps should work:

<code lang=bash>
$ cd /nobackup/proj/comet_myproject
$ mkdir MADflow.ca
$ cd MADflow.ca
$ unzip $HOME/madflow.zip
$ cd Linux-cluster/madflow
$ nano madflow
</code>

At this point you will have the ''madflow'' script open in your editor:

<code lang=bash title=madflow>
#!/bin/bash
# MADflow launcher for clusters whose compute nodes cannot run the engine
# directly (e.g. Red Hat 9): runs it inside an Ubuntu 24.04 container with
# Apptainer. Put this file in the MADflow folder, the engine and its key in
# engine/ next to it, set IMAGE below, and make both executable once:
#     chmod +x madflow engine/madflow
# See ReadMe.txt in this folder, step (3b).
#
# IMAGE  the site's Ubuntu 24.04 container image (.sif)
# BIND   extra folders the container must see, e.g. /scratch,/project
#        (your home directory and the current folder are always visible)
# --nv   gives the engine the node's NVIDIA GPU (driver 550 or newer)

IMAGE=/image.sif
BIND=
here=$(dirname "$(readlink -f "$0")")
exec apptainer exec --nv ${BIND:+--bind "$BIND"} "$IMAGE" "$here/engine/madflow" "$@"
</code>

Make the following changes to the ''madflow'' script to use the Ubuntu container we have provided (the ''IMAGE'' variable) **and** to map the standard Comet filesystem locations (the ''BIND'' variable). Note we //also// need to load the ''apptainer'' module to ensure it is available at runtime (''module load apptainer''):

<code lang=bash title=madflow>
#!/bin/bash
# MADflow launcher for clusters whose compute nodes cannot run the engine
# directly (e.g. Red Hat 9): runs it inside an Ubuntu 24.04 container with
# Apptainer. Put this file in the MADflow folder, the engine and its key in
# engine/ next to it, set IMAGE below, and make both executable once:
#     chmod +x madflow engine/madflow
# See ReadMe.txt in this folder, step (3b).
#
# IMAGE  the site's Ubuntu 24.04 container image (.sif)
# BIND   extra folders the container must see, e.g. /scratch,/project
#        (your home directory and the current folder are always visible)
# --nv   gives the engine the node's NVIDIA GPU (driver 550 or newer)

# Path to the Ubuntu container for MADflow on Comet
IMAGE=/nobackup/shared/containers/madflow.sif
# Add the standard /nobackup and /scratch filesystems
BIND=/nobackup,/scratch

here=$(dirname "$(readlink -f "$0")")

# Needed to make the apptainer commands available
module load apptainer

exec apptainer exec --nv ${BIND:+--bind "$BIND"} "$IMAGE" "$here/engine/madflow" "$@"
</code>

You are now ready to use one of the example Slurm job files below as the basis of your own job file.

----
===== Example Slurm Job =====

All examples assume you have installed MADflow.ca software as per the instructions above, and you are running from the ''comet_myproject'' project directory. Adjust to your //own// installation path and //own// project account code accordingly.

==== CPU Job ====

Use standard CPU based compute for MADflow to process a __single__ input file.

Note:
   * If you want to use the reserved CPU cores in the //paid// partition, then substitute the **default_paid** partition name - but you //must// have a paid/funded HPC project.
   * Below example allocates 16 CPU cores, you may need to set a different value, e.g. <code>#SBATCH -c 64</code> for a 64 core example.
   * Default memory allocation (1GB), you may need to set a different value, e.g. <code>#SBATCH --mem 16G</code> for a 16GB RAM example.

<code lang=bash title=madflow_cpu.sh>
#!/bin/bash
#SBATCH --job-name=madflow_cpu
#SBATCH --partition=default_free
#SBATCH --account=comet_myproject
#SBATCH --cpus-per-task=16
#SBATCH --nodes=1
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@120
# Assuming all of your input files are stored here
cd "/nobackup/proj/comet_myproject"
# A single job with 16 cores
exec /nobackup/proj/comet_myproject/MADflow.ca/Linux-cluster/madflow/madflow my_model -p 16
</code>

==== CPU Job (array task) ====

Use standard CPU based compute for MADflow to process __multiple input files in parallel__.

Note:
   * If you want to use the reserved CPU cores in the //paid// partition, then substitute the **default_paid** partition name - but you //must// have a paid/funded HPC project.
   * Below example runs 20 //parallel tasks//, each with a unique input file. The default approach below assumes each of the input files is numbered sequentially.
   * Below example allocates 16 CPU cores //per task// (16 x 20 = 320 CPU cores), you may need to set a different value, e.g. <code>#SBATCH -c 64</code> for a 64 core example.
   * Default memory allocation (1GB), you may need to set a different value, e.g. <code>#SBATCH --mem-per-cpu 1G</code> to allocate 1GB per CPU core.

<code lang=bash title=madflow_cpu.sh>
#!/bin/bash
#SBATCH --job-name=madflow_cpu
#SBATCH --partition=default_free
#SBATCH --account=comet_myproject
#SBATCH --array=1-20
#SBATCH --cpus-per-task=16
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@120
# Assuming all of your input files are stored here
cd "/nobackup/proj/comet_myproject"  
# Use one job per model, with a job array (input files should be named case_1.ini ... case_20.ini) - this is 20 parallel jobs, each with 16 cores
exec /nobackup/proj/comet_myproject/MADflow.ca/Linux-cluster/madflow/madflow case_${SLURM_ARRAY_TASK_ID} -p 16
</code>

==== GPU Job ====

Request a single L40-S GPU in the **gpu-s_free** partition to use Nvidia GPU based compute acceleration for MADflow.

Note:
   * If you want to use the reserved cards in the //paid// partition, then substitute the **gpu-s_paid** partition name - but you //must// have a paid/funded HPC project.
   * If you want to use the H100 cards, then substitute the **gpu-l_paid** partition name, and <code>#SBATCH --gres=gpu:H100:1</code> for the resource type instead - but you //must// have a paid/funded HPC project.
   * Default CPU core allocation (1 core), you may need to set a different value, e.g. <code>#SBATCH -c 8</code> for an 8 core example.
   * Default memory allocation (2GB), you may need to set a different value, e.g. <code>#SBATCH --mem 16G</code> for a 16GB RAM example.
   * You can combine the task array based CPU example with GPU to launch parallel GPU compute jobs, but free HPC projects are subject to simultaneous GPU use restrictions that make this impractical if you are not a paid/funded HPC project member.

<code lang=bash title=madflow_gpu.sh>
#!/bin/bash
#SBATCH --job-name=madflow_gpu
#SBATCH --partition=gpu-s_free
#SBATCH --account=comet_myproject
#SBATCH --gres=gpu:L40:1
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@120
# Assuming all of your input files are stored here
cd "/nobackup/proj/comet_myproject"
# Single GPU job
exec /nobackup/proj/comet_myproject/MADflow.ca/Linux-cluster/madflow/madflow my_model -gpu 0
</code>
----

[[:advanced:software|Back to Advanced Software index]]