MADflow

Originally developed for academic research, MADflow is a light-weight yet highly accurate numerical simulation tool for mobility analysis of gravity-driven flows of tailings, soil, rock and/or water mixtures. It is particularly suited for studying phenomena such as debris flows, tailings flow, mudflows, mud floods, flowslides and avalanches.

Being able to integrate complex 3D terrain or simplified 2D sections, MADflow stands out as a robust, strictly volume-conservative, depth-averaged dynamical model. It incorporates widely used rheological constitutive relationships tailored for non-Newtonian flows (including granular flows), ranging from frictional-based, visco-plastic, to their combination, to capture various flow behaviour from low to high solids concentrations. The model ensures realistic stress response to strain during movement, especially in flows with high solids concentration.

MADflow supports both Windows and Linux environments and incorporates multi-level local grid refinement within a fully parallelized framework. The code achieves near-optimal scalability through OpenMP on multicore CPUs and CUDA on NVIDIA GPUs, delivering substantial speedups that reduce wall-clock times from days to hours, hours to minutes, or minutes to seconds relative to conventional, non-accelerated solvers..

Please note: The software discussed above, from madflow.ca is not the same as https://madflow.readthedocs.io/en/latest/index.html - these are two different projects.

Licensing

Whilst MADflow.ca software is freely available, it is distributed on request directly from the author with a license key. If you wish to use a copy of MADflow on our HPC systems, please use the contact information provided on their website to request a version for your own use:

MADflow is not installed globally on Comet, but the information provided below will assist you in configuring it for your use based on our HPC architecture.

Configuring madflow script

The MADflow.ca software requires a correctly configured Ubuntu operating system to run; whilst the Comet HPC is RedHat based, we have provided a minimal Ubuntu 24 LTS container image to run MADflow. You can find the image here:

  • /nobackup/shared/containers/madflow.sif

You do not need to make a copy or move the Ubuntu container - the instructions below configure your MADflow software install to use it correctly.

Once you have downloaded your MADflow.ca software bundle you will need to copy it to Comet and then follow these steps:

  • Copy the MADflow.ca zip file to your Comet $HOME directory
  • Either create a new folder in your home (if for your personal use only) or project directory (if it will be shared with other team members)
  • Unzip the MADflow.ca zip file
  • Change the Linux-cluster/madflow folder which has been created
  • Open the madflow shell script with a text editor

Assuming you downloaded the file to your $HOME directory, and you want to create the MADflow.ca software in your project folder named comet_myproject, the following steps should work: 

$ cd /nobackup/proj/comet_myproject
$ mkdir MADflow.ca
$ cd MADflow.ca
$ unzip $HOME/madflow.zip
$ cd Linux-cluster/madflow
$ nano madflow

At this point you will have the madflow script open in your editor: 

#!/bin/bash
# MADflow launcher for clusters whose compute nodes cannot run the engine
# directly (e.g. Red Hat 9): runs it inside an Ubuntu 24.04 container with
# Apptainer. Put this file in the MADflow folder, the engine and its key in
# engine/ next to it, set IMAGE below, and make both executable once:
#     chmod +x madflow engine/madflow
# See ReadMe.txt in this folder, step (3b).
#
# IMAGE  the site's Ubuntu 24.04 container image (.sif)
# BIND   extra folders the container must see, e.g. /scratch,/project
#        (your home directory and the current folder are always visible)
# --nv   gives the engine the node's NVIDIA GPU (driver 550 or newer)

IMAGE=/image.sif
BIND=
here=$(dirname "$(readlink -f "$0")")
exec apptainer exec --nv ${BIND:+--bind "$BIND"} "$IMAGE" "$here/engine/madflow" "$@"

Make the following changes to the madflow script to use the Ubuntu container we have provided (the IMAGE variable) and to map the standard Comet filesystem locations (the BIND variable). Note we also need to load the apptainer module to ensure it is available at runtime (module load apptainer): 

#!/bin/bash
# MADflow launcher for clusters whose compute nodes cannot run the engine
# directly (e.g. Red Hat 9): runs it inside an Ubuntu 24.04 container with
# Apptainer. Put this file in the MADflow folder, the engine and its key in
# engine/ next to it, set IMAGE below, and make both executable once:
#     chmod +x madflow engine/madflow
# See ReadMe.txt in this folder, step (3b).
#
# IMAGE  the site's Ubuntu 24.04 container image (.sif)
# BIND   extra folders the container must see, e.g. /scratch,/project
#        (your home directory and the current folder are always visible)
# --nv   gives the engine the node's NVIDIA GPU (driver 550 or newer)

# Path to the Ubuntu container for MADflow on Comet
IMAGE=/nobackup/shared/containers/madflow.sif
# Add the standard /nobackup and /scratch filesystems
BIND=/nobackup,/scratch

here=$(dirname "$(readlink -f "$0")")

# Needed to make the apptainer commands available
module load apptainer

exec apptainer exec --nv ${BIND:+--bind "$BIND"} "$IMAGE" "$here/engine/madflow" "$@"

You are now ready to use one of the example Slurm job files below as the basis of your own job file.

Example Slurm Job

All examples assume you have installed MADflow.ca software as per the instructions above, and you are running from the comet_myproject project directory. Adjust to your own installation path and own project account code accordingly.

CPU Job

Use standard CPU based compute for MADflow to process a single input file.

Note:

  • If you want to use the reserved CPU cores in the paid partition, then substitute the default_paid partition name - but you must have a paid/funded HPC project.
  • Below example allocates 16 CPU cores, you may need to set a different value, e.g. 
    #SBATCH -c 64
    for a 64 core example.
  • Default memory allocation (1GB), you may need to set a different value, e.g. 
    #SBATCH --mem 16G
    for a 16GB RAM example.

#!/bin/bash
#SBATCH --job-name=madflow_cpu
#SBATCH --partition=default_free
#SBATCH --account=comet_myproject
#SBATCH --cpus-per-task=16
#SBATCH --nodes=1
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@120
# Assuming all of your input files are stored here
cd "/nobackup/proj/comet_myproject"
# A single job with 16 cores
exec /nobackup/proj/comet_myproject/MADflow.ca/Linux-cluster/madflow/madflow my_model -p 16

CPU Job (array task)

Use standard CPU based compute for MADflow to process multiple input files in parallel.

Note:

  • If you want to use the reserved CPU cores in the paid partition, then substitute the default_paid partition name - but you must have a paid/funded HPC project.
  • Below example runs 20 parallel tasks, each with a unique input file. The default approach below assumes each of the input files is numbered sequentially.
  • Below example allocates 16 CPU cores per task (16 x 20 = 320 CPU cores), you may need to set a different value, e.g. 
    #SBATCH -c 64
    for a 64 core example.
  • Default memory allocation (1GB), you may need to set a different value, e.g. 
    #SBATCH --mem-per-cpu 1G
    to allocate 1GB per CPU core.

#!/bin/bash
#SBATCH --job-name=madflow_cpu
#SBATCH --partition=default_free
#SBATCH --account=comet_myproject
#SBATCH --array=1-20
#SBATCH --cpus-per-task=16
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@120
# Assuming all of your input files are stored here
cd "/nobackup/proj/comet_myproject"  
# Use one job per model, with a job array (input files should be named case_1.ini ... case_20.ini) - this is 20 parallel jobs, each with 16 cores
exec /nobackup/proj/comet_myproject/MADflow.ca/Linux-cluster/madflow/madflow case_${SLURM_ARRAY_TASK_ID} -p 16

GPU Job

Request a single L40-S GPU in the gpu-s_free partition to use Nvidia GPU based compute acceleration for MADflow.

Note:

  • If you want to use the reserved cards in the paid partition, then substitute the gpu-s_paid partition name - but you must have a paid/funded HPC project.
  • If you want to use the H100 cards, then substitute the gpu-l_paid partition name, and 
    #SBATCH --gres=gpu:H100:1
    for the resource type instead - but you must have a paid/funded HPC project.
  • Default CPU core allocation (1 core), you may need to set a different value, e.g. 
    #SBATCH -c 8
    for an 8 core example.
  • Default memory allocation (2GB), you may need to set a different value, e.g. 
    #SBATCH --mem 16G
    for a 16GB RAM example.
  • You can combine the task array based CPU example with GPU to launch parallel GPU compute jobs, but free HPC projects are subject to simultaneous GPU use restrictions that make this impractical if you are not a paid/funded HPC project member.

#!/bin/bash
#SBATCH --job-name=madflow_gpu
#SBATCH --partition=gpu-s_free
#SBATCH --account=comet_myproject
#SBATCH --gres=gpu:L40:1
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@120
# Assuming all of your input files are stored here
cd "/nobackup/proj/comet_myproject"
# Single GPU job
exec /nobackup/proj/comet_myproject/MADflow.ca/Linux-cluster/madflow/madflow my_model -gpu 0

Back to Advanced Software index