====== Ansys Forte ======

[[https://www.ansys.com/en-gb/products/fluids/ansys-forte|ANSYS Forte]] is included in the University ANSYS license package, it is normally installed as part of the larger ANSYS software module. e.g:

<code lang=shell-session>
$ module avail ANSYS

------------------------------------------------------------ /mnt/storage/apps/eb/modules/all ------------------
   ANSYS/17.0    ANSYS/18.1    ANSYS/19.4    ANSYS/2020-R2    ANSYS/2021    ANSYS/2022-R1    ANSYS/2024R1 (D)

  Where:
   D:  Default Module
$
</code>

You should normally design your project on your desktop or workstation. The use of the HPC facility is generally intended to run analysis on existing projects, though for **Comet**, you may also choose to (optionally) run the interactive ANSYS Workbench on HPC via [[advanced:interactive|Open OnDemand]] in your browser.

----

===== Running =====

The path to all ANSYS components is //not// set when loading the ANSYS module (it adds a //lot// to the PATH, so not practical to do this for every component); **Forte** is one of the products which is not brought into the path.

To work around this on Ansys 2024R1:

<code lang=shell-session>
$ module load ANSYS/2024R1
$ export PATH=$PATH:/mnt/storage/apps/eb/software/ANSYS/2024R1/v241/reaction/forte.linuxx8664/bin
$ source /mnt/storage/apps/eb/software/ANSYS/2024R1/v241/reaction/forte.linuxx8664/bin/forte_setup.ksh
</code>

The Ansys Forte directory is now in your path and the correct environment variables have been set by ''forte_setup.ksh''. 

<WRAP box tip round half>
Use ''forte_setup.csh'' if you are using ''csh'' instead of ''bash'' for your HPC shell.
</WRAP>

----

===== Sample Job Scripts =====

All ANSYS Forte jobs run via MPI (even restricted to a single node), however the simplest version running on a single node requires a lot less Slurm headers to be set. For maximum performance consider the //multi node// version, but note that you may need to run several iterations in order to establish the most optimal resource settings for your Slurm job script.

=== Single Node ===

This method of running ANSYS Forte is about the smallest allocation you can get away with. The recipe allocates the following:

   * A single compute node (//no// ''--nodes='' option)
   * Two tasks per node, the minimum required to run Forte (''--ntasks-per-node=2'')
   * Four CPU cores per task (''--cpus-per-task=4'')
   * 8GB RAM (''--mem=8G'')
   * Up to 8 hours of runtime
   * The **INPUT_FILE** //Dual_Fuels_82percentGasoline_AMG_Tutorial.ftsim//

<WRAP box info round half>
Whilst Forte can run on a single node, you must __still__ allocate at least two //tasks// in Slurm. Set your Slurm resource values (CPU, RAM) accordingly to the complexity of your project.
</WRAP>

<code lang=bash title=forte_mpi.sh>
#!/bin/bash

# Request 1 core
#SBATCH --cpus-per-task=4
#SBATCH --ntasks-per-node=2
# Set requested amount of memory
#SBATCH --mem=8G
# Request 8 hours of runtime
#SBATCH --time=08:00:00
# Set partition/queue
# defq allows up to 2 days of runtime
#SBATCH --partition=defq

# Load ANSYS and add on the Forte runtime directory
module load ANSYS/2024R1
export PATH=$PATH:/mnt/storage/apps/eb/software/ANSYS/2024R1/v241/reaction/forte.linuxx8664/bin
source /mnt/storage/apps/eb/software/ANSYS/2024R1/v241/reaction/forte.linuxx8664/bin/forte_setup.ksh

forte -i JOB_INPUT_FILE \
      -rn Run_001 \
      -o JOB_OUTPUT_FILE \
      -$FORTE_ARGS </dev/null >MONITOR 2>&1
</code>

Note that the amount of RAM and CPU cores allocated to each task will be dependent on:

   * The size and complexity of your model
   * The number of processors allowed under the ANSYS licensing model

=== Multi Node ===

   * **Three** compute node (''--nodes=3'')
   * **Four** tasks per node; two are the minimum required to run Forte (''--ntasks-per-node=4'')
   * **Eight** CPU core per task (''--cpus-per-task=8'')
   * **8GB** RAM per node (''--mem=8G'')
   * Up to **24 hours** of runtime using the **defq** partition
   * The **INPUT_FILE** //Dual_Fuels_82percentGasoline_AMG_Tutorial.ftsim//

This configuration allows for an aggregate total of //24// processors being used by ANSYS Forte. Increase or reduce your Slurm resource values according to the complexity of your Forte project:

<code lang=bash title=forte_mpi.sh>
#!/bin/bash

# Request 3 physical servers
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=8
#SBATCH --mem=8G
#SBATCH --time=24:00:00
#SBATCH --partition=defq

echo ""
echo "Started at: `date`"

echo ""
echo "Slurm: Using $SLURM_CPUS_PER_TASK CPU cores per task"
echo "Slurm: Using $SLURM_TASKS_PER_NODE task per node"
echo "Slurm: Using $SLURM_JOB_NUM_NODES nodes"
echo "Slurm: $SLURM_CPUS_PER_TASK * $SLURM_TASKS_PER_NODE * $SLURM_JOB_NUM_NODES = Total CPU"

# Load ANSYS and add on the Forte runtime directory
echo ""
echo "Loading ANSYS module"
module load ANSYS/2024R1
export PATH=$PATH:/mnt/storage/apps/eb/software/ANSYS/2024R1/v241/reaction/forte.linuxx8664/bin
source /mnt/storage/apps/eb/software/ANSYS/2024R1/v241/reaction/forte.linuxx8664/bin/forte_setup.ksh

# Load Intel MPI runtime
module load intel

# Get the list of nodes we have been allocated
echo ""
echo "Extracting Slurm hostnames..."
HOSTNAMES=`scontrol show hostnames $SLURM_JOB_NODELIST`
echo "Job will run on:
$HOSTNAMES"

# Set to your Forte input file
INPUT_FILE=Dual_Fuels_82percentGasoline_AMG_Tutorial.ftsim
OUTPUT_FILE=jobOutput.txt

echo ""
echo "Calling Ansys Forte"
# We generate this string so that we can record the actual command we are going to run
cmd="forte -i $INPUT_FILE -rn Run_001 -o $OUTPUT_FILE"
echo "Running: $cmd"

# Run the command script
srun $cmd

echo ""
echo "Finished at: `date`"
</code>

This method of running ANSYS Forte allows you to scale the compute resources to almost any level necessary, but the Slurm job script is somewhat more complex due to the need to identify which hosts/nodes are allocated for your job, and then dynamically generating the ''cmd.sh'' file which is launched via ''srun''.

It appears that ANSYS Forte uses one of the tasks launched as a //communication// process for all other compute processes. Hence launching with //two// tasks only has one actively processing. Your total tasks (''nodes'' * ''tasks_per_node'') should be specified with this in mind.

<WRAP box round important>
You may need to do some performance profiling to understand whether it is better to have __more__ tasks with fewer cpu cores per task, or __fewer__ tasks with more cpu cores per task, e.g:

   * ''--ntasks-per-node=4'' //and// ''--cpus-per-task=8''

//or//...

   * ''--ntasks-per-node=32'' //and// ''--cpus-per-task=1''

This represents the //same// number of cores, but presented via a //different// mix of tasks.
</WRAP>

----

[[advanced:software|Back to Advanced Software Index]]