====== Building a Parallel Task Array Solution ====== This is an //example// scenario, but the principles of how you would approach it (e.g. first building a pipeline to process a single input file, understanding your input data structure, and finally converting your pipeline to a task array) are applicable to __all__ problems which are based on the analysis of many //independent// data files. ===== Steps to a solution ===== It's not often feasible to jump straight to a parallel task array solution, and we encourage HPC users to approach the problem in several stages: * Identify the problem * Build a single job solution * Test the single job process * Refactoring your input data * Building a multi job solution * Testing a multi job solution * Scaling up & resource limits ---- ===== Identify the problem ===== In this example scenario we are faced with the following problem of analysing a large amount of image data: > We need to analyse files generated by an instrument. > We need to transform each image to remove uncessary data or to highlight certain features. > We have been provided with //tens of thousands// of images, all of which need to be processed before we can generate some metrics from the image. In our example we will use the [[https://zenodo.org/records/7711810#.ZAm3k-zMKEA|EuroSAT RGB dataset]] as our dataset. This is large enough to demonstrate the file //quantity// issue (over 27000 images), but each image is //small enough// to not require substantial processing time. In reality you will probably have datasets that are both large in number, //as well as// in complexity/size. The processing of image data could mean anything, it may involve heuristics to identify features, the application of some image transformation algorithm or similar. //But//, in our case, let us //assume// that the workflow for analysing each input image is a relatively simple sequence of: * Converting the input file from RGB to greyscale * Extracting the min, max and median brightness of all pixels in the image * Recording the brightness values of the image filename for later analysis We know that processing thousands of these files //could// be performed on a local desktop, but it may take days or weeks depending on the number (and size) of files we need to analyse. We want to offload the processing to the HPC so that we don't need to keep our local machine running 24x7. **Following this example** If you want to work through this example, you can download the EuroSAT RGB image dataset and set it up with the following script: $ wget -nc -q https://zenodo.org/records/7711810/files/EuroSAT_RGB.zip?download=1 -O EUROSAT.zip $ unzip -q -n EUROSAT.zip The resulting directory tree should look like this: * EuroSAT_RGB * EuroSAT_RGB/Forest * EuroSAT_RGB/Highway * EuroSAT_RGB/Residential * ... Each of the directories should contain several //thousand// images, such as: {{:advanced:eurosat_examples.png?1000|}} ---- ===== A single job solution ===== The first step is to build an image processing pipeline for a //single// file which performs each step of the workflow. Remember we need to: * Grab the name of the input image file * Convert the file from a colour RGB image to greyscale * Calculate the minimum, maximum and mean brightness of the image * Save the brightness value and filename for later processing So each step of the pipeline should produce, in memory, something similar to this: **Step 1.** Load //source// image ''Residential_78.jpg'' {{:advanced:image_a.jpg?256|}} **Step 2.** Convert RGB to //greyscale// {{:advanced:image_b.jpg?256|}} **Step 3.** Detect and extract the //minimum//, //maximum// and //mean// brightness levels {{:advanced:image_c.jpg?384|}} Here we have written a very basic Python application which takes the name of an input filename (''-file''), and the name of an output directory (''-out''). It then uses the ''PIL'' module to do the image transformation we mandated, and then extracts the min, max and mean brightness of the image, finally it saves an output file containing the filename and the pixel values we extracted, to be used in later processing. Download the file below and save it as ''image_processor.py'': #!/usr/bin/env python3 import argparse import sys import os from PIL import Image, ImageStat def process_an_image(filename = None, outdir = None): """ Process an image """ print(f"Opening source image: {filename}") i = Image.open(filename) # Convert to greyscale print(f"Converting to greyscale") i_greyscale = i.convert('L') # Extract pixel data print(f"Extracting pixel data") i_stat = ImageStat.Stat(i_greyscale) # Save metadata for later processing data = f"filename:{filename} min:{i_stat.extrema[0][0]} max:{i_stat.extrema[0][1]} mean:{i_stat.mean[0]}" output_filename = os.path.split(filename)[1] out_name = outdir + "/" + output_filename + ".txt" print(f"Saving pixel data as: {out_name}") f = open(out_name, "w") f.write(data) f.close # Exit sys.exit(0) if __name__ == "__main__": parser = argparse.ArgumentParser("image_processor") parser.add_argument("-file", help="Name of file to process", type=str) parser.add_argument("-out", help="Name of output directory", type=str) args = parser.parse_args() filename = None outdir = None if args.file: filename = args.file if args.out: outdir = args.out if filename and outdir: process_an_image(filename, outdir) else: print("You need to provide -file and -out arguments.") print("See: ./image_processor -h") sys.exit(1) As this is a relatively lightweight process, on a single image file, we can test this out on a single image file from the EuroSAT dataset by running directly on one of the login nodes as follows: # Load the Python module to ensure we are not relying on a system version of Python $ module load Python/3.12.3 # Create a directory to hold our output data $ mkdir metadata # Now run our image processor on a single file $ python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_1.jpg -out metadata Opening source image: EuroSAT_RGB/Residential/Residential_1.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_1.jpg.txt $ Checking the output file we can see it has recorded the min/max/mean pixel brightness values as intended: $ cat metadata/Residential_1.jpg.txt filename:EuroSAT_RGB/Residential/Residential_1.jpg min:57 max:253 mean:82.02685546875 $ In terms of sequence of operations, this looks like: stateDiagram-v2 Open_Residential_1.jpg --> Process_Residential_1.jpg Process_Residential_1.jpg --> Save_Metadata We have tested this directly at the command line, next we should check it functions correctly under Slurm. ---- ===== Testing a single job ===== You can now run this as a Slurm job. Here's a simple job script which would process a single image file, just as we did at the command line above. Save the contents of the file below as ''image_job.sh'', and submit as ''sbatch image_job.sh'': #!/bin/bash #SBATCH --partition=short_free #SBATCH --account=comet_abc123 # Remember to use your own account code #SBATCH --ntasks=1 #SBATCH --mem=1g #SBATCH -t 00:02:00 module load Python/3.12.3 python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_1.jpg -out metadata This should work //identically// to running it at the command line, though we have to wait for the job to be scheduled. In theory if we were processing //very// large files then we may have had to submit the Slurm job //anyway//, as the login node may not have had enough RAM to handle even a single large image. Let's submit the simple job file and check that it //does// work as intended: $ sbatch image_job.sh Submitted batch job 1607386 $ cat slurm-1607386.out Opening source image: EuroSAT_RGB/Residential/Residential_1.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_1.jpg.txt $ cat metadata/Residential_1.jpg.txt filename:EuroSAT_RGB/Residential/Residential_1.jpg min:60 max:155 mean:92.23583984375 $ So **yes**, it works as well under Slurm as it did directly at the command line, and the sequence of operations is similar to before: stateDiagram-v2 Slurm_start_task --> Open_Residential_1.jpg Open_Residential_1.jpg --> Process_Residential_1.jpg Process_Residential_1.jpg --> Save_Metadata If we didn't have many images to process, we //could// now just add the additional images in to the script and have them processed in turn, for example: #!/bin/bash #SBATCH --partition=short_free #SBATCH --account=comet_abc123 #SBATCH --ntasks=1 #SBATCH --mem=1g #SBATCH -t 00:02:00 module load Python/3.12.3 python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_1.jpg -out metadata python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_2.jpg -out metadata python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_3.jpg -out metadata python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_4.jpg -out metadata python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_5.jpg -out metadata However, there are __more than 27000__ images included in the EuroSAT data set, so it quickly becomes impractical to both list them all individually, and the time to finish increases linearly as we add each image processing command. If we were to run like that, the sequence would start to look like this: stateDiagram-v2 Slurm_start_task --> Open_Residential_1.jpg Open_Residential_1.jpg --> Process_Residential_1.jpg Process_Residential_1.jpg --> Save_Metadata1 Save_Metadata1 --> Open_Residential_2.jpg Open_Residential_2.jpg --> Process_Residential_2.jpg Process_Residential_2.jpg --> Save_Metadata2 Save_Metadata2 --> Open_Residential_3.jpg Open_Residential_3.jpg --> Process_Residential_3.jpg Process_Residential_3.jpg --> Save_Metadata3 So it would get //very// long, and //many// times slower. It would certainly __work__, but is not the best use of resources or time. **However**, we do know our analysis pipeline //works//. This is the first step complete. ---- ===== Refactoring your input data ===== We know the image processing pipeline works for a single input file, but you are reading this guide because you presumably have hundreds or thousands of files to analyse. So how do we do that? Well, at this point there is a decision to be made, and the ease of implementation will vary depending on the situation you find yourself in with your input data, as described in the two options below: **Scenario 1** Your files are __already__ sequentially named, or have __complete control__ over your input data files naming convention and can set them to be named or numbered sequentially. {{:advanced:files_sequential.png?700|}} **Scenario 2** You do __not__ have control over your input data files, they are __not__ sequentially named, or they __already exist__ with some naming convention which needs to be __maintained__. {{:advanced:files_nonsequential.png?650|}} ==== Scenario 1 - Control of file naming ==== By far the easiest mechanism of moving to a task array based solution is if you have control over your input data naming convention. If you are able to arrange your input data files with **sequential filenames**, then it becomes almost trivial to convert to a task array. By sequential naming, we refer to a filename which has some part of the name numbered in sequence - it does //not necessarily need to start from 1//. Example sequential numbering schemes: * ''file1.jpg'', ''file2.jpg'', ''file3.jpg'' * ''data.1.dat'', ''data.2.dat'', ''data.3.dat'' * ''sequence_data.5000.seq'', ''sequence_data.5001.seq'', ''sequence_data.5002.seq'' * ''1001.txt'', ''1002.txt'', ''1003.txt'' * etc. If you already have files named like this, //or can rename them to be like this//, then you will find that you can easily convert your ''image_job.sh'' job file to a task array with almost zero effort. We'll show this below. ==== Scenario 2 - Existing file naming conventions ==== If you have directory trees of input files, naming schemes that are random or otherwise unable to be ordered sequentially by number, then you will need to generate an additional **input file** where those files //can// be read sequentially. Although this sounds complicated, it is actually quite simple. If you have a single directory of files, you can use the ''ls'' command and //redirection// to send all of your filenames to an input file: Assuming we wanted all of the files in the ''EuroSAT_RGB/Residential'' directory we could do: $ ls EuroSAT_RGB/Residential/*.jpg | sort > residential_files.txt This command: * Lists all of the files ending in ''.jpg'' in the ''EuroSAT_RGB/Residential'' directory * Sorts them by name * Send the sorted list to a file named ''residential_files.txt'' Looking at the top of the ''residential_files.txt'' file we can see the files are now listed: $ head residential_files.txt EuroSAT_RGB/Residential/Residential_1000.jpg EuroSAT_RGB/Residential/Residential_1001.jpg EuroSAT_RGB/Residential/Residential_1002.jpg EuroSAT_RGB/Residential/Residential_1003.jpg EuroSAT_RGB/Residential/Residential_1004.jpg EuroSAT_RGB/Residential/Residential_1005.jpg EuroSAT_RGB/Residential/Residential_1006.jpg EuroSAT_RGB/Residential/Residential_1007.jpg EuroSAT_RGB/Residential/Residential_1008.jpg EuroSAT_RGB/Residential/Residential_1009.jpg $ ... and it has all //3000// files: $ wc -l residential_files.txt 3000 residential_files.txt $ **We will use this ''residential_files.txt'' input file later, in our task array job**. If we //cannot// rename our files to be sequentially numbered, then this input file is the best way of moving to a task array based setup. If you have complicated subdirectory trees with your input files spread across multiple directories, then you may need to use something like ''find'' to list your files, rather than the simple ''ls'' command. ---- ===== Building a task array solution ===== How you write a task array job file is determined by whether you have sequentially named input data, or are using the input data file method listed above. ==== Task array solution - sequential filenames ==== To convert our existing ''image_job.sh'' job script to a task array, we can make the following modifications: * Add the ''--array='' parameter, which tells Slurm the //quantity// and //numbers// of the tasks we want to run - this range should correspond to the sequential numbering range of your input files The //numbers// of the tasks Slurm will start for us get turned in to the variable ''${SLURM_ARRAY_TASK_ID}''. In the //small// example below, we are asking Slurm to launch **four** tasks and each task will get the number from **1 to 4** (''--array=1-4''). Save the file below as ''image_job_seq.sh'' and run as ''sbatch image_job_seq.sh'': #!/bin/bash #SBATCH --partition=default_free #SBATCH --account=comet_abc123 # Remember to use your own account code #SBATCH --mem=1g #SBATCH --nodes=1 #SBATCH --tasks=4 #SBATCH --array=1-4 #SBATCH --cpus-per-task=1 module load Python/3.12.3 python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_${SLURM_ARRAY_TASK_ID}.jpg -out metadata We use the fact each task gets a unique number to make each task //also// use a differently named //input image file//. In this case Slurm will start the four tasks and the commands run by each one will be one of: python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_1.jpg -out metadata python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_2.jpg -out metadata python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_3.jpg -out metadata python3 image_processor.py -file EuroSAT_RGB/Residential/Residential_4.jpg -out metadata You don't need to do anything else - Slurm will automatically take care of launching each ''image_processor.py'' task with a differently named input file, and since we wrote ''image_processor.py'' to work completely independently of any other file, there will be no conflicts and each task will run simultaneously with the other three. In terms of sequence, it looks like this: stateDiagram-v2 Submit_Job --> Start_Tasks Start_Tasks --> Open_Residential_1.jpg Start_Tasks --> Open_Residential_2.jpg Start_Tasks --> Open_Residential_3.jpg Start_Tasks --> Open_Residential_4.jpg Open_Residential_1.jpg --> Process_Residential_1.jpg Process_Residential_1.jpg --> Save_Metadata_1 Open_Residential_2.jpg --> Process_Residential_2.jpg Process_Residential_2.jpg --> Save_Metadata_2 Open_Residential_3.jpg --> Process_Residential_3.jpg Process_Residential_3.jpg --> Save_Metadata_3 Open_Residential_4.jpg --> Process_Residential_4.jpg Process_Residential_4.jpg --> Save_Metadata_4 Because each of the tasks executes entirely independently and in parallel, this gives us a **x4** speed up versus processing the four images in sequence. ==== Task array solution - using an input data file ==== To convert our existing ''image_job.sh'' job script to a task array //and instead use an input file//, we can make the following modifications: * Add the ''--array='' parameter, which tells Slurm the //quantity// and //numbers// of the tasks we want to run - this range should correspond to the sequential numbering range of your input files The //numbers// of the tasks Slurm will start for us get turned in to the variable ''${SLURM_ARRAY_TASK_ID}''. In the //small// example below, we are asking Slurm to launch **four** and each task will get the number from **1 to 4** (''--array=1-4''). Save the file below as ''image_job_inputfile.sh'' and run with ''sbatch image_job_inputfile.sh'': #!/bin/bash #SBATCH --partition=default_free #SBATCH --account=comet_abc123 # Remember to use your own account code #SBATCH --mem=1g #SBATCH --nodes=1 #SBATCH --array=1-4 #SBATCH --cpus-per-task=1 module load Python/3.12.3 # This next line generates a unique input data file for each task which is launched, as long as we generated the 'residential_files.txt' input as # per our earlier example. INPUT_FILE=$(awk NR==${SLURM_ARRAY_TASK_ID} residential_files.txt) # Then each task launches image_processor with a unique INPUT_FILE name python3 image_processor.py -file $INPUT_FILE -out metadata Because we don't have sequentially numbered input files we use a little trick with the ''awk'' command to read a specific line from our input file. In //this case// each task reads its own line from the input file; task **one** reading the filename on //line one//, task **two** reading the filename on //line two//, and so on. The sequence of operations is //exactly// the same as the sequentially named file example, as we have not changed anything other than the method that the filenames are retrieved: stateDiagram-v2 Submit_Job --> Start_Tasks Start_Tasks --> Open_Residential_1.jpg Start_Tasks --> Open_Residential_2.jpg Start_Tasks --> Open_Residential_3.jpg Start_Tasks --> Open_Residential_4.jpg Open_Residential_1.jpg --> Process_Residential_1.jpg Process_Residential_1.jpg --> Save_Metadata_1 Open_Residential_2.jpg --> Process_Residential_2.jpg Process_Residential_2.jpg --> Save_Metadata_2 Open_Residential_3.jpg --> Process_Residential_3.jpg Process_Residential_3.jpg --> Save_Metadata_3 Open_Residential_4.jpg --> Process_Residential_4.jpg Process_Residential_4.jpg --> Save_Metadata_4 This therefore gets exactly the same **x4** speed up as the previous example. The only difference is the potential order the files are processed and that they don't need to be sequentially named. This achieves the same goal as the sequentially numbered task array example (four tasks launched, each reading a different input file), and while the ''awk'' line is a //little opaque// for those who have not written a great deal of shell script, you can treat it as an automatic filename supplier, as long as you give it your input file (in this case, ''residential_files.txt'' we created earlier). ---- ===== Testing a multi job solution ===== ==== Testing the sequential filename solution ==== First, let's test the sequentially numbered task array solution: $ sbatch image_job_seq.sh Submitted batch job 1607653 # Check for output logs $ ls slurm-1607653* slurm-1607653_1.out slurm-1607653_2.out slurm-1607653_3.out slurm-1607653_4.out # See contents of output logs $ cat slurm-1607653* Opening source image: EuroSAT_RGB/Residential/Residential_1.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_1.jpg.txt Opening source image: EuroSAT_RGB/Residential/Residential_2.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_2.jpg.txt Opening source image: EuroSAT_RGB/Residential/Residential_3.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_3.jpg.txt Opening source image: EuroSAT_RGB/Residential/Residential_4.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_4.jpg.txt $ Has the data been generated correctly? $ ls metadata/ Residential_1.jpg.txt Residential_2.jpg.txt Residential_3.jpg.txt Residential_4.jpg.txt $ cat metadata/Residential_1.jpg.txt filename:EuroSAT_RGB/Residential/Residential_1.jpg min:60 max:155 mean:92.23583984375 $ cat metadata/Residential_2.jpg.txt filename:EuroSAT_RGB/Residential/Residential_2.jpg min:54 max:254 mean:115.12548828125 $ cat metadata/Residential_3.jpg.txt filename:EuroSAT_RGB/Residential/Residential_3.jpg min:62 max:255 mean:102.17138671875 $ cat metadata/Residential_4.jpg.txt filename:EuroSAT_RGB/Residential/Residential_4.jpg min:56 max:158 mean:90.14990234375 $ **Yes**, the four files which were processed in parallel were analysed successfully, using our existing, sequential ''image_process.py'' application. ==== Testing the input data file solution ==== Now, let's test the solution which uses the input data file, and does not rely on sequentially named image files. # Submit job $ sbatch image_job_inputfile.sh Submitted batch job 1607718 # Did Slurm logs get created for each task? $ ls slurm-1607718_* slurm-1607718_1.out slurm-1607718_2.out slurm-1607718_3.out slurm-1607718_4.out # Check Slurm output logs $ cat slurm-1607718_* Opening source image: EuroSAT_RGB/Residential/Residential_1000.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_1000.jpg.txt Opening source image: EuroSAT_RGB/Residential/Residential_1001.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_1001.jpg.txt Opening source image: EuroSAT_RGB/Residential/Residential_1002.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_1002.jpg.txt Opening source image: EuroSAT_RGB/Residential/Residential_1003.jpg Converting to greyscale Extracting pixel data Saving pixel data as: metadata/Residential_1003.jpg.txt $ So it seems to have ran okay (note that it processed the files in a different order, based on the ''sort'' method with used to generate ''residential_files.txt''). What about the analysed data files? # Did the output data files get created? $ ls metadata/ Residential_1000.jpg.txt Residential_1001.jpg.txt Residential_1002.jpg.txt Residential_1003.jpg.txt # Check output data $ cat metadata/Residential_1000.jpg.txt filename:EuroSAT_RGB/Residential/Residential_1000.jpg min:47 max:218 mean:94.780517578125 $ cat metadata/Residential_1001.jpg.txt filename:EuroSAT_RGB/Residential/Residential_1001.jpg min:61 max:254 mean:85.656982421875 $ cat metadata/Residential_1002.jpg.txt filename:EuroSAT_RGB/Residential/Residential_1002.jpg min:66 max:254 mean:114.40478515625 $ cat metadata/Residential_1003.jpg.txt filename:EuroSAT_RGB/Residential/Residential_1003.jpg min:59 max:153 mean:79.7548828125 Again, **yes**; the four parallel tasks ran successfully, and they each generated (presumably) correct data from their independent image files. We can be confident that both the sequential filename method and the input data file method do produce the //same// output - although the ordering of the image files could differ between the two, depending on how we order/sort the input file. But hold on, we have //thousands// of input files, and we have only processed //four// of them! Well //yes//; as always we should start off small and test that things __work__ before we scale up. Read the //next section// to understand how to scale up your task arrays jobs. ---- ===== Scaling up & resource limits ===== If we look at the ''SBATCH'' headers we set in both ''input_job_seq.'' and ''input_job_inputfile.sh'' they are the same: #SBATCH --partition=default_free #SBATCH --account=comet_abc123 # Remember to use your own account code #SBATCH --mem=1g #SBATCH --nodes=1 #SBATCH --array=1-4 #SBATCH --cpus-per-task=1 How we increase the number of parallel copies of ''image_process.py'' is to configure the ''--array'' values to better suit our needs. Remember that: * ''--array='' controls the quantity of tasks (**4**, in the case above) and the numbering of the ''${SLURM_ARRAY_TASK_ID}'' variable which each task inherits (**1 to 4** in the previous example) It would seem that we could increase ''tasks'' to some //huge// number to run //all// of our input data files in parallel... but we are constrained by several factors: * In the case of Comet __we do not have__ //27000// CPU cores! (each task requires a //minimum// of 1 CPU core) * Even if we did have enough cores to launch an obscene number of tasks, this risks a detrimental impact of the HPC service to all other users! * Each HPC project will have a [[started:comet_resources|resource limit]] on the number of cores and jobs (or tasks) it can submit or have running at any time - a paid project may therefore be able to launch a //higher// number of tasks from an array job than an unfunded project When increasing the ''array'' figure the form below is used: * array=**start**-**end**%**max_running** The ''%max_running'' argument is optional, and if not set, the number of running tasks is the sequence from **start** to **end**, e.g: # Four running tasks, numbered 1, 2, 3, 4 #SBATCH --array=1-4 # Twenty eight running tasks, numbered 100 to 127 #SBATCH --array=100-127 If you //include// the ''%max_running'' argument, then this applies a limit to the simultaneous running tasks, e.g.: # Four tasks, limited to two running at a time, numbered 1, 2, 3, 4 #SBATCH --array=1-4%2 # Twenty eight tasks, limited to five running at a time, numbered 100 to 127 #SBATCH --array=100-127%5 A key point to remember is that Slurm will create a job entry for **each element of your array**. If you have an array which defines **28** entries, then you will get **28** new jobs created... if you have an array with **200** entries then you will create **200** new Slurm job entries. This means that large arrays can create huge numbers of Slurm jobs very, very quickly. For this reason we do currently have some limits set on the number of jobs both users and their entire group can create (see [[:started:comet_resources|Resource Limits - Understanding what they mean]] for more details on the current limits - **MaxSubmitJobs** and **GrpSubmitJobs** limits are applicable to the maximum size of the ''--array'' setting, but also take in to account //everything else// currently running or pending against your account code). If you have //huge// numbers of data files to process, then you may need to approach this in several batches to avoid overload Slurm and the job submission system, and to stay inside the limits enforced on your project (you can use our ''sproject'' tool from [[:started:simple_slurm_tools|Simple Slurm Tools]] to quickly check your project limits). ==== Possible workarounds to array task limits ==== Whilst you may have limits in the number of individual tasks that you can launch as part of a Slurm job (and //most// HPC systems do have limits of some sort), these can easily be worked around with a little bit of extra shell script logic. In the case of the EuroSAT image database, with 27000 files, there is little chance of using being able to launch 27K tasks at the same time, so we have a number of approaches. **Option 1 - Launch multiple Slurm jobs, manually** The simplest method is to split the file list of 27000 files down to more manageable sizes, in batches of 250, 500, 1000 entries or similar - based on what your actual **MaxSubmitJobs** and **GrpSubmitJobs** limits are set to. Launch all 250/500/1000 tasks, wait for them to finish, and then submit another batch. This is the easiest, requires no additional scripting, but does mean that you need to watch for your jobs finishing and then submitting the next batch. This may take some time with tens of thousands, or even //millions// of input files. **Option 2 - Have each array job handle multiple input files** Presently our solution has each array task handle just //one// input file... but there's no reason why we have to keep it like that. We could add a little more logic to the Slurm script and instead have each task process //a list// of files... so although we only launch 250 tasks (//for example//), each one could loop over ''1/250'' of the input files. Here's a //possible// solution for the 27000 file data set we are using in this example. Generate a list of **all input files**: $ find EuroSAT_RGB/ -type f -print > all_files.txt The file ''all_files.txt'' now contains all 27000 files in the dataset: $ wc -l all_files.txt 27000 all_files.txt $ Now use the command line tool ''split'' to create chunks of the ''all_files.txt'' by the number of tasks we will have in our Slurm array, in this case we'll limit it to 250 tasks: $ split --numeric-suffixes=001 -n l/250 -d all_files.txt $ ls download.sh x004 x016 x028 x040 x052 x064 x076 x088 x100 x112 x124 x136 x148 x160 x172 x184 x196 x208 x220 x232 x244 EuroSAT_RGB x005 x017 x029 x041 x053 x065 x077 x089 x101 x113 x125 x137 x149 x161 x173 x185 x197 x209 x221 x233 x245 EUROSAT.zip x006 x018 x030 x042 x054 x066 x078 x090 x102 x114 x126 x138 x150 x162 x174 x186 x198 x210 x222 x234 x246 all_files.txt x007 x019 x031 x043 x055 x067 x079 x091 x103 x115 x127 x139 x151 x163 x175 x187 x199 x211 x223 x235 x247 image_job_inputfile.sh x008 x020 x032 x044 x056 x068 x080 x092 x104 x116 x128 x140 x152 x164 x176 x188 x200 x212 x224 x236 x248 image_job_seq.sh x009 x021 x033 x045 x057 x069 x081 x093 x105 x117 x129 x141 x153 x165 x177 x189 x201 x213 x225 x237 x249 image_job.sh x010 x022 x034 x046 x058 x070 x082 x094 x106 x118 x130 x142 x154 x166 x178 x190 x202 x214 x226 x238 x250 image_processor.py x011 x023 x035 x047 x059 x071 x083 x095 x107 x119 x131 x143 x155 x167 x179 x191 x203 x215 x227 x239 README.txt x012 x024 x036 x048 x060 x072 x084 x096 x108 x120 x132 x144 x156 x168 x180 x192 x204 x216 x228 x240 x001 x013 x025 x037 x049 x061 x073 x085 x097 x109 x121 x133 x145 x157 x169 x181 x193 x205 x217 x229 x241 x002 x014 x026 x038 x050 x062 x074 x086 x098 x110 x122 x134 x146 x158 x170 x182 x194 x206 x218 x230 x242 x003 x015 x027 x039 x051 x063 x075 x087 x099 x111 x123 x135 x147 x159 x171 x183 x195 x207 x219 x231 x243 This gives us **250** files, each with a portion of the files listed in ''all_files.txt''. The files are also sequentially numbered (in this case ''x001'' through to ''x250'' - perfectly matching the number format used by the Slurm array tasks). With a little bit of modification to our earlier sequential input data file Slurm job (e.g. the one //not// using ''awk''), we can make each of the 250 tasks we run read from their own input files: #!/bin/bash #SBATCH --partition=default_free #SBATCH --account=comet_training # Remember to use your own account code #SBATCH --mem=1g #SBATCH --nodes=1 #SBATCH --array=1-250 #SBATCH --cpus-per-task=1 module load Python/3.12.3 # Turn the non-zero padded SLURM_ARRAY_TASK_ID (e.g. 7) # into a zero-padded string which matches the filenames (e.g. x007) INPUT_DATA_FILE="x`printf %03d ${SLURM_ARRAY_TASK_ID}`" # Read from the x001 - x250 input file, which contains a portion of ALL the input files we need to process cat ${INPUT_DATA_FILE} | while read INPUT_FILENAME do python3 image_processor.py -file ${INPUT_FILENAME} -out metadata done So this allows us to start tasks up to the limit allowed by our Slurm account code (in this case we assume **250** is our limit - see the output from ''sproject'' for your Slurm account for your real limit), and each task will then individually process multiple files in //it's own short loop//, until each one is completed. In this //specific// case, it means each running task will process **108** input files (''27000 / 250'') in the short loop section. ---- ===== Downloads ===== You can download all of the scripts used in this example using the link below: * {{ :advanced:slurm_task_array_example.zip |}} ----- [[:advanced:slurm|Back to Advanced Slurm Job Optimisation]]