Why is my Slurm job stuck with reason PartitionNodeLimit?

If you submit a job and it gets stuck in state PENDING with the reason code PartitionNodeLimit, then the Slurm documentation describes that code as follows:

PartitionNodeLimit
The number of nodes required by this job is outside of its partition's current limits. Can also indicate that required nodes are DOWN or DRAINED.

Taken from: https://slurm.schedmd.com/squeue.html

In most cases, this will be because your script has asked for more compute nodes than are available within the partition you selected. You can get a summary of the available resources across all nodes in a partition as follows - in the case of the default_free partition:

$ sinfo -p default_free -N -l
NODELIST    NODES    PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
compute009      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute010      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute011      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute012      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute013      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute014      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute015      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute016      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute017      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute018      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute019      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute020      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute021      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute022      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute023      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute024      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute025      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute026      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute027      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute028      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute029      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute030      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute031      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute032      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute033      1 default_free    drained* 256   2:128:1 150000        0     10   (null) In maintenance           
compute034      1 default_free        idle 256   2:128:1 150000        0     10   (null) none                
compute035      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute036      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
hmem009         1 default_free   allocated 256   2:128:1 150000        0      1   (null) none                
hmem010         1 default_free   allocated 256   2:128:1 150000        0      1   (null) none
$

However, there is a scenario where your job may inadvertently request more compute nodes than you realise.

If you are using the --ntasks parameter without using the --nodes parameter, then Slurm may calculate the number of nodes you require to be different to what you expect. The Slurm documentation states:

-n, –ntasks=<number>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the Slurm controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. If the –nodes option is not used, this option is used to implicitly determine the node count. The default is one task per node, but note that the --cpus-per-task option will change this default.

In some cases, we have seen examples where a user has set --ntasks=64 without --nodes, and Slurm turns this into a resource request for 1 to 64 nodes, and thus beyond what is available in the partition.

If you want to check whether it is the calculated number of nodes which is putting your job into this state, you can check using scontrol, e.g:

$ scontrol show job JOB_ID
...
...
   NumNodes=1-64 NumCPUs=64 NumTasks=64 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=64,mem=128G,node=1,billing=64
   AllocTRES=()
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=128G MinTmpDiskNode=0
...
$

If the NumNodes value is not what you expected, then it is likely due to the missing --nodes parameter.

Be explicit - always use the --nodes parameter if you are using --ntasks.

Back to FAQ index