====== Why is my Slurm job stuck with reason PartitionNodeLimit? ======

If you submit a job and it gets //stuck// in state **PENDING** with the reason code **PartitionNodeLimit**, then the Slurm documentation describes that code as follows:

> **PartitionNodeLimit**
> The number of nodes required by this job is outside of its partition's current limits. Can also indicate that required nodes are DOWN or DRAINED.

   * Taken from: https://slurm.schedmd.com/squeue.html

In //most// cases, this will be because your script has asked for **more** compute nodes than are available within the partition you selected. You can get a summary of the available resources across all nodes in a partition as follows - in the case of the **default_free** partition:

<code lang=bash>
$ sinfo -p default_free -N -l
NODELIST    NODES    PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
compute009      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute010      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute011      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute012      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute013      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute014      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute015      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute016      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute017      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute018      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute019      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute020      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute021      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute022      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute023      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute024      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute025      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute026      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute027      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute028      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute029      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute030      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute031      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute032      1 default_free   allocated 256   2:128:1 150000        0     10   (null) none                
compute033      1 default_free    drained* 256   2:128:1 150000        0     10   (null) In maintenance           
compute034      1 default_free        idle 256   2:128:1 150000        0     10   (null) none                
compute035      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
compute036      1 default_free       mixed 256   2:128:1 150000        0     10   (null) none                
hmem009         1 default_free   allocated 256   2:128:1 150000        0      1   (null) none                
hmem010         1 default_free   allocated 256   2:128:1 150000        0      1   (null) none
$
</code>

However, there is a scenario where your job may inadvertently request more compute nodes than you realise.

If you are using the ''%%--ntasks%%'' parameter //without// using the ''%%--nodes%%'' parameter, then Slurm may calculate the number of nodes you require to be different to what you expect. The Slurm documentation states:

> -n, --ntasks=<number>
> sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the Slurm controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. If the --nodes option is not used, this option is used to implicitly determine the node count. The default is one task per node, but note that the ''%%--cpus-per-task%%'' option will change this default.

In some cases, we have seen examples where a user has set ''%%--ntasks=64%%'' //without// ''%%--nodes%%'', and Slurm turns this into a resource request for **1 to 64 nodes**, and thus beyond what is available in the partition.

If you want to check whether it is the //calculated// number of nodes which is putting your job into this state, you can check using ''scontrol'', e.g:

<code lang=bash>
$ scontrol show job JOB_ID
...
...
   NumNodes=1-64 NumCPUs=64 NumTasks=64 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=64,mem=128G,node=1,billing=64
   AllocTRES=()
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=128G MinTmpDiskNode=0
...
$
</code>

If the **NumNodes** value is __not__ what you expected, then it is //likely// due to the missing ''%%--nodes%%'' parameter.

**Be explicit - __always__ use the ''%%--nodes%%'' parameter if you are using ''%%--ntasks%%''.**

----

[[:faq:index|Back to FAQ index]]