If you submit a job and it gets stuck in state PENDING with the reason code PartitionNodeLimit, then the Slurm documentation describes that code as follows:
PartitionNodeLimit
The number of nodes required by this job is outside of its partition's current limits. Can also indicate that required nodes are DOWN or DRAINED.
In most cases, this will be because your script has asked for more compute nodes than are available within the partition you selected. You can get a summary of the available resources across all nodes in a partition as follows - in the case of the default_free partition:
$ sinfo -p default_free -N -l
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
compute009 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute010 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute011 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute012 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute013 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute014 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute015 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute016 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute017 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute018 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute019 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute020 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute021 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute022 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute023 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute024 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute025 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute026 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute027 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute028 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute029 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute030 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute031 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute032 1 default_free allocated 256 2:128:1 150000 0 10 (null) none
compute033 1 default_free drained* 256 2:128:1 150000 0 10 (null) In maintenance
compute034 1 default_free idle 256 2:128:1 150000 0 10 (null) none
compute035 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
compute036 1 default_free mixed 256 2:128:1 150000 0 10 (null) none
hmem009 1 default_free allocated 256 2:128:1 150000 0 1 (null) none
hmem010 1 default_free allocated 256 2:128:1 150000 0 1 (null) none
$
However, there is a scenario where your job may inadvertently request more compute nodes than you realise.
If you are using the --ntasks parameter without using the --nodes parameter, then Slurm may calculate the number of nodes you require to be different to what you expect. The Slurm documentation states:
-n, –ntasks=<number>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the Slurm controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. If the –nodes option is not used, this option is used to implicitly determine the node count. The default is one task per node, but note that the--cpus-per-taskoption will change this default.
In some cases, we have seen examples where a user has set --ntasks=64 without --nodes, and Slurm turns this into a resource request for 1 to 64 nodes, and thus beyond what is available in the partition.
If you want to check whether it is the calculated number of nodes which is putting your job into this state, you can check using scontrol, e.g:
$ scontrol show job JOB_ID
...
...
NumNodes=1-64 NumCPUs=64 NumTasks=64 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=64,mem=128G,node=1,billing=64
AllocTRES=()
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=128G MinTmpDiskNode=0
...
$
If the NumNodes value is not what you expected, then it is likely due to the missing --nodes parameter.
Be explicit - always use the --nodes parameter if you are using --ntasks.