How Can I List My Current Scheduled Jobs?

Also: “Why is my job not running yet?”

You can get a list of jobs that you have submitted and which are waiting to run by using the squeue -t PD –me command: 

$ squeue -t PD --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          18020727    bigmem generati    n1234 PD       0:00      1 (Resources)
          18020208    bigmem   AOH_a5    n1234 PD       0:00      1 (Resources)
          17824926 bigmem,de vep4-Sen    n1234 PD       0:00      1 (DependencyNeverSatisfied)
$

The PD option is a shorthand for jobs of status type PENDING. See man squeue for a full list of all squeue options and job status type codes.

The NODELIST(REASON) column shows the reason that your job is not currently running.

More Detailed Explanations

You can use the Simple Slurm Tools utilities to obtain more information about why a job may not be running. In particular the use of sproject may prove useful to explain current restrictions on your project account.

e.g.: sproject PROJECT_NAME:

The output above shows that for project comet_imlibrw two jobs are pending, both requesting GPU resources, and they are not running because one job is receiving a Node down or drained message, and the other because gpu004 is unavailable.

It also highlights that in this case, the total number of GPU cards requested is beyond the project limits, so not all jobs will run even when the pending reasons are resolved.

Reasons

For reference, the possible REASON status codes are listed below:

Status Description
AssociationJobLimit The job's association has reached its maximum job count.
AssociationResourceLimit The job's association has reached some resource limit.
AssociationTimeLimit The job's association has reached its time limit.
BadConstraints The job's constraints can not be satisfied.
BeginTime The job's earliest start time has not yet been reached.
Cleaning The job is being requeued and still cleaning up from its previous execution.
Dependency This job is waiting for a dependent job to complete.
FrontEndDown No front end node is available to execute this job.
InactiveLimit The job reached the system InactiveLimit.
InvalidAccount The job's account is invalid.
InvalidQOS The job's QOS is invalid.
JobHeldAdmin The job is held by a system administrator.
JobHeldUser The job is held by the user.
JobLaunchFailure The job could not be launched. This may be due to a file system problem, invalid program name, etc.
Licenses The job is waiting for a license.
NodeDown A node required by the job is down.
NonZeroExitCode The job terminated with a non-zero exit code.
PartitionDown The partition required by this job is in a DOWN state.
PartitionInactive The partition required by this job is in an Inactive state and not able to start jobs.
PartitionNodeLimit The number of nodes required by this job is outside of its partition's current limits. Can also indicate that required nodes are DOWN or DRAINED.
PartitionTimeLimit The job's time limit exceeds its partition's current time limit.
Priority One or more higher priority jobs exist for this partition or advanced reservation.
Prolog Its PrologSlurmctld program is still running.
QOSJobLimit The job's QOS has reached its maximum job count.
QOSResourceLimit The job's QOS has reached some resource limit.
QOSTimeLimit The job's QOS has reached its time limit.
ReqNodeNotAvail Some node specifically required by the job is not currently available. The node may currently be in use, reserved for another job, in an advanced reservation, DOWN, DRAINED, or not responding. Nodes which are DOWN, DRAINED, or not responding will be identified as part of the job's “reason” field as “UnavailableNodes”. Such nodes will typically require the intervention of a system administrator to make available.
Reservation The job is waiting its advanced reservation to become available.
Resources The job is waiting for resources to become available. (i.e. There are more jobs waiting for slots in this partition than there are free resources available)
SystemFailure Failure of the Slurm system, a file system, the network, etc.
TimeLimit The job exhausted its time limit.
QOSUsageThreshold Required QOS threshold has been breached.
WaitingForScheduling No reason has been set for this job yet. Waiting for the scheduler to determine the appropriate reason.

Back to FAQ