Core Slurm commands

Job submission:

sbatch job.sh                        # Submit batch script
srun --gres=gpu:1 -N 1 cmd          # Run interactive/parallel job
salloc -N 2 --gres=gpu:4            # Allocate resources interactively

Monitoring:

sinfo                                # Node and partition status
sinfo -N                             # Node-centric view
squeue                               # All queued and running jobs
squeue -u $USER                      # My jobs only
sacct -j <jobid>                     # Accounting data for a job
scontrol show job <id>               # Detailed job information
scontrol show node <name>            # Detailed node information

Job control:

scancel <jobid>                      # Cancel a job
scontrol hold <jobid>                # Hold a pending job
scontrol release <jobid>             # Release a held job
scontrol update NodeName=<n> State=DRAIN Reason="maintenance"
scontrol update NodeName=<n> State=RESUME

Batch script example:

#!/bin/bash
#SBATCH --job-name=my_ai_job
#SBATCH --nodes=1
#SBATCH --gres=gpu:4
#SBATCH --time=01:00:00
#SBATCH --partition=gpu

srun python train.py

GPU scheduling:

sbatch --gres=gpu:1 job.sh           # Request 1 GPU
sbatch --gres=gpu:a100:2 job.sh      # Request 2 A100 GPUs specifically
sbatch -N 2 --gres=gpu:4 job.sh      # 2 nodes, 4 GPUs each