nvidia-smi reference

nvidia-smi (NVIDIA System Management Interface) is the primary tool for monitoring and managing NVIDIA GPUs.

Common commands:

nvidia-smi                          # Summary view
nvidia-smi -L                       # List all GPUs with UUIDs
nvidia-smi -q                       # Full query, all attributes
nvidia-smi -q -d MEMORY             # Memory only
nvidia-smi -q -d TEMPERATURE        # Temperature only
nvidia-smi -q -d POWER              # Power only
nvidia-smi -q -d UTILIZATION        # Utilization only
nvidia-smi -i 0                     # Target GPU 0 only

CSV/selective queries:

nvidia-smi --query-gpu=name,utilization.gpu,memory.used,temperature.gpu \
  --format=csv,noheader

# Continuous monitoring every 1 second
nvidia-smi --query-gpu=timestamp,name,utilization.gpu,memory.used \
  --format=csv -l 1

Key query fields:

Field	Description
`name`	GPU product name
`uuid`	GPU UUID
`memory.total / used / free`	GPU memory stats
`utilization.gpu`	Compute utilization %
`utilization.memory`	Memory bandwidth utilization %
`temperature.gpu`	Core temperature (°C)
`power.draw`	Current power draw (W)
`power.limit`	Power limit (W)
`driver_version`	Driver version

Device modifications (require root):

sudo nvidia-smi -pm 1               # Enable persistence mode
sudo nvidia-smi -i 0 -e 1           # Enable ECC
sudo nvidia-smi -i 0 -c 0           # Compute mode: DEFAULT
sudo nvidia-smi -i 0 -c 1           # Compute mode: EXCLUSIVE_PROCESS
sudo nvidia-smi -i 0 -pl 250        # Set power limit to 250 W
sudo nvidia-smi --gpu-reset -i 0    # Reset GPU (requires no processes)

Topology and NVLink:

nvidia-smi topo -m                  # GPU topology matrix
nvidia-smi nvlink -s                # NVLink status
nvidia-smi nvlink -e                # NVLink error counters

Return codes:

0 = Success
2 = Invalid argument or flag
6 = Object not found
15 = GPU fell off the bus