DCGM reference

NVIDIA DCGM (Data Center GPU Manager) is the enterprise GPU monitoring library. Unlike nvidia-smi (point-in-time queries), DCGM runs as a persistent daemon for continuous monitoring at scale.

Components:

Modes:

• Standalone mode — nv-hostengine runs as a daemon; multiple clients connect. NVIDIA's preferred mode.
• Embedded mode — 3rd-party agent loads DCGM as a shared library for fine-grained control.

Installation (Ubuntu):

CUDA_VERSION=$(nvidia-smi -q | sed -E -n 's/CUDA Version[ :]+([0-9]+)[.].*/\1/p')

sudo apt-get install --yes --install-recommends \
  datacenter-gpu-manager-4-cuda${CUDA_VERSION}

sudo systemctl --now enable nvidia-dcgm

System requirements: Min 16 GB RAM; CPU cores ≥ number of GPUs.

For NVSwitch systems (DGX/HGX): Fabric Manager package required in addition to DCGM.

Key dcgmi commands:

dcgmi discovery -l                  # List all GPUs and NVSwitches
dcgmi health -c                     # GPU health check
dcgmi diag -r 1                     # Short diagnostic (~1 min)
dcgmi diag -r 2                     # Medium diagnostic (~2 min)
dcgmi diag -r 3                     # Long/stress diagnostic (10+ min)
dcgmi dmon -e 150,155,203           # Monitor temp, power, utilization
dcgmi group -c -g mygroup           # Create GPU group
dcgmi group -a -g mygroup -i 0,1    # Add GPUs 0 and 1 to group

Diagnostic levels: 1 = short software checks, 2 = medium + memory bandwidth, 3 = comprehensive stress test.