Troubleshooting Fabric Manager and NVLink

Issue: Version mismatch

nvidia-smi | grep Driver                       # Check driver version
apt list --installed | grep fabricmanager      # Check FM version
# Major version must match

Issue: FM not running — multi-GPU jobs fail

Multi-GPU training fails with NVLink errors when FM is not running
Solution: Start FM before launching workloads

GPU reset behavior with NVLink:

Condition	Ampere+ with FM running	Ampere+ without FM / Pre-Ampere
Reset behavior	Each GPU can be reset individually	All NVLink-connected GPUs must be reset together

sudo nvidia-smi --gpu-reset -i 0    # Reset GPU 0
sudo nvidia-smi --gpu-reset         # Reset all GPUs

NVLink diagnostics:

nvidia-smi nvlink -e                # Error counters
nvidia-smi nvlink -s                # Status

NVSwitch diagnostics (via dcgmi — NVSwitches appear in dcgmi discovery -l):

NSCQ package required for Hopper and earlier (e.g., libnvidia-nscq-565)
NVSDM package required for Blackwell and later