NCCL troubleshooting

NCCL provides multi-GPU and multi-node collective communications for distributed training.

Useful environment variables:

export NCCL_DEBUG=INFO              # Enable NCCL debug output
export NCCL_DEBUG_SUBSYS=ALL        # All subsystems verbose
export NCCL_IB_DISABLE=0            # Enable InfiniBand (default)
export NCCL_IB_DISABLE=1            # Force TCP, disable IB
export NCCL_SOCKET_IFNAME=eth0      # Force specific network interface

Common NCCL issues:

• Slow collective operations → Check InfiniBand connectivity or NVLink errors
• NCCL timeout → Check network latency and firewall rules between nodes
• NCCL crash → Check for NVLink XID errors, driver issues