Common BCM and Slurm issues

Node stuck in provisioning:

• Check provisioning network connectivity
• Verify BCM DHCP is serving the correct IP to the node
• Check BCM logs: journalctl -u cmd
• Verify the node can PXE boot (check BIOS boot order)

Node in DOWN or FAIL state in Slurm:

sinfo -N                                     # Check node state
scontrol show node <name>                    # Look at the Reason field
journalctl -u slurmctld                      # Controller logs
journalctl -u slurmd                         # Node daemon logs

# Resume a drained node after resolving the issue
scontrol update NodeName=<n> State=RESUME

Reinstalling a node via cmsh:

cmsh> device set <node> state provisioning
cmsh> device commit

Slurm log location: Check SlurmctldLogFile and SlurmdLogFile in slurm.conf for configured log paths.