gpu training hangs

[shyakocat]

Related Issue

training with --gpus 0, --gpus 0,1 hangs at initializing ddp ...

I've tried, change ddp to dp, change nccl to gloo, upgrade lightning or torch version... still not work.

(only cpu works)

[dyflional10]

@shyakocat I encountered the same issue, have you resolved it?

[froestiago]

try running export NCCL_P2P_DISABLE=1 before running the script
it worked for me