Lotus daemon crash after upgrde to new version ! Please help

[seantw1974]

2023-04-27T15:49:53.019+0800 WARN modules modules/chain.go:90 loading chain state from disk: loading tipset: get block bafy2bzaceal3xax2bfwtzxf2osegdqmm2neaia5k6owxxjky4ngxtzvlrry6y: ipld: could not find bafy2bzaceal3xax2bfwtzxf2osegdqmm2neaia5k6owxxjky4ngxtzvlrry6y
2023-04-27T15:49:53.029+0800 INFO p2pnode lp2p/addrs.go:101 Swarm listening at: [/ip4/127.0.0.1/tcp/19317 /ip4/192.168.100.80/tcp/19317 /ip4/211.21.182.10/tcp/19317 /ip6/::1/tcp/22403]
2023-04-27T15:49:53.031+0800 INFO modules modules/services.go:163 subscribing to pubsub topic /fil/blocks/testnetnet
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1ae9c0e]

goroutine 886 [running]:
github.com/filecoin-project/lotus/chain/types.(*TipSet).Height(...)
/root/lotus/chain/types/tipset.go:152
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).checkMessage(0xc00038e540, {0x72791d8, 0xc01fef2640}, 0xc01efb8480)
/root/lotus/chain/messagepool/messagepool.go:728 +0x10e
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).addLoaded(0xc00038e540, {0x72791d8, 0xc01fef2640}, 0xc01efb8480)
/root/lotus/chain/messagepool/messagepool.go:919 +0x34
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).loadLocal(0xc00038e540, {0x72791d8, 0xc01fef2640})
/root/lotus/chain/messagepool/messagepool.go:1582 +0x28e
github.com/filecoin-project/lotus/chain/messagepool.New.func2()
/root/lotus/chain/messagepool/messagepool.go:434 +0x6b
created by github.com/filecoin-project/lotus/chain/messagepool.New
/root/lotus/chain/messagepool/messagepool.go:432 +0xbe5

[rjan90]

Hey @seantw1974! Do you have another disk that you can initialize and run the chain on? Based on the panic here it does sound a lot similar to this: link to slack message

[rjan90]

Which Lotus version is this,v1.23.0? On v1.23.0 CUDA is enabled by default, so if you want to build with OpenCL you will have to export: FFI_USE_OPENCL=1, or export FFI_NO_GPU=1 if you want to disable GPU.

Note that disabling GPU is fine for the deamon process, but you should not do that if your are also using the lotus-miner process with this build.

[seantw1974]

Yes just download from github ..seems nvidia driver issue ...I am checking ..thank you

[seantw1974]

even I use FFI_NO_GPU=1 but the compile still checking the nvcc ?? why??
Compiling proc-macro-crate v1.1.3
Compiling multihash-derive v0.8.1
error: failed to run custom build command for bellperson v0.24.1

Caused by:
process didn't exit successfully: /root/lotus/extern/filecoin-ffi/rust/target/release/build/bellperson-6598815fc6014bb4/build-script-build (exit status: 101)
--- stdout
cargo:rustc-cfg=nightly

--- stderr
nvcc fatal : Unknown option '-threads'
thread 'main' panicked at 'nvcc failed. See the kernel source at /root/lotus/extern/filecoin-ffi/rust/target/release/build/bellperson-347b226f04617894/out/22e62c5450c4a1a81043cc36710bfcd865d62713e714b37c0fbffc3d8fbd9e7f.cu', /root/.cargo/registry/src/gitpro.ttaallkk.top-1ecc6299db9ec823/ec-gpu-gen-0.5.2/src/source.rs:660:13
stack backtrace:

[seantw1974]

After I remove old nvcc binary ...seems work ..
compile passed

[seantw1974]

Compile passed
but still same issue
2023-04-27T16:51:47.339+0800 INFO modules modules/services.go:163 subscribing to pubsub topic /fil/blocks/testnetnet
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1aeea8e]

goroutine 919 [running]:
github.com/filecoin-project/lotus/chain/types.(*TipSet).Height(...)
/root/lotus/chain/types/tipset.go:152
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).checkMessage(0xc0005e1180, {0x729a1d8, 0xc01e7282c0}, 0xc031342000)
/root/lotus/chain/messagepool/messagepool.go:728 +0x10e
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).addLoaded(0xc0005e1180, {0x729a1d8, 0xc01e7282c0}, 0xc031342000)
/root/lotus/chain/messagepool/messagepool.go:919 +0x34
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).loadLocal(0xc0005e1180, {0x729a1d8, 0xc01e7282c0})
/root/lotus/chain/messagepool/messagepool.go:1582 +0x28e
github.com/filecoin-project/lotus/chain/messagepool.New.func2()
/root/lotus/chain/messagepool/messagepool.go:434 +0x6b
created by github.com/filecoin-project/lotus/chain/messagepool.New
/root/lotus/chain/messagepool/messagepool.go:432 +0xbe5

[marco-storswift]

I got the same case. I set the EnableSplitstore = false.

[seantw1974]

I already set that . but still fail ...

[seantw1974]

Resolved:
I remove chain data directory
and download new snapshot image
After import snapshot ......the issue is resolved

[donkabat]

filecoin-project/lotus-docs#555