runtime: SIGSEGV in readgstatus via gcBgMarkWorker

[bcmills]

2021-01-19T12:49:13-d047c91/linux-arm64-packet

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2e430]

runtime stack:
runtime.throw(0x59c4d8, 0x2a)
	/workdir/go/src/runtime/panic.go:1112 +0x54
runtime.sigpanic()
	/workdir/go/src/runtime/signal_unix.go:714 +0x29c
runtime.readgstatus(...)
	/workdir/go/src/runtime/proc.go:830
runtime.markroot(0x4000038e98, 0x4a)
	/workdir/go/src/runtime/mgcmark.go:199 +0x2b0
runtime.gcDrain(0x4000038e98, 0x3)
	/workdir/go/src/runtime/mgcmark.go:1014 +0x100
runtime.gcBgMarkWorker.func2()
	/workdir/go/src/runtime/mgc.go:1980 +0x74
runtime.systemstack(0xffff47ffe930)
	/workdir/go/src/runtime/asm_arm64.s:248 +0xa0
runtime.mstart()
	/workdir/go/src/runtime/proc.go:1244

2019-08-27T16:04:01-ba9fcec/linux-ppc64le-power9osu

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2d220]

runtime stack:
runtime.throw(0x93df99, 0x2a)
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/panic.go:774 +0x5c
runtime.sigpanic()
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/signal_unix.go:378 +0x474
runtime.readgstatus(...)
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/proc.go:707
runtime.markroot(0xc000025770, 0xc)
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgcmark.go:195 +0x300
runtime.gcDrain(0xc000025770, 0x7)
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgcmark.go:915 +0x118
runtime.gcBgMarkWorker.func2()
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/mgc.go:1936 +0x178
runtime.systemstack(0x42150)
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/asm_ppc64x.s:269 +0x94
runtime.mstart()
	/tmp/workdir-host-linux-ppc64le-power9-osu/go/src/runtime/proc.go:1182

Given the gap in time and the difference in architecture I'm not sure whether these are due to the same root cause, but they sure seem to have the same stack trace.

CC @aclements @cherrymui @prattmic

[prattmic]

Despite the not-to-useful ... argument in the stack trace, this is presumably a nil gp, which should have come from allgs: https://cs.opensource.google/go/go/+/master:src/runtime/mgcmark.go;l=190-199;drc=ecf4ebf10054f70e51a0ce759b2ae91aa4febd1a;bpv=1;bpt=1?q=runtime%2Fproc.go:830&ss=go

[alxn]

Hello, x86_64 checking in:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0x422d52]
runtime stack:
runtime.throw(0x12d67fc, 0x2a)
	GOROOT/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
	GOROOT/src/runtime/signal_unix.go:726 +0x4ac
runtime.scanobject(0xc010552140, 0xc00004b698)
	GOROOT/src/runtime/mgcmark.go:1371 +0x212
runtime.gcDrain(0xc00004b698, 0xb)
	GOROOT/src/runtime/mgcmark.go:1143 +0x22b
runtime.gcBgMarkWorker.func2()
	GOROOT/src/runtime/mgc.go:1979 +0x191
runtime.systemstack(0xc000000f00)
	GOROOT/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	GOROOT/src/runtime/proc.go:1116

Didn't manage to get a core yet.

[bcmills]

2021-01-27T12:18:00-cd176b3/linux-mips64-rtrk

[bcmills]

2021-02-23T10:04:13-a78b0e6/linux-mips64-rtrk

[bcmills]

2021-06-07T21:12:46-909dd5e/linux-arm64-packet
2021-05-26T16:11:00-bfd7798/linux-mips64le-rtrk
2021-05-21T21:07:03-3c65644/linux-mips64-rtrk
2021-05-17T18:03:56-a2c07a9/linux-mips64-rtrk
2021-05-07T17:43:31-b44c78b/linux-arm64-packet
2021-05-05T21:49:59-0714010/linux-mips64-rtrk
2021-04-23T20:57:54-691e1b8/linux-mips64le-rtrk
2021-04-16T21:20:31-02a2ff4/linux-mips64-rtrk
2021-03-24T20:56:30-d1beb07/linux-mips64-rtrk
2021-03-16T16:44:54-77fc1ea/linux-mips64le-rtrk
2021-03-12T01:47:01-a607408/linux-mips64le-rtrk
2021-03-08T20:41:06-a08adda/linux-mips64le-rtrk
2021-03-05T18:46:36-51d8d35/linux-mips64le-rtrk

[bcmills]

$ greplogs --dashboard -md --since=2021-06-08 -l -e (?ms)SIGSEGV.*^runtime\.readgstatus.*^runtime\.gcBgMarkWorker

2021-08-08T18:37:08-507cc34/linux-mips64-rtrk
2021-06-25T17:31:39-d01bc57/linux-mips64-rtrk

[gopherbot]

Change https://golang.org/cl/368134 mentions this issue: runtime: fix racy allgs access on weak memory architectures