Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[x/sync] Concurrent Map Write Panic #1750

Closed
patrick-ogrady opened this issue Jul 24, 2023 · 2 comments · Fixed by #1756
Closed

[x/sync] Concurrent Map Write Panic #1750

patrick-ogrady opened this issue Jul 24, 2023 · 2 comments · Fixed by #1756
Labels
bug Something isn't working
Milestone
v1.10.7

Comments

@patrick-ogrady
Copy link
Contributor

patrick-ogrady commented Jul 24, 2023 •
edited
Loading

I ran into this panic when running CI on hypersdk:

2023-07-22T21:39:11.7256452Z [0;0m[0;37m[sync_concurrent] fatal error: concurrent map writes
2023-07-22T21:39:11.7256844Z [0;0m[0;37m[sync_concurrent] fatal error: concurrent map writes
2023-07-22T21:39:11.7328777Z [0;0m[0;37m[sync_concurrent] 
2023-07-22T21:39:11.7329088Z [0;0m[0;37m[sync_concurrent] goroutine 27175 [running]:
2023-07-22T21:39:11.7329544Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/utils/math.(*averagerHeapBackend).Push(0xc0005314d0, {0xeceaa0?, 0xc009d03740})
2023-07-22T21:39:11.7330072Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/utils/math/averager_heap.go:134 +0x54
2023-07-22T21:39:11.7330504Z [0;0m[0;37m[sync_concurrent] container/heap.Push({0x12238a0, 0xc0005314d0}, {0xeceaa0?, 0xc009d03740?})
2023-07-22T21:39:11.7331052Z [0;0m[0;37m[sync_concurrent] 	/opt/hostedtoolcache/go/1.20.6/x64/src/container/heap/heap.go:52 +0x37
2023-07-22T21:39:11.7331569Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/utils/math.averagerHeap.Add({0x0?}, {0x1, 0x41, 0xaf, 0x5a, 0xea, 0xc4, 0xc8, 0xe3, 0xfb, ...}, ...)
2023-07-22T21:39:11.7332056Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/utils/math/averager_heap.go:77 +0x125
2023-07-22T21:39:11.7332586Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*peerTracker).TrackBandwidth(0xc000816320, {0x1, 0x41, 0xaf, 0x5a, 0xea, 0xc4, 0xc8, 0xe3, 0xfb, ...}, ...)
2023-07-22T21:39:11.7333079Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/peer_tracker.go:171 +0x1b1
2023-07-22T21:39:11.7333613Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*networkClient).request(0xc0005caf00, {0x1222160, 0xc0012082d0}, {0x1, 0x41, 0xaf, 0x5a, 0xea, 0xc4, 0xc8, ...}, ...)
2023-07-22T21:39:11.7334114Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/network_client.go:263 +0x725
2023-07-22T21:39:11.7334623Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*networkClient).RequestAny(0xc0005caf00, {0x1222160, 0xc0012082d0}, 0x0, {0xc009062780, 0x55, 0x55})
2023-07-22T21:39:11.7335090Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/network_client.go:192 +0x2d5
2023-07-22T21:39:11.7335594Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*client).get(0xc0012123c0, {0x1222160, 0xc0012082d0}, {0xc009062780, 0x55, 0x55})
2023-07-22T21:39:11.7336037Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/client.go:252 +0x16c
2023-07-22T21:39:11.7336619Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.getAndParse[...]({0x1222160?, 0xc0012082d0}, 0xc0012123c0, {0xc009062780, 0x55, 0x55}, 0xc008d17e98)
2023-07-22T21:39:11.7337074Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/client.go:200 +0xd5
2023-07-22T21:39:11.7337639Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*client).GetChangeProof(0x6ff1e7fe4db623b8?, {0x1222160, 0xc0012082d0}, 0xc00903dc20, {0x7f7d40a20e98?, 0xc000147e60?})
2023-07-22T21:39:11.7338244Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/client.go:128 +0x133
2023-07-22T21:39:11.7338748Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*Manager).getAndApplyChangeProof(0xc00123a300, {0x1222160, 0xc0012082d0}, 0xc007ee5c20)
2023-07-22T21:39:11.7339220Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/manager.go:256 +0x1d4
2023-07-22T21:39:11.7339714Z [0;0m[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*Manager).doWork(0xc00123a300, {0x1222160, 0xc0012082d0}, 0xc007ee5c20)
2023-07-22T21:39:11.7340216Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/manager.go:241 +0xb8
2023-07-22T21:39:11.7340620Z [0;0m[0;37m[sync_concurrent] created by github.com/ava-labs/avalanchego/x/sync.(*Manager).sync
2023-07-22T21:39:11.7341037Z [0;0m[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/manager.go:192 +0x198

Other usage:

2023-07-22T21:24:47.1228492Z �[0;0m�[0;37m[sync_concurrent] fatal error: concurrent map writes
2023-07-22T21:24:47.1258845Z �[0;0m�[0;37m[sync_concurrent] 
2023-07-22T21:24:47.1259364Z �[0;0m�[0;37m[sync_concurrent] goroutine 160871 [running]:
2023-07-22T21:24:47.1259949Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/utils/math.(*averagerHeapBackend).Pop(0xc0005b0960)
2023-07-22T21:24:47.1260724Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/utils/math/averager_heap.go:142 +0x7a
2023-07-22T21:24:47.1261372Z �[0;0m�[0;37m[sync_concurrent] container/heap.Pop({0x12238a0, 0xc0005b0960})
2023-07-22T21:24:47.1262215Z �[0;0m�[0;37m[sync_concurrent] 	/opt/hostedtoolcache/go/1.20.6/x64/src/container/heap/heap.go:63 +0x6b
2023-07-22T21:24:47.1262865Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/utils/math.averagerHeap.Pop({0xc000385d78?})
2023-07-22T21:24:47.1263525Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/utils/math/averager_heap.go:98 +0x6e
2023-07-22T21:24:47.1264191Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*peerTracker).GetAnyPeer(0xc00077e9b0, 0x0)
2023-07-22T21:24:47.1264910Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/peer_tracker.go:135 +0x193
2023-07-22T21:24:47.1265717Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*networkClient).RequestAny(0xc00099f080, {0x1222160, 0xc00091e460}, 0x0, {0xc00f61bb60, 0x52, 0x52})
2023-07-22T21:24:47.1266635Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/network_client.go:182 +0x12e
2023-07-22T21:24:47.1267397Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*client).get(0xc0009383c0, {0x1222160, 0xc00091e460}, {0xc00f61bb60, 0x52, 0x52})
2023-07-22T21:24:47.1268219Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/client.go:252 +0x16c
2023-07-22T21:24:47.1268937Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.getAndParse[...]({0x1222160?, 0xc00091e460}, 0xc0009383c0, {0xc00f61bb60, 0x52, 0x52}, 0xc01004fe98)
2023-07-22T21:24:47.1269509Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/client.go:200 +0xd5
2023-07-22T21:24:47.1270125Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*client).GetChangeProof(0x48e98cc5fed4284f?, {0x1222160, 0xc00091e460}, 0xc01022b710, {0x7f5fc0de3c98?, 0xc000182ea0?})
2023-07-22T21:24:47.1270739Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/client.go:128 +0x133
2023-07-22T21:24:47.1271387Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*Manager).getAndApplyChangeProof(0xc000922500, {0x1222160, 0xc00091e460}, 0xc0101b4480)
2023-07-22T21:24:47.1271930Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/manager.go:256 +0x1d4
2023-07-22T21:24:47.1272519Z �[0;0m�[0;37m[sync_concurrent] github.com/ava-labs/avalanchego/x/sync.(*Manager).doWork(0xc000922500, {0x1222160, 0xc00091e460}, 0xc0101b4480)
2023-07-22T21:24:47.1273194Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/manager.go:241 +0xb8
2023-07-22T21:24:47.1273786Z �[0;0m�[0;37m[sync_concurrent] created by github.com/ava-labs/avalanchego/x/sync.(*Manager).sync
2023-07-22T21:24:47.1274259Z �[0;0m�[0;37m[sync_concurrent] 	/home/runner/go/pkg/mod/github.com/ava-labs/avalanchego@v1.10.5/x/sync/manager.go:192 +0x198
@patrick-ogrady patrick-ogrady added the bug Something isn't working label Jul 24, 2023
@patrick-ogrady patrick-ogrady mentioned this issue Jul 24, 2023
Closed
@patrick-ogrady
Copy link
Contributor Author

From what I can tell, this is caused by concurrent modification of the bandwidthHeap here:

@patrick-ogrady patrick-ogrady mentioned this issue Jul 24, 2023
Closed
@patrick-ogrady patrick-ogrady added this to the v1.10.6 milestone Jul 24, 2023
@patrick-ogrady patrick-ogrady mentioned this issue Jul 24, 2023
Merged
@patrick-ogrady patrick-ogrady linked a pull request Jul 24, 2023 that will close this issue
x/sync: add locks for peerTracker #1756
Merged
@StephenButtolph
Copy link
Contributor

Resolved by #1756

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
No open projects
Ski Patrol
Archived in project
Milestone
v1.10.7
2 participants
@patrick-ogrady @StephenButtolph