Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests of executor package hang #27966

Closed
Tracked by #25899
tisonkun opened this issue Sep 11, 2021 · 10 comments
Closed
Tracked by #25899

tests of executor package hang #27966

tisonkun opened this issue Sep 11, 2021 · 10 comments
Assignees
@guo-shaoge
Labels
component/test severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.

Comments

@tisonkun
Copy link
Contributor 

[2021-09-11T15:46:55.410Z] coverage: 73.2% of statements
[2021-09-11T15:46:55.410Z] panic: test timed out after 9m0s
[2021-09-11T15:46:55.410Z] 
[2021-09-11T15:46:55.410Z] goroutine 584632 [running]:
[2021-09-11T15:46:55.410Z] testing.(*M).startAlarm.func1()
[2021-09-11T15:46:55.410Z] 	/usr/local/go/src/testing/testing.go:1700 +0xe5
[2021-09-11T15:46:55.410Z] created by time.goFunc
[2021-09-11T15:46:55.410Z] 	/usr/local/go/src/time/sleep.go:180 +0x45
[2021-09-11T15:46:55.410Z] 
[2021-09-11T15:46:55.410Z] goroutine 1 [chan receive, 8 minutes]:
[2021-09-11T15:46:55.410Z] testing.(*T).Run(0xc001fe3080, 0x4081c76, 0x5, 0x42446c8, 0x1339e01)
[2021-09-11T15:46:55.410Z] 	/usr/local/go/src/testing/testing.go:1239 +0x2da
[2021-09-11T15:46:55.410Z] testing.runTests.func1(0xc000c18c00)
[2021-09-11T15:46:55.410Z] 	/usr/local/go/src/testing/testing.go:1511 +0x78
[2021-09-11T15:46:55.410Z] testing.tRunner(0xc000c18c00, 0xc000f3fc60)
[2021-09-11T15:46:55.410Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-11T15:46:55.410Z] testing.runTests(0xc0010c7890, 0x5f3ff40, 0xa, 0xa, 0xc04751b1d0b00899, 0x7dbd8817be, 0x6633780, 0x38)
[2021-09-11T15:46:55.410Z] 	/usr/local/go/src/testing/testing.go:1509 +0x2fe
[2021-09-11T15:46:55.410Z] testing.(*M).Run(0xc0010c0680, 0x0)
[2021-09-11T15:46:55.410Z] 	/usr/local/go/src/testing/testing.go:1417 +0x1eb
[2021-09-11T15:46:55.410Z] go.uber.org/goleak.VerifyTestMain(0x46cb5e0, 0xc0010c0680, 0xc007245e60, 0x3, 0x3)
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.uber.org/goleak@v1.1.10/testmain.go:53 +0x38
[2021-09-11T15:46:55.410Z] github.com/pingcap/tidb/executor.TestMain(0xc0010c0680)
[2021-09-11T15:46:55.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/main_test.go:31 +0x125
[2021-09-11T15:46:55.410Z] main.main()
[2021-09-11T15:46:55.410Z] 	_testmain.go:313 +0x1f6
[2021-09-11T15:46:55.410Z] 
[2021-09-11T15:46:55.410Z] goroutine 185 [chan receive]:
[2021-09-11T15:46:55.410Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000464468)
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3ac
[2021-09-11T15:46:55.410Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x85
[2021-09-11T15:46:55.410Z] 
[2021-09-11T15:46:55.410Z] goroutine 73 [select]:
[2021-09-11T15:46:55.410Z] go.opencensus.io/stats/view.(*worker).start(0xc000cd1b00)
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xcd
[2021-09-11T15:46:55.410Z] created by go.opencensus.io/stats/view.init.0
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x68
[2021-09-11T15:46:55.410Z] 
[2021-09-11T15:46:55.410Z] goroutine 135 [chan receive]:
[2021-09-11T15:46:55.410Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0002b9ce0)
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3ac
[2021-09-11T15:46:55.410Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x85
[2021-09-11T15:46:55.410Z] 
[2021-09-11T15:46:55.410Z] goroutine 188 [chan receive]:
[2021-09-11T15:46:55.410Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000464618)
[2021-09-11T15:46:55.410Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3ac
[2021-09-11T15:46:55.411Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
[2021-09-11T15:46:55.411Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x85
[2021-09-11T15:46:55.411Z] 
[2021-09-11T15:46:55.411Z] goroutine 6093 [chan receive, 8 minutes]:
[2021-09-11T15:46:55.411Z] testing.(*T).Parallel(0xc001fe2900)
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-11T15:46:55.411Z] github.com/pingcap/tidb/executor_test.TestBasicCTE(0xc001fe2900)
[2021-09-11T15:46:55.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:26 +0x57
[2021-09-11T15:46:55.411Z] testing.tRunner(0xc001fe2900, 0x4244638)
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-11T15:46:55.411Z] created by testing.(*T).Run
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-11T15:46:55.411Z] 
[2021-09-11T15:46:55.411Z] goroutine 6094 [chan receive, 8 minutes]:
[2021-09-11T15:46:55.411Z] testing.(*T).Parallel(0xc001fe2a80)
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-11T15:46:55.411Z] github.com/pingcap/tidb/executor_test.TestUnionDistinct(0xc001fe2a80)
[2021-09-11T15:46:55.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:77 +0x57
[2021-09-11T15:46:55.411Z] testing.tRunner(0xc001fe2a80, 0x42446d0)
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-11T15:46:55.411Z] created by testing.(*T).Run
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-11T15:46:55.411Z] 
[2021-09-11T15:46:55.411Z] goroutine 6095 [chan receive, 8 minutes]:
[2021-09-11T15:46:55.411Z] testing.(*T).Parallel(0xc001fe2d80)
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-11T15:46:55.411Z] github.com/pingcap/tidb/executor_test.TestCTEMaxRecursionDepth(0xc001fe2d80)
[2021-09-11T15:46:55.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:106 +0x57
[2021-09-11T15:46:55.411Z] testing.tRunner(0xc001fe2d80, 0x4244668)
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-11T15:46:55.411Z] created by testing.(*T).Run
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-11T15:46:55.411Z] 
[2021-09-11T15:46:55.411Z] goroutine 6097 [select]:
[2021-09-11T15:46:55.411Z] github.com/pingcap/check.(*suiteRunner).doRun(0xc05fb4c090, 0xc047190000)
[2021-09-11T15:46:55.411Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:667 +0x128
[2021-09-11T15:46:55.411Z] github.com/pingcap/check.(*suiteRunner).run(0xc05fb4c090, 0xc001164358)
[2021-09-11T15:46:55.411Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:696 +0xb0
[2021-09-11T15:46:55.411Z] github.com/pingcap/check.Run(0x404a680, 0xc001164358, 0xc02e0fbe58, 0xc09b69c3f0)
[2021-09-11T15:46:55.411Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:150 +0x4d
[2021-09-11T15:46:55.411Z] github.com/pingcap/check.RunAll(0xc02e0fbe58, 0x0)
[2021-09-11T15:46:55.411Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:136 +0x6b5
[2021-09-11T15:46:55.411Z] github.com/pingcap/check.TestingT(0xc001fe3080)
[2021-09-11T15:46:55.411Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:99 +0x45c
[2021-09-11T15:46:55.411Z] github.com/pingcap/tidb/executor_test.TestT(0xc001fe3080)
[2021-09-11T15:46:55.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor_test.go:109 +0x216
[2021-09-11T15:46:55.411Z] testing.tRunner(0xc001fe3080, 0x42446c8)
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-11T15:46:55.411Z] created by testing.(*T).Run
[2021-09-11T15:46:55.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-11T15:46:55.411Z] 
[2021-09-11T15:46:55.411Z] goroutine 584420 [select]:
[2021-09-11T15:46:55.411Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeLockWorker.run(0xc0962ac540, 0xc05d152140)
[2021-09-11T15:46:55.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:148 +0x3e5
[2021-09-11T15:46:55.411Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open
[2021-09-11T15:46:55.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:206 +0xdc
[2021-09-11T15:46:55.411Z] 
[2021-09-11T15:46:55.411Z] goroutine 584369 [chan receive, locked to thread]:
[2021-09-11T15:46:55.411Z] github.com/pingcap/badger.(*writeWorker).runWriteLSM(0xc07f8f42a0, 0xc074e77f80)
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:141 +0x195
[2021-09-11T15:46:55.412Z] created by github.com/pingcap/badger.startWriteWorker
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:65 +0x1bc
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 6096 [chan receive, 8 minutes]:
[2021-09-11T15:46:55.412Z] testing.(*T).Parallel(0xc001fe2f00)
[2021-09-11T15:46:55.412Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-11T15:46:55.412Z] github.com/pingcap/tidb/executor_test.TestCTEWithLimit(0xc001fe2f00)
[2021-09-11T15:46:55.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:149 +0x57
[2021-09-11T15:46:55.412Z] testing.tRunner(0xc001fe2f00, 0x4244670)
[2021-09-11T15:46:55.412Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-11T15:46:55.412Z] created by testing.(*T).Run
[2021-09-11T15:46:55.412Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 584421 [select]:
[2021-09-11T15:46:55.412Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).runUpdateSafePointLoop(0xc0a0178000)
[2021-09-11T15:46:55.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1550 +0x28a
[2021-09-11T15:46:55.412Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.NewMVCCStore
[2021-09-11T15:46:55.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:85 +0x269
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 584424 [select]:
[2021-09-11T15:46:55.412Z] github.com/tikv/client-go/v2/internal/locate.(*RegionCache).asyncCheckAndResolveLoop(0xc0a0178100, 0xdf8475800)
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/internal/locate/region_cache.go:397 +0xdc
[2021-09-11T15:46:55.412Z] created by github.com/tikv/client-go/v2/internal/locate.NewRegionCache
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/internal/locate/region_cache.go:368 +0x259
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 584302 [select]:
[2021-09-11T15:46:55.412Z] github.com/pingcap/badger/epoch.(*ResourceManager).collectLoop(0xc05da0ccc0, 0xc074eaf980)
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/epoch/manager.go:101 +0xdc
[2021-09-11T15:46:55.412Z] created by github.com/pingcap/badger/epoch.NewResourceManager
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/epoch/manager.go:79 +0xa7
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 584722 [select]:
[2021-09-11T15:46:55.412Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc06c9d32c0, 0xc078fae0c0, 0xc078a79a00)
[2021-09-11T15:46:55.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-11T15:46:55.412Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc08fc70600, 0x47212c0, 0xc000124010)
[2021-09-11T15:46:55.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-11T15:46:55.412Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-11T15:46:55.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 584299 [select]:
[2021-09-11T15:46:55.412Z] github.com/pingcap/check.(*resultTracker)._loopRoutine(0xc05fb4c000)
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:464 +0xbc
[2021-09-11T15:46:55.412Z] created by github.com/pingcap/check.(*resultTracker).start
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:444 +0x3f
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 579983 [chan receive]:
[2021-09-11T15:46:55.412Z] gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0xc08a0d2180)
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:379 +0x5d
[2021-09-11T15:46:55.412Z] created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:390 +0x75
[2021-09-11T15:46:55.412Z] 
[2021-09-11T15:46:55.412Z] goroutine 584368 [select]:
[2021-09-11T15:46:55.412Z] github.com/pingcap/badger.(*writeWorker).runWriteVLog(0xc07f8f42a0, 0xc074e77f80)
[2021-09-11T15:46:55.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:94 +0x1dc
[2021-09-11T15:46:55.413Z] created by github.com/pingcap/badger.startWriteWorker
[2021-09-11T15:46:55.413Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:64 +0x190
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 580014 [chan receive]:
[2021-09-11T15:46:55.413Z] gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0xc088598180)
[2021-09-11T15:46:55.413Z] 	/nfs/cache/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:379 +0x5d
[2021-09-11T15:46:55.413Z] created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
[2021-09-11T15:46:55.413Z] 	/nfs/cache/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:390 +0x75
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584714 [select]:
[2021-09-11T15:46:55.413Z] github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop.func1(0xc09c18d400, 0xc06f09f438, 0x6fc23ac00, 0x4778df8, 0xc074fe9200)
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:933 +0x125
[2021-09-11T15:46:55.413Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:924 +0xf6
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584619 [select]:
[2021-09-11T15:46:55.413Z] github.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs(0xc065588e00)
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:263 +0x2d9
[2021-09-11T15:46:55.413Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:351 +0x2d5
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584605 [select]:
[2021-09-11T15:46:55.413Z] github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop.func1(0xc09c18d400, 0x47432a8, 0xc08a37bf20, 0x4778df8, 0xc07934c400)
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1050 +0xde
[2021-09-11T15:46:55.413Z] created by github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1043 +0x8f
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584724 [select]:
[2021-09-11T15:46:55.413Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc06c9d3500, 0xc078fae0c0, 0xc078a79a00)
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-11T15:46:55.413Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc08fc70680, 0x47212c0, 0xc000124010)
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-11T15:46:55.413Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584604 [select]:
[2021-09-11T15:46:55.413Z] github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1(0xc09c18d400, 0x47432a8, 0xc08a37bf20)
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1015 +0x157
[2021-09-11T15:46:55.413Z] created by github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop
[2021-09-11T15:46:55.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1002 +0x73
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584428 [select]:
[2021-09-11T15:46:55.413Z] github.com/dgraph-io/ristretto.(*Cache).processItems(0xc0962ac7e0)
[2021-09-11T15:46:55.413Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:257 +0x9d
[2021-09-11T15:46:55.413Z] created by github.com/dgraph-io/ristretto.NewCache
[2021-09-11T15:46:55.413Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:155 +0x205
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584418 [chan receive]:
[2021-09-11T15:46:55.413Z] github.com/pingcap/badger.(*writeWorker).runMergeLSM(0xc07f8f42a0, 0xc074e77f80)
[2021-09-11T15:46:55.413Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:154 +0xad
[2021-09-11T15:46:55.413Z] created by github.com/pingcap/badger.startWriteWorker
[2021-09-11T15:46:55.413Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:66 +0x1e8
[2021-09-11T15:46:55.413Z] 
[2021-09-11T15:46:55.413Z] goroutine 584422 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection.func1(0xc0a0178000)
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1286 +0x8a
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1284 +0x78
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 584362 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/badger.(*blobGCHandler).run(0xc05d152040, 0xc074e77f08)
[2021-09-11T15:46:55.414Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/blob.go:468 +0xd4
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/badger.(*blobManager).Open
[2021-09-11T15:46:55.414Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/blob.go:292 +0x618
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 584642 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/tidb/domain.(*Domain).infoSyncerKeeper(0xc09c18d400)
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:456 +0x138
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/tidb/domain.(*Domain).Init
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:775 +0x70a
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 584303 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/badger.Open.func4(0xc091544010)
[2021-09-11T15:46:55.414Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:342 +0x17c
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/badger.Open
[2021-09-11T15:46:55.414Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:339 +0x10a8
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 584689 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc06c9d31a0, 0xc078fae0c0, 0xc078a79a00)
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-11T15:46:55.414Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc08fc705c0, 0x47212c0, 0xc000124010)
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 584621 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc078f09290, 0xc078f49a40)
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 579977 [chan receive]:
[2021-09-11T15:46:55.414Z] gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0xc034a24180)
[2021-09-11T15:46:55.414Z] 	/nfs/cache/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:379 +0x5d
[2021-09-11T15:46:55.414Z] created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
[2021-09-11T15:46:55.414Z] 	/nfs/cache/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:390 +0x75
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 584666 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop.func1(0xc09c18d400, 0xc0ab21ad50, 0x45d964b800, 0x4778df8, 0xc078ebaa00)
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:884 +0x125
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:875 +0x1f0
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.414Z] goroutine 584620 [select]:
[2021-09-11T15:46:55.414Z] github.com/pingcap/tidb/ddl.(*delRange).startEmulator(0xc01f643a90)
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:144 +0x128
[2021-09-11T15:46:55.414Z] created by github.com/pingcap/tidb/ddl.(*delRange).start
[2021-09-11T15:46:55.414Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:126 +0x6a
[2021-09-11T15:46:55.414Z] 
[2021-09-11T15:46:55.415Z] goroutine 584301 [select]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/badger.(*DB).updateSize(0xc074d5f200, 0xc074eaf968)
[2021-09-11T15:46:55.415Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:1039 +0x108
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/badger.Open
[2021-09-11T15:46:55.415Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:328 +0xe14
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584643 [select]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/tidb/domain.(*Domain).topologySyncerKeeper(0xc09c18d400)
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:482 +0x193
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/tidb/domain.(*Domain).Init
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:779 +0x770
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584625 [select]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/tidb/domain.(*Domain).topNSlowQueryLoop(0xc09c18d400)
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:424 +0x1e5
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/tidb/domain.(*Domain).Init
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:772 +0x6ca
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584622 [select]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc078f09340, 0xc078f49a40)
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584427 [select]:
[2021-09-11T15:46:55.415Z] github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc074edaa20)
[2021-09-11T15:46:55.415Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:96 +0xa5
[2021-09-11T15:46:55.415Z] created by github.com/dgraph-io/ristretto.newDefaultPolicy
[2021-09-11T15:46:55.415Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:80 +0x129
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584363 [select]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/badger.(*levelsController).runWorker(0xc074a95b20, 0xc074e77f20, 0x3d1fc00)
[2021-09-11T15:46:55.415Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/levels.go:212 +0x185
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/badger.(*levelsController).startCompact
[2021-09-11T15:46:55.415Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/levels.go:180 +0x93
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584419 [select]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeDBWorker.run(0xc0962ac4e0, 0xc05d152140)
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:96 +0x205
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:199 +0x8f
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584725 [select]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc06c9d3680, 0xc078fae0c0, 0xc078a79a00)
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-11T15:46:55.415Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc08fc706c0, 0x47212c0, 0xc000124010)
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-11T15:46:55.415Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-11T15:46:55.415Z] 
[2021-09-11T15:46:55.415Z] goroutine 584364 [chan receive]:
[2021-09-11T15:46:55.415Z] github.com/pingcap/badger.(*DB).runFlushMemTable(0xc074d5f200, 0xc074eafef0, 0x0, 0x0)
[2021-09-11T15:46:55.415Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:920 +0x190
[2021-09-11T15:46:55.415Z] created by github.com/pingcap/badger.Open
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:361 +0x17b0
[2021-09-11T15:46:55.416Z] 
[2021-09-11T15:46:55.416Z] goroutine 584651 [select]:
[2021-09-11T15:46:55.416Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc09c18d400)
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1113 +0xd8
[2021-09-11T15:46:55.416Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1106 +0x7f
[2021-09-11T15:46:55.416Z] 
[2021-09-11T15:46:55.416Z] goroutine 584425 [select]:
[2021-09-11T15:46:55.416Z] github.com/tikv/client-go/v2/tikv.(*KVStore).runSafePointChecker(0xc0af7a8000)
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/tikv/kv.go:264 +0x136
[2021-09-11T15:46:55.416Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/tikv/kv.go:187 +0x38a
[2021-09-11T15:46:55.416Z] 
[2021-09-11T15:46:55.416Z] goroutine 584650 [select]:
[2021-09-11T15:46:55.416Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop.func1(0xc09c18d400, 0x4778df8, 0xc078a79400)
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1084 +0x12f
[2021-09-11T15:46:55.416Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1076 +0x2b8
[2021-09-11T15:46:55.416Z] 
[2021-09-11T15:46:55.416Z] goroutine 584426 [select]:
[2021-09-11T15:46:55.416Z] github.com/tikv/client-go/v2/tikv.(*KVStore).safeTSUpdater(0xc0af7a8000)
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/tikv/kv.go:542 +0x194
[2021-09-11T15:46:55.416Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/tikv/kv.go:188 +0x3ac
[2021-09-11T15:46:55.416Z] 
[2021-09-11T15:46:55.416Z] goroutine 584652 [runnable]:
[2021-09-11T15:46:55.416Z] github.com/pingcap/errors.callersSkip(0x4, 0xc07d876930)
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/pingcap/errors@v0.11.5-0.20210425183316-da1aaba5fb63/stack.go:203 +0x76
[2021-09-11T15:46:55.416Z] github.com/pingcap/errors.callers(...)
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/pingcap/errors@v0.11.5-0.20210425183316-da1aaba5fb63/stack.go:196
[2021-09-11T15:46:55.416Z] github.com/pingcap/errors.Wrapf(0x46c69c0, 0xc000118cc0, 0x4092be6, 0xb, 0xc0793bf148, 0x1, 0x1, 0x40a9c80, 0x12)
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/pingcap/errors@v0.11.5-0.20210425183316-da1aaba5fb63/errors.go:240 +0x113
[2021-09-11T15:46:55.416Z] github.com/pingcap/failpoint.(*Failpoints).Eval(0x6631d40, 0xc07d879640, 0x33, 0xc0793bf210, 0xe54f71b0e859d9b3, 0x0, 0xc0793bf1d8)
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/pingcap/failpoint@v0.0.0-20210316064728-7acb0f0a3dfd/failpoints.go:209 +0x266
[2021-09-11T15:46:55.416Z] github.com/pingcap/failpoint.Eval(0xc07d879640, 0x33, 0x20, 0x4679fc0, 0x1, 0x40a9c80)
[2021-09-11T15:46:55.416Z] 	/nfs/cache/mod/github.com/pingcap/failpoint@v0.0.0-20210316064728-7acb0f0a3dfd/failpoints.go:273 +0x4b
[2021-09-11T15:46:55.416Z] github.com/pingcap/tidb/executor.(*HashAggExec).getPartialResults(0xc07934d600, 0xc005c2a1ba, 0x5, 0xc07c7d3d40, 0xc01bd107d0, 0x0)
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/aggregate.go:1091 +0x3a5
[2021-09-11T15:46:55.416Z] github.com/pingcap/tidb/executor.(*HashAggExec).unparallelExec(0xc07934d600, 0x47212c0, 0xc000124010, 0xc01bd107d0, 0x0, 0xad7a35e9438000)
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/aggregate.go:917 +0xb5
[2021-09-11T15:46:55.416Z] github.com/pingcap/tidb/executor.(*HashAggExec).Next(0xc07934d600, 0x47212c0, 0xc000124010, 0xc01bd107d0, 0xc01bd107d0, 0x0)
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/aggregate.go:766 +0x78
[2021-09-11T15:46:55.416Z] github.com/pingcap/tidb/executor.Next(0x47212c0, 0xc000124010, 0x47265b8, 0xc07934d600, 0xc01bd107d0, 0x0, 0x0)
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-11T15:46:55.416Z] github.com/pingcap/tidb/executor.(*ExplainExec).executeAnalyzeExec(0xc07927d290, 0x47212c0, 0xc000124010, 0x0, 0x0)
[2021-09-11T15:46:55.416Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/explain.go:95 +0x137
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/executor.(*ExplainExec).generateExplainInfo(0xc07927d290, 0x47212c0, 0xc000124010, 0x0, 0xffffffffffffffff, 0xc0793bf570, 0x16494b1, 0x3e28ca0)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/explain.go:105 +0x4d
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/executor.(*ExplainExec).Next(0xc07927d290, 0x47212c0, 0xc000124010, 0xc01bd10780, 0xc061b56380, 0x12aefbb)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/explain.go:59 +0x229
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/executor.Next(0x47212c0, 0xc000124010, 0x4726538, 0xc07927d290, 0xc01bd10780, 0x0, 0x0)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/executor.(*recordSet).Next(0xc01bd106e0, 0x47212c0, 0xc000124010, 0xc01bd10780, 0x0, 0x0)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/adapter.go:146 +0xc9
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/session.GetRows4Test(0x47212c0, 0xc000124010, 0x4778df8, 0xc078a79a00, 0x4721c60, 0xc078a5b980, 0x1, 0x1, 0x1, 0xc078a79a10, ...)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/session/tidb.go:308 +0x1be
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/session.ResultSetToStringSlice(0x47212c0, 0xc000124010, 0x47ac7c8, 0xc078a79a00, 0x4721c60, 0xc078a5b980, 0x18, 0xc0793bf9a8, 0x3187479, 0xc05f8fa000, ...)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/session/tidb.go:326 +0xa5
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/util/testkit.(*TestKit).ResultSetToResultWithCtx(0xc0793bfb58, 0x47212c0, 0xc000124010, 0x4721c60, 0xc078a5b980, 0x46c7ae0, 0xc0795fde00, 0x1)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:428 +0x85
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/util/testkit.(*TestKit).ResultSetToResult(...)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:423
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/util/testkit.(*TestKit).MustQuery(0xc0793bfb58, 0x41b5a3e, 0x58, 0x0, 0x0, 0x0, 0x45d)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:366 +0x3c7
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/executor_test.(*testSerialSuite).TestAggInDisk(0xc001164358, 0xc05f8fa000)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/aggregate_test.go:1489 +0x348
[2021-09-11T15:46:55.417Z] reflect.Value.call(0x404a680, 0xc001164358, 0x2213, 0x4080141, 0x4, 0xc0004efef0, 0x1, 0x1, 0x666b160, 0xc01052ce28, ...)
[2021-09-11T15:46:55.417Z] 	/usr/local/go/src/reflect/value.go:476 +0x8e7
[2021-09-11T15:46:55.417Z] reflect.Value.Call(0x404a680, 0xc001164358, 0x2213, 0xc01052cef0, 0x1, 0x1, 0x0, 0xc01052cf58, 0x3238f25)
[2021-09-11T15:46:55.417Z] 	/usr/local/go/src/reflect/value.go:337 +0xb9
[2021-09-11T15:46:55.417Z] github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc05f8fa000)
[2021-09-11T15:46:55.417Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x6b5
[2021-09-11T15:46:55.417Z] github.com/pingcap/check.(*suiteRunner).forkCall.func1(0xc05fb4c090, 0xc05f8fa000, 0xc097056000)
[2021-09-11T15:46:55.417Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0xbf
[2021-09-11T15:46:55.417Z] created by github.com/pingcap/check.(*suiteRunner).forkCall
[2021-09-11T15:46:55.417Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:734 +0x233
[2021-09-11T15:46:55.417Z] 
[2021-09-11T15:46:55.417Z] goroutine 584423 [select]:
[2021-09-11T15:46:55.417Z] github.com/tikv/client-go/v2/oracle/oracles.(*pdOracle).updateTS(0xc07bf000e0, 0x47212c0, 0xc000124018, 0x77359400)
[2021-09-11T15:46:55.417Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/oracle/oracles/pd.go:229 +0x131
[2021-09-11T15:46:55.417Z] created by github.com/tikv/client-go/v2/oracle/oracles.NewPdOracle
[2021-09-11T15:46:55.417Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210902062307-4fc565e203a9/oracle/oracles/pd.go:77 +0xd3
[2021-09-11T15:46:55.417Z] 
[2021-09-11T15:46:55.417Z] goroutine 584723 [select]:
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc06c9d33e0, 0xc078fae0c0, 0xc078a79a00)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-11T15:46:55.417Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc08fc70640, 0x47212c0, 0xc000124010)
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-11T15:46:55.417Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-11T15:46:55.417Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-11T15:46:55.417Z] FAIL	github.com/pingcap/tidb/executor	540.428s

It seems likely cte_test.go parallelize wrongly. cc @wjhuang2016
@tisonkun tisonkun added the type/bug The issue is confirmed as a bug. label Sep 11, 2021
@tisonkun tisonkun mentioned this issue Sep 11, 2021
Closed
@wjhuang2016
Copy link
Member

@guo-shaoge PTAL

@unconsolable
Copy link
Contributor

I think a possible reason might be timeout in TestT. I am not quite sure, but I made a sample test file in this gist https://gist.github.com/unconsolable/2c94e5bd5dac14e963c356b14d279846
and run results as follows
https://gist.github.com/unconsolable/837c257bca55fefecb277d14369eb3b4
Looks like the call stacks are similar.

@guo-shaoge
Copy link
Collaborator

/assign

@unconsolable
Copy link
Contributor

Another occurrence https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/35868/pipeline

[2021-09-27T09:49:55.421Z] panic: test timed out after 9m0s
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 474906 [running]:
[2021-09-27T09:49:55.421Z] testing.(*M).startAlarm.func1()
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1700 +0xe5
[2021-09-27T09:49:55.421Z] created by time.goFunc
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/time/sleep.go:180 +0x45
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 1 [chan receive, 8 minutes]:
[2021-09-27T09:49:55.421Z] testing.(*T).Run(0xc000de5c80, 0x40a41a1, 0x5, 0x4268b88, 0x133bb01)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1239 +0x2da
[2021-09-27T09:49:55.421Z] testing.runTests.func1(0xc000b72c00)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1511 +0x78
[2021-09-27T09:49:55.421Z] testing.tRunner(0xc000b72c00, 0xc000ebfc60)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-27T09:49:55.421Z] testing.runTests(0xc000590a68, 0x5f6cf60, 0xa, 0xa, 0xc04c82ce95f6a5a0, 0x7deb78fe7d, 0x6663b00, 0x38)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1509 +0x2fe
[2021-09-27T09:49:55.421Z] testing.(*M).Run(0xc000387300, 0x0)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1417 +0x1eb
[2021-09-27T09:49:55.421Z] go.uber.org/goleak.VerifyTestMain(0x46f1ea0, 0xc000387300, 0xc005c61e60, 0x3, 0x3)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.uber.org/goleak@v1.1.11-0.20210813005559-691160354723/testmain.go:53 +0x38
[2021-09-27T09:49:55.421Z] github.com/pingcap/tidb/executor.TestMain(0xc000387300)
[2021-09-27T09:49:55.421Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/main_test.go:31 +0x125
[2021-09-27T09:49:55.421Z] main.main()
[2021-09-27T09:49:55.421Z] 	_testmain.go:311 +0x1f6
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 5688 [chan receive, 8 minutes]:
[2021-09-27T09:49:55.421Z] testing.(*T).Parallel(0xc000de5800)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-27T09:49:55.421Z] github.com/pingcap/tidb/executor_test.TestUnionDistinct(0xc000de5800)
[2021-09-27T09:49:55.421Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:77 +0x57
[2021-09-27T09:49:55.421Z] testing.tRunner(0xc000de5800, 0x4268b90)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-27T09:49:55.421Z] created by testing.(*T).Run
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 60 [select]:
[2021-09-27T09:49:55.421Z] go.opencensus.io/stats/view.(*worker).start(0xc000a79d80)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xcd
[2021-09-27T09:49:55.421Z] created by go.opencensus.io/stats/view.init.0
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x68
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 146 [chan receive]:
[2021-09-27T09:49:55.421Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000590c18)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3ac
[2021-09-27T09:49:55.421Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x85
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 5687 [chan receive, 8 minutes]:
[2021-09-27T09:49:55.421Z] testing.(*T).Parallel(0xc000de5680)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-27T09:49:55.421Z] github.com/pingcap/tidb/executor_test.TestBasicCTE(0xc000de5680)
[2021-09-27T09:49:55.421Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:26 +0x57
[2021-09-27T09:49:55.421Z] testing.tRunner(0xc000de5680, 0x4268af8)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-27T09:49:55.421Z] created by testing.(*T).Run
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 196 [chan receive]:
[2021-09-27T09:49:55.421Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0003ff230)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3ac
[2021-09-27T09:49:55.421Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x85
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 199 [chan receive]:
[2021-09-27T09:49:55.421Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0003ff368)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3ac
[2021-09-27T09:49:55.421Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x85
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 5689 [chan receive, 8 minutes]:
[2021-09-27T09:49:55.421Z] testing.(*T).Parallel(0xc000de5980)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-27T09:49:55.421Z] github.com/pingcap/tidb/executor_test.TestCTEMaxRecursionDepth(0xc000de5980)
[2021-09-27T09:49:55.421Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:106 +0x57
[2021-09-27T09:49:55.421Z] testing.tRunner(0xc000de5980, 0x4268b28)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-27T09:49:55.421Z] created by testing.(*T).Run
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 5690 [chan receive, 8 minutes]:
[2021-09-27T09:49:55.421Z] testing.(*T).Parallel(0xc000de5b00)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1059 +0x119
[2021-09-27T09:49:55.421Z] github.com/pingcap/tidb/executor_test.TestCTEWithLimit(0xc000de5b00)
[2021-09-27T09:49:55.421Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:149 +0x57
[2021-09-27T09:49:55.421Z] testing.tRunner(0xc000de5b00, 0x4268b30)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-27T09:49:55.421Z] created by testing.(*T).Run
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 5691 [semacquire, 8 minutes]:
[2021-09-27T09:49:55.421Z] sync.runtime_Semacquire(0xc00636bd58)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/runtime/sema.go:56 +0x45
[2021-09-27T09:49:55.421Z] sync.(*WaitGroup).Wait(0xc00636bd50)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/sync/waitgroup.go:130 +0x65
[2021-09-27T09:49:55.421Z] github.com/pingcap/check.RunAll(0xc004cabe58, 0x0)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:129 +0x516
[2021-09-27T09:49:55.421Z] github.com/pingcap/check.TestingT(0xc000de5c80)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:99 +0x45c
[2021-09-27T09:49:55.421Z] github.com/pingcap/tidb/executor_test.TestT(0xc000de5c80)
[2021-09-27T09:49:55.421Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor_test.go:109 +0x216
[2021-09-27T09:49:55.421Z] testing.tRunner(0xc000de5c80, 0x4268b88)
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef
[2021-09-27T09:49:55.421Z] created by testing.(*T).Run
[2021-09-27T09:49:55.421Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 6953 [select]:
[2021-09-27T09:49:55.421Z] github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc00552c210)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:96 +0xa5
[2021-09-27T09:49:55.421Z] created by github.com/dgraph-io/ristretto.newDefaultPolicy
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:80 +0x129
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 6949 [select]:
[2021-09-27T09:49:55.421Z] github.com/tikv/client-go/v2/oracle/oracles.(*pdOracle).updateTS(0xc004a18000, 0x4747da0, 0xc0000520d0, 0x77359400)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/oracle/oracles/pd.go:229 +0x131
[2021-09-27T09:49:55.421Z] created by github.com/tikv/client-go/v2/oracle/oracles.NewPdOracle
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/oracle/oracles/pd.go:77 +0xd3
[2021-09-27T09:49:55.421Z] 
[2021-09-27T09:49:55.421Z] goroutine 6954 [select, 8 minutes]:
[2021-09-27T09:49:55.421Z] github.com/dgraph-io/ristretto.(*Cache).processItems(0xc006e80900)
[2021-09-27T09:49:55.421Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:257 +0x9d
[2021-09-27T09:49:55.421Z] created by github.com/dgraph-io/ristretto.NewCache
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:155 +0x205
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6950 [select]:
[2021-09-27T09:49:55.422Z] github.com/tikv/client-go/v2/internal/locate.(*RegionCache).asyncCheckAndResolveLoop(0xc000356180, 0xdf8475800)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/internal/locate/region_cache.go:397 +0xdc
[2021-09-27T09:49:55.422Z] created by github.com/tikv/client-go/v2/internal/locate.NewRegionCache
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/internal/locate/region_cache.go:368 +0x259
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6951 [select]:
[2021-09-27T09:49:55.422Z] github.com/tikv/client-go/v2/tikv.(*KVStore).runSafePointChecker(0xc0057685a0)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/tikv/kv.go:264 +0x136
[2021-09-27T09:49:55.422Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/tikv/kv.go:187 +0x38a
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6952 [select]:
[2021-09-27T09:49:55.422Z] github.com/tikv/client-go/v2/tikv.(*KVStore).safeTSUpdater(0xc0057685a0)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/tikv/kv.go:542 +0x194
[2021-09-27T09:49:55.422Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/tikv/kv.go:188 +0x3ac
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6948 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection.func1(0xc0045f8e80)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1286 +0x8a
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1284 +0x78
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6673 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeDBWorker.run(0xc006b37320, 0xc0056ca0c0)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:96 +0x205
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:199 +0x8f
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6672 [chan receive]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/badger.(*writeWorker).runMergeLSM(0xc006f08b00, 0xc0067d2030)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:154 +0xad
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/badger.startWriteWorker
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:66 +0x1e8
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6947 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).runUpdateSafePointLoop(0xc0045f8e80)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1550 +0x28a
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.NewMVCCStore
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:85 +0x269
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6946 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeLockWorker.run(0xc006b374a0, 0xc0056ca0c0)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:148 +0x3e5
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:206 +0xdc
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6671 [chan receive, locked to thread]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/badger.(*writeWorker).runWriteLSM(0xc006f08b00, 0xc0067d2030)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:141 +0x195
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/badger.startWriteWorker
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:65 +0x1bc
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 474782 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc0a9e950e0, 0xc069f30900, 0x40c4a3a)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc0b1e59b40, 0x4747da0, 0xc0000520a8)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6160 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/check.(*resultTracker)._loopRoutine(0xc005272480)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:464 +0xbc
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/check.(*resultTracker).start
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:444 +0x3f
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6161 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/check.(*suiteRunner).doRun(0xc005272510, 0xc0005eb2c0)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:667 +0x128
[2021-09-27T09:49:55.422Z] github.com/pingcap/check.(*suiteRunner).asyncRun.func1(0xc005272510, 0xc00636bd50, 0xc006794de0)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:650 +0xaa
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/check.(*suiteRunner).asyncRun
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:646 +0x15b
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 474513 [semacquire]:
[2021-09-27T09:49:55.422Z] sync.runtime_Semacquire(0xc069aaa3e8)
[2021-09-27T09:49:55.422Z] 	/usr/local/go/src/runtime/sema.go:56 +0x45
[2021-09-27T09:49:55.422Z] sync.(*WaitGroup).Wait(0xc069aaa3e0)
[2021-09-27T09:49:55.422Z] 	/usr/local/go/src/sync/waitgroup.go:130 +0x65
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/store/copr.(*copIteratorTaskSender).run(0xc097490600)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:459 +0xc6
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/store/copr.(*copIterator).open
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:436 +0x407
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 14432 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc073294c00)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1133 +0xd8
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1126 +0x7f
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6657 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/badger.(*levelsController).runWorker(0xc005db0c40, 0xc004880ba0, 0x3d39900)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/levels.go:212 +0x185
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/badger.(*levelsController).startCompact
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/levels.go:180 +0x93
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 474785 [select]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc0a9e95440, 0xc069f30900, 0x40c4a3a)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-27T09:49:55.422Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc0b1e59c00, 0x4747da0, 0xc0000520a8)
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-27T09:49:55.422Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 6656 [select, 8 minutes]:
[2021-09-27T09:49:55.422Z] github.com/pingcap/badger.(*blobGCHandler).run(0xc005970040, 0xc004880b88)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/blob.go:468 +0xd4
[2021-09-27T09:49:55.422Z] created by github.com/pingcap/badger.(*blobManager).Open
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/blob.go:292 +0x618
[2021-09-27T09:49:55.422Z] 
[2021-09-27T09:49:55.422Z] goroutine 427532 [select]:
[2021-09-27T09:49:55.422Z] github.com/tikv/client-go/v2/txnkv/transaction.keepAlive(0xc0743c5500, 0xc06349c0c0, 0xc0b1484018, 0x13, 0x13, 0xc08c808000)
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/txnkv/transaction/2pc.go:902 +0x105
[2021-09-27T09:49:55.422Z] created by github.com/tikv/client-go/v2/txnkv/transaction.(*ttlManager).run
[2021-09-27T09:49:55.422Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/txnkv/transaction/2pc.go:875 +0xfe
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 6749 [select]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/badger.(*DB).updateSize(0xc0065fbb00, 0xc004d106f0)
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:1039 +0x108
[2021-09-27T09:49:55.423Z] created by github.com/pingcap/badger.Open
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:328 +0xe14
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 6750 [select]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/badger/epoch.(*ResourceManager).collectLoop(0xc006f31080, 0xc004d10708)
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/epoch/manager.go:101 +0xdc
[2021-09-27T09:49:55.423Z] created by github.com/pingcap/badger/epoch.NewResourceManager
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/epoch/manager.go:79 +0xa7
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 6751 [select, 8 minutes]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/badger.Open.func4(0xc004b0c000)
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:342 +0x17c
[2021-09-27T09:49:55.423Z] created by github.com/pingcap/badger.Open
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:339 +0x10a8
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 6818 [chan receive, 8 minutes]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/badger.(*DB).runFlushMemTable(0xc0065fbb00, 0xc004d10c78, 0x0, 0x0)
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:920 +0x190
[2021-09-27T09:49:55.423Z] created by github.com/pingcap/badger.Open
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:361 +0x17b0
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 6670 [select]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/badger.(*writeWorker).runWriteVLog(0xc006f08b00, 0xc0067d2030)
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:94 +0x1dc
[2021-09-27T09:49:55.423Z] created by github.com/pingcap/badger.startWriteWorker
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:64 +0x190
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 434992 [chan receive]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/executor.(*ProjectionExec).parallelExecute(0xc096e56700, 0x4747da0, 0xc0000520a8, 0xc05c802960, 0x0, 0x5e31134b9d1100)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:218 +0xb5
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/executor.(*ProjectionExec).Next(0xc096e56700, 0x4747da0, 0xc0000520a8, 0xc05c802960, 0xc08b35bab0, 0x12b0d9b)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:182 +0xe5
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/executor.Next(0x4747da0, 0xc0000520a8, 0x474d698, 0xc096e56700, 0xc05c802960, 0x0, 0x0)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/executor.(*recordSet).Next(0xc05c802910, 0x4747da0, 0xc0000520a8, 0xc05c802960, 0x0, 0x0)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/adapter.go:147 +0xc9
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/session.GetRows4Test(0x4747da0, 0xc0000520a8, 0x47c86b0, 0xc0662be000, 0x4748740, 0xc0982faa80, 0x1, 0x1, 0x1, 0xc0662be010, ...)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/session/tidb.go:311 +0x1be
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/session.ResultSetToStringSlice(0x4747da0, 0xc0000520a8, 0x47d3ce8, 0xc0662be000, 0x4748740, 0xc0982faa80, 0x18, 0xc022647168, 0x3209399, 0xc061a84000, ...)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/session/tidb.go:329 +0xa5
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/util/testkit.(*TestKit).ResultSetToResultWithCtx(0xc022647378, 0x4747da0, 0xc0000520a8, 0x4748740, 0xc0982faa80, 0x46ee3c0, 0xc05f6b9350, 0x1)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:428 +0x85
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/util/testkit.(*TestKit).ResultSetToResult(...)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:423
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/util/testkit.(*TestKit).MustQuery(0xc022647378, 0xc000a55110, 0x64, 0x0, 0x0, 0x0, 0x64)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:366 +0x3c7
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/executor_test.(*partitionTableSuite).TestParallelApply(0xc000602958, 0xc061a84000)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/partition_table_test.go:2127 +0x1c65
[2021-09-27T09:49:55.423Z] reflect.Value.call(0x407eb20, 0xc000602958, 0x9213, 0x40a2667, 0x4, 0xc0046b9ef0, 0x1, 0x1, 0x669b500, 0xc0046b1628, ...)
[2021-09-27T09:49:55.423Z] 	/usr/local/go/src/reflect/value.go:476 +0x8e7
[2021-09-27T09:49:55.423Z] reflect.Value.Call(0x407eb20, 0xc000602958, 0x9213, 0xc0046b16f0, 0x1, 0x1, 0x0, 0xc08c7a9b80, 0x1)
[2021-09-27T09:49:55.423Z] 	/usr/local/go/src/reflect/value.go:337 +0xb9
[2021-09-27T09:49:55.423Z] github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc061a84000)
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x6b5
[2021-09-27T09:49:55.423Z] github.com/pingcap/check.(*suiteRunner).forkCall.func1(0xc005272510, 0xc061a84000, 0xc06d3d8020)
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0xbf
[2021-09-27T09:49:55.423Z] created by github.com/pingcap/check.(*suiteRunner).forkCall
[2021-09-27T09:49:55.423Z] 	/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:734 +0x233
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 12682 [select]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc064a4c8f0, 0xc064a138c0)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315
[2021-09-27T09:49:55.423Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5
[2021-09-27T09:49:55.423Z] 
[2021-09-27T09:49:55.423Z] goroutine 474512 [runnable]:
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/sessionctx/variable.GetSysVars(0x0)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/sessionctx/variable/sysvar.go:600 +0x1e5
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/sessionctx/variable.NewMockGlobalAccessor(0xffffffffffffffff)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/sessionctx/variable/mock_globalaccessor.go:40 +0x8b
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/util/mock.NewContext(0xc07e3d2e50)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/mock/context.go:358 +0x205
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/expression.newDistSQLFunctionBySig(0xc082dc8900, 0xc000000c2c, 0xc05b60e300, 0xc07e3d2e50, 0x1, 0x1, 0xc0a881a5a0, 0x0, 0x0, 0x50)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/expression/distsql_builtin.go:1062 +0x34
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/expression.PBToExpr(0xc0ab339d40, 0xc07e3d2ce0, 0x2, 0x2, 0xc082dc8900, 0x2, 0x47ca140, 0xc0737199a0, 0x0)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/expression/distsql_builtin.go:1148 +0x591
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/expression.PBToExpr(0xc0ab339cb0, 0xc07e3d2ce0, 0x2, 0x2, 0xc082dc8900, 0x47ca140, 0xc0737199a0, 0x0, 0x0)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/expression/distsql_builtin.go:1142 +0x459
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/store/mockstore/unistore/cophandler.convertToExprs(0xc082dc8900, 0xc07e3d2ce0, 0x2, 0x2, 0xc07e3d2c20, 0x2, 0x2, 0x2, 0xc0a4af84e0, 0xc000102ae0, ...)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/closure_exec.go:189 +0xd7
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/store/mockstore/unistore/cophandler.buildClosureExecutorFromExecutorList(0xc05b60e420, 0xc0a29f0300, 0x3, 0x4, 0xc061d60c00, 0x0, 0xc061d60c00)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/closure_exec.go:134 +0x753
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/store/mockstore/unistore/cophandler.buildClosureExecutor(0xc05b60e420, 0xc071c93440, 0xc0ab3395f0, 0xc05b60e420, 0xc071c93440)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/closure_exec.go:179 +0xf6
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/store/mockstore/unistore/cophandler.handleCopDAGRequest(0xc073719630, 0xc005397080, 0xc0ab3395f0, 0xc0a881a460)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/cop_handler.go:115 +0x217
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/store/mockstore/unistore/cophandler.HandleCopRequestWithMPPCtx(0xc073719630, 0xc005397080, 0xc0ab3395f0, 0x0, 0xc0a881a3c0)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/cop_handler.go:68 +0x65
[2021-09-27T09:49:55.423Z] github.com/pingcap/tidb/store/mockstore/unistore/cophandler.HandleCopRequest(...)
[2021-09-27T09:49:55.423Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/cop_handler.go:56
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*Server).Coprocessor(0xc0056ca280, 0x4747d68, 0xc097490640, 0xc0ab3395f0, 0x0, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/server.go:565 +0x15f
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/mockstore/unistore.(*RPCClient).SendRequest(0xc00090fbf0, 0x4747d68, 0xc097490640, 0xc0059aab10, 0x6, 0xc0700d37a0, 0xdf8475800, 0x46ed2a0, 0x12ad81f, 0xc000480000)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/rpc.go:236 +0x2386
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/mockstore.(*clientRedirector).SendRequest(0xc00552c030, 0x4747d68, 0xc097490640, 0xc0059aab10, 0x6, 0xc0700d37a0, 0xdf8475800, 0xdf8475800, 0x7efee6fa9800, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/redirector.go:61 +0x170
[2021-09-27T09:49:55.424Z] github.com/tikv/client-go/v2/internal/client.reqCollapse.SendRequest(0x4720cc8, 0xc00552c030, 0x4747d68, 0xc097490640, 0xc0059aab10, 0x6, 0xc0700d37a0, 0xdf8475800, 0x615193ba, 0xc015ddb7cc, ...)
[2021-09-27T09:49:55.424Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/internal/client/client_collapse.go:74 +0x12a
[2021-09-27T09:49:55.424Z] github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).sendReqToRegion(0xc0003338a8, 0xc0ab339560, 0xc02ff64620, 0xc0700d37a0, 0xdf8475800, 0x0, 0x0, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/internal/locate/region_request.go:1128 +0x897
[2021-09-27T09:49:55.424Z] github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx(0xc0003338a8, 0xc0ab339560, 0xc0700d37a0, 0x2, 0x1, 0x1, 0xdf8475800, 0x7efebc2eb900, 0xc000333a70, 0x0, ...)
[2021-09-27T09:49:55.424Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/internal/locate/region_request.go:980 +0x474
[2021-09-27T09:49:55.424Z] github.com/tikv/client-go/v2/txnkv/txnsnapshot.(*ClientHelper).SendReqCtx(0xc0974905c0, 0xc0ab339560, 0xc0700d37a0, 0x2, 0x1, 0x1, 0xdf8475800, 0xc074cded00, 0x0, 0x0, ...)
[2021-09-27T09:49:55.424Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20210926100628-3cc2459779ca/txnkv/txnsnapshot/client_helper.go:108 +0x17e
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce(0xc068ab1e00, 0xc081c561c0, 0xc0737192c0, 0xc068ab1d40, 0xc068ab1e00, 0xc081c561c0, 0x2, 0x2, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:730 +0x632
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask(0xc068ab1e00, 0x4747e10, 0xc005e07ad0, 0xc0737192c0, 0xc068ab1d40)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:645 +0x1b3
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/copr.(*copIteratorWorker).run(0xc068ab1e00, 0x4747e10, 0xc005e07ad0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:382 +0xcb
[2021-09-27T09:49:55.424Z] created by github.com/pingcap/tidb/store/copr.(*copIterator).open
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:419 +0xa5
[2021-09-27T09:49:55.424Z] 
[2021-09-27T09:49:55.424Z] goroutine 474780 [select]:
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/copr.(*copIterator).recvFromRespCh(0xc069aaa370, 0x4747da0, 0xc0000520a8, 0xc068ab1d40, 0x0, 0xc0976e0000)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:469 +0x16d
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/store/copr.(*copIterator).Next(0xc069aaa370, 0x4747da0, 0xc0000520a8, 0x0, 0x0, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:568 +0x48f
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/distsql.(*selectResult).fetchResp(0xc0625bc0e0, 0x4747da0, 0xc0000520a8, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/distsql/select_result.go:192 +0x1cd
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/distsql.(*selectResult).Next(0xc0625bc0e0, 0x4747da0, 0xc0000520a8, 0xc073719400, 0x3d05801, 0xc0976e7350)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/distsql/select_result.go:256 +0x65
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*tableResultHandler).nextChunk(0xc005e07860, 0x4747da0, 0xc0000520a8, 0xc073719400, 0xc009503388, 0x2)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/table_reader.go:432 +0x70
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*TableReaderExecutor).Next(0xc000a3e4e0, 0x4747da0, 0xc0000520a8, 0xc073719400, 0x8, 0xc009503410)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/table_reader.go:233 +0x1be
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.Next(0x4747da0, 0xc0000520a8, 0x474dcd8, 0xc000a3e4e0, 0xc073719400, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*TopNExec).loadChunksUntilTotalLimit(0xc096eab2c0, 0x4747da0, 0xc0000520a8, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/sort.go:430 +0x2e5
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*TopNExec).Next(0xc096eab2c0, 0x4747da0, 0xc0000520a8, 0xc073719360, 0x139d685, 0xc0ad956040)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/sort.go:396 +0x2b1
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.Next(0x4747da0, 0xc0000520a8, 0x474dd98, 0xc096eab2c0, 0xc073719360, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*StreamAggExec).consumeCurGroupRowsAndFetchChild(0xc096e56600, 0x4747da0, 0xc0000520a8, 0xc0737193b0, 0x13a30f5, 0xc0ad956000)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/aggregate.go:1367 +0xc9
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*StreamAggExec).consumeOneGroup(0xc096e56600, 0x4747da0, 0xc0000520a8, 0xc0737193b0, 0x0, 0x5e31135bbd1f00)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/aggregate.go:1295 +0x64f
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*StreamAggExec).Next(0xc096e56600, 0x4747da0, 0xc0000520a8, 0xc0737193b0, 0x203001, 0x12b0d9b)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/aggregate.go:1284 +0x9b
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.Next(0x4747da0, 0xc0000520a8, 0x474dc58, 0xc096e56600, 0xc0737193b0, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*SelectionExec).Next(0xc09c2ccdd0, 0x4747da0, 0xc0000520a8, 0xc05c802500, 0x20300b, 0x20300b)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:1302 +0x292
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.Next(0x4747da0, 0xc0000520a8, 0x474d8d8, 0xc09c2ccdd0, 0xc05c802500, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*NestedLoopApplyExec).fetchAllInners(0xc000406c40, 0x4747da0, 0xc0000520a8, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/join.go:963 +0x40a
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*NestedLoopApplyExec).Next(0xc000406c40, 0x4747da0, 0xc0000520a8, 0xc05c8029b0, 0xc0045a8e28, 0x12a9045)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/join.go:1016 +0x8de
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.Next(0x4747da0, 0xc0000520a8, 0x474d518, 0xc000406c40, 0xc05c8029b0, 0x0, 0x0)
[2021-09-27T09:49:55.424Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x32e
[2021-09-27T09:49:55.424Z] github.com/pingcap/tidb/executor.(*projectionInputFetcher).run(0xc096e56780, 0x4747da0, 0xc0000520a8)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:381 +0x294
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:272 +0x746
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 474784 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc0a9e95320, 0xc069f30900, 0x40c4a3a)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc0b1e59bc0, 0x4747da0, 0xc0000520a8)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 12687 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).infoSyncerKeeper(0xc073294c00)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:471 +0x138
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).Init
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:795 +0x70a
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 12683 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc064a4c9a0, 0xc064a138c0)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 12686 [select, 8 minutes]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).topNSlowQueryLoop(0xc073294c00)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:439 +0x1e5
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).Init
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:792 +0x6ca
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 14286 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop.func1(0xc073294c00, 0xc07cf9b470, 0x6fc23ac00)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:953 +0x125
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:944 +0xc5
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 12681 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/ddl.(*delRange).startEmulator(0xc064a3caf0)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:144 +0x128
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/ddl.(*delRange).start
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:126 +0x6a
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 12680 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs(0xc064bb6e00)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:263 +0x2d9
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:351 +0x2d5
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 12688 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).topologySyncerKeeper(0xc073294c00)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:497 +0x193
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).Init
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:799 +0x770
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 13056 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop.func1(0xc073294c00, 0x476a268, 0xc071718780, 0x47c86b0, 0xc0660e7c00)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1070 +0xde
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1063 +0x8f
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 13055 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1(0xc073294c00, 0x476a268, 0xc071718780)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1035 +0x157
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1022 +0x73
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 14431 [select, 8 minutes]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop.func1(0xc073294c00, 0x47c86b0, 0xc06fc4a800)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1104 +0x12f
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1096 +0x2b8
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 14158 [select, 3 minutes]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop.func1(0xc073294c00, 0xc07e92b4d8, 0x45d964b800, 0x47c86b0, 0xc065d58a00)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:904 +0x125
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:895 +0x1f0
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 474783 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc0a9e95200, 0xc069f30900, 0x40c4a3a)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc0b1e59b80, 0x4747da0, 0xc0000520a8)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-27T09:49:55.425Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-27T09:49:55.425Z] 
[2021-09-27T09:49:55.425Z] goroutine 474781 [select]:
[2021-09-27T09:49:55.425Z] github.com/pingcap/tidb/executor.readProjectionInput(0xc0a9e94f60, 0xc069f30900, 0x40c4a3a)
[2021-09-27T09:49:55.425Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:460 +0x90
[2021-09-27T09:49:55.426Z] github.com/pingcap/tidb/executor.(*projectionWorker).run(0xc0b1e59b00, 0x4747da0, 0xc0000520a8)
[2021-09-27T09:49:55.426Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:428 +0x14e
[2021-09-27T09:49:55.426Z] created by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare
[2021-09-27T09:49:55.426Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/projection.go:276 +0x799
[2021-09-27T09:49:55.426Z] FAIL	github.com/pingcap/tidb/executor	541.211s

@qw4990
Copy link
Contributor

qw4990 commented Oct 29, 2021

https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/41682/

@qw4990 qw4990 mentioned this issue Oct 29, 2021
Merged
12 tasks
@guo-shaoge
Copy link
Collaborator

guo-shaoge commented Oct 29, 2021 •
edited
Loading

https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/41682/

Looks like all coroutines are blocked in t.Parallel(), still wroking on it.

@guo-shaoge
Copy link
Collaborator

guo-shaoge commented Oct 29, 2021 •
edited
Loading

I think a possible reason might be timeout in TestT. I am not quite sure, but I made a sample test file in this gist https://gist.github.com/unconsolable/2c94e5bd5dac14e963c356b14d279846 and run results as follows https://gist.github.com/unconsolable/837c257bca55fefecb277d14369eb3b4 Looks like the call stacks are similar.

Looks like reasonable. Have we met this kind of hang before? @tisonkun

I will delete t.Parallel() in cte_test.go and give it a try(#29243).

@tisonkun
Copy link
Contributor Author

Have we met this kind of hang before?

Sorry I don't have an idea.

@guo-shaoge
Copy link
Collaborator

I can reproduce the ci error in #29243, the following cases all appears in the error log:

24:[2021-10-29T08:15:56.102Z]   /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/main_test.go:47 +0x1eb
56:[2021-10-29T08:15:56.102Z]   /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/adapter_test.go:26 +0x45
66:[2021-10-29T08:15:56.102Z]   /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/slow_query_test.go:488 +0x45
76:[2021-10-29T08:15:56.102Z]   /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:29 +0x57
86:[2021-10-29T08:15:56.102Z]   /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:79 +0x54
96:[2021-10-29T08:15:56.103Z]   /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:111 +0x54
106:[2021-10-29T08:15:56.103Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:176 +0x57
116:[2021-10-29T08:15:56.103Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:127 +0x45
126:[2021-10-29T08:15:56.103Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:144 +0x45
136:[2021-10-29T08:15:56.103Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:159 +0x54
146:[2021-10-29T08:15:56.103Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:106 +0x57
164:[2021-10-29T08:15:56.103Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/executor_test.go:94 +0x3c
174:[2021-10-29T08:15:56.104Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:77 +0x57
184:[2021-10-29T08:15:56.104Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:26 +0x57
194:[2021-10-29T08:15:56.104Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/batch_point_get_test.go:323 +0x45
204:[2021-10-29T08:15:56.104Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/cte_test.go:149 +0x57
214:[2021-10-29T08:15:56.104Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/delete_test.go:114 +0x45
224:[2021-10-29T08:15:56.104Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/delete_test.go:27 +0x57
278:[2021-10-29T08:15:56.105Z]  /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/executor/explainfor_test.go:331 +0x945

So IMHO, it's because ci timeout and cases in cte_test.go are waiting to run.

@jebter jebter added the sig/execution SIG execution label Dec 7, 2021
@tiancaiamao
Copy link
Contributor

After #30822, this kind of hang is avoided.
Close for now, if it still reproduce, feel free to reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@guo-shaoge guo-shaoge

Labels
component/test severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@tiancaiamao @jebter @ChenPeng2013 @guo-shaoge @qw4990 @unconsolable @wjhuang2016 @tisonkun