Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

query analyze_status table got panic #48835

Closed
aytrack opened this issue Nov 23, 2023 · 2 comments · Fixed by #50421
Closed

query analyze_status table got panic #48835

aytrack opened this issue Nov 23, 2023 · 2 comments · Fixed by #50421
Assignees
Labels

Comments

@aytrack
Copy link
Contributor

aytrack commented Nov 23, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. disable auto analyze
  2. write some data with partition table
  3. enable analyze
  4. query analyze status
  5. wait some time, query again

2. What did you expect to see? (Required)

query analyze status success

3. What did you see instead (Required)

query information_schema.analyze_status got panic, wait some time query again and query success

2023/06/08 11:22:01.735 +08:00] [INFO] [owner_daemon.go:56] ["daemon became owner"] [id=9caa5755-4f21-4325-ba08-e3f3211aa223] [daemon-id=LogBackup::Advancer]
[2023/06/08 11:22:06.471 +08:00] [WARN] [collate.go:221] ["The collation latin1_swedish_ci specified on connection is not supported when new collation is enabled, switch to the default collation: utf8mb4_bin"]
[2023/06/08 11:23:38.668 +08:00] [INFO] [set.go:161] ["set global var"] [conn=5585158429288169529] [name=tidb_enable_auto_analyze] [val=ON]
[2023/06/08 11:23:40.755 +08:00] [INFO] [update.go:1182] ["[stats] auto analyze triggered"] [sql="analyze table `xxx`.`xxx`"] [reason="table unanalyzed"]
[2023/06/08 11:23:41.420 +08:00] [INFO] [coprocessor.go:1270] ["[TIME_COP_PROCESS] resp_time:643.755824ms txnStartTS:18446744073709551615 region_id:110 store_addr:tikv-1-peer:20160 kv_process_ms:549 kv_wait_ms:0 kv_read_ms:0 processed_versions:12000 total_versions:12001 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:23999 rocksdb_cache_hit_count:0 rocksdb_read_count:0 rocksdb_read_byte:0"]
[2023/06/08 11:23:44.844 +08:00] [INFO] [handle.go:1425] ["[stats] incrementally update modifyCount"] [tableID=90] [curModifyCnt=12000] [results.BaseModifyCnt=12000] [modifyCount=0]
[2023/06/08 11:23:44.844 +08:00] [INFO] [handle.go:1447] ["[stats] directly update count"] [tableID=90] [results.Count=12000] [count=12000]
[2023/06/08 11:23:46.561 +08:00] [ERROR] [adapter.go:148] ["execute sql panic"] [conn=5585158429288169531] [sql="select * from information_schema.analyze_status where table_schema = 'xxx' and table_name = 'xxx' and job_info like 'auto analyze %' and start_time >= '2023-06-07 16:11:06' order by start_time limit 1"] [stack="github.com/pingcap/tidb/executor.(*recordSet).Next.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:148
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:884
runtime.goPanicIndex
	/usr/local/go/src/runtime/panic.go:113
github.com/pingcap/tidb/util/chunk.setMutRowBytes
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:335
github.com/pingcap/tidb/util/chunk.MutRow.SetDatum
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:305
github.com/pingcap/tidb/util/chunk.MutRow.SetDatums
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:288
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:131
github.com/pingcap/tidb/executor.Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:329
github.com/pingcap/tidb/executor.(*SelectionExec).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1707
github.com/pingcap/tidb/executor.Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:329
github.com/pingcap/tidb/executor.(*TopNExec).loadChunksUntilTotalLimit
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/sort.go:440
github.com/pingcap/tidb/executor.(*TopNExec).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/sort.go:406
github.com/pingcap/tidb/executor.Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:329
github.com/pingcap/tidb/executor.(*ExecStmt).next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1195
github.com/pingcap/tidb/executor.(*recordSet).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:151
github.com/pingcap/tidb/server.(*tidbResultSet).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:428
github.com/pingcap/tidb/server.(*clientConn).writeChunks
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2326
github.com/pingcap/tidb/server.(*clientConn).writeResultSet
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2269
github.com/pingcap/tidb/server.(*clientConn).handleStmt
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2142
github.com/pingcap/tidb/server.(*clientConn).handleQuery
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1903
github.com/pingcap/tidb/server.(*clientConn).dispatch
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1390
github.com/pingcap/tidb/server.(*clientConn).Run
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1170
github.com/pingcap/tidb/server.(*Server).onConn
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:670"]
[2023/06/08 11:23:46.561 +08:00] [INFO] [conn.go:1201] ["command dispatched failed"] [conn=5585158429288169531] [connInfo="id:5585158429288169531, addr:127.0.0.1:34590 status:10, collation:latin1_swedish_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="select * from information_schema.analyze_status where table_schema = 'xxx' and table_name = 'xxx' and job_info like 'auto analyze %' and start_time >= '2023-06-07 16:11:06' order by start_time limit 1"] [txn_mode=PESSIMISTIC] [timestamp=0] [err="runtime error: index out of range [1] with length 0
github.com/pingcap/tidb/executor.(*recordSet).Next.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:147
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:884
runtime.goPanicIndex
	/usr/local/go/src/runtime/panic.go:113
github.com/pingcap/tidb/util/chunk.setMutRowBytes
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:335
github.com/pingcap/tidb/util/chunk.MutRow.SetDatum
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:305
github.com/pingcap/tidb/util/chunk.MutRow.SetDatums
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:288
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:131
github.com/pingcap/tidb/executor.Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:329
github.com/pingcap/tidb/executor.(*SelectionExec).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1707
github.com/pingcap/tidb/executor.Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:329
github.com/pingcap/tidb/executor.(*TopNExec).loadChunksUntilTotalLimit
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/sort.go:440
github.com/pingcap/tidb/executor.(*TopNExec).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/sort.go:406
github.com/pingcap/tidb/executor.Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:329
github.com/pingcap/tidb/executor.(*ExecStmt).next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1195
github.com/pingcap/tidb/executor.(*recordSet).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:151
github.com/pingcap/tidb/server.(*tidbResultSet).Next
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:428
github.com/pingcap/tidb/server.(*clientConn).writeChunks
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2326
github.com/pingcap/tidb/server.(*clientConn).writeResultSet
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2269
github.com/pingcap/tidb/server.(*clientConn).handleStmt
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2142
github.com/pingcap/tidb/server.(*clientConn).handleQuery
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1903
github.com/pingcap/tidb/server.(*clientConn).dispatch
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1390
github.com/pingcap/tidb/server.(*clientConn).Run
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1170
github.com/pingcap/tidb/server.(*Server).onConn
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:670
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1598"]
[2023/06/08 11:23:50.390 +08:00] [INFO] [2pc.go:685] ["[BIG_TXN]"] [session=0] ["key sample"=7480000000000000165f698000000000000001040622641a18b80007038000000000000001] [size=6637194] [keys=94797] [puts=94618] [dels=178] [locks=1] [checks=0] [txnStartTS=442025803708628993]
[2023/06/08 11:23:50.683 +08:00] [INFO] [analyze.go:589] ["analyze table `xxx`.`xxx` has finished"] [partition=] ["job info"="auto analyze table all columns with 256 buckets, 500 topn, 1 samplerate"] ["start time"=2023/06/08 11:23:40.774 +08:00] ["end time"=2023/06/08 11:23:50.679 +08:00] [cost=9.904442555s]

4. What is your TiDB version? (Required)

MySQL [(none)]> select tidb_version();
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v7.2.0-alpha
Edition: Community
Git Commit Hash: 20a442fead53a37e7cda1e3acf6ae2368b34347a
Git Branch: heads/refs/tags/v7.2.0-alpha
UTC Build Time: 2023-06-07 14:34:14
GoVersion: go1.20.3
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+---------------------------------------
@aytrack aytrack added type/bug The issue is confirmed as a bug. sig/planner SIG: Planner severity/major affects-7.5 labels Nov 23, 2023
@ti-chi-bot ti-chi-bot bot added may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Nov 23, 2023
@Rustin170506
Copy link
Member

Can you provide your test table schema and DML?

@Rustin170506
Copy link
Member

The stack:

[2024/01/12 16:10:33.598 +08:00] [WARN] [memtable_reader.go:143] ["CCCCCCCC: stacktrace from panic: \ngoroutine 3703 [running]:\nruntime/debug.Stack()\n\t/root/.go/src/runtime/debug/stack.go:24 +0x5e\ngitpro.ttaallkk.top/pingcap/tidb/pkg/executor.(*MemTableReaderExec).Next.func1()\n\t/root/code/rustin/tidb/pkg/executor/memtable_reader.go:143 +0x85\npanic({0x56f1ee0?, 0xc00864d248?})\n\t/root/.go/src/runtime/panic.go:914 +0x21f\ngitpro.ttaallkk.top/pingcap/tidb/pkg/util/chunk.setMutRowBytes(...)\n\t/root/code/rustin/tidb/pkg/util/chunk/mutrow.go:335\ngitpro.ttaallkk.top/pingcap/tidb/pkg/util/chunk.MutRow.SetDatum({0xc00cb6e780?, 0x2569cc6?}, 0xb, {0x5, 0x0, 0x0, 0x0, {0x59ef2e0, 0xb}, {0xc00d1c2900, ...}, ...})\n\t/root/code/rustin/tidb/pkg/util/chunk/mutrow.go:305 +0x928\ngitpro.ttaallkk.top/pingcap/tidb/pkg/util/chunk.MutRow.SetDatums(...)\n\t/root/code/rustin/tidb/pkg/util/chunk/mutrow.go:288\ngitpro.ttaallkk.top/pingcap/tidb/pkg/executor.(*MemTableReaderExec).Next(0xc00afd4960, {0x63e9c88, 0xc00b9806f0}, 0xc007452c30)\n\t/root/code/rustin/tidb/pkg/executor/memtable_reader.go:147 +0xe05\ngitpro.ttaallkk.top/pingcap/tidb/pkg/executor/internal/exec.Next({0x63e9c88, 0xc00b9806f0}, {0x6411220, 0xc00afd4960}, 0xc007452c30)\n\t/root/code/rustin/tidb/pkg/executor/internal/exec/executor.go:314 +0x2c3\ngitpro.ttaallkk.top/pingcap/tidb/pkg/executor.(*ExecStmt).next(0xc0001c65a0, {0x63e9c88, 0xc00b9806f0}, {0x6411220, 0xc00afd4960}, 0x2d3f7d3?)\n\t/root/code/rustin/tidb/pkg/executor/adapter.go:1252 +0x6e\ngitpro.ttaallkk.top/pingcap/tidb/pkg/executor.(*recordSet).Next(0xc00ced2a20, {0x63e9c88?, 0xc00b9806f0?}, 0xc007452c30)\n\t/root/code/rustin/tidb/pkg/executor/adapter.go:157 +0xb2\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server/internal/resultset.(*tidbResultSet).Next(0x5a1eeb1?, {0x63e9c88?, 0xc00b9806f0?}, 0x1bfcf68?)\n\t/root/code/rustin/tidb/pkg/server/internal/resultset/resultset.go:64 +0x25\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server.(*clientConn).writeChunks(0xc0072c2340, {0x63e9c88, 0xc00b9806f0}, {0x6401a38, 0xc00cde2a40}, 0x0, 0x1bf?)\n\t/root/code/rustin/tidb/pkg/server/conn.go:2315 +0x18a\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server.(*clientConn).writeResultSet(0xc0072c2340, {0x63e9c88, 0xc00b9806f0}, {0x6401a38, 0xc00cde2a40}, 0xa0?, 0x2, 0xc0035dcc60?)\n\t/root/code/rustin/tidb/pkg/server/conn.go:2258 +0x2f0\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server.(*clientConn).handleStmt(0xc0072c2340, {0x63e9cc0, 0xc00bec85a0}, {0x63fecc8?, 0xc00b258240}, {0x0, 0x0, 0x0}, 0x1)\n\t/root/code/rustin/tidb/pkg/server/conn.go:2051 +0x52a\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server.(*clientConn).handleQuery(0xc0072c2340, {0x63e9cc0, 0xc00bec85a0}, {0xc001263861, 0x2f})\n\t/root/code/rustin/tidb/pkg/server/conn.go:1771 +0xb4c\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server.(*clientConn).dispatch(0xc0072c2340, {0x63e9c88?, 0xc0072cd2f0?}, {0xc001263860, 0x30, 0x30})\n\t/root/code/rustin/tidb/pkg/server/conn.go:1345 +0xf8b\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server.(*clientConn).Run(0xc0072c2340, {0x63e9c88, 0xc0072cd2f0})\n\t/root/code/rustin/tidb/pkg/server/conn.go:1121 +0x53e\ngitpro.ttaallkk.top/pingcap/tidb/pkg/server.(*Server).onConn(0xc00311ca20?, 0xc0072c2340)\n\t/root/code/rustin/tidb/pkg/server/server.go:713 +0x89d\ncreated by github.com/pingcap/tidb/pkg/server.(*Server).startNetworkListener in goroutine 556\n\t/root/code/rustin/tidb/pkg/server/server.go:519 +0x78d\n"]
[2024/01/12 16:10:33.691 +08:00] [INFO] [coprocessor.go:1337] ["[TIME_COP_PROCESS] resp_time:2.542687984s txnStartTS:18446744073709551615 region_id:288 store_addr:tikv-1-peer:20160 kv_process_ms:2512 kv_wait_ms:0 kv_read_ms:0 processed_versions:62023 total_versions:62024 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:124045 rocksdb_cache_hit_count:7 rocksdb_read_count:140 rocksdb_read_byte:910393"]
[2024/01/12 16:10:36.282 +08:00] [INFO] [coprocessor.go:1337] ["[TIME_COP_PROCESS] resp_time:2.591155037s txnStartTS:18446744073709551615 region_id:290 store_addr:tikv-1-peer:20160 kv_process_ms:2572 kv_wait_ms:0 kv_read_ms:0 processed_versions:64710 total_versions:64711 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:129419 rocksdb_cache_hit_count:5 rocksdb_read_count:149 rocksdb_read_byte:1073528"]
[2024/01/12 16:10:38.547 +08:00] [INFO] [coprocessor.go:1337] ["[TIME_COP_PROCESS] resp_time:2.264629626s txnStartTS:18446744073709551615 region_id:148 store_addr:tikv-1-peer:20160 kv_process_ms:2246 kv_wait_ms:0 kv_read_ms:0 processed_versions:57366 total_versions:57367 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:114731 rocksdb_cache_hit_count:5 rocksdb_read_count:131 rocksdb_read_byte:959174"]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants