changefeed can't be removed when meeting ErrGCTTLExceeded error

[amyangfei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

2. What did you expect to see?

3. What did you see instead?

   1. Use cdc cli to remove changefeed, and met following error
      
   2. After some time, changefeed list
      
   3. show meta

tiup cdc cli unsafe show-metadata --pd=http://192.168.149.156:2409
Starting component `cdc`: /home/tidb/.tiup/components/cdc/v5.1.0/cdc cli unsafe show-metadata --pd=http://192.168.149.156:2409
Key: /tidb/cdc/capture/edebc987-a43c-4870-b256-22ac4ff3bb6f, Value: {"id":"edebc987-a43c-4870-b256-22ac4ff3bb6f","address":"192.168.149.155:8302","version":"v5.1.0"}
Key: /tidb/cdc/capture/f9930cbe-f43c-46cc-9684-9b120439bc34, Value: {"id":"f9930cbe-f43c-46cc-9684-9b120439bc34","address":"192.168.149.156:8302","version":"v5.1.0"}
Key: /tidb/cdc/changefeed/info/rainbow-task, Value: {"sink-uri":"mysql://root:Usu84DWl_q2l_n9M@172.29.238.212:4920/?worker-count=8\u0026max-txn-row=5000","opts":

{"_changefeed_id":"cli-verify"}
,"create-time":"2021-07-22T15:06:52.599968298+08:00","start-ts":426491933508239361,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","config":{"case-sensitive":true,"enable-old-value":true,"force-replicate":false,"check-gc-safe-point":true,"filter":

{"rules":["*.*"],"ignore-txn-start-ts":null}
,"mounter":{"worker-num":16},"sink":{"dispatchers":null,"protocol":"default"},"cyclic-replication":{"enable":false,"replica-id":0,"filter-replica-ids":null,"id-buckets":0,"sync-ddl":false},"scheduler":{"type":"table-number","polling-time":-1}},"state":"error","history":[1627353897950,1627353898150,1627353898350,1627353898551,1627353898750,1627353898950,1627353899151,1627353899350,1627353899550,1627353899750,1627353899950,1627353900150,1627353900350,1627353900550,1627353900750,1627353900950,1627353901150,1627353901350,1627353901550,1627353901750,1627353901950,1627353902151,1627353902350,1627353902551,1627353902750,1627353902951,1627353903151,1627353903350,1627353903550,1627353903750,1627353903950,1627353904150,1627353904350,1627353904550,1627353904750,1627353904950,1627353905150,1627353905350,1627353905550,1627353905750,1627353905950,1627353906150,1627353906351,1627353906550,1627353906751,1627353906950,1627353907151,1627353907350,1627353907550,1627353907750,1627353907950,1627353908150,1627353908350,1627353908550,1627353908750,1627353908950,1627353909151,1627353909350,1627353909551,1627353909750,1627353909950,1627353910150,1627353910350,1627353910550,1627353910750,1627353910951,1627353911151,1627353911350,1627353911550,1627353911750,1627353911950,1627353912150,1627353912350,1627353912550,1627353912750,1627353912950,1627353913151,1627353913350,1627353913550,1627353913750,1627353913950,1627353914150,1627353914350,1627353914550,1627353914750,1627353914950,1627353915150,1627353915350,1627353915550,1627353915750,1627353915951,1627353916150,1627353916350,1627353916550,1627353916750,1627353916950,1627353917150,1627353917350,1627353917550,1627353917750,1627353917950,1627353918150,1627353918350,1627353918551,1627353918750,1627353918950,1627353919150,1627353919350,1627353919550,1627353919751,1627353919950,1627353920150,1627353920350,1627353920550,1627353920750,1627353920950,1627353921151,1627353921350,1627353921550,1627353921751,1627353921950,1627353922150,1627353922350,1627353922551,1627353922750,1627353922950,1627353923150,1627353923350,1627353923550,1627353923750,1627353923951,1627353924151,1627353924350,1627353924550,1627353924750,1627353924950,1627353925150,1627353925350,1627353925550,1627353925750,1627353925950,1627353926150,1627353926350,1627353926551,1627353926750,1627353926950,1627353927151,1627353927351,1627353927550,1627353927750,1627353927951,1627353928151,1627353928350,1627353928550,1627353928750,1627353928950,1627353929150,1627353929350,1627353929550,1627353929750,1627353929950,1627353930150,1627353930350,1627353930551,1627353930750,1627353930950,1627353931150,1627353931350,1627353931550,1627353931750,1627353931951,1627353932150,1627353932350,1627353932550,1627353932750,1627353932951,1627353933150,1627353933350,1627353933550,1627353933750,1627353933950,1627353934150,1627353934350,1627353934550,1627353934751,1627353934950,1627353935150,1627353935350,1627353935550,1627353935750,1627353935950,1627353936150,1627353936350,1627353936550,1627353936750,1627353936950,1627353937150,1627353937351,1627353937550,1627353937750,1627353937950,1627353938151,1627353938350,1627353938550,1627353938750,1627353938950,1627353939150,1627353939350,1627353939550,1627353939750,1627353939950,1627353940150,1627353940350,1627353940550,1627353940750,1627353940951,1627353941150,1627353941351,1627353941550,1627353941750,1627353941950,1627353942150,1627353942350,1627353942550,1627353942750,1627353942950,1627353943150,1627353943350,1627353943550,1627353943750,1627353943950,1627353944150,1627353944350,1627353944550,1627353944751,1627353944950,1627353945150,1627353945350,1627353945550,1627353945751,1627353945950,1627353946150,1627353946351,1627353946550,1627353946750,1627353946950,1627353947150,1627353947350,1627353947551,1627353947750,1627353947950,1627353948150,1627353948350,1627353948550,1627353948750,1627353948950,1627353949150,1627353949351,1627353949550,1627353949750,1627353949950,1627353950151,1627353950350,1627353950551,1627353950751,1627353950950,1627353951150,1627353951350,1627353951550,1627353951750,1627353951950,1627353952150,1627353952350,1627353952550,1627353952750,1627353952950,1627353953150,1627353953350,1627353953550,1627353953750,1627353953951,1627353954150,1627353954350,1627353954550,1627353954750,1627353954950,1627353955150,1627353955350,1627353955550,1627353955750,1627353955950,1627353956151,1627353956350,1627353956551,1627353956750,1627353956950,1627353957150,1627353957350,1627353957550,1627353957750,1627353957951,1627353958150,1627353958350,1627353958550,1627353958750,1627353958950,1627353959150,1627353959350,1627353959550,1627353959750,1627353959950,1627353960151,1627353960351,1627353960550,1627353960750,1627353960950,1627353961150,1627353961351,1627353961551,1627353961751,1627353961951,1627353962150,1627353962351,1627353962551,1627353962751,1627353962952,1627353963150,1627353963351,1627353963550,1627353963750,1627353963950,1627353964150,1627353964350,1627353964550,1627353964750,1627353964951,1627353965151,1627353965350,1627353965550,1627353965750,1627353965950,1627353966150,1627353966351,1627353966550,1627353966750,1627353966950,1627353967150,1627353967350,1627353967550,1627353967750,1627353967950,1627353968150,1627353968350,1627353968550,1627353968750,1627353968950,1627353969150,1627353969350,1627353969550,1627353969750,1627353969951,1627353970150,1627353970350,1627353970550,1627353970750,1627353970950,1627353971150,1627353971351,1627353971550,1627353971750,1627353971950,1627353972150,1627353972351,1627353972550,1627353972750,1627353972950,1627353973151,1627353973350,1627353973550,1627353973750,1627353973950,1627353974150,1627353974351,1627353974550,1627353974750,1627353974950,1627353975150,1627353975350,1627353975550,1627353975750,1627353975950,1627353976150,1627353976351,1627353976550,1627353976750,1627353976951,1627353977150,1627353977350,1627353977551,1627353977750,1627353977950,1627353978151,1627353978350,1627353978550,1627353978750,1627353978950,1627353979150,1627353979350,1627353979550,1627353979750,1627353979950,1627353980150,1627353980351,1627353980551,1627353980750,1627353980950,1627353981150,1627353981350,1627353981551,1627353981750,1627353981951,1627353982150,1627353982350,1627353982550,1627353982750,1627353982950,1627353983150,1627353983351,1627353983550,1627353983750,1627353983950,1627353984151,1627353984350,1627353984550,1627353984750,1627353984950,1627353985150,1627353985350,1627353985550,1627353985750,1627353985951,1627353986150,1627353986350,1627353986550,1627353986750,1627353986950,1627353987150,1627353987350,1627353987550,1627353987750,1627353987950,1627353988150,1627353988350,1627353988550,1627353988750,1627353988950,1627353989151,1627353989350,1627353989550,1627353989751,1627353989950,1627353990150,1627353990351,1627353990550,1627353990750,1627353990950,1627353991150,1627353991350,1627353991550,1627353991750,1627353991950,1627353992150,1627353992351,1627353992550,1627353992750,1627353992950,1627353993151,1627353993351,1627353993550,1627353993750,1627353993950,1627353994150,1627353994351,1627353994550,1627353994750,1627353994950,1627353995150,1627353995350,1627353995551,1627353995750,1627353995950,1627353996150,1627353996350,1627353996550,1627353996750,1627353996950,1627353997150,1627353997350,1627353997550,1627353997750,1627353997950,1627353998150,1627353998350,1627353998550,1627353998751,1627353998950,1627353999150,1627353999350,1627353999550,1627353999750,1627353999950,1627354000150,1627354000350,1627354000550,1627354000750,1627354000950,1627354001150,1627354001351,1627354001550,1627354001750,1627354001950,1627354002150,1627354002350,1627354002550,1627354002750,1627354002951,1627354003150,1627354003350,1627354003550,1627354003750,1627354003950,1627354004150,1627354004350,1627354004551,1627354004750,1627354004950,1627354005150,1627354005351,1627354005550,1627354005750,1627354005950,1627354006150,1627354006350,1627354006551,1627354006750,1627354006951,1627354007150,1627354007351,1627354007550,1627354007750,1627354007950,1627354008150,1627354008350,1627354008550,1627354008750,1627354008950,1627354009150,1627354009350,1627354009550,1627354009750,1627354009951,1627354010150,1627354010350,1627354010550,1627354010750,1627354010950,1627354011150,1627354011350,1627354011550,1627354011750,1627354011951,1627354012150,1627354012350,1627354012550,1627354012750,1627354012951,1627354013150,1627354013351,1627354013550,1627354013750,1627354013950,1627354014150,1627354014350,1627354014550,1627354014750,1627354014950,1627354015151,1627354015351,1627354015550,1627354015750,1627354015950,1627354016150,1627354016350,1627354016550,1627354016750,1627354016950,1627354017150,1627354017350,1627354017550,1627354017750],"error":{"addr":"192.168.149.155:8302","code":"CDC:ErrGCTTLExceeded","message":"[CDC:ErrGCTTLExceeded]the checkpoint-ts(426491997759733761) lag of the changefeed(rainbow-task) %!d(MISSING) has exceeded the GC TTL"},"sync-point-enabled":false,"sync-point-interval":600000000000,"creator-version":"v5.1.0"}
Key: /tidb/cdc/job/rainbow-task, Value: {"resolved-ts":426491997759733761,"checkpoint-ts":426491997759733761,"admin-job-type":1}
Key: /tidb/cdc/owner/11ec7ae1abe34311, Value: f9930cbe-f43c-46cc-9684-9b120439bc34
Key: /tidb/cdc/owner/3e5b7ae1abcf2e79, Value: edebc987-a43c-4870-b256-22ac4ff3bb6f
Show 6 KVs

4. After TiCDC servers are reloaded, the changefeed can't be removed. cdc owner logs this error every 200ms

[2021/07/27 14:21:52.057 +08:00] [ERROR] [changefeed.go:106] ["an error occurred in Owner"] [changefeedID=rainbow-task] [error="[CDC:ErrGCTTLExceeded]the checkpoint-ts(426491997759733761) lag of the changefeed(rainbow-task) %!d(MISSING) has exceeded the GC TTL"] [errorVerbose="[CDC:ErrGCTTLExceeded]the checkpoint-ts(426491997759733761) lag of the changefeed(rainbow-task) %!d(MISSING) has exceeded the GC TTL\ngitpro.ttaallkk.top/pingcap/errors.AddStack\n\tgitpro.ttaallkk.top/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngitpro.ttaallkk.top/pingcap/errors.(*Error).GenWithStackByArgs\n\tgitpro.ttaallkk.top/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngitpro.ttaallkk.top/pingcap/ticdc/cdc/owner.(*gcManager).CheckStaleCheckpointTs\n\tgitpro.ttaallkk.top/pingcap/ticdc/cdc/owner/gc_manager.go:128\ngitpro.ttaallkk.top/pingcap/ticdc/cdc/owner.(*changefeed).tick\n\tgitpro.ttaallkk.top/pingcap/ticdc/cdc/owner/changefeed.go:128\ngitpro.ttaallkk.top/pingcap/ticdc/cdc/owner.(*changefeed).Tick\n\tgitpro.ttaallkk.top/pingcap/ticdc/cdc/owner/changefeed.go:105\ngitpro.ttaallkk.top/pingcap/ticdc/cdc/owner.(*Owner).Tick\n\tgitpro.ttaallkk.top/pingcap/ticdc/cdc/owner/owner.go:134\ngitpro.ttaallkk.top/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\n\tgitpro.ttaallkk.top/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:168\ngitpro.ttaallkk.top/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\n\tgitpro.ttaallkk.top/pingcap/ticdc/cdc/capture/capture.go:261\ngitpro.ttaallkk.top/pingcap/ticdc/cdc/capture.(*Capture).campaignOwner\n\tgitpro.ttaallkk.top/pingcap/ticdc/cdc/capture/capture.go:240\ngitpro.ttaallkk.top/pingcap/ticdc/cdc/capture.(*Capture).run.func2\n\tgitpro.ttaallkk.top/pingcap/ticdc/cdc/capture/capture.go:169\nruntime.goexit\n\truntime/asm_amd64.s:1371"]

detail cdc logs:
cdc_new.zip

4. Versions of the cluster

   • TiCDC version (execute cdc version):

     v5.1.0