Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docdb] tserver core dump when restoring using PITR #9162

Closed
rao-vasireddy opened this issue Jul 1, 2021 · 0 comments
Closed

[docdb] tserver core dump when restoring using PITR #9162

rao-vasireddy opened this issue Jul 1, 2021 · 0 comments
Assignees
@spolitov
Labels
kind/bug This issue is a bug priority/high High Priority

Comments

@rao-vasireddy
Copy link
Contributor

There were a few RESTORE's in progress and the previous snapshot schedule was deleted and a new one was started and a restore was done using the new schedule. All tservers seem to affected,

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/home/yugabyte/tserver/bin/yb-tserver --flagfile /home/yugabyte/tserver/conf/se'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f9292e15bdf in yb::util::FastDecodeDescendingSignedVarInt (slice=slice@entry=0x7f92735a11e0) at ../../src/yb/util/fast_varint.cc:240
240 ../../src/yb/util/fast_varint.cc: No such file or directory.
(gdb) bt
#0 0x00007f9292e15bdf in yb::util::FastDecodeDescendingSignedVarInt (slice=slice@entry=0x7f92735a11e0) at ../../src/yb/util/fast_varint.cc:240
#1 0x00007f9294d7ec7a in yb::DocHybridTime::DecodeFrom (this=this@entry=0x7f92735a1310, slice=slice@entry=0x7f92735a11e0) at ../../src/yb/common/doc_hybrid_time.cc:91
#2 0x00007f9294d7f150 in yb::DocHybridTime::FullyDecodeFrom (this=this@entry=0x7f92735a1310, encoded=...) at ../../src/yb/common/doc_hybrid_time.cc:130
#3 0x00007f9294d7fca4 in yb::DocHybridTime::DecodeFromEnd (encoded_key_with_ht_at_end=encoded_key_with_ht_at_end@entry=0x7f92735a1340) at ../../src/yb/common/doc_hybrid_time.cc:145
#4 0x00007f929ba8cc4d in yb::docdb::(anonymous namespace)::HybridTimeFilteringIterator::Satisfied (this=0x4a21140, key=...) at ../../src/yb/docdb/docdb_rocksdb_util.cc:351
#5 0x00007f929ba8d7d7 in rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:102
#6 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#7 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#8 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#9 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#10 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#11 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#12 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#13 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#14 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#15 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#16 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#17 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#18 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#19 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#20 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#21 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#22 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#23 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#24 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#25 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#26 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#27 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#28 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#29 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#30 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#31 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#32 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#33 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#34 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#35 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#36 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#37 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#38 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#39 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#40 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#41 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#42 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#43 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#44 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#45 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
#46 0x00007f929ba8d816 in Next (this=0x4a21140) at ../../src/yb/rocksdb/table/filtering_iterator.h:64
#47 rocksdb::FilteringIterator::ApplyFilter (this=this@entry=0x4a21140, backward=false) at ../../src/yb/rocksdb/table/filtering_iterator.h:106
@rao-vasireddy rao-vasireddy added kind/bug This issue is a bug priority/high High Priority labels Jul 1, 2021
spolitov added a commit that referenced this issue Jul 2, 2021
@spolitov
[#9162] Fix stack overflow in filtering iterator
124af42 
Summary:
Filtering iterator could make a call to Next/Prev while checking filter.
But those methods also would check filter.
So, iterating over one skipped entry results in adding two more stack frames, one for Next() or Prev() and one more for ApplyFilter(), and the stack grows proportionally to the number of skipped entries, resulting in a potential stack overflow.

Actually we should call Next/Prev of the underlying iterator in this case, as done in this diff. This approach would only utilize a constant amount of space on the stack and not linear space based on the number of skipped entries.

Test Plan: ybd debug --gtest_filter BackupTxnTest.PointInTimeBigSkipRestore

Reviewers: bogdan, mbautin

Reviewed By: mbautin

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D12143
spolitov added a commit that referenced this issue Jul 2, 2021
@spolitov
[Backport 2.6] [#9162] PITR: Fix stack overflow in filtering iterator
6494075 
Summary:
Filtering iterator could make a call to Next/Prev while checking filter.
But those methods also would check filter.
So, iterating over one skipped entry results in adding two more stack frames, one for Next() or Prev() and one more for ApplyFilter(), and the stack grows proportionally to the number of skipped entries, resulting in a potential stack overflow.

Actually we should call Next/Prev of the underlying iterator in this case, as done in this diff. This approach would only utilize a constant amount of space on the stack and not linear space based on the number of skipped entries.

Original commit: 124af42/D12143

Test Plan:
ybd debug --gtest_filter BackupTxnTest.PointInTimeBigSkipRestore
Jenkins: rebase: 2.6

Reviewers: bogdan, mbautin

Reviewed By: mbautin

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D12153
@spolitov spolitov closed this as completed Jul 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spolitov spolitov

Labels
kind/bug This issue is a bug priority/high High Priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@spolitov @rao-vasireddy