Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add snapshot for prefix iterate. #3717

Merged
merged 13 commits into from
Jan 21, 2022
Merged

Add snapshot for prefix iterate. #3717

merged 13 commits into from
Jan 21, 2022

Conversation

SuperYoko
Copy link
Contributor

What type of PR is this?

  • bug
  • feature
  • enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:
Use prefix to iter data on the same snapshot of rocksdb.

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

  • Unit test(positive and negative cases)
  • Function test
  • Performance test
  • N/A

Affects:

  • Documentation affected (Please add the label if documentation needs to be modified.)
  • Incompatibility (If it breaks the compatibility, please describe it and add the label.)
  • If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
  • Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

@critical27
Copy link
Contributor

Close #3710

@critical27
Copy link
Contributor

Very quick fix 👍

@SuperYoko SuperYoko changed the title TMP ADD SNAPSHOT Add snapshot for prefix iterate. Jan 17, 2022
@SuperYoko SuperYoko marked this pull request as ready for review January 17, 2022 03:57
@SuperYoko SuperYoko added ready-for-testing PR: ready for the CI test and removed ready-for-testing PR: ready for the CI test labels Jan 17, 2022
@SuperYoko SuperYoko added the cherry-pick-v3.0 PR: need cherry-pick to this version label Jan 17, 2022
}

void ReleaseSnapshot(const void* snapshot) override {
db_->ReleaseSnapshot(reinterpret_cast<const rocksdb::Snapshot*>(snapshot));
Copy link
Contributor

@darionyaphet darionyaphet Jan 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReleaseSnapshot would remove files from file system?

Copy link
Contributor Author

@SuperYoko SuperYoko Jan 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, It's only a memory snapshot. But it do prevent files to be deleted before snapshot is released.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK snapshot will create some hard link about the previous SST files why don't remove them?

Copy link
Contributor Author

@SuperYoko SuperYoko Jan 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK snapshot will create some hard link about the previous SST files why don't remove them?

https://github.com/facebook/rocksdb/wiki/Snapshot

A snapshot captures a point-in-time view of the DB at the time it's created. Snapshots do not persist across DB restarts.

Maybe you mean Checkpoint.

https://github.com/facebook/rocksdb/wiki/Checkpoints

@liwenhui-soul
Copy link
Contributor

what does void* mean? it doesn't have a type?

@SuperYoko
Copy link
Contributor Author

SuperYoko commented Jan 18, 2022

what does void* mean? it doesn't have a type?

I wan't to create a wrapper like iterator first, but it didn't help much understanding.
Anyway, Rocksdb Snapshot pointer should pass to prefix function, and there should be a container to save pointer.

Maybe a warpper of vector boost::any is okey, but it still pass a pointer. I don't have another clear idea. WDYT?

@critical27 critical27 merged commit e14aba8 into vesoft-inc:master Jan 21, 2022
Sophie-Xie added a commit that referenced this pull request Jan 22, 2022
* Add snapshot for prefix iterate.

* Add header comment

* Add happy test

* Fix lint

* Modify engine snap argument position

* Fix other subclass

* Fix code

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
@HarrisChu
Copy link
Contributor

#3782

seems db_upgrader crashed by this

Sophie-Xie added a commit that referenced this pull request Jan 26, 2022
* Add snapshot for prefix iterate.

* Add header comment

* Add happy test

* Fix lint

* Modify engine snap argument position

* Fix other subclass

* Fix code

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
critical27 added a commit that referenced this pull request Jan 26, 2022
* drop host only when it registe heartbeat (#3739)

* Fix race of create tag and edge (#3735)

* fix rece

* polish

* fix review

* Add version for `show hosts` (#3702)

* version

* tck

* pytest

* pytest

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* Update client version to v3.0.0 (#3741)

* Update python package name to v3

* Update client version white list

* Update go mod to v3

* Change the python package name used in tests

* Update client version white list

* Fix create fulltext index failed (#3747)

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* fix writternBy delete by optimizer (#3746)

* fix bugs

* add tck features

* style

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix with & optimizer rule error (#3736)

* fix error

* fix collapseProjectRule

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix upgrade bug (#3772)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* fix 3727 (#3737)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Balance fix (#3668)

* fix core balance in zone removing the only host in zone

* fix balance plan that make redundant part

* fix deadlock when runInternal and finish callback called in the same thread

* fix transleader return TERM_OUT_OF_DATE

* continue running balance task from last status

* fix deadlock when removing part

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Add snapshot for prefix iterate. (#3717)

* Add snapshot for prefix iterate.

* Add header comment

* Add happy test

* Fix lint

* Modify engine snap argument position

* Fix other subclass

* Fix code

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* suppress undesired ASAN error (#3705)

* suppress undesired error

* fix unstable test case

* fix upgrade (#3787)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Create CODEOWNERS (#3770)

Confirm with sophie.xie

* disable data balance (#3756)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix iterator with snapshot(parameter position) (#3785)

Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>
Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com>
Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com>
Co-authored-by: jimingquan <mingquan.ji@vesoft.com>
Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com>
Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com>
Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
@panda-sheep
Copy link
Contributor

Good job

critical27 added a commit that referenced this pull request Feb 8, 2022
* add cmake git hooks options

* githooks option

* include what you use

* make name follow convention

* linker support lld, which is the fastest linker. If ci
install lld when pull request , will set lld as default linker.

* log rotate

* delete werror

* pragma once => ifndef

* pragma once => ifndef

* Werror bison

* format code

* fix issue 3601 (#3666)

* fix meta upgrade for multi instance (#3734)

* drop host only when it registe heartbeat (#3739)

* Fix race of create tag and edge (#3735)

* fix rece

* polish

* fix review

* Add version for `show hosts` (#3702)

* version

* tck

* pytest

* pytest

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* Enhancement/set expression depth by flag (#3722)

* Set expression depth limit by flag.

* Format.

Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>

* Update client version to v3.0.0 (#3741)

* Update python package name to v3

* Update client version white list

* Update go mod to v3

* Change the python package name used in tests

* Update client version white list

* Fix create fulltext index failed (#3747)

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* add comments and adjust log level for meta/processors/admin (#3662)

* add comments and adjust log level for meta admin processors

* change agent hb log level

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* fix writternBy delete by optimizer (#3746)

* fix bugs

* add tck features

* style

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix with & optimizer rule error (#3736)

* fix error

* fix collapseProjectRule

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix upgrade bug (#3772)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* fix 3727 (#3737)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Balance fix (#3668)

* fix core balance in zone removing the only host in zone

* fix balance plan that make redundant part

* fix deadlock when runInternal and finish callback called in the same thread

* fix transleader return TERM_OUT_OF_DATE

* continue running balance task from last status

* fix deadlock when removing part

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Eliminte the useless row collect. (#3363)

* Eliminte the useless row collect.

* Remove comment.

* Format.

* Rebase.

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Add snapshot for prefix iterate. (#3717)

* Add snapshot for prefix iterate.

* Add header comment

* Add happy test

* Fix lint

* Modify engine snap argument position

* Fix other subclass

* Fix code

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Read lock zone when create space. (#3693)

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* suppress undesired ASAN error (#3705)

* suppress undesired error

* fix unstable test case

* 1. add comments and change log level for meta (#3709)

2. remove some unused meta kv files
3. remove unused tests
4. remove unused kv relative tests in metaclient

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* fix upgrade (#3787)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Create CODEOWNERS (#3770)

Confirm with sophie.xie

* disable data balance (#3756)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix iterator with snapshot(parameter position) (#3785)

Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable meta upgrade from v1 (#3798)

* Add gflag validator for client_idle_timeout_secs (#3807)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable zone commands (#3776)

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* Add ci for standalone compile and ut. (#3637)

* Add standalone workflow

* remove tmp note

* Add meta v2 thrift obj for sa

* Add tck test

* Fix port and dist tck

* skip some test on sa

* Fix merge

* Fix add host

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix the promise/future. (#3814)

Co-authored-by: jimingquan <mingquan.ji@vesoft.com>

* add comments & accumulate bug fix for TOSS (#3643)

* 1. add comments & adjust some log level
2. accumulate bug fix for TOSS
  2.1 one space may be affected by another space. when recover prime edges.
  2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to
      leader_lease_failed.
  2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly.
  2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to
      "setAutoUnlock(bool)"

modify update

fix compile error

adjust finish()

adjust finish()

add rcPrepare_ = succeeded

adjust delete local processor

adjust log

use rcPrepare, rcRemote rcCommit

fix resume add edges

adjust UT

disable ioThreadPoolForMeta_

make int64 vid printable

adjust some log to vlog 2

only print prepare succeeded result

only print readable in finish() of resume processor

adjust some log

some log not print term

desc class name in uuid

debug

add some more log

looks like factory may fail?

add leader lease error code

add check when get lock

report leader lease

do some clean

change table prefix to NebulaKeyUtil Style

change some print

disable UT for short

clean

compile error

disable pool monitor

* unify add/delete finish() as update

* not stable will add it back later

* fix a clang warning

* fix a clang10 warning 2

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable add hosts into zone (#3825)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Check active zone before create space (#3822)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* abandon upgrader for old version (#3818)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* put dataVersionKey on create space for storage (#3817)

* put dataVersionKey on create space for storage

* address some comments

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Update PR template (#3809)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* fix meta http replace host handler (#3794)

* fix meta http replace host handler

* fix http replace test and add zone test

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix `Match cannot be followed by use space` (#3748)

* fix bug

* add tck

* remove set

* fix

* rm header

* fix

* remove changes in match

* fix

* enhance annotation

* fix review

* fix

* fix review

* remove redundant

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* support toSet function (#3594)

add test

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Enhance date parser (#3797)

* parse date

* enhance datetime

* add test

* fix review

* rename

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* raft: fix always loop when no log need to send (#3819)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Update CODEOWNERS (#3835)

* Update CODEOWNERS

* Update CODEOWNERS

Confirm with sophie.xie

* refactor AdminClient (#3781)

* refactor AdminClient

* fix test case, unify all getAdminAddrFromStoreAddr into AdminClient

* add more test cases

* address @panda-sheep's comment

* Feature/optimizer trait match (#3769)

* Add trait for pattern match.

* Implement match plan by trait and reduce some reduncant rules.

* Remove unused header.

* Fix warning.

* Let optimize rule decide collection of matched nodes.

Co-authored-by: kyle.cao <kyle.cao@vesoft.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Simplify datetime tostring. (#3712)

* Simplify datetime tostring.

* Remove unused code.

* Fix error.

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* pragma once not allowd (#3842)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* refactor LOG and add comment for meta, meta/processors/job (#3686)

* refactor LOG and add comment for meta, meta/processors/job

* refactor LOG and add comment for meta/processors/zone

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* replace STRING to name_label for zone's name (#3780)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Add test for rc. (#3847)

* Add test for rc.

* Fix cxx.

Confirm with sophie.xie

* cpp lint

Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com>
Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com>
Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com>
Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>
Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com>
Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com>
Co-authored-by: jimingquan <mingquan.ji@vesoft.com>
Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com>
Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com>
Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com>
Co-authored-by: kyle.cao <kyle.cao@vesoft.com>
liwenhui-soul added a commit to liwenhui-soul/nebula that referenced this pull request Feb 15, 2022
* add cmake git hooks options

* githooks option

* include what you use

* make name follow convention

* linker support lld, which is the fastest linker. If ci
install lld when pull request , will set lld as default linker.

* log rotate

* delete werror

* pragma once => ifndef

* pragma once => ifndef

* Werror bison

* format code

* fix issue 3601 (vesoft-inc#3666)

* fix meta upgrade for multi instance (vesoft-inc#3734)

* drop host only when it registe heartbeat (vesoft-inc#3739)

* Fix race of create tag and edge (vesoft-inc#3735)

* fix rece

* polish

* fix review

* Add version for `show hosts` (vesoft-inc#3702)

* version

* tck

* pytest

* pytest

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* Enhancement/set expression depth by flag (vesoft-inc#3722)

* Set expression depth limit by flag.

* Format.

Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>

* Update client version to v3.0.0 (vesoft-inc#3741)

* Update python package name to v3

* Update client version white list

* Update go mod to v3

* Change the python package name used in tests

* Update client version white list

* Fix create fulltext index failed (vesoft-inc#3747)

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* add comments and adjust log level for meta/processors/admin (vesoft-inc#3662)

* add comments and adjust log level for meta admin processors

* change agent hb log level

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* fix writternBy delete by optimizer (vesoft-inc#3746)

* fix bugs

* add tck features

* style

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix with & optimizer rule error (vesoft-inc#3736)

* fix error

* fix collapseProjectRule

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix upgrade bug (vesoft-inc#3772)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* fix 3727 (vesoft-inc#3737)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Balance fix (vesoft-inc#3668)

* fix core balance in zone removing the only host in zone

* fix balance plan that make redundant part

* fix deadlock when runInternal and finish callback called in the same thread

* fix transleader return TERM_OUT_OF_DATE

* continue running balance task from last status

* fix deadlock when removing part

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Eliminte the useless row collect. (vesoft-inc#3363)

* Eliminte the useless row collect.

* Remove comment.

* Format.

* Rebase.

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Add snapshot for prefix iterate. (vesoft-inc#3717)

* Add snapshot for prefix iterate.

* Add header comment

* Add happy test

* Fix lint

* Modify engine snap argument position

* Fix other subclass

* Fix code

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Read lock zone when create space. (vesoft-inc#3693)

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* suppress undesired ASAN error (vesoft-inc#3705)

* suppress undesired error

* fix unstable test case

* 1. add comments and change log level for meta (vesoft-inc#3709)

2. remove some unused meta kv files
3. remove unused tests
4. remove unused kv relative tests in metaclient

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* fix upgrade (vesoft-inc#3787)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Create CODEOWNERS (vesoft-inc#3770)

Confirm with sophie.xie

* disable data balance (vesoft-inc#3756)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix iterator with snapshot(parameter position) (vesoft-inc#3785)

Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable meta upgrade from v1 (vesoft-inc#3798)

* Add gflag validator for client_idle_timeout_secs (vesoft-inc#3807)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable zone commands (vesoft-inc#3776)

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* Add ci for standalone compile and ut. (vesoft-inc#3637)

* Add standalone workflow

* remove tmp note

* Add meta v2 thrift obj for sa

* Add tck test

* Fix port and dist tck

* skip some test on sa

* Fix merge

* Fix add host

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix the promise/future. (vesoft-inc#3814)

Co-authored-by: jimingquan <mingquan.ji@vesoft.com>

* add comments & accumulate bug fix for TOSS (vesoft-inc#3643)

* 1. add comments & adjust some log level
2. accumulate bug fix for TOSS
  2.1 one space may be affected by another space. when recover prime edges.
  2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to
      leader_lease_failed.
  2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly.
  2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to
      "setAutoUnlock(bool)"

modify update

fix compile error

adjust finish()

adjust finish()

add rcPrepare_ = succeeded

adjust delete local processor

adjust log

use rcPrepare, rcRemote rcCommit

fix resume add edges

adjust UT

disable ioThreadPoolForMeta_

make int64 vid printable

adjust some log to vlog 2

only print prepare succeeded result

only print readable in finish() of resume processor

adjust some log

some log not print term

desc class name in uuid

debug

add some more log

looks like factory may fail?

add leader lease error code

add check when get lock

report leader lease

do some clean

change table prefix to NebulaKeyUtil Style

change some print

disable UT for short

clean

compile error

disable pool monitor

* unify add/delete finish() as update

* not stable will add it back later

* fix a clang warning

* fix a clang10 warning 2

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable add hosts into zone (vesoft-inc#3825)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Check active zone before create space (vesoft-inc#3822)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* abandon upgrader for old version (vesoft-inc#3818)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* put dataVersionKey on create space for storage (vesoft-inc#3817)

* put dataVersionKey on create space for storage

* address some comments

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Update PR template (vesoft-inc#3809)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* fix meta http replace host handler (vesoft-inc#3794)

* fix meta http replace host handler

* fix http replace test and add zone test

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix `Match cannot be followed by use space` (vesoft-inc#3748)

* fix bug

* add tck

* remove set

* fix

* rm header

* fix

* remove changes in match

* fix

* enhance annotation

* fix review

* fix

* fix review

* remove redundant

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* support toSet function (vesoft-inc#3594)

add test

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Enhance date parser (vesoft-inc#3797)

* parse date

* enhance datetime

* add test

* fix review

* rename

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* raft: fix always loop when no log need to send (vesoft-inc#3819)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Update CODEOWNERS (vesoft-inc#3835)

* Update CODEOWNERS

* Update CODEOWNERS

Confirm with sophie.xie

* refactor AdminClient (vesoft-inc#3781)

* refactor AdminClient

* fix test case, unify all getAdminAddrFromStoreAddr into AdminClient

* add more test cases

* address @panda-sheep's comment

* Feature/optimizer trait match (vesoft-inc#3769)

* Add trait for pattern match.

* Implement match plan by trait and reduce some reduncant rules.

* Remove unused header.

* Fix warning.

* Let optimize rule decide collection of matched nodes.

Co-authored-by: kyle.cao <kyle.cao@vesoft.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Simplify datetime tostring. (vesoft-inc#3712)

* Simplify datetime tostring.

* Remove unused code.

* Fix error.

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* pragma once not allowd (vesoft-inc#3842)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* refactor LOG and add comment for meta, meta/processors/job (vesoft-inc#3686)

* refactor LOG and add comment for meta, meta/processors/job

* refactor LOG and add comment for meta/processors/zone

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* replace STRING to name_label for zone's name (vesoft-inc#3780)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Add test for rc. (vesoft-inc#3847)

* Add test for rc.

* Fix cxx.

Confirm with sophie.xie

* cpp lint

Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com>
Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com>
Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com>
Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>
Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com>
Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com>
Co-authored-by: jimingquan <mingquan.ji@vesoft.com>
Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com>
Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com>
Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com>
Co-authored-by: kyle.cao <kyle.cao@vesoft.com>
yixinglu added a commit to yixinglu/nebula that referenced this pull request Mar 21, 2022
* add cmake git hooks options

* githooks option

* include what you use

* make name follow convention

* linker support lld, which is the fastest linker. If ci
install lld when pull request , will set lld as default linker.

* log rotate

* delete werror

* pragma once => ifndef

* pragma once => ifndef

* Werror bison

* format code

* fix issue 3601 (vesoft-inc#3666)

* fix meta upgrade for multi instance (vesoft-inc#3734)

* drop host only when it registe heartbeat (vesoft-inc#3739)

* Fix race of create tag and edge (vesoft-inc#3735)

* fix rece

* polish

* fix review

* Add version for `show hosts` (vesoft-inc#3702)

* version

* tck

* pytest

* pytest

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* Enhancement/set expression depth by flag (vesoft-inc#3722)

* Set expression depth limit by flag.

* Format.

Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>

* Update client version to v3.0.0 (vesoft-inc#3741)

* Update python package name to v3

* Update client version white list

* Update go mod to v3

* Change the python package name used in tests

* Update client version white list

* Fix create fulltext index failed (vesoft-inc#3747)

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes

* add comments and adjust log level for meta/processors/admin (vesoft-inc#3662)

* add comments and adjust log level for meta admin processors

* change agent hb log level

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* fix writternBy delete by optimizer (vesoft-inc#3746)

* fix bugs

* add tck features

* style

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix with & optimizer rule error (vesoft-inc#3736)

* fix error

* fix collapseProjectRule

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* fix upgrade bug (vesoft-inc#3772)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* fix 3727 (vesoft-inc#3737)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Balance fix (vesoft-inc#3668)

* fix core balance in zone removing the only host in zone

* fix balance plan that make redundant part

* fix deadlock when runInternal and finish callback called in the same thread

* fix transleader return TERM_OUT_OF_DATE

* continue running balance task from last status

* fix deadlock when removing part

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Eliminte the useless row collect. (vesoft-inc#3363)

* Eliminte the useless row collect.

* Remove comment.

* Format.

* Rebase.

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Add snapshot for prefix iterate. (vesoft-inc#3717)

* Add snapshot for prefix iterate.

* Add header comment

* Add happy test

* Fix lint

* Modify engine snap argument position

* Fix other subclass

* Fix code

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Read lock zone when create space. (vesoft-inc#3693)

Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* suppress undesired ASAN error (vesoft-inc#3705)

* suppress undesired error

* fix unstable test case

* 1. add comments and change log level for meta (vesoft-inc#3709)

2. remove some unused meta kv files
3. remove unused tests
4. remove unused kv relative tests in metaclient

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* fix upgrade (vesoft-inc#3787)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Create CODEOWNERS (vesoft-inc#3770)

Confirm with sophie.xie

* disable data balance (vesoft-inc#3756)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix iterator with snapshot(parameter position) (vesoft-inc#3785)

Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable meta upgrade from v1 (vesoft-inc#3798)

* Add gflag validator for client_idle_timeout_secs (vesoft-inc#3807)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable zone commands (vesoft-inc#3776)

Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* Add ci for standalone compile and ut. (vesoft-inc#3637)

* Add standalone workflow

* remove tmp note

* Add meta v2 thrift obj for sa

* Add tck test

* Fix port and dist tck

* skip some test on sa

* Fix merge

* Fix add host

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix the promise/future. (vesoft-inc#3814)

Co-authored-by: jimingquan <mingquan.ji@vesoft.com>

* add comments & accumulate bug fix for TOSS (vesoft-inc#3643)

* 1. add comments & adjust some log level
2. accumulate bug fix for TOSS
  2.1 one space may be affected by another space. when recover prime edges.
  2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to
      leader_lease_failed.
  2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly.
  2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to
      "setAutoUnlock(bool)"

modify update

fix compile error

adjust finish()

adjust finish()

add rcPrepare_ = succeeded

adjust delete local processor

adjust log

use rcPrepare, rcRemote rcCommit

fix resume add edges

adjust UT

disable ioThreadPoolForMeta_

make int64 vid printable

adjust some log to vlog 2

only print prepare succeeded result

only print readable in finish() of resume processor

adjust some log

some log not print term

desc class name in uuid

debug

add some more log

looks like factory may fail?

add leader lease error code

add check when get lock

report leader lease

do some clean

change table prefix to NebulaKeyUtil Style

change some print

disable UT for short

clean

compile error

disable pool monitor

* unify add/delete finish() as update

* not stable will add it back later

* fix a clang warning

* fix a clang10 warning 2

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* disable add hosts into zone (vesoft-inc#3825)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Check active zone before create space (vesoft-inc#3822)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* abandon upgrader for old version (vesoft-inc#3818)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* put dataVersionKey on create space for storage (vesoft-inc#3817)

* put dataVersionKey on create space for storage

* address some comments

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Update PR template (vesoft-inc#3809)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* fix meta http replace host handler (vesoft-inc#3794)

* fix meta http replace host handler

* fix http replace test and add zone test

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Fix `Match cannot be followed by use space` (vesoft-inc#3748)

* fix bug

* add tck

* remove set

* fix

* rm header

* fix

* remove changes in match

* fix

* enhance annotation

* fix review

* fix

* fix review

* remove redundant

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>

* support toSet function (vesoft-inc#3594)

add test

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Enhance date parser (vesoft-inc#3797)

* parse date

* enhance datetime

* add test

* fix review

* rename

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* raft: fix always loop when no log need to send (vesoft-inc#3819)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Update CODEOWNERS (vesoft-inc#3835)

* Update CODEOWNERS

* Update CODEOWNERS

Confirm with sophie.xie

* refactor AdminClient (vesoft-inc#3781)

* refactor AdminClient

* fix test case, unify all getAdminAddrFromStoreAddr into AdminClient

* add more test cases

* address @panda-sheep's comment

* Feature/optimizer trait match (vesoft-inc#3769)

* Add trait for pattern match.

* Implement match plan by trait and reduce some reduncant rules.

* Remove unused header.

* Fix warning.

* Let optimize rule decide collection of matched nodes.

Co-authored-by: kyle.cao <kyle.cao@vesoft.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* Simplify datetime tostring. (vesoft-inc#3712)

* Simplify datetime tostring.

* Remove unused code.

* Fix error.

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* pragma once not allowd (vesoft-inc#3842)

Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>

* refactor LOG and add comment for meta, meta/processors/job (vesoft-inc#3686)

* refactor LOG and add comment for meta, meta/processors/job

* refactor LOG and add comment for meta/processors/zone

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* replace STRING to name_label for zone's name (vesoft-inc#3780)

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* Add test for rc. (vesoft-inc#3847)

* Add test for rc.

* Fix cxx.

Confirm with sophie.xie

* cpp lint

Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com>
Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com>
Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com>
Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>
Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com>
Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com>
Co-authored-by: jimingquan <mingquan.ji@vesoft.com>
Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com>
Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com>
Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com>
Co-authored-by: kyle.cao <kyle.cao@vesoft.com>

Co-authored-by: yuehua.jia <3423893+jiayuehua@users.noreply.github.com>
Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com>
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com>
Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com>
Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com>
Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com>
Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com>
Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com>
Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com>
Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com>
Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com>
Co-authored-by: jimingquan <mingquan.ji@vesoft.com>
Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com>
Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com>
Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>
Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com>
Co-authored-by: kyle.cao <kyle.cao@vesoft.com>
yixinglu pushed a commit to yixinglu/nebula that referenced this pull request Mar 21, 2022
## What type of PR is this?
- [ ] bug
- [ ] feature
- [X] enhancement

## What problem(s) does this PR solve?
#### Issue(s) number: 

#### Description:

This is some prepare work for later data balance test, mainly two changes:
1. Follow up of vesoft-inc#3717(In that pr we use snapshot when doing data scan, but the commit log id/term is not taken care). So in this PR, we create a rocksdb snapshot at first, read commit log id/term, and use the same snapshot to scan data.
2. We can't simply skip the special raft command, because if we skip the special log after rebooting, the state of raft peer will be wrong.
3. Add some metrics in raft, remove the `SlowOpTracker`.

## How do you solve it?

## Special notes for your reviewer, ex. impact of this fix, design document, etc:

## Checklist:
Tests:
- [X] Unit test(positive and negative cases)
- [ ] Function test
- [ ] Performance test
- [ ] N/A

Affects:
- [ ] Documentation affected (Please add the label if documentation needs to be modified.)
- [ ] Incompatibility (If it breaks the compatibility, please describe it and add the label.)
- [ ] If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
- [ ] Performance impacted: Consumes more CPU/Memory


## Release notes:
not related



Migrated from vesoft-inc#3846

Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>
@SuperYoko SuperYoko deleted the prefix_a_snap branch September 28, 2022 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick-v3.0 PR: need cherry-pick to this version ready-for-testing PR: ready for the CI test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants