JobManager related fix #4742
Merged
JobManager related fix #4742
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
critical27
added
ready-for-testing
critical27
requested review from
liwenhui-soul,
panda-sheep,
SuperYoko and
pengweisong
SuperYoko
reviewed
Oct 18, 2022
| @@ -81,6 +81,12 @@ ErrOrHosts StorageJobExecutor::getLeaderHost(GraphSpaceID space) { | |||
| it->second.emplace_back(partId); | |||
| } | |||
| } | |||
| // If storage has not report leader distribution to meta and we don't report error here, | |||
| // JobMananger will think of the job consists of 0 task, and the task will not send to any | |||
| // storage. And the job will always be RUNNING. | |||
There was a problem hiding this comment.
Here I got an issue for @liangliangc . That Explorer will execute "SUBMIT JOB STATS" after create space and wait a heartbeat, and job is always running, what do you think? Do we need to handle this?
There was a problem hiding this comment.
Yeah, that is what I told him. To be precise, a heartbeat is not enough, we must wait until leader distribution in meta is not empty. Dashboard or explorer don't want to handle this logic (by checking show hosts or show parts)... So I changed in core. It is indeed a rare case, if storage report even once, it won't reproduce.
There was a problem hiding this comment.
BTW, I am not sure whether we should check leader count same as partition count here... WDYT?
There was a problem hiding this comment.
Maybe a flag to identify a space is ready to supply service.
Maybe use leader count == partition,both okay for me.
SuperYoko
approved these changes
Oct 19, 2022
panda-sheep
approved these changes
Oct 19, 2022
panda-sheep
reviewed
Oct 19, 2022
| @@ -853,6 +871,15 @@ TEST_F(JobManagerTest, StoppableJob) { | |||
| code = jobMgr->stopJob(spaceId, jobId); | |||
| ASSERT_EQ(code, nebula::cpp2::ErrorCode::SUCCEEDED); | |||
|
|
|||
| // check job status again, it still should be running | |||
There was a problem hiding this comment.
Good job, the comments here should be be stopped.
dutor
approved these changes
Oct 19, 2022
caton-hpg
pushed a commit
to caton-hpg/nebula
that referenced
this pull request
Oct 19, 2022
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Sophie-Xie
added a commit
that referenced
this pull request
Oct 20, 2022
Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com>
Sophie-Xie
added a commit
that referenced
this pull request
Oct 20, 2022
* fix dep of loop in go planner (#4736) * fix inappropriate error code from raft (#4737) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix variable types collected and graph crash (#4724) * Fix variable types collected and graph crash add test cases small fix * unskip some test cases related to multiple query parts * small delete * fmt * Fix ldbc BI R5 implementation Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * stats handle the flag use_vertex_key (#4738) * JobManager related fix (#4742) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * download job related fix (#4754) * fixed case-when error (#4744) * fixed case-when error * fix tck * fix tck * fix tck Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Refine go planner (#4750) * refine go planner * update * fix ctest Co-authored-by: jie.wang <38901892+jievince@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: kyle.cao <kyle.cao@vesoft.com> Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: canon <87342612+caton-hpg@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
What problem(s) does this PR solve?
Issue(s) number:
Description:
Two bugs are fixed:
show job, the job is marked as STOPPED, which is not expected. It looks like as below:After this PR, these unstoppable job will still keep in RUNNING state.
2. Some job need to run in storage leader. There is a corner case that, when we create a new space, before storage report leader distribution to meta, we start a job, e.g. STATS. For now JobManager can't select a storage to run the job, and the task won't be sent to storage for execution. Since no storage ever run it, the job will always stay in RUNNING.
How do you solve it?
Special notes for your reviewer, ex. impact of this fix, design document, etc:
Checklist:
Tests:
Affects:
Release notes:
Please confirm whether to be reflected in release notes and how to describe: