Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add session tool to manage and analysis sessions #3696

Closed
wants to merge 3 commits into from

Conversation

carrey-feng
Copy link
Contributor

@carrey-feng carrey-feng commented Jan 11, 2022

What type of PR is this?

  • bug
  • feature
  • enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:
Metad OOM because some users have frequently created sessions and did not release those sessions.
Finally by more than 8000000 sessions were created. When I tried to restart cluster, metad OOM occured again.
I found that graphd/storaged called waitForMetadReady function of MetaClient to wait for Metad become be available,
Metad loaded all sessions to memory in waitForMetadReady function. Too many sessions caused metad OOM.

How do you solve it?

I develop a session tool to delete extra sessions in the way offline. It directly delete sessions from rocksdb data of all metad after cluster shutdown. It can delete all sessions and delete the sessions for the specified user. like this:

delete all sessions:
session_tool -path data/meta/nebula/0/data -operator delete (need execute it on every metad node)
image

delete all sessions of root user:
session_tool -path data/meta/nebula/0/data -operator delete -user root (need execute it on every metad node)
image

In addition it also counts the number of sessions in rocksdb of metad in real time, like this:
The total number of sessions in cluster:
session_tool -path data/meta/nebula/0/data -operator count
image

The number of sessions per user:
session_tool -path data/meta/nebula/0/data -operator groupCount
image

It can also show sessions info, like this:
Show N sessions:
session_tool -path data/meta/nebula/0/data -operator list -limit N
image

Show N sessions of user root:
session_tool -path data/meta/nebula/0/data -operator list -limit N -user root
image

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

  • Unit test(positive and negative cases)
  • Function test
  • Performance test
  • N/A

Affects:

  • Documentation affected (Please add the label if documentation needs to be modified.)
  • Incompatibility (If it breaks the compatibility, please describe it and add the label.)
  • If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
  • Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

add a session tool to manage and analysis sessions

@CLAassistant
Copy link

CLAassistant commented Jan 11, 2022

CLA assistant check
All committers have signed the CLA.

@Hera-han
Copy link

Hera-han commented Feb 28, 2022

@CPWstatic this PR has been open for a long time, please take a look and give feedback. thanks~

@Sophie-Xie Sophie-Xie added the community Source: who proposed the issue label Feb 28, 2022
@Sophie-Xie Sophie-Xie removed the request for review from critical27 February 28, 2022 08:15

{
if ("groupCount" == op) {
auto map = std::unordered_map<std::string, int>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why don't use std::unordered_map<std::string, int> map directly?

return Status::OK();
}

static Status deleteSession(const char* path, const std::string user) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why don't merge sessionsCount and deleteSession into one ?

@CPWstatic
Copy link
Contributor

The session could be manipulated via admin command. We support read sessions for now, i suggest you implement a delete command rather than a tool.

@carrey-feng
Copy link
Contributor Author

carrey-feng commented Mar 7, 2022

The session could be manipulated via admin command. We support read sessions for now, i suggest you implement a delete command rather than a tool.

@CPWstatic
Too many sessions causing Metad OOM. Cluster can't start normally,admin command can not be used before cluster is available. I can only use offline repair mode to delete sessions to ensure that the cluster is started normally. You can get the reason for the cluster cannot be started normally from issue description

@xtcyclist
Copy link
Contributor

This PR has been opened for too long. It is addressing a valid problem, although I'm not sure whether this problem is still there after almost a year. I've rebased the PR to check the CI results.

@xtcyclist
Copy link
Contributor

Given that the CI here run into errors and this issue has been opened for too long, I'm closing this PR fow now. It is welcomed for anyone to pick this up later, fix related issues, and reopen it.

@xtcyclist xtcyclist closed this Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community Source: who proposed the issue ready-for-testing PR: ready for the CI test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants