Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] Support initialize llumlet by manager #11

Merged
merged 21 commits into from
Aug 21, 2024
Merged

[BugFix] Support initialize llumlet by manager #11

merged 21 commits into from
Aug 21, 2024

Conversation

s5u13b
Copy link
Contributor

@s5u13b s5u13b commented Aug 15, 2024

When the migration backend is gloo, during scale_up/scale_down, llumnix needs to re-initialize the migration communication process group. However, Ray requires that all the processes that attempt to initialize the process group share the same driver process to ensure that the job_id is consistent. This means that all the llumlets should be initialized in the same process in llumnix. Previously, llumnix initialize llumlet in the entrypoint process, and we can have multiple entrypoints in llumnix, so the driver process of llumlets can be different. Therefore, we support initializing llumlet by the manager as another option in llumnix to ensure that all the llumlets can share the same driver process(manager).

docs/Arguments.md Outdated Show resolved Hide resolved
llumnix/backends/utils.py Show resolved Hide resolved
llumnix/entrypoints/vllm/api_server.py Show resolved Hide resolved
@s5u13b s5u13b merged commit 309c296 into main Aug 21, 2024
4 checks passed
@s5u13b s5u13b changed the title [BugFix] Support initialize llumlet in manager [BugFix] Support initialize llumlet by manager Aug 22, 2024
@s5u13b s5u13b deleted the init-llumlet branch August 23, 2024 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants