Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do vtensor need 64K/128K physical memory policy? #24

Open
nalinaly opened this issue Aug 8, 2024 · 0 comments
Open

Do vtensor need 64K/128K physical memory policy? #24

nalinaly opened this issue Aug 8, 2024 · 0 comments

Comments

@nalinaly
Copy link

nalinaly commented Aug 8, 2024

vAttention said that: if use 2M pageSize, 128M physical memory can be wasted per-request in the worst-case in Llama-3-8B (TP-1), but if use 64KB, 128M would be only 4M
Do vtensor have the same problem？ Will vtensor integrate 64K/128K pageSize in the future?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nalinaly