Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Cuda compatibility of vllm-openai image #2393

Open
RonanKMcGovern opened this issue Jan 9, 2024 · 2 comments
Open

Improve Cuda compatibility of vllm-openai image #2393

RonanKMcGovern opened this issue Jan 9, 2024 · 2 comments

Comments

@RonanKMcGovern
Copy link
Contributor

Currently the 'https://hub.docker.com/r/vllm/vllm-openai/' image uses Cuda 12.1 - this runs into a lot of cuda versioning issues depending on the drivers used on the underlying GPU.

This makes the image an inconsistent starting point for running on services like vast ai or runpod.

Could the docker image be updated to more dynamically support cuda versions from 11.8 and up?

@RonanKMcGovern
Copy link
Contributor Author

for comparison, the text-generation-inference docker image doesn't have these issues. See here

@LronDC
Copy link

LronDC commented May 20, 2024

really expect an official image that supports cuda 11.8

or please 🙏 provide a guide on how to build cuda 11.8 version vllm-openai image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants