could not load model (no success): Unexpected err=ModuleNotFoundError(\"No module named 'optimum'\"), type(err)=<class 'ModuleNotFoundError'

[noblerboy2004]

LocalAI version:

Docker 3.6 Running on Docker-Container
Environment, CPU architecture, OS, and Version:

Linux 162fd9400319 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug

I'm trying to use https://huggingface.co/intfloat/multilingual-e5-base for embedding. I install model from localhost:8080. When running command $ curl http://localhost:8080/v1/embeddings -H "Content-Type: application/json" -d '{"input": "Test","model": "openvino-all-MiniLM-L6-v2"}'

Output will be: Administrator@mainserver MINGW64 /d/Flowise
$ curl http://localhost:8080/v1/embeddings -H "Content-Type: application/json" -d '{"input": "Test","model": "openvino-all-MiniLM-L6-v2"}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 250 100 196 100 54 31 8 0:00:06 0:00:06 --:--:-- 49{"error":{"code":500,"message":"could not load model (no success): Unexpected err=ModuleNotFoundError("No module named 'optimum'"), type(err)=\u003cclass 'ModuleNotFoundError'\u003e","type":""}}
I tried to install pip install optimum, but the result is the same.

To Reproduce

Expected behavior

Logs

08-07 12:36:10 5:36AM DBG GRPC(all-MiniLM-L6-v2-127.0.0.1:38999): stderr Server started. Listening on: 127.0.0.1:38999
2024-08-07 12:36:11 5:36AM DBG GRPC Service Ready
2024-08-07 12:36:11 5:36AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:all-MiniLM-L6-v2 ContextSize:8192 Seed:940775446 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:true NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:32 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/all-MiniLM-L6-v2 Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:OVModelForFeatureExtraction FlashAttention:false NoKVOffload:false}
2024-08-07 12:36:11 5:36AM DBG GRPC(all-MiniLM-L6-v2-127.0.0.1:38999): stderr Error: No module named 'optimum'
2024-08-07 12:36:11 5:36AM ERR Server error error="could not load model (no success): Unexpected err=ModuleNotFoundError("No module named 'optimum'"), type(err)=<class 'ModuleNotFoundError'>" ip=172.18.0.1 latency=6.15637602s method=POST status=500 url=/v1/embeddings
2024-08-07 12:36:58 5:36AM INF Success ip=127.0.0.1 latency="86.705µs"

Additional context

[mudler]

works here with the following container image:

quay.io/go-skynet/local-ai:master-cublas-cuda12

# File: /build/models/all-MiniLM-L6-v2.yaml 
backend: sentencetransformers
name: all-MiniLM-L6-v2
parameters:
  model: all-MiniLM-L6-v2

and:

curl http://localhost:8080/embeddings -X POST -H "Content-Type: application/json" -d '{
  "input": "Your text string goes here",
  "model": "all-MiniLM-L6-v2"
}' | jq "."

[mudler]

ah - got it sorry. You are trying to use the openvino-* models, but that requires to use the container images built for intel (the ones that have sycl in the tag name)

[noblerboy2004]

I tried to pull these below image. But when running container, docker show error "start failed, container intit failed.."