Hi everyone,
I'm currently comparing different embedding models and the compatibility an performance in openvino. As I'm looking for top performing multilingual models I tried to use Alibaba-NLP/gte-Qwen2-1.5B-instruct and BAAI/bge-multilingual-gemma2. I tried the conversion to openvino format with export_model.py and optimum cli.
With export_model.py the conversion cancels during the process. I used these commands:
python export_model.py embeddings --source_model BAAI/bge-multilingual-gemma2 --weight-format fp16 --config_file_path models/config_all.json
python export_model.py embeddings --source_model Alibaba-NLP/gte-Qwen2-1.5B-instruct --weight-format int8 --config_file_path models/config_all.json
With optimum-cli the conversion works but the models cannot be loaded. This is the error which I get doing a request to the model:
APIStatusError: {"error": "Mediapipe graph precondition failed - FAILED_PRECONDITION: CalculatorGraph::Run() failed in Run:
Calculator::Open() for node "OpenVINOModelServerSessionCalculator_1" failed: ; OpenVINOModelServerSessionCalculator failed to load the model
Calculator::Open() for node "OpenVINOModelServerSessionCalculator_2" failed: ; OpenVINOModelServerSessionCalculator failed to load the model"}
Request:
from openai import OpenAI
import numpy as np
client = OpenAI(
base_url="http://localhost:8000/v3",
api_key="unused"
)
model = "Alibaba-NLP/gte-Qwen2-1.5B-instruct"
embedding_responses = client.embeddings.create(
input=[
"That is a happy person",
"That is a happy very person"
],
model=model,
)
embedding_from_string1 = np.array(embedding_responses.data[0].embedding)
embedding_from_string2 = np.array(embedding_responses.data[1].embedding)
cos_sim = np.dot(embedding_from_string1, embedding_from_string2)/(np.linalg.norm(embedding_from_string1)*np.linalg.norm(embedding_from_string2))
print("Similarity score as cos_sim", cos_sim)
Folder:
And the graph file:
input_stream: "REQUEST_PAYLOAD:input"
output_stream: "RESPONSE_PAYLOAD:output"
node {
calculator: "OpenVINOModelServerSessionCalculator"
output_side_packet: "SESSION:tokenizer"
node_options: {
[type.googleapis.com / mediapipe.OpenVINOModelServerSessionCalculatorOptions]: {
servable_name: "Alibaba-NLP/gte-Qwen2-1.5B-instruct_tokenizer_model"
}
}
}
node {
calculator: "OpenVINOModelServerSessionCalculator"
output_side_packet: "SESSION:embeddings"
node_options: {
[type.googleapis.com / mediapipe.OpenVINOModelServerSessionCalculatorOptions]: {
servable_name: "Alibaba-NLP/gte-Qwen2-1.5B-instruct_embeddings_model"
}
}
}
node {
input_side_packet: "TOKENIZER_SESSION:tokenizer"
input_side_packet: "EMBEDDINGS_SESSION:embeddings"
calculator: "EmbeddingsCalculator"
input_stream: "REQUEST_PAYLOAD:input"
output_stream: "RESPONSE_PAYLOAD:output"
node_options: {
[type.googleapis.com / mediapipe.EmbeddingsCalculatorOptions]: {
normalize_embeddings: true,
}
}
}
I think there is a problem with the mediapipe graph which I created manually. Is there any documentation on how to create the folder structure and mediapipe graph? During the export with optimum a lot of addtional files get exported, not like it is with export_model.py. Are there any hints on how to get these models running with openvino?
Thanks!
Link Copied
