Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

Runtime Error with OVModelForCausalLM

Berdy
New Contributor I
3,180 Views

Hi,

 

I am running a OVModelForCausalLM sample code which results in a RuntimeError while compiling.

My code:

def ovConvertAlt(optm_dir, model_id, device):

'''OpenVINO Convert Alternate - Conduct model optimization via Intel OpenVINO.

 

Args:

optm_dir:= path to save OpenVINO model

model_id:= model repo identifier

device:= one of "CPU", "GPU"

Return: None

'''

from optimum.intel.openvino import OVModelForCausalLM

 

model_ov = OVModelForCausalLM.from_pretrained(model_id, export=True, device=device)

model_ov.save_pretrained(optm_dir)

 

device is set to 'GPU'

optm_dir is set to the path to save the model

model_id is set to "databricks/dolly-v2-3b"

The execution ends in RuntimeError: with being identical to that of that listed at

https://docs.openvino.ai/2023.0/notebooks/240-dolly-2-instruction-following-with-output.html

 

Saving external data to one file...
Compiling the model...
Set CACHE_DIR to /tmp/tmpdi3xa6vi/model_cache
Traceback (most recent call last):

...

File "/home/jellybean/workspace/optimum_hf/src/./ovino_optimum.py", line 115, in ovConvertAlt
model_ov = OVModelForCausalLM.from_pretrained(model_id, export=True, device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

...

File "/home/jellybean/.virtualenvs/optimum_hf/lib64/python3.11/site-packages/openvino/runtime/ie_api.py", line 398, in compile_model
super().compile_model(model, device_name, {} if config is None else config),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Check 'false' failed at src/inference/src/core.cpp:114:
clEnqueueMapBuffer

 

Can someone help decipher this issue?

 

Thanks

 

 

 

 

 

0 Kudos
1 Solution
Berdy
New Contributor I
2,755 Views

Hairul,

 

Thanks for the link, I had been on that page before, but somehow I missed the Dolly v2 notebook on it.

I ran the code to confirm that it does work. I am however still having issues with the same code running as a Python module in my venv rather than as a notebook.

Checking thru installed packages there are different versions for torch, cuda, and some others. I will try to realign the packages and try again therafter.

For the time being lets consider the issue closed.

 

Thanks,

Berdy

View solution in original post

0 Kudos
8 Replies
Hairul_Intel
Moderator
3,133 Views

Hi Berdy,

Thank you for reaching out to us.

 

We're investigating this issue and will update you on any findings as soon as possible.

 

 

Regards,

Hairul


0 Kudos
Berdy
New Contributor I
3,116 Views

Hi Hairul_Intel,

Thanks for the reply. Perhaps the following provides further info for the investigation.

 

I attempted to first convert databricks/dolly-v2-3b to ONNX, and then convert from ONNX to OpenVINO with mo from openvino.tools. I never get to create the OpenVINO model since the conversion to ONNX fails. ONNX conversion code:

 

def onnxModel(onnx_file_path, model_id):

'''ONNX Model - Convert HF Pytorch to ONNX model

 

Args:

onnx_file_path:= path to onnx model file name

model_id:= HF model repo identifier

Return: None

'''

 

from transformers import AutoModelForCausalLM, AutoTokenizer

from transformers.onnx import export, FeaturesManager

 

tokenizer = AutoTokenizer.from_pretrained(model_id)

hf_model = AutoModelForCausalLM.from_pretrained(model_id)

model_kind, model_onnx_config = FeaturesManager.check_supported_model_or_raise(hf_model, feature="text-generation")

onnx_config = model_onnx_config(hf_model.config)

 

# export to onnx format

export(preprocessor=tokenizer, model=hf_model, config=onnx_config, opset=onnx_config.default_onnx_opset, output=onnx_file_path)

 

This results in KeyError: "gpt-neox is not supported yet..., at line

 

model_features = FeaturesManager.get_supported_features_for_model_type(model_type, model_name=model_name)

 

Indicates that gpt-neo is supported, but not gpt-neox which is the basis for Dolly-v2.

 

The strange thing is that with the previously posted ovConvertAlt() function, which uses OVModelForCausalLM.from_pretrained

shows the conversion to ONNX being completed. That code fails during compile, not during ONNX conversion. Perhaps the compile fails because model.onnx is configured incorrectly.

 

Best,

Berdy

 

 

 

0 Kudos
Hairul_Intel
Moderator
3,062 Views

Hi Berdy,

We've escalated this issue to the relevant teams for further investigation and will update you on any findings as soon as possible.

 

 

Regards,

Hairul


0 Kudos
Hairul_Intel
Moderator
2,792 Views

Hi Berdy,

Thank for your patience.

 

We've received feedback from the Engineering team regarding this issue. This issue is no longer observed on the Master branch. Here is the result from running the notebook from the Master branch:

 image-2023-10-13-13-47-18-903.png

 

 

Please have a try using the Master branch and let us know if you require any further assistance.

 

 

Regards,

Hairul

 

0 Kudos
Berdy
New Contributor I
2,776 Views

Hi Hairul,

 

Thanks for the reply.

Can you kindly post the link for the notebook on the Master branch? I know that I should be able to find it but somehow I can't. I would like to verify the code there against mine since I am still getting the same issue so far.

 

Thank you,

Berdy

0 Kudos
Hairul_Intel
Moderator
2,767 Views

Hi Berdy,

The current Master branch would be the "main" branch.

Screenshot (447).png

 

Running the default git clone command should get you the main branch:

 

git clone --depth=1 https://github.com/openvinotoolkit/openvino_notebooks.git

 

 

Regards,

Hairul

 

0 Kudos
Berdy
New Contributor I
2,756 Views

Hairul,

 

Thanks for the link, I had been on that page before, but somehow I missed the Dolly v2 notebook on it.

I ran the code to confirm that it does work. I am however still having issues with the same code running as a Python module in my venv rather than as a notebook.

Checking thru installed packages there are different versions for torch, cuda, and some others. I will try to realign the packages and try again therafter.

For the time being lets consider the issue closed.

 

Thanks,

Berdy

0 Kudos
Hairul_Intel
Moderator
2,737 Views

Hi Berdy,

This thread will no longer be monitored. If you need any additional information from Intel, please submit a new question.

 

 

Regards,

Hairul


0 Kudos
Reply