Runtime Error with OVModelForCausalLM

[Berdy]

Hi,

I am running a OVModelForCausalLM sample code which results in a RuntimeError while compiling.

My code:

def ovConvertAlt(optm_dir, model_id, device):

'''OpenVINO Convert Alternate - Conduct model optimization via Intel OpenVINO.

Args:

optm_dir:= path to save OpenVINO model

model_id:= model repo identifier

device:= one of "CPU", "GPU"

Return: None

'''

from optimum.intel.openvino import OVModelForCausalLM

model_ov = OVModelForCausalLM.from_pretrained(model_id, export=True, device=device)

model_ov.save_pretrained(optm_dir)

device is set to 'GPU'

optm_dir is set to the path to save the model

model_id is set to "databricks/dolly-v2-3b"

The execution ends in RuntimeError: with being identical to that of that listed at

https://docs.openvino.ai/2023.0/notebooks/240-dolly-2-instruction-following-with-output.html

Saving external data to one file...
Compiling the model...
Set CACHE_DIR to /tmp/tmpdi3xa6vi/model_cache
Traceback (most recent call last):

...

File "/home/jellybean/workspace/optimum_hf/src/./ovino_optimum.py", line 115, in ovConvertAlt
model_ov = OVModelForCausalLM.from_pretrained(model_id, export=True, device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

...

File "/home/jellybean/.virtualenvs/optimum_hf/lib64/python3.11/site-packages/openvino/runtime/ie_api.py", line 398, in compile_model
super().compile_model(model, device_name, {} if config is None else config),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Check 'false' failed at src/inference/src/core.cpp:114:
clEnqueueMapBuffer

Can someone help decipher this issue?

Thanks

[Hairul_Intel]

Hi Berdy,

Thank you for reaching out to us.

We're investigating this issue and will update you on any findings as soon as possible.

Regards,

Hairul

[Berdy]

Hi Hairul_Intel,

Thanks for the reply. Perhaps the following provides further info for the investigation.

I attempted to first convert databricks/dolly-v2-3b to ONNX, and then convert from ONNX to OpenVINO with mo from openvino.tools. I never get to create the OpenVINO model since the conversion to ONNX fails. ONNX conversion code:

def onnxModel(onnx_file_path, model_id):

'''ONNX Model - Convert HF Pytorch to ONNX model

Args:

onnx_file_path:= path to onnx model file name

model_id:= HF model repo identifier

Return: None

'''

from transformers import AutoModelForCausalLM, AutoTokenizer

from transformers.onnx import export, FeaturesManager

tokenizer = AutoTokenizer.from_pretrained(model_id)

hf_model = AutoModelForCausalLM.from_pretrained(model_id)

model_kind, model_onnx_config = FeaturesManager.check_supported_model_or_raise(hf_model, feature="text-generation")

onnx_config = model_onnx_config(hf_model.config)

# export to onnx format

export(preprocessor=tokenizer, model=hf_model, config=onnx_config, opset=onnx_config.default_onnx_opset, output=onnx_file_path)

This results in KeyError: "gpt-neox is not supported yet..., at line

model_features = FeaturesManager.get_supported_features_for_model_type(model_type, model_name=model_name)

Indicates that gpt-neo is supported, but not gpt-neox which is the basis for Dolly-v2.

The strange thing is that with the previously posted ovConvertAlt() function, which uses OVModelForCausalLM.from_pretrained

shows the conversion to ONNX being completed. That code fails during compile, not during ONNX conversion. Perhaps the compile fails because model.onnx is configured incorrectly.

Best,

Berdy

[Hairul_Intel]

Hi Berdy,

We've escalated this issue to the relevant teams for further investigation and will update you on any findings as soon as possible.

Regards,

Hairul

[Hairul_Intel]

Hi Berdy,

Thank for your patience.

We've received feedback from the Engineering team regarding this issue. This issue is no longer observed on the Master branch. Here is the result from running the notebook from the Master branch:

 

Please have a try using the Master branch and let us know if you require any further assistance.

Regards,

Hairul

[Berdy]

Hi Hairul,

Thanks for the reply.

Can you kindly post the link for the notebook on the Master branch? I know that I should be able to find it but somehow I can't. I would like to verify the code there against mine since I am still getting the same issue so far.

Thank you,

Berdy

[Hairul_Intel]

Hi Berdy,

The current Master branch would be the "main" branch.

Running the default git clone command should get you the main branch:

git clone --depth=1 https://github.com/openvinotoolkit/openvino_notebooks.git

Regards,

Hairul

[Berdy · Accepted solution]

Hairul,

Thanks for the link, I had been on that page before, but somehow I missed the Dolly v2 notebook on it.

I ran the code to confirm that it does work. I am however still having issues with the same code running as a Python module in my venv rather than as a notebook.

Checking thru installed packages there are different versions for torch, cuda, and some others. I will try to realign the packages and try again therafter.

For the time being lets consider the issue closed.

Thanks,

Berdy

[Hairul_Intel]

Hi Berdy,

This thread will no longer be monitored. If you need any additional information from Intel, please submit a new question.

Regards,

Hairul