benchmark_app fails at step 8

[Brijender_B_Intel]

Not sure, what is missing:

(base) c:\AI\CodeGen\cg350model\ov_model>benchmark_app -m openvino_model.xml
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.1.0-15008-f4afc983258-releases/2024/1
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 173.06 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
[ INFO ] attention_mask (node: attention_mask) : i64 / [...] / [?,?]
[ INFO ] position_ids (node: position_ids) : i64 / [...] / [?,?]
[ INFO ] beam_idx (node: beam_idx) : i32 / [...] / [?]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.model/aten::to/Convert) : f32 / [...] / [?,?,51200]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]
[ INFO ] attention_mask (node: attention_mask) : i64 / [...] / [?,?]
[ INFO ] position_ids (node: position_ids) : i64 / [...] / [?,?]
[ INFO ] beam_idx (node: beam_idx) : i32 / [...] / [?]
[ INFO ] Model outputs:
[ INFO ] logits (node: __module.model/aten::to/Convert) : f32 / [...] / [?,?,51200]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 12218.41 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ ERROR ] Exception from src\inference\src\cpp\compiled_model.cpp:145:
Exception from src\plugins\intel_cpu\src\node.cpp:503:
Node Broadcast_141508 contains less child edges than 1

Traceback (most recent call last):
File "C:\Users\bbharti\AppData\Local\anaconda3\Lib\site-packages\openvino\tools\benchmark\main.py", line 452, in main
value = compiled_model.get_property(k)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Exception from src\inference\src\cpp\compiled_model.cpp:145:
Exception from src\plugins\intel_cpu\src\node.cpp:503:
Node Broadcast_141508 contains less child edges than 1

[Vipin_Singh1]

Hi Brijender, we would like to inform you that we are routing your query to the dedicated team for further assistance.

[Megat_Intel]

Hi Brijender_B_Intel,

Thank you for reaching out to us.

We believe the error you received is due to the model itself. Could you please provide us with the model and the model information for further investigation?

On my end, I was able to run the OpenVINO™ 2024.1.0 benchmark_app on the model efficientdet-d1-tf:

 

Regards,

Megat

[Brijender_B_Intel]

hi.

thanks for looking into this. The model is  hugging face model  salesforce/codegen-350M-muli converted to openvino through optimum command:

optimum-cli export openvino --model salesforce/codegen-350M-multi openvino_model


thanks
Brijender 

[Megat_Intel]

Hi Brijender,

Thank you for providing us with the details.

For your information, I encountered a similar error when running the benchmark_app on the CPU device:

However, I successfully run the benchmark_app on the GPU device. You will need to specify the data shape since the model is dynamic. Please try to run the benchmark_app on the GPU device and specify the shape value using the parameter -shape "input_ids[1,1],attention_mask[1,1],position_ids[1,1],beam_idx[1]":

 

If you encountered the error "Incompatible MatMul matrix dimension", please reduce the number of iterations. On my end, selecting -niter 500 resolves the error.

On another note, I was able to load the model and successfully ran the inference using the code below for both CPU and GPU devices. The error seems to be related to the benchmark_app. We will investigate this issue and get back to you soon.

 

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model = OVModelForCausalLM.from_pretrained("openvino_model")
#model.to("GPU") - for GPU device
prompt = "def hello_world():"
tokenizer = AutoTokenizer.from_pretrained("openvino_model")
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

Regards,

Megat

[Brijender_B_Intel]

hi Megat.

Thanks for debugging it. Yes model inference runs fine but this tool was giving error. Looking forward for cpu fix.

-regards

Brijender