topic Re:Benchmark_app error in Intel® Distribution of OpenVINO™ Toolkit

Benchmark_app error

KW5 — Thu, 31 Oct 2024 16:41:13 GMT

Dear Sir or Madam,

Excuse me for my dumb question as I'm new to AI domain.

I'm doing some simple benchmark to some LLM models on my company's hardware. I cloned some LLM models (for example, mistral-7b-instruct-v0.1-int8-ov) from HF openvino page (for example, https://huggingface.co/OpenVINO/mistral-7b-instruct-v0.1-int8-ov).

When I ran it, it showed I need to provide data shapes as "input_ids" is dynamic. The typical error message is as follows.

Exception: Input input_ids is dynamic. Provide data shapes!

I have read some docs online (mainly at "https://docs.openvino.ai/"), but still could not figure it out.

For an IR format LLM model, how could I figure out what data shapes I should provide? For a new LLM model on HF, I tried to convert it to IR format by adding data shapes but always failed. Do you have a step-by-step example to show me how to do it?

Your prompt response is appreciated!

Kevin

Re: Benchmark_app error

Aznie_Intel — Sat, 02 Nov 2024 02:08:59 GMT

Hi KW5,

Thanks for reaching out.

For LLM models you can specify the shape value using the parameter -shape "input_ids[1,1],attention_mask[1,1],position_ids[1,1],beam_idx[1]"

Regards,

Aznie

Re: Benchmark_app error

KW5 — Sat, 02 Nov 2024 04:41:14 GMT

Hi Aznie,

Many thanks for your reply!

Once I applied that input shape, I got the following error message. Could you advise what I should do? Thanks a lot!

[Step 7/11] Loading the model to the device
[ ERROR ] Exception from src/inference/src/cpp/core.cpp:104:
Exception from src/inference/src/dev/plugin.cpp:53:
Exception from src/plugins/intel_cpu/src/cpu_memory.cpp:410:
Can not create StaticMemory object. The memory desc is undefined

Traceback (most recent call last):
File "/home/amd/openvino_env/lib/python3.12/site-packages/openvino/tools/benchmark/main.py", line 408, in main
compiled_model = benchmark.core.compile_model(model, benchmark.device, device_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/amd/openvino_env/lib/python3.12/site-packages/openvino/runtime/ie_api.py", line 543, in compile_model
super().compile_model(model, device_name, {} if config is None else config),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Exception from src/inference/src/cpp/core.cpp:104:
Exception from src/inference/src/dev/plugin.cpp:53:
Exception from src/plugins/intel_cpu/src/cpu_memory.cpp:410:
Can not create StaticMemory object. The memory desc is undefined

Re: Benchmark_app error

KW5 — Sat, 02 Nov 2024 04:46:54 GMT

Hi Aznie,

Btw, I just noticed from your screenshot that you ran your LLM on GPU. But in my case, I tried to benchmark it on CPU and got that error ("Can not create StaticMemory object. The memory desc is undefined").

Thanks again!

Kevin

Re: Benchmark_app error

KW5 — Sat, 02 Nov 2024 05:23:59 GMT

Hi Aznie,

I just googled the internet and one posting recommends to using llm_bench tool for LLM models instead of using benmark_app. I will try it and let you know. Thanks!

-Kevin

Re: Benchmark_app error

Aznie_Intel — Mon, 04 Nov 2024 23:42:02 GMT

Hi KW5,

How do you download the models and generate the IR files? I didn’t observe any error when running on the CPU plugin. Below is the result when I ran on CPU:

Regards,

Aznie

Re: Benchmark_app error

KW5 — Tue, 05 Nov 2024 22:28:51 GMT

Hi Aznie,

Which model are you running with your above example?

I downloaded existing openvino models from https://huggingface.co/OpenVINO website, and I used
"git clone" to download them.

Could you teach me your regular way to download and convert a model? Thanks!

-Kevin

Re: Benchmark_app error

Aznie_Intel — Wed, 06 Nov 2024 07:30:00 GMT

Hi KW5,

I am using optimum-cli command to download LLM models. However, you can also use Git clone. I observed the same error when running the mistral-7b-instruct-v0.1-int8-ov models with the shapes parameter. I have tried with llm_bench and encountered the error below:

I will check this with the developer and get back to you soon.

Regards,

Aznie

Re:Benchmark_app error

Witold_Intel — Wed, 06 Nov 2024 19:55:46 GMT

Hi Kevin,

could you give me more details of your setup so I can pick the best machine for reproducing your issue?

OpenVINO package version
compute runtime driver version if you're using one
Ubuntu or other OS version
CPU architecture, eg. Elkhart Lake
other details that you consider important

Thank you in advance,

Re:Benchmark_app error

Witold_Intel — Mon, 11 Nov 2024 18:49:15 GMT

Hi Kevin,

Can I have your setup details please?

Re:Benchmark_app error

Witold_Intel — Wed, 13 Nov 2024 20:02:39 GMT

Hi Kevin,

could you share your setup details with us please?

Re:Benchmark_app error

Witold_Intel — Fri, 15 Nov 2024 21:27:21 GMT

Hi Kevin,

please share your setup details with me. Otherwise I won't be able to reproduce your case and will have to close it without response in 3 business days.

Re:Benchmark_app error

Witold_Intel — Mon, 18 Nov 2024 22:23:17 GMT

This is a reminder that your setup details are needed to reproduce the issue.

Re: Benchmark_app error

KW5 — Mon, 18 Nov 2024 23:41:54 GMT

Hi Witold_Intel,

Excuse me for my late response as I was on vacation last couple of days. My setup is not special, and just a regular x86 server with Ubuntu 24.04.

Aznie (above) said he could reproduce the issue, and I'm waiting for his further feedback.

At the same time, I can use "llm_bench" tool to conduct some LLM related testing. So I'm good now.

I met some other questions, I will initiate a new thread as it is a new question.

Re:Benchmark_app error

Witold_Intel — Thu, 21 Nov 2024 17:14:32 GMT

Thank you for your response. Indeed, Aznie has tried to reproduce, I can see it now. In this case I can open a Jira issue with OpenVino developers to investigate further. Can you post the link to the other topic here for completeness?

Re:Benchmark_app error

Witold_Intel — Wed, 27 Nov 2024 18:50:44 GMT

Hi Kevin, did you open a new thread or can we continue to support you in the current one?

Re:Benchmark_app error

Witold_Intel — Fri, 29 Nov 2024 15:02:10 GMT

Hi Kevin, did you open a new thread or can we continue to support you in the current one?

Re:Benchmark_app error

Aznie_Intel — Thu, 12 Dec 2024 01:40:05 GMT

Hi Kevin,

Thank you for your question. If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored.

Regards,

Aznie