- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We want to check the count setting of streams and requests for each device in multi-threads ( refer to : Object Detection C++ Demo — OpenVINO™ documentationCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboard — Version(2023.2)).
For GPU can be set requests as 1, 2, 4.
For NPU just can be set requests as 1, when we set it as 2, just one request can work.
For further investigation, we use get_property to get the information for stream, request, but we cannot get the needed information for NPU.
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JohnnyWang1992,
Thank you for reaching out to us.
We are checking with the relevant team regarding this issue. We will get back to you once we receive any feedback from them. Thank you for your patience.
Regards,
Megat
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Helllo JohnnyWang1992,
- The inference execution through the NPU plugin is entirely offloaded to the NPU device, no processing occurs on the CPU.
- The MTL firmware does not support real HW concurrency ( executing multiple inferences in parallel on different tiles)
Because of the above, the NPU plugin forces NPU_STREAMS=1.
However, multiple inference requests can still be triggered concurrently to improve the throughput of the application.
The recommendation for this is to create multiple inference requests from the application.
The object detection demo supports this argument: "-nireq ": "Optional. Number of infer requests. If this option is omitted, number of infer " "requests is determined automatically.";[ https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/object_detection_demo/cpp/main.cpp#L403|https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/object_detection_demo/cpp/main.cpp#L403]
As mentioned in the source code, the application already queries the NPU plugin to get the optimal_number_of_inference_requests. By default, NPU plugin will attempt to minimize latency and will return "1", but this behavior can be modified through the ov::hint::performance_mode property. When set to THROUGHPUT instead of LATENCY (default), plugin will return "4" as the optimal number. Applications might not expose this config in the command line however and few changes in the source code might be required.
Can you please try the below recommendations -
- Explicit nireq=1,2,4 provided as an argument to the demo
- ov::hint::performance_mode=THROUGHPUT provided in the config to core.compile_model()
Please let me know your results.
Thanks
Haarika M
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have tried the multi-request for NPU, if we set the nireq as 1, all of the results are correct, but once we set it more than 1 (like 2), only half of the results are correct, the other ones are wrong, we suspect that the dispatching for NPU request got something wrong.
We also tried to query related attributes “optimal_number_of_infer_requests” in auto device mode, the result is always 1. Even we modified the LATENCY, THROUGHPUT, there is no change. Would like to know the recommended settings for the number of npu, gpu, stream, and request (in throughput mode). For example, GPU, a stream, corresponds to 2 requests.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JohnnyWang1992,
To investigate this issue further, could you please provide us with more information?
What is the model that you used when running the Object Detection C++ Demo and could you provide us the results that you received which show only half the result is correct.
Regards,
Megat
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JohnnyWang1992,
Thank you for your question. If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored.
Regards,
Megat
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page