Slow inference on model converted from tensorflow 2 object detection api

[Aisia]

I have trained a tensorflow object detection model with tensorflow 2.7.0 and converted it to IR with openvino 2021.4.582. Consuming it with either a python implementation or a C++ one yields slow inference time. The model converted was trained using the ssd_resnet50_v1_fpn_640x640_coco17_tpu-8, which is from the tf2 model repository and using ssd_support_api_v2.4.json.

I tried using the object_detection_sample_ssd thinking there was doing something wrong with my implementations but the problem persists with an inference time of 400 ms. I tested the same object_detection_sample_ssd sample with the person-detection-retail-0013 intel model and inference time was 70 ms.

[IntelSupport]

Hi Aisia,

Thanks for reaching out.

Which device plugin and model precision format do you use when inferencing? For your information, in OpenVINO there are 2 ways to enhance the performance:

• During development: Post-training Optimization tool (POT), Neural Network Compression Framework (NNCF), Model Optimizer
• During deployment: tuning inference parameters and optimizing model execution

 You can also combine both ways. Refer to  Performance Optimization Guide for more information.

Apart from that, you can test your model with Benchmark_app to measure your model's performance.

Regards,

Aznie

[Aisia]

Im running inference on CPU with a FP32 model. I have run both my model and the person-detection-retail-0013 with the benchmark app and got this results on a Intel(R) Core(TM) i9-10900 2.80 Ghz with 16 gb ram and windows 10. There is a really big diference in speed even though are both object detection models. 

My model:

Total time: 250136 microseconds
Total CPU time: 250136 microseconds

Count: 240 iterations
Duration: 60230.43 ms
Latency: 253.81 ms
Throughput: 3.94 FPS

person-detection-retail-0013

Total time: 5118 microseconds
Total CPU time: 5118 microseconds

Count: 11204 iterations
Duration: 60005.14 ms
Latency: 5.18 ms
Throughput: 193.16 FPS

The original ssd_resnet50_v1_fpn_640x640_coco17_tpu-8 from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md converted using ssd_support_api_v2.0.json because 2.4 gave me an error.

Total time: 312223 microseconds
Total CPU time: 312223 microseconds

Count: 192 iterations
Duration: 60105.76 ms
Latency: 315.35 ms
Throughput: 3.17 FPS

The original model infered using tensorflow gave me a latency of aprox 470 ms.

Though there is some speed improvement over the original model could i get some insight as to where T2 ODAPI models are supposed to take this long when converted. Could it be a mismatch between tensorflow/object detecion api/openvino versions. Thx in advance for any information you can provide.

[IntelSupport]

Hi Aisia,

The inference speed is affected based on the hardware configuration and also the model layer framework. In your case, your model might have layers that consume more time to be executed. Refer to the  Intel® Distribution of OpenVINO™ toolkit Benchmark Results for the inference performance on a specified hardware configuration and selected models.

Apart from that, check out the Supported Devices , Supported Model Formats and Supported Layers documentation for more information.

Meanwhile, latency measures the inference time (ms) required to process a single input. For batch input, we measure throughput (frames per second, FPS). Check out this What Is Performance documentation for the calculation of the FPS.

Also, the ssd_resnet50_v1_fpn_640x640_coco17_tpu-8 from TensorFlow 2 Detection Model Zoo is trained using Tensorflow 2.2, so the compatible configuration file is ssd_support_api_v.2.0.json. You may refer to the respective configuration file, for a particular TensorFlow version.

Regards,

Aznie

[IntelSupport]

Hi Aisia,

This thread will no longer be monitored since we have provided the information. If you need any additional information from Intel, please submit a new question.

Regards,

Aznie