Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.

OpenVino image scale & benchmark app

DQader
Beginner
401 Views

Hi, 

 

I am using OpenVino to compile my mobilenet (in onnx format) based model to then later deploy on FPGA (CoreDLA).

When running the compiled IR graph using the benchmark_app on CPU or FPGA I get a major accuracy drop as compared to running it in Pytorch (>20% difference). My thought is that I am missing something in the image preprocessing step. 

 

In pytorch below is the preprocessing thats done on the image before feeding to model: 

To preprocess the image before feeding it to the model, we apply the following torch transform. Which means convert the input image from range [0, 255] to [0.0, 1.0] first, and then subtract the Imagenet Mean [0.485, 0.456, 0.406],  and divide by Standard deviation [0.229, 0.224, 0.225] in the order of R, G, B.   

 

  transform = Compose([

        ToTensor(),

        Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),

    ])

 

When running the model optimizer I have tried using mean_values argument along with scale_values for standard deviation. I tried all different variations and even tried completely dropping the scaling and in all cases I see the accuracy hover around the same. 

model optimizer command: 

python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --input_model=./model _256.onnx --mean_values=[0.485,0.456,0.406] --scale_values=[0.229,0.224,0.225] --data_type=FP32

Any thoughts on: 

1. Best way to pre-process the image (maybe add the implementation to benchmark_app?)

2. is there a way to convert the images to a 0,1 range before applying mean and s.d.  

3. Is there something else I am missing?

 

Thank you!

 

 

0 Kudos
3 Replies
IntelSupport
Community Manager
362 Views

 

Hi Dqader,

Thanks for reaching out to us.

We are working on this and will share the information with you soon.

 

Regards,

Aznie


IntelSupport
Community Manager
351 Views

Hi Dgader,

My answer to your query is as below.

 

1. Best way to pre-process the image (maybe add the implementation to benchmark_app?)

 

We would suggest you utilize our Post-Training Optimization Tool where the Accuracy_Checker offers the Preprocessing features. Check out: Preprocessors — OpenVINO™ documentation. You can use aspect_ratio_scale for image rescaling in the accuracy_check configuration file.

Accuracy Checker tools allowed you to change the required setting in order to obtain the highest possible accuracy, you can use this tool to find the best-intended result.

 

Example of accuracy-check.yml for mobilenet-v2 that you can refer to : https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/mobilenet-v2/accuracy-ch...

 

Video on Post-training Optimization Tool (POT): https://www.youtube.com/watch?v=cGQesbWuRhk

 

2. Is there a way to convert the images to a 0,1 range before applying mean and s.d.  

 

By using our Post-training Optimization Tool (POT) and Accuracy Checker (AC), you do not required to convert (unlike in PyTorch transform as mentioned above), further explanation can be found here: Converting a Model to Intermediate Representation (IR) — OpenVINO™ documentation

 

You can directly apply Mean and Standard deviation in the accuracy-check.yml

 

Regards,

Aznie


IntelSupport
Community Manager
290 Views

Hi Dgader,

This thread will no longer be monitored since we have provided the answers. If you need any additional information from Intel, please submit a new question.


Regards,

Aznie


Reply