OpenVino image scale & benchmark app

[DQader]

Hi, 

I am using OpenVino to compile my mobilenet (in onnx format) based model to then later deploy on FPGA (CoreDLA).

When running the compiled IR graph using the benchmark_app on CPU or FPGA I get a major accuracy drop as compared to running it in Pytorch (>20% difference). My thought is that I am missing something in the image preprocessing step. 

In pytorch below is the preprocessing thats done on the image before feeding to model: 

To preprocess the image before feeding it to the model, we apply the following torch transform. Which means convert the input image from range [0, 255] to [0.0, 1.0] first, and then subtract the Imagenet Mean [0.485, 0.456, 0.406],  and divide by Standard deviation [0.229, 0.224, 0.225] in the order of R, G, B.   

  transform = Compose([

        ToTensor(),

        Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),

    ])

When running the model optimizer I have tried using mean_values argument along with scale_values for standard deviation. I tried all different variations and even tried completely dropping the scaling and in all cases I see the accuracy hover around the same. 

model optimizer command: 

python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --input_model=./model _256.onnx --mean_values=[0.485,0.456,0.406] --scale_values=[0.229,0.224,0.225] --data_type=FP32

Any thoughts on: 

1. Best way to pre-process the image (maybe add the implementation to benchmark_app?)

2. is there a way to convert the images to a 0,1 range before applying mean and s.d.  

3. Is there something else I am missing?

Thank you!

[IntelSupport]

Hi Dqader,

Thanks for reaching out to us.

We are working on this and will share the information with you soon.

Regards,

Aznie

[IntelSupport]

Hi Dgader,

My answer to your query is as below.

1. Best way to pre-process the image (maybe add the implementation to benchmark_app?)

We would suggest you utilize our Post-Training Optimization Tool where the Accuracy_Checker offers the Preprocessing features. Check out: Preprocessors — OpenVINO™ documentation. You can use aspect_ratio_scale for image rescaling in the accuracy_check configuration file.

Accuracy Checker tools allowed you to change the required setting in order to obtain the highest possible accuracy, you can use this tool to find the best-intended result.

Example of accuracy-check.yml for mobilenet-v2 that you can refer to : https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/mobilenet-v2/accuracy-check.yml

Video on Post-training Optimization Tool (POT): https://www.youtube.com/watch?v=cGQesbWuRhk

2. Is there a way to convert the images to a 0,1 range before applying mean and s.d.  

By using our Post-training Optimization Tool (POT) and Accuracy Checker (AC), you do not required to convert (unlike in PyTorch transform as mentioned above), further explanation can be found here: Converting a Model to Intermediate Representation (IR) — OpenVINO™ documentation

You can directly apply Mean and Standard deviation in the accuracy-check.yml

Regards,

Aznie

[IntelSupport]

Hi Dgader,

This thread will no longer be monitored since we have provided the answers. If you need any additional information from Intel, please submit a new question.

Regards,

Aznie