Hi,
I am using OpenVino to compile my mobilenet (in onnx format) based model to then later deploy on FPGA (CoreDLA).
When running the compiled IR graph using the benchmark_app on CPU or FPGA I get a major accuracy drop as compared to running it in Pytorch (>20% difference). My thought is that I am missing something in the image preprocessing step.
In pytorch below is the preprocessing thats done on the image before feeding to model:
To preprocess the image before feeding it to the model, we apply the following torch transform. Which means convert the input image from range [0, 255] to [0.0, 1.0] first, and then subtract the Imagenet Mean [0.485, 0.456, 0.406], and divide by Standard deviation [0.229, 0.224, 0.225] in the order of R, G, B.
transform = Compose([
ToTensor(),
Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
When running the model optimizer I have tried using mean_values argument along with scale_values for standard deviation. I tried all different variations and even tried completely dropping the scaling and in all cases I see the accuracy hover around the same.
model optimizer command:
python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --input_model=./model _256.onnx --mean_values=[0.485,0.456,0.406] --scale_values=[0.229,0.224,0.225] --data_type=FP32
Any thoughts on:
1. Best way to pre-process the image (maybe add the implementation to benchmark_app?)
2. is there a way to convert the images to a 0,1 range before applying mean and s.d.
3. Is there something else I am missing?
Thank you!
链接已复制
Hi Dgader,
My answer to your query is as below.
1. Best way to pre-process the image (maybe add the implementation to benchmark_app?)
We would suggest you utilize our Post-Training Optimization Tool where the Accuracy_Checker offers the Preprocessing features. Check out: Preprocessors — OpenVINO™ documentation. You can use aspect_ratio_scale for image rescaling in the accuracy_check configuration file.
Accuracy Checker tools allowed you to change the required setting in order to obtain the highest possible accuracy, you can use this tool to find the best-intended result.
Example of accuracy-check.yml for mobilenet-v2 that you can refer to : https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/mobilenet-v2/accuracy-check.yml
Video on Post-training Optimization Tool (POT): https://www.youtube.com/watch?v=cGQesbWuRhk
2. Is there a way to convert the images to a 0,1 range before applying mean and s.d.
By using our Post-training Optimization Tool (POT) and Accuracy Checker (AC), you do not required to convert (unlike in PyTorch transform as mentioned above), further explanation can be found here: Converting a Model to Intermediate Representation (IR) — OpenVINO™ documentation
You can directly apply Mean and Standard deviation in the accuracy-check.yml
Regards,
Aznie
