NCS2 produce different result as CPU

Hu__Yiqian · ‎09-05-2019

Dear all,

I've trained a PyTorch model to generate embeddings from images. However, when I use the trained model for inference, CPU and NCS2 produce totally different embeddings and I believe CPU generates the correct result. My code is attached.

import cv2 as cv
import numpy as np

# Read images
image_color = cv.imread('con_localized/0.jpg')

# Load the model.
net = cv.dnn.readNet('models/model_color.xml',
                     'models/model_color.bin')

# Specify target device.
net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU)
# net.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD)

# Prepare input blob and perform an inference.
blob = cv.dnn.blobFromImage(image_color, size=(227, 227), ddepth=cv.CV_8U)
net.setInput(blob)
out = net.forward()
print(out)

The CPU setting is run on macOS and MYRIAD setting is run on Raspberry. Everything else is totally same. Could anyone help me? Many thanks

Shubha_R_Intel · ‎09-06-2019

Dear Hu, Yiqian,

Can you retry this code using traditional Inference Engine Core API, i.e.

        slog::info << "Loading model to the device" << slog::endl;
        ExecutableNetwork executable_network = ie.LoadNetwork(network, FLAGS_d);
        // -----------------------------------------------------------------------------------------------------

        // --------------------------- 5. Create infer request -------------------------------------------------
        slog::info << "Create infer request" << slog::endl;
        InferRequest inferRequest = executable_network.CreateInferRequest();
        // ---------------------------------------------------------------------------

Because otherwise, your question is an OpenCV question, and it really should be posted here:

https://github.com/opencv/opencv/issues/

It could also be a Model Optimizer issue though. What version of OpenVino are you using ? Are you on the latest and greatest 2019R2.01 ? If so, maybe upgrade and see if the problem persists.

If it wouldn't be too much trouble to convert your code to IE Core instead of OpenCV, it would be good to get this data-point so that I can help you.

Thanks,

Shubha

Hu__Yiqian · ‎09-06-2019

Shubha R. (Intel) wrote:
Dear Hu, Yiqian,
Can you retry this code using traditional Inference Engine Core API, i.e.
        slog::info << "Loading model to the device" << slog::endl;
        ExecutableNetwork executable_network = ie.LoadNetwork(network, FLAGS_d);
        // -----------------------------------------------------------------------------------------------------

        // --------------------------- 5. Create infer request -------------------------------------------------
        slog::info << "Create infer request" << slog::endl;
        InferRequest inferRequest = executable_network.CreateInferRequest();
        // ---------------------------------------------------------------------------
Because otherwise, your question is an OpenCV question, and it really should be posted here:

https://github.com/opencv/opencv/issues/

It could also be a Model Optimizer issue though. What version of OpenVino are you using ? Are you on the latest and greatest 2019R2.01 ? If so, maybe upgrade and see if the problem persists.

If it wouldn't be too much trouble to convert your code to IE Core instead of OpenCV, it would be good to get this data-point so that I can help you.

Thanks,

Shubha

Hi Shubha,

Thanks for the reply. My openvino is the latest 2019.R2 version.

I tried the IECore code but the result is the same. However I tried other models and found that the difference is quite small and acceptable. Is this a problem of my model? If so, why and how shall I fix it?

BTW, what do you mean by data-point? Like my .xml and .bin along with my input?

Many thanks again

Shubha_R_Intel · ‎09-09-2019

Dear Hu, Yiqian,

Of course the result should be the same between the OpenCV/Inference Engine approach versus the Inference Engine Core API approach but as I mentioned previously, this is the OpenVino forum (not the OpenCV forum).

I am not sure why your results would be "different" from the CPU. That sounds odd. If you can attach your original model as well as your Model Optimizer command, I can try myself.

It may be a bug.

Thanks,

Shubha

Hu__Yiqian · ‎09-09-2019

Shubha R. (Intel) wrote:
Dear Hu, Yiqian,
Of course the result should be the same between the OpenCV/Inference Engine approach versus the Inference Engine Core API approach but as I mentioned previously, this is the OpenVino forum (not the OpenCV forum).
I am not sure why your results would be "different" from the CPU. That sounds odd. If you can attach your original model as well as your Model Optimizer command, I can try myself.
It may be a bug.
Thanks,
Shubha

Dear Shubha

Thanks for the reply.

Here's my model structure.

class AlexNet(nn.Module):

    def __init__(self, num_channels):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(num_channels, 96, kernel_size=11, stride=4),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(96, 256, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(256, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Linear(6 * 6 * 256, 2048),
            nn.ReLU(inplace=True),
            nn.Linear(2048, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, 128),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

I convert the pytorch model to onnx first.

model_color = Alexnet(3)

image_con_color = images['image_color' ] // load image data
image_con_color = Variable(image_con_color.float()).to(device)
color_output = model_color(image_con_color)

torch.onnx.export(model_color, image_con_color, 'model_color.onnx', input_names=['image_con_color'], output_names=['color_output'])

After that I convert the .onnx into .xml and .bin

cd /opt/intel/openvino/deployment_tools/model_optimizer/
python mo.py --input_model ~/input_dir/model_color.onnx --data_type FP16 --output_dir ~/output_dir

In the end I use the code in the very first post to run on CPU by macOS and NCS2 by raspberry pi. and get the different results.

Result on CPU:

[ 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 1850.8931
8222.642 0. 0. 0. 2135.309 3193.6743
8222.642 4932.039 1411.8556 0. 2135.309 2135.309
0. 0. 0. 10461.765 10461.765 0.
0. 0. 0. 10461.765 14551.347 14551.347
0. 0. 0. 0. 14551.347 14908.998
0. 0. 0. 0. 4479.435 17829.059
0. 0. 545.42224 0. 9005.181 17829.059
0. 0. 545.42224 0. 7507.1333 7507.1333
0. 0. 0. 0. 0. 0.
0. 1268.1013 951.8981 0. 0. 0.
0. 4062.216 3555.8015 0. 366.30554 366.30554
0. 6334.16 6334.16 0. 0. 0.
0. 6334.16 6334.16 0. ]

Result on NCS2:

[[ 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
232.25 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 261. 0. 56.84375 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0.
0. 0. ]]

PS Before training the results don't differ very much. At least zeros and non-zeros aren't messed up. After several epoches' training, it becomes the above result.

Many thanks again!

Shubha_R_Intel · ‎09-10-2019

Dear Hu, Yiqian,

Thanks for all the info. I will attempt to reproduce and file a bug if I can reproduce the discrepancy same as you. I will let you know the results on this forum.

Shubha

Shubha_R_Intel · ‎09-16-2019

Dear Hu, Yiqian,

The code you provided me above is not a complete script.

I reproduced it here:

import torch
import torch.nn as nn
from torchvision.models import AlexNet


class AlexNet(nn.Module):

    def __init__(self, num_channels):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(num_channels, 96, kernel_size=11, stride=4),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(96, 256, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(256, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Linear(6 * 6 * 256, 2048),
            nn.ReLU(inplace=True),
            nn.Linear(2048, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, 128),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

    if __name__ == "__main__":
        model_color = AlexNet(3)

        image_con_color = images['image_color'] // load
        image_con_color = Variable(image_con_color.float()).to(device)
        color_output = model_color(image_con_color)

        torch.onnx.export(model_color, image_con_color, 'model_color.onnx', input_names=['image_con_color'],
                          output_names=['color_output'])

But I get an error:

image_con_color = images['image_color'] // load
NameError: name 'images' is not defined

In order to go further with this issue, I need the original ONNX model, not just the IR. I actually sent you a Private Message about this.

Thanks,

Shubha

Shubha_R_Intel · ‎09-18-2019

Dear Hu, Yiqian,

The problem is fixed in OpenVino 2019R3 as you can see by the results below. I regenerated FP16 IR using R3, ran your inference script on both CPU and MYRIAD and as you can see below, the results are identical.

net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) - FP16

C:\Program Files (x86)\IntelSWTools\openvino_2019.3.324\inference_engine\samples\python_samples\forum_sample>python inference_.py
[[ -0. -0. -0. -0. -0. 397571.22 -0.
248193.4 -0. -0. -0. -0. -0. -0.
227947.9 -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. 72395.18 -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. ]]

net.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD) - FP16

C:\Program Files (x86)\IntelSWTools\openvino_2019.3.324\inference_engine\samples\python_samples\forum_sample>python inference_.py
[[ -0. -0. -0. -0. -0. 397571.22 -0.
248193.4 -0. -0. -0. -0. -0. -0.
227947.9 -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. 72395.18 -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. -0. -0. -0. -0. -0.
-0. -0. ]]

R3 should be arriving soon, perhaps in 3-4 weeks maybe earlier. So sorry about the inconvenience but this problem is resolved in R3 !

Shubha