Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

NCS2 produce different result as CPU

Hu__Yiqian
Beginner
766 Views

Dear all,

I've trained a PyTorch model to generate embeddings from images. However, when I use the trained model for inference, CPU and NCS2 produce totally different embeddings and I believe CPU generates the correct result. My code is attached.

import cv2 as cv
import numpy as np

# Read images
image_color = cv.imread('con_localized/0.jpg')

# Load the model.
net = cv.dnn.readNet('models/model_color.xml',
                     'models/model_color.bin')

# Specify target device.
net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU)
# net.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD)

# Prepare input blob and perform an inference.
blob = cv.dnn.blobFromImage(image_color, size=(227, 227), ddepth=cv.CV_8U)
net.setInput(blob)
out = net.forward()
print(out)

The CPU setting is run on macOS and MYRIAD setting is run on Raspberry. Everything else is totally same. Could anyone help me? Many thanks

0 Kudos
7 Replies
Shubha_R_Intel
Employee
766 Views

Dear Hu, Yiqian,

Can you retry this code using traditional Inference Engine Core API, i.e.

        slog::info << "Loading model to the device" << slog::endl;
        ExecutableNetwork executable_network = ie.LoadNetwork(network, FLAGS_d);
        // -----------------------------------------------------------------------------------------------------

        // --------------------------- 5. Create infer request -------------------------------------------------
        slog::info << "Create infer request" << slog::endl;
        InferRequest inferRequest = executable_network.CreateInferRequest();
        // ---------------------------------------------------------------------------

Because otherwise, your question is an OpenCV question, and it really should be posted here:

https://github.com/opencv/opencv/issues/

It could also be a Model Optimizer issue though. What version of OpenVino are you using ?  Are you on the latest and greatest 2019R2.01 ? If so, maybe upgrade and see if the problem persists.

If it wouldn't be too much trouble to convert your code to IE Core instead of OpenCV, it would be good to get this data-point so that I can help you.

Thanks,

Shubha

0 Kudos
Hu__Yiqian
Beginner
766 Views

Shubha R. (Intel) wrote:

Dear Hu, Yiqian,

Can you retry this code using traditional Inference Engine Core API, i.e.

        slog::info << "Loading model to the device" << slog::endl;
        ExecutableNetwork executable_network = ie.LoadNetwork(network, FLAGS_d);
        // -----------------------------------------------------------------------------------------------------

        // --------------------------- 5. Create infer request -------------------------------------------------
        slog::info << "Create infer request" << slog::endl;
        InferRequest inferRequest = executable_network.CreateInferRequest();
        // ---------------------------------------------------------------------------

Because otherwise, your question is an OpenCV question, and it really should be posted here:

https://github.com/opencv/opencv/issues/

It could also be a Model Optimizer issue though. What version of OpenVino are you using ?  Are you on the latest and greatest 2019R2.01 ? If so, maybe upgrade and see if the problem persists.

If it wouldn't be too much trouble to convert your code to IE Core instead of OpenCV, it would be good to get this data-point so that I can help you.

Thanks,

Shubha

 

Hi Shubha,

Thanks for the reply. My openvino is the latest 2019.R2 version.

I tried the IECore code but the result is the same. However I tried other models and found that the difference is quite small and acceptable. Is this a problem of my model? If so, why and how shall I fix it?

BTW, what do you mean by data-point? Like my .xml and .bin along with my input?

Many thanks again

0 Kudos
Shubha_R_Intel
Employee
766 Views

Dear Hu, Yiqian,

Of course the result should be the same between the OpenCV/Inference Engine approach versus the Inference Engine Core API approach but as I mentioned previously, this is the OpenVino forum (not the OpenCV forum). 

I am not sure why your results would be "different" from the CPU. That sounds odd. If you can attach your original model as well as your Model Optimizer command, I can try myself.

It may be a bug. 

Thanks,

Shubha

 

0 Kudos
Hu__Yiqian
Beginner
766 Views

Shubha R. (Intel) wrote:

Dear Hu, Yiqian,

Of course the result should be the same between the OpenCV/Inference Engine approach versus the Inference Engine Core API approach but as I mentioned previously, this is the OpenVino forum (not the OpenCV forum). 

I am not sure why your results would be "different" from the CPU. That sounds odd. If you can attach your original model as well as your Model Optimizer command, I can try myself.

It may be a bug. 

Thanks,

Shubha

 

Dear Shubha

Thanks for the reply.

Here's my model structure.

class AlexNet(nn.Module):

    def __init__(self, num_channels):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(num_channels, 96, kernel_size=11, stride=4),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(96, 256, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(256, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Linear(6 * 6 * 256, 2048),
            nn.ReLU(inplace=True),
            nn.Linear(2048, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, 128),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

I convert the pytorch model to onnx first.

model_color = Alexnet(3)

image_con_color = images['image_color' ] // load image data
image_con_color = Variable(image_con_color.float()).to(device)
color_output = model_color(image_con_color)

torch.onnx.export(model_color, image_con_color, 'model_color.onnx', input_names=['image_con_color'], output_names=['color_output'])

After that I convert the .onnx into .xml and .bin

cd /opt/intel/openvino/deployment_tools/model_optimizer/
python mo.py --input_model ~/input_dir/model_color.onnx --data_type FP16 --output_dir ~/output_dir

In the end I use the code in the very first post to run on CPU by macOS and NCS2 by raspberry pi. and get the different results.

Result on CPU: 

[    0.          0.          0.          0.          0.          0.
     0.          0.          0.          0.          0.          0.
     0.          0.          0.          0.          0.          0.
     0.          0.          0.          0.          0.       1850.8931
  8222.642       0.          0.          0.       2135.309    3193.6743
  8222.642    4932.039    1411.8556      0.       2135.309    2135.309
     0.          0.          0.      10461.765   10461.765       0.
     0.          0.          0.      10461.765   14551.347   14551.347
     0.          0.          0.          0.      14551.347   14908.998
     0.          0.          0.          0.       4479.435   17829.059
     0.          0.        545.42224     0.       9005.181   17829.059
     0.          0.        545.42224     0.       7507.1333   7507.1333
     0.          0.          0.          0.          0.          0.
     0.       1268.1013    951.8981      0.          0.          0.
     0.       4062.216    3555.8015      0.        366.30554   366.30554
     0.       6334.16     6334.16        0.          0.          0.
     0.       6334.16     6334.16        0.     ]

Result on NCS2:

[[  0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
  232.25      0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.      261.        0.       56.84375   0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.        0.        0.        0.        0.        0.
    0.        0.     ]]

PS Before training the results don't differ very much. At least zeros and non-zeros aren't messed up. After several epoches' training, it becomes the above result.

Many thanks again!

0 Kudos
Shubha_R_Intel
Employee
766 Views

Dear Hu, Yiqian,

Thanks for all the info. I will attempt to reproduce and file a bug if I can reproduce the discrepancy same as you. I will let you know the results on this forum.

Shubha

0 Kudos
Shubha_R_Intel
Employee
766 Views

Dear Hu, Yiqian,

The code you provided me above is not a complete script. 

I reproduced it here:

import torch
import torch.nn as nn
from torchvision.models import AlexNet


class AlexNet(nn.Module):

    def __init__(self, num_channels):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(num_channels, 96, kernel_size=11, stride=4),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(96, 256, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(256, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Linear(6 * 6 * 256, 2048),
            nn.ReLU(inplace=True),
            nn.Linear(2048, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, 128),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

    if __name__ == "__main__":
        model_color = AlexNet(3)

        image_con_color = images['image_color'] // load
        image_con_color = Variable(image_con_color.float()).to(device)
        color_output = model_color(image_con_color)

        torch.onnx.export(model_color, image_con_color, 'model_color.onnx', input_names=['image_con_color'],
                          output_names=['color_output'])

 

But I get an error:

image_con_color = images['image_color'] // load
NameError: name 'images' is not defined
 

In order to go further with this issue, I need the original ONNX model, not just the IR. I actually sent you a Private Message about this.

Thanks,

Shubha

0 Kudos
Shubha_R_Intel
Employee
766 Views

Dear Hu, Yiqian,

The problem is fixed in OpenVino 2019R3 as you can see by the results below. I regenerated FP16 IR using R3, ran your inference script on both CPU and MYRIAD and as you can see below, the results are identical.

net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) - FP16

C:\Program Files (x86)\IntelSWTools\openvino_2019.3.324\inference_engine\samples\python_samples\forum_sample>python inference_.py
[[    -0.       -0.       -0.       -0.       -0.   397571.22     -0.
  248193.4      -0.       -0.       -0.       -0.       -0.       -0.
  227947.9      -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.    72395.18     -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.  ]]

 

net.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD) - FP16

C:\Program Files (x86)\IntelSWTools\openvino_2019.3.324\inference_engine\samples\python_samples\forum_sample>python inference_.py
[[    -0.       -0.       -0.       -0.       -0.   397571.22     -0.
  248193.4      -0.       -0.       -0.       -0.       -0.       -0.
  227947.9      -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.    72395.18     -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.       -0.       -0.       -0.       -0.       -0.
      -0.       -0.  ]]
 

R3 should be arriving soon, perhaps in 3-4 weeks maybe earlier. So sorry about the inconvenience but this problem is resolved in R3 !

Shubha

0 Kudos
Reply