Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6392 Discussions

Please tell me how to infer the Segmentation model that inputs RGB and Depth.

Yumi_ques
Novice
838 Views

System Information
Ubuntu 18.04.5 LTS
OpenVINO:2022.1.0.643
PyTorch:1.8.1 + cu102

[issue]
We are trying to infer indoor segmentation using RGB and Depth images.
The model for indoor segmentation (*ESANet) was converted to IR and segmentation was performed using OpenVINO's inference API.
(Adapted from Hello Image Segmentaion (https://docs.openvino.ai/latest/notebooks/003-hello-segmentation-with-output.html))
I expected that the inference results of ESANet's PyTorch model and the converted IR model would be similar, but there is a difference.
The conversion to the IR model is well done, so it is hard to imagine that the conversion would result in poor accuracy.
We believe that this is due to an error in either the use of OpenVINO's API or the plotting of results in the homebrew code.
Could you please tell me what I need to modify in my code to get the same results as ESANet?

Thank you in advance.

[Expected Results]
The results of inference using the "ESANetR34-NBt1D" trained model on the ESANet NYUv2 dataset are as follows

・result_pytorch.jpg

The PyTorch inference execution command, available in the ESANet repository, is shown below.
python inference_samples.py\
--dataset nyuv2\
--ckpt_path ./trained_models/nyuv2/r34_NBt1D.pth \
--depth_scale 0.1 \
--raw_depth


[Actual Results]
The results of the inference after converting to OpenVINO's IR format are as follows.
・result_IRmodel.jpg

The inference execution command is as follows

python inference_samples.py\
--dataset nyuv2\
--ckpt_path ./ESANET_to_IR/model.xml\
--depth_scale 0.1\
--raw_depth

The detailed source code is below.

---------------------------------------------------


import argparse
from glob import glob
import os

import cv2
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn.functional as F

from src.args import ArgumentParserRGBDSegmentation
from src.build_model import build_model
from src.prepare_data import prepare_data
import sys
from openvino.runtime import Core


if __name__ == '__main__':

# arguments
parser = ArgumentParserRGBDSegmentation(
description='Efficient RGBD Indoor Sematic Segmentation (Inference)',
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.set_common_args()
parser.add_argument('--ckpt_path', type=str,
help='Path to the checkpoint of the trained model.')
parser.add_argument('--depth_scale', type=float,
default=1.0,
help='Additional depth scaling factor to apply.')
args = parser.parse_args()

args.pretrained_on_imagenet = False # we are loading other weights anyway
dataset, preprocessor = prepare_data(args, with_input_orig=True)
n_classes = dataset.n_classes_without_void

#load the model
ie = Core()
model = ie.read_model(model=args.ckpt_path)
compiled_model = ie.compile_model(model=model,device_name="CPU")

input_layer_ir_f = compiled_model.inputs[0]
input_layer_ir_s = compiled_model.inputs[1]
output_layer_ir = next(iter(compiled_model.outputs))

#load image
image_r = cv2.imread("/path/to/image/sample_rgb.png",cv2.IMREAD_UNCHANGED)
image_d = cv2.imread("/path/to/image/samples/sample_depth.png",cv2.IMREAD_UNCHANGED)
_rgb_image = cv2.cvtColor(image_r, cv2.COLOR_BGR2RGB)

# get samples
n,c,h_r,w_r = input_layer_ir_f.shape
n,c,h_d,w_d = input_layer_ir_s.shape
resize_rgb_image = cv2.resize(_rgb_image, (w_r,h_r,))
resize_depth_image = cv2.resize(image_d, (w_d,h_d,))
rgb_input_image = np.expand_dims(resize_rgb_image.transpose(2,0,1),0)

depth_input_image = np.expand_dims(resize_depth_image,0)
depth_input_image = np.expand_dims(depth_input_image,0)* args.depth_scale

# Run the inference
input_image_data = [rgb_input_image,depth_input_image]
result = compiled_model(input_image_data)[output_layer_ir]

segmentation_mask = np.argmax (result,axis=1)
t_seg_mask = torch.from_numpy(segmentation_mask.astype(np.float32)).clone()
segmentation_mask = t_seg_mask.cpu().numpy().squeeze().astype(np.uint8)
segmentation_mask_colored = dataset.color_label(segmentation_mask,with_void=False)
fig, axs = plt.subplots(1, 3, figsize=(16, 3))
[ax.set_axis_off() for ax in axs.ravel()]
axs[0].imshow(_rgb_image)
axs[1].imshow(image_d, cmap='gray')
axs[2].imshow(segmentation_mask_colored)
plt.savefig('./result.jpg', dpi=150)
#plt.show()
---------------------------
*ESANet is a repository that uses rgb and depth images to infer indoor segmentation.
The model used was ESANet's NYUv2(test) ESANetR34-NBt1D.
https://github.com/TUI-NICR/ESANet

*I asked about the conversion of the ESANet model here.
https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/Conversion-from-ONNX-model-to-IR-model-fails/m-p/1389445/emcs_t/S2h8ZW1haWx8dG9waWNfc3Vic2NyaXB0aW9ufEwzV0c3WjRSMktMTlBLfDEzODk0NDV8U1VCU0NSSVBUSU9OU3xoSw#M27408

0 Kudos
1 Solution
Yumi_ques
Novice
709 Views

Thanks for the reply.

I will change the code with reference to Image Segmentation Python Demo.
We will consider this matter resolved.
If I have any further questions, I will ask them again.

Thank you for your response.

Yumi_ques

View solution in original post

5 Replies
Wan_Intel
Moderator
804 Views

Hi Yumi_ques,

Thanks for reaching out to us.

We’re investigating this issue and we will update you as soon as possible.

 

 

Regards,

Wan


0 Kudos
Yumi_ques
Novice
781 Views

Thank you very much.
We hope you will update us as soon as you know more.

I really appreciate your great support.

0 Kudos
Wan_Intel
Moderator
728 Views

Hi Yumi_ques,

Thanks for your patience.

Based on the feedback from the relevant team, the ONNX model you have converted is correct and we verified your model with the benchmark app as well.

 

For your information, the Hello Image Segmentation sample code is designed for road-segmentation-adas-0001 based on the input data of the model.

 

To test your ESANet model, we can advise referring to the Image Segmentation Python Demo sample code. For this sample code, ESANet is not in the list of the supported models, but you can refer to the code and modify it to suit your ESANet model parameter, for example, the input shape of the model etc.

 

Hope this information helps

 

 

Regards,

Wan


0 Kudos
Yumi_ques
Novice
710 Views

Thanks for the reply.

I will change the code with reference to Image Segmentation Python Demo.
We will consider this matter resolved.
If I have any further questions, I will ask them again.

Thank you for your response.

Yumi_ques

Wan_Intel
Moderator
697 Views

Hi Yumi_ques,

Thanks for your question.

This thread will no longer be monitored since we have provided suggestions. 

If you need any additional information from Intel, please submit a new question.

 

 

Best regards,

Wan


0 Kudos
Reply