Problem convertin fastrcnn inseption v2

[Warday]

the network was trained on tensorflow 1.15.x using Tensorflow objet detection api (for tensorflow 1.x)
conversion script (jupiter notebook)
`import platform

mo_tf_path = '/opt/intel/openvino_2021/deployment_tools/model_optimizer/mo_tf.py'

pb_file = '/home/warday/Documentos/CNN/Convert/Convertido/my_model/frozen_inference_graph.pb'
checkpoint = '/home/warday/Documentos/CNN/Convert/Convertido/my_model/model.ckpt.index'
output_dir = '/home/warday/Documentos/CNN/Convert/Openvivo'
configuration_file = '/opt/intel/openvino_2021/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support_api_v1.15.json'

pipeline = '/home/warday/Documentos/CNN/Convert/Convertido/my_model/pipeline.config'

input_shape = [1,800,800,3]
input_shape_str = str(input_shape).replace(' ','')
input_shape_str

!python3 {mo_tf_path} --input_model {pb_file} --output_dir {output_dir} --transformations_config {configuration_file} --tensorflow_object_detection_api_pipeline_config {pipeline} --input_shape {input_shape_str} --reverse_input_channels --data_type FP16 `

if I use that I get the error

Failed to match nodes from custom replacement description with id 'ObjectDetectionAPIProposalReplacement':

If I use aster_rcnn_support.json' it works but I got the next error in inverence
ValueError: could not broadcast input array from shape (3,600,600) into shape (1,3)

I make a costum faster_rcnn_support_api_v1.15.json replaceing only ObjectDetectionAPIProposalReplacement from faster_rcnn_support.json and I got the same error when i try inference
ValueError: could not broadcast input array from shape (3,600,600) into shape (1,3)

When conversion is saccesfull it sed Suitable inputs {'image_tensor': [1, 3, 600, 600]} so I reshepe image to that instead of 800 * 800
I have no problem to share .pb and checkpoint files

I try using tensorflow 1.14.0 but same problem
thax

[Warday]

IMPORTANT: I try to convert and test original model (without retraining) and with faster_rcnn_support.json. Transform worked good but inference fail. I sent inference code. I have successfully make inference with custom trained inceptionv3 without problem.

ie = IECore()
net = ie.read_network(model=model_xml , weights=model_bin)
exec_net = ie.load_network(network=net, device_name="CPU")
input_blob = next(iter(exec_net.inputs))
image = Image.open(IMAGE_DIR)
n, c, h, w = [1, 3, 1024, 1024]
processed_img = image.resize((h,w), resample=Image.BILINEAR)
processed_img = np.array(processed_img).astype(np.uint8)
processed_img = processed_img.transpose((2, 0, 1))
processed_img = processed_img.reshape((n, c, h, w))
print(processed_img.shape)
result = exec_net.infer(inputs = {input_blob: processed_img})

[Iffa_Intel]

Greetings,

Could you share your files so that we could replicate on our side?

Sincerely,

Iffa

[Warday]

Hi, I send you the exported model (full retrained model) with .pb, cheackpoint, pipeline, etc.

https://drive.google.com/drive/folders/10iyjT6ywVQJbuIPixwKjpRU8NWMUEusr?usp=sharing

pretreined model downloaded from
https://docs.openvinotoolkit.org/2020.2/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html
and orifinal sourse (I tasted both in case there ware an error)
Note. I transformed the original model and try inference. It seems that from version 2019 they added a secondary input check (https://github.com/openvinotoolkit/cvat/pull/545) so

input_blob = (next(iter(exec_net.inputs)))

Doasent work eny more i try

input_blob = 'image_tensor'

but it make python crash

network.inputs = ['image_info', 'image_tensor']
network.outputs = ['detection_output']

Thenx for all

[Iffa_Intel]

Noted, we are investigating this and will get back to you asap.

Sincerely,

Iffa

[Warday · Accepted solution]

Soved with:

python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo.py \
--framework tf \
--input_model model/frozen_inference_graph.pb \
--reverse_input_channels \
--data_type FP16 \
--tensorflow_object_detection_api_pipeline_config model/pipeline.config \
--transformations_config json/faster_rcnn_support_api_v1.10.json \
--input_shape [1,600,1024,3] \
--static_shape

inference:

from openvino.inference_engine import IENetwork 
from openvino.inference_engine import IECore
from matplotlib import pyplot as plt
import sys
from PIL import Image
import numpy as np
import glob
import cv2
import tensorflow as tf
import ngraph as ng
model_xml = '/home/warday/Documentos/CNN/Convert/original/frozen_inference_graph.xml'
model_bin = '/home/warday/Documentos/CNN/Convert/original/frozen_inference_graph.bin'
IMAGE_DIR = '/home/warday/Documentos/CNN/Convert/Convertido/fast_insep_v2/images/test/hojasred_4.jpg'
# We prepare input image
n, c, h, w = [1, 3, 600, 1024]
images = np.ndarray(shape=(n, c, h, w))
images_hw = []
image = cv2.imread(IMAGE_DIR)
ih, iw = image.shape[:-1]
images_hw.append((ih, iw))
if image.shape[:-1] != (h, w):
    image = cv2.resize(image, (w, h))
image2 = image.transpose((2, 0, 1))
images[0] = image2
# We load network
ie = IECore()
net = ie.read_network(model=model_xml , weights=model_bin) 
# we prepare input blob
# As older version of Openvino worked with 1 input and newer with 2. 
# first we check and prepare inputs (1 or 2) and the we fill it with the data.
# as we use Openvino 2021 we have 2 inputs ( ['image_info', 'image_tensor'])
# you and check that using print('network.inputs = ' + str(list(exec_net.inputs)))
# we prepare both inputs in a variable called data
for input_key in net.input_info:
        if len(net.input_info[input_key].layout) == 4:
            input_name = input_key
            net.input_info[input_key].precision = 'U8'
        elif len(net.input_info[input_key].layout) == 2:
            input_info_name = input_key
            net.input_info[input_key].precision = 'FP32'
            if net.input_info[input_key].input_data.shape[1] != 3 and net.input_info[input_key].input_data.shape[1] != 6 or \
                net.input_info[input_key].input_data.shape[0] != 1:
                log.error('Invalid input info. Should be 3 or 6 values length.')
data = {}
data[input_name] = images
if input_info_name != "":
        infos = np.ndarray(shape=(n, c), dtype=float)
        for i in range(n):
            infos[i, 0] = h
            infos[i, 1] = w
            infos[i, 2] = 1.0
        data[input_info_name] = infos
# we prepare output blob
out_blob = next(iter(net.outputs)) # more than enough
# prepare net
exec_net = ie.load_network(network=net, device_name="CPU")
#execute
image_tensor = tf.convert_to_tensor(images)
result = exec_net.infer(inputs = data)
# if you wont to pritn the boxes
res = result[out_blob]
boxes, classes = {}, {}
data = res[0][0]
for number, proposal in enumerate(data):
    if proposal[2] > 0:
        imid = np.int(proposal[0])
        ih, iw = images_hw[imid]
        label = np.int(proposal[1])
        confidence = proposal[2]
        xmin = np.int(iw * proposal[3])
        ymin = np.int(ih * proposal[4])
        xmax = np.int(iw * proposal[5])
        ymax = np.int(ih * proposal[6])
        print("[{},{}] element, prob = {:.6}    ({},{})-({},{}) batch id : {}" \
                .format(number, label, confidence, xmin, ymin, xmax, ymax, imid), end="")
        if proposal[2] > 0.5:
            print(" WILL BE PRINTED!")
            if not imid in boxes.keys():
                boxes[imid] = []
            boxes[imid].append([xmin, ymin, xmax, ymax])
            if not imid in classes.keys():
                classes[imid] = []
            classes[imid].append(label)
        else:
            print()
for imid in classes:
    tmp_image = cv2.imread(IMAGE_DIR)
    for box in boxes[imid]:
        cv2.rectangle(tmp_image, (box[0], box[1]), (box[2], box[3]), (232, 35, 244), 2)
    cv2.imwrite("out.bmp", tmp_image)
    plt.imshow(tmp_image) # wrong colors but useful to check
    plt.show()

[Iffa_Intel]

Glad to know that you had solved it.

In case you want to know more regarding the conversion you can refer here.

Sincerely,

Iffa