Not able to run inference with the NCS2 on Raspberry Pi 4b

[publio]

Hi, I've been using a tensorflow model for a year that was converted to IR. However, I had to retrain the model from scratch, but since I had migrated to Windows 11, I had to re-install OpenVino toolkit therefore I am currently using the latest available version (2023.1) to perform the model conversion to IR.

Additionally, I am also trying to deploy the model to a Raspberry Pi 4B that is accelerated by a VPU which still has the older OpenVino 2022.1 installed for inference.  

I must admit that I am confused on how to properly compress the model to float16 and convert it to IR as It was done a year ago.

This is code used to run inference on RPi 4B with the NCS2 device:

import numpy as np
from openvino.inference_engine import IECore
from openvino.runtime import Core
import datetime
import cv2
import os

def min_max_norm(img):
    img = np.asarray(img, dtype="int16")
    x_min = img.min(axis=(1, 2), keepdims=True)
    x_max = img.max(axis=(1, 2), keepdims=True)
    img = (img - x_min) / (x_max - x_min)
    return img

def load_images_from_folder(folder):
    images = []
    for filename in os.listdir(folder):
        img = cv2.imread(os.path.join(folder,filename))
        img = min_max_norm(img)
        if img is not None:
            images.append(img)
    return images

y_labels_files_1 = [0]*50
y_labels_files_2 = [1]*50
y_labels_files_3 = [0]*50
y_labels_files_4 = [1]*50
y_labels_files_5 = [0]*50
y_labels_files_6 = [1]*50

y_test = np.array(y_labels_files_1 + y_labels_files_2 + y_labels_files_3 + y_labels_files_4 + y_labels_files_5 + y_labels_files_6)

x_test = np.concatenate([
    load_images_from_folder('/home/pi/Desktop/NCS2/test_data/files_1'),
    load_images_from_folder('/home/pi/Desktop/NCS2/test_data/files_2'),
    load_images_from_folder('/home/pi/Desktop/NCS2/test_data/files_3'),
    load_images_from_folder('/home/pi/Desktop/NCS2/test_data/files_4'),
    load_images_from_folder('/home/pi/Desktop/NCS2/test_data/files_5'),
    load_images_from_folder('/home/pi/Desktop/NCS2/test_data/files_6')
])

ie = IECore()
ov = Core()

net = ie.read_network(model="saved_model.xml", weights="saved_model.bin")
compiled_net = ov.compile_model(net, "MYRIAD")

inference_times = []
input_key = next(iter(compiled_net.input_info))
output_key = next(iter(compiled_net.outputs.keys()))
print("Successfully detected VPU")
idx = 0
for k in x_test:
    input_image = np.expand_dims(k.transpose(2, 0, 1), 0)
    start = datetime.datetime.now()
    _ = compiled_net.infer(inputs={input_key: input_image})[output_key]
    end = datetime.datetime.now()
    delta = end - start
    t = round(float(delta.total_seconds() * 1000), 4)
    idx += 1
    print(f"Inferencing time: {t}ms for frame {idx}")
    inference_times.append(t)

inference_times = np.array(inference_times)
inference_times_average = np.mean(inference_times)
inference_times_std = np.std(inference_times)

print(f"Inference Time mean: {inference_times_average:.6f}")
print(f"Inference Time std: {inference_times_std:.6f}")

The following code will generate this error:

Traceback (most recent call last):
File "/home/pi/Desktop/NCS2/image_classifier.py", line 58, in <module>
compiled_net = ov.compile_model(net, "MYRIAD")
File "/opt/intel/openvino_2022/python/python3.9/openvino/runtime/ie_api.py", line 387, in compile_model
super().compile_model(model, device_name, {} if config is None else config),
TypeError: compile_model(): incompatible function arguments. The following argument types are supported:
1. (self: openvino.pyopenvino.Core, model: openvino.pyopenvino.Model, device_name: str, properties: Dict[str, object]) -> ov::CompiledModel
2. (self: openvino.pyopenvino.Core, model: openvino.pyopenvino.Model, properties: Dict[str, object]) -> ov::CompiledModel
3. (self: openvino.pyopenvino.Core, model_path: str, device_name: str, properties: Dict[str, object]) -> ov::CompiledModel
4. (self: openvino.pyopenvino.Core, model_path: str, properties: Dict[str, object]) -> ov::CompiledModel

Invoked with: <openvino.runtime.ie_api.Core object at 0x7f900e48b0>, <openvino.inference_engine.ie_api.IENetwork object at 0x7f900ea270>, 'MYRIAD', {}

My model was saved as a SavedModel format and was converted using the following command: mo --saved_model_dir my_model_model-openvino --input_shape [1,288,384,3]

[Wan_Intel]

Hello Publio,

Thanks for reaching out to us.

I noticed that the code read Model with OpenVINO™ Inference Engine API and loaded Model with OpenVINO™ API 2.0:

ie = IECore()

ov = Core()

net = ie.read_network(model="saved_model.xml", weights="saved_model.bin")

compiled_net = ov.compile_model(net, "MYRIAD")

For your information, starting from OpenVINO™ 2022.1, OpenVINO™ API 2.0 has been introduced. You may refer to the link below to understand how to migrate OpenVINO™ Inference Engine-based applications to OpenVINO™ API 2.0:

https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html#more-information

Regards,

Wan

[publio · Accepted solution]

Hi Wan, thank you very much for your reply. I managed to run inference following this example: Hello Image Classification — OpenVINO™ documentationCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboard — Version(2023.0)

[Wan_Intel]

Hi Publio,

Glad to know that your issue has been resolved. If you need additional information from Intel, please submit a new question as this thread will no longer be monitored.

Regards,

Wan