Model accuracy drops drastically after converting TF2 model to float16 for the Intel NCS2

[publio]

Hello, I am currently trying to deploy a model to Intel NCS2 for Raspberry Pi but I am struggling with accuracy drop.  I used to be able to do this sometimes with accuracy gain or minimal accuracy drop. However, I had to do a fresh install of the Raspberry Pi and I also had to change my Tensorflow Version on Desktop. So, I installed OpenVino 2024.6 on Windows 11 to convert the model to IR and I am running the last Intel NCS2 compatible version that is the OpenVino 2022.1 for the Raspberry Pi which is in this guide: Building Open Source OpenVINO™ Toolkit for Debian* OS and.

I am currently using tensorflow 2.14 on my Desktop to train the model and convert it and deploy to a Bullseye OS running OpenVino 2022 and Intel NCS2. The command I used to convert my model was:

mo --input_model /mnt/e/MyModel --input_shape [1,384,288,3] --output_dir /mnt/e/MyModel /IR --compress_to_fp16=True

and 

ovc "/mnt/e/MyModel" --output_model "/mnt/e/MyModel/IR/model.xml" --input "[1,288,384,3]" --compress_to_fp16 True

In both cases the accuracy dropped below 50%. I am currently comparing with the Edge TPU accelerator which requires int8 quantization, the accuracy change is negligible. In order to ensure the images were not corrupted and the preprocessing wasn't done differently, I tried simplifying the code to do a simple transfer learning and train the model and run it on my CPU instead of the Raspberry Pi, the same problem with accuracy drop happened. 

If it helps, the dataset I am using to train, for reproducibility purposes:

Dataset:

https://drive.google.com/file/d/1iNDixUR_E54YXzQNFzX19Hjbc2srkBJ-/view?usp=sharing

Tensorflow in Saved Model format:

https://drive.google.com/file/d/1fHrwxFaeifnK4GZTM3K6srv0ej_qAwyi/view?usp=drive_link

And the code used for training the model was this one: 

import os
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications import VGG16
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.optimizers import Adam
from sklearn.metrics import classification_report
import numpy as np
from collections import Counter


train_path = '/mnt/e/Dataset Split/train'
val_path = '/mnt/e/Dataset Split/val'
test_path = '/mnt/e/Dataset Split/test'

image_size = (384, 288)
batch_size = 32

train_datagen = ImageDataGenerator(rescale=1.0/255.0)
val_datagen = ImageDataGenerator(rescale=1.0/255.0)
test_datagen = ImageDataGenerator(rescale=1.0/255.0)

train_generator = train_datagen.flow_from_directory(
train_path,
target_size=image_size,
batch_size=batch_size,
class_mode='categorical'
)

val_generator = val_datagen.flow_from_directory(
val_path,
target_size=image_size,
batch_size=batch_size,
class_mode='categorical'
)

test_generator = test_datagen.flow_from_directory(
test_path,
target_size=image_size,
batch_size=batch_size,
class_mode='categorical',
shuffle=False
)


class_counts = Counter(train_generator.classes)
total_samples = sum(class_counts.values())
class_weights = {class_index: total_samples / (len(class_counts) * count) for class_index, count in class_counts.items()}
print(f"Class weights: {class_weights}")


base_model = VGG16(weights='imagenet', include_top=False, input_shape=(384, 288, 3))


base_model.trainable = False

x = Flatten()(base_model.output)
output = Dense(2, activation='softmax')(x)

model = Model(inputs=base_model.input, outputs=output)


model.compile(optimizer=Adam(learning_rate=1e-5), loss='categorical_crossentropy', metrics=['accuracy'])


history = model.fit(
train_generator,
validation_data=val_generator,
epochs=15,
verbose=1,
class_weight=class_weights
)


test_loss, test_accuracy = model.evaluate(test_generator, verbose=1)
print(f"Test Accuracy: {test_accuracy * 100:.2f}%")


test_generator.reset()
predictions = model.predict(test_generator)
predicted_classes = np.argmax(predictions, axis=1)
true_classes = test_generator.classes
class_labels = list(test_generator.class_indices.keys())


report = classification_report(true_classes, predicted_classes, target_names=class_labels)
print(report)

It seems to me that this is a version problem, I am unsure if that is the case or is there something wrong with the command I am using to convert my model to IR. Additionally, this is the code I used to run inference on my images on the test set after conversion:

import os
import time
import numpy as np
from openvino.runtime import Core
from glob import glob
from PIL import Image

# Paths
model_path_xml = '/mnt/e//IR/model.xml'
model_path_bin = '/mnt/e/Dataset Split/IR/model.bin'
test_path = '/mnt/e/Dataset Split/test'

# Constants
image_size = (384, 288) # Same as training
class_labels = ['Class 0', 'Class 1']

# Load the OpenVINO model
ie = Core()
compiled_model = ie.compile_model(model=model_path_xml, device_name="CPU")
input_layer = compiled_model.input(0)
output_layer = compiled_model.output(0)

# Function to preprocess images
def preprocess_image(image_path):
img = Image.open(image_path).convert('RGB')
img = img.resize(image_size)
img_array = np.array(img, dtype=np.float32) / 255.0 
img_array = np.expand_dims(img_array, axis=0) 
img_array = np.ascontiguousarray(img_array) 
return img_array

# Evaluation variables
correct = 0
total = 0
start_time = time.time()

# Gather all test images
image_paths = []
labels = []

for label, class_name in enumerate(class_labels):
class_path = os.path.join(test_path, class_name)
for image_path in glob(os.path.join(class_path, '*.*')):
if image_path.lower().endswith(('.jpg', '.jpeg', '.png', '.tiff')):
image_paths.append(image_path)
labels.append(label)

# Total images
num_images = len(image_paths)
if num_images == 0:
raise ValueError("No images found in the test dataset.")

# Inference loop
for idx, (image_path, true_label) in enumerate(zip(image_paths, labels)):
# Preprocess the image
img = preprocess_image(image_path)

# Perform inference
result = compiled_model([img])[output_layer]
predicted_label = np.argmax(result, axis=1)[0]
confidence = result[0][predicted_label]

# Accuracy calculation
is_correct = (predicted_label == true_label)
correct += int(is_correct)
total += 1

# Time calculations
elapsed_time = time.time() - start_time
remaining_time = (elapsed_time / total) * (num_images - total)

# Print per-image result
print(f"Image: {os.path.abspath(image_path)}")
print(f"Predicted: {class_labels[predicted_label]}, True: {class_labels[true_label]}")
print(f"Confidence: {confidence:.2f}")
print(f"Correct: {is_correct}, Total Accuracy So Far: {correct / total:.2%}")
print(f"Classified {total}/{num_images}, Time Passed: {elapsed_time:.2f}s, Time Remaining: {remaining_time:.2f}s")
print("-" * 50)

# Final accuracy
total_accuracy = correct / total
print(f"Total Images: {total}")
print(f"Correctly Classified: {correct}")
print(f"Final Accuracy: {total_accuracy:.2%}")
print(f"Total Time: {time.time() - start_time:.2f}s")

I used to have a script to convert the models, which never gave me any problems until I had to use ovc or mo to convert the models.  I would activate the variables like this:

import os
os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'
os.system('C:\\Program Files (x86)\\Intel\\openvino_2022\\bin\\setupvars.bat') 

And then, I would easily convert the model passing this script:

import subprocess

# Define paths
saved_model_path = '/mnt/e/models/final_model.h5'
output_dir = '/mnt/e/models/NCS2/OV_IR_VGG16_leaf_disease'
input_shape = "[1,288,384,3]"

# Form the Model Optimizer command
mo_command = f"mo --saved_model_dir {saved_model_path} --input_shape {input_shape} --output_dir {output_dir}"

# Run the command
result = subprocess.run(mo_command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

# Print the output (optional)
print(result.stdout)
print(result.stderr)

Since I had to re-install, I no longer have any Intel folders in my Program Files (x86). So, I would like some guidance on what is going on and what can I do to stop the accuracy from drastically going down, if possible, please and thanks in advance. Feel free to ask me anything, if my problem isn't clear or you need details. 

[publio]

Hi, i am sorry, I just verified this is an issue on my side, please close and/or delete this.

[Zulkifli_Intel]

Thank you for your question. If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored.