Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6392 Discussions

Custom binary classification model using openvino on ncs2

P__kalaiselvan
Beginner
1,016 Views

Hello, hope you doing good.

I'm running a custom 2 layer CNN classification model on images on Openvino target using ncs2. My results are always class id ="0" irrespective of all the images. The model is giving expected results in the backend(when i load the h5 file and predict), whereas it fails while i do it on inference samples called "classification sample". Hope you can help me out.

Here is the model structure

model = Sequential()
    model.add(Conv2D(4, (3, 3), strides=(2, 2), input_shape=(img_width, img_width,1), padding='valid'))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    
    model.add(Conv2D(1, (3, 3),padding='valid'))
    #model.add(BatchNormalization())
    model.add(Activation('relu'))
    #model.add(MaxPooling2D(pool_size=(2, 2)))
    #model.add(GlobalMaxPooling2D())

    model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
    
    #model.add(GlobalMaxPooling2D())
    model.add(Dense(1))
    model.add(Activation('sigmoid'))

The openvino version im using is 2019.1.144

The model optimizer command is

python3 mo.py --input_model 'weights/2layer_tyqi.pb' --output_dir /ir_files --input_shape [1,100,100,1] --data_type FP16

Here is the h5 file containing the model, and pb file and the script i used to convert from h5 to pb in the attatchment.

running this same model on mnist dataset with last layer modeified to 10 classes works. Any idea about that?

I suspect that binary classification needs some special configuration to be done. Is it something like that. My dataset consists of two classes and sigmoid layer at the end. Please provide your opinion on this. Any help is appreciated. 

Thank you!

0 Kudos
9 Replies
RandallMan_B_Intel
1,016 Views

Hi kalaiselvan,

Thank you for reaching out.
It seems you have an older Openvino™ toolkit version, please try to run your commands again using the latest version and tell us if the results persist.  
Also, if you trained your model with images in RGB channel order, you need to use the flag --reverse_input_channels when converting the model to IR. This is because the OpenVINO samples load the images in the BGR channel order.  
 
Please give this a try, let us know if you have additional questions. 

Regards,

Randall B.

0 Kudos
P__kalaiselvan
Beginner
1,016 Views

Hi, i have tried the following approaches you mentioned and still the results persist. Im getting class id as always 0 in my custom binary classification model trained on my custom dataset. where as im able to get accurate results for the same model trained on mnist dataset.

And i would like to add a point. Reversing input channels can be applied only if number of channels in my input is 3 right?. My input is a single channel image.

Can you have a look at it and help me out if possible.

Thank you!

 

0 Kudos
RandallMan_B_Intel
1,016 Views

Hi kalaiselvan,

Are you using our classification sample, right? Can you provide the command you are using to run the sample?

Please check this when to reverse input Channels.

Regards,

Randall B.

0 Kudos
P__kalaiselvan
Beginner
1,016 Views

python3 mo.py --input_model 'models/acrylic_white_model.pb' --output_dir /models/ir_files/FP_16 -b 1 --reverse_input_channels --data_type FP16
 

Here is the command i use to generate the ir file using model optimizer. And i checked the link you provided. Seems like i did the same. Still doesnt make any difference in my result.

0 Kudos
P__kalaiselvan
Beginner
1,016 Views

Hi i am still waiting for the clarification. As i have tried revsersing input channels and still it predicts all samples as a single class. As i dont see any problem in the model or in the conversion process. Since the same model is working on mnist dataset. Any idea what might be the problem here?

Seems like some problem in handling the input during inference part. Kindly respond.

0 Kudos
RandallMan_B_Intel
1,016 Views

Hi kalaiselvan, 

Using the classification sample application we are unable to run the converted model. 

Did you modify the sample application code? If so, could you share the file, the command used to run the sample application, and also a couple of sample images you are using for us to test? 

Regards, 

Randall B. 

0 Kudos
Anonymous
Not applicable
1,005 Views

I have exactly the same issue, I created a custom network based on DenseNet (https://github.com/COVID-19-AI-Research-Project/AI-Classification/blob/master/Projects/2/Classes/Model.py) The network achieves around 92% accuracy and when testing in a real world system on a separate test dataset it gets 1 false negative. 

I converted the model to IR (After around a week of following incomplete and wrong Intel documentation that was full of bugs) and then just when I thought I was out of the woods, I hit this issue. It classifies 0 for everything.

I used the following to convert the frozen model to IR again after hitting problem after problem: 

sudo python3 mo_tf.py --input_model /Users/Adam-Milton-Barker/Desktop/frozen_graph.pb --output_dir /Users/Adam-Milton-Barker/Desktop/ --input_shape [1,64,64,3] --generate_deprecated_IR_V7 --reverse_input_channels


Finally the IR would load on a Raspberry Pi 4, my struggles were over I thought, but then:

2020-06-28 11:22:06,116 - Model - INFO - Loaded test image Model/Data/0/Non-Covid (1).png
[0.58496094 0.41503906]
0
2020-06-28 11:22:08,767 - Model - INFO - Loaded test image Model/Data/0/Non-Covid (54).png
[0.59277344 0.4074707 ]
0
2020-06-28 11:22:08,788 - Model - INFO - Loaded test image Model/Data/0/Non-Covid (104).png
[0.5541992 0.4453125]
0
2020-06-28 11:22:08,805 - Model - INFO - Loaded test image Model/Data/0/Non-Covid (389).png
[0.546875 0.45336914]
0
2020-06-28 11:22:08,826 - Model - INFO - Loaded test image Model/Data/0/Non-Covid (582).png
[0.57714844 0.42236328]
0
2020-06-28 11:22:08,846 - Model - INFO - Loaded test image Model/Data/1/Covid (10).png
[0.55078125 0.44970703]
0
2020-06-28 11:22:08,864 - Model - INFO - Loaded test image Model/Data/1/Covid (76).png
[0.546875 0.45336914]
0
2020-06-28 11:22:08,884 - Model - INFO - Loaded test image Model/Data/1/Covid (156).png
[0.58496094 0.41503906]
0
2020-06-28 11:22:08,904 - Model - INFO - Loaded test image Model/Data/1/Covid (356).png
[0.53125 0.46875]
0
2020-06-28 11:22:08,920 - Model - INFO - Loaded test image Model/Data/1/Covid (675).png
[0.57714844 0.42236328]
0

 

In comparison to running on the Raspberry Pi TF2 without OpenVINO:

2020-06-29 20:33:52,953 - Model - INFO - Sending request for: Model/Data/0/Non-Covid (1).png
2020-06-29 20:34:00,720 - Model - INFO - COVID-19 correctly not detected (True Negative) with confidence: 0.9981186985969543

2020-06-29 20:34:07,728 - Model - INFO - Sending request for: Model/Data/0/Non-Covid (54).png
2020-06-29 20:34:08,083 - Model - INFO - COVID-19 correctly not detected (True Negative) with confidence: 0.987395703792572

2020-06-29 20:34:15,090 - Model - INFO - Sending request for: Model/Data/0/Non-Covid (104).png
2020-06-29 20:34:15,463 - Model - INFO - COVID-19 correctly not detected (True Negative) with confidence: 0.9958042502403259

2020-06-29 20:34:22,470 - Model - INFO - Sending request for: Model/Data/0/Non-Covid (389).png
2020-06-29 20:34:22,830 - Model - INFO - COVID-19 correctly not detected (True Negative) with confidence: 0.9944603443145752

2020-06-29 20:34:29,837 - Model - INFO - Sending request for: Model/Data/0/Non-Covid (582).png
2020-06-29 20:34:30,205 - Model - INFO - COVID-19 correctly not detected (True Negative) with confidence: 0.9024257063865662

2020-06-29 20:34:37,212 - Model - INFO - Sending request for: Model/Data/1/Covid (10).png
2020-06-29 20:34:37,572 - Model - INFO - COVID-19 correctly detected (True Positive) with confidence: 0.9873468279838562

2020-06-29 20:34:44,579 - Model - INFO - Sending request for: Model/Data/1/Covid (76).png
2020-06-29 20:34:44,948 - Model - INFO - COVID-19 correctly detected (True Positive) with confidence: 0.9913552403450012

2020-06-29 20:34:51,955 - Model - INFO - Sending request for: Model/Data/1/Covid (156).png
2020-06-29 20:34:52,303 - Model - INFO - COVID-19 correctly detected (True Positive) with confidence: 0.9963597655296326

2020-06-29 20:34:59,311 - Model - INFO - Sending request for: Model/Data/1/Covid (356).png
2020-06-29 20:34:59,699 - Model - INFO - COVID-19 correctly detected (True Positive) with confidence: 0.9782198667526245

2020-06-29 20:35:06,707 - Model - INFO - Sending request for: Model/Data/1/Covid (675).png
2020-06-29 20:35:07,052 - Model - INFO - COVID-19 incorrectly not detected (False Negative) with confidence: 0.8346897959709167

2020-06-29 20:35:14,059 - Model - INFO - Images Classifier: 10
2020-06-29 20:35:14,060 - Model - INFO - True Positives: 4
2020-06-29 20:35:14,061 - Model - INFO - False Positives: 0
2020-06-29 20:35:14,061 - Model - INFO - True Negatives: 5
2020-06-29 20:35:14,062 - Model - INFO - False Negatives: 1

 Incidentally, if you install OpenVINO on any device that has Tensorflow installed it completely breaks the Tensorflow installation, before I was aware of this it caused me to reformat an entire laptop and reinstall everything, losing a lot of data and projects due to the crash. This issue means that I cannot use OpenVINO on any of my devices that require Tensorflow. To get this to work I had to:

- Train the model on my Linux / GPU device

- Install OpenVINO on my Mac which did not have Tensorflow installed so no danger of OpenVINO wrecking anything. 

- Move the converted IR to the Raspberry 

Only to be hit with this issue, you can understand my frustration. Please can someone offer a suggestion that will put me out of my misery, when I started converting this project to OpenVINO I had no idea it would take 2 weeks almost to do this supposedly simple task. 

Thanks in advance

0 Kudos
Max_L_Intel
Moderator
983 Views

Hi @Anonymous 

Old threads are no longer monitored by OpenVINO staff, so please create a new one.
Thanks.

0 Kudos
Anonymous
Not applicable
977 Views

That makes no sense, you have just replied, and the post would be a duplicate.  Just more hassle. 

0 Kudos
Reply