Considerable reduction in accuracy when using AI Analytics Toolkit

[AdamMiltonBarker]

Hi all. I created the following project using TF MKL earlier in the year, https://github.com/aiial/hias-all-oneapi-classifier/blob/main/documentation/getting-started.md as you can see from the results the outcome after training was a model that gets 2 incorrect classifications on a set of 20 unseen test images of Acute Lymphoblastic Leukemia positive and negative peripheral blood samples. 

2021-05-03 22:56:11,409 - Agent - INFO - Images Classified: 20
2021-05-03 22:56:11,409 - Agent - INFO - True Positives: 9
2021-05-03 22:56:11,409 - Agent - INFO - False Positives: 1
2021-05-03 22:56:11,409 - Agent - INFO - True Negatives: 9
2021-05-03 22:56:11,409 - Agent - INFO - False Negatives: 1
2021-05-03 22:56:11,409 - Agent - INFO - Total Time Taken: 1.8600921630859375

Today I have updated the project using Tensorflow from the AI Analytics Toolkit again ( I received the same results on a previous attempt to upgrade earlier in the year) and the performance has been damaged considerably.

2021-09-18 21:34:25,900 - Agent - INFO - Images Classified: 20
2021-09-18 21:34:25,900 - Agent - INFO - True Positives: 5
2021-09-18 21:34:25,900 - Agent - INFO - False Positives: 0
2021-09-18 21:34:25,900 - Agent - INFO - True Negatives: 10
2021-09-18 21:34:25,900 - Agent - INFO - False Negatives: 5

This issue now affects a number of our projects that use this classifier.

[AdamMiltonBarker]

Me again After retraining with the steps from our original tutorial using Tensorflow MKL (which have been reproduced on numerous devices over the last year with the same results) it now has completely different results. This type of change is not expected, our results need to be reproducible, is this type of instability expected to continue ?

2021-09-19 00:03:35,883 - Agent - INFO - Images Classified: 20
2021-09-19 00:03:35,883 - Agent - INFO - True Positives: 8
2021-09-19 00:03:35,883 - Agent - INFO - False Positives: 0
2021-09-19 00:03:35,883 - Agent - INFO - True Negatives: 10
2021-09-19 00:03:35,883 - Agent - INFO - False Negatives: 2

Accuracy, precision and recall also took a negative hit.

[RahulU_Intel]

Hi,

Thanks for posting in Intel forums. We are working on the use case you mentioned. We will get back to you with an update. We have reached out to you through private message for some details.

Thanks and Regards

Rahul

[AdamMiltonBarker]

Hi Rahul, thanks for the reply I have responded to your message. An update on this I have published the updated code, all is not lost converting the model to IR with openvino gave the same results after I retrained it using Tensorflow MKL. 

So using:

conda create -n hias-all-oneapi-classifier python=3 
conda activate hias-all-oneapi-classifier 
conda install tensorflow -c anaconda

Gave the same results as above mentioned: (Documentation with accuracy plots etc and testing results: https://hias-acute-lymphoblastic-leukemia-oneapi-classifier.readthedocs.io/en/latest/usage/ubuntu/ )

2021-09-19 00:03:35,883 - Agent - INFO - Images Classified: 20
2021-09-19 00:03:35,883 - Agent - INFO - True Positives: 8
2021-09-19 00:03:35,883 - Agent - INFO - False Positives: 0
2021-09-19 00:03:35,883 - Agent - INFO - True Negatives: 10
2021-09-19 00:03:35,883 - Agent - INFO - False Negatives: 2

however after converting to IR the final results were as before:

2021-09-19 00:32:03,210 - Agent - INFO - Images Classifier: 20 
2021-09-19 00:32:03,210 - Agent - INFO - True Positives: 9 
2021-09-19 00:32:03,211 - Agent - INFO - False Positives: 1 
2021-09-19 00:32:03,211 - Agent - INFO - True Negatives: 9 
2021-09-19 00:32:03,211 - Agent - INFO - False Negatives: 1

The original version however had the same results as directly above in all cases (Before converting to openvino) 

I haven't tested training with AI analytics and converting to IR yet as I don't have the time right now but the worse case scenario no longer exists as we can still get the same performance after converting to IR.  ( Documentation and testing results: https://hias-acute-lymphoblastic-leukemia-oneapi-classifier.readthedocs.io/en/latest/usage/raspberry-pi/ ) 

[AdamMiltonBarker]

Something else that may be useful, I found that without the following:

conda install -c conda-forge defusedxml 
pip3 install test-generator==0.1.1

Then converting to IR will fail when installing OpenVINO with conda, this isn't mentioned in the guide that I saw as required dependencies. 

[RahulU_Intel]

Hi,

Thanks for your reply. We are working on this internally. We are trying to do the inference with an open source dataset. We will keep you posted about our findings.

Thanks and Regards

Rahul

[AdamMiltonBarker]

Hi thank you no worries, sorry I couldn't provide the dataset directly to you, it is required to apply for this dataset. 

[RahulU_Intel]

Hi,

Thank you for your patience. We were able to reproduce your results at our side. We are checking on this internally, we will get back to you. Could you please share which version of tensorflow you are using.

Thanks and Regards

Rahul

[AdamMiltonBarker]

Hi Rahul thank you for the update. For TF MKL I installed Tensorflow using: 

conda create -n hias-all-oneapi-classifier python=3 
conda activate hias-all-oneapi-classifier 
conda install tensorflow -c anaconda

In that environment the version installed is '2.0.0-beta1'.

Regarding the AI Analytics installation which has the problem it is intel-aikit-tensorflow 2021.1.1 and the version of Tensorflow listed is tensorflow 2.2.0 

[Shailen_Sobhee]

Hello,

We would gladly like to assist you in understanding the performance issues encountered. However, I may be missing data here. 

Do I understand this right, you are getting poorer performance with Intel Tensorflor version 2.5.x from the AI Kit, compared to stock TensorFlow (same version)? 

Can you please tell me more about your workload or potentially share with us a sample of the workload including the dataset? Also, can you please describe the hardware you are using?

Thank you and kind regards,

Shailen Sobhee

AI Software Solutions Engineer

[AdamMiltonBarker]

Hi how are you? Printing tensorflow version from Python gives 2.0.0-beta1, from the conda list gives 1.3.1 from Intel Channel. This works ok  although the results differ from when the model was originally trained.

From AI DevKit with the latest install it is 2.2.0 from the Intel channel, however I remember at some point during installation TF was downgraded by Conda. 

The AI Devkit Tensorfow provides poor results compared to the original model, and the newer model created. 

Unfortunately as mentioned I cannot provide you with the dataset as it is against the terms of the license, you would need to apply for it here: https://homes.di.unimi.it/scotti/all/#download 

The device I am running on has an Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, however I was able to replicate the exact same scenario on an Intel NUC i7. 

Hope this helps. 

[Shailen_Sobhee]

Hi Adam,

I apologise for the delay in getting back to you on this issue. I will attempt to resolve this as fast as I can.

First of all, I would highly recommend that you use the latest software package. Right now, the latest version of Intel optimization for TensorFlow is 2.5.0 and this is available via conda (from the Intel channel, i.e with -c intel) or via the latest Intel AI Analytics Toolkit.

This is how you can easily install the latest Intel AI Analytics Toolkit:

conda create --name aikit python=3.7 intel-aikit -c intel 

You should get TensorFlow 2.5.0 with the above command. Please kindly let me know if that is not the case.

Now, regarding the dataset, I can apply for access from the link as well. Is there a way you can share with me (privately is possible) your model so that I can replicate the results on my end.

Thank you and kind regards,

Shailen.

[Shailen_Sobhee]

Hi Adam,

Since we haven't heard from you for a while, I a proceeding with closing this ticket. If by any chance, you encouter a similar issue, please feel free to open a new ticket.

Thank you and kind regards,

Shailen