Community
cancel
Showing results for 
Search instead for 
Did you mean: 
526 Views

Considerable reduction in accuracy when using AI Analytics Toolkit

Hi all. I created the following project using TF MKL earlier in the year, https://github.com/aiial/hias-all-oneapi-classifier/blob/main/documentation/getting-started.md as you can see from the results the outcome after training was a model that gets 2 incorrect classifications on a set of 20 unseen test images of Acute Lymphoblastic Leukemia positive and negative peripheral blood samples. 

2021-05-03 22:56:11,409 - Agent - INFO - Images Classified: 20
2021-05-03 22:56:11,409 - Agent - INFO - True Positives: 9
2021-05-03 22:56:11,409 - Agent - INFO - False Positives: 1
2021-05-03 22:56:11,409 - Agent - INFO - True Negatives: 9
2021-05-03 22:56:11,409 - Agent - INFO - False Negatives: 1
2021-05-03 22:56:11,409 - Agent - INFO - Total Time Taken: 1.8600921630859375

Today I have updated the project using Tensorflow from the AI Analytics Toolkit again ( I received the same results on a previous attempt to upgrade earlier in the year) and the performance has been damaged considerably.

2021-09-18 21:34:25,900 - Agent - INFO - Images Classified: 20
2021-09-18 21:34:25,900 - Agent - INFO - True Positives: 5
2021-09-18 21:34:25,900 - Agent - INFO - False Positives: 0
2021-09-18 21:34:25,900 - Agent - INFO - True Negatives: 10
2021-09-18 21:34:25,900 - Agent - INFO - False Negatives: 5

This issue now affects a number of our projects that use this classifier.

0 Kudos
10 Replies
505 Views

Me again After retraining with the steps from our original tutorial using Tensorflow MKL (which have been reproduced on numerous devices over the last year with the same results) it now has completely different results. This type of change is not expected, our results need to be reproducible, is this type of instability expected to continue ?

2021-09-19 00:03:35,883 - Agent - INFO - Images Classified: 20
2021-09-19 00:03:35,883 - Agent - INFO - True Positives: 8
2021-09-19 00:03:35,883 - Agent - INFO - False Positives: 0
2021-09-19 00:03:35,883 - Agent - INFO - True Negatives: 10
2021-09-19 00:03:35,883 - Agent - INFO - False Negatives: 2

Accuracy, precision and recall also took a negative hit.

RahulU_Intel
Moderator
478 Views

Hi,


Thanks for posting in Intel forums. We are working on the use case you mentioned. We will get back to you with an update. We have reached out to you through private message for some details.


Thanks and Regards

Rahul


474 Views

Hi Rahul, thanks for the reply I have responded to your message. An update on this I have published the updated code, all is not lost converting the model to IR with openvino gave the same results after I retrained it using Tensorflow MKL. 

 

So using:

conda create -n hias-all-oneapi-classifier python=3 
conda activate hias-all-oneapi-classifier
conda install tensorflow -c anaconda

Gave the same results as above mentioned: (Documentation with accuracy plots etc and testing results: https://hias-acute-lymphoblastic-leukemia-oneapi-classifier.readthedocs.io/en/latest/usage/ubuntu/ )

2021-09-19 00:03:35,883 - Agent - INFO - Images Classified: 20
2021-09-19 00:03:35,883 - Agent - INFO - True Positives: 8
2021-09-19 00:03:35,883 - Agent - INFO - False Positives: 0
2021-09-19 00:03:35,883 - Agent - INFO - True Negatives: 10
2021-09-19 00:03:35,883 - Agent - INFO - False Negatives: 2

however after converting to IR the final results were as before:

2021-09-19 00:32:03,210 - Agent - INFO - Images Classifier: 20 
2021-09-19 00:32:03,210 - Agent - INFO - True Positives: 9
2021-09-19 00:32:03,211 - Agent - INFO - False Positives: 1
2021-09-19 00:32:03,211 - Agent - INFO - True Negatives: 9
2021-09-19 00:32:03,211 - Agent - INFO - False Negatives: 1

The original version however had the same results as directly above in all cases (Before converting to openvino) 

I haven't tested training with AI analytics and converting to IR yet as I don't have the time right now but the worse case scenario no longer exists as we can still get the same performance after converting to IR.  ( Documentation and testing results: https://hias-acute-lymphoblastic-leukemia-oneapi-classifier.readthedocs.io/en/latest/usage/raspberry... ) 

 

470 Views

Something else that may be useful, I found that without the following:

conda install -c conda-forge defusedxml 
pip3 install test-generator==0.1.1

Then converting to IR will fail when installing OpenVINO with conda, this isn't mentioned in the guide that I saw as required dependencies. 

RahulU_Intel
Moderator
409 Views

Hi,


Thanks for your reply. We are working on this internally. We are trying to do the inference with an open source dataset. We will keep you posted about our findings.


Thanks and Regards

Rahul


339 Views

Hi thank you no worries, sorry I couldn't provide the dataset directly to you, it is required to apply for this dataset. 

RahulU_Intel
Moderator
258 Views

Hi,


Thank you for your patience. We were able to reproduce your results at our side. We are checking on this internally, we will get back to you. Could you please share which version of tensorflow you are using.


Thanks and Regards

Rahul


167 Views

Hi Rahul thank you for the update. For TF MKL I installed Tensorflow using: 

conda create -n hias-all-oneapi-classifier python=3 
conda activate hias-all-oneapi-classifier
conda install tensorflow -c anaconda

In that environment the version installed is '2.0.0-beta1'.

 

Regarding the AI Analytics installation which has the problem it is intel-aikit-tensorflow 2021.1.1 and the version of Tensorflow listed is tensorflow 2.2.0 

Shailen_Sobhee
Employee
84 Views

Hello,


We would gladly like to assist you in understanding the performance issues encountered. However, I may be missing data here. 

 

Do I understand this right, you are getting poorer performance with Intel Tensorflor version 2.5.x from the AI Kit, compared to stock TensorFlow (same version)? 

Can you please tell me more about your workload or potentially share with us a sample of the workload including the dataset? Also, can you please describe the hardware you are using?

 

Thank you and kind regards,

 

Shailen Sobhee

AI Software Solutions Engineer

 


77 Views

Hi how are you? Printing tensorflow version from Python gives 2.0.0-beta1, from the conda list gives 1.3.1 from Intel Channel. This works ok  although the results differ from when the model was originally trained.

From AI DevKit with the latest install it is 2.2.0 from the Intel channel, however I remember at some point during installation TF was downgraded by Conda. 

The AI Devkit Tensorfow provides poor results compared to the original model, and the newer model created. 

Unfortunately as mentioned I cannot provide you with the dataset as it is against the terms of the license, you would need to apply for it here: https://homes.di.unimi.it/scotti/all/#download 

The device I am running on has an Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, however I was able to replicate the exact same scenario on an Intel NUC i7. 

Hope this helps. 

Reply