Error for accuracy calculation on own model of Imagenet style data

[kalocsai]

This is the error message from accuracy calculation on the DL Workbench:

-------------------------------------------------------------------------------------------------------

[ACCURACY CHECKER] 800 objects total
[ACCURACY CHECKER] 800 batches total
[ACCURACY CHECKER] Measurement Done
[ACCURACY CHECKER] 800 objects total
[ACCURACY CHECKER] 800 batches total

Traceback (most recent call last):

File "/opt/intel/openvino_2021/deployment_tools/tools/workbench/wb/main/console_tool_wrapper/accuracy_tools/accuracy/check_accuracy.py", line 95, in <module> main()

File "/opt/intel/openvino_2021/deployment_tools/tools/workbench/wb/main/console_tool_wrapper/accuracy_tools/accuracy/check_accuracy.py", line 82, in main model_evaluator.process_dataset(None, progress_reporter=progress_reporter)

File "/opt/intel/openvino/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/evaluators/model_evaluator.py", line 333, in process_dataset_sync for batch_id, (batch_input_ids, batch_annotation, batch_input, batch_identifiers) in enumerate(self.dataset):

File "/opt/intel/openvino/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/dataset.py", line 243, in __getitem__ return self.data_provider[item]

File "/opt/intel/openvino/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/dataset.py", line 583, in __getitem__ batch_input = [self.data_reader(identifier=identifier) for identifier in batch_identifiers]

File "/opt/intel/openvino/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/dataset.py", line 583, in <listcomp> batch_input = [self.data_reader(identifier=identifier) for identifier in batch_identifiers]

File "/opt/intel/openvino/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/data_readers/data_reader.py", line 220, in __call__ return self.read_item(identifier)

File "/opt/intel/openvino/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/data_readers/data_reader.py", line 319, in read_item identifier=data_id if not isinstance(data_id, ListIdentifier) else list(data_id.values)

File "/opt/intel/openvino/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/data_readers/data_reader.py", line 79, in __init__ self.metadata['image_size'] = data.shape if not isinstance(data, list) else np.shape(data[0])

AttributeError: 'NoneType' object has no attribute 'shape'

------------------------------------------------------------------------------------------------------------

Any help on fixing this would be greatly appreciated.

[Wan_Intel]

Hi Kalocsai,

Thank you for reaching out to us.

Could you please share your model and dataset with us for further replication from our side?

Regards,

Wan

[kalocsai]

Hi Wan,

Sure thing, here is the data you requested: 

TestPackage.zip

This zip file contains the following:

1. SavedModel folder containing the tf2 model

2. A zip file containing the test data and labels.txt file in Imagenet format

3. ConfigForWorkbench.yml - the config file I used in DL Workbench

4. ConfigForAccuracyChecker.yml -the almost identical config file I used with the command line Accuracy Checker tool

5. WorkbenchError.txt - the error message produced by DL Workbench when running accuracy

So, the command line  Accuracy Checker tool could run this model, data combination without any problem. The workbench could also run this combination just not for accuracy. Accuracy produces the above error. I have also tried a .h5 Keras version of the model and the IR converted version as well. All producing this same error. 

The data doesn't need preprocessing, ready to go as is. The data is 150x150, 6 channel numpy arrays saved in .npy format. Please let me know if you need anything else or if you have any questions about this setup.

Thank you very much for your help.

peter

[kalocsai]

Hi Wan,

I had sent you the requested data last Thursday. Do you happen to have any update on this for me by any chance?

Should you have any questions about the model or data, then please don't hesitate to ask.

I think the DL Workbench is a really great tool and I would love to use it for comparing various model and precision configurations, but, of course, for that I would need the accuracy part of the tool work with my setup as well.

Thanks again very much for your help with this!

Best regards,

peter

[Wan_Intel]

Hi Kalocsai,

Thank you for sharing your model and dataset with us.

For your information, I have validated the Command Line Accuracy Checker tool with your model and dataset.

Also, I encountered the same error as you when using the DL Workbench Accuracy Checker tool with your model and dataset.

I have forwarded this case to the development team for further investigation and we will update you at the earliest.

Regards,

Wan

[Wan_Intel]

Hi Kalocsai,

Thanks for your patience.

Based on the development team's response, DL Workbench supports only the format of the ImageNet validation dataset published in 2012 which is in JPEG format as mentioned here:

https://docs.openvinotoolkit.org/latest/workbench_docs_Workbench_DG_Dataset_Types.html#imagenet

Hence, a different dataset format is not supported, thus the error is expected.

On another note, if you want to use the .npy format, we suggest you utilize our command line Accuracy Checker tool from the OpenVINO toolkit as it was executed successfully.

Regards,

Wan

[kalocsai]

Hi Wan,

Thank you for looking into this!

Of course, I am familiar with the above mentioned reference, but thank you anyway! I suppose the confusion might come from the fact that DL Workbench appears to be perfectly capable of processing my .npy data for latency and throughput calculations and only not for accuracy. So the question has always been: if the tool can handle a given data for some type of calculations, then why not another type? But I suppose this might be sorted out in some future versions of DL Workbench. 

 Yes, as I mentioned earlier, I already ran the model and data with the Accuracy Checker. It's only that DL Workbench's analytical and graphing capabilities were very appealing to me, as Intel also seems to encourage user's to take advantage of the Workbench as much as possible. 

In any case, I'll try to use some other tool for now and thank you again for taking the time to look into this!

Best regards,

peter 

[Wan_Intel]

Hi Kalocsai,

This thread will no longer be monitored since we have provided a solution.

If you need any additional information from Intel, please submit a new question.

Regards,

Wan