OpenVINO calibration tool: Segmentation fault

[Ermak__Aleksei]

Hi,

I am trying to generate INT8 model from my detection FP32 IR model by calibration_tool, but getting segmentation fault during calibration.

I found this happens if the model weights are set to non-zero values (ones for example). I tried to generate model with zero weights and it succeeded without any problems (of course accuracy is zero).

Here is the calibration output:

[ INFO ] InferenceEngine: 
        API version ............ 1.4
        Build .................. 19154
[ INFO ] Parsing input parameters
[ INFO ] Loading plugin

        API version ............ 1.5
        Build .................. lnx_20181004
        Description ....... MKLDNNPlugin
[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Batch size is 1
[ INFO ] Collecting accuracy metric in FP32 mode to get a baseline, collecting activation statistics
Progress: [......              ] 30.00% done./calibrate.sh: 
line 16: 22614 Segmentation fault      (core dumped)

Output from GDB debugging:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[ INFO ] InferenceEngine: 
        API version ............ 1.4
        Build .................. 19154
[ INFO ] Parsing input parameters
[ INFO ] Loading plugin

        API version ............ 1.5
        Build .................. lnx_20181004
        Description ....... MKLDNNPlugin
[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Batch size is 1
[New Thread 0x7fffee7b6700 (LWP 22720)]
[New Thread 0x7fffedfb5700 (LWP 22721)]
[New Thread 0x7fffed5b3780 (LWP 22722)]
[ INFO ] Collecting accuracy metric in FP32 mode to get a baseline, collecting activation statistics
[New Thread 0x7fffeca92700 (LWP 22723)]
[New Thread 0x7fffdffff700 (LWP 22724)]
[New Thread 0x7fffdf7fe700 (LWP 22725)]
[Thread 0x7fffeca92700 (LWP 22723) exited]
[New Thread 0x7fffeca92700 (LWP 22726)]
[Thread 0x7fffedfb5700 (LWP 22721) exited]
[Thread 0x7fffee7b6700 (LWP 22720) exited]
Progress: [..                  ] 10.00% done
Thread 1 "calibration_too" received signal SIGSEGV, Segmentation fault.
__memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:931
931     ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory.

I assume there may be a problem with outputs of my model and the incorrectly organized calibration dataset. Dataset is prepared by tutorial, but it is not clear for me what must be in annotation and how calibration tool use it. To reproduce issue I attached zip archive with dataset example, two types of my model (with zero weights and ones weights) and bash script with parameters I run calibration tool with. My OS is Ubuntu 16.04

Can you help me please with that problem?

Thanks 

[Edward_S_Intel1]

Hi Aleksei,

New Python Calibration Tool will be announced in 2019 R1 release. New tool supports more datasets and topologies. Can you, please, use new tool after release? 

Let me know if you still have any questions

Best regards, Edward

[Ermak__Aleksei]

Hi Edward,

thanks you for reply. Does this mean that the current calibration tool doesn't support custom topologies?

Where can I read how works existing calibration tool and how it use supported dataset annotation?

I mean maybe my model must have a certain output format for calibration with described VOC dataset.

When can I expect 2019 R1 release? I can not find the exact date.

Regards,

Aleksei

[Shubha_R_Intel]

Dear  Aleksei,

2019 R1 was just released today.

Thanks for using OpenVino !

Shubha

[cbohr1]

hi i am running a code usingopenvino+ ncs its having 3 models of cnn each 70 mb and output of these 3 given input to fourth which is lstm based model .it shows segmentation fault.does openvino+ncs doenot support cnn+lstm.do reply .it works well on cpu.

[Shubha_R_Intel]

Dear bohra, chandni,

Please download and try OpenVino 2019R2 if you haven't already. Lots of improvements and fixes have been made. Are you getting the seg fault on 2019R2 ?

Thanks,

Shubha

[cbohr1]

NO  mam i m not getting segmentation fault on Openvino R2 .I am getting some other error if i m using cpu+ncs its not able to load lstm model which is taking output of three cnn model as input .its showing some plugin error .

[Shubha_R_Intel]

Dear bohra, chandni,

Thank you for using OpenVino R2.

First I think you're missing some steps. You can't just jump to calibrate.py. Please follow my response to github post 171 .

With the calibration tools, there are some very important steps.

First, please make sure you install all the prerequisites:

using pip install -r requirements.txt

C:\Program Files (x86)\IntelSWTools\openvino_2019.2.242\python\python3.7\openvino\tools\calibration\requirements.txt

Make sure you can do an

python accuracy_check.py --h

python calibrate.py --h

Personally I found it much easier to install the prerequisites in requirements.txt on Ananconda than a regular Python installation. 

Make sure you can do --h successfully, that's how you know that you have all the pre-requisites. Then kindly follow the steps outlined in the aforementioned github post.

Hope it helps,

Thanks,

Shubha

[cbohr1]

[ INFO ] Loading models to the plugin Traceback (most recent call last): File "LSTM_face_detection_dlib.py", line 504, in sys.exit(main() or 0) File "LSTM_face_detection_dlib.py", line 195, in main exec_net4 = plugin.load(network=net4) File "ie_api.pyx", line 551, in openvino.inference_engine.ie_api.IEPlugin.load File "ie_api.pyx", line 561, in openvino.inference_engine.ie_api.IEPlugin.load RuntimeError: AssertionFailed: StatusCode::OK == network->getLayerByName(layerName.c_str(), layer, &resp) Mam I m not using calibration.py inference .I generate my own inference file to run this model in that i m getting above error while using Myriad but without using Myriad its working very fine.Without using LSTM with MYRIAD also working fine.

[cbohr1]

hi shobha in the openvino R2 toolkit the text recognition model is using bidirectional lstm but when i am running it using ncs it responding 0 fps and readme file of test detection demo is showing that this demo doesnot support ncs so it concludes that lstm does not support ncs... Thanku...do reply

[Shubha_R_Intel]

Dear bohra, chandni,

LSTM is definitely supported by NCS, but GRU is not.

If you have a sample model which contains LSTM which fails inference on NCS2, I'd be happy to file a bug on your behalf. Please attach the *.zip to this ticket.

Thanks kindly !

Shubha

[cbohr1]

dear shobha i pm u my model that using lstm with tanh but u r not replying please do reply.. i m waiting

[Shubha_R_Intel]

Dearest bohra, chandni,

I never got your attachment ! I will PM you again. Please retry sending it. Honestly, I never got your PM. But sorry for your trouble. Let's try this again.

Thanks,

Shubha