I am trying to generate INT8 model from my detection FP32 IR model by calibration_tool, but getting segmentation fault during calibration.
I found this happens if the model weights are set to non-zero values (ones for example). I tried to generate model with zero weights and it succeeded without any problems (of course accuracy is zero).
Here is the calibration output:
[ INFO ] InferenceEngine: API version ............ 1.4 Build .................. 19154 [ INFO ] Parsing input parameters [ INFO ] Loading plugin API version ............ 1.5 Build .................. lnx_20181004 Description ....... MKLDNNPlugin [ INFO ] Loading network files [ INFO ] Preparing input blobs [ INFO ] Batch size is 1 [ INFO ] Collecting accuracy metric in FP32 mode to get a baseline, collecting activation statistics Progress: [...... ] 30.00% done./calibrate.sh: line 16: 22614 Segmentation fault (core dumped)
Output from GDB debugging:
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [ INFO ] InferenceEngine: API version ............ 1.4 Build .................. 19154 [ INFO ] Parsing input parameters [ INFO ] Loading plugin API version ............ 1.5 Build .................. lnx_20181004 Description ....... MKLDNNPlugin [ INFO ] Loading network files [ INFO ] Preparing input blobs [ INFO ] Batch size is 1 [New Thread 0x7fffee7b6700 (LWP 22720)] [New Thread 0x7fffedfb5700 (LWP 22721)] [New Thread 0x7fffed5b3780 (LWP 22722)] [ INFO ] Collecting accuracy metric in FP32 mode to get a baseline, collecting activation statistics [New Thread 0x7fffeca92700 (LWP 22723)] [New Thread 0x7fffdffff700 (LWP 22724)] [New Thread 0x7fffdf7fe700 (LWP 22725)] [Thread 0x7fffeca92700 (LWP 22723) exited] [New Thread 0x7fffeca92700 (LWP 22726)] [Thread 0x7fffedfb5700 (LWP 22721) exited] [Thread 0x7fffee7b6700 (LWP 22720) exited] Progress: [.. ] 10.00% done Thread 1 "calibration_too" received signal SIGSEGV, Segmentation fault. __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:931 931 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory.
I assume there may be a problem with outputs of my model and the incorrectly organized calibration dataset. Dataset is prepared by tutorial, but it is not clear for me what must be in annotation and how calibration tool use it. To reproduce issue I attached zip archive with dataset example, two types of my model (with zero weights and ones weights) and bash script with parameters I run calibration tool with. My OS is Ubuntu 16.04
Can you help me please with that problem?
New Python Calibration Tool will be announced in 2019 R1 release. New tool supports more datasets and topologies. Can you, please, use new tool after release?
Let me know if you still have any questions
Best regards, Edward
thanks you for reply. Does this mean that the current calibration tool doesn't support custom topologies?
Where can I read how works existing calibration tool and how it use supported dataset annotation?
I mean maybe my model must have a certain output format for calibration with described VOC dataset.
When can I expect 2019 R1 release? I can not find the exact date.
Dear bohra, chandni,
Please download and try OpenVino 2019R2 if you haven't already. Lots of improvements and fixes have been made. Are you getting the seg fault on 2019R2 ?
NO mam i m not getting segmentation fault on Openvino R2 .I am getting some other error if i m using cpu+ncs its not able to load lstm model which is taking output of three cnn model as input .its showing some plugin error .
Dear bohra, chandni,
Thank you for using OpenVino R2.
First I think you're missing some steps. You can't just jump to calibrate.py. Please follow my response to github post 171 .
With the calibration tools, there are some very important steps.
First, please make sure you install all the prerequisites:
using pip install -r requirements.txt
C:\Program Files (x86)\IntelSWTools\openvino_2019.2.242\python\python3.7\openvino\tools\calibration\requirements.txt
Make sure you can do an
python accuracy_check.py --h
python calibrate.py --h
Personally I found it much easier to install the prerequisites in requirements.txt on Ananconda than a regular Python installation.
Make sure you can do --h successfully, that's how you know that you have all the pre-requisites. Then kindly follow the steps outlined in the aforementioned github post.
Hope it helps,
Dear bohra, chandni,
LSTM is definitely supported by NCS, but GRU is not.
If you have a sample model which contains LSTM which fails inference on NCS2, I'd be happy to file a bug on your behalf. Please attach the *.zip to this ticket.
Thanks kindly !
Dearest bohra, chandni,
I never got your attachment ! I will PM you again. Please retry sending it. Honestly, I never got your PM. But sorry for your trouble. Let's try this again.