Converted yolo models crash on NCS

idata · ‎12-14-2017

Hi,

I have made my own tinyer yolo and trained it and it works with darknet without issues.

But when I compile it and run it on the stick I get a hug error I dont understand.

Can anyone guide me ?

I have attached the model and the weights here.

https://drive.google.com/open?id=1XnNkMyhZp0DBJvDqh_hzI1Fv_roY__r5

Here is the error I get

    Found stale device, resetting
    Device 0 Address: 13.1 - VID/PID 03e7:2150
    Starting wait for connect with 2000ms timeout
    Found Address: 13.1 - VID/PID 03e7:2150
    Found EP 0x81 : max packet size is 512 bytes
    Found EP 0x01 : max packet size is 512 bytes
    Found and opened device
    Performing bulk write of 865212 bytes...
    Successfully sent 865212 bytes of data in 74.994229 ms (11.002586 MB/s)
    Boot successful, device address 13.1
    Device 0 Address: 4.1 - VID/PID 03e7:f63b
    Found Address: 4.1 - VID/PID 03e7:f63b
    done
    Booted 4.1 -> VSC
    /usr/local/lib/python3.5/dist-packages/skimage/transform/_warps.py:84: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
      warn("The default mode, 'constant', will be changed to 'reflect' in "
    *** Error in `python3': malloc(): memory corruption: 0x00000000027527d0 ***
    ======= Backtrace: =========
    /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fc1c8cf07e5]
    /lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7fc1c8cfb13e]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fc1c8cfd184]
    /lib/x86_64-linux-gnu/libc.so.6(qsort_r+0x79)[0x7fc1c8cb2349]
    /media/anilas1/data/Ncs/YoloV2NCS/detectionExample/libpydetector.so(_ZN6Region13GetDetectionsEPfiiiiiiffiRSt6vectorI14DetectedObjectSaIS2_EE+0x56d)[0x7fc1b482225d]
    /media/anilas1/data/Ncs/YoloV2NCS/detectionExample/libpydetector.so(_ZN12YoloDetector6DetectEN5boost6python3api6objectEiiiiiiffi+0xe0)[0x7fc1b4825160]
    /media/anilas1/data/Ncs/YoloV2NCS/detectionExample/libpydetector.so(_ZN5boost6python6detail12caller_arityILj11EE4implIM12YoloDetectorFNS0_4listENS0_3api6objectEiiiiiiffiENS0_21default_call_policiesENS_3mpl8vector12IS6_RS5_S8_iiiiiiffiEEEclEP7_objectSI_+0x617)[0x7fc1b4825f17]
    /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0(_ZNK5boost6python7objects8function4callEP7_objectS4_+0x26d)[0x7fc1b3f6e00d]
    /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0(+0x29208)[0x7fc1b3f6e208]
    /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0(_ZN5boost6python21handle_exception_implENS_9function0IvEE+0x73)[0x7fc1b3f76053]
    /usr/lib/x86_64-linux-gnu/libboost_python-py35.so.1.58.0(+0x26409)[0x7fc1b3f6b409]
    python3(PyObject_Call+0x47)[0x5c1797]
    python3(PyEval_EvalFrameEx+0x4ec6)[0x53bba6]
    python3(PyEval_EvalFrameEx+0x4b04)[0x53b7e4]
    python3[0x540199]
    python3(PyEval_EvalCode+0x1f)[0x540e4f]
    python3[0x60c272]
    python3(PyRun_FileExFlags+0x9a)[0x60e71a]
    python3(PyRun_SimpleFileExFlags+0x1bc)[0x60ef0c]
    python3(Py_Main+0x456)[0x63fb26]
    python3(main+0xe1)[0x4cfeb1]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fc1c8c99830]
    python3(_start+0x29)[0x5d6049]
    ======= Memory map: ========
    00400000-007a9000 r-xp 00000000 fd:00 10485774                           /usr/bin/python3.5
    009a9000-009ab000 r--p 003a9000 fd:00 10485774                           /usr/bin/python3.5
    009ab000-00a42000 rw-p 003ab000 fd:00 10485774                           /usr/bin/python3.5

idata · ‎12-15-2017

@ashwinnair14, looks like the issue is @ the app level. I was able to successfully run your model through mvNCProfiler, which means that your graph file is 'syntactically correct'. I do not have your python app, so I tried loading your graph file into https://github.com/movidius/ncappzoo/blob/master/caffe/TinyYolo/run.py, but since the number of outputs in your network is different from the one that's on NCAppZoo I had to hack your 8.prototxt:

Last layer of your 8.prototxt

layer {
  name: "conv8"
  type: "Convolution"
  bottom: "scale7"
  top: "conv8"
  convolution_param {
    num_output: 45
    kernel_size: 1
    stride: 1
    pad: 0
  }
}

I added an fc9 to your 8.prototxt

layer {
  name: "conv8"
  type: "Convolution"
  bottom: "scale7"
  top: "conv8"
  convolution_param {
    num_output: 45
    kernel_size: 1
    stride: 1
    pad: 0
  }
}
layer {
  name: "fc9"
  type: "InnerProduct"
  bottom: "conv8"
  top: "fc9"
  inner_product_param {
    num_output: 1470
  }
}

The app runs fine, but the network doesn't detect any objects (output is a matrix of zeros). As a sanity check, I would recommend retraining your model with an fc layer that has 1470 outputs, and use https://github.com/movidius/ncappzoo/blob/master/caffe/TinyYolo/run.py for inferencing.

idata · ‎12-18-2017

Hello @AshwinVijayakumar ,

Thats some great insight Reading your answer I fell I had not given all details of the problem.

I had used the original darknet (https://github.com/pjreddie/darknet) to train my model with 4 classes which is why the last conv layer has 45 outputs.

I have used the scripts in github https://github.com/duangenquan/YoloV2NCS to convert the darknet to caffe model which is y they dont have last FC layer and use their YoloV2NCS/detectionExample/Main.py to run where I get the same error.

Is there an other way to convert the darknet models to NCS so that this is done automatically? Or do I need to train the network with some other framework?

I checked their code and found out that I get an error where the outputs are used to get bounding boxes and perform NMS.

I can probably try to find the error if not.