Using retrain MobileNet-SSD's problems

[idata]

Hi, I am try to using retrained MobileNet-SSD model, there have some problems! The retrained was fetched on https://github.com/zeusees/SSD_License_Plate_Detection.

I merge the bn by merge_bn.py and rewrite the conf name to remove "_new" by my own script (https://github.com/zhoujustin/MobileNetSSD_Tools/blob/master/mobssd_rm_new.py).When I using the caffe.net, I can get the result. BUT mvNCCompile the model and weight, the result always return NAN. How can I debug it?The coverted caffemodel prototxt and graph can be found in my google drive(https://drive.google.com/file/d/1wMQWSN4zfDPaXXEuF4BlmsLJvNvCIWxk/view?usp=sharing)Thanks

[idata]

@Zhoustin Hi, I'm having some problems compiling and checking your provided files.

WARNING: Logging before InitGoogleLogging() is written to STDERR F0403 13:39:33.733206 9083 blob.cpp:507] Check failed: count_ == proto.data_size() (864 vs. 0) *** Check failure stack trace: *** Aborted (core dumped)

Can you please double check your caffe and weights file and send me another link?

[idata]

Sorry for the model which compiled on the jetson TX platform.

I redo it on my pc and there is the link(https://drive.google.com/open?id=1xNGQbszsFmAv4Npk02QV5adDqyVHMLQX)Appreciate for your reply!

[idata]

@Zhoujustin It looks like some of the values from the conv2 layer are nearing the limit of fp16.

Result: (128, 160, 120)

1) 426420 nan2) 1707207 nan3) 1707198 nan4) 1707197 nan5) 1707196 nanExpected: (128, 160, 120)1) 1896613 62300.02) 1896012 62240.03) 1882207 62200.04) 1893698 61950.05) 1887356 60960.0

I'll keep digging. Will keep you posted.

[idata]

Thanks for your digging. We look forward to good news.

[idata]

@Zhoujustin The prototxt file uses the same layer names for multiple top and bottom parameters. Our compiler doesn't seem to like that so I made some adjustments (i.e. unique top name for each Conv/BatchNorm/Scale, & same bottom/top name for Relus).

The input size is pretty large and I get nans if I run the graph, so I reduced the size of the input to 320 x 240 and it runs to the end (no more nans in conv2 layer also), but it results in invalid classification results. In order to get valid classification results, I think you may need to retrain the network with a smaller size.

You can view the edited prototxt file here.

[idata]

Thank you for your reply, I probably understand your revised content, I read the discussion on the issue of mobile net-ssd above 300 pixels image, I hope this problem can be resolved sooner.

[idata]

@Tome_at_Intel I'm retrain another graph with 300X300 model. This train detected the bread. The result was as same as above. I already using the prototxt you edited.

I already test the model in caffe, it worked well. But in the movidius it always return [ 1.000e+00 -1.882e-04 -1.346e-02 … -8.743e-03 1.944e-02 -2.725e+04]. Please give me some suggestions. Thanks.link: https://drive.google.com/file/d/19bG-YTM-cYLzmb5SFgSpNez-SLR2CHgn/view?usp=sharing

[idata]

The prototxt's input dim should be changed to 300 x 300. Sorry for it.

[idata]

@Zhoujustin Okay, I'll test it out with the new dims and let you know. Thanks.

[idata]

@Tome_at_Intel I found the problems. Because of the nvcaffe not the as ssd-caffe. When I changed into the ssd-caffe, there are no problems. All the model seems work well.

Thanks for your attention.

[idata]

@Zhoujustin Glad you were able to solve the problem.

[idata]

@Tome_at_Intel Thanks, Hope the anothers will avoid this 'bug'.

[idata]

@Zhoujustin, could you explain in detail how to fix it, i met the same problem

[idata]

@voqtuyen Using the https://github.com/chuanqi305/MobileNet-SSD, and https://github.com/weiliu89/caffe/tree/ssd. Don't using the other version.