Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

Using retrain MobileNet-SSD's problems

idata
Employee
1,048 Views

Hi, I am try to using retrained MobileNet-SSD model, there have some problems! The retrained was fetched on https://github.com/zeusees/SSD_License_Plate_Detection.

 

I merge the bn by merge_bn.py and rewrite the conf name to remove "_new" by my own script (https://github.com/zhoujustin/MobileNetSSD_Tools/blob/master/mobssd_rm_new.py).

 

When I using the caffe.net, I can get the result. BUT mvNCCompile the model and weight, the result always return NAN. How can I debug it?

 

The coverted caffemodel prototxt and graph can be found in my google drive(https://drive.google.com/file/d/1wMQWSN4zfDPaXXEuF4BlmsLJvNvCIWxk/view?usp=sharing)

 

Thanks
0 Kudos
14 Replies
idata
Employee
717 Views

@Zhoustin Hi, I'm having some problems compiling and checking your provided files.

 

WARNING: Logging before InitGoogleLogging() is written to STDERR F0403 13:39:33.733206 9083 blob.cpp:507] Check failed: count_ == proto.data_size() (864 vs. 0) *** Check failure stack trace: *** Aborted (core dumped)

 

Can you please double check your caffe and weights file and send me another link?

0 Kudos
idata
Employee
717 Views

Sorry for the model which compiled on the jetson TX platform.

 

I redo it on my pc and there is the link(https://drive.google.com/open?id=1xNGQbszsFmAv4Npk02QV5adDqyVHMLQX)

 

Appreciate for your reply!
0 Kudos
idata
Employee
717 Views

@Zhoujustin It looks like some of the values from the conv2 layer are nearing the limit of fp16.

 

Result: (128, 160, 120)

 

1) 426420 nan

 

2) 1707207 nan

 

3) 1707198 nan

 

4) 1707197 nan

 

5) 1707196 nan

 

Expected: (128, 160, 120)

 

1) 1896613 62300.0

 

2) 1896012 62240.0

 

3) 1882207 62200.0

 

4) 1893698 61950.0

 

5) 1887356 60960.0

 

I'll keep digging. Will keep you posted.

0 Kudos
idata
Employee
717 Views

Thanks for your digging. We look forward to good news.

0 Kudos
idata
Employee
717 Views

@Zhoujustin The prototxt file uses the same layer names for multiple top and bottom parameters. Our compiler doesn't seem to like that so I made some adjustments (i.e. unique top name for each Conv/BatchNorm/Scale, & same bottom/top name for Relus).

 

The input size is pretty large and I get nans if I run the graph, so I reduced the size of the input to 320 x 240 and it runs to the end (no more nans in conv2 layer also), but it results in invalid classification results. In order to get valid classification results, I think you may need to retrain the network with a smaller size.

 

You can view the edited prototxt file here.

0 Kudos
idata
Employee
717 Views

Thank you for your reply, I probably understand your revised content, I read the discussion on the issue of mobile net-ssd above 300 pixels image, I hope this problem can be resolved sooner.

0 Kudos
idata
Employee
717 Views

@Tome_at_Intel I'm retrain another graph with 300X300 model. This train detected the bread. The result was as same as above. I already using the prototxt you edited.

 

I already test the model in caffe, it worked well. But in the movidius it always return [ 1.000e+00 -1.882e-04 -1.346e-02 … -8.743e-03 1.944e-02 -2.725e+04].

 

Please give me some suggestions. Thanks.

 

link: https://drive.google.com/file/d/19bG-YTM-cYLzmb5SFgSpNez-SLR2CHgn/view?usp=sharing
0 Kudos
idata
Employee
717 Views

The prototxt's input dim should be changed to 300 x 300. Sorry for it.

0 Kudos
idata
Employee
717 Views

@Zhoujustin Okay, I'll test it out with the new dims and let you know. Thanks.

0 Kudos
idata
Employee
717 Views

@Tome_at_Intel I found the problems. Because of the nvcaffe not the as ssd-caffe. When I changed into the ssd-caffe, there are no problems. All the model seems work well.

 

Thanks for your attention.
0 Kudos
idata
Employee
717 Views

@Zhoujustin Glad you were able to solve the problem.

0 Kudos
idata
Employee
717 Views

@Tome_at_Intel Thanks, Hope the anothers will avoid this 'bug'.

0 Kudos
idata
Employee
717 Views

@Zhoujustin, could you explain in detail how to fix it, i met the same problem

0 Kudos
idata
Employee
717 Views

@voqtuyen Using the https://github.com/chuanqi305/MobileNet-SSD, and https://github.com/weiliu89/caffe/tree/ssd. Don't using the other version.

0 Kudos
Reply