poor performance of caffe ssd-mobilenet after upgrade ncsdk from 2.05 to 2.08.01.02

[idata]

Hi all,

The caffe ssd-mobilenet works perfect with ncsdk 2.05, after I upgrade the ncsdk to 2.08.01.02, the detection result does not make sense any more, any suggestion?

I train the caffe ssd-mobilenet model based on https://github.com/chuanqi305/MobileNet-SSD

and convert to graph using the cmd:

root@31373c8647eb:/opt/MobileNet-SSD# mvNCCompile /data/train/MobileNetSSD_deploy.prototxt -w /data/train/MobileNetSSD_deploy.caffemodel -o /data/train/graph -s 12 /usr/local/bin/ncsdk/Controllers/Parsers/TensorFlowParser/Convolution.py:46: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(False, "Layer type not supported by Convolution: " + obj.type) /usr/local/bin/ncsdk/Controllers/Parsers/Phases.py:322: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(len(pred) == 1, "Slice not supported to have >1 predecessors") mvNCCompile v02.00, Copyright @ Intel Corporation 2017 Fusing Pad and Convolution2D Fusing BatchNorm and Scale after Convolution Replacing BN with Bias&Scale Fusing Permute and Flatten Fusing Eltwise and Relu Eliminate layers that have been parsed as NoOp Evaluating input and weigths for each hw layer -------------------------------------- Skipping data#120 Skipping conv11#143 Skipping conv13#147 Skipping conv14_2#149 Skipping conv15_2#151 Skipping conv16_2#153 Skipping conv17_2#155 Adding edge between data#120 and input_convert#266 Adding edge between input_convert#266 and data_converted#265 Successors before mutation ['conv0#1', 'conv11_mbox_priorbox#77', 'conv13_mbox_priorbox#84', 'conv14_2_mbox_priorbox#91', 'conv15_2_mbox_priorbox#98', 'conv16_2_mbox_priorbox#105', 'conv17_2_mbox_priorbox#112'] Adding edge between data#120 and input_convert#268 Adding edge between input_convert#268 and data_converted#267 Successors before mutation ['conv0#1', 'conv13_mbox_priorbox#84', 'conv14_2_mbox_priorbox#91', 'conv15_2_mbox_priorbox#98', 'conv16_2_mbox_priorbox#105', 'conv17_2_mbox_priorbox#112', 'input_convert#266'] Adding edge between data#120 and input_convert#270 Adding edge between input_convert#270 and data_converted#269 Successors before mutation ['conv0#1', 'conv14_2_mbox_priorbox#91', 'conv15_2_mbox_priorbox#98', 'conv16_2_mbox_priorbox#105', 'conv17_2_mbox_priorbox#112', 'input_convert#266', 'input_convert#268'] Adding edge between data#120 and input_convert#272 Adding edge between input_convert#272 and data_converted#271 Successors before mutation ['conv0#1', 'conv15_2_mbox_priorbox#98', 'conv16_2_mbox_priorbox#105', 'conv17_2_mbox_priorbox#112', 'input_convert#266', 'input_convert#268', 'input_convert#270'] Adding edge between data#120 and input_convert#274 Adding edge between input_convert#274 and data_converted#273 Successors before mutation ['conv0#1', 'conv16_2_mbox_priorbox#105', 'conv17_2_mbox_priorbox#112', 'input_convert#266', 'input_convert#268', 'input_convert#270', 'input_convert#272'] Adding edge between data#120 and input_convert#276 Adding edge between input_convert#276 and data_converted#275 Successors before mutation ['conv0#1', 'conv17_2_mbox_priorbox#112', 'input_convert#266', 'input_convert#268', 'input_convert#270', 'input_convert#272', 'input_convert#274'] Adding edge between conv11#143 and conv11/relu_convert#278 Adding edge between conv11/relu_convert#278 and conv11_converted#277 Successors before mutation ['conv11_mbox_conf_new#74', 'conv11_mbox_loc#71', 'conv11_mbox_priorbox#77', 'conv12/dw#47'] Adding edge between conv13#147 and conv13/relu_convert#280 Adding edge between conv13/relu_convert#280 and conv13_converted#279 Successors before mutation ['conv13_mbox_conf_new#81', 'conv13_mbox_loc#78', 'conv13_mbox_priorbox#84', 'conv14_1#55'] Adding edge between conv14_2#149 and conv14_2/relu_convert#282 Adding edge between conv14_2/relu_convert#282 and conv14_2_converted#281 Successors before mutation ['conv14_2_mbox_conf_new#88', 'conv14_2_mbox_loc#85', 'conv14_2_mbox_priorbox#91', 'conv15_1#59'] Adding edge between conv15_2#151 and conv15_2/relu_convert#284 Adding edge between conv15_2/relu_convert#284 and conv15_2_converted#283 Successors before mutation ['conv15_2_mbox_conf_new#95', 'conv15_2_mbox_loc#92', 'conv15_2_mbox_priorbox#98', 'conv16_1#63'] Adding edge between conv16_2#153 and conv16_2/relu_convert#286 Adding edge between conv16_2/relu_convert#286 and conv16_2_converted#285 Successors before mutation ['conv16_2_mbox_conf_new#102', 'conv16_2_mbox_loc#99', 'conv16_2_mbox_priorbox#105', 'conv17_1#67'] Adding edge between conv17_2#155 and conv17_2/relu_convert#288 Adding edge between conv17_2/relu_convert#288 and conv17_2_converted#287 Successors before mutation ['conv17_2_mbox_conf_new#109', 'conv17_2_mbox_loc#106', 'conv17_2_mbox_priorbox#112'] ---------------------- ---------------------- # Network Input tensors ['data#120'] # Network Output tensors ['detection_out#204'] /usr/local/bin/ncsdk/Controllers/FileIO.py:65: UserWarning: You are using a large type. Consider reducing your data sizes for best performance Blob generated

[idata]

code for decode the output:

from mvnc import mvncapi def _ncsdk_caffe_mobilenetssd_detect(self, image): # adjust values to range between -1.0 and +1.0 image = image - 127.5 image = image * 0.007843 # Read and pre-process input (data type must match input Fifo data type) input_tensor = image.astype(numpy.float32) # Write the image to the input queue and queue the inference in one call self._ncsdk_graph.queue_inference_with_fifo_elem(self._ncsdk_input_fifo, self._ncsdk_output_fifo, input_tensor, None) # Get the results from the output queue output, user_obj = self._ncsdk_output_fifo.read_elem() num_valid_bboxes = int(output[0]) detections = [] for bbox_index in range(num_valid_bboxes): base_index = 7+ bbox_index * 7 if (not numpy.isfinite(output[base_index]) or not numpy.isfinite(output[base_index + 1]) or not numpy.isfinite(output[base_index + 2]) or not numpy.isfinite(output[base_index + 3]) or not numpy.isfinite(output[base_index + 4]) or not numpy.isfinite(output[base_index + 5]) or not numpy.isfinite(output[base_index + 6])): # bboxes with non infinite (inf, nan, etc) numbers must be ignored # print('bbox at index: ' + str(bbox_index) + ' has nonfinite data, ignoring it') continue # clip the bboxes to the image size incase network returns bboxes outside of the image x1 = max(0, float(output[base_index + 3])) y1 = max(0, float(output[base_index + 4])) x2 = min(1, float(output[base_index + 5])) y2 = min(1, float(output[base_index + 6])) score = output[base_index + 2] #class index. classid = output[base_index + 1] detections.append({ "bbox": [x1,y1,x2,y2], "score": float(score), "class": int(classid) }) return detections #singleton device object device_list = mvncapi.enumerate_devices() device = mvncapi.Device(device_list[0]) device.open() # Initialize a graph from file with open(self.model_path, mode='rb') as f: graph_buffer = f.read() graph = mvncapi.Graph(self._name) # Allocate the graph to the device and create input/output Fifos with default options in one call input_fifo, output_fifo = graph.allocate_with_fifos(device, graph_buffer) self._ncsdk_graph = graph self._ncsdk_input_fifo = input_fifo self._ncsdk_output_fifo = output_fifo

[idata]

@cfdcfc

Do you change the situation when you use the graph file with the URL below?It is a compiled graph with NCSDK v2.08.01.https://github.com/PINTO0309/MobileNet-SSD-RealSense/raw/master/graph

[idata]

@PINTO

Thanks for the graph, the detection results make sense to me now:raw [{'bbox': [-1.0, -1.0, -1.0, -1.0], 'class': -1, 'score': -1.0}] raw [{'bbox': [-1.0, -1.0, -1.0, -1.0], 'class': -1, 'score': -1.0}] raw [{'bbox': [-1.0, -1.0, -1.0, -1.0], 'class': -1, 'score': -1.0}] raw [{'bbox': [0.1387939453125, 0.474609375, 0.2064208984375, 0.5615234375], 'class': 7, 'score': 0.33349609375}] raw [{'bbox': [-1.0, -1.0, -1.0, -1.0], 'class': -1, 'score': -1.0}] raw [{'bbox': [-1.0, -1.0, -1.0, -1.0], 'class': -1, 'score': -1.0}]

here's the detection results with my own graph:

raw [{'score': 0.4892578125, 'class': 1, 'bbox': [85.3125, 85.3125, 85.3125, 85.3125]}, {'score': 0.474365234375, 'class': 1, 'bbox': [85.3125, 85.3125, 85.3125, 85.3125]}, {'score': 0.36962890625, 'class': 1, 'bbox': [85.3125, 85.3125, 85.3125, 85.3125]}] raw [{'score': 0.60302734375, 'class': 1, 'bbox': [85.3125, 85.3125, 85.3125, 85.3125]}, {'score': 0.426513671875, 'class': 1, 'bbox': [85.3125, 85.3125, 85.3125, 85.3125]}, {'score': 0.3525390625, 'class': 1, 'bbox': [85.3125, 85.3125, 85.3125, 85.3125]}, {'score': 0.261962890625, 'class': 1, 'bbox': [85.3125, 85.3125, 85.3125, 85.3125]}]

Can you suggest any modification should be done after upgrade ncsdk from 2.05 to 2.08.01.02?

[idata]

@cfdcfc

1. First of all, did my graph return the expected results?
2. Can you publish .prototxt?

[idata]

@PINTO

1. yes
2. https://github.com/cfdcfc/ncsdk/blob/master/MobileNetSSD_deploy.prototxt

[idata]

@cfdcfc

”conv17_2_mbox_conf_new”

Is this layer not connected correctly?

.prototxt appears to be wrong.I think that ”_new” is unnecessary.Please check other parts as well.layer { name: "conv17_2_mbox_conf_new" type: "Convolution" bottom: "conv17_2" top: "conv17_2_mbox_conf" param { lr_mult: 1.0 decay_mult: 1.0 } param { lr_mult: 2.0 decay_mult: 0.0 } convolution_param { num_output: 12 kernel_size: 1 weight_filler { type: "msra" } bias_filler { type: "constant" value: 0.0 } } }

[idata]

@PINTO

the prototxt file and the model comes from

https://github.com/chuanqi305/MobileNet-SSD/blob/master/template/MobileNetSSD_deploy_template.prototxt

I train my own model based on that.