Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Patrick_T_Intel
Employee
382 Views

Error for ssd_mobilenet_v2_coco optimization

Hello,

When I am optimizing the ssd_mobilenet_v2_coco model trained on tensorflow, the error always return:

C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer>python mo.py --input_model=c:\export_dir\frozen_inference_graph.pb --output_dir c:\mo_output_dir --tensorflow_use_custom_operations_config extensions\front\tf\ssd_v2_support.json --output="detection_boxes,detection_scores,detection_classes,num_detections"
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      c:\export_dir\frozen_inference_graph.pb
        - Path for generated IR:        c:\mo_output_dir
        - IR output name:       frozen_inference_graph
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        detection_boxes,detection_scores,detection_classes,num_detections
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Offload unsupported operations:       False
        - Path to model dump for TensorBoard:   None
        - Update the configuration file with input/output node names:   None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\extensions\front\tf\ssd_v2_support.json
Model Optimizer version:        1.2.110.59f62983
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
[ ERROR ]  FusedBatchNorm doesn't support is_training=True
The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.
[ ERROR ]  -------------------------------------------------
[ ERROR ]  ----------------- INTERNAL ERROR ----------------
[ ERROR ]  Unexpected exception happened.
[ ERROR ]  Please contact Model Optimizer developers and forward the following information:
[ ERROR ]  'data_format'
[ ERROR ]  Traceback (most recent call last):
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\mo\main.py", line 291, in main
    return driver(argv)
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\mo\main.py", line 233, in driver
    mean_scale_values=mean_scale)
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\mo\pipeline\tf.py", line 186, in tf2nx
    class_registration.apply_replacements(graph, class_registration.ClassType.MIDDLE_REPLACER)
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 102, in apply_replacements
    replacer.find_and_replace_pattern(graph)
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\mo\utils\replacement_pattern.py", line 28, in find_and_replace_pattern
    apply_pattern(graph, **self.pattern(), action=self.replace_pattern)
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\mo\middle\pattern_match.py", line 54, in apply_pattern
    action(graph, match)
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\extensions\middle\FusedBatchNormNonConstant.py", line 43, in replace_pattern
    if (node.data_format != b'NHWC' or
  File "C:\Intel\computer_vision_sdk_2018.2.317\deployment_tools\model_optimizer\mo\graph\graph.py", line 188, in __getattr__
    return self.graph.node[self.node]
KeyError: 'data_format'
 
[ ERROR ]  ---------------- END OF BUG REPORT --------------
[ ERROR ]  -------------------------------------------------

 

Anything wrong with my trained model? I would appreciate if someone could provide your assistance.

0 Kudos
14 Replies
Monique_J_Intel
Employee
382 Views

Hi Patrick,

Try to use this command to run the model:

sudo python3 mo.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config extensions/front/tf/ssd_v2_support.json --output="detection_boxes,detection_scores,num_detections" --tensorflow_object_detection_api_pipeline_config ../model_downloader/object_detection/common/ssd_mobilenet_v2_coco/tf/ssd_mobilenet_v2_coco_2018_03_29/pipeline.config

This command uses the pipeline config and the  json config as R3(I'm assuming you are using R3, if not I highly recommend you upgrade) has changed it's way of handling the Tensorflow Object Detection API . If this doesn't work I suggest you supply your own pipeline config file. If you don't know how I can help with this.

Also, for future post we have the inside blue page for internal employees. 

Kind Regards,

Monique Jones

Mark_L_Intel1
Moderator
382 Views

Hi Patrick:

As Monique point out, you seems use R3 but the directory shows R2. The feature you are using started from R3.

I also noticed you are working on Windows and I think there might be related to the environment like Python version you are using.

I run exact command like yours on Ubuntu 16.04 without problem but I got a problem even install the prerequisties script. So you might stay on the Linux system.

Mark

Patrick_T_Intel
Employee
382 Views

Monique, Mark,

Thanks for your advice.

My situation is, the OpenVINO installed on my computer can optimize the pre-trained model downloaded from TensorFlow model zoo successfully. However, after training with my own data set, the trained model is unable to be optimized by OpenVINO with following errors (I updated my OpenVINO to r3 based on Monique's advice):

C:\Intel\computer_vision_sdk_2018.3.343\deployment_tools\model_optimizer>python mo.py --input_model c:\export_dir\frozen_inference_graph.pb --tensorflow_use_custom_operations_config extensions/front/tf/ssd_v2_support.json --output="detection_boxes,detection_scores,num_detections" --tensorflow_object_detection_api_pipeline_config C:\tensorflow\models\research\object_detection\samples\configs\pipeline.config --output_dir c:\mo_output_dir
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      c:\export_dir\frozen_inference_graph.pb
        - Path for generated IR:        c:\mo_output_dir
        - IR output name:       frozen_inference_graph
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        detection_boxes,detection_scores,num_detections
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Offload unsupported operations:       False
        - Path to model dump for TensorBoard:   None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  C:\tensorflow\models\research\object_detection\samples\configs\pipeline.config
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  C:\Intel\computer_vision_sdk_2018.3.343\deployment_tools\model_optimizer\extensions/front/tf/ssd_v2_support.json
Model Optimizer version:        1.2.185.5335e231
[ ERROR ]  Failed to convert tokens to dictionary: Wrong character "/ssd_mobilenet_v2_coco_2018_03_29/model.ckpt" in position 344
[ ERROR ]  Failed to generate dictionary representation of file with content: model {
  ssd {
    num_classes: 90
    image_resizer {
      fixed_shape_resizer {
        height: 300
        width: 300
      }
    }
    feature_extractor {
      type: "ssd_mobilenet_v2"
      depth_multiplier: 1.0
      min_depth: 16
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 3.99999989895e-05
          }
        }
        initializer {
          truncated_normal_initializer {
            mean: 0.0
            stddev: 0.0299999993294
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.999700009823
          center: true
          scale: true
          epsilon: 0.0010000000475
          train: true
        }
      }
      use_depthwise: true
    }
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    box_predictor {
      convolutional_box_predictor {
        conv_hyperparams {
          regularizer {
            l2_regularizer {
              weight: 3.99999989895e-05
            }
          }
          initializer {
            truncated_normal_initializer {
              mean: 0.0
              stddev: 0.0299999993294
            }
          }
          activation: RELU_6
          batch_norm {
            decay: 0.999700009823
            center: true
            scale: true
            epsilon: 0.0010000000475
            train: true
          }
        }
        min_depth: 0
        max_depth: 0
        num_layers_before_predictor: 0
        use_dropout: false
        dropout_keep_probability: 0.800000011921
        kernel_size: 3
        box_code_size: 4
        apply_sigmoid_to_scores: false
      }
    }
    anchor_generator {
      ssd_anchor_generator {
        num_layers: 6
        min_scale: 0.20000000298
        max_scale: 0.949999988079
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 3.0
        aspect_ratios: 0.333299994469
      }
    }
    post_processing {
      batch_non_max_suppression {
        score_threshold: 0.300000011921
        iou_threshold: 0.600000023842
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SIGMOID
    }
    normalize_loss_by_num_matches: true
    loss {
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      classification_loss {
        weighted_sigmoid {
        }
      }
      hard_example_miner {
        num_hard_examples: 3000
        iou_threshold: 0.990000009537
        loss_type: CLASSIFICATION
        max_negatives_per_positive: 3
        min_negatives_per_image: 3
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
  }
}
train_config {
  batch_size: 24
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    ssd_random_crop {
    }
  }
  optimizer {
    rms_prop_optimizer {
      learning_rate {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.00400000018999
          decay_steps: 800720
          decay_factor: 0.949999988079
        }
      }
      momentum_optimizer_value: 0.899999976158
      decay: 0.899999976158
      epsilon: 1.0
    }
  }
  fine_tune_checkpoint: "C:/ssd_mobilenet_v2_coco_2018_03_29/model.ckpt"
  num_steps: 10
  fine_tune_checkpoint_type: "detection"
}
train_input_reader {
  label_map_path: "C:/dataset/raccoon_dataset/pet_label_map.pbtxt"
  tf_record_input_reader {
    input_path: "C:/dataset/raccoon_dataset/train.record"
  }
}
eval_config {
  num_examples: 1
  max_evals: 10
  use_moving_averages: false
}
eval_input_reader {
  label_map_path: "C:/dataset/raccoon_dataset/pet_label_map.pbtxt"
  shuffle: false
  num_readers: 1
  tf_record_input_reader {
    input_path: "C:/dataset/raccoon_dataset/train.record"
  }
}
 
[ ERROR ]  Failed to parse pipeline.config file C:\tensorflow\models\research\object_detection\samples\configs\pipeline.config
 
C:\Intel\computer_vision_sdk_2018.3.343\deployment_tools\model_optimizer>
 
The pipeline.config is the same as I used for training. Does it make sense to reuse the pipeline.config for OpenVINO optimization?
 
Monique_J_Intel
Employee
382 Views

Hi Patrick,

I recommend doing a diff between the two files to see what the differences are. You can try to use the standard pipeline config file that's included in the OpenVINO package but my guess is that since you re=trained the model that some of the parameter values in your pipeline config may be different such as you restoring your model from the checkpoint I would maybe take that line out as it seems model optimizer can't parse that correctly . However, once you do the diff you could sync the value changes with the current file in the OpenVINO package.

Kind Regards,

Monique Jones

Monique_J_Intel
Employee
382 Views

Hi Patrick,

This looks like a bug with the parser in Model Optimizer. Can you please attach your files so that we can reproduce and resolve the issue?

Kind Regards,

Monique Jones

Patrick_T_Intel
Employee
382 Views

Thanks, Monique.

My config file attached. Ext file name 'config' is not allowed to upload, so I zip it accordingly.

Because my OpenVINO is windows version, there are many characters like ':', '\' in the file. May be escape character is required.

Patrick_T_Intel
Employee
382 Views

OH, I just did some test, and found the char ":" in the path of file is the root cause of this error. Any escape char should I add?

Monique_J_Intel
Employee
382 Views

Hi Patrick,

I have achieved the conversion of your model on my side by removing ":" and switching "\" char to "/" in all the paths in the pipeline.config file.

There shouldn't be any other escape chars to add. I've attached the updated pipeline config file that converted for me. Please remember I'm working on Linux instead of Windows but let me know if you run into issues or if you successfully convert the model with your pipeline config while taking out the ":" char.

Kind Regards,

Monique Jones

Kamarol__Amalina
Beginner
382 Views

I have trained a custom SSD mobilenet v1 using Tensorflow Object Detection API. I managed to freeze the graph and successfully used it in inferencing with Tensorflow. I plan to use it with the object_detection_sample_ssd in OpenVINO. However, I was unable to convert the model using model optimizer using the following command:

python3 ./mo_tf.py --input_model /home/amalina/tf-demo/models/research/object_detection/inference_graph/frozen_inference_graph.pb --tensorflow_use_custom_operations_config extensions/front/tf/ssd_v2_support.json --tensorflow_object_detection_api_pipeline_config /home/amalina/tf-demo/models/research/object_detection/inference_graph/pipeline.config --reverse_input_channels

These are the errors encountered:

The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.
[ ERROR ]  Cannot infer shapes or values for node "MultipleGridAnchorGenerator/ToFloat_11".
[ ERROR ]  NodeDef mentions attr 'Truncate' not in Op<name=Cast; signature=x:SrcT -> y:DstT; attr=SrcT:type; attr=DstT:type>; NodeDef: MultipleGridAnchorGenerator/ToFloat_11 = Cast[DstT=DT_FLOAT, SrcT=DT_INT32, Truncate=false](MultipleGridAnchorGenerator/ToFloat_11/x_port_0_ie_placeholder). (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <function tf_native_tf_node_infer at 0x7ff7796f6510>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Stopped shape/value propagation at "MultipleGridAnchorGenerator/ToFloat_11" node. 
 For more information please refer to Model Optimizer FAQ (<INSTALL_DIR>/deployment_tools/documentation/docs/MO_FAQ.html), question #38. 

Please advice. The frozen graph is attached.

Severine_H_Intel
Employee
382 Views

Dear Amalina,

can you upgrade your Tensorflow with a recent version? I solved a very similar issue by upgrading to TF 1.11.

This is the command to do so: 

pip3 install --upgrade tensorflow

Best, 

Severine

Kamarol__Amalina
Beginner
382 Views

Dear Severine,

Thanks for the response. I upgraded Tensorflow to version 1.11 and retrained the model. However I'm still getting the same error from Model Optimizer.

Kamarol__Amalina
Beginner
382 Views

Hi,

Update - I managed to generate BIN and XML files by running model optimizer in a virtual environment where Tensorflow is installed.

Truong__Dien_Hoa
New Contributor II
382 Views

Hi,

I am trying to optimize ssd_mobilenet_v1. However I encounter an error

[ ERROR ]  Failed to convert tokens to dictionary: Wrong character "Use" in position 62

My config doesn't have escape character in the path

numb question but how can I know what character is in position 62 ?

Thank you in advance

 

382 Views

Hello,

I'm having the same problem as Kamarol above (except instead of ToFloat_11 it is ToFloat_3). Using tensorflow 1.09.

Thanks.

Reply