shape mismatch when optimize resnet_v1_50.pb

Dongjie_S_Intel · ‎04-03-2019

Hi,

I downloaded resnet_v1_50_2016_08_28.tar.gz from http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz, and use dldt 2018_R5 to optimize it. Get shape mismatch error. Please help.

Commands:

python3 tf_models/research/slim/export_inference_graph.py \
--model_name resnet_v1_50 \
--output_file ov_irs/resnet_v1_50.pb

python3 ~/dldt/model-optimizer/mo/utils/summarize_graph.py --input_model ov_irs/resnet_v1_50.pb

wget http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz
tar xvf resnet_v1_50_2016_08_28.tar.gz -C ov_irs

python3 ~/dldt/model-optimizer/mo_tf.py --input_model ./ov_irs/resnet_v1_50.pb --input_checkpoint ./ov_irs/resnet_v1_50.ckpt -b 1 --mean_value [103.94,116.78,123.68] --scale 1

output:

Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /home/glorysdj/./ov_irs/resnet_v1_50.pb
- Path for generated IR: /home/glorysdj/.
- IR output name: resnet_v1_50
- Log level: ERROR
- Batch: 1
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: Not specified, inherited from the model
- Mean values: [103.94,116.78,123.68]
- Scale values: Not specified
- Scale factor: 1.0
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: False
TensorFlow specific parameters:
- Input model in text protobuf format: False
- Offload unsupported operations: False
- Path to model dump for TensorBoard: None
- List of shared libraries with TensorFlow custom layers implementation: None
- Update the configuration file with input/output node names: None
- Use configuration file used to generate the model with Object Detection API: None
- Operations to offload: None
- Patterns to offload: None
- Use the config file: None
Model Optimizer version: 1.5.12.49d067a0
[ FRAMEWORK ERROR ] Cannot load input model: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]
[[node save/Assign_266 (defined at /home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py:105) = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/weights, save/RestoreV2:266)]]

Caused by op 'save/Assign_266', defined at:
File "/home/glorysdj/dldt/model-optimizer/mo_tf.py", line 31, in <module>
sys.exit(main(get_tf_cli_parser(), 'tf'))
File "/home/glorysdj/dldt/model-optimizer/mo/main.py", line 325, in main
return driver(argv)
File "/home/glorysdj/dldt/model-optimizer/mo/main.py", line 267, in driver
mean_scale_values=mean_scale)
File "/home/glorysdj/dldt/model-optimizer/mo/pipeline/tf.py", line 127, in tf2nx
saved_model_tags=argv.saved_model_tags)
File "/home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py", line 208, in load_tf_graph_def
graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
File "/home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py", line 105, in freeze_checkpoint
tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1102, in __init__
self.build()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 119, in restore
self.op.get_shape().is_fully_defined())
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
validate_shape=validate_shape)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]
[[node save/Assign_266 (defined at /home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py:105) = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/weights, save/RestoreV2:266)]]

Shubha_R_Intel · ‎04-04-2019

Dear Dongjie S,

Please make sure your Tensorflow model is a frozen protobuf first before you use Model Optimizer. Below are instructions on using Model Optimizer to convert a Tensorflow model:

https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html

Also you are using an older version of OpenVino (though not too old). I strongly encourage you to download the latest 2019 R1 release.

Thanks !

Shubha

Dan_L_Intel · ‎08-02-2019

Dear Dongjie S,

I encourted exactly the same error as you described "Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]" when modifying resnet_v1_50.pb model using OpenVINO 2019 R1.

I also downloaded resnet_v1_50_2016_08_28.tar.gz from http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz, from where our guide https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html suggests.

So did you solve this issue? Looking forward to your reply.

Thanks,

Una

Shubha_R_Intel · ‎08-02-2019

Dear everyone,

By now we have 2019R2. In fact it was just released last week. Please update to the latest release and try again.

Thanks,

Shubha

Bin_X_Intel · ‎09-22-2019

Hi Shubha, I'm using 2019R2, I still suffer same issue, any idea how to solve this? Thanks

Shubha_R_Intel · ‎09-24-2019

Dear Bin X,

Can you kindly be more explicit about your error ? This thread is pretty old. Is it a mismatch error using Model Optimizer ? And are you using the resnet_v1_50 model just like the Original Poster ? What is your exact model optimizer command ?

Thanks,

Shubha

Bin_X_Intel · ‎09-25-2019

Hi Shubha, I tried 2019R2 and internal 2019R3, all fail with same issue as Dongjie mentioned:

1. wget http://download.tensorflow.org/models/ resnet_v1_50_2016_08_28.tar.gz

2. tar xvf resnet_v1_50_2016_08_28.tar.gz

3. python3 research/slim/export_inference_graph.py --model_name resnet_v1_50 --output_file resnet_v1_50_inference_graph.pb

4. python3 /opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/utils/summarize_graph.py --input_model ./resnet_v1_50_inference_graph.pb

5. python3 /usr/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py --input_graph ./resnet_v1_50_inference_graph.pb --input_checkpoint resnet_v1_50.ckpt --input_binary=true --output_graph frozen_resnet_v1_50.pb --output_node_names=resnet_v1_50/predictions/Reshape_1

then error showed as attachment

Shubha_R_Intel · ‎09-27-2019

Dear Bin X.,

Why do you think that this is an OpenVino problem ? In looking at your *.jpg it's clearly a Tensorflow (freeze_graph.py) error. In fact I found this Stack Overflow link about Tensorflow Assign error about the very same error you're getting.

This is the Model Optimizer Documentation for Tensorflow Slim . It looks like you followed the steps in this document, based on what I see in your post. But in your steps, you decided to use freeze_graph.py while the MO document does not recommend this step. Instead it jumps immediately to (keep in mind it's a different model than yours):

<MODEL_OPTIMIZER_INSTALL_DIR>/mo_tf.py --input_model ./inception_v1_inference_graph.pb --input_checkpoint ./inception_v1.ckpt -b 1 --mean_value [127.5,127.5,127.5] --scale 127.5

Hope it helps,

Thanks,

Shubha

Bin_X_Intel · ‎09-29-2019

Hi Shubha, I run tf_mo.py directly as you pointed out, but still encounter similar issue, are you able to duplicate this problem?

[root@localhost 00464629]# python3.6 /opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo_tf.py --input_model ./resnet_v1_50_inference_graph.pb --input_checkpoint ./resnet_v1_50.ckpt -b 1 --mean_value [127.5,127.5,127.5] --scale 127.5
Model Optimizer arguments:
Common parameters:
   - Path to the Input Model:    /home/xiebin/00464629/./resnet_v1_50_inference_graph.pb
   - Path for generated IR:    /home/xiebin/00464629/.
   - IR output name:    resnet_v1_50_inference_graph
   - Log level:    ERROR
   - Batch:    1
   - Input layers:    Not specified, inherited from the model
   - Output layers:    Not specified, inherited from the model
   - Input shapes:    Not specified, inherited from the model
   - Mean values:    [127.5,127.5,127.5]
   - Scale values:    Not specified
   - Scale factor:    127.5
   - Precision of IR:    FP32
   - Enable fusing:    True
   - Enable grouped convolutions fusing:    True
   - Move mean values to preprocess section:    False
   - Reverse input channels:    False
TensorFlow specific parameters:
   - Input model in text protobuf format:    False
   - Path to model dump for TensorBoard:    None
   - List of shared libraries with TensorFlow custom layers implementation:    None
   - Update the configuration file with input/output node names:    None
   - Use configuration file used to generate the model with Object Detection API:    None
   - Operations to offload:    None
   - Patterns to offload:    None
   - Use the config file:    None
Model Optimizer version:    2019.2.0-436-gf5827d4
[ FRAMEWORK ERROR ] Cannot load input model: Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
   [[Node: save/Assign_265 = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/biases, save/RestoreV2:265)]]

Caused by op 'save/Assign_265', defined at:
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo_tf.py", line 31, in <module>
sys.exit(main(get_tf_cli_parser(), 'tf'))
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/main.py", line 302, in main
return driver(argv)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/main.py", line 251, in driver
is_binary=not argv.input_model_is_text)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 82, in tf2nx
saved_model_tags=argv.saved_model_tags)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 206, in load_tf_graph_def
graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 103, in freeze_checkpoint
tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
self.build()
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1296, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
restore_sequentially, reshape)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 113, in restore
self.op.get_shape().is_fully_defined())
File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 219, in assign
validate_shape=validate_shape)
File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
use_locking=use_locking, name=name)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
[[Node: save/Assign_265 = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/biases, save/RestoreV2:265)]]

Shubha_R_Intel · ‎09-30-2019

Dear Bin X.

Thanks for posting the detailed exception log. But it's clear that it's still not a Model Optimizer issue. Please take a look. Model Optimizer is simply passing Tensorflow's error along:

File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 206, in load_tf_graph_def
graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 103, in freeze_checkpoint
tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
self.build()
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1296, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
restore_sequentially, reshape)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))

Hope it helps. It looks like your Tensorflow Model is encountering difficulties getting frozen. Why that is, I don't know.

Thanks,

Shubha

yu__jia · ‎11-06-2019

Hi Bin X.

Can you solve this problem? I have the same problem as you. If you have already solved it,can you share how you solved it.

Thanks,

Yujia