topic Dear Dongjie S, in Intel® Distribution of OpenVINO™ Toolkit

shape mismatch when optimize resnet_v1_50.pb

Dongjie_S_Intel — Thu, 04 Apr 2019 02:55:39 GMT

Hi,

I downloaded resnet_v1_50_2016_08_28.tar.gz from http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz, and use dldt 2018_R5 to optimize it. Get shape mismatch error. Please help.

Commands:

python3 tf_models/research/slim/export_inference_graph.py \
--model_name resnet_v1_50 \
--output_file ov_irs/resnet_v1_50.pb

python3 ~/dldt/model-optimizer/mo/utils/summarize_graph.py --input_model ov_irs/resnet_v1_50.pb

wget http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz
tar xvf resnet_v1_50_2016_08_28.tar.gz -C ov_irs

python3 ~/dldt/model-optimizer/mo_tf.py --input_model ./ov_irs/resnet_v1_50.pb --input_checkpoint ./ov_irs/resnet_v1_50.ckpt -b 1 --mean_value [103.94,116.78,123.68] --scale 1

output:

Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /home/glorysdj/./ov_irs/resnet_v1_50.pb
- Path for generated IR: /home/glorysdj/.
- IR output name: resnet_v1_50
- Log level: ERROR
- Batch: 1
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: Not specified, inherited from the model
- Mean values: [103.94,116.78,123.68]
- Scale values: Not specified
- Scale factor: 1.0
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: False
TensorFlow specific parameters:
- Input model in text protobuf format: False
- Offload unsupported operations: False
- Path to model dump for TensorBoard: None
- List of shared libraries with TensorFlow custom layers implementation: None
- Update the configuration file with input/output node names: None
- Use configuration file used to generate the model with Object Detection API: None
- Operations to offload: None
- Patterns to offload: None
- Use the config file: None
Model Optimizer version: 1.5.12.49d067a0
[ FRAMEWORK ERROR ] Cannot load input model: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]
[[node save/Assign_266 (defined at /home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py:105) = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/weights, save/RestoreV2:266)]]

Caused by op 'save/Assign_266', defined at:
File "/home/glorysdj/dldt/model-optimizer/mo_tf.py", line 31, in <module>
sys.exit(main(get_tf_cli_parser(), 'tf'))
File "/home/glorysdj/dldt/model-optimizer/mo/main.py", line 325, in main
return driver(argv)
File "/home/glorysdj/dldt/model-optimizer/mo/main.py", line 267, in driver
mean_scale_values=mean_scale)
File "/home/glorysdj/dldt/model-optimizer/mo/pipeline/tf.py", line 127, in tf2nx
saved_model_tags=argv.saved_model_tags)
File "/home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py", line 208, in load_tf_graph_def
graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
File "/home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py", line 105, in freeze_checkpoint
tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1102, in __init__
self.build()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 119, in restore
self.op.get_shape().is_fully_defined())
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
validate_shape=validate_shape)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Dear Dongjie S,

Shubha_R_Intel — Thu, 04 Apr 2019 21:36:36 GMT

Dear Dongjie S,

Please make sure your Tensorflow model is a frozen protobuf first before you use Model Optimizer. Below are instructions on using Model Optimizer to convert a Tensorflow model:

https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html

Also you are using an older version of OpenVino (though not too old). I strongly encourage you to download the latest 2019 R1 release.

Thanks !

Shubha

Dear Dongjie S,

Dan_L_Intel — Fri, 02 Aug 2019 08:31:56 GMT

Dear Dongjie S,

I encourted exactly the same error as you described "Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]" when modifying resnet_v1_50.pb model using OpenVINO 2019 R1.

I also downloaded resnet_v1_50_2016_08_28.tar.gz from http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz, from where our guide https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html suggests.

So did you solve this issue? Looking forward to your reply.

Thanks,

Una

Dear everyone,

Shubha_R_Intel — Fri, 02 Aug 2019 20:54:52 GMT

Dear everyone,

By now we have 2019R2. In fact it was just released last week. Please update to the latest release and try again.

Thanks,

Shubha

Hi Shubha, I'm using 2019R2,

Bin_X_Intel — Mon, 23 Sep 2019 03:33:00 GMT

Hi Shubha, I'm using 2019R2, I still suffer same issue, any idea how to solve this? Thanks

Dear Bin X,

Shubha_R_Intel — Tue, 24 Sep 2019 22:32:56 GMT

Dear Bin X,

Can you kindly be more explicit about your error ? This thread is pretty old. Is it a mismatch error using Model Optimizer ? And are you using the resnet_v1_50 model just like the Original Poster ? What is your exact model optimizer command ?

Thanks,

Shubha

Hi Shubha, I tried 2019R2 and

Bin_X_Intel — Wed, 25 Sep 2019 08:24:36 GMT

Hi Shubha, I tried 2019R2 and internal 2019R3, all fail with same issue as Dongjie mentioned:

1. wget http://download.tensorflow.org/models/ resnet_v1_50_2016_08_28.tar.gz

2. tar xvf resnet_v1_50_2016_08_28.tar.gz

3. python3 research/slim/export_inference_graph.py --model_name resnet_v1_50 --output_file resnet_v1_50_inference_graph.pb

4. python3 /opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/utils/summarize_graph.py --input_model ./resnet_v1_50_inference_graph.pb

5. python3 /usr/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py --input_graph ./resnet_v1_50_inference_graph.pb --input_checkpoint resnet_v1_50.ckpt --input_binary=true --output_graph frozen_resnet_v1_50.pb --output_node_names=resnet_v1_50/predictions/Reshape_1

then error showed as attachment

Dear Bin X.,

Shubha_R_Intel — Fri, 27 Sep 2019 19:54:45 GMT

Dear Bin X.,

Why do you think that this is an OpenVino problem ? In looking at your *.jpg it's clearly a Tensorflow (freeze_graph.py) error. In fact I found this Stack Overflow link about Tensorflow Assign error about the very same error you're getting.

This is the Model Optimizer Documentation for Tensorflow Slim . It looks like you followed the steps in this document, based on what I see in your post. But in your steps, you decided to use freeze_graph.py while the MO document does not recommend this step. Instead it jumps immediately to (keep in mind it's a different model than yours):

<MODEL_OPTIMIZER_INSTALL_DIR>/mo_tf.py --input_model ./inception_v1_inference_graph.pb --input_checkpoint ./inception_v1.ckpt -b 1 --mean_value [127.5,127.5,127.5] --scale 127.5

Hope it helps,

Thanks,

Shubha

Hi Shubha, I run tf_mo.py

Bin_X_Intel — Mon, 30 Sep 2019 03:01:34 GMT

Hi Shubha, I run tf_mo.py directly as you pointed out, but still encounter similar issue, are you able to duplicate this problem?

[root@localhost 00464629]# python3.6 /opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo_tf.py --input_model ./resnet_v1_50_inference_graph.pb --input_checkpoint ./resnet_v1_50.ckpt -b 1 --mean_value [127.5,127.5,127.5] --scale 127.5
Model Optimizer arguments:
Common parameters:
   - Path to the Input Model:    /home/xiebin/00464629/./resnet_v1_50_inference_graph.pb
   - Path for generated IR:    /home/xiebin/00464629/.
   - IR output name:    resnet_v1_50_inference_graph
   - Log level:    ERROR
   - Batch:    1
   - Input layers:    Not specified, inherited from the model
   - Output layers:    Not specified, inherited from the model
   - Input shapes:    Not specified, inherited from the model
   - Mean values:    [127.5,127.5,127.5]
   - Scale values:    Not specified
   - Scale factor:    127.5
   - Precision of IR:    FP32
   - Enable fusing:    True
   - Enable grouped convolutions fusing:    True
   - Move mean values to preprocess section:    False
   - Reverse input channels:    False
TensorFlow specific parameters:
   - Input model in text protobuf format:    False
   - Path to model dump for TensorBoard:    None
   - List of shared libraries with TensorFlow custom layers implementation:    None
   - Update the configuration file with input/output node names:    None
   - Use configuration file used to generate the model with Object Detection API:    None
   - Operations to offload:    None
   - Patterns to offload:    None
   - Use the config file:    None
Model Optimizer version:    2019.2.0-436-gf5827d4
[ FRAMEWORK ERROR ] Cannot load input model: Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
   [[Node: save/Assign_265 = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/biases, save/RestoreV2:265)]]

Caused by op 'save/Assign_265', defined at:
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo_tf.py", line 31, in <module>
sys.exit(main(get_tf_cli_parser(), 'tf'))
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/main.py", line 302, in main
return driver(argv)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/main.py", line 251, in driver
is_binary=not argv.input_model_is_text)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 82, in tf2nx
saved_model_tags=argv.saved_model_tags)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 206, in load_tf_graph_def
graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 103, in freeze_checkpoint
tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
self.build()
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1296, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
restore_sequentially, reshape)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 113, in restore
self.op.get_shape().is_fully_defined())
File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 219, in assign
validate_shape=validate_shape)
File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
use_locking=use_locking, name=name)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
[[Node: save/Assign_265 = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/biases, save/RestoreV2:265)]]

Dear Bin X.

Shubha_R_Intel — Mon, 30 Sep 2019 19:12:00 GMT

Dear Bin X.

Thanks for posting the detailed exception log. But it's clear that it's still not a Model Optimizer issue. Please take a look. Model Optimizer is simply passing Tensorflow's error along:

File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 206, in load_tf_graph_def
graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 103, in freeze_checkpoint
tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
self.build()
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1296, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
restore_sequentially, reshape)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))

Hope it helps. It looks like your Tensorflow Model is encountering difficulties getting frozen. Why that is, I don't know.

Thanks,

Shubha

Hi Bin X.

yu__jia — Thu, 07 Nov 2019 01:53:47 GMT

Hi Bin X.

Can you solve this problem? I have the same problem as you. If you have already solved it,can you share how you solved it.

Thanks,

Yujia