Can't convert mobilenetV2-ssdlite tensorflow model to IR

[imazaike]

Hello!

I trained a mobilenetV2-ssdlite model using my own made dataset and want to put it on a NCS2.

I have tried to convert the graph using the following command:

python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py 
--input_model saved_to_frozen.pb 
--tensorflow_object_detection_api_pipeline_config pipeline.config 
--data_type FP16

The command above returned the following output:

Model Optimizer arguments:
Common parameters:
	- Path to the Input Model: 	/home/imazaike/saved_to_frozen.pb
	- Path for generated IR: 	/home/imazaike/.
	- IR output name: 	saved_to_frozen
	- Log level: 	ERROR
	- Batch: 	Not specified, inherited from the model
	- Input layers: 	Not specified, inherited from the model
	- Output layers: 	Not specified, inherited from the model
	- Input shapes: 	Not specified, inherited from the model
	- Mean values: 	Not specified
	- Scale values: 	Not specified
	- Scale factor: 	Not specified
	- Precision of IR: 	FP16
	- Enable fusing: 	True
	- Enable grouped convolutions fusing: 	True
	- Move mean values to preprocess section: 	False
	- Reverse input channels: 	False
TensorFlow specific parameters:
	- Input model in text protobuf format: 	False
	- Path to model dump for TensorBoard: 	None
	- List of shared libraries with TensorFlow custom layers implementation: 	None
	- Update the configuration file with input/output node names: 	None
	- Use configuration file used to generate the model with Object Detection API: 	/home/imazaike/pipeline.config
	- Use the config file: 	None
Model Optimizer version: 	
/home/imazaike/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/imazaike/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/imazaike/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/imazaike/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/imazaike/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/imazaike/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/imazaike/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/imazaike/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/imazaike/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/imazaike/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/imazaike/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/imazaike/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/imazaike/anaconda3/envs/tf/lib/python3.6/site-packages/dask/dataframe/utils.py:13: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm
[ ANALYSIS INFO ]  It looks like there is IteratorGetNext as input
Run the Model Optimizer with:
		--input "IteratorGetNext:1[16 300 300 3],IteratorGetNext:5[16 100 4],IteratorGetNext:6[16 100 2],IteratorGetNext:12[16 100],IteratorGetNext:13[16]"
And replace all negative values with positive values
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.front.output_cut.OutputCut'>): Graph contains 0 node after executing <class 'extensions.front.output_cut.OutputCut'>. It considered as error because resulting IR will be empty which is not usual

So I tried the following to resolve the error.

1. change the tensorflow-gpu version to 1.14.0

2. add --tensorflow_use_custom_operations_config /opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json

3. recreated frozen file with following command.

python3 tensorflow/python/tools/freeze_graph.py \
--input_meta_graph model.ckpt-1000000.meta \
--output_node_names "Merge/MergeSummary" \
--output_graph saved_model.pb \
--input_checkpoint model.ckpt-1000000 \
--input_binary=true \
--output_graph=saved_to_frozen.pb

But none of them worked. Is my pbfile corrupted?

I would appreciate if anyone helped to solve this issue.

Thanks for your answers.

Imazaike

[samontab]

Did you try the suggestion shown?

[ ANALYSIS INFO ]  It looks like there is IteratorGetNext as input
Run the Model Optimizer with:
		--input "IteratorGetNext:1[16 300 300 3],IteratorGetNext:5[16 100 4],IteratorGetNext:6[16 100 2],IteratorGetNext:12[16 100],IteratorGetNext:13[16]"
And replace all negative values with positive values

[imazaike]

Thank you for your quick response.

Yes. I already tried. Below is the output.

[ ERROR ]  Cannot infer shapes or values for node "gradients/Loss/stack_76_grad/tuple/group_deps".
[ ERROR ]  "The name 'gradients/Loss/stack_76_grad/tuple/group_deps:0' refers to a Tensor which does not exist. The operation, 'gradients/Loss/stack_76_grad/tuple/group_deps', exists but only has 0 outputs."
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <function tf_native_tf_node_infer at 0x7f1c286d7378>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ANALYSIS INFO ]  It looks like there is IteratorGetNext as input
Run the Model Optimizer with:
		--input "IteratorGetNext:1[16 300 300 3],IteratorGetNext:5[16 100 4],IteratorGetNext:6[16 100 2],IteratorGetNext:12[16 100],IteratorGetNext:13[16]"
And replace all negative values with positive values
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "gradients/Loss/stack_76_grad/tuple/group_deps" node. 
 For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38. 

I referred to Model Optimizer FAQ #38 but it didn't solve the problem.

What does "replace all negative values with positive values" mean?

Thanks again

Imazaike

[imazaike]

Below is my environment

openvino : 2020 R3

tensorflow-gpu : 1.14.0

Ubuntu : 18.04

[Sahira_Intel]

Hi Imazaike,

I am currently looking into this error and will let you know when I find something.

Sincerely,

Sahira 

[imazaike]

Hi Sahira,

Sorry for late reply.

I haven't been able to resolve the error yet. So, I am looking forward to your reply.

Best regards,

Imazaike

[Sahira_Intel]

Hi,

I am running into some errors when trying to convert this model as well.

Was the model trained with TF API 1.14? The TF versions must match.

When trying to freeze the model, I got an error involving the 'ParallelInterleaveDataset' which is an unsupported operation. How did you work around this error? 

Here is the command used to freeze the model:

python3 tensorflow/python/tools/freeze_graph.py \
--input_meta_graph model.ckpt-1000000.meta \
--output_node_names "Merge/MergeSummary" \
--output_graph saved_model.pb \
--input_checkpoint model.ckpt-1000000 \
--input_binary=true \
--output_graph=saved_to_frozen.pb

Sincerely,

Sahira 

[imazaike]

Hi,

Thanks for your reply.

Was the model trained with TF API 1.14? The TF versions must match.

> I don't remember because I changed the TF version. So, I will match the versions and then train again. And I will report the result.

When trying to freeze the model, I got an error involving the 'ParallelInterleaveDataset' which is an unsupported operation. How did you work around this error? 

> I've never faced that error, so I'm not sure. Sorry.

Sincerely,

Imazaike

[imazaike]

Hi,

I changed all tensorflow versions to 1.15.2 and retrained. Then, I execute mo_tf.py. But I got the same error.

Is "Merge/MergeSummary" the wrong output node name?

Thank you for all your help.

Imazaike

[Sahira_Intel]

Hi,

It looks like your input and output nodes are not specified, which could be causing the error.

Can you please provide the output after you run the following command:

python3 tensorflow/python/tools/freeze_graph.py \

--input_meta_graph model.ckpt-1000000.meta \

--output_node_names "Merge/MergeSummary" \

--output_graph saved_model.pb \

--input_checkpoint model.ckpt-1000000 \

--input_binary=true \

--output_graph=saved_to_frozen.pb

Thank you,

Sahira 

[imazaike]

Hi,

The following is the result of executing the command.

root@a4cd4b9b19f3:/tensor-15/tensorflow# python3 tensorflow/python/tools/freeze_graph.py --input_meta_graph model.ckpt-1000000.meta --output_node_names "Merge/MergeSummary" --output_graph saved_model.pb --input_checkpoint model.ckpt-1000000 --input_binary=true --output_graph=saved_to_frozen.pb
2020-08-05 05:29:35.977614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loaded meta graph file 'model.ckpt-1000000.meta
WARNING:tensorflow:From tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0805 05:29:36.841789 140091924260672 deprecation.py:323] From tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2020-08-05 05:29:37.006440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-08-05 05:29:37.691574: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:37.692713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.695
pciBusID: 0000:01:00.0
2020-08-05 05:29:37.692800: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-05 05:29:38.481283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-05 05:29:38.568483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-05 05:29:38.632295: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-05 05:29:38.779205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-05 05:29:38.893226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-05 05:29:39.327638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-05 05:29:39.327986: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:39.329234: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:39.330304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-08-05 05:29:39.332941: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:39.334070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.695
pciBusID: 0000:01:00.0
2020-08-05 05:29:39.334154: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-05 05:29:39.334222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-05 05:29:39.334277: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-05 05:29:39.334334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-05 05:29:39.334390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-05 05:29:39.334444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-05 05:29:39.334499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-05 05:29:39.334682: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:39.335836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:39.336853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-08-05 05:29:39.348477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-05 05:29:49.660593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-05 05:29:49.660680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-08-05 05:29:49.660717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-08-05 05:29:49.703693: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:49.704922: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:49.706105: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-05 05:29:49.707175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4962 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from model.ckpt-1000000
I0805 05:29:53.008160 140091924260672 saver.py:1284] Restoring parameters from model.ckpt-1000000
WARNING:tensorflow:From tensorflow/python/tools/freeze_graph.py:226: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
W0805 05:29:54.511643 140091924260672 deprecation.py:323] From tensorflow/python/tools/freeze_graph.py:226: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
W0805 05:29:54.511796 140091924260672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
INFO:tensorflow:Froze 405 variables.
I0805 05:29:55.304905 140091924260672 graph_util_impl.py:334] Froze 405 variables.
INFO:tensorflow:Converted 405 variables to const ops.
I0805 05:29:55.481544 140091924260672 graph_util_impl.py:394] Converted 405 variables to const ops.

Sincerely,

Imazaike

[Max_L_Intel]

Hi @imazaike 

We are not able to freeze your inference graph most likely due to environment specifics (stream_executor error).

/opt/intel/openvino_2020.3.194/deployment_tools/tools/model_downloader$ python3 /home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py --input_meta_graph ~/Downloads/mobilenetv2_ssdlite_openvino/model.ckpt-1000000.meta --output_node_names "Merge/MergeSummary" --output_graph model_frozen.pb --input_checkpoint ~/Downloads/mobilenetv2_ssdlite_openvino/model.ckpt-1000000 --input_binary=true --output_graph=saved_to_frozen.pb
Loaded meta graph file Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0731 18:43:22.437623 140412314171200 deprecation.py:323] From /home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2020-07-31 18:43:22.459599: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-07-31 18:43:22.480259: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 1999965000 Hz
2020-07-31 18:43:22.480813: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x9828150 executing computations on platform Host. Devices:
2020-07-31 18:43:22.480863: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 491, in <module>
    run_main()
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 487, in run_main
    app.run(main=my_main, argv=[sys.argv[0]] + unparsed)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 486, in <lambda>
    my_main = lambda unused_args: main(unused_args, flags)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 378, in main
    flags.saved_model_tags, checkpoint_version)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 361, in freeze_graph
    checkpoint_version=checkpoint_version)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 154, in freeze_graph_with_def_protos
    input_meta_graph_def, clear_devices=True)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1449, in import_meta_graph
    **kwargs)[0]
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1473, in _import_meta_graph_with_return_elements
    **kwargs))
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 857, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 400, in import_graph_def
    _RemoveDefaultAttrs(op_dict, producer_op_list, graph_def)
  File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 160, in _RemoveDefaultAttrs
    op_def = op_dict[node.op]
KeyError: 'ParallelInterleaveDataset'

Meanwhile we are able to successfully convert Open Model Zoo version of mobilenetV2-ssdlite TensorFlow frozen model with the following recommended command:

/usr/bin/python3 -- /opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP16 --output_dir=/opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/public/ssdlite_mobilenet_v2/FP16 --model_name=ssdlite_mobilenet_v2 --reverse_input_channels '--input_shape=[1,300,300,3]' --input=image_tensor --output=detection_scores,detection_boxes,num_detections --transformations_config=/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json --tensorflow_object_detection_api_pipeline_config=/opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/public/ssdlite_mobilenet_v2/ssdlite_mobilenet_v2_coco_2018_05_09/pipeline.config --input_model=/opt/intel/openvino_2020.3.194/deployment_tools/open_model_zoo/tools/downloader/public/ssdlite_mobilenet_v2/ssdlite_mobilenet_v2_coco_2018_05_09/frozen_inference_graph.pb

However, that does not work for your model that was frozen manually. None of these nodes were found in your model: image_tensor, detection_scores, detection_boxes, num_detections.
So I think you should try to freeze your model with specifying input and output nodes. As the output node names you could try the same detection_scores, detection_boxes, num_detections instead of Merge/MergeSummary, and also specify image_tensor as input node. And then please try that Model Optimizer example command that provided above. And since this is manually trained model, you also need to replace ssd_v2_support.json with ssd_support_api_v1.14.json or ssd_support_api_v1.15.json depending on TF API version used.

The output we got during Open Model Zoo mobilenetV2-ssdlite model conversion is:

- IR output name: 	ssdlite_mobilenet_v2
- Log level: 	ERROR
- Batch: 	Not specified, inherited from the model
- Input layers: 	image_tensor
- Output layers: 	detection_scores,detection_boxes,num_detections
- Input shapes: 	[1,300,300,3]

While for your model there're no input shapes, no input and output layers specified:

- IR output name: 	saved_to_frozen
- Log level: 	ERROR
- Batch: 	Not specified, inherited from the model
- Input layers: 	Not specified, inherited from the model
- Output layers: 	Not specified, inherited from the model
- Input shapes: 	Not specified, inherited from the model

[imazaike]

Hi @Max_L_Intel 

sorry for late reply.

We are not able to freeze your inference graph most likely due to environment specifics (stream_executor error).

-> Am I getting this error because I trained using the voc dataset?

So I think you should try to freeze your model with specifying input and output nodes.

->Could you give me the correct command to freeze the graph by specifying an output node?

I tried to freeze the graph with following command but error occurred.

root@a4cd4b9b19f3:/tensor-15/tensorflow# python3 tensorflow/python/tools/freeze_graph.py --input_meta_graph model.ckpt-1000000.meta --output_node_names "detection_scores,detection_boxes,num_detections" --output_graph saved_model.pb --input_checkpoint model.ckpt-1000000 --input_binary=true
2020-08-19 04:16:00.225687: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loaded meta graph file 'model.ckpt-1000000.meta
~~~~~~~~~
abridgement
~~~~~~~~~
2020-08-19 04:16:01.808963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4928 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from model.ckpt-1000000
I0819 04:16:04.744784 140517555464000 saver.py:1284] Restoring parameters from model.ckpt-1000000
WARNING:tensorflow:From tensorflow/python/tools/freeze_graph.py:226: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
W0819 04:16:06.170308 140517555464000 deprecation.py:323] From tensorflow/python/tools/freeze_graph.py:226: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
W0819 04:16:06.170460 140517555464000 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
Traceback (most recent call last):
  File "tensorflow/python/tools/freeze_graph.py", line 491, in <module>
    run_main()
  File "tensorflow/python/tools/freeze_graph.py", line 487, in run_main
    app.run(main=my_main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "tensorflow/python/tools/freeze_graph.py", line 486, in <lambda>
    my_main = lambda unused_args: main(unused_args, flags)
  File "tensorflow/python/tools/freeze_graph.py", line 378, in main
    flags.saved_model_tags, checkpoint_version)
  File "tensorflow/python/tools/freeze_graph.py", line 361, in freeze_graph
    checkpoint_version=checkpoint_version)
  File "tensorflow/python/tools/freeze_graph.py", line 226, in freeze_graph_with_def_protos
    variable_names_blacklist=variable_names_blacklist)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py", line 277, in convert_variables_to_constants
    inference_graph = extract_sub_graph(input_graph_def, output_node_names)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py", line 197, in extract_sub_graph
    _assert_nodes_are_present(name_to_node, dest_nodes)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py", line 152, in _assert_nodes_are_present
    assert d in name_to_node, "%s is not in graph" % d
AssertionError: detection_scores is not in graph

I can't think of any other solution. so.... It's a maybe bad idea, but I'll create and train a coco-format dataset.

[Max_L_Intel · Accepted solution]

Hi @imazaike 

Verified SSD Lite MobileNet V2 model's version from Open Model Zoo has been trained on COCO dataset, but I don't think this might be related.

For the correct command you need to identify the nodes for your model.
Please refer to export_inference_graph.py method to freeze the model https://stackoverflow.com/questions/45017356/converting-ssd-to-frozen-graph-in-tensorflow-which-output-node-names-must-be-us

And then refer to summarize_graph for inspecting the model's nodes https://github.com/tensorflow/models/issues/2623

Hope this helps.

[Sahira_Intel]

Hi Imazaike,

As we have not received a response from you, this thread will no longer be monitored. If you have any additional questions, please submit a new post.

Sincerely,

Sahira