- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
While running mo.py script with a tensorflow protobuf file, the following error occurred, with suggestion to forward to MO devs.
[ ERROR ] -------------------------------------------------
[ ERROR ] ----------------- INTERNAL ERROR ----------------
[ ERROR ] Unexpected exception happened.
[ ERROR ] Please contact Model Optimizer developers and forward the following information:
[ ERROR ] Graph contains a cycle.
[ ERROR ] Traceback (most recent call last):
File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/main.py", line 222, in main
return driver(argv)
File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/main.py", line 190, in driver
mean_scale_values=mean_scale)
File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 143, in tf2nx
partial_infer(graph)
File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/middle/passes/infer.py", line 55, in partial_infer
nodes = nx.topological_sort(graph)
File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/venv/lib/python3.5/site-packages/networkx/algorithms/dag.py", line 157, in topological_sort
raise nx.NetworkXUnfeasible("Graph contains a cycle.")
networkx.exception.NetworkXUnfeasible: Graph contains a cycle.
[ ERROR ] ---------------- END OF BUG REPORT --------------
[ ERROR ] -------------------------------------------------
Model Optimizer arguments
Batch: 1
Precision of IR: FP32
Enable fusing: True
Enable gfusing: True
Names of input layers: inherited from the model
Path to the Input Model: matroid.pb
Input shapes: inherited from the model
Log level: ERROR
Mean values: ()
IR output name: inherited from the model
Names of output layers: inherited from the model
Path for generated IR: /opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer
Reverse input channels: False
Scale factor: None
Scale values: ()
Version: 0.3.61.37271eb9
Input model in text protobuf format: False
Offload unsupported operations: False
Path to model dump for TensorBoard: None
Update the configuration file with input/output node names: None
Operations to offload: None
Patterns to offload: None
Use the config file: None
I'm not able to find anything in the documentation about how to handle cycles within the graph, but imagine it may require offloading into TensorFlow. Any information regarding this would be extremely helpful. Thanks!
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Done. I have forwarded this to the Model Optimizer Devs. Thank You Mark.
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Mark I have contacted Intel Developers and they would like you to share a freezed model. Can you kindly attach that ?
Thanks,
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
This model is actually proprietary from one of our ISVs, and I would not want to send it to a public forum. Is there an email address I could forward a copy of this model to to have it looked at? Thanks.
Mark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
In few days ago, I tried to convert TensorFlow models: SSD MobileNet and SSD InceptionV2 via MO, but encountered similar issue. Then I followed direction in "C:/Intel/computer_vision_sdk_2018.1.249/deployment_tools/documentation/docs/TensorFlowObjectDetectionSSD.html", and this issue is resolved.
Take SSD MobileNet for example, you may need use mo_tf.py to covert it, like
python mo_tf.py --input_model "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\ssd_mobilenet_v1_coco_2017_11_17\frozen_inference_graph.pb" --input=1:Preprocessor/mul --input_shape="(1,300,300,3)" --tensorflow_use_custom_operations_config extensions/front/tf/ssd_support.json --output="detection_boxes,detection_scores,num_detections" --output_dir "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\intel_models\ssd_mobilenet_v1_coco_2017_11_17"
I'm still not familiar with this Python module, and above just a trail-and-error result.
Hope it would be helpful.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Based on symptoms, this is a graph with control flow that may or may not be an essential part of the model. Cycles in graphs are supported for predefined number of models – SSDs built with TF Detection API – where they are just eliminated. In general case cycles are not supported. Common recipe: the graph should be analyzed (for example in TensorBoard) to identify straightforward convolutional parts that consist of supported TF operations without cycles. These parts should be converted to IR. All other parts should be done as pre-/post-processing outside IE IR.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Sergey
Could you please explain this a little more?
1. Why is the control flow cyclic graph not an essential part of the model?
2. Could you please give an example of where the SSD graph could have a cycle and how eliminating it is ok from an IR perspective?
3. Is it very hard to visualize graph cycles in tensorboard so if you can throw some light on where in the seq2seq model, there could be a potential graph cycle, that will help a lot
Regards
Deepa
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi There,
I am stuck at the same issue. I did train an SSD InceptionV2 with tensorflow, afterwards freezing up the net, now having my frozen.pb net.
I want it to be optimized for the AWS deeplens - basically the wrapper around the intel model optimizer is a little sub-optimal, so you do end up with searching a lot for the actual error.
[ERROR]-mo.py:160,[ WARNING ]
Detected not satisfied dependencies:
numpy: installed: 1.14.5, required: 1.13.0
Please install required versions of components or use install_prerequisites script
/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites_tf.sh
Note that install_prerequisites scripts may install additional components.
[ ERROR ] -------------------------------------------------
[ ERROR ] ----------------- INTERNAL ERROR ----------------
[ ERROR ] Unexpected exception happened.
[ ERROR ] Please contact Model Optimizer developers and forward the following information:
[ ERROR ] Graph contains a cycle.
[ ERROR ] Traceback (most recent call last):
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/main.py", line 222, in main
return driver(argv)
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/main.py", line 190, in driver
mean_scale_values=mean_scale)
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 143, in tf2nx
partial_infer(graph)
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/middle/passes/infer.py", line 55, in partial_infer
nodes = nx.topological_sort(graph)
File "/usr/local/lib/python3.5/dist-packages/networkx/algorithms/dag.py", line 157, in topological_sort
raise nx.NetworkXUnfeasible("Graph contains a cycle.")
networkx.exception.NetworkXUnfeasible: Graph contains a cycle.
[ ERROR ] ---------------- END OF BUG REPORT --------------
[ ERROR ] -------------------------------------------------
The model optimizer is called through the mo wrapper method - https://docs.aws.amazon.com/deeplens/latest/dg/deeplens-model-optimizer-api-functions_and_objects.html
I would optimize the model manually, but the solution above ...
python mo_tf.py --input_model "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\ssd_mobilenet_v1_coco_2017_11_17\frozen_inference_graph.pb" --input=1:Preprocessor/mul --input_shape="(1,300,300,3)" --tensorflow_use_custom_operations_config extensions/front/tf/ssd_support.json --output="detection_boxes,detection_scores,num_detections" --output_dir "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\intel_models\ssd_mobilenet_v1_coco_2017_11_17" |
...does not translate to my problem
Applying the call (without input shape) leads to "Sub-graph contains network input node "image_tensor". For more ... FAQ75
Specifying the input shape of (1,224,224,3) leads to "Other inputs needed for output computation" 27
In the first place I would like to get rid of the circle without bumping in substituion problems.
Any ideas?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I finally solved the problem, which was occuring on AWS Deeplens with the deployed version of the model optimizer (awscam.mo). It occured due to the fact that an old Intel Open Vino version was deployed & the underlying net didn't have the right version. I documented the solution in my blog article that I was writing ...
https://blog.codecentric.de/2018/07/deep-learning-diesel-aws-deeplens/
It's in German, an English version will follow later. Regards, Kai
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I ran into the same issue here when running mo_tf.py on my frozen.pb. I am suspecting that the cycle issue is rooted in tf.while_loop(). Do you know if MO is able to handle dynamic RNN layers?
Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Richard,
Currently MO doesn't support dynamic RNN layers or RNN networks but I will create a feature request for this.
Kind Regards,
Monique Jones
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have updated the description how I finally managed to work around the issue. Hope this helps: https://blog.codecentric.de/en/2018/11/aws-deep-lens-diesel-car-detection/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In case I wish to fix the graph using the Point 97 in the Mo_FAQ.html document in the VINO documentation docs,
which nodes do I include in the first command -
python3 mo.py --input_model model/frozen_inference_graph.pb --tensorflow_subgraph_pattern ""FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/BatchNorm/FusedBatchNorm, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/Relu6, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_depthwise/depthwise, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_depthwise/BatchNorm/FusedBatchNorm, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_depthwise/Relu6,..
Which of the nodes do I put above to offload a sub-graph of operations?
(Actual .pbtxt file had about 100 nodes)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Monique, can you please confirm that MO still doesn't support RNNs? We're getting "Graph contains a cycle" when trying to optimize a GRU network.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Matsak, Art,
OpenVino does support several RNN-like layers such as GRU, LSTM, and RNN sequence layers.
If you peruse the Release Notes and do a search on RNN you will find out exactly what OpenVino supports.
May I know more about the model you're using ?
Thanks,
Shubha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Shubha, thank you. We're using a bi-directional GRU recurrent network. The model is trained with Keras/TensorFlow and we're trying to convert it with mo_tf.py. However, we're running into the "Graph contains a cycle" error. Hence my question regarding mo_tf.py (not OpenVINO itself) not supporting RNN conversion yet. Please let me know.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Shubha, thank you for your comments. The problem seems to be not in OpenVINO itself but in the mo_tf.py script. We use it to optimize/convert a TensorFlow bi-directional GRU model, and it fails with the "Graph contains a cycle" error. Hence my question regarding MO (not OpenVINO) not supporting RNNs yet.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page