[Bug Report] Exception: Graph Contains a Cycle

[Mark_A_Intel]

While running mo.py script with a tensorflow protobuf file, the following error occurred, with suggestion to forward to MO devs.

[ ERROR ]  -------------------------------------------------
[ ERROR ]  ----------------- INTERNAL ERROR ----------------
[ ERROR ]  Unexpected exception happened.
[ ERROR ]  Please contact Model Optimizer developers and forward the following information:
[ ERROR ]  Graph contains a cycle.
[ ERROR ]  Traceback (most recent call last):
  File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/main.py", line 222, in main
    return driver(argv)
  File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/main.py", line 190, in driver
    mean_scale_values=mean_scale)
  File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 143, in tf2nx
    partial_infer(graph)
  File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/mo/middle/passes/infer.py", line 55, in partial_infer
    nodes = nx.topological_sort(graph)
  File "/opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer/venv/lib/python3.5/site-packages/networkx/algorithms/dag.py", line 157, in topological_sort
    raise nx.NetworkXUnfeasible("Graph contains a cycle.")
networkx.exception.NetworkXUnfeasible: Graph contains a cycle.

[ ERROR ]  ---------------- END OF BUG REPORT --------------
[ ERROR ]  -------------------------------------------------
Model Optimizer arguments
    Batch:     1
    Precision of IR:     FP32
    Enable fusing:     True
    Enable gfusing:     True
    Names of input layers:     inherited from the model
    Path to the Input Model:     matroid.pb
    Input shapes:     inherited from the model
    Log level:     ERROR
    Mean values:     ()
    IR output name:     inherited from the model
    Names of output layers:     inherited from the model
    Path for generated IR:     /opt/intel/computer_vision_sdk_2018.0.234/deployment_tools/model_optimizer
    Reverse input channels:     False
    Scale factor:     None
    Scale values:     ()
    Version:     0.3.61.37271eb9
    Input model in text protobuf format:     False
    Offload unsupported operations:     False
    Path to model dump for TensorBoard:     None
    Update the configuration file with input/output node names:     None
    Operations to offload:     None
    Patterns to offload:     None
    Use the config file:     None

I'm not able to find anything in the documentation about how to handle cycles within the graph, but imagine it may require offloading into TensorFlow.  Any information regarding this would be extremely helpful.  Thanks!

[Shubha_R_Intel]

Done. I have forwarded this to the Model Optimizer Devs.  Thank You Mark.

Shubha

[Shubha_R_Intel]

Mark I have contacted Intel Developers and they would like you to share a freezed model. Can you kindly attach that ?

Thanks,

Shubha

[Mark_A_Intel]

Hello,

This model is actually proprietary from one of our ISVs, and I would not want to send it to a public forum.  Is there an email address I could forward a copy of this model to to have it looked at?  Thanks.
 

Mark

[chen__ives]

Hello,

In few days ago, I tried to convert TensorFlow models: SSD MobileNet and SSD InceptionV2 via MO, but encountered similar issue. Then I followed direction in "C:/Intel/computer_vision_sdk_2018.1.249/deployment_tools/documentation/docs/TensorFlowObjectDetectionSSD.html", and this issue is resolved.

Take SSD MobileNet for example, you may need use mo_tf.py to covert it, like

python mo_tf.py --input_model "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\ssd_mobilenet_v1_coco_2017_11_17\frozen_inference_graph.pb" --input=1:Preprocessor/mul --input_shape="(1,300,300,3)" --tensorflow_use_custom_operations_config extensions/front/tf/ssd_support.json --output="detection_boxes,detection_scores,num_detections" --output_dir "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\intel_models\ssd_mobilenet_v1_coco_2017_11_17"

I'm still not familiar with this Python module, and above just a trail-and-error result.

Hope it would be helpful. 

[Sergey_L_Intel2]

Hi,

Based on symptoms, this is a graph with control flow that may or may not be an essential part of the model. Cycles in graphs are supported for predefined number of models – SSDs built with TF Detection API – where they are just eliminated. In general case cycles are not supported. Common recipe: the graph should be analyzed (for example in TensorBoard) to identify straightforward convolutional parts that consist of supported TF operations without cycles. These parts should be converted to IR. All other parts should be done as pre-/post-processing outside IE IR.

[mohan__deepa]

Hi Sergey

Could you please explain this a little more?

1. Why is the control flow cyclic graph not an essential part of the model?

2. Could you please give an example of where the SSD graph could have a cycle and how eliminating it is ok from an IR perspective?

3.  Is it very hard to visualize graph cycles in tensorboard so if you can throw some light on where in the seq2seq model, there could be a potential graph cycle, that will help a lot

Regards

Deepa

[Herings__Kai]

Hi There, 

I am stuck at the same issue. I did train an SSD InceptionV2 with tensorflow, afterwards freezing up the net, now having my frozen.pb net.
I want it to be optimized for the AWS deeplens - basically the wrapper around the intel model optimizer is a little sub-optimal, so you do end up with searching a lot for the actual error.

[ERROR]-mo.py:160,[ WARNING ] 
Detected not satisfied dependencies:
numpy: installed: 1.14.5, required: 1.13.0

Please install required versions of components or use install_prerequisites script
/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites_tf.sh
Note that install_prerequisites scripts may install additional components.
[ ERROR ] -------------------------------------------------
[ ERROR ] ----------------- INTERNAL ERROR ----------------
[ ERROR ] Unexpected exception happened.
[ ERROR ] Please contact Model Optimizer developers and forward the following information:
[ ERROR ] Graph contains a cycle.
[ ERROR ] Traceback (most recent call last):
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/main.py", line 222, in main
return driver(argv)
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/main.py", line 190, in driver
mean_scale_values=mean_scale)
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 143, in tf2nx
partial_infer(graph)
File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/middle/passes/infer.py", line 55, in partial_infer
nodes = nx.topological_sort(graph)
File "/usr/local/lib/python3.5/dist-packages/networkx/algorithms/dag.py", line 157, in topological_sort
raise nx.NetworkXUnfeasible("Graph contains a cycle.")
networkx.exception.NetworkXUnfeasible: Graph contains a cycle.

[ ERROR ] ---------------- END OF BUG REPORT --------------
[ ERROR ] -------------------------------------------------

The model optimizer is called through the mo wrapper method - https://docs.aws.amazon.com/deeplens/latest/dg/deeplens-model-optimizer-api-functions_and_objects.html

I would optimize the model manually, but the solution above ...

python mo_tf.py --input_model "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\ssd_mobilenet_v1_coco_2017_11_17\frozen_inference_graph.pb" --input=1:Preprocessor/mul --input_shape="(1,300,300,3)" --tensorflow_use_custom_operations_config extensions/front/tf/ssd_support.json --output="detection_boxes,detection_scores,num_detections" --output_dir "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\intel_models\ssd_mobilenet_v1_coco_2017_11_17"

 ...does not translate to my problem

Applying the call (without input shape) leads to "Sub-graph contains network input node "image_tensor". For more ... FAQ75

Specifying the input shape of (1,224,224,3) leads to "Other inputs needed for output computation" 27

In the first place I would like to get rid of the circle without bumping in substituion problems.

Any ideas?

[Herings__Kai]

I finally solved the problem, which was occuring on AWS Deeplens with the deployed version of the model optimizer (awscam.mo). It occured due to the fact that an old Intel Open Vino version was deployed & the underlying net didn't have the right version. I documented the solution in my blog article that I was writing ... 

https://blog.codecentric.de/2018/07/deep-learning-diesel-aws-deeplens/

It's in German, an English version will follow later. Regards, Kai

[Zicheng_R_Intel]

Hello,

I ran into the same issue here when running mo_tf.py on my frozen.pb. I am suspecting that the cycle issue is rooted in tf.while_loop(). Do you know if MO is able to handle dynamic RNN layers?

Thanks!

[Monique_J_Intel]

Hi Richard,

Currently MO doesn't support dynamic RNN layers or RNN networks but I will create a feature request for this.

Kind Regards,

Monique Jones

[Herings__Kai]

I have updated the description how I finally managed to work around the issue. Hope this helps: https://blog.codecentric.de/en/2018/11/aws-deep-lens-diesel-car-detection/

[K__Mike]

In case I wish to fix the graph using the Point 97 in the Mo_FAQ.html document in the VINO documentation docs,

which nodes do I include in  the first command -
python3 mo.py --input_model model/frozen_inference_graph.pb --tensorflow_subgraph_pattern ""FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/BatchNorm/FusedBatchNorm, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/Relu6, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_depthwise/depthwise, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_depthwise/BatchNorm/FusedBatchNorm, FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_1_depthwise/Relu6,..

Which of the nodes do I put above to offload a sub-graph of operations?

(Actual .pbtxt file had about 100 nodes)

[Matsak__Art]

Monique, can you please confirm that MO still doesn't support RNNs? We're getting "Graph contains a cycle" when trying to optimize a GRU network.

[Shubha_R_Intel]

Dear Matsak, Art,

OpenVino does support several RNN-like layers such as GRU, LSTM, and RNN sequence layers.  

If you peruse the  Release Notes and do a search on RNN you will find out exactly what OpenVino supports.

May I know more about the model you're using ? 

Thanks,

Shubha

[Matsak__Art]

Dear Shubha, thank you. We're using a bi-directional GRU recurrent network. The model is trained with Keras/TensorFlow and we're trying to convert it with mo_tf.py. However, we're running into the "Graph contains a cycle" error. Hence my question regarding mo_tf.py (not OpenVINO itself) not supporting RNN conversion yet. Please let me know.

[Matsak__Art]

Dear Shubha, thank you for your comments. The problem seems to be not in OpenVINO itself but in the mo_tf.py script. We use it to optimize/convert a TensorFlow bi-directional GRU model, and it fails with the "Graph contains a cycle" error. Hence my question regarding MO (not OpenVINO) not supporting RNNs yet.