Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Rik_A_Intel
Employee
80 Views

BUG - Subgraph replacement attempt to iterate over nodes that have been removed

Hi. I'm seeing exceptions thrown when using the subgraph replacement.

In "repalcement.py" you have a function

   def find_and_replace_pattern(self, graph: nx.MultiDiGraph):
        for match in find_pattern_matches(graph, **self.pattern()):
            match = {v: k for k, v in match.items()} 
            for k in match:
                match = Node(graph, match)
            self.replace_sub_graph(graph, match)

 

If this finds  a subgraph to match and replace, then the "replace_sub_graph(...)" method in my extension derived from FrontReplacementSubgraph is called.

If this calls "replace_node(..)" in graph.py, then that removes a node from the graph. This is modifying the graph while the "find_and_replace_pattern()" is iterating over it - and causes an exception when the iterator tries to find the node that has already been removed in the graph.

I have worked around this with changes to replace_node(...) in graph.py - not calling graph.remove_node() but simply removing the outgoing edges from the node that is being replaced. I suspect this is not how you intend the code to work, but it is letting me proceed.

I will attempt to create a customer agnostic example that I can share. Note that the behaviour of the script is non-deterministic since find_pattern_matches() will search the graph in different orders on different runs, so sometimes the pattern of execution is ok for a given graph, sometimes it throws the exception.

If you have a better arena for me to raise issues with model_optimiser as an Intel employee then I'd be happy to follow your advice on how to register and track issues. 

 

 

0 Kudos
3 Replies
Rik_A_Intel
Employee
80 Views

Attached a zip file with a trivial test. 

Test input in "input"

XML Results with no subgraph replacement in Results/ncr_mult_no_subgraph_replacement.xml

The results I need are Results/ncr_mult_required.xml

My extensions are ncr_graph.py - copy to extensions/front - and neg_concat_relu.py - copy to extensions/ops

I changed caffe.py to call .. (See another forum post)

FrontReplacementSubgraph.find_and_replace_all_patterns(graph) # rik

Without the change to graph.py I get ...

[ 2018-06-14 11:59:59,971 ] [ DEBUG ] [ ncr_graph:39 ]  replace_sub_graph NCR
***replace_node ncr3/relu
[ 2018-06-14 12:00:02,823 ] [ DEBUG ] [ ncr_graph:39 ]  replace_sub_graph NCR
***replace_node ncr2/relu
***replace_node ncr1/relu
[ 2018-06-14 12:00:04,401 ] [ DEBUG ] [ ncr_graph:39 ]  replace_sub_graph NCR
[ 2018-06-14 12:00:05,150 ] [ ERROR ] [ main:230 ]  -------------------------------------------------
[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:231 ]  ----------------- INTERNAL ERROR ----------------
[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:232 ]  Unexpected exception happened.
[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:233 ]  Please contact Model Optimizer developers and forward the following information:
[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:234 ]  'ncr2/relu'
[ 2018-06-14 12:00:05,157 ] [ ERROR ] [ main:235 ]  Traceback (most recent call last):
  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\main.py", line 222, in main
    return driver(argv)
  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\main.py", line 202, in driver
    custom_layers_mapping_path=custom_layers_mapping_path)
  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\pipeline\caffe.py", line 112, in driver
    FrontReplacementSubgraph.find_and_replace_all_patterns(graph) # rik
  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\utils\replacement_pattern.py", line 42, in find_and_replace_all_patterns
    replacer.find_and_replace_pattern(graph)
  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\front\common\replacement.py", line 121, in find_and_replace_pattern
    for match in find_pattern_matches(graph, **self.pattern()):
  File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 372, in subgraph_isomorphisms_iter
    for mapping in self.match():
  File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 308, in match
    if self.syntactic_feasibility(G1_node, G2_node):
  File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 632, in syntactic_feasibility
    for predecessor in self.G1.pred[G1_node]:
KeyError: 'ncr2/relu'
 
Hope this makes sense and I'm not wasting your time.
Gleb_K_Intel
Employee
80 Views

Hi Rik!

Thank you for reporting this issue!

I have checked model conversion on R2 release version with model you attached and everything works fine, and I see the same IR you sent.

Could you please check this issue on you side with R2 MO version?  and could you please provide your MO version ?

 

Rik_A_Intel
Employee
80 Views

Thanks for checking. we are hoping to catch up to version 2 in the next sprint and will confirm the fix. It is good that it implies you are now supporting subgraph replacement in caffe - I did see substantial changes in that area.

Rik

 

Reply