BUG - Subgraph replacement attempt to iterate over nodes that have been removed

Rik_A_Intel · ‎06-14-2018

Hi. I'm seeing exceptions thrown when using the subgraph replacement.

In "repalcement.py" you have a function

   def find_and_replace_pattern(self, graph: nx.MultiDiGraph):
        for match in find_pattern_matches(graph, **self.pattern()):
            match = {v: k for k, v in match.items()} 
            for k in match:
                match = Node(graph, match)
            self.replace_sub_graph(graph, match)

If this finds a subgraph to match and replace, then the "replace_sub_graph(...)" method in my extension derived from FrontReplacementSubgraph is called.

If this calls "replace_node(..)" in graph.py, then that removes a node from the graph. This is modifying the graph while the "find_and_replace_pattern()" is iterating over it - and causes an exception when the iterator tries to find the node that has already been removed in the graph.

I have worked around this with changes to replace_node(...) in graph.py - not calling graph.remove_node() but simply removing the outgoing edges from the node that is being replaced. I suspect this is not how you intend the code to work, but it is letting me proceed.

I will attempt to create a customer agnostic example that I can share. Note that the behaviour of the script is non-deterministic since find_pattern_matches() will search the graph in different orders on different runs, so sometimes the pattern of execution is ok for a given graph, sometimes it throws the exception.

If you have a better arena for me to raise issues with model_optimiser as an Intel employee then I'd be happy to follow your advice on how to register and track issues.

Rik_A_Intel · ‎06-14-2018

Attached a zip file with a trivial test.

Test input in "input"

XML Results with no subgraph replacement in Results/ncr_mult_no_subgraph_replacement.xml

The results I need are Results/ncr_mult_required.xml

My extensions are ncr_graph.py - copy to extensions/front - and neg_concat_relu.py - copy to extensions/ops

I changed caffe.py to call .. (See another forum post)

FrontReplacementSubgraph.find_and_replace_all_patterns(graph) # rik

Without the change to graph.py I get ...

[ 2018-06-14 11:59:59,971 ] [ DEBUG ] [ ncr_graph:39 ] replace_sub_graph NCR

***replace_node ncr3/relu

[ 2018-06-14 12:00:02,823 ] [ DEBUG ] [ ncr_graph:39 ] replace_sub_graph NCR

***replace_node ncr2/relu

***replace_node ncr1/relu

[ 2018-06-14 12:00:04,401 ] [ DEBUG ] [ ncr_graph:39 ] replace_sub_graph NCR

[ 2018-06-14 12:00:05,150 ] [ ERROR ] [ main:230 ] -------------------------------------------------

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:231 ] ----------------- INTERNAL ERROR ----------------

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:232 ] Unexpected exception happened.

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:233 ] Please contact Model Optimizer developers and forward the following information:

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:234 ] 'ncr2/relu'

[ 2018-06-14 12:00:05,157 ] [ ERROR ] [ main:235 ] Traceback (most recent call last):

File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\main.py", line 222, in main

return driver(argv)

File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\main.py", line 202, in driver

custom_layers_mapping_path=custom_layers_mapping_path)

File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\pipeline\caffe.py", line 112, in driver

FrontReplacementSubgraph.find_and_replace_all_patterns(graph) # rik

File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\utils\replacement_pattern.py", line 42, in find_and_replace_all_patterns

replacer.find_and_replace_pattern(graph)

File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\front\common\replacement.py", line 121, in find_and_replace_pattern

for match in find_pattern_matches(graph, **self.pattern()):

File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 372, in subgraph_isomorphisms_iter

for mapping in self.match():

File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 308, in match

if self.syntactic_feasibility(G1_node, G2_node):

File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 632, in syntactic_feasibility

for predecessor in self.G1.pred[G1_node]:

KeyError: 'ncr2/relu'

Hope this makes sense and I'm not wasting your time.

Gleb_K_Intel · ‎06-28-2018

Hi Rik!

Thank you for reporting this issue!

I have checked model conversion on R2 release version with model you attached and everything works fine, and I see the same IR you sent.

Could you please check this issue on you side with R2 MO version? and could you please provide your MO version ?

Rik_A_Intel · ‎06-28-2018

Thanks for checking. we are hoping to catch up to version 2 in the next sprint and will confirm the fix. It is good that it implies you are now supporting subgraph replacement in caffe - I did see substantial changes in that area.

Rik