Community
cancel
Showing results for 
Search instead for 
Did you mean: 
addoic
Beginner
243 Views

Failing to Compile InferenceEngine Openvino using MVN operation

I'm in progress converting the MBART model from HuggingFace Transformer to the OpenVino IF format and I've "successfully" brokendown the original pytorch model graph into 3 seperate ONNX models. I then used the mo.py to convert the onnx model to the IF format model to use the inference engine in openvino for "MYRIAD" Neural Compute Stick 2 . Its essentially two encoders and one decoder. I'm trying to test the first encoder model to see if a simple inference engine load works.

I'm geting the following error:

[ INFO ] Loading Inference Engine
[ INFO ] Loading network:
[ INFO ]     c:\protoc\models\translator\encoder\model.xml
[ INFO ] Device info:
[ INFO ]         MYRIAD
[ INFO ]         MKLDNNPlugin version ......... 2.1
[ INFO ]         Build ........... 2021.3.0-2774-d6ebaa2cd8e-refs/pull/4731/head
[ INFO ] Inputs number: 2
[ INFO ]     Input name: attention_mask
[ INFO ]     Input shape: [1, 92]
....

RuntimeError: Failed to compile layer "Div_25": Unsupported combination of indices in layer "Div_25". Only across channel and full batch supported.

I check the Div_25 layer and it looks like:


<layer id="30" name="Div_25" type="MVN" version="opset6">
            <data eps="9.999999747378752e-06" eps_mode="inside_sqrt" normalize_variance="true"/>
            <input>
                <port id="0">
                    <dim>1</dim>
                    <dim>92</dim>
                    <dim>1024</dim>
                </port>
                <port id="1">
                    <dim>1</dim>
                </port>
            </input>
            <output>
                <port id="2" precision="FP32" names="231">
                    <dim>1</dim>
                    <dim>92</dim>
                    <dim>1024</dim>
                </port>
            </output>
        </layer>

Reading the MVN documentation, and tried putting various values for port id="1" under dim with not luck. I unfortunately don't fully understand what its warning me about to fix it.

0 Kudos
5 Replies
Zulkifli_Intel
Moderator
214 Views

Hello Hovaness Bartamian,


Greetings to you.


Please share with us your model and necessary files for us to recreate the issue.


Sincerely,

Zulkifli


addoic
Beginner
186 Views

The link to the models: https://drive.google.com/file/d/1X85QqooE2A4p9Zgi1h7XMMDLzcfc3HE_/view?usp=sharing 

 

Change `model = 'c:\\models\\encoder\\model.xml'` to where ever you extrac the "encoder" model.

 

'''
 Copyright (c) 2018-2021 Intel Corporation

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
'''
from __future__ import print_function
from argparse import ArgumentParser, SUPPRESS
import logging as log
import os
import sys

import ngraph as ng
import numpy as np
from openvino.inference_engine import IECore
from transformers import AutoTokenizer


def main():
    log.basicConfig(format='[ %(levelname)s ] %(message)s', level=log.INFO, stream=sys.stdout)

    log.info('Loading Inference Engine')
    ie = IECore()

    # ---1. Read a model in OpenVINO Intermediate Representation (.xml and .bin files) or ONNX (.onnx file) format ---
    model = 'c:\\models\\encoder\\model.xml'
    log.info(f'Loading network:')
    log.info(f'    {model}')
    net = ie.read_network(model=model)
    # -----------------------------------------------------------------------------------------------------

    # ------------- 2. Load Plugin for inference engine and extensions library if specified --------------
    log.info('Device info:')
    versions = ie.get_versions('MYRIAD')
    log.info(f'        MYRIAD')
    log.info(f'        MKLDNNPlugin version ......... {versions["MYRIAD"].major}.{versions["MYRIAD"].minor}')
    log.info(f'        Build ........... {versions["MYRIAD"].build_number}')
    # -----------------------------------------------------------------------------------------------------

    # --------------------------- 3. Read and preprocess input --------------------------------------------

    log.info(f'Inputs number: {len(net.input_info.keys())}')
    assert len(net.input_info.keys()) == 2, 'Sample supports network with two inputs'
    assert len(net.outputs.keys()) == 1, 'Sample supports clean one output'
    input_name = list(net.input_info.keys())[0]
    input_info = net.input_info[input_name]
    supported_input_dims = 2  # Supported input layout - NHWC

    log.info(f'    Input name: {input_name}')
    log.info(f'    Input shape: {str(input_info.input_data.shape)}')
    print()
    if len(input_info.input_data.layout) == supported_input_dims:
        n, c = input_info.input_data.shape
        assert n == 1, 'Sample supports topologies with one '
    else:
        raise AssertionError('Sample supports input with NC shape only')


    # -----------------------------------------------------------------------------------------------------

    # --------------------------- 4. Configure input & output ---------------------------------------------
    # --------------------------- Prepare input blobs -----------------------------------------------------
    log.info('Preparing input blobs')

    if len(input_info.layout) == supported_input_dims:
        input_info.precision = 'U8'

    tokenizer = AutoTokenizer.from_pretrained('facebook/mbart-large-50-many-to-many-mmt')
    sample_input = "The universe is a dark forest." 
    model_inputs = tokenizer(sample_input,  max_length=92, padding='max_length', truncation=True, return_tensors="np")
    input_ids = model_inputs["input_ids"]
    attention_mask = model_inputs["attention_mask"]

    data = {"input_ids":input_ids ,"attention_mask":attention_mask}

    # --------------------------- Prepare output blobs ----------------------------------------------------

    # -----------------------------------------------------------------------------------------------------

    # --------------------------- Performing inference ----------------------------------------------------
    log.info('Loading model to the device')
    exec_net = ie.load_network(network=net, device_name='MYRIAD') 
    log.info('Creating infer request and starting inference')
    exec_result = exec_net.infer(inputs=data)
    # -----------------------------------------------------------------------------------------------------

    # --------------------------- Read and postprocess output ---------------------------------------------
    log.info('Processing output blobs')
    result = exec_result[output_name]
    boxes = {}
    detections = result[0][0]  # [0][0] - location of detections in result blob
    for number, proposal in enumerate(detections):
        imid, label, confidence, coords = np.int(proposal[0]), np.int(proposal[1]), proposal[2], proposal[3:]
        if confidence > 0.5:
            # correcting coordinates to actual image resolution
            xmin, ymin, xmax, ymax = w_new * coords[0], h_new * coords[1], w_new * coords[2], h_new * coords[3]

            log.info(f'    [{number},{label}] element, prob = {confidence:.6f}, '
                f'bbox = ({xmin:.3f},{ymin:.3f})-({xmax:.3f},{ymax:.3f}), batch id = {imid}')
            if not imid in boxes.keys():
                boxes[imid] = []
            boxes[imid].append([xmin, ymin, xmax, ymax])

    imid = 0  # as sample supports one input image only, imid in results will always be 0

    tmp_image = cv2.imread(args.input)
    for box in boxes[imid]:
        # drawing bounding boxes on the output image
        cv2.rectangle(
            tmp_image, 
            (np.int(box[0]), np.int(box[1])), (np.int(box[2]), np.int(box[3])), 
            color=(232, 35, 244), thickness=2)
    cv2.imwrite('out.bmp', tmp_image)
    log.info('Image out.bmp created!')
    # -----------------------------------------------------------------------------------------------------

    log.info('Execution successful\n')
    log.info(
        'This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool')


if __name__ == '__main__':
    sys.exit(main() or 0)
Zulkifli_Intel
Moderator
150 Views

Hello Hovaness bartamian,


It seems that the ONNX model that you are using with the Div_25 layer is not yet supported by OpenVINO. You may refer to these documents on the Supported Framework layer.


Sincerely,

Zulkifli


Zulkifli_Intel
Moderator
130 Views

Hello Hovaness Bartamian,


This thread will no longer be monitored since we have provided a solution. If you need any additional information from Intel, please submit a new question.


Sincerely,

Zulkifli


Zulkifli_Intel
Moderator
97 Views

Hello Hovaness Bartamian,

We have revisited the case. Looking at the error, MVN-1 operation would cater to the across channels flag.

We would like to suggest you try using MVN - 1 operation instead of MVN - 6 operation.

We would also encourage you to try using the OpenVINO 2021.4, which has been released recently.

 

Sincerely,

Zulkifli

Reply