Speech recognition with NCS2

[Kano]

Hi, 

I'm trying to run speech recognition sample with NCS2(OpenVINO 2018R5/2019R1) + Raspberry Pi3 according to document.

I downloaded models and files from here and optimized them to .xml on Mac.

I used this command, but the following error was returned.

./speech_sample -d GNA_AUTO -bs 2 -i feat1_10.ark -m rm_cnn4a.xml -o score.ark -r score1_10.ark

[ INFO ] InferenceEngine: 
	API version ............ 1.6
	Build .................. 22443
[ INFO ] Parsing input parameters
[ INFO ] No extensions provided
[ INFO ] Loading plugin
[ ERROR ] Cannot find plugin to use :

I tried replacing the '-d' and '-m' arguments, but the same error was returned for all device. 

Please give me some advice how to run the speech recognition sample, or kaldi models which were confirmed that work on OpenVINO.

(In the first place, it seems that few people work on speech recognition with NCS2. Is it not suitable for speech recognition ?)

Thanks,

Kano

[Luis_at_Intel]

It looks like the models used by the speech_sample are not supported by the VPU plugins. The sample makes use of the GNA plugin which based on the Supported Devices in the OpenVINO™ Toolkit documentation, the device types are Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver processor J5005, Intel® Celeron® processor J4005, Intel® Core™ i3-8121U processor. 

It may not be possible to run these particular models on the NCS2/RPi as the Myriad plugin doesn't seem to have support for the networks used by the speech_sample. Let me know if you have any other questions.

Regards,

@Luis_at_Intel

[Eugene_S_Intel]

Hi Kano,  

I see following solution: to run rm_cnn4a model on ncs2 stick you need to use fp16 ir, so please configure ModelOptimiser accordingly. Also -bs 2 didn't work for some reason.

Actual errors with -d GNA_AUTO, CPU, ... on rpi relates to just missed plugins for that architecture, please check your installation. I've double checked on my side (not on raspberry pi though) rm_cnn4a might work on -d MYRIAD device it will produce large error, even comparing to -d GPU with same fp16 model. 

speech_sample -m rm_cnn4a_fp16.xml -i rm_cnn4a_smbr_feat1_10.ark -r rm_cnn4a_smbr_score1_10.ark -d MYRIAD
[ INFO ] InferenceEngine:
    API version ............ 1.6
    Build .................. custom_master-2
    Description ....... API
[ INFO ] Parsing input parameters
[ INFO ] No extensions provided
[ INFO ] Loading Inference Engine
[ INFO ] Device info:
    MYRIAD
    myriadPlugin version ......... 1.6
    Build ........... custom_master-2

[ INFO ] Loading network files
[ INFO ] Batch size is 1
[ INFO ] Using scale factor of 3075.15 calculated from first utterance.
[ INFO ] Loading model to the device
[ INFO ] Model loading time 2873.59 ms
Utterance 0:
Total time in Infer (HW and SW):    2849.58 ms
Frames in utterance:            250 frames
Average Infer time per frame:        11.3983 ms
         max error: 27.3883
         avg error: 2.8932
     avg rms error: 3.71711
       stdev error: 2.64166


End of Utterance 0

[顾__博]

Hi Eugene Smirnov,

It looks like you successfully converted the model rm_cnn4a.nnet ,  I converted the model succeed on toolkit (2019_R1),but I on the latest version of the toolkit (2019_R3) failed,

 ./mo_kaldi.py --input_model /home/gubo/Downloads/wsj/wsj_dnn5b.nnet --counts /home/gubo/Downloads/wsj/wsj_dnn5b.counts --remove_output_softmax
Model Optimizer arguments:
Common parameters:
    - Path to the Input Model:     /home/gubo/Downloads/wsj/wsj_dnn5b.nnet
    - Path for generated IR:     /opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/.
    - IR output name:     wsj_dnn5b
    - Log level:     ERROR
    - Batch:     Not specified, inherited from the model
    - Input layers:     Not specified, inherited from the model
    - Output layers:     Not specified, inherited from the model
    - Input shapes:     Not specified, inherited from the model
    - Mean values:     Not specified
    - Scale values:     Not specified
    - Scale factor:     Not specified
    - Precision of IR:     FP32
    - Enable fusing:     True
    - Enable grouped convolutions fusing:     True
    - Move mean values to preprocess section:     False
    - Reverse input channels:     False
Kaldi specific parameters:
    - A file name with full path to the counts file:     /home/gubo/Downloads/wsj/wsj_dnn5b.counts
    - Removes the SoftMax layer that is the output layer:     True
Model Optimizer version:     2019.2.0-436-gf5827d4
[ ERROR ]  -------------------------------------------------
[ ERROR ]  ----------------- INTERNAL ERROR ----------------
[ ERROR ]  Unexpected exception happened.
[ ERROR ]  Please contact Model Optimizer developers and forward the following information:
[ ERROR ]  Exception occurred during running replacer "None (<class 'extensions.front.kaldi.set_ports.SetPortsPattern'>)": 'list' object has no attribute 'keys'
[ ERROR ]  Traceback (most recent call last):
  File "/opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 273, in apply_replacements
    for_graph_and_each_sub_graph_recursively(graph, replacer.find_and_replace_pattern)
  File "/opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/mo/middle/pattern_match.py", line 58, in for_graph_and_each_sub_graph_recursively
    func(graph)
  File "/opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/extensions/front/kaldi/set_ports.py", line 35, in find_and_replace_pattern
    nodes = graph.nodes(data=False).keys()
AttributeError: 'list' object has no attribute 'keys'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/mo/main.py", line 302, in main
    return driver(argv)
  File "/opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/mo/main.py", line 273, in driver
    ret_res = mo_kaldi.driver(argv, argv.input_model, model_name, argv.output_dir)
  File "/opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/mo/pipeline/kaldi.py", line 140, in driver
    class_registration.apply_replacements(graph, class_registration.ClassType.FRONT_REPLACER)
  File "/opt/intel/openvino_2019.2.242/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 299, in apply_replacements
    )) from err
Exception: Exception occurred during running replacer "None (<class 'extensions.front.kaldi.set_ports.SetPortsPattern'>)": 'list' object has no attribute 'keys'

[ ERROR ]  ---------------- END OF BUG REPORT --------------
[ ERROR ]  -------------------------------------------------

Thanks

顾博

[JAVIERJOSE_A_Intel]

Hi 顾 博,  

I noticed you are using openvino_2019.2.242 version. Please update OpenVINO™ toolkit to the latest version (2019.3.376) and try to run it with the same command again.

Regards,  

Javier A.  

[顾__博]

Hi,

 I failed on the latest version of the toolkit (both 2019_R3 and 2019_R2 with the same error)

[David_C_Intel]

Hi 博 顾,  

We tested it from our end with the latest OpenVINO™ toolkit version. To do so, we followed these steps:

- Check for prerequisites by running these script located in <INSTALL_DIR>/deployment_tools/model_optimizer/install_prerequisites directory:

install_prerequisites_kaldi.sh

- Then on a folder you have all permissions on, copy the .nnet and the .counts files and run the following:

python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_kaldi.py \
--input_model wsj_dnn5b.nnet \
--counts wsj_dnn5b.counts \
--remove_output_softmax

Please, let me know if it works for you.

Best regards,  

David

[顾__博]

Hi, David

Thanks for your advice,but I failed with the same ERROR

python3 mo_kaldi.py --input_model /home/gubo/Downloads/wsj/wsj_dnn5b.nnet --counts /home/gubo/Downloads/wsj/wsj_dnn5b.counts --remove_output_softma
Model Optimizer arguments:
Common parameters:
    - Path to the Input Model:     /home/gubo/Downloads/wsj/wsj_dnn5b.nnet
    - Path for generated IR:     /opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/.
    - IR output name:     wsj_dnn5b
    - Log level:     ERROR
    - Batch:     Not specified, inherited from the model
    - Input layers:     Not specified, inherited from the model
    - Output layers:     Not specified, inherited from the model
    - Input shapes:     Not specified, inherited from the model
    - Mean values:     Not specified
    - Scale values:     Not specified
    - Scale factor:     Not specified
    - Precision of IR:     FP32
    - Enable fusing:     True
    - Enable grouped convolutions fusing:     True
    - Move mean values to preprocess section:     False
    - Reverse input channels:     False
Kaldi specific parameters:
    - A file name with full path to the counts file:     /home/gubo/Downloads/wsj/wsj_dnn5b.counts
    - Removes the Memory layer and use additional inputs and outputs instead:     False
    - Removes the SoftMax layer that is the output layer:     True
Model Optimizer version:     2019.3.0-408-gac8584cb7
[ ERROR ]  -------------------------------------------------
[ ERROR ]  ----------------- INTERNAL ERROR ----------------
[ ERROR ]  Unexpected exception happened.
[ ERROR ]  Please contact Model Optimizer developers and forward the following information:
[ ERROR ]  Exception occurred during running replacer "None (<class 'extensions.front.kaldi.set_ports.SetPortsPattern'>)": 'list' object has no attribute 'keys'
[ ERROR ]  Traceback (most recent call last):
  File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 273, in apply_replacements
    for_graph_and_each_sub_graph_recursively(graph, replacer.find_and_replace_pattern)
  File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/mo/middle/pattern_match.py", line 58, in for_graph_and_each_sub_graph_recursively
    func(graph)
  File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/extensions/front/kaldi/set_ports.py", line 35, in find_and_replace_pattern
    nodes = graph.nodes(data=False).keys()
AttributeError: 'list' object has no attribute 'keys'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/mo/main.py", line 298, in main
    return driver(argv)
  File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/mo/main.py", line 269, in driver
    ret_res = mo_kaldi.driver(argv, argv.input_model, model_name, argv.output_dir)
  File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/mo/pipeline/kaldi.py", line 148, in driver
    class_registration.apply_replacements(graph, class_registration.ClassType.FRONT_REPLACER)
  File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 299, in apply_replacements
    )) from err
Exception: Exception occurred during running replacer "None (<class 'extensions.front.kaldi.set_ports.SetPortsPattern'>)": 'list' object has no attribute 'keys'

[ ERROR ]  ---------------- END OF BUG REPORT --------------
[ ERROR ]  -------------------------------------------------

the .nnet please see https://download.01.org/openvinotoolkit/2018_R3/models_contrib/GNA/wsj_dnn5b_smbr/

[Luis_at_Intel]

Hi 顾, 博

We are able to convert the model from the provided link just fine using the latest release (see log below). It might be a configuration issue or something else going on with the software.

I could suggest double check all install steps are completed, and verify the installation using the verification scripts before attempting to convert the model. Make sure also to run the setupvars.bat and the install_prerequisites_kaldi.bat.

C:\Users\user name\Downloads\kaldi>python "C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\mo_kaldi.py" --input_model wsj_dnn5b.nnet --counts wsj_dnn5b.counts --remove_output_softmax
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      C:\Users\user name\Downloads\kaldi\wsj_dnn5b.nnet
        - Path for generated IR:        C:\Users\user name\Downloads\kaldi\.
        - IR output name:       wsj_dnn5b
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
Kaldi specific parameters:
        - A file name with full path to the counts file:        C:\Users\user name\Downloads\kaldi\wsj_dnn5b.counts
        - Removes the Memory layer and use additional inputs and outputs instead:       False
        - Removes the SoftMax layer that is the output layer:   True
Model Optimizer version:        2019.3.0-408-gac8584cb7
[ WARNING ]
Detected not satisfied dependencies:
        numpy: installed: 1.18.0, required: 1.13.0

Please install required versions of components or use install_prerequisites script
C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\install_prerequisites\install_prerequisites_kaldi.bat
Note that install_prerequisites scripts may install additional components.

[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: C:\Users\user name\Downloads\kaldi\.\wsj_dnn5b.xml
[ SUCCESS ] BIN file: C:\Users\user name\Downloads\kaldi\.\wsj_dnn5b.bin
[ SUCCESS ] Total execution time: 18.60 seconds.

If the issue persists, please provide more details about your setup (i.e. versions of python, cmake, etc.) so we can attempt to reproduce and help find a solution. 

Regards,

Luis

[顾__博 · Accepted solution]

Hi，for the Linux there is a error for python3.5.2

 File "/opt/intel/openvino_2019.3.376/deployment_tools/model_optimizer/extensions/front/kaldi/set_ports.py", line 35, in find_and_replace_pattern
    nodes = graph.nodes(data=False).keys()
AttributeError: 'list' object has no attribute 'keys'

just change the nodes = graph.nodes(data=False).keys() to nodes = graph.nodes(data=False)

and run successed

Thanks