Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Kulecz__Walter
New Contributor I
92 Views

Clues about this error message?

I've got my NCS v1 SDK python3 code ported and running nicely on OpenVINO using NCS or NCS2.  But occasionally when I start the python3 program I get this error:

terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
  what():  New shapes [1,256,38,38] make Kernels(3x3), Channels(256), Output depth(32), Groups(1) not matching weights size: 73728 vs 864

So far, immediately restarting the program has always succeeded.  I'd like to get some idea of what the error message is trying to tell me.   

The code seems solid one its starts, as its been processing ~25 fps with a mix of NCS2 and CPU AI threads from 15 1920x1080 RTSP cameras for as long as three days in my testing so far.  Ultimately it needs to run 24/7/365 with a "watchdog" to restart/reboot in case of failure, which is where I think this intermittent error could cause me grief.

 

 

0 Kudos
9 Replies
JesusE_Intel
Moderator
92 Views

Hi Walter,

Could you please provide additional information about your setup?

  • What version of the OpenVINO toolkit are you using and which operating system?
  • Which model are you using?

Regards,

Jesus

Kulecz__Walter
New Contributor I
92 Views

OpenVINO 2019.1.144 on Ubuntu 16.04.  I've observed it using  NCS and NCS2 USB stick.

 

JesusE_Intel
Moderator
92 Views

Hi Walter,

I would like to reproduce your issue, could you share the network model that you converted to IR format and the model optimizer command used?

Regards,

Jesus

Kulecz__Walter
New Contributor I
92 Views

I didn't convert the model, I used the IR format from this tutorial, which was my starting point:

openvino tutorial where I got the IR format model.

I'm not developing AI, I'm trying to use it for an add-on to existing security camera setups.  My NCS SDK v1 code is here:

Security DVR add-on

In the extras directory is my very first working version Using OpenVINO, I haven't uploaded the current version yet, but virtually all the changes are to the image acquisition "front ends"

So far the code appears to run perfectly once it starts, its just sometimes I get that error on start up, but so far its always worked on the second try, this is the issue I'm trying to understand.

 

Two instances have been running for over a day now on two different test systems, one using NCS and the other NCS2.  I don't start and stop them any more than necessary as I develop my code on a third system.

JesusE_Intel
Moderator
92 Views

Hi Walter,

It looks like you are using the Deep Neural Networks module with OpenCV to inference with the Neural Compute Stick. Have you tried converting your Caffe model with the model optimzer included in the OpenVINO toolkit and run inference with one of our demo apps?

Converting Caffe model to IR format:

https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_C...

 

Object Detection Demo with SSD Networks:

http://docs.openvinotoolkit.org/latest/_inference_engine_samples_object_detection_sample_ssd_README....

 

Regards,

Jesus

 

 

Kulecz__Walter
New Contributor I
92 Views

I'm going to try converting MobileNetSSD-v2 as discussed in this thread:   https://software.intel.com/en-us/forums/computer-vision/topic/807958  as so far it performs much better on the Coral TPH than does the v1 on the NCS2.  I'll put the effort into trying to get this version 2 MobileNetSSD-coco  model running on the NCS2, as it should be worth the effort.

 

But I'm looking for a clue about what this error message is trying to tell me, not a treasure hunt with random attempts to change things in hope it goes away.  This approach is generally doomed with an error that is this infrequent.  Why does it fail, but then succeed on the next launch?  Background library threads not terminated before I restart the program?  I've started and stopped the program probably half a dozen times since my initial post and haven't seen the error again.

JEspi1
New Contributor I
92 Views

Hi Walter,

I apologize for the delay in my response. I was told by the development team that error comes from the network validation and can be caused when the file system and weights file is not completely read. Please let us know if you continue to see the error message persistently and if you could provide steps to reproduce.

Regards,

Jesus

JesusE_Intel
Moderator
92 Views

Hi Walter,

I apologize for the delay in my response. I was told by the development team that error comes from the network validation and can be caused when the file system and weights file is not completely read. Please let us know if you continue to see the error message persistently and if you could provide steps to reproduce.

Regards,

Jesus

Kulecz__Walter
New Contributor I
92 Views

It'll be real hard to reproduce as it happens rarely.

It seems most likely to occur after I terminate a test run and switch from an NCS2 to NCS or vice-versa.

I continue to modify my testing code but an earlier version of my code where I first saw it occur is here:

https://github.com/wb666greene/AI_enhanced_video_security/blob/master/README.md

It the AI_OVmt.py that supports both NCS and NCS2

I saw it again last week, and it happened  after I unplugged the NCS2 and plugged io the NCS.  And an immediate program restart worked perfectly.

I appreciate the effort, but if there are not some internal library threads that may not have terminated by the time I restart I'm at a loss as to what it could be, other than perhaps a udev interaction or "race"

My code is showing close to linear speed up mixing NCS and NCS 2  i.e ~11 fps with NCS, ~21 fps with NCS2 and ~ 30 fps with NCS1 + NCS2, and approx 20 fps with two NCS.  My input sourcing may be the limit at about 35 fps.  I on;y have a single NCS2 at present.