Openvino Smart classroom sample Broken

[Bird__James]

Hi Guys,

The Inteligent classroom demo is giving real problems, Acuracy is getting worse and worse with each openvino release. Let me demostrate what im talking about, Please Openvino team get this resolved.

Quick Video grab of runing the demo in R3 and the bad results:

https://youtu.be/5GcR6nym7Is

I did a video of the sample running on R1.1 which still was not working as should, but gave better results than this version:

https://youtu.be/07QdEA4VHMU

And here's a video from your official openvino channel on youtube that shows the system working?, Please find a fix for this.

https://youtu.be/ACtMs0ETpBg

[Luis_at_Intel]

Thanks for reaching out. I am testing the smart_classroom_demo as is in both R1.1 and R3.1 and I see the same results in both releases. The student from back-left is detected for some period of time as "standing" while sitting (on both releases), and all faces/students are detected (on both releases). I used FP32 for my tests, if there is any information that could help us see the issue that would be great.

P.S. the videos were a great way to show the issue you are observing, so thank you for that. 

See the pictures attached from my testing.  

Left : Windows 10 R1.1 (v2019.1.148, API version = 1.6)

Right: Ubuntu 16.04 R3.1 (v2019.3.376, API version = 2.1)

Edit: By the way, these are the models that I used (for both versions). Are these the same ones you used? 

./smart_classroom_demo
-i ~/Downloads/classroom.mp4
-m_act ~/models/intel/person-detection-action-recognition-0005/FP32/person-detection-action-recognition-0005.xml
-m_fd ~/models/intel/face-detection-retail-0004/FP32/face-detection-retail-0004.xml
-m_lm ~/models/intel/landmarks-regression-retail-0009/FP32/landmarks-regression-retail-0009.xml
-m_reid ~/models/intel/face-reidentification-retail-0095/FP32/face-reidentification-retail-0095.xml

[Bird__James]

Hi Luis,

Appreciate your time. Ive run the simple Classroom demo and get the same results as you as shown in the following video:

https://youtu.be/tlXRWbkvypY

But Im on a 6 core 12 thread CPU, and half the threads dont even get used. There seems to be some threading or Async issues for a start.

But the Classroom example I first showed in the videos is a more production environment example that runs data results out the demo into influxDB and into Grafana dashboarding software, which clearly is a more realistic production release example. The code for the more complex Classroom example is located at https://github.com/intel-iot-devkit/intelligent-classroom.

Try both the R1 version then the R3 version branch's, R1 behaves more like the sinple demo but still lower acuracy, and R3 versions runs very bad in terms of accuracy compared to even R1 version

Try the above example from Intels Github page and compare to my results from the first R1 and R3 videos I posted. Im useing exactly the same Fp32 inference models (in the simple demo example and the complex example but very different results) as you posted but you get very different results with the more complex example. Looks like a Asysnc or threading issue with Openvino to me at this point.

Also can intel open source it's pre trained models so we can do transfer learning with them, at least then we can try to increase acuracy with our own data sets.Or Intel to build and provide much higher acuracy versions of models that partners could license for our product releases

Look forward to working with you and all at Intel to resolve such issues, to build partnerships that allow us to release products with Intel inside and Openvino software stack.

[Luis_at_Intel]

Hi James,

Thanks for clarifying which sample you used. I have tried the intelligent-classroom demo available in GitHub on Ubuntu* 16. I tested the R3 and R2 versions, there wasn't an R1 available when I accessed the repository. Either way, I wanted to share that I see very similar results using both versions where one of the students is identified as standing while sitting (back left) when using "face-detection-retail-0004", and when I use the default model "face-detection-adas-0001" the students in the back do not get identified. See the attached pdf for my results. Have you tried using the face-detection-retail-0004 model instead of the face-detection-adas-0001? Something else you could try too is a different input that has better lighting conditions where student faces could be easily or better identified, this is just a thought I had that came to mind.

As far as the pre-trained models go, I can't comment if these are going to be open sourced at some point in time. However, I can take your feedback and share it with engineering for consideration. You can take a look at the Open Model Zoo repo, it has details for some of the Intel models that might be helpful. 

Regards,

Luis