Stuck during execution on Mustang-F100-A10, Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA

[VDemc2]

Hello,

we have Mustang F100-A10 and TANK-870 AIoT Dev. Kit.

We have followed the official guide for the latest openvino to setup the 2019R1 openvino and our FPGA card.

We were able to run "aocl diagnose" and we see there is the card as you can see below.

ieisw@ieisw-SER0:~$ aocl diagnose   --------------------------------------------------------------------   Device Name:   acl0       BSP Install Location:   /opt/altera/aocl-pro-rte/aclrte-linux64/board/a10_1150_sg1       Vendor: Intel(R) Corporation       Phys Dev Name Status Information       acla10_1150_sg10Passed Intel Vision Accelerator Design with Intel Arria 10 FPGA (acla10_1150_sg10)   PCIe dev_id = 2494, bus:slot.func = 01:00.00, Gen3 x8   FPGA temperature = 61.6406 degrees C.       DIAGNOSTIC_PASSED   --------------------------------------------------------------------  

The 

./demo_squeezenet_download_convert_run.sh -d HETERO:FPGA,CPU

succeeded as well as demo_security_barrier_camera.sh with device HETERO:FPGA,CPU

However when we try to run classification_sample demo as below, it sometimes stuck forever. Not after certain amount of inference calls but randomly. Sometimes it finished with -ni set to 3000 but sometimes it stuck when we set -ni to 10. Below is the command we used.

/<path>/inference_engine_samples_build/intel64/Release/classification_sample -i /opt/intel/openvino/deployment_tools/demo/car.png -m /<path>/squeezenet1.1_FP16/squeezenet1.1.xml -d HETERO:FPGA,CPU -ni 10

The same thing happens with our custom code, which runs without issue on CPU.

While this execution is running we are not able to call "aocl diagnose" at all.

We are really out of ideas and we are starting to suspect that the FPGA card is broken. So any help or suggestion is highly appreciated.

My second question is if it's possible to call "aocl diagnose" from two different terminals simultaneously.

PS: We have skipped the part in documentation which is about USB Blaster (we dont have this thing). 

[JohnT_Intel]

Hi,

May I know if everything is running on local machine without docker, are you still observing the issue? I would like to see if the Message broker is running in local machine is also causing the freeze issue or not.

[VDemc2]

Yes if everything is running on local machine without any docker container we didnt see the issue.

We tried to run message broker - zookeeper and kafka - locally on host and we didnt see the freezing issue.

[JohnT_Intel]

Hi,

It looks like the docker container is causing the issue. You will need to debug to see why the debug container is causing the OpenVINO to hang.

[VDemc2]

It didnt happen on R5. docker version is the same as it was while we were using openvino R5. So IMHO it looks like there is nothing to debug on our side (docker containers we use are not our).

However could you please :

1. list all log files where FPGA plugin and HETERO:FPGA,CPU plugin and fpga itself store any logs
2. Any way how to get any informations regarding utilization of FPGA while there is something running on the board?

[JohnT_Intel]

1. list all log files where FPGA plugin and HETERO:FPGA,CPU plugin and fpga itself store any logs

No, log file is available. You will need to modify your host code to see where it is stuck.

1. Any way how to get any informations regarding utilization of FPGA while there is something running on the board?

No. The bitstream and driver is different as we update it and optimize to have a better performance.

[VDemc2]

we’ve found another way how to reproduce the freezing issue which is happening with intel’s human_pose_estimation demo as we.

From some reason the human_pose demo freezes when we open Google Chrome and Chromium browser. it is not happening for Firefox browser.

It seems is not just about docker but a virtualization in general.  

How to reproduce:

1. Build and run intel’s human pose demo with an input from a video file on HETERO:FPGA,CPU device.

~/inference_engine_samples_build/intel64/Release/human_pose_estimation_demo -i ~/Downloads/yt-openvino-interactive-face-detection-demo.mp4 -m /opt/intel/openvino/deployment_tools/tools/mod_downloader/Transportation/human_pose_estimation/mobilenet-v1/dldt/human-pose-estimation-0001-fp16.xml -d HETERO:FPGA,CPU

2. Open Google Chrome browser

3. Demo should freeze

[JohnT_Intel]

Hi.

We are able to duplicate this issue and already feedback this issue to engineering to fixed the issue.

[JohnT_Intel]

Hi,

We are still looking into the FPGA freeze issue.

[VDemc2]

Thanks for letting us know.

[VDemc2]

please keep us informed on the progress

[JohnT_Intel]

Hi,

We have tested the issue in upcoming release (2019R3) and it is no longer observed. The new OpenVINO 2019R3 is schedule to be available next week.

[VDemc2]

that's cool. Looking forward to test it on 2019R3 next week!

[JohnT_Intel]

Hi,

The 2019R3 has been released. Please test it out and let me know if you are still observing the issue

[VDemc2]

Hi JohnT,

it looks like the freezing issue has been resolved and it is working for both approaches:

1. Openvinotest application and kafka + zookeeper docker. https://github.com/VladoDemcak/ovdebug/blob/master/openvinotest.py
2. Human pose and chrome. But I don’t see better performance on HETERO:FPGA,CPU compared to CPU i7. Is it ok? What are your observations or what do you thing?

BTW I had another issue with the pose_estimation demo when it stuck on “Parsing input parameters” while running with HETERO:FPGA,CPU. In few cases the whole machine stopped and I needed to hard reset.

Maybe I was using invalid bitstream. Will try to test it next week and with our application.

Which bitstreams are suitable for human_pose_estimation model?

[JohnT_Intel]

Hi,

I am glad that you are able to make it work.

May I know which bitstream are you using to run on the Human Pose?

We performed the benchmark using 2019R3_PV_PL1_FP16_ELU or 2019R3_PV_PL1_FP11_YoloV3_ELU bitstream.

[VDemc2]

We use 2019R3_PV_PL1_FP16_MobileNet_Clamp.aocx

[JohnT_Intel]

Hi,

Could you check the performance using 2019R3_PV_PL1_FP16_ELU or 2019R3_PV_PL1_FP11_YoloV3_ELU bitstream?