Stuck during execution on Mustang-F100-A10, Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA

[VDemc2]

Hello,

we have Mustang F100-A10 and TANK-870 AIoT Dev. Kit.

We have followed the official guide for the latest openvino to setup the 2019R1 openvino and our FPGA card.

We were able to run "aocl diagnose" and we see there is the card as you can see below.

ieisw@ieisw-SER0:~$ aocl diagnose   --------------------------------------------------------------------   Device Name:   acl0       BSP Install Location:   /opt/altera/aocl-pro-rte/aclrte-linux64/board/a10_1150_sg1       Vendor: Intel(R) Corporation       Phys Dev Name Status Information       acla10_1150_sg10Passed Intel Vision Accelerator Design with Intel Arria 10 FPGA (acla10_1150_sg10)   PCIe dev_id = 2494, bus:slot.func = 01:00.00, Gen3 x8   FPGA temperature = 61.6406 degrees C.       DIAGNOSTIC_PASSED   --------------------------------------------------------------------  

The 

./demo_squeezenet_download_convert_run.sh -d HETERO:FPGA,CPU

succeeded as well as demo_security_barrier_camera.sh with device HETERO:FPGA,CPU

However when we try to run classification_sample demo as below, it sometimes stuck forever. Not after certain amount of inference calls but randomly. Sometimes it finished with -ni set to 3000 but sometimes it stuck when we set -ni to 10. Below is the command we used.

/<path>/inference_engine_samples_build/intel64/Release/classification_sample -i /opt/intel/openvino/deployment_tools/demo/car.png -m /<path>/squeezenet1.1_FP16/squeezenet1.1.xml -d HETERO:FPGA,CPU -ni 10

The same thing happens with our custom code, which runs without issue on CPU.

While this execution is running we are not able to call "aocl diagnose" at all.

We are really out of ideas and we are starting to suspect that the FPGA card is broken. So any help or suggestion is highly appreciated.

My second question is if it's possible to call "aocl diagnose" from two different terminals simultaneously.

PS: We have skipped the part in documentation which is about USB Blaster (we dont have this thing). 

[JohnT_Intel]

Hi,

There is some issue with the 2019R1 released for this board. Please upgrade your OpenVINO to 2019R1.1 released.

You may call "aocl diagnose" from two different terminals simultaneously as long as you are able to confirm that there is no application is running on the FPGA.

[VDemc2]

Hi,

thanks for your answer.

we downloaded "l_openvino_toolkit_fpga_p_2019.1.144" so we do have 2019R1.1.

[JohnT_Intel]

Hi,

Is that mean that you are also facing the same issue on 2019R1.1?

[VDemc2]

Yes we are facing the issue on 2019R1.1. We use the latest openvino.

Hovewer we haven't tried on 2019R1 (I thought R1.1 and R1 is almost the same so I incorrectly wrote 2019R1 in my original post). But I assume it wouldn't help.

We tried 2018R5, because our second FPGA, which we received and installed like 6months ago, worked well (we have 2 FPGAs, the new with the problem on 2019R1.1 and the second one - works well but not on the latest version of openvino)

Unfortunately with the new FPGA we were not able to call "aocl diagnose" with success message on 2018R1 version.

[JohnT_Intel]

Hi,

May I know how frequent are you observing the hang issue?

[VDemc2]

Hi,

it's like 95% for classification_sample demo from openvino toolkit when we set number of iterations to 10. When we run just classification_sample without number of iteration (without -ni parameter, so there is only 1 itteration) it works well without freezing problem.

But when we set number of iterations to 100+ it always freezes.

From stream input e.g. from camera it stuck every time after ~0-3 inferences so it's useless in cases like that.

[JohnT_Intel]

Hi,

We are in the progress of looking into the freeze issue and will be fixed in 2019R2 released.

[MČule]

Hi John, please do you know, when the 2019R2 for FPGA will be released? For now, there is just R1.1 available.

thank you!

Martin

[JohnT_Intel]

Hi,

The 2019R2 for FPGA is schedule to be released in 2 weeks time.

[VDemc2]

Hi JohnT,

now the 2019 for FPGA has been released. Could you please confirm if the issue is resolved? I cannot find it in Release notes.

We will definitely try it but would be good to know any details what was wrong etc.

In the near future we want to create FPGA docker for FPGA as it is described in the documentation. There is a similar freezing issue (marked as #31 in issue list) and it's still in the "Known Issues" (in Release notes) and it looks like it's still not resolved.

31Inference on FPGA freezes when run in Docker*Inference EngineN/A

Thanks

[JohnT_Intel]

Hi, If you look into the release note below, the bitstream for Mustang F100-A10 is not included. We are still in the progress of debugging the hang issue. "Bitstreams for Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Mustang-F100-A10) Speed Grade 1 and Speed Grade 2 are not included in the OpenVINO R2 release. These bitstreams will be added at a future date."

[VDemc2]

Hi,

we have prepared some steps how to reproduce our issue. I might help you in the process of debugging

https://github.com/VladoDemcak/ovdebug

[JohnT_Intel]

Hi, The link provided is issue observed when using docker container. If it is running without docker container then no issue is observed. May I know what is the reason that you need to run in docker container? The reason is that the demo application is not validated in docker container environment.

[VDemc2]

The link provided is issue observed when a FPGA program is running locally and there is another docker containers running on the same host - it this example kafka and zookeeper.

We would like to run FPGA in docker in the future, but for now we run FPGA locally however we have another components which run in docker.

[JohnT_Intel]

Hi,

From the link provided, I observed it mention that it is successfull process 3 time and 0 time failed when performed using "HETERO:FPGA,CPU without docker kafka"

Could you confirm if my understanding is correct?

[VDemc2]

Yes exactly ... but when we run the same application on the same host and there is running a docker container (eg kafka with zookeeper) it freezes.

[JohnT_Intel]

Hi,

I understand but unfortunately currently the OpenVINO for FPGA does not support docker as it is not validated in this setup

[VDemc2]

"OpenVINO for FPGA does not support docker" I am confused a bit.

We run FPGA on host (not in docker) and there are 2 docker containers running on the same host BUT he FPGA application does not communicate with those docker containers.

They run as separated processes.

I assume there is a simultaneous access from FPGA and docker engine which cause freezing issue.

[JohnT_Intel]

Hi,

What do you mean by running 2 docker container running on the same host? May I know what is the docker container is running?

The reason is that from your information, the issue is not observed when the docker engine is not run. This looks like it is related to your PC issue rather than FPGA issue.

[VDemc2]

The issue has been observed on two different computers with FPGA so I dont think it's related to PC/card issue.

As I stated in the beginning we have IEI Tank and Mustang F100 inside with Ubuntu.

We have docker installed on the Ubuntu

We have Openvino R1 installed on the Ubuntu.

When we run simple openvino application which runs on HETERO:FPGA, CPU device it worked and process whole video file which has around 600 frames.

When we start up docker-compose with completely separated logic (just kafka and zookeeper as message broker) and we run the simple openvino application again the openvino application freezes.

This freezing issue doesn't have anything with "running FPGA in docker" since as I mentioned we run openvino application locally on host (openvino is installed on host directly, aocl is installed and programmed locally directly on the host).

In short the freezing issue has been observed when we had following setup at once:

1. We have Message broker system (doesnt have anything related to openvino/FPGA) - We have 2 docker containers running ( official zookeeper image + kafka confluent official image)
2. We run openvino application with device HETERO:FPGA,CPU locally

When we stop both docker containers (docker engine is still running as service but no docker containers are running) and we run the openvino application it doesnt freeze.