we have Mustang F100-A10 and TANK-870 AIoT Dev. Kit.
We have followed the official guide for the latest openvino to setup the 2019R1 openvino and our FPGA card.
We were able to run "aocl diagnose" and we see there is the card as you can see below.
ieisw@ieisw-SER0:~$ aocl diagnose -------------------------------------------------------------------- Device Name: acl0 BSP Install Location: /opt/altera/aocl-pro-rte/aclrte-linux64/board/a10_1150_sg1 Vendor: Intel(R) Corporation Phys Dev Name Status Information acla10_1150_sg10Passed Intel Vision Accelerator Design with Intel Arria 10 FPGA (acla10_1150_sg10) PCIe dev_id = 2494, bus:slot.func = 01:00.00, Gen3 x8 FPGA temperature = 61.6406 degrees C. DIAGNOSTIC_PASSED --------------------------------------------------------------------
./demo_squeezenet_download_convert_run.sh -d HETERO:FPGA,CPU
succeeded as well as demo_security_barrier_camera.sh with device HETERO:FPGA,CPU
However when we try to run classification_sample demo as below, it sometimes stuck forever. Not after certain amount of inference calls but randomly. Sometimes it finished with -ni set to 3000 but sometimes it stuck when we set -ni to 10. Below is the command we used.
/<path>/inference_engine_samples_build/intel64/Release/classification_sample -i /opt/intel/openvino/deployment_tools/demo/car.png -m /<path>/squeezenet1.1_FP16/squeezenet1.1.xml -d HETERO:FPGA,CPU -ni 10
The same thing happens with our custom code, which runs without issue on CPU.
While this execution is running we are not able to call "aocl diagnose" at all.
We are really out of ideas and we are starting to suspect that the FPGA card is broken. So any help or suggestion is highly appreciated.
My second question is if it's possible to call "aocl diagnose" from two different terminals simultaneously.
PS: We have skipped the part in documentation which is about USB Blaster (we dont have this thing).
There is some issue with the 2019R1 released for this board. Please upgrade your OpenVINO to 2019R1.1 released.
You may call "aocl diagnose" from two different terminals simultaneously as long as you are able to confirm that there is no application is running on the FPGA.
Yes we are facing the issue on 2019R1.1. We use the latest openvino.
Hovewer we haven't tried on 2019R1 (I thought R1.1 and R1 is almost the same so I incorrectly wrote 2019R1 in my original post). But I assume it wouldn't help.
We tried 2018R5, because our second FPGA, which we received and installed like 6months ago, worked well (we have 2 FPGAs, the new with the problem on 2019R1.1 and the second one - works well but not on the latest version of openvino)
Unfortunately with the new FPGA we were not able to call "aocl diagnose" with success message on 2018R1 version.
it's like 95% for classification_sample demo from openvino toolkit when we set number of iterations to 10. When we run just classification_sample without number of iteration (without -ni parameter, so there is only 1 itteration) it works well without freezing problem.
But when we set number of iterations to 100+ it always freezes.
From stream input e.g. from camera it stuck every time after ~0-3 inferences so it's useless in cases like that.
now the 2019 for FPGA has been released. Could you please confirm if the issue is resolved? I cannot find it in Release notes.
We will definitely try it but would be good to know any details what was wrong etc.
In the near future we want to create FPGA docker for FPGA as it is described in the documentation. There is a similar freezing issue (marked as #31 in issue list) and it's still in the "Known Issues" (in Release notes) and it looks like it's still not resolved.
31Inference on FPGA freezes when run in Docker*Inference EngineN/A
The link provided is issue observed when a FPGA program is running locally and there is another docker containers running on the same host - it this example kafka and zookeeper.
We would like to run FPGA in docker in the future, but for now we run FPGA locally however we have another components which run in docker.
From the link provided, I observed it mention that it is successfull process 3 time and 0 time failed when performed using "HETERO:FPGA,CPU without docker kafka"
Could you confirm if my understanding is correct?
Yes exactly ... but when we run the same application on the same host and there is running a docker container (eg kafka with zookeeper) it freezes.
"OpenVINO for FPGA does not support docker" I am confused a bit.
We run FPGA on host (not in docker) and there are 2 docker containers running on the same host BUT he FPGA application does not communicate with those docker containers.
They run as separated processes.
I assume there is a simultaneous access from FPGA and docker engine which cause freezing issue.
What do you mean by running 2 docker container running on the same host? May I know what is the docker container is running?
The reason is that from your information, the issue is not observed when the docker engine is not run. This looks like it is related to your PC issue rather than FPGA issue.
The issue has been observed on two different computers with FPGA so I dont think it's related to PC/card issue.
As I stated in the beginning we have IEI Tank and Mustang F100 inside with Ubuntu.
We have docker installed on the Ubuntu
We have Openvino R1 installed on the Ubuntu.
When we run simple openvino application which runs on HETERO:FPGA, CPU device it worked and process whole video file which has around 600 frames.
When we start up docker-compose with completely separated logic (just kafka and zookeeper as message broker) and we run the simple openvino application again the openvino application freezes.
This freezing issue doesn't have anything with "running FPGA in docker" since as I mentioned we run openvino application locally on host (openvino is installed on host directly, aocl is installed and programmed locally directly on the host).
In short the freezing issue has been observed when we had following setup at once:
When we stop both docker containers (docker engine is still running as service but no docker containers are running) and we run the openvino application it doesnt freeze.