Unable to run OpenVINO on three or more FPGAs

[Tony7]

Hi,

I have created this post since my previous post is no longer monitored. The problem is that OpenVINO works on any two of the three FPGAs but not on all three FPGAs. In my previous post, the suggested solution is to follow the steps in FPGA Plugin - OpenVINO Toolkit(version 2018_R5). Since I am using a newer version of OpenVINO, I have followed the steps in FPGA Plugin - OpenVINO(version 2020.03) instead.

This is the source code of my program, which loads two different models on the three FPGAs.

InferenceEngine::Core core;
auto network1 = core.ReadNetwork("/home/autodrive/dla/new_openvino_test/benchmark/zeus_inference_engine/SqueezeDetTest/lightNSign_720x1280_noMean_bgr_R5.xml");
auto network2 = core.ReadNetwork("/home/autodrive/planning_ws/src/fpga_stuff/squeezenet/objectDet_384x1248_noMean_bgr_R5.xml");
auto network3 = core.ReadNetwork("/home/autodrive/planning_ws/src/fpga_stuff/squeezenet/objectDet_384x1248_noMean_bgr_R5.xml");

auto exeNetwork1 = core.LoadNetwork(network1, "FPGA.0");
auto exeNetwork3 = core.LoadNetwork(network3, "FPGA.2");
auto exeNetwork2 = core.LoadNetwork(network2, "FPGA.1");

I ran the program, and the following error was shown in the console.

Error initializing DMA: 5
Error initializing mmd dma
Error initializing bsp
Error Code: 147
Error Description: Error: Failure due to generic standard exception.User requested device 1 is invalid
File: /home/jenkins/agent/workspace/private-ci/ie/build-linux-ubuntu16/b/repos/fpga-plugin/thirdparty/dla/runtime/core/src/new_device.cpp
Function: initialize
Line #: 101
terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
what(): Failed to create FPGA device handle with ID 1
Aborted (core dumped)

Regards,

Tony

[Zulkifli_Intel]

Hello Tony,

Greetings to you.

Although you run 3 FPGA cards simultaneously, but the same model was running in 2 FPGA cards which as our solution provided, multiple

FPGA devices work with different models in each respective card.

Please try our proposed configuration:

Card 1: Object Detection with SSD-VGG Sample

Card 2: Image Segmentation Sample with FCN-8.

Card 3: Object Detection for Faster R-CNN Sample

Supported OpenVINO FPGA Configuration

• FPGA with the same model - maximum 2 cards
• FPGA with a different model - more than 2 cards with different models in each FPGA cards

Regards,

Zulkifli

[Tony7]

Hi Zulkifli,

I have tried the proposed solutions with different models in each FPGA card, but I am still getting the same error. 

Here is the configuration:

card0: Alexnet

card1: SqueezeNet

card2: ResNet-50

Here is the source code:

InferenceEngine::Core core;
    auto network1 = core.ReadNetwork("/home/autodrive/planning_ws/src/fpga_stuff/alexnet_models/traffic_lights.xml");
    auto network2 = core.ReadNetwork("/home/autodrive/planning_ws/src/fpga_stuff/squeezenet/objectDet_384x1248_noMean_bgr_R5.xml");
    auto network3 = core.ReadNetwork("/home/autodrive/planning_ws/src/fpga_stuff/resnet_models/lightNSign_384x1248_noMean_bgr_R5.xml");

   auto exeNetwork1 = core.LoadNetwork(network1, "FPGA.0");
   auto exeNetwork3 = core.LoadNetwork(network3, "FPGA.2");
   auto exeNetwork2 = core.LoadNetwork(network2, "FPGA.1");

Here is the error:

(aocl) autodrive@AUTODRIVE-BOLT:~/dla/new_openvino_test/benchmark_build$ ./devel/lib/zeus_inference_engine/YoloTest 
Error initializing DMA: 5
Error initializing mmd dma
Error initializing bsp
Error Code: 147
Error Description: Error: Failure due to generic standard exception.User requested device 1 is invalid
File: /home/jenkins/agent/workspace/private-ci/ie/build-linux-ubuntu16/b/repos/fpga-plugin/thirdparty/dla/runtime/core/src/new_device.cpp
Function: initialize
Line #: 101
terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
  what():  Failed to create FPGA device handle with ID 1
Aborted (core dumped)

Regards,

Tony

[Zulkifli_Intel]

Hello Tony,

We suggest you use Open Model Zoo models to run the test (such as squeezenet and the security barrier camera) to isolate a problem and eliminate a root cause with custom models.

Regards,

Zulkifli

[Zulkifli_Intel]

Hello Tony C,

This thread will no longer be monitored since we have provided information on FPGA configuration. If you need any additional information from Intel, please submit a new question.

Regards,

Zulkifli