Error running DLA Benchmark

[RubenPadial]

Hello,

I am working with Intel FPGA AI suite on a Intel Arria 10  SoC Dev Board.

I tried to run DLA benchmark several times and I receive the following messages  after some inferences:

"void MmdWrapper::WriteToDDR(int, uint64_t, uint64_t, const void*) const: Assertion `status == 0' failed"

The error comes after run the benchmark about 400 times

Intel FPGA AI Suite version: 2023.2 Running custom NN over A10_FP16_Performance.arch example.

Could anyone help me to avoid this problem?

Thank you in advance.

[JohnT_Intel]

Hi,

Just would like to clarify, your issue only observed after running 400 times? May I know which bitstream and which NN model are you using?

Please also provide me your board setup and DDR used?

[JohnT_Intel]

Hi,

May I know if you have the information requested?

[JohnT_Intel]

Hi RubenPadial,

Do you have update on this?

Thanks

[RubenPadial]

Hello @JohnT_Intel 

The problem comes after 400  dla benchmark execution approximately as I mentioned in the first message. I used the A10_FP16_Performance architecture to build the bitstream and a custom CNN.

I'm using Arria10 SoC Dev Kit with default memory configuration.

[JohnT_Intel]

Hi,

Can you share with me the step you use to generate the bitstream, board setup including memory used and if possible share with me your bitstream and model so that I can try to duplicate from my side to understand the issue?

[RubenPadial]

Hello @JohnT_Intel,

To generate the bitstream, you can follow the Intel FPGA AI Suite. SoC Design Example User Guide by changing the architure. You can find the IR for the NN in this link

[JohnT_Intel]

Hi,

May I know which DDR memory are you using in your setup? I am facing issue to download the file with the link you provided

[RubenPadial]

Hello @JohnT_Intel ,

Try again in this link. I'm using the default DDR configuration for Arria10 SoC Dev Kit: https://www.intel.la/content/www/xl/es/products/details/fpga/development-kits/arria/10-sx.html

[JohnT_Intel]

HI,

Just to confirm, you are using the HILO board shipped with the dev kit? Let me know if you are using different memory. Please also confirm you are using the correct memory on HPS and FPGA memory.

• HPS memory size (HiLo card):
  • 2 GB DDR3 (256 Mb x 40 x dual rank)
  • 1 GB DDR3 (256 Mb x 40 x single rank)
  • 1 GB DDR4 (256 Mb x 40 x single rank) - ships with kit
• FPGA memory size (HILO Card):
  • 4 GB DDR3 (256Mb x72 x dual rank)
  • 2 GB DDR3 (256Mb x72 x single rank)
  • 2 GB DDR4 (256Mb x 72 x single rank) - ships with kit
  • 16 MB QDRV (4Mb x 36)
  • 128 MB RLDRAM3(16Mb x 72)

[RubenPadial]

Hello @JohnT_Intel,

Yes, I'm using the HILO cards shipped with the dev kit.
Both are "ALTERA DDR4 x72 DAUGHTER CARD"

[JohnT_Intel]

Hi,

Are you running M2M or S2M? Can you confirm FPGA is connected to 2GB RAM and HPS is 1GB RAM?

[RubenPadial]

Hello @JohnT_Intel,

Both HILO cards seems to be equal. Any tip to know the memory size?

HPS "grep MemTotal /proc/meminfo" --> MemTotal: 1025000 kB 

[JohnT_Intel]

HI,

Have you try with the prebuilt bitstream and sdcard image (https://www.intel.com/content/www/us/en/docs/programmable/768979/2024-3/writing-the-sd-card-image-wic-to-an-sd-card.html)?

[JohnT_Intel]

Hi,

The HILO memory module should have the label on it. May I know if you are able to see the label?

[RubenPadial]

Hello @JohnT_Intel , both boards label show "ALTERA DDR4 x72 DAUGHTER CARD"

[JohnT_Intel]

Hi,

I try to convert the model provided by you and it looks like it is not be able to fully run in FPGA. The model will need to run on both CPU and FPGA.

I suspect that the hang issue might be due to the CPU side. The testing that was done is on Resnet model which can be fully run in FPGA.

Below is the summary of the analysis when converting the model.

List of reasons for unsupporting layers:

- Transpose_1482: General reshape is not supported when the input surface size (HxWxD) is greater than 10 or when CVEC != KVEC or when target_surface_size * CVEC (0) > 8192.

- Transpose_1482: General reshape is not supported when the target channel dimension is greater than the input channel dimension

Original graph per layer support information analysis:

Original Layer Name: Identity

 Causative reasons for CPU placement:

 - Transpose_1482: General reshape is not supported when the input surface size (HxWxD) is greater than 10 or when CVEC != KVEC or when target_surface_size * CVEC (0) > 8192.

 - Transpose_1482: General reshape is not supported when the target channel dimension is greater than the input channel dimension

Original Layer Name: Transpose_1482

 Causative reasons for CPU placement:

 - Transpose_1482: General reshape is not supported when the input surface size (HxWxD) is greater than 10 or when CVEC != KVEC or when target_surface_size * CVEC (0) > 8192.

 - Transpose_1482: General reshape is not supported when the target channel dimension is greater than the input channel dimension

Original Layer Name: Constant_5670

 Causative reasons for CPU placement:

 - Transpose_1482: General reshape is not supported when the input surface size (HxWxD) is greater than 10 or when CVEC != KVEC or when target_surface_size * CVEC (0) > 8192.

 - Transpose_1482: General reshape is not supported when the target channel dimension is greater than the input channel dimension

[RubenPadial]

Hello @JohnT_Intel,

But the graph obtained after compiling the model for the target architecture shows all the  laryers are computed on the FPGA. Please, find attached the .dot files

[JohnT_Intel]

Hi,

Can you try to run the model that we have verified to see if the issue persist as well?

[JohnT_Intel]

Hi,

If you look the svg file, the 1st 2 node is run on CPU and not FPGA.

<!-- Reshape_552 -->

<g id="node1" class="node">

<title>Reshape_552</title>

<polygon fill="cornflowerblue" stroke="black" points="4751,-202 4645,-202 4645,-119 4751,-119 4751,-202"/>

<text text-anchor="middle" x="4698" y="-186.8" font-family="Times,serif" font-size="14.00">Transpose_1482</text>

<text text-anchor="middle" x="4698" y="-171.8" font-family="Times,serif" font-size="14.00">Reshape_552</text>

<text text-anchor="middle" x="4698" y="-156.8" font-family="Times,serif" font-size="14.00">Reshape</text>

<text text-anchor="middle" x="4698" y="-141.8" font-family="Times,serif" font-size="14.00">subgraph=82</text>

<text text-anchor="middle" x="4698" y="-126.8" font-family="Times,serif" font-size="14.00">device=CPU</text>

</g>

<!-- Result_556 -->

<g id="node2" class="node">

<title>Result_556</title>

<polygon fill="cornflowerblue" stroke="crimson" stroke-width="1.5" points="4742,-83 4654,-83 4654,0 4742,0 4742,-83"/>

<text text-anchor="middle" x="4698" y="-67.8" font-family="Times,serif" font-size="14.00">Identity</text>

<text text-anchor="middle" x="4698" y="-52.8" font-family="Times,serif" font-size="14.00">Result_556</text>

<text text-anchor="middle" x="4698" y="-37.8" font-family="Times,serif" font-size="14.00">Result</text>

<text text-anchor="middle" x="4698" y="-22.8" font-family="Times,serif" font-size="14.00">subgraph=82</text>

<text text-anchor="middle" x="4698" y="-7.8" font-family="Times,serif" font-size="14.00">device=CPU</text>

</g>

[JohnT_Intel]

Hi,

May I know if you have any other queries or update on this?