MCDMA/PCIE max number of channels - how to get 256 physical channels - Page 2

grspbr · ‎01-05-2023

I am recreating this question because the older one was closed. There is a problem with notifications. I am not getting notifications of replies to my posts. How can I correct this?

To get to the problem, The MCDMA/PCIE core has 11 bits of DMA channel address when using multiple logical channels on one physical interface. We planned to use 256 channels (physically) but probably only a dozen or so logical channels. However, when we try to drive it using Intel's DPDK, we seem to get corruption of the D2H DMA descriptors if we use more than 64 channels. Why are we seeing this limitation? Is it in the DPDK itself? Also, it seems we cannot use logical channels that we can map to a physical channel address. It seems that the creation of channels automatically uses sequential physical channel addresses starting from 0. Is it not possible to map logical channels to physical channels?

To answer your question, we are using Quartus 21.2 for Agilex AGFB014 (P+E Tile).

In our build settings, we have:

PCIe0 Settings/PCIe0 IP Settings/PCIe0 PCI Express/PCI Capabilities/PCIe0 Device/PF0 = 256; and PCIe0 Settings/PCIe0 IP Settings/MCDMA Settings/D2H Prefetch channels = 256; and Maximum Descriptor Fetch = 16

grspbr · ‎01-19-2023

Hi @wchiah

I have sent you a link (by email) where you can retrieve the QAR files from our cloud server.

Best regards,

Greg

wchiah · ‎01-08-2023

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

grspbr · ‎01-09-2023

Hi Wincnet, I still need to review your last response. I should have something tomorrow.

Regards,

Greg

wchiah · ‎01-12-2023

Hi,

I wish to follow up with you about this IPS case.

Hoping to hear back from you so that we can proceed for next step.

Regards,

Wincent_Intel

wchiah · ‎01-17-2023

Hi Greg,

Thanks for the .qar file and evidence.

I had rise an internal engineering ticket for this.

The related team will work on it, will follow up closely, let you know if there is any update.

Regards,

Wei Chuan

wchiah · ‎01-18-2023

Hi Greg,

We might need perfq_log_20230106_143546.txt and 22.4 ED qar file.

can you put it in google drive and share me the link ?

Regards,

Wei Chuan

wchiah · ‎01-19-2023

Hi Greg,

I can access to your link.
Can you please attach perfq_log_20230106_143546.txt for Quartus v22.4 as well ?
with 256 channel fail log.

Regards,

Wincent_Intel.

grspbr · ‎01-24-2023

Hi Wincent,

Unfortunately, I don't have the compiled project anymore for v22.4. But it behaved exactly the same as v21.2. It was an intel example project that I just changed the settings for 256 channel.

Regards,

Greg

wchiah · ‎01-25-2023

Hi Greg,

I have tested 22.4 acs branch and 22.4 acs Qshell using design example from Quartus with given command, and not observing any issue.

Please find bitstream and release software in attachment. Also confirm are you using DCA enabled ZTE sof? if so this Quartus build won't support DCA feature.

Test output:

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 8192 -l 2 -z -c 64 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

Allocating 64 Channels...

----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 64

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 8192 Bytes

D2H Payload Size per descriptor: 8192 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 8192 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 8192 Bytes

PKG Gen Files: 1

File Size: 8192 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues Time_elpsd B_trnsfrd TBW d_drop_cnt Passed Failed %passed

Tx 64 00:03:457 26185368.00KB 17.11GBPS 0 64 0 100.00%

Rx 64 00:03:457 24388064.00KB 09.46GBPS 0 64 0 100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 26.57GBPS, 3.48MPPS

Total TX Bandwidth: 17.11GBPS, 2.24MPPS

Total RX Bandwidth: 9.46GBPS, 1.24MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW: Total Bandwidth

IBW: Interval Bandwidth

MIBW: Mean Interval Bandwidth

HIBW: Highest Interval Bandwidth

LIBW: Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120711.txt for more details

total_drops:0

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 8192 -l 2 -z -c 65 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

Allocating 65 Channels...

----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 65

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 8192 Bytes

D2H Payload Size per descriptor: 8192 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 8192 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 8192 Bytes

PKG Gen Files: 1

File Size: 8192 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues Time_elpsd B_trnsfrd TBW d_drop_cnt Passed Failed %passed

Tx 65 00:03:617 28762960.00KB 16.93GBPS 0 65 0 100.00%

Rx 65 00:03:617 25945592.00KB 09.44GBPS 0 65 0 100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 26.38GBPS, 3.46MPPS

Total TX Bandwidth: 16.93GBPS, 2.22MPPS

Total RX Bandwidth: 9.44GBPS, 1.24MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW: Total Bandwidth

IBW: Interval Bandwidth

MIBW: Mean Interval Bandwidth

HIBW: Highest Interval Bandwidth

LIBW: Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120720.txt for more details

total_drops:0

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 256 -l 2 -z -c 65 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

Allocating 65 Channels...

----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 65

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 256 Bytes

D2H Payload Size per descriptor: 256 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 256 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 256 Bytes

PKG Gen Files: 1

File Size: 256 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues Time_elpsd B_trnsfrd TBW d_drop_cnt Passed Failed %passed

Tx 65 00:03:902 15322550.00KB 07.68GBPS 0 65 0 100.00%

Rx 65 00:03:902 1771967.50KB 00.58GBPS 0 65 0 100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 8.26GBPS, 34.65MPPS

Total TX Bandwidth: 7.68GBPS, 32.21MPPS

Total RX Bandwidth: 0.58GBPS, 2.44MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW: Total Bandwidth

IBW: Interval Bandwidth

MIBW: Mean Interval Bandwidth

HIBW: Highest Interval Bandwidth

LIBW: Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120734.txt for more details

total_drops:0

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 256 -l 2 -z -c 256 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 8086:15f3 net_igc

Allocating 256 Channels...

----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 256

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 256 Bytes

D2H Payload Size per descriptor: 256 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 256 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 256 Bytes

PKG Gen Files: 1

File Size: 256 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues Time_elpsd B_trnsfrd TBW d_drop_cnt Passed Failed %passed

Tx 256 00:03:976 8357743.00KB 04.02GBPS 0 256 0 100.00%

Rx 256 00:03:976 6494101.75KB 02.08GBPS 0 256 0 100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 6.10GBPS, 25.57MPPS

Total TX Bandwidth: 4.02GBPS, 16.86MPPS

Total RX Bandwidth: 2.08GBPS, 8.71MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW: Total Bandwidth

IBW: Interval Bandwidth

MIBW: Mean Interval Bandwidth

HIBW: Highest Interval Bandwidth

LIBW: Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120746.txt for more details

total_drops:0

[root@BAPVECISE040T perfq]#

wchiah · ‎01-29-2023

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

wchiah · ‎01-30-2023

Hi

We have not hear from you and this Case is idling. It is not recommended to idle for too long.

Therefore following our support policy, I have to put this case in close status. My apologies if any inconvenience cause

Hence, This thread will be transitioned to community support.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If you feel your support experience was less than a 9 or 10,

please allow me to correct it before closing or let me know the cause so that I may improve your future support experience.

Regards,

Wincent_Intel

grspbr · ‎02-09-2023

Hi @wchiah

I'm sorry for the long delay but as we went with a workaround which was to use 64 channels, we were able to move forward. Still we will try again with project you sent when time permits.

I have looked at the package you sent and I have a couple of notes: 1) contains Stratix 10 target whereas we use Agilex, so IP had to be upgraded, which was successful; 2) Settings did not include CVP which is how we load the design - presumably not a problem; 3) We need to provide PCIE pinout to try in our eval board; 4) Design does indeed use 256 channels and I noted that the test report indicated it was 100% successful when configured for more than 64 channels, up to 256 channels - we will try again from our end when time permits and open another thread if necessary.

Thank you for your support. You were very helpful.

Regards,

Greg