FPGA Intellectual Property
PCI Express*, Networking and Connectivity, Memory Interfaces, DSP IP, and Video IP
Announcements
The Intel sign-in experience is changing in February to support enhanced security controls. If you sign in, click here for more information.
6065 Discussions

MCDMA/PCIE max number of channels - how to get 256 physical channels

grspbr
Novice
1,118 Views

Hi @wchiah ,

I am recreating this question because the older one was closed. There is a problem with notifications. I am not getting notifications of replies to my posts. How can I correct this?

To get to the problem, The MCDMA/PCIE core has 11 bits of DMA channel address when using multiple logical channels on one physical interface. We planned to use 256 channels (physically) but probably only a dozen or so logical channels. However, when we try to drive it using Intel's DPDK, we seem to get corruption of the D2H DMA descriptors if we use more than 64 channels. Why are we seeing this limitation? Is it in the DPDK itself? Also, it seems we cannot use logical channels that we can map to a physical channel address. It seems that the creation of channels automatically uses sequential physical channel addresses starting from 0. Is it not possible to map logical channels to physical channels?

To answer your question, we are using Quartus 21.2 for Agilex AGFB014 (P+E Tile).

In our build settings, we have:

PCIe0 Settings/PCIe0 IP Settings/PCIe0 PCI Express/PCI Capabilities/PCIe0 Device/PF0 = 256; and PCIe0 Settings/PCIe0 IP Settings/MCDMA Settings/D2H Prefetch channels = 256; and Maximum Descriptor Fetch = 16

 

0 Kudos
31 Replies
grspbr
Novice
169 Views

Hi @wchiah 

 

I have sent you a link (by email) where you can retrieve the QAR files from our cloud server.

Best regards,

Greg

 

wchiah
Employee
389 Views

Hi,

 

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

 

Regards,

Wincent_Intel


grspbr
Novice
367 Views

Hi Wincnet, I still need to review your last response.  I should have something tomorrow.

Regards,

Greg

 

wchiah
Employee
244 Views

Hi,

I wish to follow up with you about this IPS case.

Hoping to hear back from you so that we can proceed for next step.

Regards,

Wincent_Intel


wchiah
Employee
196 Views

Hi Greg,


Thanks for the .qar file and evidence.

I had rise an internal engineering ticket for this.

The related team will work on it, will follow up closely, let you know if there is any update.


Regards,

Wei Chuan


wchiah
Employee
174 Views

Hi Greg,


We might need perfq_log_20230106_143546.txt and 22.4 ED qar file.

can you put it in google drive and share me the link ?


Regards,

Wei Chuan


wchiah
Employee
157 Views

Hi Greg,

I can access to your link.
Can you please attach perfq_log_20230106_143546.txt for Quartus v22.4 as well ?
with 256 channel fail log.

Regards,

Wincent_Intel.

grspbr
Novice
136 Views

Hi Wincent,

Unfortunately, I don't have the compiled project anymore for v22.4. But it behaved exactly the same as v21.2. It was an intel example project that I just changed the settings for 256 channel. 

 

Regards,

Greg

 

wchiah
Employee
117 Views

Hi Greg,


I have tested 22.4 acs branch and 22.4 acs Qshell using design example from Quartus with given command, and not observing any issue.

Please find bitstream and release software in attachment. Also confirm are you using DCA enabled ZTE sof? if so this Quartus build won't support DCA feature. 


Test output:

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 8192 -l 2 -z -c 64 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

Allocating 64 Channels...


----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 64

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 8192 Bytes

D2H Payload Size per descriptor: 8192 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 8192 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 8192 Bytes

PKG Gen Files: 1

File Size: 8192 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues  Time_elpsd   B_trnsfrd      TBW   d_drop_cnt Passed Failed %passed

Tx   64   00:03:457   26185368.00KB  17.11GBPS    0    64   0   100.00%

Rx   64   00:03:457   24388064.00KB  09.46GBPS    0    64   0   100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 26.57GBPS, 3.48MPPS

Total TX Bandwidth: 17.11GBPS, 2.24MPPS

Total RX Bandwidth: 9.46GBPS, 1.24MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW:      Total Bandwidth

IBW:      Interval Bandwidth

MIBW:      Mean Interval Bandwidth

HIBW:      Highest Interval Bandwidth

LIBW:      Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120711.txt for more details

     total_drops:0

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 8192 -l 2 -z -c 65 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

Allocating 65 Channels...


----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 65

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 8192 Bytes

D2H Payload Size per descriptor: 8192 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 8192 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 8192 Bytes

PKG Gen Files: 1

File Size: 8192 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues  Time_elpsd   B_trnsfrd      TBW   d_drop_cnt Passed Failed %passed

Tx   65   00:03:617   28762960.00KB  16.93GBPS    0    65   0   100.00%

Rx   65   00:03:617   25945592.00KB  09.44GBPS    0    65   0   100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 26.38GBPS, 3.46MPPS

Total TX Bandwidth: 16.93GBPS, 2.22MPPS

Total RX Bandwidth: 9.44GBPS, 1.24MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW:      Total Bandwidth

IBW:      Interval Bandwidth

MIBW:      Mean Interval Bandwidth

HIBW:      Highest Interval Bandwidth

LIBW:      Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120720.txt for more details

     total_drops:0

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 256 -l 2 -z -c 65 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

Allocating 65 Channels...


----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 65

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 256 Bytes

D2H Payload Size per descriptor: 256 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 256 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 256 Bytes

PKG Gen Files: 1

File Size: 256 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues  Time_elpsd   B_trnsfrd      TBW   d_drop_cnt Passed Failed %passed

Tx   65   00:03:902   15322550.00KB  07.68GBPS    0    65   0   100.00%

Rx   65   00:03:902   1771967.50KB   00.58GBPS    0    65   0   100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 8.26GBPS, 34.65MPPS

Total TX Bandwidth: 7.68GBPS, 32.21MPPS

Total RX Bandwidth: 0.58GBPS, 2.44MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW:      Total Bandwidth

IBW:      Interval Bandwidth

MIBW:      Mean Interval Bandwidth

HIBW:      Highest Interval Bandwidth

LIBW:      Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120734.txt for more details

     total_drops:0

[root@BAPVECISE040T perfq]# ./build/mcdma-test -- -b 0000:01:00.0 -p 256 -l 2 -z -c 256 -a 1 -d 4

EAL: Detected 16 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 1172:0 net_mcdma

PMD: ifc_mcdma_get_hw_version(): MCDMA RTL VERSION : 0x50001

PMD: ifc_mcdma_get_device_caps(): Max Supported by DPDK: 1024

EAL: PCI device 0000:04:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

EAL: PCI device 0000:05:00.0 on NUMA socket -1

EAL:  Invalid NUMA socket, default to 0

EAL:  probe driver: 8086:15f3 net_igc

Allocating 256 Channels...


----------------------------------------------------

BDF: 0000:01:00.0

Channels Allocated: 256

QDepth 508

Number of pages: 8

Completion mode: WB

H2D Payload Size per descriptor: 256 Bytes

D2H Payload Size per descriptor: 256 Bytes

H2D SOF on descriptor: 1

H2D EOF on descriptor: 1

H2D File Size: 256 Bytes

D2H SOF on descriptor: 1

D2H EOF on descriptor: 1

D2H File Size: 256 Bytes

PKG Gen Files: 1

File Size: 256 Bytes

Tx Batch Size: 127 Descriptors

Rx Batch Size: 127 Descriptors

TID FIFO Checks: OFF

AVST Example Design Interface

DCA: OFF

----------------------------------------------------------

Thread initialization in progress ...

Thread is in READY state...

Thread initialization done

All Threads exited

TIME OUT while waiting for completions

Leaving...

-------------------------------------OUTPUT SUMMARY------------------------------------------

Dir #queues  Time_elpsd   B_trnsfrd      TBW   d_drop_cnt Passed Failed %passed

Tx   256   00:03:976   8357743.00KB   04.02GBPS    0    256   0   100.00%

Rx   256   00:03:976   6494101.75KB   02.08GBPS    0    256   0   100.00%

----------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------

Total Bandwidth: 6.10GBPS, 25.57MPPS

Total TX Bandwidth: 4.02GBPS, 16.86MPPS

Total RX Bandwidth: 2.08GBPS, 8.71MPPS

Total data drop count :0

---------------------------------------------------------------------------------------------

Full Forms:

TBW:      Total Bandwidth

IBW:      Interval Bandwidth

MIBW:      Mean Interval Bandwidth

HIBW:      Highest Interval Bandwidth

LIBW:      Lowest Interval Bandwidth

Please refer to perfq_log_20230125_120746.txt for more details

     total_drops:0


[root@BAPVECISE040T perfq]#



wchiah
Employee
95 Views

Hi,

 

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

 

Regards,

Wincent_Intel


wchiah
Employee
89 Views

Hi

 

We have not hear from you and this Case is idling. It is not recommended to idle for too long.

Therefore following our support policy, I have to put this case in close status. My apologies if any inconvenience cause

Hence, This thread will be transitioned to community support.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If you feel your support experience was less than a 9 or 10,

please allow me to correct it before closing or let me know the cause so that I may improve your future support experience.

 

Regards,

Wincent_Intel


Reply