OFS : Data Path of QSFP from HE_HSSI

[attdoner]

Hi Team,

I am using N6001 with Intel OFS (OFS PCIe Attach) . I wanted to test data transfer from Host1 to Host2 as below.
Host1 -> N6001(Port0) -> OtherNIC(Port0) -> Host2.
Is there any sample application available in OPAE/OFS which performs this transaction? Can you please suggest.

In addition to above query, I wants to know how HE-HSSI sends data to QSFP?. The hssi application shows "No eth interface, so not honoring --eth-loopback". ( https://ofs.github.io/ofs-2024.1-1/hw/n6001/user_guides/ug_qs_ofs_n6001/ug_qs_ofs_n6001/  Section 7. Send traffic through the 10G AFU). Does this mean the HE_HSSI will not communicate with E-Tile or QSFPs (FIM) ?

Do I need to enable any parameters to pass over QSFPs?

Thankyou.

[JohnT_Intel]

Hi,

From the screenshot provided, you enable the loopback on your HSSI testing. You will need to disable to loopback if you are communicating between 2 boards.

You may refer to https://ofs.github.io/ofs-2024.2-1/sw/fpga_tools/hssi/hssi/#description on the argument available.

Thanks.

[attdoner]

Hi @JohnT_Intel ,
Thanks for the update.
I will test and update the same.

In addition to my queries, whether the ice driver will list any interface in Host for N6001? So that interface will list under 'ip a'/'ifconfig'. With this interface, can I create any application to send and receive packets from board to board without HSSI in-built application?. Is it possible to run DPDK application in this FIM?

Best regards,
Ayshathul Thuhara

[JohnT_Intel]

Hi,

Currently there is no way to run DPDK on this design as it. We do not have application or driver for it to show in the OS level such ipconfig command.

[attdoner]

Hi @JohnT_Intel ,

I hope you’re doing well. I’ve noticed that there’s an IFPGA driver in DPDK designed for the N6000 & N3000. Would it be feasible to port this driver for the N6001 design with minimal modifications?

Thank you for your insights!

[JohnT_Intel]

Hi,

Unfortunately that is not possible. The reason is that the DPDK is communicating to the E810 NIC controller directly through PCIe x4 interface.

Thanks.

John Tio

[JohnT_Intel]

Hi,

May I know if you have any other queries?

[JohnT_Intel]

We do not receive any response from you to the previous question/reply/answer that I have provided. This thread will be transitioned to community support. If you have a new question, feel free to open a new thread to get the support from Intel experts. Otherwise, the community users will continue to help you on this thread. Thank you.

[attdoner]

Hi @JohnT_Intel ,

Sorry for the inconvenience.
I was out of Office for many days.
I was testing these  HSSI applications with --eth-loopback off parameter. But I could not see any packets transferred/received in another PC(where another NIC is connected).  I was checking ifconfig <nic-interface> | grep 'RX\ packets'  in another PC to know whether packets are received or not.
Is there any other applications/method I can validate that the packets moves out of QSFP1 of N6001 to QSFP2 of other NIC?

Thanks and regards,
Ayshathul Thuhara

[JohnT_Intel]

HI,

Have you turn off the ethernet loopback on both board? The reason is that the loopback is turn on by default.

Can you provide the log file of 2 system after you have run the "hssi" cmd. Please provide the log after running this command "hssistats [-h] [--pcie-address PCIE_ADDRESS, -P PCIE_ADDRESS]"

[attdoner]

Hi @JohnT_Intel ,
Yes I had disabled using --eth-loopback=off. Below logs contains interface stats after executing hssi commands.

In Host1 where N6001 is connected,
Log:  "sudo hssi hssi_10g --eth-loopback off --num-packets 100"

log: sudo hssistats

In Host2 where Other NIC,
user@user-OptiPlex-7080:~$ sudo ifconfig enp1s0f0np0
enp1s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 04:3f:72:a4:11:bc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Host2 (NIC2) is validated with other tests and it receives data. But HE-HSSI applications the packet counts are 0

[JohnT_Intel]

Hi,

Do you know the other host Ethernet address? If you look at the screenshot provided, it is not honoring the ethernet loopback off due to not able to find the ethernet destination. You might need to specify the receiver address.

Can you print out the receiver FPGA ethernet information as well?

[attdoner]

Hi @JohnT_Intel  ,
Thanks for your response.

I had even tried adding destination address using --dst-addr in hssi applicaition. Still I was not able to detect the packet increment count in ifconfig.

Host1:

$ sudo hssi hssi_10g --eth-loopback off --num-packets 50 --src-addr 11:22:33:44:55:66 --dest-addr 04:3f:72:a4:11:bc --src-port 0
10G loopback test
Tx/Rx port: 99
Tx port: 0
Rx port: 0
eth_loopback: off
he_loopback: none
num_packets: 50
packet_length: 64
src_address: 11:22:33:44:55:66
(bits): 0x665544332211
dest_address: 04:3f:72:a4:11:bc
(bits): 0xbc11a4723f04
random_length: fixed
random_payload: incremental
rnd_seed0: 5eed0000
rnd_seed1: 5eed0001
rnd_seed2: 25eed
eth:

No eth interface, so not honoring --eth-loopback. Try using the hssiloopback command instead.
0x40000 ETH_AFU_DFH: 0x1000010000001000
0x40008 ETH_AFU_ID_L: 0xbb370242ac130002
0x40010 ETH_AFU_ID_H: 0x823c334c98bf11ea
0x40030 TRAFFIC_CTRL_CMD: 0x0000000000000000
0x40038 TRAFFIC_CTRL_DATA: 0x0000000100000000
0x40040 TRAFFIC_CTRL_PORT_SEL: 0x0000000000000000
0x40048 AFU_SCRATCHPAD: 0x0000000045324511

0x3c00 number_packets: 0x00000032
0x3c01 random_length: 0x00000000
0x3c02 random_payload: 0x00000000
0x3c03 start: 0x00000000
0x3c04 stop: 0x00000000
0x3c05 source_addr0: 0x44332211
0x3c06 source_addr1: 0x00006655
0x3c07 dest_addr0: 0xa4723f04
0x3c08 dest_addr1: 0x0000bc11
0x3c09 packet_tx_count: 0x00000032
0x3c0a rnd_seed0: 0x5eed0000
0x3c0b rnd_seed1: 0x5eed0001
0x3c0c rnd_seed2: 0x00025eed
0x3c0d pkt_length: 0x00000040
0x3cf4 tx_sta_tstamp: 0x029238c8
0x3cf5 tx_end_tstamp: 0x02923a90
0x3d00 num_pkt: 0xffffffff
0x3d01 pkt_good: 0x00000000
0x3d02 pkt_bad: 0x00000000
0x3d07 avst_rx_err: 0x00000000
0x3d0b rx_sta_tstamp: 0x029427da
0x3d0c rx_end_tstamp: 0xaa91bc27
0x3e00 mac_loop: 0x00000000

HSSI performance:
Selected clock frequency : 402.832 MHz
Latency minimum : 529638 ns
Latency maximum : 6.99681e+09 ns
Achieved Tx throughput : 19.7882 GB/s

No eth interface, so not showing stats. Try using the hssistats command instead.

In Host2,

user@user-OptiPlex-7080:~$ sudo ifconfig enp1s0f0np0
enp1s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::694b:f9a2:b595:f22b prefixlen 64 scopeid 0x20<link>
ether 04:3f:72:a4:11:bc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[JohnT_Intel]

Hi,

Just would like to understand, are you connecting to the Host 2 normal ethernet port? Can you provide me the detail diagram of host1 to host2 connection?

[attdoner]

Hi @JohnT_Intel ,

I am connecting N6001 in Host1 and Mellanox NIC in Host2.

In Host1:
Application : HSSI, HSSISTATS

In Host2:
Application : ifconfig port0(interface), tcpdump on port0(interface). -- interface handled using Mellanox driver.

Its not normal ethernet (RJ45). I am using QSFP module on both the end devices connected via MTP cable. (Optical cable)

[JohnT_Intel]

Hi,

Do you face the same issue when you have the connection loopback from port 0 to port 1 of the FPGA board?

[attdoner]

Hi @JohnT_Intel ,

Testcase1:  In Loopback mode: Server1

Here, In server1 I am using HSSI application to send packets. I could see the packets received using hssistats in same Server1. I could see out the values of TX and RX packet counts in hssistats application. This testcase works.

Testcase2:  In External Board to Board mode: Server1 -> Server2

Here, In server1 I am using HSSI application to send packets. In server2 I am using normal ifconfig. I could see no packets received in Server2. I could see out the values of TX counts in hssistats application in Server1. Even executed tcpdump in server2. This testcase doesnot work.

Let me know is there anyway I can show that HSSI sending packets to other cards. Since I have only one N6001 and a Mellanox card.

[JohnT_Intel]

Hi,

Can you try running "sudo fpgainfo phy -B 0xb1"?

Can you also try disabling the loopback by "sudo hssiloopback --loopback disable --pcie-address 0000:b1:00.0"?

[attdoner]

Hi @JohnT_Intel 

[user@user ~]$ sudo fpgainfo phy -B 0x8
Intel Acceleration Development Platform N6001
Board Management Controller NIOS FW version: 131.15.0 
Board Management Controller Build version: 3.15.0 
//****** PHY ******//
Interface                        : DFL
Object Id                        : 0xF100000
PCIe s:b:d.f                     : 0000:08:00.0
Vendor Id                        : 0x8086
Device Id                        : 0xBCCE
SubVendor Id                     : 0x8086
SubDevice Id                     : 0x1771
Socket Id                        : 0x00
Ports Num                        : 01
Bitstream Id                     : 0x5010202FB22E12D
Bitstream Version                : 5.0.1
Pr Interface Id                  : 8b153d79-5e54-5416-8096-25480eb9b02f
QSFP0                            : Connected 
QSFP1                            : Connected 
//****** HSSI information ******//
HSSI version                     : 1.0  
Number of ports                  : 8  
Port0                            :25GbE         UP
Port1                            :25GbE         UP
Port2                            :25GbE         UP
Port3                            :25GbE         UP
Port4                            :25GbE         UP
Port5                            :25GbE         UP
Port6                            :25GbE         UP
Port7                            :25GbE         UP
[user@user ~]$

[user@user ~]$ sudo hssiloopback --pcie-address 0000:08:00.0 --loopback disable
args Namespace(loopback='disable', ncsi_ch_sel=None, pcie_address='0000:08:00.0', port=0)
sbdf: 0000:08:00.0
FPGA dev: {'segment': 0, 'bus': 8, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:08:00.0'}
args.hssi_grps[('dfl_dev.3', ['/sys/bus/pci/devices/0000:08:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.3/uio/uio1'], '0000:08:00.0', 21)]
fpga uio dev:dfl_dev.3

--------HSSI INFO START-------
DFH                     :0x3000000010002015
HSSI ID                 :0x15
DFHv                    :0.5
guidl                   :0x99a078ad18418b9d
guidh                   :0x4118a7cbd9db4a9b
HSSI version            :1.0
Firmware Version        :1
HSSI num ports          :8
Port0                   :25GbE                    
Port1                   :25GbE                    
Port2                   :25GbE                    
Port3                   :25GbE                    
Port4                   :25GbE                    
Port5                   :25GbE                    
Port6                   :25GbE                    
Port7                   :25GbE                    
--------HSSI INFO END-------

hssi loopback disabled to port0
[user@user ~]$ 

[JohnT_Intel]

HI,

I am working internally on this issue. Once I will keep you posted once I have progress on this.

[attdoner]

Hi @JohnT_Intel ,

I have found some updation at Mellanox side, by enabling promiscuous mode, I could see the packets in Mellanox.

In Mellanox,
user@user:~$ sudo ifconfig enp1s0f0np0
enp1s0f0np0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 0c:42:a1:8c:d5:cc  txqueuelen 1000  (Ethernet)
RX packets 110  bytes 6600 (6.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

But in N6001, hssi stats I could see some random counts(4040438296505124148,....) in statistics parameters as in image. Can you please let me know what are those values?. Even after hssistats is cleared using "--clear", those values still exist under hssistats.
In N6001,

user@user:~$ sudo hssistats 
Namespace(pcie_address=None, clear=False)
sbdf: 0000:01:00.0
FPGA dev: {'segment': 0, 'bus': 1, 'dev': 0, 'func': 0, 'path': '/sys/class/fpga_region/region0', 'pcie_address': '0000:01:00.0'}
args.hssi_grps[('dfl_dev.4', ['/sys/bus/pci/devices/0000:01:00.0/fpga_region/region0/dfl-fme.0/dfl_dev.4/uio/uio1'], '0000:01:00.0', 21)]
fpga uio dev:dfl_dev.4

--------HSSI INFO START-------
DFH                     :0x3000000010003015
HSSI ID                 :0x15
DFHv                    :0.5
guidl                   :0x99a078ad18418b9d
guidh                   :0x4118a7cbd9db4a9b
HSSI version            :2.0
Firmware Version        :16
HSSI num ports          :4
Port0                   :25GbE                    
Port1                   :25GbE                    
Port2                   :25GbE                    
Port3                   :25GbE                    
--------HSSI INFO END-------

HSSI num ports: 4
HSSI Ports                       |             port:0|             port:1|             port:2|             port:3|

tx_packets                       |4040438296505124148|4040438296505124148|4040438296505124148|                110|
rx_packets                       |4040438296505124148|4040438296505124148|4040438296505124148|                  6|
rx_crc_errors                    |         3735929054|         3735929054|         3735929054|                  0|
rx_align_errors                  |                  0|                  0|                  0|                  0|
tx_payload_bytes                 |16045693110842147038|16045693110842147038|16045693110842147038|               5060|
rx_payload_bytes                 |16045693110842147038|16045693110842147038|16045693110842147038|                950|
tx_pause                         |16045693110842147038|16045693110842147038|16045693110842147038|                  0|
rx_pause                         |16045693110842147038|16045693110842147038|16045693110842147038|                  0|
rx_errors                        |         3735929054|         3735929054|         3735929054|                  0|
tx_errors                        |         3735929054|         3735929054|         3735929054|                  0|
rx_unicast                       |13644642147974742460|13644642147974742460|13644642147974742460|                  0|
rx_multicast                     |13644642147974742460|13644642147974742460|13644642147974742460|                  4|
rx_broadcast                     |13644642147974742460|13644642147974742460|13644642147974742460|                  2|
tx_discards                      |                  0|                  0|                  0|                  0|
tx_unicast                       |13644642147974742460|13644642147974742460|13644642147974742460|                110|
tx_multicast                     |13644642147974742460|13644642147974742460|13644642147974742460|                  0|
tx_broadcast                     |13644642147974742460|13644642147974742460|13644642147974742460|                  0|
ether_drops                      |         3735929054|         3735929054|         3735929054|                  0|
rx_total_bytes                   |16045693110842147038|16045693110842147038|16045693110842147038|               1058|
rx_total_packets                 |4040438318920698472|4040438318920698472|4040438318920698472|                  6|
rx_undersize                     |         3735929054|         3735929054|         3735929054|                  0|
rx_oversize                      |         3735929054|         3735929054|         3735929054|                  0|
rx_64_bytes                      |16045693110842147038|16045693110842147038|16045693110842147038|                  0|
rx_65_127_bytes                  |16045693110842147038|16045693110842147038|16045693110842147038|                  4|
rx_128_255_bytes                 |16045693110842147038|16045693110842147038|16045693110842147038|                  0|
rx_256_511_bytes                 |16045693110842147038|16045693110842147038|16045693110842147038|                  2|
rx_512_1023_bytes                |16045693110842147038|16045693110842147038|16045693110842147038|                  0|
rx_1024_1518_bytes               |16045693110842147038|16045693110842147038|16045693110842147038|                  0|
rx_gte_1519_bytes                |16045693110842147038|16045693110842147038|16045693110842147038|                  0|
rx_jabbers                       |         3735929054|         3735929054|         3735929054|                  0|
rx_runts                         |         3735929054|         3735929054|         3735929054|                  0|