FPGA, SoC, And CPLD Boards And Kits
FPGA Evaluation and Development Kits
6192 Discussions

Link status on BD-NVV-N3000-3 card

Daehyeok_Kim
New Contributor I
2,457 Views

Hello,

I've installed a BD-NVV-N3000-3 card on our server and connected it to an Intel Tofino-based 32-port 100Gbps switch using a QSFP+ cable. And I've followed the instruction in the user guide (https://www.intel.com/content/www/us/en/programmable/documentation/zsf1588015530773.html) to install the required runtime stack. I am trying to use it as a basic NIC with a 4 x 25G configuration. 

When I checked the status of Retimer using "sudo fpgainfo phy", it showed that the all four ports are down:

 

-bash-4.2$ sudo fpgainfo phy
Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** PHY ******//
Object Id                     : 0xEC00000
PCIe s:b:d.f                  : 0000:08:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
//****** PHY GROUP 0 ******//
Direction                     : Line side
Speed                         : 25 Gbps
Number of PHYs                : 4
//****** PHY GROUP 1 ******//
Direction                     : Host side
Speed                         : 40 Gbps
Number of PHYs                : 4
//****** Intel C827 Retimer ******//
Port0 25G                     : Down
Port1 25G                     : Down
Port2 25G                     : Down
Port3 25G                     : Down
Retimer A Version             : 101c.1064
Retimer B Version             : 0000.0000

 

On the switch side, I could see the port status for all four lanes is down:

 

-----+----+---+----+-------+----+---+---+---+--------+----------------+----------------+-
PORT |MAC |D_P|P/PT|SPEED  |FEC |RDY|ADM|OPR|LPBK    |FRAMES RX       |FRAMES TX       |E
-----+----+---+----+-------+----+---+---+---+--------+----------------+----------------+-
25/0 |31/0|188|3/60|25G    | RS |YES|ENB|DWN|  NONE  |               0|               0|
25/1 |31/1|189|3/61|25G    | RS |YES|ENB|DWN|  NONE  |               0|               0|
25/2 |31/2|190|3/62|25G    | RS |YES|ENB|DWN|  NONE  |               0|               0|
25/3 |31/3|191|3/63|25G    | RS |YES|ENB|DWN|  NONE  |               0|               0|

 

Could you let me know how to activate the ports?

Please let me know if you need any further information.

Thanks,

Daehyeok

0 Kudos
1 Solution
JonWay_C_Intel
Employee
2,294 Views

Hi @Daehyeok_Kim 

Thanks for your update on your debug status. Im glad to know that we have ruled out the cable or card issue. For the switch related queries, I am not the subject matter expert. Could you file a separate forum thread on the Ethernet products Forum (https://community.intel.com/t5/Ethernet-Products/bd-p/ethernet-products). They will be able to guide you from there. Appreciate if you can put in the link to the new thread for the benefit of others who face similar issue.

 

 

View solution in original post

0 Kudos
8 Replies
JonWay_C_Intel
Employee
2,366 Views

Please follow the below debug steps:

1)  Check if you are using the correct board OPN.

BD-NFV-N3000-1 :Production 10 GbE

OR

BD-NVV-N3000-2: Production 25 GbE

You should not program the BD-NFV-N3000-1 (Production 10 GbE) with a 25GbE image, vice versa. You cannot use the cards interchangeably.

2) Connect the QSFP cages with a loopback module. Run “fpgainfo phy” and see if all ports are UP.

3) If Step 2 is passing but failing only when you are connecting the card QSFP to other QSFP cage on other devices.

0 Kudos
Daehyeok_Kim
New Contributor I
2,362 Views

Thanks for your reply! I have some follow-up questions.

1) I am using BD-NVV-N3000-3 and haven't programmed the board with other images. I followed the instruction up to section 4 (https://www.intel.com/content/www/us/en/programmable/documentation/zsf1588015530773.html#phn1588111278439).

Could you let me know how to verify which version is loaded? Also, what does OPN stands for?

2) By the "loopback module", are you referring to the module shown in Fig. 4 (https://www.intel.com/content/www/us/en/programmable/documentation/zsf1588015530773.html#xzd1588094394268)? Is it included in the original package?

Thanks,

Daehyeok

0 Kudos
JonWay_C_Intel
Employee
2,350 Views

1) Thanks. I think you have answered me.
OPN refers to BD-NVV-N3000-3
Version loaded refers to the MAX10 build/NIOS version + Bitstream ID.

 2) Yes, loopback module as shown in the link you referred:  https://www.intel.com/content/www/us/en/programmable/documentation/zsf1588015530773.html#xzd1588094394268

No, it is not included in the package. You will have to purchase it separately from the module makers.

If you do not have a loopback module at hand, you can order for 2 (one for each cage). It will be useful for future debugging as well. Meanwhile, you can try connect the ethernet cable from QSFP cage A to cage B, and check fpgainfo phy. If port is up, then it would be the connection to the external switch issue. If it is still down, it would either be a card or a cable issue. To rule out if it is a cable issue, you will definitely need a loopback module. 

0 Kudos
Daehyeok_Kim
New Contributor I
2,344 Views

Thanks! We just checked the port status by connecting two cages using an Ethernet cable, but 
"fpgainfo phy" command still shows they are down:

 

Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** PHY ******//
Object Id                     : 0xEC00000
PCIe s:b:d.f                  : 0000:08:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
//****** PHY GROUP 0 ******//
Direction                     : Line side
Speed                         : 25 Gbps
Number of PHYs                : 4
//****** PHY GROUP 1 ******//
Direction                     : Host side
Speed                         : 40 Gbps
Number of PHYs                : 4
//****** Intel C827 Retimer ******//
Port0 25G                     : Down
Port1 25G                     : Down
Port2 25G                     : Down
Port3 25G                     : Down
Retimer A Version             : 101c.1064
Retimer B Version             : 0000.0000

 

I suspect this is a cable issue because all four cards in our testbed show the same symptom. Is there a particular spec for a QSFP cable that is compatible with this card?

By the way, if I understand correctly, the default mode of BD-NVV-N3000-3 is 4x25GbE, where only one of the QSFP cage (i.e., cage A) is active. In this case, when we connect the Ethernet cable from QSFP cage A to cage B, do you still expect that the ports are up if there's no cable or card issue?

Regarding the loopback module, would the one in the following link work for this card?:

https://cablesondemand.com/qsfp-100g-loopback-adapter-module-for-qsfp28-port-testing-3db-attenuation-2-5w-power-consumption-100-gigabit-ethernet-ready-sf-100glb2-5w-3db?utm_source=google&utm_medium=cpc&adpos&scid=scplpSF-100GLB2.5W-3DB&sc_intid=SF-100GLB2.5W-3DB&gclid=Cj0KCQjwi7yCBhDJARIsAMWFScMuBbJujTg3weD0ZE1Z7H88h9qr_b3eVW1sUXOmBlNcgBSjR58T0TQaA...

Thanks,

Daehyeok

0 Kudos
JonWay_C_Intel
Employee
2,329 Views

For validated cable, please refer to datasheet: https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ds/ds-pac-n3000-n.pdf (Table 2)

Yes, default user image is 4x25G. You need to switch to Factory Boot page for the 2x2x25G image. Then you can test with cable from cageA to cageB.

To switch to Factory Boot page (2x2x25G), run "rsu fpga -f"

 

0 Kudos
Daehyeok_Kim
New Contributor I
2,300 Views

Thanks for your reply!

I switched the card configuration to 2x2x25G, tested it with cable from Cage A to Cage B, and checked that the port status is up:

[root@h12 daehyeok]# fpgainfo phy
Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** PHY ******//
Object Id                     : 0xF100000
PCIe s:b:d.f                  : 0000:08:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000410010310
Bitstream Version             : 0.2.3
Pr Interface Id               : a5d72a3c-c8b0-4939-912c-f715e5dc10ca
//****** PHY GROUP 0 ******//
Direction                     : Line side
Speed                         : 25 Gbps
Number of PHYs                : 4
//****** PHY GROUP 1 ******//
Direction                     : Host side
Speed                         : 40 Gbps
Number of PHYs                : 4
//****** Intel C827 Retimer ******//
Port0 25G                     : Up
Port1 25G                     : Up
Port2 25G                     : Up
Port3 25G                     : Up
Retimer A Version             : 101c.1064
Retimer B Version             : 101c.1064

With this, I've confirmed that there's no issue with the cable and card. 

Since even in 2x2x25G mode, it still didn't work when I connected the card with the switch, I suspect there might be some issue with the configuration on the switch side. When I configured the switch port connected to the card, I set the rate to 25G and the FEC mode to RS for each lane (I checked the card is configured to use RS).

Could you advise what else should I check on the switch side?    

Thanks,

Daehyeok

 

 

 

0 Kudos
JonWay_C_Intel
Employee
2,295 Views

Hi @Daehyeok_Kim 

Thanks for your update on your debug status. Im glad to know that we have ruled out the cable or card issue. For the switch related queries, I am not the subject matter expert. Could you file a separate forum thread on the Ethernet products Forum (https://community.intel.com/t5/Ethernet-Products/bd-p/ethernet-products). They will be able to guide you from there. Appreciate if you can put in the link to the new thread for the benefit of others who face similar issue.

 

 

0 Kudos
Daehyeok_Kim
New Contributor I
2,267 Views

Thanks for your reply.

I figured out what causes the issue. Since the factory/user image on FPGA does not support auto-negotiation, I had to manually disable auto-negotiation on the switch side. Otherwise, it seems to use auto-negotiation by default, and something went wrong.

Thanks for your help!

Daehyeok

0 Kudos
Reply