how to test FPGA PAC N3000 network port

[fzhan51]

FPGA PAC N3000, configed in 8x10G mode, attached to the Xeron serverA with pcie x16 。we got 40GbE servrB/ 10GbE serverC , QSFP+40G SR-L optical module and 12core mo3 fiber。test case:1) serverA-->one port QSFP and fiber--> serverB: leds out 2)serverA(B)-->one port to another port-->serverA(B);leds on

(a).should we puchase E40GQSFPSR to displace QSFP+40G SR-L? (b)give some advices.

by the way,I didn't enable Vx-Tx Vx-d(not found in the bios),may it help? thanks a lot

[JonWay_C_Intel]

Hi @fzhan51​  First of all, what is the objective of your test? A simple functionality test? If yes, you can refer to:

Test by DPDK

https://www.intel.com/content/www/us/en/programmable/documentation/xgz1560360700260.html#hnb1561577285390

Test by fpgadiag:

https://www.intel.com/content/www/us/en/programmable/documentation/xgz1560360700260.html#hll1561577008230

[fzhan51]

thank you. we want a simple test fot the connection between n3000 qsfp port and the port of other 40GbE networkcard , a ping test etc. beside,

today,I got empty when entered lspci | grep 0d58 or lspci | grep 0cf8 ,what may cause that?

[JonWay_C_Intel]

Is the card alive? Did you connect the AUX cable? Can this recover by reset? Is it overheated?

any status logs e.g. fpgainfo fme? fpgainfo bmc? etc

...i can help more if there is more info.

[fzhan51]

the cable is 12core mo3 fiber; we are purchasing X4DACBL3 to try; reset not fix;

I just found that the Image may be upgraded to Production Version for trying the connection, how can I revert it?

we bought a E40GQSFPSR optic mode to test with 12core mo3 fiber and other optic mode,seems no help.

for now,the 40G serverB is not available, i connect the netport QSFPa(QSFP+40G SR-L) to netportQSFPb(QSFP+40G SR-L) with 12core mo3 fiber

-----------------------------------------------

[fpga@localhost ~]$sudo fpgainfo fme

Board Management Controller, MAX10 NIOS FW version D.2.0.19

Board Management Controller, MAX10 Build version D.2.0.6

//****** FME ******//

Object Id           : 0xF500000

PCIe s:b:d.f         : 0000:07:00.0

Device Id           : 0x0b30

Numa Node           : 0

Ports Num           : 01

Bitstream Id         : 0x23000010010309

Bitstream Version       : 0.2.3

Pr Interface Id        : 901dd697-ca79-4b05-b843-8138cefa2846

Boot Page           : user

----------------------------------------------

[fpga@localhost ~]$ sudo fpgainfo phy

[sudo] password for fpga: 

Board Management Controller, MAX10 NIOS FW version D.2.0.19

Board Management Controller, MAX10 Build version D.2.0.6

//****** PHY ******//

Object Id           : 0xF500000

PCIe s:b:d.f         : 0000:07:00.0

Device Id           : 0x0b30

Numa Node           : 0

Ports Num           : 01

Bitstream Id         : 0x23000010010309

Bitstream Version       : 0.2.3

Pr Interface Id        : 901dd697-ca79-4b05-b843-8138cefa2846

//****** PHY GROUP 0 ******//

Direction           : Line side

Speed             : 10 Gbps

Number of PHYs        : 8

//****** PHY GROUP 1 ******//

Direction           : Host side

Speed             : 10 Gbps

Number of PHYs        : 8

//****** Intel C827 Retimer ******//

Port0 10G           : Up

Port1 10G           : Up

Port2 10G           : Up

Port3 10G           : Up

Port4 10G           : Up

Port5 10G           : Up

Port6 10G           : Up

Port7 10G           : Up

Retimer A Version       : 101c.1064

Retimer B Version       : 101c.1064

--------------------------------------------

[fpga@localhost ~]$ sudo fpgainfo bmc

Board Management Controller, MAX10 NIOS FW version D.2.0.19

Board Management Controller, MAX10 Build version D.2.0.6

//****** BMC SENSORS ******//

Object Id           : 0xF500000

PCIe s:b:d.f         : 0000:07:00.0

Device Id           : 0x0b30

Numa Node           : 0

Ports Num           : 01

Bitstream Id         : 0x23000010010309

Bitstream Version       : 0.2.3

Pr Interface Id        : 901dd697-ca79-4b05-b843-8138cefa2846

( 1) Board Power       : 69.56 Watts

( 2) 12V Backplane Current  : 3.20 Amps

( 3) 12V Backplane Voltage  : 12.15 Volts

( 4) 1.2V Voltage       : 1.18 Volts

( 6) 1.8V Voltage       : 1.80 Volts

( 8) 3.3V Voltage       : 3.25 Volts

(10) FPGA Core Voltage    : 0.90 Volts

(11) FPGA Core Current    : 15.07 Amps

(12) FPGA Die Temperature   : 69.00 Celsius

(13) Board Temperature    : 43.50 Celsius

(14) QSFP0 Supply Voltage   : 3.27 Volts

(15) QSFP0 Temperature    : 51.00 Celsius

(24) 12V AUX Current     : 2.53 Amps

(25) 12V AUX Voltage     : 12.11 Volts

(37) QSFP1 Supply Voltage   : 3.27 Volts

(38) QSFP1 Temperature    : 54.00 Celsius

(44) PKVL0 Core Temperature  : 68.00 Celsius

(45) PKVL0 SerDes Temperature : 68.50 Celsius

(46) PKVL1 Core Temperature  : 69.00 Celsius

(47) PKVL1 SerDes Temperature : 69.50 Celsius

---------------------------------------------------------------------------

how to test the connection in this case ,and how to check tx and rx ?

[fzhan51]

for above connection ,if disconnect the fiber

[fpga@localhost ~]$ sudo fpgainfo phy

Board Management Controller, MAX10 NIOS FW version D.2.0.19

Board Management Controller, MAX10 Build version D.2.0.6

//****** PHY ******//

Object Id           : 0xF500000

PCIe s:b:d.f         : 0000:07:00.0

Device Id           : 0x0b30

Numa Node           : 0

Ports Num           : 01

Bitstream Id         : 0x23000010010309

Bitstream Version       : 0.2.3

Pr Interface Id        : 901dd697-ca79-4b05-b843-8138cefa2846

//****** PHY GROUP 0 ******//

Direction           : Line side

Speed             : 10 Gbps

Number of PHYs        : 8

//****** PHY GROUP 1 ******//

Direction           : Host side

Speed             : 10 Gbps

Number of PHYs        : 8

//****** Intel C827 Retimer ******//

Port0 10G           : Down

Port1 10G           : Down

Port2 10G           : Down

Port3 10G           : Down

Port4 10G           : Down

Port5 10G           : Up

Port6 10G           : Up

Port7 10G           : Down

Retimer A Version       : 101c.1064

Retimer B Version       : 101c.1064

-------------------------

why Port5/Port6 always Up ?

[JonWay_C_Intel]

Hi @fzhan51​  Lets take a step back. So i suppose you are able to see output from lspci | grep ocf8, right?

Let me get to your last question... If you have removed all the cable, i would expect port 5/6 to be "Down"...

Could you try "rsu bmcimg <BDF>" and run fpgainfo phy again to see if that recovers the card?

[fzhan51]

@JwChin​ :   lspci | grep ocf8 ,the output is ocf8 before my upgradation ,but now, i can't see ;

"rsu bmcimg <BDF>" is good for port 5/6 up problem. now all show "Down". one thing to mention, I forggot to do "sudo rm -rf /usr/share/opae/n3000/one-time-update/*/"

[JonWay_C_Intel]

So, you still cant see 0CF8?

Then, lets fix this first.

I know you have provided some info, but could you fill up some basic info as below (if too long, then put in a text file and attach it here). Thanks.

Operating System Vendor and Version

Kernel Version

Server Vendor and Model

Intel Acceleration Stack release i.e. Beta, Production

Ethernet Port Configuration i.e 2x2x25G, 4x25G, 8x10G

I40e and iavf driver version

I40e and iavf firmware version

log or screenshot of fpgainfo fme, fpgainfo bmc, fpgainfo phy, fpgainfo errors

dmesg log file

lspci -vvv log file

Run "lspci -vt"

and also lspci | grep Ethernet

[fzhan51]

Dear JwChin:

[3days ago,the system installed by others]

Operating System Vendor and Version:  linux-4.19.8

Kernel Version:     4.19.8

rpm -qa | grep kernel-header: kernel-headers-3.10.0-1062.18.1.el7.x86_64

Server Vendor and Model: HUAWEI/H22H-03

Intel Acceleration Stack release i.e. Beta, Production: IntelAccelerationStack release.jpg Alpha-1 D.1.0.13

Ethernet Port Configuration i.e 2x2x25G, 4x25G, 8x10G: 8x10G

I40e and iavf driver version: i40e-2.9.21

I40e and iavf firmware version: iavf-3.7.53 

[now]

Operating System Vendor and Version: CentOS-7-x86_64-Everything-1611.iso(maybe)

Kernel Version:         3.10.0-1062.18.1.el7.x86_64

rpm -qa | grep kernel-header: kernel-headers-3.10.0-1062.18.1.el7.x86_64

Server Vendor and Model:     HUAWEI/H22H-03

Intel Acceleration Stack release i.e. Beta, Production: Production D.2.0.6

Ethernet Port Configuration i.e 2x2x25G, 4x25G, 8x10G: 8x10G

I40e and iavf driver version:    i40e-2.10.19.30(i40e-2.9.21 compiled failed)

I40e and iavf firmware version: iavf-3.7.61.20(when install,it says invalid private key)

maybe there were some mistakes when I did OTSU/RSU

[JonWay_C_Intel]

hi @fzhan51​  It doesnt seems right that 0CF8 DID is not showing.

From experience, it could be the kernel (im using 3.10.0-957.el7.x86_64). Second, it could be the OTSU/RSU.

If it were me, I would probably check the correct OS and kernel version and to perform the OTSU/RSU again....might be a long-winded process but at least you can watch for any error and foolproof it.

[fzhan51]

Dear JwChin: reinstall Centos 7.6 does help,0cf8 shows correctly.thank you very much .

[JonWay_C_Intel]

Hi @fzhan51​  Thank you for your update. So, it might be something wrong with the kernel, and it resolved over re-installing. This is great input for others watching the forum, thanks.

[fzhan51]

Dear JwChin I would try it.note that, Retimer A/B Version are zero after "rsu bmcimg <BDF>", and port0-7 all Down now even connect the fiber. if it become Porduct version,it can't revert back to Alpha Version,isn't it ? if no, it means that I have to sign the image first

[fzhan51]

Dear JwChin I would try it . note that Retimer A/B Version become 0000.0000, port0-port7 always Down after "rsu bmcimg <BDF>". if it become product version,cant it revet to Alpha version ? last time i try the $ sudo super-rsu /usr/share/opae/n3000/one-time-update/<25G or 10G>/\

rollback-*.json --with-rsu in section "B.1.2. Root Entry Hash Not Programmed",if failed. that means I do have to prepare signed image?

[JonWay_C_Intel]

Hi @fzhan51​  Unfortunately, no you cannot revert from Production to Alpha.

[fzhan51]

Dear JwChin,I switch the kernel to 4.19.8; and tried the section "B.1.2.",attached the log.

I remember that I didn't get any signed image,and the last operation i follow to do is the section "B.1.2."

seemed that it went through,then the version become Product version.

I just read the ug-pac-security-n3000.pdf,it says "This operation is permanent and irreversible."on page 21.Does that mean root entry hash bitstream is programmed,I even don't know any keys(root key/csks),then the card is dead for me? could you help me please.

[JonWay_C_Intel]

Hi @fzhan51​  If you did not create + program any root hash into the device, then it should be fine. You can still program unsigned images.

[fzhan51]

would you help to login our n3000 server to check what's going wrong? much appreciated if there is your email to send teamviewer id /psw. i just send it to supportreplies@intel.com i don't know whether you can received it or not.

i Update the Intel XL710 Firmware,and after rsu bmcimg <FPGA PCIe B:D.F> the server hang.attached is the sreenshot

[JonWay_C_Intel]

Hi @fzhan51​  I see you have created another thread https://forums.intel.com/s/question/0D50P00004dmzbn/pac-n3000-8x10g-xl710-update-failed?language=en_US&t=1586513388811 on the same issue.

But i probably continue to help from here, and close the duplicated.

Could you run lspci -s BDF_OfTheEthernetCtrller -x

Expected to see something like:

00: 86 80 f8 0c xx xx xx

I see that your version is up-to-date, and if the lspci command as above is ok...then probably you can try to move on

[fzhan51]

so glade to get your answer.

[root@localhost Factory_Image]# lspci -s 07:00.0 -x

07:00.0 Processing accelerators: Intel Corporation Device 0b30

00: 86 80 30 0b 46 05 10 00 00 00 00 12 08 00 00 00

10: 0c 00 f0 ff 3f 02 00 00 0c 00 e0 ff 3f 02 00 00

20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 00 00

30: 00 00 00 00 68 00 00 00 00 00 00 00 0a 01 00 00