Programmable Devices
CPLDs, FPGAs, SoC FPGAs, Configuration, and Transceivers
20638 Discussions

Is the Intel N3000 FPGA PAC compatible with Supermicro SYS-2029GP-TR server.

Anup_Agarwal
Beginner
703 Views

TLDR: Is the Intel N3000 FPGA PAC compatible with Supermicro SYS-2029GP-TR server. If so, what kind of power connections does it expect from the server.

Power ports on the server:
According to the manual [2], the Supermicro SYS-2029GP-TR has 5 ports (JPW1-4) meant for GPUs and the server came with 8 pin to 8 pin connectors for these 5 ports (their pin layout is similar to EPS12V seen in [3]). The server also has 4 undocumented ports on the daughter board. 2 of these ports are similar to EPS12V and other 2 similar to traditional 8 pin PCIE/GPU power ports as also noted by [4]. The daughter board connects the power supply and motherboard.

Ports needed by the FPGA: From the docs [6], the Intel N3000 PAC requires a 12 V Auxiliary 2×3 power connector (6pin). I am not sure if this needs to be served via an EPS12V port (using a cable like [5]) or the PCIE power ports on the daughter board using a standard 8pin to 6 pin pcie power cable (e.g. [9]). We tried the PCIE power ports, the FPGA blinks 2 green and 2 yellow lights. The FPGA docs don't specify what this means. The closest behavior in the data sheet is 4 yellow blinking lights signalling power issues. The NIC does not show up in lspci when the server is booted.
We also tried using an adapter to convert the EPS12V (JPW5/6 on the motherboard) to 8 pin PCIE (as seen in [3]) and then using the standard PCIE cable (with the understanding that this setup would mimic cable [5]). The result was the same as before. We can buy the cable [5] from EPS12V to 6 pin PCIE and try it out but I don't expect results to be any different.

Potentially informative aside: We have previously used this NIC/FPGA board successfully on a Dell PowerEdge R720 using another cable [7]. For the cable, the 6 pin side is the same as PCIE 6pin [3] while the 8 pin side is shaped like EPS12V while pins are laid out like PCIE 8 pin. This I think is how the pins are present on the Dell server. This is based on the fact that the cable has yellow (12V) cables on the non-notch side [7] same as PCIE 8 pin layout in [3] while it naturally fits in an EPS12V port (Note, I haven't personally inspected the Dell server port/manual). If I am correct about the pin layout/shape of the Dell server 8 pin port by extrapolating from the cable, is there a name for this port?

Power on the server itself:
The server manual [2] also says:
Quote "
The main power connection on the X11DPG-SN must be connected to the power supply. The wiring is included with the power supply.
• 8-pin Processor Power (JPWR2)
Main ATX Power
The power supply modules plug directly into the X11DPG-SN, so there is no cable connection that needs to be made for main power. However you must also connect the 8-pin power connectors (JPW1 - JPW4) to your power supply (see below) to provide power to the GPUs.
GPU Power Headers
JPW3-JPW7 are the 8-pin 12V DC power output for the GPUs.
"
From the Main ATX power section, the JPW1/2 are already connected to the power supply. The JPW3/4 however are EPS12V ports as seen in Figure 1-2. (X11DPG-SN Motherboard Layout) of document [8], I thought these are for output to GPUs and not input from power supply. The next section about GPU power headers mentions these ports again as power supply to GPUs. Are these ports meant for supply from the mother board to supply to the motherboard from power supply? If these are meant to give power to the motherboard, which ports on the power supply should be connected to the JPW3/4 then?

[1] Xilinx Alveo U280 docs: https://www.xilinx.com/support/docu...ing-started-guide-alveo-accelerator-cards.pdf
[2] Supermicro SYS-2029GP-TR server manual: https://www.supermicro.com/manuals/superserver/2U/MNL-2068.pdf
[3] EPS12V and PCIE pin layout: What are "sense" pins in 8-pin PCI Express power plug?
[4] Undocumented power ports: https://forums.servethehome.com/index.php?threads/non-tesla-gpu-in-a-supermicro-2029gp-tr.26013/
[5] EPS12V to 6 pin PCIE cable: PROVANTAGE: Supermicro Cable-PWEX-1017 Supermicro Cabel Cable-Pwex-1017 6+2-Pin PCIE Power Connector 20CM Brown Box
[6] Intel N3000 FPGA PAC docs: https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ds/ds-pac-n3000.pdf
[7] Dell 8pin to 6pin cable: Amazon.com: New for Dell Split GPU Power Adapter Cable 8Pin to 6Pin and 8Pin N08NH 9H6FV J30DG PowerEdge R720 R730 R7910 : Everything Else
[8] https://www.tmc-uk.com/fileuploader/download/download/?d=0&file=custom/upload/File-1521111686.pdf
[9] NeweggBusiness - Supermicro CBL-PWEX-0582 30cm 8-pin to two 6+2 Pin 12V GPU Power Cable

0 Kudos
5 Replies
wchiah
Employee
668 Views

Hi,

 

Can I know which version of N3000 PAC card you are using? 10G or 25G?

There is two Supermicro that is qualified by Intel

  1. SuperServer 1019P-FHN2T
  2. SuperServer 2029U-MTNRV

You may refer the link below for detail
https://www.intel.com/content/www/us/en/programmable/products/boards_and_kits/dev-kits/altera/intel-fpga-pac-n3000/buy.html

 

I don’t see Supermicro SYS-2029GP-TR server in the Intel Qualified list

BUT…..

  • Whenever that is not qualified, does not mean that it cannot work with VC.
  • Please ensure that your server has enough airflow and power to the card
  • Other than that, you might need to check the compatibility of the server based on mechanical fitting, software BIOS, is it able to support Gen3 and other etc…

 

For the Supermicro server power connection,

  • I wish to answer that, but I afraid my answer would not be so accurate
  • I do suggest you refer back to Supermicro if possible

 

For N3000 Led Blinking,

  • Can I know which LED that you mention? Connectivity LED ? Activity LED
  • The Intel FPGA PAC N3000 follows PCIe standards for 150 W add-in cards where the maximum current from the 12 V slot power source is 5.5 A (max) and the 12 V Auxilary connector is 6.25 A (max).
  • Initially, I do suggest that you can try to remove the card from the slot and ensure edge connector is clean
  • Last, please ensure the fan is at maximum speed
  • In some case, setting server fan to maximum and having connected an AUX does not mean that you have enough cooling or power to the card.
  • You may try to monitor fpgainfo bmc on “Die temperature” “12V AUX Voltage” to confirm it

 

N3000 Power port detail

 

Let me know if this helps.

Regards,

WeiChuan_C_Intel

0 Kudos
Anup_Agarwal
Beginner
659 Views

Compatibility:

We set max fan speed through IPMI. Mechanically the NIC fits in the PCIE riser in the rear end of server. The slot is PCIE 3.0 x16.

 

Power connection:

We contacted Supermicro for this. They suggested us to use cable [1] to connect the PCIE power port on the power distribution board to the 6 pin PCIE power port of the FPGA. We did this. This cable respects the pinout of the N3000-N PAC.

 

LEDs:

Currently both the activity LEDs (for QSFP A, B) blink in green and both the connectivity LEDs blink yellow (FYI this issue is also raised by us in [2], I am not sure if you want to merge these posts). This is when nothing is connected to the QSFP ports.

Note we have the Intel N3000-N PAC. This does not support 10G configuration and the data sheet never mentions connectivity LED with yellow. Only mentions all 4 leds yellow in case of power/fan issue.

We have tried removing and installing the board again. This did not work. The servers we have are brand new and there is no dust and the edge connectors are clean. We ensure that the fans are at full speed.

We even plugged in the FPGA into a Dell PowerEdge R720 server. We observe the same behavior (2 yellow, 2 green blinking LEDs and board not showing in lspci) on it. Previously the board worked perfectly fine on another Dell PowerEdge R720 server. Based on this we think the board should be getting sufficient power/airflow. Is it possible that the board is damaged in some way? How can we confirm?

fpgainfo bmc just says `No FPGA resources found.`

 

PCIE Enumeration:

We cannot see the accelerator in `lspci | grep -i accel` or `lspci -d :0b30`. The server is brand new, no dust. We ensured server fans are at full speed through IPMI as well as by inspecting server fan noise.

 

Other potentially useful information:

Environment:

We are using Centos 7.9 and linux kernel 4.19. We started with Centos 7.6 compiled linux kernel 4.19 with real-time patch following instructions in the user guide. Somewhere in between a `sudo yum update` caused update from Centos 7.6 to 7.9. We then installed the runtime stack for N3000-N.

I would imagine we should still be able to see the board in lspci irrespective of Centos 7.6 or 7.9. Am I wrong here? If so, I can reinstall Centos 7.6 and ensure we don't update to Centos 7.9.

 

Also, the user guide says:

  • Enable the following options in the BIOS:
    • Intel VT-x (Intel Virtualization Technology for IA-32 and Intel 64 Processors)
    • Intel VT-d (Intel Virtualization Technology for Directed I/O)

we couldn't find these in the BIOS so skipped this step.

 

dmesg output:

Please find attached the output of dmesg.

 

Data sheet: https://www.intel.com/content/www/us/en/programmable/documentation/dlq1585950463484.html

User guide: https://www.intel.com/content/www/us/en/programmable/documentation/zsf1588015530773.html#kdq15984767...

 

[1] Supermicro CBL-PWEX-0581 8 Pin to Two 6+2 Pin 12V GPU Power Cable, 65 cm

[2] https://community.intel.com/t5/Programmable-Devices/What-do-two-yellow-and-two-green-blinking-LEDs-on-the-Intel/m-p/1319766

0 Kudos
Anup_Agarwal
Beginner
659 Views

lspci output also for reference.

0 Kudos
Anup_Agarwal
Beginner
615 Views

We tried putting the NIC back in the original Dell PowerEdge R720 servers. The boards earlier worked perfectly on these machines. However, now we see the same behavior (2 yellow/green blinking LEDs and nothing about the board shows up in lspci).

Is there a way that we can check if the boards are physically damaged?

0 Kudos
wchiah
Employee
601 Views

Hi,

For the following question about LED and fpgainfo error,
I will answer your question on another page, as both of them having the same server, to avoid any confusion.
Please refer https://community.intel.com/t5/Programmable-Devices/What-do-two-yellow-and-two-green-blinking-LEDs-on-the-Intel/m-p/1319766#M81666

 

I think I had answered your question about "N3000 FPGA PAC compatible with Supermicro SYS-2029GP-TR server"

If you have any further questions about the compatibility please let me know, Else I will close this loop after 3 days.

 

Regards,

WeiChuan_C_Intel

0 Kudos
Reply