S2600CW NIC Configuration BIOS page blank

[MShepard]

I'm bringing up a new S2600CW2R board with a single E5-2680v4 CPU in CPU1 and 4x 32GB DIMMs in A1-D1.

During startup, the board does not discover the onboard NICs - the BIOS page for "NIC Configuration" is completely blank - and only HBAs in PCIe slots 5 & 6 connected to CPU1 show their startup screens.

Have latest firmware updates:  BIOS 01.01.1029; BMC FW 1.61.12390; Boot FW 1.07; SDR pkg D1.14; ME FW .3.01.03.072 and reset BIOS NVRAM to defaults.

RMM4 works fine using Dedicated Management LAN.  Could not connect using baseboard LAN.

Don't know if this is a hardware problem or BIOS setup problem or next steps to take.  Debug Log from BMC Web Console is attached.

Thanks!!

[JoseH_Intel]

Hello MShepard,

Thank you for joining the Intel community

There are 2 RJ45 ports on the back of this board. One of them (yellow) is the management/NIC. So as I understand you are able to communicate with the BMC of the system but not possible to use it for LAN connectivity. Have you tested both of the ports? Do you have activity LED from the NICs after a LAN cable is connected? In BIOS do you have the option to enable PXE boot from boot devices?

We received your debug log. They are password protected and we cannot see them. If you could attach a SysInfo log will be a lot easier for us to check them and find any faults. You can download the SysInfo tool from this URL: Download System Information Retrieval Utility (SysInfo) (intel.com)

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[MShepard]

Thank you for your help, Jose.

Actually, there are 3 RJ45 ports on the back of the board - this is the S2600CW board that went EOL July 2020.  I am using the Dedicated Management LAN port to access the BMC and it is green on this board.  I was able to set the static IP address for this port and find the BMC at that address.  I have the RMM4LITE key and the full remote management Javascript console works reliably as long as I use IE11 - I thought it was HTML5 now? 

After setting a different static IP for the BMC LAN in BIOS I am still unable to reach the BMC through either RJ45 port for the onboard NICs.  When connected to a GbE switch, both RJ45s show a solid green and solid orange LED where the Dedicated Management port has two green LEDs with one flashing.

I believe the BIOS options for PXE Boot are in the NIC Configuration page, which is completely blank - it seems like the BIOS does not find the i350 NICs.

Attached are the sysinfo log files you requested.  There is no RAID key and the SATA controller is used only for CD/DVD so I did not generate those logs.

There are 4 PCIe cards inserted: two HBAs on the CPU1 PCI bus (slots 5&6) which do run their BIOS at boot and a HBA and PCIe AHCI SSD on the CPU2 PCI bus (slots 2&3) which do not.

Best regards,

...Mark

[JoseH_Intel]

Hello MShepard,

Both the embedded NIC cards seem to show up on the System Even Log and Retrieval information log. One thing that pulled my attention is that in the SEL log shows the temp reading for both NICs as UNAVAILABLE, where the temp for the BMC LAN shows up correctly. Let me analyze this logs with our senior team and will let you know as soon as I have updates.

In the mean time please try to set the BIOS to factory defaults F9 and reboot.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[PCProfessionals]

I too am having the EXACT same problem, NIC area in BIOS is blank on the server I'm building up.

[MShepard]

Thank you, Jose.

I did the F9 reset to defaults yesterday after I did the firmware (incl BIOS) updates.  It did not affect this issue and I have only changed a few BIOS values since.

Reviewing again, I noticed that the SDR was not upgraded and was showing rev 1.14 instead of 1.15.  I performed a manual upgrade procedure as described in the IBMC(R) FRU/SDR update package S2600CW_FRUSDR_115 release notes with the following commands in the EFI shell:

frusdr.efi -d fru
frusdr.efi -fru S2600CW.fru
frusdr.efi -cfg master.cfg with option 3 to program both FRU and SDR

After this I shut down and removed AC power until the DC buses collapsed then rebooted to the BIOS setup screen.  The NIC Configuration page was still blank, but System Information now shows

SDR Package Version :

SDR Package 1.15

I will await the results of your analysis.

Again, thank you for your support.

...Mark

[MShepard]

Thank you, Jose-

Answers to your questions:
The problem occurred on board startup. It is a brand new board and has never been used (mfg 3/20). The firmware package was upgraded before testing the board. The original boot screen had a red warning that the board had factory mfg firmware load and must be updated before use.

1. This S2600CW2R board is being installed to replace the S5500HCV server board in a SC5600LX server chassis which has been operating for ~10 years. The SC5600LX chassis has 2 HSBP, a 6-slot AXX6DRV3GR non-expanded SAS/SATA backplane with LSI 9211-8i HBA and a 4-slot AXX4DRV3GEXP expander backplane with LSI SAS9211-4i HBA. I have not been able to connect the IPMI cables from the HSBPs to the server board because it has only one 3-pin HSBP connector where the S5500HCV has two 4-pin HSBP connectors.  The BMC does not see either HSBP.

2. When installing SDR package 1.15, the following messages are displayed
" Parsing..... The chassis type cannot be auto-detected"
"S2600CW platform FRU & SDR package version 1.15"
" Intel Server Board S2600CW platform"
"Detecting HSBP, please wait a moment."
"Hot swap backplane is not detected"
"Front Panel FRU is not detected"  (SC5600LX front panel is connected)
"SDR auto-update failed due to chassis type could not be detected"
"Falling back to Other Chassis. Please update SDR by legacy process"
"SDR auto-update failed"
"Retry with legacy update process? (Y/n)"

3. The previous sysinfo logs I sent were made with SDR version 1.14.

4. I re-installed the complete firmware package
a. Downloaded a new copy of the firmware package from Intel website
"S2600CW_EFI_BIOS_R01.01.1029_ME_030103072_BMC_0161r12390_FRUSDR_115"
b. Extracted the ZIP file and copied its contents to a blank USB flash drive.
c. Inserted the USB flash and booted to EFI shell which auto-executed startup.nsh

5. I reset all BIOS settings to factory default
a. Boot to BIOS Setup (F2)
b. Move BIOS recovery jumper to recovery position 2-3 (as specified in Release Notes)
c. Press F9, then F10.
d. Remove AC power and let DC bus decay and rest for several minutes
d. After reboot to BIOS Setup, some settings have changed but the Management LAN IPv4 settings were not reset.
e. The NIC Configuration BIOS setup screen is still completely blank

6. Created new Sysinfo log files (attached).

Best regards,

...Mark

[JoseH_Intel]

Hello MShepard,

After analyzing the logs we have a couple more questions and some other findings

When did this problem occur? Has the onboard LAN ports ever worked, or worked and then failed?

When did you update the BIOS to the latest version (SE5C610.86B.01.01.1029)? (you already answered this)

This message appears in the log many times since 12/01/2020 -

Line 481: 9  12/09/2020-00:55:16  

Management Subsystem Health, BMC FW Health (#0x10)  

Warning event: BMC FW Health reports SAS IOC Temp has failed and may not be providing a valid reading.  

Integrated BMC - LUN#0 (Channel#0) 

Also, these items are found in the log:

Line 69: 0 DIMM Thrm Mrgn 2 177 32 Degrees Celsius READING_UNAVAILABLE

Line 33: 0 SAS IOC Temp     214  32  Degrees Celsius READING_UNAVAILABLE  

Line 41: 0 LAN NIC Temp     47  32  Degrees Celsius READING_UNAVAILABLE  

Line 46: 0 DIMM Thrm Mrgn 3      178  32  Degrees Celsius READING_UNAVAILABLE  

Line 47: 0 DIMM Thrm Mrgn 4      179  32  Degrees Celsius READING_UNAVAILABLE

The recommendation was to reinstall the latest available BIOS again, then power off/on the server after the BIOS has been updated, and send us a new set of sysinfo and logs. It seems some sensors did not get properly updated or are not being read correctly for some reason. Since you already reinstalled the BIOS and SDR/FRU to the latest you can just attach the new set of sysinfo logs for us to compare

We'll look forward for your updates

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello MShepard,

Thank you very much for the updates. We will analyze this new set of logs provided and will let you know our findings.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[MShepard]

Another observation about the PCI bus.  From the PCI_log.txt file:

BIOS and EFI shell see both channels of the I350 Ethernet Controller as functions 0 and 1
00 01 00 00 ==> Network Controller - Ethernet controller
Vendor 8086 Device 151F Prog Interface 0
00 01 00 01 ==> Network Controller - Ethernet controller
Vendor 8086 Device 151F Prog Interface 0

And the LSI MegaRAID 9260-4i in Slot 6 (CPU1) on bus 3
00 03 00 00 ==> Mass Storage Controller - RAID controller
Vendor 1000 Device 0079 Prog Interface 0

And the LSI SAS9211-4i HBA in Slot 5 (CPU1) on bus 4
00 04 00 00 ==> Mass Storage Controller - Serial Attached SCSI (SAS) controller
Vendor 1000 Device 0070 Prog Interface 0

But it does not see the LSI SAS9211-8i HBA in Slot 3 (CPU2)

And if I move the 9211-4i from Slot 5 (CPU1) to Slot 4 (CPU2), it is no longer recognized

When these controllers show up in the PCI log, they also run their BIOS as part of the boot process and display the controller screens with specifics and attached devices.  When they do not show up in the log, they also are not part of the boot process.

[JoseH_Intel]

Hello MShepard,

Could you please confirm that you installed an s2600CW board into a chassis that a 5500HCV server board was removed from?

The SC5600LX server chassis is not a supported chassis for the s2600CW. From what I see in ark.intel.com is that the support chassis for an s2600CW server board is the Intel® Server Chassis P4304XXMUXX (4U Rack or Pedestal). It sure seems like the errors reported is that the newer board is not liking the chassis it was installed in.

I will look forward for your updates

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[MShepard]

Yes, that is correct.  I removed the S5500HCV server board form my SC5600LX chassis and installed the S2600CW board.  All of the critical connectors and wiring harness are the same across product generations.

I anticipate some issues with a few chassis peripherals.  However, the S2600CW has provisions to be mated with non-Intel chassis by system integrators, according to the Intel documentation, so a prior-generation Intel chassis should also be workable.  The P4304XXMUXX chassis does not support 10 3.5" HDD which I require in my application, only 4x3.5" and 8x2.5", else I would have upgraded the chassis as well.  2.5" HDD are not available in the large sizes as 3.5" HDD.

And in any case, I do not see how a failure to recognize on-board NICs or other PCIe devices is related to the details of the chassis.  Can you describe what specific chassis features are a problem?  The redundant power supplies and power distribution board are the same and fully compatible, in particular the PMbus management communications.  The same redundant fans are supported.  Differences in the front panel controls are minor and the SC5600LX front panel controls have the same 24-pin connector and all the controls function properly.  It's really just the HSBP management connections that differ, and I do not see how that impacts the inability to recognize the on-board NICs and CPU2 PCI devices.  The base version of the P4304XXMUXX  chassis does not have HSBP modules, and thus the S2600CW must function properly without them.  Please be specific here to help me understand the problem you see. 

The presenting problem seems to be a board issue, not a chassis issue.

Thank you.

Mark Shepard

[JoseH_Intel]

Hello MShepard,

Thank you very much for your confirmation about the configuration board/chassis.

I understand you point, however when a board is not installed on a non supported chassis unexpected issues might happen. I think part of the issue could be that you are selecting an Intel chassis option during the SDR/FRU update. In this case you should select non-Intel chassis in order to prevent weird issues. You can try this and let me know. In the mean time I will research for the specific issues that might be generating a conflict here.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[MShepard]

Hi, Jose--

I repeated the FRUSDR update from the latest update package on a USB flash:
fs5:>frusdr.efi /cfg master.cfg

The utility reported that it could not auto-sense the chassis type and I chose the "Other chassis" option.
It then asked me questions about which fans were installed, whether chassis intrusion detection was present, and if the front panel supported an NMI button, and then completed the configuration.
After rebooting, the NIC Configuration BIOS page is still blank and PCIe cards on CPU2 are still not enumerated.

Next, I tried replacing the E5-2680v4 CPU with a E5-2678v3 CPU with exactly the same results.

At this point, unless you can identify some configuration issue, I think we have to conclude that this S2600CW2R server board is defective and initiate a warranty claim.

Best regards,

Mark Shepard

[JoseH_Intel]

Hello MShepard,

I definitely understand your point. As a very last try could you please remove the board from the chassis and test it outside, with just the very minimal connections as 1 processor, 1 memory and power supply. Of course just make sure to lay the board down to a non conductive surface. Try to power it on that way and check the NIC screen under BIOS. Let me know how it goes

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[MShepard]

Thank you, Jose.

I do not have an EEB power supply with long enough cables to power the board on a bench.  However, I now have a P4304XXMUXX chassis and am installing the board in that chassis.  I am waiting for the 750W power supplies, but will re-load the FRUSDR for this chassis and test the board in a few days when the power supplies arrive and let you know then.

Best regards,

Mark Shepard

[JoseH_Intel]

Hello MShepard,

That sounds like a plan. I will wait for your updates.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[JoseH_Intel]

Hello MShepard,

I am just following up just to check if you have any updates about this issue. I will try to reach you back on next Thursday 31st.

We'll look forward for your updates

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[MShepard]

Thanks for checking, Jose.

Still waiting for power supplies, delayed due to vendor closing for holidays.  They shipped yesterday (12/28), delivery scheduled tomorrow (12/30).  Tests should not take long after I receive them, hopefully answers sometime 12/31.

Mark

[MShepard]

Hi, Jose-

I have received and installed power supplies.  My S2600CW2R board is installed in a P4304XXMUXX chassis.  I re-flashed the FRUSDR and it seemed to recognize the chassis and configure properly without asking about fans, NMI, etc.

The BIOC NIC configuration screen is still blank.

Attached are Sysinfo logs from this configuration.  It does not appear that the NIC problem is chassis dependent, it just seems to be a defective board.  Do you agree?  Do you have other configuration suggestions that might resolve the problem?  If not, how do I initiate a warranty claim?

Best regards,

Mark 

[JoseH_Intel]

Hello MShepard,

Happy new year and thanks for the update. I think you have done everything we have suggested and the issue persists. Please allow me some time to check with our senior team and if they agree we will be able to process a warranty replacement.

I will update you soon.

Regards

Jose A.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios