Cannot see disks attached to retimer AXXP3RTX16040

[STurc]

Hello!

We are using R1208WFTYS system with retimer card AXXP3RTX16040. We see disks in slots 0-3 connected to on-board OCuLink connectors with no problem in BIOS and OS (Linux). But we are unable to see disks connected to retimer card neither in OS nor in BIOS.

Retimer installation was done according to the R1000WF_SystemIntegration_and_ServiceGuide.pdf # 2.8.3: retimer installed in Riser#2, PEM Cable to motherboard, and 4-opt OCuLink Cable Bundle connected accordingly. We inserted two drives into Slot#5 and #6 and yet none of them is visible in BIOS "PCI Configuration\UEFI Option ROM Control", although I see there drives from slot #0-3. Same with OS Linux - we see drives in slots #0-3 but not those that are connected to retimer.

We tried changing RiserSlot#2 biffurcation settings: Auto, x16, x4x4x4x4 - same result - no disks.

BMC sensors that monitor HDD Slots occupation on the backplane show that disk is inserted (HDD x Status | Drive Slot | ... |  'Drive Presence').

BUT! RMM4 Web Console UI somehow is able to see them on page "System\NVME Information" - I have no idea how RMM/BMC was able to obtain that information.

We also have VROC STDMOD activator key on this system and we tried setting VMD to "off" and to "on" in PCI Link settings with no success: we see drives in slots #0-3 (on-board OCuLink), but not in slot #5-6 (retimer).

[JoseH_Intel]

Hello STurc,

Thank you for joining the community.

Since you say you followed the steps from the integration guide, I will assume that all the steps are followed included the PEM cable installed and the NVMe population rules for VROC. Now, a couple questions more, do you have 2 processors installed in this board, and what RAID configuration do you have currently setup, RAID mode or VROC?

The riser #2 and #3 will only be recognized if CPU #2 is populated. About the RAID configuration, RAID mode will use the embedded RAID controller in the board (RSTe, ESRT2), while VROC will use the RAID on Chip managed by the CPU directly.

Will look forward for your udpates.

Regards

Jose A.

Intel Customer Support

For firmware updates and troubleshooting tips, visit:

https://intel.com/support/serverbios

[STurc]

Hello!

I do have CPU #2 installed, of course. (by the way, drives in bay #2 and #3 are routed to CPU#2 - and I do see them).

As for RAID - I did not configure RAID in BIOS.  I use NVMe drives as a raw storage to OS.

As I said - I see first 4 dives (bay0-3) connected to OCuLink connectors on motherboard but not those that are connected to retimer.

An output of 'lspci -v | grep NVMe' also shows only first 4 dives (bay0-3) in output.

[JoseH_Intel]

Hello STurc,

Thanks for the updates. Let me research a bit more on your issue and will let you know as soon as I have updates

Regards

Jose A.

Intel Customer Support

[JoseH_Intel]

Hello STurc,

Could you please check if VMD support is enabled ?, if so, that is only needed for RAID setups. (disabled on raw storage)

In addition to the previous, please capture a sysinfo log using the following tool: https://downloadcenter.intel.com/download/29677/System-Information-Retrieval-Utility-SysInfo-

And last but not least please provide the brand and model of your drives and tell if the ones in the onboard OCuLink ports are the same as the ones used in the retimer card.

We will look forward for your updates.

Regards

Jose A.

Intel Customer Support

[STurc]

Hello!

> Could you please check if VMD support is enabled ...

We tried both with VMD enabled and disabled. Same result: we see only drives in slot 0-3.

(By the way, VMD is need not only in RAID set-ups but also it is a must for backplane led management, at least "ledctl" utility works only with VMD enabled)

> please capture a sysinfo 

done, attached.

> please provide the brand and model of your drives and tell if the ones in the onboard OCuLink ports are the same as the ones used in the retimer card.

Intel P4610 1.6TB in all of 0-3,5,6 drive bays. So, yes, they all the same.

And yet, as I said, somehow RMM Web UI "NVMe" information page shows those drives (in 0-3,5,6 slots) with their serial numbers and so on..... One could argue that this is an OS issue, But BIOS also sees drives only in 0-3 bays .

[STurc · Accepted solution]

An update. We actually bought two retimer cards. Second card arrived just recently. We installed it to the same server and it works just perfect - we see disks attached to retimer! So it appears that the first retimer card is defective.  Will open a tiket for RMA.

Jose A., thank you for your suggestions anyway. Your help is really appreciated.

With best regards,

Sergei Turchanov.