SSD Data Center tool in ESXi not detecting NVMe SSD DC P4500

[LTran7]

Hi,

I'm setting up an all-flash vSAN 6.6 platform and my capacity tier is Intel DC P4500 SSDPE2KX020T7 NVMe drives with firmware QDV1013D. vSAN shows warning on controller firmware in health check, it recommends VDV10110. I installed SSD Data Center Tool version 3.0.9 (tried older versions too) onto ESXi 6.5 U1 to update firmware but it does not work. Here are the steps per instructions:

esxcli software acceptance set --level=CommunitySupported

esxcli software acceptance get

CommunitySupported

esxcli software vib install -v /tmp/intel_ssd_data_center_tool-3.0.9-400.vib

esxcli software vib list | grep intel

intel_ssd_data_center_tool 3.0.9-400 Intel CommunitySupported 2017-12-18

cd /opt/intel/isdct

./isdct show -intelssd

No results

Enable logging:

./isdct set -system LogFile = 'log.txt'

./isdct set -system EnableLog = 'true'

Set EnableLog successful.

./isdct show -intelssd

No results

more log.txt

21:22:08:164981 [debug] tdk_interface_dll.cpp(259) SendXMLMessage: Entering

21:22:08:167676 [debug] TDKInterface.cpp(522) scan: Entering

21:22:08:167697 [debug] DeviceFinder_NVME.cpp(63) scan: Entering

21:22:08:167709 [debug] DeviceFinder_NVME.cpp(100) scanPhysicalDevices: Entering

21:22:08:167854 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:167883 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme0

21:22:08:167950 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:167966 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme1

21:22:08:168013 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:168027 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme2

21:22:08:168072 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:168086 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme3

21:22:08:168129 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:168144 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme4

21:22:08:168187 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:168202 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme5

21:22:08:168245 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:168259 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme6

21:22:08:168303 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:168317 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme7

21:22:08:168360 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19

21:22:08:168374 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme8

Any ideas how I can go about updating the firmware so that vSAN can verify it? TIA!

[idata]

Hi virtualtran,

Thank you for contacting our support community. We understand that you are not able to update the firmware of your SSD.

In order to better understand your situation, could you please provide us the following information:

*Could you please confirm if the SSD is connected directly to the motherboard?

*How many SSDs are affected?

We look forward to hearing back from you.

Regards,

Junior M.

[LTran7]

Hi Junior,

It's a Supermicro 2028U-TN24R4T+ server. There are 20-2TB SSD DC P4500 drives. The drives are connected directly to the backplane controller. Drive has firmware QDV1013D, but according to the Firmware Listing page https://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Firmware Versions for Intel® Solid State Drives and Intel® Optane™... QDV10130 is the latest. Is QDV10130 newer than QDV1013D? I also tried Firmware Update Tool and it reports QDV1013D is up to date. Thanks!

[idata]

Hi virtualtran,

Thanks for the reply.

We would like to let you know that the latest firmware version for the Intel® SSD DC P4500 Series is QDV10150. This information is available in the Intel® SSD Data Center Tool https://downloadmirror.intel.com/27248/eng/Intel_SSD_Data_Center_Tool_3_0_9_Release_Notes_022.pdf release notes. We will provide the feedback in order to update the https://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Firmware Versions for Intel® https://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Solid Statehttps://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Drives and Intel® Optane™ Memory article.

Since you have the SSDs connected to the backplane controller, we recommend you to connect one of the SSDs directly to the motherboard in order to perform the firmware update. Otherwise, the tool will not be to perform it.

Also, We would like to let you know that the Intel® SSD Firmware Update Tool does not support this SSD.

We look forward to hearing back from you.

Regards,

Junior M.

[EOdeg]

Hi!

This usually occurs when using the inbox-driver (driver/module name "nvme", aka "vmw"-version) - DCT won't find anything. You need to use the intel-version of the nvme-driver (called intel-nvme, also referred to as "partner async native driver") for the DCT to detect your intel-device.

In your case (to follow VMware VSAN HCL on VSAN 6.6), that's intel-nvme 1.2.1.15, for P4500. Install the driver + reboot (ESXi should now load the intel-nvme module, or you may need to remove other nvme-drivers)

Direct link to the intel-nvme 1.2.1.15 for P4500, supported on vSAN 6.6 (HCL):

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-INTEL-NVME-12115&productId=614 VMware ESXi 6.5 intel-nvme 1.2.1.15 NVMe Driver for Intel(R) Solid-State Drive DC P3700, P3600, P3500, P3520, P4500, D3600, and P4800X NVM Express SSDs

HCL (VSAN/SSD-category) for P4500:

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=41822 Intel® SSD DC P4500 Series SSDPE2KX020T7 (2 TB, 2.5-inch)

Other drivers on HCL (not necessary on VSAN HCL!), including download links:

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=43948&vcl=true Intel SSD DC P4500 Series SSDPE2KX020T7 (2.0 TB, 2.5-inch)

Merry x-mas :-).

Regards,

Espen Ødegaard

[LTran7]

Hi Espen,

Thanks for replying! I forgot to mention that I also have Intel DC P3700 SSD drives for caching tier and its firmware is 8DV1071. According to vsAN HCL page for P3700 https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=40168&deviceCategory=ssd&details=1&vsan_type=vsanssd&ssd_partner=46&ssd_releases=278,279&keyword=P3700&vsanrncomp=true&page=1&display_interval=500&sortColumn=Partner&sortOrder=Asc

the intel-nvme 1.2.1.15 is not even listed. I'm running nvme 1.2.0.32-2vmw and it satisfies with 8DV1071 (P3700) but does not with QDV1013D (P4500). That's my dilemma right now.

Thanks!

[EOdeg]

Well. This is almost funny, as we sort of have the same "issue". That's with the P3700 (for cache), but P4600 for capacity, currently on 6.0 U3 (hopefully upgraded soon). For now, it's running stable with intel-nvme 1.2.0.8 and FW 8DV101H0. Luckily we have the option to separate the DGs with, leaving the P3700s on different hosts (and other capacity devices), and using P4800X (Optane) and P4600 together (which currently have matching drivers, at least on 6.5 U1).

Hopefully the VSAN HCL certification process will be better matched in the future, at least by the same vendor/devices. There will always be a balance between good drivers, testing, and early releases (which might include issues). Yet, this leaves a lot of customers on older software/versions, pending updated VSAN HCL certification, and sometimes not being able to even use their hardware investment (e.g. device is supported, but newer stock firmware on delivered device, which is not certified yet.. Intel won't usually recommend downgrade (makes sense), and VSAN certification takes it time.. leaving the customer with a "supported device, but not on your current firmware level).

Who knows, maybe VMware will support loading different modules (drivers) based on e.g. PCI/VEN/DID/SVID (etc.) in the future. For now, we'll just have to wait (I know..).

Anyways, back to your case; if you like to change the firmware with the DCT (isdct), you'll need "intel-nvme" drivers (won't find any devices with the inbox driver/"vmw").

Br.

Espen

[LTran7]

So I installed intel-nvme 1.2.1.15 driver and drives are now detected as P3520 PCIe instead of NVMe P4500 . Latest build, ESXi, 6.5.0, 7388607.

and firmware is "N/A":

[EOdeg]

Would't worry much about the "display name" - rather focus on actual supported versions (on VSAN HCL). Sometimes the health check is also buggy (even seen wrong device IDs in the VSAN HCL-databasen/JSON-file).

But 1.2.1.15 isn't supported on P3700, right.. The intel-nvme 1.2.1.15 only applied when just using P4500. So I'd maybe rather go with the 1.2.0.32-2 (which is supported by both devices, on VSAN 6.6), and temporary use a intel-nvme (version "x"), just to use the DCT, changing firmware to a VSAN HCL-supported version (means downgrading P4500 firmware in your case.. which Intel won't like, but hey..). When using the DCT - make sure you specify the correct devices (see documentation on DCT).

.espen

[LTran7]

Thanks for the pointers about buggy health check and HCL takes time. For now I will settle with 1.2.0.32-5 vs 1.2.0.32-4 recommended driver. Health check is no longer complaining about firmware after installing ESXi latest build. Merry Xmas and thanks for your help!

[EOdeg]

I would stick to the online version of the VSAN HCL (web version), not the health check recommendations (which may be incorrect, based on the display name being wrong). If you locate the hardware IDs of your P4500 device, you'll know for sure. I've posted about this in VMTN, so just have a quick look for the commands here: https://communities.vmware.com/message/2598822# 2598822 Intel DC P3700 Firmware |VMware Communities

Also, I don't see 1.2.0.32-5 or 1.2.0.32-4 for VSAN "6.6".. but then again, 1.2.0.32-4 is supported for "ESXi 6.5 U1", which is VSAN 6.6.1, so it depends on how you read it . Have fun with that one! Always verify with VMware (make SR) if you need to know for sure. IMO health check (at least previous versions) have been so "buggy" with P3700, that I'll rather verify the hardware IDs & the online VSAN HCL manually. Hopefully it will be fixed.. in version.next.