I'm setting up an all-flash vSAN 6.6 platform and my capacity tier is Intel DC P4500 SSDPE2KX020T7 NVMe drives with firmware QDV1013D. vSAN shows warning on controller firmware in health check, it recommends VDV10110. I installed SSD Data Center Tool version 3.0.9 (tried older versions too) onto ESXi 6.5 U1 to update firmware but it does not work. Here are the steps per instructions:
esxcli software acceptance set --level=CommunitySupported
esxcli software acceptance get
esxcli software vib install -v /tmp/intel_ssd_data_center_tool-3.0.9-400.vib
esxcli software vib list | grep intel
intel_ssd_data_center_tool 3.0.9-400 Intel CommunitySupported 2017-12-18
./isdct show -intelssd
./isdct set -system LogFile = 'log.txt'
./isdct set -system EnableLog = 'true'
Set EnableLog successful.
./isdct show -intelssd
21:22:08:164981 [debug] tdk_interface_dll.cpp(259) SendXMLMessage: Entering
21:22:08:167676 [debug] TDKInterface.cpp(522) scan: Entering
21:22:08:167697 [debug] DeviceFinder_NVME.cpp(63) scan: Entering
21:22:08:167709 [debug] DeviceFinder_NVME.cpp(100) scanPhysicalDevices: Entering
21:22:08:167854 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:167883 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme0
21:22:08:167950 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:167966 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme1
21:22:08:168013 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:168027 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme2
21:22:08:168072 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:168086 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme3
21:22:08:168129 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:168144 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme4
21:22:08:168187 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:168202 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme5
21:22:08:168245 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:168259 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme6
21:22:08:168303 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:168317 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme7
21:22:08:168360 [error] Connection_Path.cpp(91) open: vmk_MgmtUserInit error: 19
21:22:08:168374 [error] DeviceFinder_NVME.cpp(134) scanPhysicalDevices: Failed to connect to device. Ignoring intel-nvme8
Any ideas how I can go about updating the firmware so that vSAN can verify it? TIA!
Thank you for contacting our support community. We understand that you are not able to update the firmware of your SSD.
In order to better understand your situation, could you please provide us the following information:
*Could you please confirm if the SSD is connected directly to the motherboard?
*How many SSDs are affected?
We look forward to hearing back from you.
It's a Supermicro 2028U-TN24R4T+ server. There are 20-2TB SSD DC P4500 drives. The drives are connected directly to the backplane controller. Drive has firmware QDV1013D, but according to the Firmware Listing page https://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Firmware Versions for Intel® Solid State Drives and Intel® Optane™... QDV10130 is the latest. Is QDV10130 newer than QDV1013D? I also tried Firmware Update Tool and it reports QDV1013D is up to date. Thanks!
Thanks for the reply.
We would like to let you know that the latest firmware version for the Intel® SSD DC P4500 Series is QDV10150. This information is available in the Intel® SSD Data Center Tool https://downloadmirror.intel.com/27248/eng/Intel_SSD_Data_Center_Tool_3_0_9_Release_Notes_022.pdf release notes. We will provide the feedback in order to update the https://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Firmware Versions for Intel® https://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Solid Statehttps://www.intel.com/content/www/us/en/support/articles/000017245/memory-and-storage.html Drives and Intel® Optane™ Memory article.
Since you have the SSDs connected to the backplane controller, we recommend you to connect one of the SSDs directly to the motherboard in order to perform the firmware update. Otherwise, the tool will not be to perform it.
Also, We would like to let you know that the Intel® SSD Firmware Update Tool does not support this SSD.
We look forward to hearing back from you.
This usually occurs when using the inbox-driver (driver/module name "nvme", aka "vmw"-version) - DCT won't find anything. You need to use the intel-version of the nvme-driver (called intel-nvme, also referred to as "partner async native driver") for the DCT to detect your intel-device.
In your case (to follow VMware VSAN HCL on VSAN 6.6), that's intel-nvme 22.214.171.124, for P4500. Install the driver + reboot (ESXi should now load the intel-nvme module, or you may need to remove other nvme-drivers)
Direct link to the intel-nvme 126.96.36.199 for P4500, supported on vSAN 6.6 (HCL):
https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-INTEL-NVME-12115&productId=614 VMware ESXi 6.5 intel-nvme 188.8.131.52 NVMe Driver for Intel(R) Solid-State Drive DC P3700, P3600, P3500, P3520, P4500, D3600, and P4800X NVM Express SSDs
HCL (VSAN/SSD-category) for P4500:
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=41822 Intel® SSD DC P4500 Series SSDPE2KX020T7 (2 TB, 2.5-inch)
Other drivers on HCL (not necessary on VSAN HCL!), including download links:
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=43948&vcl=true Intel SSD DC P4500 Series SSDPE2KX020T7 (2.0 TB, 2.5-inch)
Merry x-mas :-).
Thanks for replying! I forgot to mention that I also have Intel DC P3700 SSD drives for caching tier and its firmware is 8DV1071. According to vsAN HCL page for P3700 https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=40168&deviceCategory=ssd&details=1&vsan_type=vsanssd&ssd_partner=46&ssd_releases=278,279&keyword=P3700&vsanrncomp=true&page=1&display_interval=500&sortColumn=Partner&sortOrder=Asc
the intel-nvme 184.108.40.206 is not even listed. I'm running nvme 220.127.116.11-2vmw and it satisfies with 8DV1071 (P3700) but does not with QDV1013D (P4500). That's my dilemma right now.
Well. This is almost funny, as we sort of have the same "issue". That's with the P3700 (for cache), but P4600 for capacity, currently on 6.0 U3 (hopefully upgraded soon). For now, it's running stable with intel-nvme 18.104.22.168 and FW 8DV101H0. Luckily we have the option to separate the DGs with, leaving the P3700s on different hosts (and other capacity devices), and using P4800X (Optane) and P4600 together (which currently have matching drivers, at least on 6.5 U1).
Hopefully the VSAN HCL certification process will be better matched in the future, at least by the same vendor/devices. There will always be a balance between good drivers, testing, and early releases (which might include issues). Yet, this leaves a lot of customers on older software/versions, pending updated VSAN HCL certification, and sometimes not being able to even use their hardware investment (e.g. device is supported, but newer stock firmware on delivered device, which is not certified yet.. Intel won't usually recommend downgrade (makes sense), and VSAN certification takes it time.. leaving the customer with a "supported device, but not on your current firmware level).
Who knows, maybe VMware will support loading different modules (drivers) based on e.g. PCI/VEN/DID/SVID (etc.) in the future. For now, we'll just have to wait (I know..).
Anyways, back to your case; if you like to change the firmware with the DCT (isdct), you'll need "intel-nvme" drivers (won't find any devices with the inbox driver/"vmw").
Would't worry much about the "display name" - rather focus on actual supported versions (on VSAN HCL). Sometimes the health check is also buggy (even seen wrong device IDs in the VSAN HCL-databasen/JSON-file).
But 22.214.171.124 isn't supported on P3700, right.. The intel-nvme 126.96.36.199 only applied when just using P4500. So I'd maybe rather go with the 188.8.131.52-2 (which is supported by both devices, on VSAN 6.6), and temporary use a intel-nvme (version "x"), just to use the DCT, changing firmware to a VSAN HCL-supported version (means downgrading P4500 firmware in your case.. which Intel won't like, but hey..). When using the DCT - make sure you specify the correct devices (see documentation on DCT).
Thanks for the pointers about buggy health check and HCL takes time. For now I will settle with 184.108.40.206-5 vs 220.127.116.11-4 recommended driver. Health check is no longer complaining about firmware after installing ESXi latest build. Merry Xmas and thanks for your help!
I would stick to the online version of the VSAN HCL (web version), not the health check recommendations (which may be incorrect, based on the display name being wrong). If you locate the hardware IDs of your P4500 device, you'll know for sure. I've posted about this in VMTN, so just have a quick look for the commands here: https://communities.vmware.com/message/2598822# 2598822 Intel DC P3700 Firmware |VMware Communities
Also, I don't see 18.104.22.168-5 or 22.214.171.124-4 for VSAN "6.6".. but then again, 126.96.36.199-4 is supported for "ESXi 6.5 U1", which is VSAN 6.6.1, so it depends on how you read it . Have fun with that one! Always verify with VMware (make SR) if you need to know for sure. IMO health check (at least previous versions) have been so "buggy" with P3700, that I'll rather verify the hardware IDs & the online VSAN HCL manually. Hopefully it will be fixed.. in version.next.