Help! Intel RST RAID 1 Corrupting SSDs

[Michael4000]

We have multiple HP Z2 G9 workstations using Intel RST motherboard RAID in RAID 1 configuration.  We use Western Digital Black SN850 and Samsung 990 Pro SSDs.  Over the past few months, we have experienced multiple problems, to the point of being chronic.  

Problems:

1. After the computer sits overnight, it says "Boot Device not found".   After power cycling, computer reboots normally.

2.  Intel RST reports SMART errors on one or both drives.

3.  Intel RST reports one of the drives  as "Unidentified" after a few days or weeks.

One computer was in since last November, and just in the past couple weeks is experiencing the "Boot Device not found" error.  It restarts normally after soft boot.  Driver issue with latest drivers?

We have sent multiple SSDs in under warranty to WD and Samsung.  Finally, Samsung reported RAID is corrupting drive on one of the RMA repairs.

All drivers are up to date from HP including BIOS and Intel RST.  All computers have 14 gen i7 processors.

Any help would be appreciated since we are at a standstill.

[RobbieR_Intel]

Hello Michael4000,

Thank you for bringing this to our attention. I understand how concerning it can be to encounter these chronic problems with your HP Z2 G9 workstations, especially when using a RAID 1 configuration.

To better asses the root cause of these problems, could you provide the following additional information?

• Was the system working fine before?
• When did the issue start occurring?
• Are you using the latest Intel® Optane™ Memory and Storage Management application from the Microsoft Store?
• How many systems are affected by the issue?
• Could you share which version of the Intel RST drivers are currently installed on the workstations experiencing these issues?
• You have mentioned that one computer just began experiencing the "Boot Device not found" error. Is this issue affecting all workstations in the same way, or are there variations?
• Have you noticed if the issue occurs more frequently with a particular SSD model, or is it affecting both models equally?

Please generate an SSU report to help me further analyze important details on your system. To generate the SSU report, please refer to the article: How to get the Intel® System Support Utility Logs on Windows*. Please send us the generated SSU.txt file.

We appreciate your patience as we work through this, and we'll monitor your feedback to help resolve this as soon as possible.

Best Regards,

Robbie R.

Intel Customer Support Technician

[Michael4000]

Hi Robbie,

Thank you for offering to help with this.  Here are answers to your questions.

• Was the system working fine before?  All of the computers were working fine when setup and installed for the user.  Some failed within a couple of weeks, and one failed after six months.
• When did the issue start occurring?  It has been happening for about two months now.
• Are you using the latest Intel® Optane™ Memory and Storage Management application from the Microsoft Store?  No, I am using the one that HP provides on their Support website.
• How many systems are affected by the issue?  Seven systems so far.
• Could you share which version of the Intel RST drivers are currently installed on the workstations experiencing these issues?  The HP version is 20.1.0.1015.1 Rev.F.
• You have mentioned that one computer just began experiencing the "Boot Device not found" error. Is this issue affecting all workstations in the same way, or are there variations?  Two system had {"Boot Device not found" errors.  Two systems had SMART Errors.  Two system had "Unrecognized drive" errors.
• Have you noticed if the issue occurs more frequently with a particular SSD model, or is it affecting both models equally?  We use WD SSDs and Samsung SSDs slightly more often.  Both seem to have problems about the same.

The WD drives with SMART errors, I have had to send back to WD to fix or replace.  I believe all of them have been replaced rather than be repaired.

Recently, I have found that if the computers that show "Unrecognized drive" are Samsung drives.  We have sent them in to Samsung, and they have reset the drives.  In the last week, I have started running the Windows Disk Partition "Clean" command, and reinstalled them, and rebuilt the RAID.

[RobbieR_Intel]

Hello Michael4000,

I wanted to check if you had the chance to review the questions I posted. Please let me know at your earliest convenience so that we can determine the best course of action to resolve this matter. 

Best regards,

Robbie R.

Intel Customer Support Technician

[Michael4000]

Yes, see above.

We also had another computer that is giving an error "Boot device not found".  That makes three computers.

We had another computer this morning that was frozen.  I went into the BIOS Intel RST, and one of the drives had dropped out of the RAID.  I tried to reset the disk to non-RAID, and then Rebuild with it, but it would not accept the drive (click on it and nothing happened).  I had to remove the drive, put it into a USB to NVMe adapter, run the Windows Disk Partition program "Clean" command.  I replaced the drive, and it is rebuilding the RAID now.

Something horrible has happened to Intel RST.  It is totally unreliable now.

[RobbieR_Intel]

Hello Michael4000,

Thank you for answering the question and for the additional information. I will now raise this concern to my team and we will further review this issue internally. We will get back to you as soon as possible. Thank you for your patience.

Best Regards,

Robbie R.

Intel Customer Support Technician

[Spookytemplate]

The best way to be able to help is to generate forced kernel memory dump using tool like NotMyFault (can be easily found on the web).

Please generate once you see any problem with RAID being booted to the OS. Dump file should be zipped.  From such dump we could try to extract necessary information for root causing this problem.

[Michael4000]

Thanks for the follow-up.

Do I run this after the computer fails to boot with "Boot device not found", and then after I get the computer booted?

Also, what option do I use in NotMyFault?  As you can see below, it has a lot of options.

Thanks!

[Spookytemplate]

The best moment is whenever you see anything suspicious about RAID in Windows. NotMyFault work in windows only. 

It will trigger Bugcheck and after reboot there will be memory.dmp in C:\Windows.  

Default option High IRQL is perfect. Additionally you can change dump type to Kernel dump using steps described here How to manage crash dump settings on Windows 10 | Windows Central .

[RobbieR_Intel]

Hello Michael4000,

Thank you for your patience. I would like for you to try these following recommendations:

• Ensure that the BIOS settings are correctly configured for RAID.
• Check the power settings in both the BIOS and Windows to make sure the drives are not being powered down or put into a lower-power state that they can't recover from.
• Verify the firmware for both the Western Digital Black SN850 and Samsung 990 Pro SSDs is up to date.
• Ensure that the latest Intel RST drivers are installed.
• Verify the RAID configuration in the Intel RST software. Rebuilding the RAID array might help if the configuration has become corrupted.

For more information, you may also see the following links below:

• Intel® Optane™ Memory and Storage Management Application Shows Message "At risk (SMART event)"
• How to Confirm If Intel® Rapid Storage Technology/Intel® Optane™ Memory Is Being Used in My System

Finally please try to download and use the Intel® Optane™ Memory and Storage Management software and kindly let us know the outcome.

We look forward to your response!

Best Regards,

Robbie R.

Intel Customer Support Technician

[Michael4000]

I have had two more computers showing that one of the drives in the RAID was unrecognized.  Both were HP Z2 G9 computers with Samsung 990 Pro 1 TB SSDs running RAID 1.

The first computer I unplugged the power from the system (HP Z2 G9).  After a couple minutes, I replugged the the power, booted it, and it recognized the second drive in the RAID 1, and started rebuilding on it's own after booting into Windows.  I was able to see this with the Intel RST software.

On the second computer, I ran the the Intel RST software, and it gave an error message saying "APPLICATION FAILED TO LAUNCH" with error code 0xA0080000.  It say to visit "http://www.intel.com/support/optaine-memory, which I did.  I ended up running the "Intel Driver and Support Assistant".  It did not find any out of date drivers related to the disk system.  It generated a report which is attached.

Storage

Intel Raid 1 VolumeDriver DetailsProviderVersionDateDevice DetailsCapacitySerial NumberPartitionsDevice IdDevice PathFirmware DetailsVersionC:File SystemCompressedCapacityFree Space

Microsoft

10.0.22621.3672

2006-06-21

931.50 GB

Volume1

3

SCSI\DISK&VEN_INTEL&PROD_RAID_1_VOLUME\4&23F2E5E4&0&010000

\\.\PHYSICALDRIVE0

1.0.00

NTFS

False

930.70 GB

840.20 GB

Here are answers to your questions:

• Ensure that the BIOS settings are correctly configured for RAID.

Confirmed.

• Check the power settings in both the BIOS and Windows to make sure the drives are not being powered down or put into a lower-power state that they can't recover from.

Set to never go to sleep.

• Verify the firmware for both the Western Digital Black SN850 and Samsung 990 Pro SSDs is up to date.

I was unable to do this.  When the system is set to RAID, the Samsung Magician software doesn't recognize individual drives.  I'm not sure how to check this.

• Ensure that the latest Intel RST drivers are installed.

Confirmed with both HP's Image Assistant and Intel's Driver and Support Assistant.

• Verify the RAID configuration in the Intel RST software. Rebuilding the RAID array might help if the configuration has become corrupted.

Verified.  Rebuilding seems to work, but only temporarily.

[RobbieR_Intel]

Hello Michael4000,

I hope you had the opportunity to review the information I posted. At your earliest convenience, please let me know so we can determine the best course of action to resolve this matter efficiently.

Best regards,

Robbie R.

Intel Customer Support Technician

[Michael4000]

Yes, please see above.

Thanks!

[RobbieR_Intel]

Hello Michael4000,

Thank you for the detailed update. I appreciate the time you took to provide the answers and additional context.

To further investigate the issue, kindly answer the additional questions:

• How long has this configuration been in use, especially with the RAID setup?
• To confirm, have you already backed up your current RAID data? 

I would also like to request a screenshot of your Intel RST Console showing the status of both physical and virtual drives. If possible, if you have your RAID Logs with you, kindly provide them here as well.

We look forward to your response!

Best Regards,

Robbie R.

Intel Customer Support Technician

[Michael4000]

Hi,

You're welcome.

We have been using Intel RST RAID 1 on all HP workstations since 2007.

We have been using Intel RST RAID 1 on SATA SSD workstation since 2018.

We have been using Intel RST RAID 1 on NVMe SSD HP workstations since 2019.

We have been using Intel RST RAID 1 with NVMe SSDs on HP Z2 G9 workstations (the computer model we using now and having problems with) since September 2022.

User data is stored on servers.  The operating system, programs, and all settings are not.

One of the questions asked was if the firmware on the SSDs is up-to-date.  We had another workstation on Monday have the error message "Boot device not found" after the computer sat powered on over the weekend.  The user hit F2, and it started right up.  I checked the SSD firmware for the Samsung 990 Pro 1 TB SSDs on that computer, and it was current at version 4B2QJXD7.

Today another HP Z2 G9 computer Intel RST RAID 1 Windows 11 Version 23H2 with Samsung 990 Pro SSD Firmware up-to-date, "Boot device not found".

This is chronic and repeatable.  I would recommend Intel buy one of these HP Z2 G9s and a couple of Samsung 990 Pro drives, and see why these are dying.  Let it sit with the power on, leave it over the weekend, and you'll see what I am seeing.