topic Re: Re:Help! Intel RST RAID 1 Corrupting SSDs in Rapid Storage Technology

Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Wed, 09 Oct 2024 20:22:19 GMT

We have multiple HP Z2 G9 workstations using Intel RST motherboard RAID in RAID 1 configuration. We use Western Digital Black SN850 and Samsung 990 Pro SSDs. Over the past few months, we have experienced multiple problems, to the point of being chronic.

Problems:

1. After the computer sits overnight, it says "Boot Device not found". After power cycling, computer reboots normally.

2. Intel RST reports SMART errors on one or both drives.

3. Intel RST reports one of the drives as "Unidentified" after a few days or weeks.

One computer was in since last November, and just in the past couple weeks is experiencing the "Boot Device not found" error. It restarts normally after soft boot. Driver issue with latest drivers?

We have sent multiple SSDs in under warranty to WD and Samsung. Finally, Samsung reported RAID is corrupting drive on one of the RMA repairs.

All drivers are up to date from HP including BIOS and Intel RST. All computers have 14 gen i7 processors.

Any help would be appreciated since we are at a standstill.

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Fri, 18 Oct 2024 02:38:18 GMT

Hello Michael4000,

Thank you for bringing this to our attention. I understand how concerning it can be to encounter these chronic problems with your HP Z2 G9 workstations, especially when using a RAID 1 configuration.

To better asses the root cause of these problems, could you provide the following additional information?

Was the system working fine before?
When did the issue start occurring?
Are you using the latest Intel® Optane™ Memory and Storage Management application from the Microsoft Store?
How many systems are affected by the issue?
Could you share which version of the Intel RST drivers are currently installed on the workstations experiencing these issues?
You have mentioned that one computer just began experiencing the "Boot Device not found" error. Is this issue affecting all workstations in the same way, or are there variations?
Have you noticed if the issue occurs more frequently with a particular SSD model, or is it affecting both models equally?

Please generate an SSU report to help me further analyze important details on your system. To generate the SSU report, please refer to the article: How to get the Intel® System Support Utility Logs on Windows*. Please send us the generated SSU.txt file.

We appreciate your patience as we work through this, and we'll monitor your feedback to help resolve this as soon as possible.

Best Regards,

Robbie R.

Intel Customer Support Technician

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Mon, 21 Oct 2024 21:08:47 GMT

Hello Michael4000,

I wanted to check if you had the chance to review the questions I posted. Please let me know at your earliest convenience so that we can determine the best course of action to resolve this matter.

Best regards,

Robbie R.

Intel Customer Support Technician

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Tue, 22 Oct 2024 00:50:41 GMT

Hi Robbie,

Thank you for offering to help with this. Here are answers to your questions.

Was the system working fine before? All of the computers were working fine when setup and installed for the user. Some failed within a couple of weeks, and one failed after six months.
When did the issue start occurring? It has been happening for about two months now.
Are you using the latest Intel® Optane™ Memory and Storage Management application from the Microsoft Store? No, I am using the one that HP provides on their Support website.
How many systems are affected by the issue? Seven systems so far.
Could you share which version of the Intel RST drivers are currently installed on the workstations experiencing these issues? The HP version is 20.1.0.1015.1 Rev.F.
You have mentioned that one computer just began experiencing the "Boot Device not found" error. Is this issue affecting all workstations in the same way, or are there variations? Two system had {"Boot Device not found" errors. Two systems had SMART Errors. Two system had "Unrecognized drive" errors.
Have you noticed if the issue occurs more frequently with a particular SSD model, or is it affecting both models equally? We use WD SSDs and Samsung SSDs slightly more often. Both seem to have problems about the same.

The WD drives with SMART errors, I have had to send back to WD to fix or replace. I believe all of them have been replaced rather than be repaired.

Recently, I have found that if the computers that show "Unrecognized drive" are Samsung drives. We have sent them in to Samsung, and they have reset the drives. In the last week, I have started running the Windows Disk Partition "Clean" command, and reinstalled them, and rebuilt the RAID.

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Wed, 23 Oct 2024 17:09:46 GMT

Yes, see above.

We also had another computer that is giving an error "Boot device not found". That makes three computers.

We had another computer this morning that was frozen. I went into the BIOS Intel RST, and one of the drives had dropped out of the RAID. I tried to reset the disk to non-RAID, and then Rebuild with it, but it would not accept the drive (click on it and nothing happened). I had to remove the drive, put it into a USB to NVMe adapter, run the Windows Disk Partition program "Clean" command. I replaced the drive, and it is rebuilding the RAID now.

Something horrible has happened to Intel RST. It is totally unreliable now.

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Tue, 29 Oct 2024 02:54:57 GMT

Hello Michael4000,

Thank you for answering the question and for the additional information. I will now raise this concern to my team and we will further review this issue internally. We will get back to you as soon as possible. Thank you for your patience.

Best Regards,

Robbie R.

Intel Customer Support Technician

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Spookytemplate — Thu, 31 Oct 2024 11:51:24 GMT

The best way to be able to help is to generate forced kernel memory dump using tool like NotMyFault (can be easily found on the web).

Please generate once you see any problem with RAID being booted to the OS. Dump file should be zipped. From such dump we could try to extract necessary information for root causing this problem.

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Thu, 31 Oct 2024 16:28:23 GMT

Thanks for the follow-up.

Do I run this after the computer fails to boot with "Boot device not found", and then after I get the computer booted?

Also, what option do I use in NotMyFault? As you can see below, it has a lot of options.

Thanks!

Re: Help! Intel RST RAID 1 Corrupting SSDs

Spookytemplate — Thu, 31 Oct 2024 18:06:55 GMT

The best moment is whenever you see anything suspicious about RAID in Windows. NotMyFault work in windows only.

It will trigger Bugcheck and after reboot there will be memory.dmp in C:\Windows.

Default option High IRQL is perfect. Additionally you can change dump type to Kernel dump using steps described here How to manage crash dump settings on Windows 10 | Windows Central .

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Sun, 03 Nov 2024 20:46:16 GMT

Hello Michael4000,

Thank you for your patience. I would like for you to try these following recommendations:

Ensure that the BIOS settings are correctly configured for RAID.
Check the power settings in both the BIOS and Windows to make sure the drives are not being powered down or put into a lower-power state that they can't recover from.
Verify the firmware for both the Western Digital Black SN850 and Samsung 990 Pro SSDs is up to date.
Ensure that the latest Intel RST drivers are installed.
Verify the RAID configuration in the Intel RST software. Rebuilding the RAID array might help if the configuration has become corrupted.

For more information, you may also see the following links below:

Finally please try to download and use the Intel® Optane™ Memory and Storage Management software and kindly let us know the outcome.

We look forward to your response!

Best Regards,

Robbie R.

Intel Customer Support Technician

Re: Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Wed, 06 Nov 2024 21:45:23 GMT

Hello Michael4000,

I hope you had the opportunity to review the information I posted. At your earliest convenience, please let me know so we can determine the best course of action to resolve this matter efficiently.

Best regards,

Robbie R.

Intel Customer Support Technician

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Thu, 07 Nov 2024 05:56:18 GMT

I have had two more computers showing that one of the drives in the RAID was unrecognized. Both were HP Z2 G9 computers with Samsung 990 Pro 1 TB SSDs running RAID 1.

The first computer I unplugged the power from the system (HP Z2 G9). After a couple minutes, I replugged the the power, booted it, and it recognized the second drive in the RAID 1, and started rebuilding on it's own after booting into Windows. I was able to see this with the Intel RST software.

On the second computer, I ran the the Intel RST software, and it gave an error message saying "APPLICATION FAILED TO LAUNCH" with error code 0xA0080000. It say to visit "http://www.intel.com/support/optaine-memory, which I did. I ended up running the "Intel Driver and Support Assistant". It did not find any out of date drivers related to the disk system. It generated a report which is attached.

Storage

Intel Raid 1 VolumeDriver DetailsProviderVersionDateDevice DetailsCapacitySerial NumberPartitionsDevice IdDevice PathFirmware DetailsVersionC:File SystemCompressedCapacityFree Space

Microsoft

10.0.22621.3672

2006-06-21

931.50 GB

Volume1

SCSI\DISK&VEN_INTEL&PROD_RAID_1_VOLUME\4&23F2E5E4&0&010000

\\.\PHYSICALDRIVE0

1.0.00

NTFS

False

930.70 GB

840.20 GB

Here are answers to your questions:

Ensure that the BIOS settings are correctly configured for RAID.

Confirmed.

Check the power settings in both the BIOS and Windows to make sure the drives are not being powered down or put into a lower-power state that they can't recover from.

Set to never go to sleep.

Verify the firmware for both the Western Digital Black SN850 and Samsung 990 Pro SSDs is up to date.

I was unable to do this. When the system is set to RAID, the Samsung Magician software doesn't recognize individual drives. I'm not sure how to check this.

Ensure that the latest Intel RST drivers are installed.

Confirmed with both HP's Image Assistant and Intel's Driver and Support Assistant.

Verify the RAID configuration in the Intel RST software. Rebuilding the RAID array might help if the configuration has become corrupted.

Verified. Rebuilding seems to work, but only temporarily.

Re: Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Thu, 07 Nov 2024 19:56:36 GMT

Yes, please see above.

Thanks!

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Tue, 19 Nov 2024 01:29:13 GMT

Hello Michael4000,

Thank you for the detailed update. I appreciate the time you took to provide the answers and additional context.

To further investigate the issue, kindly answer the additional questions:

How long has this configuration been in use, especially with the RAID setup?
To confirm, have you already backed up your current RAID data?

I would also like to request a screenshot of your Intel RST Console showing the status of both physical and virtual drives. If possible, if you have your RAID Logs with you, kindly provide them here as well.

We look forward to your response!

Best Regards,

Robbie R.

Intel Customer Support Technician

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Wed, 20 Nov 2024 23:27:55 GMT

Hi,

You're welcome.

We have been using Intel RST RAID 1 on all HP workstations since 2007.

We have been using Intel RST RAID 1 on SATA SSD workstation since 2018.

We have been using Intel RST RAID 1 on NVMe SSD HP workstations since 2019.

We have been using Intel RST RAID 1 with NVMe SSDs on HP Z2 G9 workstations (the computer model we using now and having problems with) since September 2022.

User data is stored on servers. The operating system, programs, and all settings are not.

One of the questions asked was if the firmware on the SSDs is up-to-date. We had another workstation on Monday have the error message "Boot device not found" after the computer sat powered on over the weekend. The user hit F2, and it started right up. I checked the SSD firmware for the Samsung 990 Pro 1 TB SSDs on that computer, and it was current at version 4B2QJXD7.

Today another HP Z2 G9 computer Intel RST RAID 1 Windows 11 Version 23H2 with Samsung 990 Pro SSD Firmware up-to-date, "Boot device not found".

This is chronic and repeatable. I would recommend Intel buy one of these HP Z2 G9s and a couple of Samsung 990 Pro drives, and see why these are dying. Let it sit with the power on, leave it over the weekend, and you'll see what I am seeing.

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Wed, 27 Nov 2024 05:21:41 GMT

Hello Michael4000,

Thank you for providing such detailed information about the history of your configurations and the specific issues you're encountering with the HP Z2 G9 Workstations.

I would like to request a SSU Log on one of the machines that is encountering the RAID 1 corruption. To generate the SSU report, please refer to the article: How to get the Intel® System Support Utility Logs on Windows*. Please send us the generated SSU.txt file.

Please answer the additional following questions:

What is the exact model number of the HP Z2 G9 systems? If possible, please provide a weblink of the exact system.

Do you have data on the SMART errors on the nvme drives that had problems?

Were you able to generate a forced kernel memory dump using NotMyFault? If so, please give me a copy of it. You may compress it using 7z format since it is large.

Does the system have option for hot swappable nvme? This can be checked in the BIOS.

We look forward to your response!

Best Regards,

Robbie R.

Intel Customer Support Technician

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Mon, 02 Dec 2024 02:30:00 GMT

Hello Michael4000,

I wanted to check if you had the chance to review the additional questions I posted. Please let me know at your earliest convenience so that we can determine the best course of action to resolve this matter.

Best regards,

Robbie R.

Intel Customer Support Technician

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Mon, 02 Dec 2024 03:40:22 GMT

Thanks for checking in Robbie. I have not be able to review the questions above. I have been out sick for over a week. I'll review in a few days when I feel better.

Thanks!

Re:Help! Intel RST RAID 1 Corrupting SSDs

RobbieR_Intel — Fri, 06 Dec 2024 00:07:59 GMT

Hello Michael4000,

Thank you for your update. I hope you get well.

I will wait for your response.

Best Regards,

Robbie R.

Intel Customer Support Technician

Re: Re:Help! Intel RST RAID 1 Corrupting SSDs

Michael4000 — Sun, 05 Jan 2025 00:40:33 GMT

Hi Robbie,

Well, that was miserable, but I have finally recovered from my illness.

I wish I could say the same for the Intel RST RAID situation. They are still failing.

I have tried a couple of changes to the new systems we are putting in to see if it had any effect.

1. I used Windows 11 Version 24H2 instead of 23H2. 24H2 is the latest version that just came out. This was with Samsung 990 Pro 1TB with Heatsinks. Unfortunately, that did not help.

It resulted in the usual "Unknow hard disk (0 bytes)" message. See below.

2. I tried using HP branded 512 GB SSDs that I had in stock. So far those are working. Even if they work long term, HP is no longer selling them, so we can't build new systems with them.

If I were to guess, I would say that Intel RST needs updating for the new faster SSDs, such as Samsung 990 Pro and Western Digital Black SN850X.

Intel needs to investigate this. Every computer running RAID 1 RST is failing on this configuration. Here are the part numbers of the systems we are building:

HP Z2 G9 P/N A1NX0UT#ABA

https://www.hp.com/us-en/shop/pdp/hp-z2-tower-g9-workstation-wolf-pro-security-edition-p-a1qh2ua-aba-1

Note: HP uses different part numbers on their online store, but this one is the same model and specs as the ones sold retail.

The single 512 GB SSD in the HP Z2 G9 is replaced by two:

Samsung 990 Pro with Heatsink 1 TB SSDs P/N MZ-V9P1T0

https://www.samsung.com/us/computing/memory-storage/solid-state-drives/990-pro-w-heatsink-pcie-4-0-nvme-ssd-1tb-mz-v9p1t0cw/

RAID 1 is used.