RSTe service crashes when Storage Spaces is in use

[LNava3]

I've got a Supermicro X9SRE-F motherboard with the 4.1.0.1026 RSTe option ROM and 5.2.2.1022 RSTe drivers & UI. I'm using RSTe to run a boot drive that consists of 2 Samsung SSD 850 PRO SSDs in RAID 1. The OS is Windows Server 2016 Standard.

This system also has an LSI/Avago 9305-16i.

I've discovered that the RSTe service (IAStorDataMgrSvc) will error out immediately upon starting if I've got powered up drives connected to the LSI card *and* the drives are in a Storage Spaces pool. RSTe starts up normally if I have a non-Storage Spaces drive connected to either the LSI card or to another onboard Intel SATA port or if I have no drives (besides the two in the RAID 1).

I'm guessing that the RSTe service is trying to probe or manage the Storage Spaces drives when it shouldn't. Is there a way to configure the RSTe service to not do this?

Here's the error that I see in the Event Viewer Application log:

Log Name: Application

Source: Application Error

Date: 3/3/2018 1:14:48 PM

Event ID: 1000

Task Category: (100)

Level: Error

Keywords: Classic

User: N/A

Computer: server.domain.com

Description:

Faulting application name: IAStorDataMgrSvc.exe, version: 5.2.2.1022, time stamp: 0x5964cea7

Faulting module name: ISDI2.dll, version: 5.2.2.1022, time stamp: 0x5964cd32

Exception code: 0xc0000409

Fault offset: 0x0006149f

Faulting process id: 0x59c

Faulting application start time: 0x01d3b334a80abb3c

Faulting application path: C:\Program Files (x86)\Intel\Intel(R) Rapid Storage Technology enterprise\IAStorDataMgrSvc.exe

Faulting module path: C:\Program Files (x86)\Intel\Intel(R) Rapid Storage Technology enterprise\ISDI2.dll

Report Id: ce7dc142-5c02-4011-90fa-961ea558953b

Faulting package full name:

Faulting package-relative application ID:

Event Xml:

http://schemas.microsoft.com/win/2004/08/events/event http://schemas.microsoft.com/win/2004/08/events/event">

1000

2

100

0x80000000000000

3566

Application

callisto.lunadesign.net

IAStorDataMgrSvc.exe

5.2.2.1022

5964cea7

ISDI2.dll

5.2.2.1022

5964cd32

c0000409

0006149f

59c

01d3b334a80abb3c

C:\Program Files (x86)\Intel\Intel(R) Rapid Storage Technology enterprise\IAStorDataMgrSvc.exe

C:\Program Files (x86)\Intel\Intel(R) Rapid Storage Technology enterprise\ISDI2.dll

ce7dc142-5c02-4011-90fa-961ea558953b

Many thanks in advance!

[idata]

Hello xyzzy6,

Thanks for contacting the Intel® Rapid Storage Technology enterprise (Intel® RSTe) support forum; in regards to the situation that you've commented we'll like to have more information about your setup to help us determine what could be causing this situation that you've described. Please reply to this message with the following information: Processor model:RAID Controller cards Manufacturer and models (If there's more than one):Amount of RAM that your equipment has installed:Amount of drives, capacity and type of all the drives connected when the error happens:If possible please attach us the LOG Files generated by RSTe, depending on your Windows Version they could be found in the C:\Users\[username]\Intel folder or also you can use the Email option on the RSTe GUI to generate a report for your equipment.Also, please remember that if you require more assistance or want to add additional details, we'll be glad to assist youRegards,Chris

[LNava3]

Hi Chris,

Here are the answers to your questions:

> Processor model:

Xeon E5-1650 V2

> RAID Controller cards Manufacturer and models (If there's more than one):

No RAID controller cards. The only storage card is the one LSI 9305-16i HBA I mentioned above.

> Amount of RAM that your equipment has installed:

64 GB ECC DDR3

> Amount of drives, capacity and type of all the drives connected when the error happens:

2 Samsung SSD 850 Pro 256 GB (MZ-7KE256), both connected to onboard SATA 3 ports

4 WD Red 8 TB (WD80EFZX), all 4 connected to the LSI HBA and configured in Storage Spaces in a single pool, with a single virtual drive that supports a single volume

> If possible please attach us the LOG Files generated by RSTe, depending on your Windows Version they could be found in the C:\Users\[username]\Intel folder or also you can use the Email option on the RSTe GUI to generate a report for your equipment.

I don't have a C:\Users\[username]\Intel folder. I did a search for all *.log files on C: and the only ones I found under an "Intel" folder were in C:\Users\[username]\AppData\Local\Temp\Intel\Logs. However, they are all dated from the day I installed the system. None of them have dates since RSTe stopped working (after the drives were set up under Storage Spaces).

The RSTe GUI (IAStorUI) no longer starts since this problem started occurring. I believe this is because it relies on the RSTe service.

Is there a way to start the RSTe service in a debug mode so it provides more info about why its failing?

[idata]

Hello xyzzy6,

Thank you for your patience and the information that you've provided us; in regards to the situation that you're experiencing with your server we'll need a brief time to further investigate and if more information it's required we'll be back with you as soon as possible. Please remember that if you require more assistance or want to add additional details, we'll be glad to assist youRegards,Chris

[LNava3]

Thanks Chris....I really appreciate your assistance as its very important that this server's boot drive is protected from drive failures (thus the RAID 1 mirror).

[LNava3]

Chris - Any update on this? I'm concerned that I've got a server in production that is "flying blind" in that I can't get any indications of array issues with the RSTe service non-functional.

[idata]

Hello xyzzy6,

We are sorry for the delay in our responce, the issue that you are experiencing is currently being investigated in order to determine the root cause of the issue,We will contact you back as soon as we get further updates Best RegardsLuis H

[LNava3]

Hi Luis,

Thanks for your response. However, are there any more details you can provide so I can plan accordingly?

For example: Has your team been able to reproduce this? Is there a rough time frame that I might expect a response (are we talking a few more days, weeks, months)? Most importantly, is the array safe to use in the meantime without the RSTe service? Are there any workarounds that I can use in the meantime to check the array's health?

Thanks!

[JTerr6]

I also have a similar issue with Windows Server 2016 on Supermicro X10SDV-TLN4F motherboard, with the same RSTe version.

RSTe CLI crashes when Windows Storage Spaces are built on AHCI passthru disks attached to the SATA controller in RAID mode.

We originally wanted to configure 3 x RAID 1 volumes on 6 disks attached to the SATA controller in RAID mode. This caused various issues with verification errors on the largest RAID 1 (1.92TB). In the RSTe GUI help I found documented a limit on number of arrays/volumes. (Feedback - It is poor that this is buried in GUI help, I cannot find this limitation documented anywhere else by Intel.). Hence we were forced to configure 1 x RAID 1 for OS boot disk on Intel RSTe, and configure an additional 2 x mirrored storage spaces on the other 4 disks in the Windows OS.

I am sorry details are sketchy, but I can provide further information if I refresh my research of a few months back. Basically Windows Storage Spaces in the host OS breaks RSTe management, whether this is via the GUI/Service or CLI. So, we too have a 'flying blind' server, where Intel RSTe cannot be managed or monitored. We rely on health status of volumes in the Windows OS, which is not ideal.

I did find a release note for RST (non-Enterprise), which referenced fixing an issue with RST GUI crashing with Windows 8 Storage Spaces. Apologies again, I do not have a reference to the document and RST version. It appears a fix was made in RST and may not have been applied also for RSTe.

Thanks,

Justin

[LNava3]

justin.terry -- Thanks for sharing your experience. It sounds like we're in the same boat.

Intel -- Any update on this? Have you been able to reproduce this? Is the array safe to use in the meantime without the RSTe service?

[LNava3]

Chris & Luis H - Any update on this? Its been a month since I originally reported this. Thanks!

[JTerr6]

Hi, I did track down the release note that I found for Intel RST (non-enterprise), it is in Intel's document 'Intel® Rapid Storage Technology 12.5.0.1066 Production Version Release Notes', 22 March 2013. Under 'Resolved Issues in 12.5.0.1032 Pre-Beta' there is: -

4161049

UI crashes if a Win 8 "Storage Spaces" volume is present

Windows 8,Windows 8 - 64

At the time I was building our server I did wonder if a fix for storage spaces had been done for RST, but never made it into the RST(e) branch.

Maybe totally irrelevant, but it would be good to hear some comment back from Intel. Windows Storage Spaces + Intel RST(e) does appear to be broken, at least on some platforms.

Thanks, Justin

[LNava3]

Is anyone from Intel still following this thread?

Any updates? Did you see @justin.terry's helpful info?

[idata]

Hello xyzzy6,

Apologies for the delay in our responceWe have gather all the information provided by Justin and it has bee forwarded to our engineering team, meanwhile in regards to the issues that you are experiencing our recommendation will be disable the Intel® VMD on those PCIE slots that the HWRAIUd controller is connected to, this is because Intel VMD its recommended to be used behind a switch.We would to gather more information about why are you trying to use multiple raid solutions at the same time?Best RegardsLuis H

[LNava3]

Hi Luis,

I'm not sure what you mean by VMD and HWRAIUd?

As for your last question, the setup I'm using isn't that uncommon. I'm using Storage Spaces and ReFS for the data drives because of its integrity checks / self-healing features. However, Windows doesn't support using either of those for a boot drive. So, I'm using the Intel onboard RAID to have a RAID 1 boot drive. (It would be a shame to have a server with resilient storage but with a single point of failure boot disk.)

[LNava3]

It just occurred to me that by "HWRAIUd" you meant HWRAID controller.

I Googled Intel VMD and it appears to be new technology on the Xeon Scalable processors for NVMe hot-plug. It appears to be something that you can disable at the BIOS level for systems using those Xeons. However, I don't think that technology was around in the E5 V2 processor / C600 chipset generation and I haven't seen BIOS settings for anything like that on my system.

[idata]

Hello xyzzy6,

We have informed our engineering team about your system not showing the Intel® VMD option on the BIOS. I will keep following up with them and as soon as we get an update we will contact you back ASAPBest RegardsLuis H

[idata]

Hello xyzzy6,

Thanks for your patience and in regards to your situation and based on the information that you've provided us we'll recommend you to contact your OEM to verify if the version of Intel® RSTe that they've provided you it's the correct one for your equipment; that's because the one you've been provided by your OEM isn't recommended for Ivy Bridge EP based equipment and beginning with RSTe v.5.3.0.1413 it's optimized for more recent systems running VMD Technology.Also, please remember that if you require more assistance or want to add additional details, we'll be glad to assist youBest regards,Chris.

[LNava3]

Hi Chris,

I downloaded this software from the OEM's site specifically from the section for my Ivy Bridge EP based motherboard and the selected OS (Windows Server 2016).

I'm also dubious of this theory as I've fully tested this version of RSTe and it works perfectly as long as there are no drives in a Storage Spaces pool. Has anyone at Intel tried reproducing my scenario?

While I understand that the OEM is the proper channel for distributing the software, Intel (as the creator of the RSTe software, chipsets and CPUs) would best know which version of RSTe is compatible with which chipsets. Surely Intel must have a support matrix doc for this? I'm sure if I ask the OEM, they will send me back to Intel.

Best regards,

xyzzy6

[idata]

Hello xyzzy6,

Thank you for your reply and in regards to your situation, the main reason why we're referring you to the OEM it's because the version of RSTe that they're providing for this motherboard isn't suitable for the Intel® Xeon® Ivy Bridge EP processor line since it requires an Intel® VMD Technology capable motherboard.Additionally as mentioned, we've investigated on your situation and we've concluded that the version of RSTe that your OEM is providing isn't the most appropriate since the VMD requirement.Intel provides the OEM with the RSTe Packages and technologies so they can make the implementation, and they're responsible in the testing of the software on their equipment to ensure that the drivers version that they provide are compatible and the most appropriate.Also, please remember that if you require more assistance or want to add additional details, we'll be glad to assist youRegards,Chris

[LNava3]

Hi Chris,

Your previous post indicated that I should check with the OEM as if you were not sure about the version compatibility. It sounds like you are now sure that the version I have is incompatible with the Ivy Bridge EP processor line.

Since our last interaction, I have submitted a support case with the OEM as I do realize they are responsible for testing. However, to facilitate my discussion with them, can you please provide me with the range of version numbers that are *intended* to be compatible with the Ivy Bridge EP processor line? That would be incredibly helpful.

Regards,

xyzzy6