Bootable VROC RAID of 3rd pary M2 SSDs on x299 MB/ i9-7980XE set up and key options

[AAbra7]

I have carefully read threads on the VROC on x299 topic but am still struggling with the answers to these questions and am hoping you can help me:

I want build a system with a bootable VROC RAID 0 of 2 Samsung SM961(M.2) on the Asus Rampage VI Extreme X299 MB with an i9-7980XE CPU. (MB have a VROC key slot.) I want to run windows 10 64 bit pro.

Is this possible?

If so what key do I need? I understand there are three different keys:

VROCISSDMOD (intel SSDs only so I assume no)

VROCSTANMOD raid 0,1,10

VROCPREMMOD raid 0,1,5,10

Can I use a raid card, i.e. ASUS hyper M.2 x16 PCIe card with one of the 3 VROC keys above and create a bootable VROC RAID 0,1, 5,10 of 4 Samsung SM961 on the same MB configuration (Asus Rampage VI Extreme X299 MB with a i9-7980XE CPU).

Thanks for your help!

IntelBu1ld3r

[BWalu]

I can't say that my experience with VROC is good. Actually it was total waste of money in my case but at least I got 4x16GB Optane drives for tests really cheap on sale.

I'm using ASRock X299 motherboard and it looks like performance with enabled VROC is worse than the performance with disabled VROC and set stripped volume in Windows disk manager.

Here you can see my results and settings using Premium VROC key:

VROC / RAID0 on 4x Optane 16GB / default settings - really low 4KiB Q1T1 performance which is pretty much the most important for daily work on a typical home/office PC

VROC / stripped volume in Windows disk manager on 4x Optane 16GB - really low sequential Q32T1 performance, VROC enabled but RAID is disabled and random 4K results go up ... 

VROC disabled / stripped volume in Windows disk manager on 4x Optane 16GB - the best results in everything once VROC mode is disabled 

Simply once I enable VROC ( no matter what PCIE settings, used slot, 2-4 drives etc. ) then performance is worse than without VROC.

Not to mention that I have 2x Patriot Hellfire M.2 SSD and both run in RAID0/VROC mode without VROC key ( there is 90 day limit in RSTe software but I don't know if it means that RAID won't work after that or what ? ). Once I use the key then drives are suddenly unsupported. I don't know if I have to comment that. Anyway I can confirm that to use VROC with hardware key is required Intel SSD.

[MLam3]

I have an Asus Rampage VI Extreme, an Asus Hyper x16, and 4x Intel 900p SSD.

I can confirm the findings from woomack that when VROC raid is enabled, the random performance is actually worse than a single drive. This has also been tested by Tomshardware and PCPerspective. When I use straight passthru, and then raid the volume using Windows disk management utility, then the 4k performance goes back to the more normal 250MB/s (for QD1) under Crystal Diskmark.

A couple of things I haven't tested yet, first is actually plugging in a standard VROC key. The thing is on its way so I will find out in a few days. But given woomack actually tested using his premium key and got the same results, I doubt the key will make a difference. Second is whether it would be different under Linux. I suspect the Intel RSTe driver is to blame on this. Maybe their Linux driver will perform differently?

Any thought from Intel would be nice. It's such a shame that after buying all these gears, we cant get it to perform correctly.

[SEric]

Are you using the Asus Hyper M.2 x 16 card?

I am also going to use an Asrock board,

and Asus is trying to tell me "Asus boards only" for the Hyper card.

I am getting pretty close here, and don't know if I can handle any more setbacks.

Thanks!

[SEric]

Optane 16 and 32 GB modules use only 2 lanes each.

Your results are using a total of 8 lanes total, not 16

In theory, double the lane bandwidth may double your score.

[BWalu]

It's not an issue of how many PCIE lanes are in use as drives in every case ( all 3 screenshots ) are using PCIE x2. Every drive is connected to separated PCIE 3.0 x8/x16 slot ( 32Gbps M.2 in case of one drive ). In use are slots marked as 2, 3, 5 and M.2_1 as this motherboard has one M.2 socket connected to VMD controller too. 3 drives are connected via Alphacool MX2 PCIE controllers which are able to use full PCIE 3.0 x4 bandwidth and it was tested with other drives.

Single 16GB drive has maximum bandwidth of about 900MB/s and 4 are showing 3700MB/s so it's scalling good in sequential read and based on that I can tell that there is no issue with maximum bandwidth.

https://www.intel.com/content/www/us/en/products/memory-storage/optane-memory/optane-16gb-m-2-80mm.html Intel® Optane™ Memory Series 16GB M.2 80mm

The only difference which I see between this setting and ASUS card is that ASUS card is using one PCIE slot and is working on one VMD ( controller ) while most PCIE slots work as different controllers or are sharing it. In theory difference is only that RAID on different VMD cannot be bootable. In my case 2 slots are sharing the same VMD ( on the screenshot you can see that too ).

It doesn't change fact that in every screenshot, drives configuration is exactly the same and drives are connected to exactly the same PCIE/VMD. As you see, random 4k performance on enabled VROC is pathetic. I also have ASUS X299 TUF Mark 2 motherboard and I was checking the same but on 2 drives and also then results were worse once I enabled VROC.

It's hard not to notice difference between ~35MB/s and ~220MB/s. 35MB/s is performance of typical SATA SSD.

Other view on this issue. When VROC is enabled then every +1 drive is causing random performance or deeper queue results to drop. So exactly in what VROC supposed to perform great and for what it was designed as server/workstation storage. Results are close to: 2 drives in RAID0 = ~140MB/s random 4k, 3 drives = ~70MB/s, 4 drives = ~35MB/s. When I disable VROC and set stripped volume then on 2, 3 and 4 drives performance in random 4k tests is about the same ... and it shouldn't be any different as in RAID, random bandwidth is not going up so this is correct behaviour.

Results on 2 drives on enabled VROC while using the same or different VMD are the same so it's not an issue of different VMD and some communication between them.

I'm using Win10 Pro x64 with the latest updates. OS was reinstalled. Tests were made on new drives. I was checking drives in every possible PCIE configuration. I don't know if there is any new VROC key firmware or any new drivers for VROC. Intel support page is empty and specification for VROC is changing every couple of weeks. ASRock had to change user's manual because of that ( I'm also in contact with motherboard vendors ).

[SEric]

I am wondering if this is a limitation of the Optane modules.

Originally I was going to setup my build as you have,

along the lines of the tweaktown.com article. (World's fastest system drive)

There is something off there, maybe a setting somewhere.

Once the 900P drives were released, I decided to go that route.

With the cost increase, the build is a bit delayed, shall we say.

I found another build using the 900P drives (below), with screen shots.

Optane d0t weebly d0t c0m

The numbers on that one look pretty good with 3 900P drives.

One thing though, he is using a "Premium key". (WTH?)

At least the Hyper card does work, much to my relief,

as I have been reading lately it is for "Asus boards only",

which made no sense to me, and still doesn't.

Think of yourself as an innovator!

When Thomas Edison had 100 non functional light bulb experiments,

he said he didn't fail.

He just found 100 ways not to build a light bulb.

Keep pushing, you'll get it.

I'm right behind you gathering resources.

Edit: Have you tried with the VROCISSDMOD key?

If you need one, I have a spare.

[BWalu]

I have only Premium key.

I don't think it's any issue with Optane drives or motherboard or anything like that. As I said, exactly the same drives, in the same PCIE slots, the same PCIE speed, the same OS/drivers, VROC key installed all the time but the only difference is that in first test VROC key is enabled / PCIE slots configured to support VMD and in last test VROC is disabled and array runs using stripped array set in Windows disk manager. So issue is in the way how VROC works, not in drives or motherboard limits.

I'm using it mainly for tests/reviews and I wouldn't care about VROC if not pure curiosity. This is also why I purchased 4x 16GB drives on black friday sale. Premium key cost me as much as Standard ( long story ) so I can live with that as long as I made everything work at least as fast as without the key. Right now results without VROC key are much better and I can set stripped volume ( so RAID0 ) without VROC using any drive. I was testing it with Patriot Hellfire drives and I could pass 5GB/s sequential read on 2x240 drives ( VROC disabled or enabled without key ).

The only issue is in random bandwidth and I have no idea why performance is getting worse with each next +1 drive also it's clearly scalling down.

On 3 drives I was getting this error ( below ) in first tests after creating new array but it was disappearing and never back after I clicked "mark as normal". Also no issues at all once I added 4th drive. So on 3 drives 1 or 2 randomly picked drives had always an error. Drives passed diagnostic tests and for last 3 days I'm using some programs without any issues 24/7 so again it's not drives issue but VROC itself.

If I'm right then ASUS Hyper X16 card is the only one which let to set RAID on single VMD/single card. I haven't seen any other controller like that and even ASUS one is not available in EU ( I live in Poland ). ASUS supposed to release these cards over half year ago.

[SEric]

BH Photo has the Asus cards in stock as of last night.

I am unsure of they ship overseas, or if there are duties involved.

B&H Photo Video website.

Get rid of spaces between name and add d0t c0m

I am a month or so away from getting to try all of this.

Already have a couple aspirin bottles ready to go.

[SEric]

Seriously, acquire the Intel basic key.

VROCISSDMOD

According to Intel, the only key that works "properly" on X299 with Intel drives,

is the one mentioned above.

The premium key might work, but I think that might be what is holding you back.

mouser d0t c0m has them here in the US, overseas, I don't know.

We have gone 12 rounds on that subject here.

[BWalu]

AFAIK Intel said nothing about other keys. One rep said only something about compatibility using basic key for Intel SSD only ( VROCISSDMOD ). Point is that official data says almost nothing about compatibility with anything and even motherboard vendors can't get clear answer from Intel about compatibility. Think that they had to change user manuals for their motherboards because Intel was changing specification for VROC during year. I'm in contact with ASRock support which is testing VROC on available drives so I guess they will say some more soon.

If I'm right then the only difference between keys is their firmware which is letting to use or only Intel or Intel+3rd party drives. I don't think it's the reason why only random performance is going down once more drives are in use.

I don't want to invest more in this platform till I get some answers. Results without VROC are good enough for me for everything I do, just that key seems useless now. I can wait for new firmware/BIOS and I have some other things to test anyway. I guess I will back here when I find some more.

[BWalu]

I just received reply from ASRock support. Their engineers performed tests on 4x 32GB Optane SSD and their results are about the same as mine so when VROC is enabled then performance is much worse than when array is configured via Windows disk manager ( for that you don't even need RAID controller, not to mention VROC ).

Looks like there is no point to set VROC RAID if you don't have something like ASUS Hyper x16 card and this is one more cost. Pretty disappointing considering that Intel wants additional money to enable VROC and drive support is limited to barely couple of SSD models.

[dmerr6]

I too am pulling my hair out . I have an asus-x299 board, I9 10core cpu. I have the asus hyper m.2 x16 pcie card. also have premium vroc key. cannot get vroc to work with three pm961 m.2 drives. Looks like Intel restricted the x299 chipset to only work with intel ssd. very disappointing. Had I known this ( intel and asus docs not clear) , I would have gone with AMD. their vroc is not restricted.

[BWalu]

Let's say list of disappointed users is quite long. Intel has changed specs couple of times and as I said, some vendors changed info in motherboard manuals ( in online version ). For sure ASRock had to change it 2 times in last months and now there is info that VROC supports only Intel SSD. Intel list of supported SSD is really short and on Intel Optane series drives that I was testing ( as you see in this thread ) and on drives that ASRock tested, performance in VROC is worse than without it.

For me it's really weird situation as VROC was introduced couple of months ago and there are still barely any official documents about it and there is still the same one product page which is clearly confusing for many users.

Some of my contacts asked Intel about VROC but they got barely any info and if large companies can't get clear info then what can do end-users. I'm Intel's partner and I can't get any support as in my area support department is not existing since last month.

[MRobi7]

Would you be surprised to find that the Facebook support for Intel is still supplying the Xeon FAQ to anyone who requests info on VROC implementation on the HEDT X299 chipset?

I'm finding this entire fiasco to be rather pathetic from a world leading tech giant. Is this how Intel panics and responds when their complacency and arrogance is shattered by a market contender?

I don't care that I may need a VROC Key - IF it is reasonably priced; lets be real though - the $8 cost is not going to protect their server SKU's from ingress by the x299 sku's.

BLocking non-intel SSD's is just a total dick move - especially when their SSDs are not competitive with the performance of other consumer products; HEDT is for prosumers and Intel has dropped the ball here massively.

[SEric]

It could have been handled better or sure.

The way the MB mfr's touted it from the X299 launch,

it was like the second coming of Jesus.

However, it is new technology and it does "belong" to Intel.

Maybe there will be firmware updates at some point for the keys.

It has been frustrating for quite a while,

and I did get some incorrect Intel "support" at one point,

and I wound up wasting $100 on a key that likely will not work.

I AM, however, determined to make this work,

and with the recent release of the 900P drives, the performance should be stunning.

It's a pretty exciting time to be doing a HEDT build.

At least Intel's CPU sockets work, and the memory works,

and you can boot from VROC.

It is not a Prius trying to be a Shelby like some others.

[MRobi7]

Does anyone know specifically if the onboard m.2 slots that x299 motherboards have are connected to allow them to function as a bootable CPU PCIe device (in RAID with VROC or otherwise) or are they all run via the PCH and the DMI link?

I'd like to know if I could run a single or dual SSD that isn't going through the DMI link. THe Dimm.2 cards etc that companies like asus offer are also a bit of a ??? for me.

How about the ones on the z370 by comparison (I figured with 16 lanes only it would be DMI / PCH linked)?

[dmerr6]

I am going through vroc headaches. Intel has not made this easy. the only way to make this work on x299 is to have a vroc key and intel m.2 drives. I am using the asus hyper m.2 x 16 , but, it is only as a passthrough. I tried intel 600 p drive with out the vroc key, nightmare. i had to remove them and now I have Samsung pm691 m.2 install in the hyper card. I will try again next week once I get the vroc key. I boot off a crucial ssd and want the m.2 drives as a raid volume through vroc. now, I am using a windows volume.

[MRobi7]

yeah the info I have out of Intel STILL conflicts with the fact that x299 will only accept Intel based SSD's for VROC. It seems the info they are providing is only relevant to the xeon chipset.

Perhaps you can provide insight into the following with question 4 being the most important :

1. Do the m.2 sockets on the motherboard allow direct CPU control of the SSD or do they all go through the PCH lanes? I am thinking the answer is 'no' but wanted to confirm.

I see you are using the Asus PCIe card so I assume you have a bios that lets you break the PCIe slot into 4 groups of lanes? A mainboard that does not allow that would require a PCIe card with a PLX chip like the new Highpoint SSD RAID card from what I have read.

2. Does RAID0 work (VROC enabled) with Intel drives? - ok from your testing it seems the answer is 'No' on x299 for it working and it being bootable...

3. Does a 'Standard' VROC key enable RAID 0 with VROC enabled, and bootable with Intel SSD?

4. Does a 'Standard' VROC key enable RAID 0 with VROC enabled, and bootable with Samsung SSD?

Intel is not sure what its own system does so I would not be surprised if it does actually function if you get the VROC key.

5. Even with a bios enabled array (VROC) a driver is enabled for installation of windows? I'm interested to know as I run scripted backups with programs like Acronis and WinPE recovery environments so have to be able to boot and 'see' my array from other than windows.

Thoughts?

From my own perspective I am really just very keen to get a bios enabled raid array rather than software, though with a decent CPU I don't see it as a hassle I guess if the array is direct to CPU. Being bootable as an array may not be so important - I can use a small SSD just as an OS drive. I don't want my main array being stuck behind the DMI bottleneck though.

Looking at the z370 and the Coffee Lake CPUs with only 16 lanes you can run one GPU at x8, your RAID array (4 drives) at x16 and that's it. I need x8 for my 10GBe card (none of the z370 boards seem to have it though some of the HEDT do). I'm confirming if my 10GBe card can run wirespeed at x4 if I'm just using a single port off it (I have a Intel x540-T2).

[BWalu]

Answers from me:

1. If M.2 socket is using VROC or not depends on the motherboard. For example ASRock X299 Taichi/XE has 1 M.2 socket connected to VROC/CPU - M.2_1 while X299 ITX has 2. Other thing is that VMD are per PCIE slot or shared. You can't make bootable RAID on more than 1 VMD.

2. VROC RAID works with Intel SSD and is bootable as long as RAID is working on 1 VMD.

3. Standard key is the same as Premium but has no RAID5 option so answer is yes. It's bootable as long as it uses one VMD.

4. I wasn't able to run VROC on any non-Intel SSD. Even on Intel SSD performance was lower than using Windows disk manager / stripped volume ( results in this thread ).

5. I'm not sure what about drivers for bootable array. I wasn't testing that.

Right now if you want to keep maximum performance then or use ASUS Hyper X16 card and pass-through mode or set stripped volume in Windows disk manager and use small SSD as OS bootable drive. In my tests VROC was causing stability issues or performance issues. Screenshots are in this thread.

[MRobi7]

When you say passthrough do you mean that the drives are on the PCIe lanes for the CPU and the RAID is just being done in windows? No VROC at all?

[BWalu]

I don't have ASUS Hyper card so I'm not sure how it works but other users said it runs in VROC passthrough mode. It still should use VROC functionality so to be connected to the CPU. If I'm right ( correct me if I'm wrong ) then ASUS card is using 1 VMD as it uses 1 PCIE slot. On ASRock motherboard you can set PCIE slots as VROC or AIC VROC ( simply SSD or add-in-card mode ) but it changes nothing for me and in both cases VROC works in the same way. I assume that for ASUS Hyper card AIC mode it may change something.

VROC key is only locking/unlocking additional modes. It's not adding anything else. If you don't have key and use 2 Intel SSD then you are able to run RAID0/1 too. What more, it's possible to set RAID 0/1 on non-Intel drives without a key but then array won't be bootable and Intel software says something about 90 day free period. I have no idea what will happen after these 90 days.