NUC8i7BEH & TEKQ Cube TB3 NVMe SSD Enclosure: HotPlug Failure in Win10

[SChau10]

I wish to report a case of hot-plug failure with TEKQ Cube TB3 NVMe SSD enclosure and NUC8i7BEH (Bean Canyon).

Attached is a table of test results and test configuration.

When a TEKQ Cube TB3 NVMe SSD enclosure with a Samsung 960 PRO NVMe PCIe M.2 1TB SSD (or 950 PRO 512GB) installed is hot-plugged into the TB3 port of my NUC8i7BEH running Win10 Pro 1809, this external SSD is not recognized by Win10 and disk partitions in the SSD do not come online. Two yellow bang errors appear in Windows device manager, with one belonging to the Intel TB3 controller in the NUC.

Failure rate of this scenario is 100% repeatable. Please note the NUC8i7BEH has no trouble supporting Win10 OS boot from this external NVMe SSD whenever it is pre-attached before OS boot. This device is also properly recognized after booting OS from internal SSD, but only if this external SSD is pre-attached to the NUC before OS boot.

I would like Intel to assist in replicating and troubleshooting this issue.

[Alberto_R_Intel]

SChau10, Thank you for posting in the Intel® Communities Support. In the following link you will find the list of tested peripherals that were tested by Intel® and were validated as compatible with the Intel® NUC Kit NUC8i7BEH, as you can see the TEKQ Cube TB3 NVMe is not validated or showing in that list, so we cannot guarantee it wil work for sure: http://compatibleproducts.intel.com/ProductDetails?activeModule=Intel%C2%AE%20NUC In this case I encourage all the peers in the community to confirm if they are currently using the TEKQ Cube TB3 NVMe with the Intel® NUC Kit NUC8i7BEH without problems and if they have further suggestions for this scenario. Just to make sure, how many NUC's do you have that are showing this problem? We can always try a BIOS update to the latest version 0064: https://downloadcenter.intel.com/download/28569/BIOS-Update-BECFL357-86A-?product=126140 Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[SChau10]

Alberto, I have just one NUC8i7BEH and one TEKQ Cube NVMe enclosure having this issue. I tried 3 different TB3 0.5m (40Gbps) cables with same results.

BIOS of this NUC8i7BEH has already been updated to 0064 with the jumper recovery method.

The TEKQ Cube NVMe enclosure uses an Alpine Ridge JHL6340 Thunderbolt 3 interface controller and a TI TPS65983 USB-C/USB PD controller. Hot-plugging of the TEKQ Cube working with my NUC7i5BNH (1 unit) & NUC6i7KYK (3 units) but not with NUC8i7BEH was unexpected. There appears to be a compatibility issue between NUC8i7BEH & TEKQ Cube, but only specifically with hot-plug. Other aspects of operation, including getting Win10 OS to boot from the pre-attached TEKQ Cube, are all normal as far as I can tell. We are looking at a corner case here.

[Alberto_R_Intel]

SChau10, Thank you for providing those details. I sent you an email asking about some specifics about your system since I am not sure you want that information to be public or not. Please verify your inbox. Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[Alberto_R_Intel]

Hi SChau10, Thank you very much for clarifying that information and for providing those details. Since Intel® has not validated the TEKQ Cube NVMe enclosure, we cannot confirm if it will work as expected with the Intel® NUC Kit NUC8i7BEH and Intel does not purchase products to test. From our side, we can recommend that you have the latest BIOS for the Intel® NUC, latest firmware for the TEKQ Cube NVMe enclosure and the latest firmware on any drives installed in the NUC or enclosure. We would also like to confirm that the Operating System is activated. We will provide feedback to the product team with your findings so they will consider reporting that the TEKQ Cube NVMe enclosure may not be compatible with the Intel® NUC Kit NUC8i7BEH per customer reporting. Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[SChau10]

Hi Alberto,

I confirm that the Windows 10 Pro 1809 OS installations in all NUCs tested with TEKQ Cube NVMe enclosure are activated: "Windows is activated with a digital license".

I believe it is too early to declare the TEKQ Cube NVMe enclosure being incompatible with NUC8i7BEH. Last night, I managed to reproduce the exact same hot-plug failure symptoms with the TEKQ Cube plugged into the TB3 port of an ASRock Z390 Phantom Gaming ITX/AC motherboard, also running Win10 Pro 1809. It is beginning to look like the TEKQ Cube has hot-plug difficulty specifically with Coffee Lake based platforms.

Fundamentally, though, I believe TEKQ Cube NVMe enclosure is well-designed. Booting Win10 OS installed into an SSD inside this enclosure has been rock solid with all my NUCs featuring TB3 port. Successfully tested NUC models include NUC8i7BEH, NUC7i5BNH, NUC6i7KYK for booting OS from this external enclosure. Same also for the ASRock Z390 ITX motherboard. I therefore believe this hot-plug issue is not related to TEKQ or motherboard hardware, but rather a corner case possibly involving device and controller firmware, TB3 device driver or Thunderbolt settings in BIOS. It almost appears some hot-plug related code in Coffee Lake platforms is running too fast and leading to a premature timeout.

TEKQ has not released any firmware update for their Cube.

More testing is called for and I will share more results as they become available. I'll try the TEKQ Cube with more Coffee Lake platforms to see if a pattern emerges. Stay tuned.

[Alberto_R_Intel]

SChau10, Thank you very much for letting us know those updates. Perfect, excellent, take your time and once you get the chance please let us know the results. Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[SChau10]

I posted a detailed update to this thread about 2 hours ago, but it disappeared completely within minutes of posting.

What's going on???

[Alberto_R_Intel]

SChau10, Yes, let me apologize for that, I saw it, I was working on the response and all of a sudden it disappear. Could you please re-post it again? Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[SChau10]

Alberto, it looks like my post got deleted again! I saw the post in "review pending" state before it disappeared.

Did you pick up the post before it got deleted? Should I send you a PM with the content?

[Alberto_R_Intel]

Hi SChau10, Yes, I saw it, I was able to save this time. Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[Alberto_R_Intel]

This is the original post provided by SChau10: I decided to focus on further hot-plug testing the combo of NUC8i7BEH & TEKQ Cube NVMe enclosure (referred below as: device). Here are some new observations (#1 & #2 reported previously): 1. Boot Win10 (from internal SSD) with device pre-attached to TB3 port: WORKS! (SSD partition comes online) 2. Hot-plug device into TB3 port after Win10 bootup: FAILS! ("STATUS_POWER_DEVICE_FAILURE" Code 10 yellow bang error in "Thunderbolt(TM) Controller - 15D9" device in Device Manager, SSD partition not online) 3. With failed device still attached, manually uninstall "Thunderbolt(TM) Controller - 15D9" yellow bang device in Device Manager, click "Scan for hard changes": FAILS! (Thunderbolt controller returns with same yellow bang error) 4. With failed device still attached, manually uninstall "Thunderbolt(TM) Controller - 15D9" yellow bang device in Device Manager, put system into Sleep (S3) for a few seconds, wake system back to Windows desktop: WORKS! 5. With working device attached, eject & unplug device and re-attach within ~10 seconds: WORKS! 6. With working device attached, eject & unplug device for more than ~15 seconds, then re-attach: FAILS! 7. With failed device still attached, restart Win10: WORKS! Observation details: 1. When detaching a working device, a Windows system sound ("da-du-dong") for device unplug is heard immediately, then ~10 seconds later, the same sound is heard again, when the entire Thunderbolt device stack, including a few "PCI Express Upstream Switch Port" and "PCI Express Downstream Switch Port" device nodes and "Thunderbolt(TM) Controller - 15D9", all disappear (de-enumerate) from Device Manager, leaving just the parent "Intel(R) PCI Express Root Port #5 - 9DBC" device node still visible. 2. If working device is detached and re-attached before the second Windows system sound is heard, the device will work again. 3. If working device is detached and re-attached after the second Windows system sound is heard, the device will FAIL with the same yellow bang error in the "Thunderbolt(TM) Controller - 15D9" device node. I essentially found a quick workaround for this hot-plug failure with NUC8i7BEH: - Manually un-install the yellow bang "Thunderbolt(TM) Controller - 15D9" device node within Device Manager, then take the system through a Sleep/Wake cycle. After waking the system, the TEKQ Cube enclosure will be back online. Also, doing Sleep/Wake first followed by manual Thunderbolt device node un-installation and "Scan for hardware changes" also restores the attached device to working. Such a workaround is of course not ideal, and the hot-plugged device should just work without needing any help from such a workaround. Please note that similar failure symptoms have been observed with other TB3 equipped systems, so this hot-plug failure is by no means limited to the NUC8i7BEH (Bean Canyon), though this NUC appears to reproduce the failure symptoms most easily and consistently. I now suspect that this hot-plug failure is power related. Since the TEKQ Cube enclosure is a bus-powered Thunderbolt device, during hot-plug both TB3 link and power delivery negotiations have to occur, and the "STATUS_DEVICE_POWER_FAILURE" message of the yellow bang error suggests the hot-plug failure is associated more with device power negotiation than with TB3 link negotiation. It may also help explain why the device works solidly in non-hot-plug scenarios, including pre-attach before OS boot: power negotiation and TB3 link negotiation are not concurrent in time for the device. The hot-plug failure appears to be a corner case involving simultaneous power and link negotiations.

[Alberto_R_Intel]

SChau10, Thank you very much for the detailed response, that information is very helpful and also will be useful for all the peers viewing this thread. Since we have not validated this device, it is difficult to say it will work as expected. Some recommendations you could try are the following to see if the issue is replicated: -Test another cable with the TEKQ Cube -Test the TEKQ Cube without any drives installed Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[SChau10]

Hi Alberto,

Thank you for the update.

I have already tested this TEKQ Cube enclosure with three TB3 0.5m (40Gbps) cables (two from StarTech), and there was no difference in issue symptoms among them.

The Amazon web page for TEKQ Cube mentions this device as being "Intel Certified" (in one of the product images). Not sure if this means the device actually completed Thunderbolt certification with Intel. It would be helpful if Intel can look up your Thunderbolt certification database for the TEKQ Cube. A confirmation of the device having passed formal Intel Thunderbolt certification would be a strong vote of confidence for this product.

As mentioned before, the TEKQ Cube uses JHL6340 Alpine Ridge 2C Thunderbolt interface controller and a Texas Instruments TPS65983 USB-C/USB PD controller in its design. Nothing unusual in the hardware design as far as I can tell. TEKQ has not released any firmware update for the Cube.

My next steps:

• Install different NVMe PCIe M.2 SSDs into the TEKQ Cube
• Change Thunderbolt Windows driver revision

[Alberto_R_Intel]

SChau10, Thank you very much for providing those updates. We will do further research on this matter and as soon as I get any details I will post them on this thread. Once you get the chance, please let us know the results of trying the next steps you mentioned previously. Regards, Alberto R. Intel Customer Support Technician Under Contract to Intel Corporation

[SChau10]

Quick update:

I just updated Thunderbolt driver package from 17.4.77.400 to 17.4.78.500 for the NUC8i7BEH. HotPlug failure symptoms with TEKQ Cube NVMe enclosure have not changed. The previously mentioned workaround involving a Sleep/Wake cycle is still effective.

BTW, NVM firmware version for the Thunderbolt 3 (15D9) controller in the NUC8i7BEH is 33.00. I believe this is the firmware shipping with Bean Canyon NUCs, and I don't remember having run a Thunderbolt firmware update for this NUC either.

[Ronny_G_Intel]

Hi SChau10, We are looking into this issue, Thunderbolt by definition is "hot-plug". By any chance, did you try the Thunderbolt Control Center App available here: https://www.microsoft.com/en-us/p/thunderbolt-control-center/9n6f0jv38ph1#activetab=pivot:overviewtab When installed, under the devices you will have an option to approve devices: Connect only once or Always connect. Can you please try that and let me know the results. Thanks, Ronny G

[SChau10]

Hi Ronny,

I can't get the Thunderbolt Control Center App installed for my NUC8i7BEH. This may be because I'm running a legacy Thunderbolt driver not a DCH Thunderbolt driver. I also don't use a Microsoft account with this NUC and not sure if that matters for the app installation.

The TEKQ Cube NVMe enclosure has been approved as "Always connect", and works quite well if pre-attached to NUC8i7BEH before booting Windows. Only the hot-plug consistently fails (has never worked even once). Same failure and symptoms also with my ASRock Z390 Phantom gaming-ITX/AC motherboard featuring one TB3 port. No hot-plug trouble with my other Thunderbolt NUCs: NUC7i5BNH and NUC6i7KYK.

[Ronny_G_Intel]

Hi SChau10, We dont have the exact same SSD enclosure for testing but we tried 2 models we have available in one of our labs. They are very similar to the one you have and both work. We also look for the specifications about the two we tried and the TEKQ and all three use the same controller and parts, I'm not sure what else to do, this seems to be a compatibility issue, I guess thats the obvious but the problem is that I dont have any recommendation at this point. What we tried: Patriot EVLVR - https://www.info.patriotmemory.com/evlvr-ssd Samsung X5 (sounds like a car) - https://www.samsung.com/semiconductor/minisite/ssd/product/portable/x5/ Regards, Ronny G

[SChau10]

Hi Ronny,

Thanks for the update.

Even though all three TB3 devices are based on the same Intel TB3 controller (Alpine Ridge), it is possible the firmware for the TB3 controller in these devices to be different?

This may or may not be a coincidence, but the hot-plug failure with the TEKQ Cube reproduces consistently only with my latest PC platforms: NUC8i7BEH and ASRock Z390 Phantom Gaming-ITX/AC motherboard. Both are Intel Coffee Lake platforms.

Also, when observing the hot-plug failure of the TEKQ Cube in action, I noticed there is usually a yellow bang Intel USB 3.1 Host Controller appearing in Windows Device Manager. I find this strange, since the TEKQ Cube should appear as PCIe x4 root port linked through TB3 back to the host, but there should be no USB 3.x host controller, especially since NVMe M.2 SSDs don't talk USB protocol. The yellow bang appears to be a spurious USB device appearing then disappearing during the hot-plug, and Windows reports a device status of "not present". I suspect that a deeper investigation of this issue should help determine whether this misbehavior is on the host-side TB3 controller or the device-side TB3 controller.

I've been testing with TB3 driver 17.4.77.400 which still appears to be the latest, and the TB3 firmware on the 3 NUC models are also up to date. I'd be happy to test newer TB3 driver and/or firmware against this hot-plug issue.

Lastly, I discovered that the TEKQ Rapide NVMe SSD (240GB, 480GB, 960GB) and the TEKQ Cube NVMe SSD TB3 enclosure use the exact same circuit board inside, so the TEKQ Rapide should be usable to replace TEKQ Cube + NVMe SSD for testing purposes. I do not have a TEKQ Rapide SSD nor do I plan on getting one. I also won't buy another TEKQ Cube enclosure at least until this hot-plug issue comes to some kind of closure.

An updated test table is attached.

[Chris_V_Intel]

Hi SChau10,

I work with Ronny and have been the one doing most of the testing on this issue. I want to apologize. I did not test this issue thoroughly enough before Ronny posted last week.

The good/bad news, I was able to see the issue, kind of. Also, we got the TEKQ so now I've tested 3 enclosures and they all act the same way.

Here's what I found:

You are correct. In the BIOS, if you set Thunderbolt Security Level to Legacy, you are not able to hot-plug any of the enclosures.

However, if you set it to the default setting, Unique ID, I am able to hot-plug all 3 enclosures without issue.

First, I will have the BIOS/engineering team look at the legacy issue to see what's going on.

Second, does Unique ID work for you? If so, is there a reason you can't use it? Just curious. :)

Lastly, whether the driver is DCH or not doesn't matter, at least if you're using Unique ID.

As soon as I have anything on Legacy either Ronny or myself will post it here.