NUC9VXQNX Won't Boot - Power LED Blinking

Hossy · ‎06-26-2020

A little bit of background... After a long wait, I finally got a NUC9VXQNX and I began the journey of installing ESXi 6.7. My plan is to install an NVIDIA Quadro P2200 and use PCI pass-through to present it to a VM. Many have done it. I didn't think it would be an issue. Alas, I've had nothing but problems with getting error code 43 in Windows (2012 R2, 2019, BIOS, EFI). I did some troubleshooting and I noticed that if I configured the BIOS to set the Primary Display to Auto or PEG, my code 43 error would go away, but this prevented Intel AMT's Remote Desktop from functioning and also killed the ESXi console mid-startup since ESXi grabs the P2200 for PCI pass-through. My goal is to use the integrated graphics (HDMI) for the ESXi console and the P2200 in the VM.

During my latest "let's try various settings and see what might work" sessions, I increase the IGD Minimum Memory and IGD Aperature Size to their maximum values. The attachment is my record of what the settings were so I could change them back. I haven't read up on these settings if I'm being honest, but I equated them to general video memory settings. I have 64GB of RAM in the server so I didn't think there was any harm in trying a larger memory size. I was wrong.

After this change, I have no video from the IGFX over HDMI or the P2200 over DP. I have two monitors connected to the NUC just in case it decides to use one of the video cards. When the NUC powers on, it takes a while and the DP monitor lights up then goes to standby. Eventually, the power button will begin to blink (0.25 seconds on/off repeatedly).

I have tried doing a BIOS Recovery using the Security Jumper. The 16GB USB is formatted as FAT32, but it doesn't appear to be doing the Recovery. Since I believe this issue to be caused by bad BIOS settings, I tried disconnecting the AC power and CMOS battery for about 10 minutes (or so), but the BIOS settings were retained.

As per usual, technical support is always closed when you actually need them, so I'm reaching out to the community here to see if anyone has an idea of what I could try next. At this point, it's a paperweight, so I'm willing to try anything.

n_scott_pearson · ‎06-27-2020

The BIOS Configuration is stored in flash, not CMOS, and has been stored this way for more than 10 years. Clearing CMOS is used as a signal that a BIOS reset is necessary, but this reset is done using the current configuration. Bottom line, this isn't going to help; you must get into BIOS Setup to clear the bad configuration.
To do this, power off, wait a few seconds and then power back on, but this time do so holding down/in the Power Button until the Power LED turns amber. This should result in display of Power Button Menu. Use F2 in this menu to get into BIOS Setup.

Hope this helps,
...S

Hossy · ‎06-27-2020

Thank you for your response. I also tried getting into the Power Button Menu. The instructions I have for this NUC say to hold the power button in for 3 seconds but to release it before 4 seconds. The Power LED is always white. The first day I was testing it to become familiar with it, but I cannot get the power button menu now. Neither display turns on.

n_scott_pearson · ‎06-27-2020

Hold button in until the LED changes color. If it goes back off, you know you've held it too long.
...S

Hossy · ‎06-28-2020

My LED doesn't change colors. It is always white. What's weird is instead of being solid, it's always flashing (0.25 seconds on/off repeatedly) when it's running. After I turn it off, it will keep flashing for about 10 seconds and then stop. Intel AMT is no longer responding or lighting up the NIC.

I've tried holding in the power button for various lengths of time (up to 10 seconds) and the NUC doesn't respond any differently. It will turn off and back on a couple of times when first starting.

MRoss5200 · ‎06-30-2020

I tried this on a Ghost i9 with BIOS 0036 and a Quartz Xeon with BIOS 0034 and had no problem getting to the Power Button Menu.

The color of the LED does not change so it always stays white.

From Power Off, hold the button down and count to 3 or 4. You should see a flash of the LED, then release the button. 5-10 seconds later the system will display the menu. This is with an HDMI monitor plugged in. If you are using a TB3 adapter to DP or VGA or a x16 PCIe video card, you may have to wait longer to see the screen.

This was done on a Ghost with a 1060 3GB x16 graphics card. The Quartz was done with the internal graphics only.

Hossy · ‎06-30-2020

MRoss5200,

Thank you for testing this and your reply. What you're describing is exactly what I've been attempting. I know it sounds like I'm clueless and not doing it correctly, but before I did anything with the box, I had actually tested and practiced bringing up the power button menu. The thing just doesn't want to come up. I have two monitors hooked up with no converter (straight cable). The Quadro P2200 outputs DP to mini-DP to one monitor. The integrated graphics outputs HDMI to HDMI to another monitor. I have even tried removing the Quadro altogether and still I get no video. Maybe because the BIOS is set the PEG for Primary Display? I would HOPE that Intel programmed the BIOS to fail back to IGFX in the absence of a PEG, but I can't confirm that right now.

LeonWaksman · ‎07-01-2020

Hi @Hossy ,

1. Your display disappeared after changing the IGD settings in BIOS. You should leave the default settings, since those settings are used for BIOS only and don't have any effect on System graphics performance. I don't know the reason why those setting can be changed by users.

2. I suggest that you try to reset BIOS settings to default, using "blind procedure":

3. Press and hold power button - count: 1001, 1002, 1003, 1004. Release power button.

4. Wait about 2 minutes and click on F2.

5. Wait 1 minutes and click on F9, followed by "y". This should set BIOS settings to default.

6. Click on F10, confirmed by "y", to save settings and exit to Windows.

Hope this will help

Leon

n_scott_pearson · ‎07-01-2020

After talking with BIOS developers, my understanding is that the Power LED used on these boards (Ghost Canyon and Quartz Canyon) is not capable of changing color. As a result, in this design, the BIOS flashes this LED to signal that it has accepted the request for the Power Button Menu (PBM). They also indicated that the PBM may take up to 30 seconds to appear (depending upon Primary Display setting), so give it a chance.

Now, my understanding is that the PBM should ignore the BIOS configuration of Primary Display and go through monitor detection. It should thus not be possible for it to not appear. Regardless, Leon's algorithm for blind BIOS Configuration reset is a good thing to remember...

...S

Hossy · ‎07-02-2020

Thank you Scott and Leon,

I did try the "blind procedure" (a few times) but without success. As far as giving time for the PBM, I have, at times, given it over an hour and sometimes overnight to appear without success either. Intel AMT, at one point, started working again and I was able to see in the event log that it was logging no system memory installed. However, because of the BIOS time being reset when I pulled the battery, I couldn't correlate the timing to my testing with Intel technical support when we powered the system up purposefully without memory. I did purchase new memory (non-ECC this time) and tried starting up with that thinking that maybe the ECC memory (Samsung M474A4G43B1-CTDQ) had gone bad, but again, no success. The non-ECC memory I tested with is the same model memory I used originally when working on the system while I waited for the ECC memory to arrive. Unfortunately, I had already returned the non-ECC memory before this problem started because I thought everything was okay. So, not the same memory, but the same model (Crucial CT2K32G4SFD8266). Since there are two SODIMMs, I even tried booting with just one installed (alternating between them) as well as alternating between SODIMM0 and SODIMM1 (even though I know a single SODIMM should be installed in slot 0, not slot 1).

I keep wanting to think that this issue is just a configuration problem which can be corrected by resetting to factory default. I've already spoken to my VAR and they are overnighting me another NUC as a replacement, but I'm starting to think that maybe this might have more to do with a bug in the BIOS and less a failure of this particular unit. I'm concerned that through testing BIOS setting changes and trying to get the Quadro to work correctly, I may inadvertently end up bricking the second NUC. I'm hoping I'm wrong, though. I can't be 100% certain, but I believe when I was testing the PBM before with the Primary Display set to PEG, I had to connect a monitor to the Quadro just to see the PBM (meaning that the PBM did not override the Primary Display setting). But, I've slept since then and I won't stand by that statement definitively.

n_scott_pearson · ‎07-02-2020

First, some side comments:

There is no SODIMM0 vs SODIMM1 to worry about; there is only one SODIMM supported per channel so issues with signal reflection, etc. don't (edit: well, shouldn't) exist. I prefer the "upper" slot so that there is more air moving underneath the SODIMM to help with cooling.
When you do install two SODIMMs, the big difference is that interleaving is enabled to make the most of having two memory channels working. This can cause issues if the SODIMMs are not well matched. If there seems to be a problem, back off to a single SODIMM and see if this helps (as you did).

I don't remember you saying and this edit box methodology doesn't let me go back and read the entire conversation (do you get browser warning about leaving the site if you attempt to use the "View discussion in a popup" feature? I do). Have you tried clearing CMOS? Power everything down, unplug from wall, disconnect the CR2032 battery for 15 minutes and then try again.

...S

Hossy · ‎07-02-2020

I haven't tried the popup discussion, but I did just have to delete all my intel cookies to get back to this thread. I did try leaving the CR2032 battery disconnected (overnight even) and that didn't help.

MRoss5200 · ‎07-02-2020

Since you can't get to the PBM using the power button, try pulling the yellow recovery jumper next to the edge of the board on the NUC Element board. This will force the system to boot to the Recovery Menu. You should be able to select F2 to get into BIOS Setup. I tried this with a Quadro card installed and POST took a little longer to display on screen but it worked. You may want to try with out the Quadro card installed too. Make sure the other yellow jumper is NOT on both pins as this is a CMOS clear jumper. If you leave it on both pins the system will not boot properly.

I tried this with a Quadro card installed and was able to get to the PBM fine. Since the card I have has (4) DP ports, I used DP 1.

Hossy · ‎07-02-2020

I have tried booting with the yellow recovery jumper off (that's how I tried to do BIOS recovery -- using the security jumper method). Interestingly, I do not have a second yellow jumper. According to the tech document, the two pins aren't for CMOS -- they are for resetting the MEBX (page 39). The tech document does not list a method for resetting CMOS, which I think is the root of the problem here. Even with the security jumper off and all extra devices removed (no Quadro, no NVMEs, literally just memory and a wireless USB dongle for the KB), the PBM still won't show up.

MRoss5200 · ‎07-02-2020

You might want to try it just in case. I assume you are not using AMT on this Xeon NUC so resetting MEbx will not affect the operation(?) of the system. Of course backup any data you need to save first just in case. If this does not work then I would see about an exchange for a new one. This clearly has some issues.

mgt7485c · ‎07-06-2020

Hey I have this exact same problem. I didn’t do a good forum search because I was multitasking and calling Intel support.

I also messed around with the VRAM settings and after setting the memory and aperture to 1024, I could NEVER get it to POST no matter what.

Not sure if you got your issue resolved but Intel was going to replace my unit but they don’t have any in stock. They advised me to do a return with my retailer so now I have another unit coming in tomorrow.

Hossy · ‎07-15-2020

Unfortunately, no. I never got it resolved. My VAR sent me a new unit. This is obviously a major bug Intel needs to fix -- "how to buy a $1700 paperweight"

firedog1024 · ‎07-15-2020

I have the exact same issue changed the video aperture size and now the my NUC9 Extreme i9 is a $1700 doorstop. I've been going mad all afternoon trying to recover or reset the BIOS, how can there be no BIOS reset jumper on these things. Reading the thread looks like I have to basically call Intel support or return my unit to B&H who need 2 to 4 weeks to special order a new unit. All this after waiting 3 months from ordering this thing to get it. Really not sure if I can trust this computer now based on this, especially at the price point.

Hossy · ‎07-15-2020

Agreed. Even with the second unit, I have found that sometimes, it just chooses not to boot. Since I have AMT activated, I can easily tell when this happens because AMT stops pinging. I have (no joke) attached an energy meter so I can tell what power state it's in and an outdoor Insteon on/off controller to the server controlled by HomeSeer so I say "Alexa tell HomeSeer to Reboot the Server". Saves me the constant getting up and walking into the other room to mess with it.

I've had this happen, I believe, more so on reboots than with power ons, but power ons are no exception.

<rant>

The thing that bothers me the most is that Intel was supposed to ship this unit Mid-March. Ok, COVID happened. I get that. But in all that time, you didn't have your BIOS developers continuing to write fixes for code problems? What was your QA doing when they weren't tending to their families? I've been working my butt off even while hiding from everyone as best I can while moving an office and standing up two more. #ITslave

The stability of this machine has me questioning the money I spent (I'm $3k+ in right now and nearing the end of the return period). And with Intel's policy of only being able to email or phone with (being polite here) a LEVEL ONE support technician on a bleeding edge system they honestly know nothing about is beyond my ability to comprehend. I've been doing IT since I could stand on a chair and reach the keyboard. I can do (and unfortunately have done) this stuff in my sleep. What I need is to talk to someone who actually knows the system. I'm all for OJT and all that. Fine. This is why we invented conference calls and ZOOM (Intel has a policy prohibiting their techs from using Zoom, FYI -- a thousand words being the preferred method of communication?). I wouldn't even mind if the level two or level three tech took time to explain who what where when why to the level one guy while I was on the phone. I would appreciate the knowledge too! The way their support is structured can only be to avoid solving issues more complex than reboot your router. I honestly don't know how they expect to fix their systems.

</rant>

mgt7485c · ‎07-16-2020

Hossy,

I've read enough instances of people having the same issue with this NUC9 that you might still have a bad one. Luckily my second unit has been trouble-free and not exhibiting the boot issue you have occasionally.

Originally I ordered mine from simplynuc but cancelled and ordered the bare-bones version because I didn't want to wait. You might consider getting this from a VAR who can provide possibly better support--or just getting a Dell/HP unit. Intel's support is very hit or miss and probably not good for mission-critical.

My support experience has been good but support were not familiar enough with the NUC9 to help much. Even the top contributors in this forum (as helpful as they are) are giving sub-par advice on this particular issue (try resetting BIOS, etc). I've seen at least three people have the same issue with this NUC9 and they're all told to "you're doing it wrong". There IS an issue with this NUC9 but that information hasn't been properly distributed.

Hossy · ‎07-16-2020

Hi mgt7485c,

First off, thank you for a bit of validation. I don't feel as lonely now.

Before I say anything else, I'm very interested in where you found the BIOS reset jumper. Maybe I can resurrect my original unit before I send it back via RMA. As firedog1024 mentioned, the "typical" physical location of the BIOS reset jumper is actually the 2-pin MEBX Reset Jumper, which IIRC is completely separate from the BIOS. Also, the MEBX Reset Jumper "conveniently" has no yellow jumper -- both units I received have only one yellow jumper installed in the "normal" position on the 3-pin BIOS Security Jumper. Previous posters showed pictures of two yellow jumpers. Maybe they paid more than I did so they get the extra plastic.

I also bought my NUC 9 from Provantage and they were very accomodating in getting a replacement unit overnighted to me. I haven't even received RMA paperwork yet (guess that's still in the works) which speaks to their customer-first focus.