Nuc does not recover after a restart (Intel® NUC Kit NUC7i7DNHE)

[Singularity9942]

I've had some issues with this Nuc previously but for the last month it's been fine. Well last night automatic updates ran and the system initiated a reboot. Well afterwards it never came back online. The Nuc had power (the power indicator was on) but I couldn't ping it or SSH into it. Additionally hooking up a monitor / mouse / keyboard to it didn't allow me to access it.

I'm at my wits end with this device. I don't think there's a solution other than just getting a new one but open to suggestions.

I'm running Ubuntu Server 20.04 LTS on Intel® NUC Kit NUC7i7DNHE. Here's the journalctl log from when it initiated the shutdown.

journalctl -b -1 | tail -20

May 11 02:00:25 scadrial systemd[1]: multipathd.service: Succeeded.
May 11 02:00:25 scadrial systemd[1]: Stopped Device-Mapper Multipath Device Controller.
May 11 02:00:25 scadrial systemd[1]: swap.img.swap: Succeeded.
May 11 02:00:25 scadrial systemd[1]: Deactivated swap /swap.img.
May 11 02:00:25 scadrial systemd[1]: swapfile.swap: Succeeded.
May 11 02:00:25 scadrial systemd[1]: Deactivated swap /swapfile.
May 11 02:00:25 scadrial systemd[1]: Reached target Unmount All Filesystems.
May 11 02:00:25 scadrial systemd[1]: systemd-remount-fs.service: Succeeded.
May 11 02:00:25 scadrial systemd[1]: Stopped Remount Root and Kernel File Systems.
May 11 02:00:25 scadrial systemd[1]: lvm2-monitor.service: Succeeded.
May 11 02:00:25 scadrial systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
May 11 02:00:25 scadrial systemd[1]: Reached target Shutdown.
May 11 02:00:25 scadrial systemd[1]: Reached target Final Step.
May 11 02:00:25 scadrial systemd[1]: systemd-reboot.service: Succeeded.
May 11 02:00:25 scadrial systemd[1]: Finished Reboot.
May 11 02:00:25 scadrial systemd[1]: Reached target Reboot.
May 11 02:00:25 scadrial systemd[1]: Shutting down.
May 11 02:00:25 scadrial systemd-shutdown[1]: Syncing filesystems and block devices.
May 11 02:00:25 scadrial systemd-shutdown[1]: Sending SIGTERM to remaining processes...
May 11 02:00:25 scadrial systemd-journald[432]: Journal stopped

I ran this and got the results at 10:30am this morning after having to hard power cycle the device. 2am is when my monitoring tool reported issues so it initiated a restart at 2am and then never came back online.

[Jean_Intel]

Hello Singularity9942,

Thank you for posting on the Intel️® communities.  

To have a better understanding of your issue, please provide me with the following:

• Can you recall what the automatic update was initiated after the reboot?
• You mentioned that the power indicator is on. I want to confirm the behavior of the power button. Is there any kind of LED pattern?
• Did your system suffer from overheating symptoms: blue screen of death, flickering, or random shut down?
• Have you tried any troubleshooting steps to solve the issue?

Best regards, 

Jean O.  

Intel Customer Support Technician

[Singularity9942]

Thanks for the reply, Jean.

• Can you recall what the automatic update was initiated after the reboot?
  • These were automatic OS updates that Ubuntu automatically installed and initiated a reboot for. From the logs the reboot process initiated properly but the computer never came back online. My assumption is that it was a problem where the device never POSTed. I've had this problem in the past where I go to restart / power cycle the machine and it never POSTs and I have to power cycle it multiple times for it to finally post.
• You mentioned that the power indicator is on. I want to confirm the behavior of the power button. Is there any kind of LED pattern?
  • The LED was solid white however when I pressed the button it turned off instantly. Usually when the machine is fully booted into an OS, you have to press and hold the power button before it'll finally power off. The fact that it turned off instantly makes me feel even more confident in my assumption that the device simply never posted. I couldn't ping it, ssh into it, nor when I hooked up a monitor / keyboard/mouse could I get it to respond in anyway.
• Did your system suffer from overheating symptoms: blue screen of death, flickering, or random shut down?
  • I've had issues with this Nuc in the past. No overheating issues as far as I know but I've had issues with it not posting upon restarting before. Additionally I posted another thread on this forum where the device would be come unresponsive if I unplugged/replugged an HDMI cable once it was already booted into Ubuntu Server 20.04 LTS.
• Have you tried any troubleshooting steps to solve the issue?
  • I've tried updating, downgrading, and updating the BIOS again.
  • I've tried upgrading the firmware on both HDMI ports.
  • I've tried messing around with various BIOS settings.
  • I've tried switching to a different m.2
  • I've run Memtest86 on the RAM to confirm no errors. 
  • I've tried running a different kernel in Ubuntu
  • I've tried running Windows on it although I did not test with Windows long enough to replicate the issues. I will try to do that a bit this weekend and see if restarting in Windows produces the same issue with it not posting properly.

Is it possible there's maybe an issue with the power brick? Replacing that would be a lot easier than replacing the Nuc itself. I currently have an M.2 SATA 2TB drive installed.

Thanks again for the help. @Jean_Intel 

[Jean_Intel]

Hello Singularity9942,

Thanks for your response. The information provided has been useful.

To confirm if the power brick is having issues you can try:

• Use a different power socket
• Test a different power adapter on the NUC

You mentioned that "I've tried messing around with various BIOS settings." Are you able to reach the BIOS settings?

Also, you mentioned that you tried running windows, is the Intel® NUC able to reach the Operating system when you installed Windows?

Best regards, 

Jean O.  

Intel Customer Support Technician

[Singularity9942]

Thanks, Jean. I will see if I can find a different power brick if you think that's a possible culprit.

I am able to access the BIOS and I was able to successfully access Windows when I had it installed.  The issue just seems to be that sporadically the machine will not POST upon restarting.

[LeonWaksman]

When your machine is stack during POST, disconnect any external devices (except mouse and keyboard) connected to the USB ports and then restart the computer. If the computer boots, try to troubleshoot which device causes this problem.

Leon

[Singularity9942]

@LeonWaksman I just spent about 2 hours troubleshooting from that idea and here's my findings.

In all of the below scenarios I always have ethernet + power cable plugged in at a minimum. To restart the Nuc I run

sudo shutdown -r now

1. Nothing plugged in = No POST

2. Monitor Plugged in OR HDMI Dummy plug plugged in = Reliably POST every single time upon initiating a restart

3. Monitor/HDMI Dummy + USB security key = sporadically POST but mostly No POST

4. Monitor/HDMI Dummy  + USB bluetooth mouse/keyboard dongle plugged in = No POST

5. Just USB Bluetooth mouse/keyboard dongle plugged in = No POST

In all of the above situations, any of the 4 USB ports made no difference. In the scenarios where it was reliably posting I restarted 4-5 times to make sure it wasn't a fluke.

There were some situations where a configuration that should be reliably POSTing was not as a result of it previously being in a bad configuration. To address I would hold down the power button until it showed red LED. This would pop up on the monitor a special menu. I would then turn it off via the power button wait a few seconds and turn it back on again and then it would behave as normal (eg reliably posting upon intiating a restarting from Ubuntu)

Any time it wasn't going to post I knew immediately because my PING command went from saying

"Request Timed Out"

to saying

"Reply from 192.168.50.9: Destination Host Unreachable"

I'm unclear why that's the IP it spits back in those scenarios because that's not the Nuc's IP.  It's the IP of the computer that I'm testing from.

In the BIOS I have the USB ports configured to continue to provide power even when the machine is off. I haven't begun testing any of this with any BIOS settings tweaked. Wanted to give you and update before I went down that rabbit hole.

I am able to plug in those USB devices after the fact and they are fine although if I plug/unplug the HDMI device too much the machine will become unresponsive (destination host unreachable)

Thank you in advance.

EDIT: I took a break and began testing again and am having success with scenario 3 so I will keep that configuration for the time being while I wait on feedback / the new ac adapter.

[LeonWaksman]

Hi @Singularity9942 

1. Your NUC should POST and boot even in your scenario 1 (i.e. nothing connected except power cord ).  I'm working in this mode with my NUC7i7DNHE. I'm accessing this NUC from other computer, using Remote Desktop function (both computers running Windows 11 Pro).

- I have the latest BIOS (0075) installed 

- Secure Boot enabled (Standard Mode)

- NUC7i7DNHE connected to network trough WiFi (no ethernet cable)

- Boot UEFI enabled, Legacy boot disabled

2. I suggest that you verify that you have latest BIOS installed (you may want to update your BIOS using BIOS Recovery by Secure Jumper method Before starting BIOS Recovery, disable drive encryption if the drive is encrypted)

    2.1 Start with reset BIOS to default setting (F9, F10)

    2.2 If you suspect that some USB device is hanging the POST, set its USB port in BIOS Advanced > Devices > USB > USB Ports to Not Detect (instead of Enable)

Leon

[Singularity9942]

@LeonWaksman 

I did the BIOS recovery using the jumper method as instructed. It completed successfully. I then restored my BIOS to default settings. I then did a full shutdown after booting into Ubuntu.

I kept  everything except power, ethernet unplugged and got no POST. I waited a few minutes just to make sure. I say I got no POST meaning that my ping to the machine never resolved.

I shutdown using the power button (and as usual it powered off immediately indicating to me that it hadn't POSTed) and plugged in my HDMI dummy plug and my USB Security Key and then powered the Nuc on. After about 45 seconds I was able to SSH in (and my pings returned successfully)

I forgot to mention when it's failing to post, the machine will powercycle itself every 1m30s.

I will try my new power cable when it arrives otherwise I will continue to use the HDMI dummy plug. I'm out of ideas on what else to try. 

I appreciate all of your help nonetheless.

[LeonWaksman]

Hi @Singularity9942 

We are talking about two different things. For me POST (Power-On Self-Test), is a sequence of tests that BIOS performs after Power ON in order to determine if the computer devices (i.e. RAM, disk drives, keyboard, USB and other hardware) are working correctly. 

Only after finishing the POST, the BIOS starts to boot the Operating System. Only after the Operating System is running, device like ethernet or WiFi may be used in order to ping your computer.

Usually, the NUC7i7DNHE, is ready with Windows running after about 25 seconds. I believe that booting Linux doesn't take longer. If your your machine is recycling after about 1:30 minutes, it must be because the O.S. is stuck.  So, check your installation. 

If you connect monitor, do you have any video output (like Intel NUC logo)? 

After "unsuccessful POST" what do you see?

I suggest, check the O.S. installation.

Leon

[Singularity9942]

@LeonWaksman 

When I say it fails to POST I mean that the screen shows nothing upon starting no matter how long I wait. Literally nothing, just black. When I say it successfully POSTs I mean that I see the Intel NUC splash screen (the one where it tells me I can press keys to enter the BIOS, or start the boot manager etc) and then it immediately boots into Ubuntu.

There is never a situation where I see the Intel NUC splash screen but then operating system doesn't start. It's all or nothing.

The issue I'm having occurs before the operating system even comes into play. I press the power button and I see nothing. If I press the power button to turn it off it turns off immediately without hesitation which does not happen if the computer has booted into Ubuntu. In that situation I have to hold it down for 3-4 seconds before it will power off.

In situations where I see nothing (fails to POST in my lingo) the machine will usually powercycle itself every 1m30s although not always. Sometimes It will just stay powered on with nothing displaying on the monitor until I turn if off.

I am happy to record some videos of the issue if that would help.

[itsme]

This 'hanging' issue in post/boot  occurs only if you reboot the system?

A cold start works w/o  monitor/HDMI dummy  connected?  

Does the system boot into graphical  or just multi-user? 

If graphical, does it restart successfully with boot option 'nomodeset', or if it's configured to boot into multi-user instead of graphical?

[Singularity9942]

@itsme it's an issue from cold start as well.

"Does the system boot into graphical  or just multi-user? 

If graphical, does it restart successfully with boot option 'nomodeset', or if it's configured to boot into multi-user instead of graphical?"

I'm not sure I follow. I'm not clear on what multi-user is. When it fails to POST I see nothing on the screen. When it's successful I see the Intel Nuc splashscreen image and then it boots into Ubuntu Server which is pure CLI

I will reinstall my Windows m.2 and run the same tests to confirm it's not an Ubuntu Server issue.

[itsme]

---

@itsme it's an issue from cold start as well.

then it's quite unlikely that it's an issue with the OS

I'm not sure I follow. I'm not clear on what multi-user is. When it fails to POST I see nothing on the screen. When it's successful I see the Intel Nuc splashscreen image and then it boots into Ubuntu Server which is pure CLI

systemctl get-default 

multi-user.target   boots into multi-user, network etc  but NO graphics

graphical.target    starts also X11/Wayland on local GPU ==  multi-user +  GUI

On a server I would expect  default target to be multi-user, but you never know

[Singularity9942]

@itsme @LeonWaksman As much as I didn't think it could be I'm starting to think it is an Ubuntu Server problem (or it's an issue with my m.2 drive)

I installed Windows 10 Pro on another SATA m.2. Updated windows, installed and updated all drivers. I had a hiccup here or there but eventually I was able to consistently restart it and get it to show up via ping and remote desktop no matter what was plugged in.

So I swapped back to my Ubuntu Server drive with nothing except power and ethernet plugged in. it booted fine, I restarted it, booted fine. So I shut it down (sudo shutdown now) and plugged in my usb security key and then it never became pingable. Every 1m30s it would power cycle. I waited and it did it about 4 times before I finally turned it off.

One thing interesting is that as soon as I plugged in my HDMI dummy plug (with the machine on) it immediately starting registering a ping. 

When I run "systemctl get-default" it shows "graphical.target" no matter whether or not there's anything plugged into the HDMI port.

So I really think it may be an issue with the HDMI ports + Ubuntu Server?

So yeah I guess it is an ubuntu server problem. I've read elsewhere that Ubuntu Server on Nucs behaves better if you legacy boot but I do not have the option to legacy boot on this device. If I disable UEFI it won't detect the OS on my m.2 (even if it was installed in Legacy Mode which I tested).

I've reinstalled Ubuntu Server previously though and it didn't solve the issue. So IDK. I've put so much time into troubleshooting this nightmare I'm thinking that I will just continue to use the hdmi dummy plug and if that doesn't solve it I'll just buy a new Nuc. That is assuming the new power adapter doesn't fix my issue. It arrives tomorrow but I doubt it's goiing to help.

[itsme]

---

@Singularity9942  

 So I shut it down (sudo shutdown now) and plugged in my usb security key and then it never became pingable.

I guess it's the usb security key, whatever this might be.  

As @LeonWaksman suggested, you could try to disable the usb ports,  or disable  booting from USB devices in case it's detected as some kind of  block device.

---

it shows "graphical.target" no matter whether or not there's anything plugged into the HDMI port.

So I really think it may be an issue with the HDMI ports + Ubuntu Server?

No,  the default target is configured with systemctl set-default  <target>  

It's like runlevels  3,4 or 5  with SystemV init.  

does the system boot  Windows WITH the usb security key connected?

[LeonWaksman]

Hi @Singularity9942 

In this case set the USB port with your usb security key to "Not Detect" See para.2.2 in my post above at 11:08 AM. 

But I have to admit, I'm confused already. In your post above your wrote "...I kept  everything except power, ethernet unplugged and got no POST...." However, in your last post you reported: "So I swapped back to my Ubuntu Server drive with nothing except power and ethernet plugged in. it booted fine, I restarted it, booted fine."

FYI, with M.2 drive removed (only RAM installed), the NUC should boot into BIOS (after pressing on F2, when you see the INTEL NUC logo).

Leon

[Singularity9942 · Accepted solution]

@LeonWaksman 

I got a new ac adapter  and installed it. I also set the USB port for my security key to "do not detect". Only other change that's been made in the BIOS is to have USB ports stay powered when the device is off. I also swapped my RAM stick into the other slot just for the heck of it.

Here is where I stand when the only things plugged in are the USB Security key into the "do not detect" usb port, power, ethernet.

Sometimes when I restart or shutdown and cold start it starts up just fine and my ping returns / I can SSH in almost immediately. within 1-2 minutes.

Sometimes it takes 10-15 minutes after starting/restarting before my ping will finally return / I can SSH. I mean literally 10-15 minutes of "Destination Host Unreachable" and then all of a sudden, pings return. The few times this has happened (because I was patient enough to wait) was when a monitor wasn't plugged in but I will test the same with a monitor plugged in (UPDATE: Results below.)

Before installing the new power cable and modifying the USB setting in the bios, I ran another Memtest86 for 8 hours with nothing plugged in except the USB drive for Memtest (not the security key) my mouse/keyboard dongle and my monitor. It returned no errors when it finished 4 passes. I hit the power button to turn it off then unplugged the USB key. Then I hit the power button to turn it back on and it turned on with nothing on the screen, after 1m30 it power cycled itself and then booted up as normal.

I have no idea why sometimes it is taking 10-15 minutes to finally boot. I'm unclear if the new power brick or USB setting made a difference.

If it will always consistently EVENTUALLY boot up normally then that's all I need. I don't really care if it takes 10-15 minutes. The only thing I can't have is if I have to manually intervene.

I will continue to test.

EDIT: Did some more testing. With only USB security key, power, and ethernet plugged in, I initiated a restart. After 3-4 minutes of no ping I tried plugging in a monitor (without turning it off) and then waited 20-30 minutes and it never became pingable. The screen is currently showing blue. I will try again with the monitor plugged in from the get go.

With Monitor plugged in I initiated a restart. I never saw the post screen, after 1m30 of blue screen I see the very end of the Ubuntu Server Boot process, successful.

Restarted again, I see the NUC splash screen and it boots boots normally (immediately, less than 20 sec).

Restarted again,  immediate boot again.

Restarted again, 5m16s later it lands on the Ubuntu server pre-boot screen where it will start booting ubuntu after 15 seconds of no selection made. no other time has it landed on this screen.

Restarted again,  immediate boot

Restarted again, immediate boot

Restarted again, immediate boot

Restarted again, 1m30s later I see the Nuc splash screen and it takes me to the Ubuntu Server pre-boot screen, then successfully booted.

Restarted again,  immediate boot

Restarted again,  immediate boot

So I feel pretty good about saying that if a monitor is installed, it's going to consistently restart (i did not test cold boot) for me with that one 5m15s time being an anomaly. I will test next with the dummy plug. Hopefully it has the same outcome since I don't really want to have to keep a monitor/tv connected.

Shutdown, unplugged monitor and plugged in HDMI dummy plug and turned on. 1m20s later it power cycled itself. Upon powering back on it took 45s to ping. 

Restarted, immediate boot (30s to ping, anything quicker than 1m is going to be considered immediate boot)

Restarted, immediate boot

Restarted, immediate boot

Restarted, immediate boot

Restarted, immediate boot.

Shutdown, powered back on. 1m15s later it power cycled. Then 1m later it pinged.

I guess I'm going to consider this resolved. With a monitor or HDMI dummy plug plugged in + the new AC adapter + the USB port bios change I haven't been able to get it to hang without resolving. Will do some more testing maybe but I'm glad it's behaving right now.

[Jean_Intel]

Hello Singularity9942,

I hope you are doing fine.

I have not heard back from you, and you mentioned that will consider your issue solved. So we will close this thread. If you need any additional information, submit a new question, as this thread will no longer be monitored.

Best regards.

Jean O. 

Intel Customer Support Technician.