NUC11TNHv5 AMT stop responding in 36 - 48 hours after NUC powered on

[Wells]

Hi Everyone,

I recently got a NUC11TNHv5 for my home lab, mainly looking to the AMT feature for out-of-band management and headless KVM access.

With PVE installed, I found the AMT will stop responding to ping and remote access in around 36 - 48 hours. I searched the web but couldn't find a clue whether it's "normal" (given there is some article saying the AMT is not a true OOB, e.g., iLO or IPMI), or there is a trick I can do to make the AMT accessibility consistent.

Here is some detailed configuration:

- NUC Model: NUC11TNHv5

- BIOS: TNTGLV57.0064.2022.0217.1533

- ME FW: 15.0.35.1951

- IP address assignment: static

- OS: PVE 7.2-5

P.S. the OS runs perfectly fine even when the AMT stops response. A soft power toggle can't restore AMT. A physical power up/down will be required to correct.

NUC11TNHv5 AMT stop responding in 36 - 48 hours after NUC powered on 

[Wells]

See the trough green lines for the period AMT is accessible after a power cycle.

[Alberto_R_Intel]

Wells, Thank you for posting in the Intel® Communities Support.

In order for us to provide the most accurate assistance on this matter or point you in the right direction for support on this scenario, we just wanted to confirm a few details about your system:

When did you purchase the Intel® NUC? (For warranty purposes)

Did the AMT feature ever work consistently on this specific Intel® NUC before?

If yes, when did the issue start?

Did you make any recent hardware/software changes that might cause this issue?

Does the problem happen at home or in the work environment?

Please attach the SSU report so we can verify further details about the components in your platform, check all the options in the report including the one that says "3rd party software logs":

https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows-?product=91600

Any questions, please let me know.

Regards,

Albert R.

Intel Customer Support Technician

[Wells]

Hi Albert, thanks for your reply, please see below for the answers:

When did you purchase the Intel® NUC? (For warranty purposes)

- I purchased it last month, June 2022

Did the AMT feature ever work consistently on this specific Intel® NUC before?

- Not really, I noticed the AMT stoping after certain period of time after my initial setup. And that's why I add a probe to smokeping to track the service availability

If yes, when did the issue start?

- N/A

Did you make any recent hardware/software changes that might cause this issue?

- No. I have the RAM and SSD installed from day 1

Does the problem happen at home or in the work environment?

- Home environment, I use this unit for my home lab

Please attach the SSU report so we can verify further details about the components in your platform, check all the options in the report including the one that says "3rd party software logs":

- Please see attachment. I ran the script on PVE, hope the information would be good enough for diagnosis.

[Alberto_R_Intel]

Hi Wells, You are very welcome, thank you very much for providing that information and the SSU report.

Accroding to the information in the SSU report, the BIOS version currently installed on your Intel® NUC is 0046, which is the latest one. It is important to mention though that this version has been removed from our website, the latest one is now 0062:

https://www.intel.com/content/www/us/en/download/19699/bios-update-tntglv57.html?wapkw=nuc11tnhv5 

But before doing any changes, we will do additional research on this matter to further assist you with this topic, as soon as I get any updates I will post all the details on this thread.

Regards,

Albert R.

Intel Customer Support Technician

[SergioS_Intel]

Hello Wells,

Thank you for waiting for our updates.

Please help us by providing the following information:

1. How are you positioning the unit, what tools are you using?

2. In what power state is the system when you lose the connection?

3. Is the unit in admin control mode or client control mode?

4. Are you using a certificate? If so can you please send us the details and pictures?

5. What is the error are you getting? Please send us picture if possible.

6. Is this the only unit having issues? Do you have other working units? if so, what are the differences in terms of settings?

We will be looking forward to your updates.

Best regards,

Sergio S.

Intel Customer Support Technician

[Wells]

Hi Sergio,

Please see my setup inline:

1. How are you positioning the unit, what tools are you using?

- I put the unit properly on a shelf.

2. In what power state is the system when you lose the connection?

- The power adapter is all the way connected to an APC BK650M2 UPS power.

3. Is the unit in admin control mode or client control mode?

- It's in admin control mode 

4. Are you using a certificate? If so can you please send us the details and pictures?

- System default certificate is being used.

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 268441 (0x41899)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Untrusted Root Certificate
Validity
Not Before: Dec 31 16:00:00 2017 GMT
Not After : Dec 30 16:00:00 2049 GMT
Subject: CN=NUC-ME, C=None, ST=None, O=None
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c5:da:8e:8e:45:0d:23:b2:11:1b:81:21:b8:9f:
9a:01:ec:74:85:37:df:c0:a2:58:5f:6d:37:3c:02:
1c:40:a1:63:84:9b:c7:ea:dd:4b:8b:e7:b4:f6:bf:
09:60:b3:e9:9a:af:6d:73:fa:c2:c8:bb:ed:22:a6:
14:0a:aa:93:b8:5b:a1:61:f6:ec:cd:a3:39:19:87:
8e:ba:96:44:00:73:0f:e0:55:25:1c:63:87:5f:1d:
36:08:d5:12:51:f1:3b:4b:4c:ee:e2:c5:ab:ae:72:
0b:e6:e3:27:9b:f0:e2:55:4d:bd:7c:b2:24:d0:99:
0e:b9:c0:1e:82:5a:be:39:3e:84:21:25:d5:7e:fd:
66:be:5d:86:2e:86:01:92:76:2a:e3:c7:c7:97:94:
14:71:c6:89:0d:3d:99:9c:ea:e6:85:62:21:e1:3e:
42:a7:cc:81:5a:06:bd:39:37:93:5e:c4:51:71:4f:
0b:21:bc:25:97:b7:f3:3c:f3:e5:c1:55:a3:b8:f6:
76:a1:ad:b6:1f:49:03:eb:48:88:36:48:5f:e5:80:
b0:07:95:16:ad:18:8e:39:07:d8:5a:ff:90:10:80:
33:47:3a:ef:0a:e5:e5:13:e9:a0:a8:83:e1:0b:88:
9c:c5:be:83:3f:59:aa:47:4a:8b:83:98:85:0e:9f:
b5:9f
Exponent: 65537 (0x10001)

5. What is the error are you getting? Please send us picture if possible.

- There is no error code since the AMT function doesn't accept any kind of connection, icmp or tcp at the moment it stops working. Yet to remind the system OS, in my case, the PVE works perfectly well.

6. Is this the only unit having issues? Do you have other working units? if so, what are the differences in terms of settings?

- This is my only unit. I couldn't say for the others

[GenuineJakob]

Dear Intel support,

I am facing similar issues.

NUC11TNKv5

TNTGLV57.0062.2021.1203.1050

1. How are you positioning the unit, what tools are you using?

Provisioned through a USB key, MeshCentral.

2. In what power state is the system when you lose the connection?

Not entirely sure, the device was probably in Modern Standby.

No UPS, connected to a wall outlet.

3. Is the unit in admin control mode or client control mode?

ACM

4. Are you using a certificate? If so can you please send us the details and pictures?

Secured using TLS, certificate attached (.cer isn't supported, you will have to unzip it.).

5. What is the error are you getting? Please send us picture if possible.

The device isn't reachable anymore. It doesn't report back via CIRA. I am unable to access the AMT web interface.

The OS, in my case Windows, is working perfectly fine, everything network related from Windows has no issues.

6. Is this the only unit having issues? Do you have other working units? if so, what are the differences in terms of settings?

I have setup a unit at home which I can easily power cycle if unreachable. It should become unreachable in about 32 hours.

[GenuineJakob]

Here is an update from my devices:

NUC11TNKv5-2 with TNTGLV57.0064.2022.0217.1533, which I setup for testing, AMT is working fine since 5 days.

NUC11TNKv5-1 with TNTGLV57.0062.2021.1203.1050, this is the machine that went unreachable. I visited the remote NUC in person and it probably has a corrupted BIOS. Conflicting power BIOS settings were selected and options were checked, that I definitely not enabled. This machine also blue screened with DRIVER_POWER_STATE_FAILURE. After unplugging the NUC and loading default BIOS settings, it is working fine for now.

[n_scott_pearson]

The 120W power adaptors shipped with these NUCs contain UL-required circuitry that can have issues operating with UPS units that generate simulated AC sine wave output while on battery. A UPS that generates a pure sine wave is recommended.

You indicated that the your UPS is an APC BK650M2. This model does NOT generate a pure sine wave. It is thus possible that, should power be interrupted, the operation of the NUC could be interrupted as well. I suggest that, as an experiment, you run without this UPS connected (or, if you have access to one, run with a UPS that generates a pure sine wave) and see if the issue still reproduces itself.

Hope this helps,

...S

[Wells]

Hi Scott,

Thank you for the insight, I have moved the power plug from UPS to a wall socket, will see what's happening in the next 2 days.

Yet I am just curious why it seems that the sine wave cause would potentially impact AMT only. I assume the adapter powers the whole unit. Yet, only the south bridge (assuming AMT is a sideband communication in between CPU and NIC) was affected, but the north bridge was immune (considering the OS doesn't report any error).

[n_scott_pearson]

It could be that this has absolutely nothing to do with the issue. It is a simple test, however, and you will know shortly (if not already) whether it makes a difference. If it does, well, pure sine wave UPS units are not much more expensive than the existing stepped sine wave UPS unit.

...S

[Wells]

An update on the issue, with more testing, here is my observation:

1) As long as the OS is not running on NUC, AMT will stay accessible

As you can see from the attachment, the unit was stable for more than three days, from Jul.23 to Jul.26. Meanwhile, the probing response time was steady.

During this period, the host OS, in my case the PVE was not running, and I switched the power source from UPS to the wall outlet as Scott suggested.

2) AMT goes down "as expected" after the host OS ran for a while

Once the OS is turned on, you can see the AMT response time starting to fluctuate as the grey peak and trough. And then it turned inaccessible after 30+ hours.

Here is my assumption of the issue:
1. The AMT will run into an issue with the host OS turned on
2. The issue is irrelevant to simulated AC sine wave UPS

[Wells]

Just out of curiosity, the issue shall be very easy to replicate. I wonder why it's not reported by QA given the AMT should be a key selling point of "v" models ...

[Wells]

I did another test, fresh install Windows 11 pro workstation, with all Intel drivers.

AMT stop working in 6 hours.

[Wells]

It's now getting even more amazing.

I download the 062 BIOS from the website, use F7 method to perform the upgrade, and the upgrade goes well. While after the upgrade/downgrade, the NUC now bricked, but there is no video output on both HDMI ports.

I then tried the BIOS recovery procedure, which seemed to make the box into a not-so-dead state.

I can access the AMT with MeshCommander, yet the system complains, "No system memory is physically installed in the system", but I am pretty sure the DIMMs are properly installed. Meanwhile, there is no display on both the physical device and AMT KVM, and the OS can't boot.

Any idea about the situation?

 

[GenuineJakob]

My NUC also bricked after downgrading the BIOS from 64 to 62. I opened a ticket with Intel to get a replacement.

[SergioS_Intel]

Hello Wells,

We appreciate the additional information please allow us to check it and we will get back to you.

Best regards,

Sergio S.

Intel Customer Support Technician

[KalleDK]

Dear Intel support,

I am facing similar issues.

NUC11TNHv7

TNTGLV57.0064.2022.0217.1533

1. How are you positioning the unit, what tools are you using?

On a table in a corner of an empty room, MeshCentral.

2. In what power state is the system when you lose the connection?

Not entirely sure, but the OS was running

No UPS, connected to a wall outlet.

3. Is the unit in admin control mode or client control mode?

ACM

4. Are you using a certificate? If so can you please send us the details and pictures?

Secured using TLS but also allowing non secured connections - both fail.

5. What is the error are you getting? Please send us picture if possible.

The device isn't reachable anymore. I am unable to access the AMT web interface. On both Wifi and LAN

The OS, in my case ProxMox, is working perfectly fine, everything network related from ProxMox has no issues.

I am using VLANS on the lan, but the default/untagged vlan is the one AMT is listening on.

I couldn't remember the ipv6 adress it also has, but I can test if that also stops working.

Also a downgrade of the BIOS would be problematic, as the Remote Desktop didn't work well when no display was attached (whis is my situation), and the BIOS update fixed this problem

6. Is this the only unit having issues? Do you have other working units? if so, what are the differences in terms of settings?

I have setup a unit at home which I can easily power cycle if unreachable. It should become unreachable in about 32 hours.

[rolandu]

Dear Intel support,

I am facing similar issues.

2 units NUC11TNKv7

1x TNTGLV57.0067.2022.0718.1757 (updated yesterday)

1x TNTGLV57.0054.2021.0316.1434 (as it was shipped)

1. How are you positioning the unit, what tools are you using?

Standing in a 19" rack

2. In what power state is the system when you lose the connection?

Powered on. No UPS, connected to a wall outlet.

3. Is the unit in admin control mode or client control mode?

ACM

4. Are you using a certificate? If so can you please send us the details and pictures?

Secured using TLS but also allowing non secured connections - both fail.

5. What is the error are you getting? Please send us picture if possible.

The device isn't reachable anymore per AMT after some hours. I am unable to access the AMT web interface. Also no connection for MeshCommander possible.

The OS, in my case ESXi, is working.

6. Is this the only unit having issues? Do you have other working units? if so, what are the differences in terms of settings?

Both units have the same issue.

In my opinion it might have something to do with the network-card, as I'm also facing another problem:

I have a Windows DHCP Server running inside ESXi (VM) with the latest community-driver (Community Networking Driver for ESXi | VMware Flings) and I have problems with broadcast-messages that not always getting "out of the NUC". Currently I'm digging deeper into this, but it seems that DHCP-Offers (broadcast) can be seen in the VM with WireShark, and also on ESXi-level with its network-trace-tools. But the DHCP-Offers do not always pop up on my network switch... Weird - it seems they are not sent out by the network-card.

KR, Roland

[MiKok]

Hi Everyone,

I can confirm the same behavior of two  BNUC11TNKV50Z00 with TNTGLV57.0067.

The device isn't reachable anymore per AMT after some hours. I am unable to access the AMT web interface. Also no connection for MeshCommander possible.

Moved to fan-less Akasa case, running Win10, EATON UPS Ellipse PRO 650, ACM, no TLS.

The OS, in my case Win10, is working.

A Windows restart does not help, a hard reset can fix it.

Both units have the same issue.

Have nice days

Michael