Issue with adopting clients to EMA with diferent ME-Firmware versions

[LucasOIZ]

Hey,

We are trying to adopt our Clients (~9'000) from Intel SCS to Intel EMA using the by Intel provided adopting PowerShell script.

We can adopt clients running AMT / ME-Firmware version 11.8.90.3987 or 12.0.70.1652 fine without issues.

On the other side we can not adopt clients Running AMT / ME-Firmware version 11.8.93.4323 or 12.0.92.2145, the client just keeps saying "Pending Configuration"

Since in a month all our clients get their BIOS updated, their AMT / ME-Firmware will also be updated to the ones not working.

About our Infrastructure:

• EMA-Server: 1.9.1.0 running on Server 2019 / Virtual-Server OnPrem
• EMA-Agent : 1.9.04
• AMT Configuration: ACM, CIRA / FQDN Source: Primary DNS / IP Address: From the DHCP Server
• Clients to adopt: ~9'000
• Using own PKI-Certificates since we run on a ".loc" domain (Cert gets loaded into the system by Thumb drive once)
• Clients are connected by Ethernet-Wire without adapters an such.

Clients working flawless to adopt:

• HP ProBook 650 G4 with Firmware 11.8.90.3987
• HP ProBook 650 G5 with Firmware 12.0.70.1652
• HP ProBook 650 G8 with Firmware 15.0.42.2235

Clients not able to adopt:

• HP ProBook 650 G4 with Firmware 11.8.93.4323
• HP ProBook 650 G5 with Firmware 12.0.92.2145

I know AMT Firmware 11.X is not officially supported, but after a talk with Dariusz (Mr. vPro) we tried it, and it worked until firmware update.

The log entries from EMA-Server look like this, for a client not able to adopt:

2023-06-01 14:23:14.2762|INFO||5724|59|StartRouter - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Message:Starting Mesh Router 52778 -> ***:16992, SYSTEM 

2023-06-01 14:23:14.2762|INFO||5724|59|PerformAction - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Message:Attempting Non-TLS Mesh phase 2 connection : (HOSTNAME1234,***). 

2023-06-01 14:23:14.2762|INFO||5724|59|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Message:AMT Profile detected : (HOSTNAME1234,***). 

2023-06-01 14:23:28.0188|WARN||5724|59|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Warning:Unable to connect to Intel AMT computer for round 2, 127.0.0.1:52778 

2023-06-01 14:23:28.0188|WARN||5724|59|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Warning:(Host=127.0.0.1, Computer=HOSTNAME1234, Domain=, Tls=False, Endpoint=(HOSTNAME1234,***), User=SYSTEM, UserId=00000000-0000-0000-0000-000000000000) 

2023-06-01 14:23:28.0188|INFO||5724|59|StartRouter - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Message:Starting Mesh Router 52793 -> ***:16993, SYSTEM

2023-06-01 14:23:28.0344|INFO||5724|59|PerformAction - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Message:Attempting TLS Mesh phase 2 connection : (HOSTNAME1234,***). 

2023-06-01 14:23:28.0344|INFO||5724|59|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Message:AMT Profile detected : (HOSTNAME1234,***). 

2023-06-01 14:23:47.9230|WARN||5724|59|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Warning:Unable to connect to Intel AMT computer for round 2, 127.0.0.1:52793 

2023-06-01 14:23:47.9230|WARN||5724|59|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.9.1.0, Culture=neutral, PublicKeyToken=*** - [1] - Warning:(Host=127.0.0.1, Computer=HOSTNAME1234, Domain=, Tls=True, Endpoint=(HOSTNAME1234,***), User=SYSTEM, UserId=00000000-0000-0000-0000-000000000000) 

From what I can understand it tries to connect to 127.0.0.1 although when testing on the EMA server directly it can resolve the correct IP of the system just fine.

Provisioning clients "out of the box" with those two "problem" firmwares works fine, it's just the migration not working.

Thanks in Advance,
Lucas

[MIGUEL_C_Intel]

Hello, LucasOIZ,

Thank you for using our software products. 

Please share the EMA server log, it will tell us more about the connection issue with the latest BIOS/firmware version.

EMA logs from the Server

[System drive]\Program File(x86)\Intel\Platform Manager\EmaLogs

Intel® EMA requires at least AMT version 11.8.79; Intel improved the security of the connections, and machines with only TLS1.0 or TLS1.1 are not supported anymore.  In addition, the Certificate chain needs encryption equal to or better than SHA2.

Please install and run the tool called EMA Configuration Tool on 1 or 2 non-working machines.  It will give us the endpoint AMT specifications. 

https://www.intel.com/content/www/us/en/download/19805/30485/intel-endpoint-management-assistant-configuration-tool-intel-ema-configuration-tool.html

Installation:

Double-click the .msi file and follow the prompts.

Run:

a-Open a command prompt (alternatively, you can run the tool from within Windows PowerShell*) as administrator.

b-Navigate to the installation folder (default C:\Program Files (x86)\Intel\EMAConfigTool).

c-Run the command: EMAConfigTool.exe –verbose

I look forward to hearing from you.

Regards,

Miguel C.

Intel Customer Support Technician

[LucasOIZ]

Hey MIGUEL_C_Intel,

Thanks for your response.

I think you already answered the cause of the problem, on our affected systems that we cannot adopt we still have SHA1 Hash, I will try with SHA2 and if the problem still occurs I will send you the requested logs.

Is there any way to send you the logs directly, as I'm a bit uncomfortable with sharing whole server-logs to the public.

Regards,

Lucas

[LucasOIZ]

Hey @MIGUEL_C_Intel 

Another update; I tested it with SHA1 certificate, that's not the issue it seems since I still can adopt SHA1 clients fine.

By running the EMA Configuration Tool I may have seen a pattern of what clients we can adopt to EMA without issues and those we can't.

In the CMD when running the EMA Configuration Tool clients not working to adopt send the following:

On those not working clients I'm unable to generate an XML but with directing the output to a log I was still able to capture the most part, although compared to a working one there's a bunch of information missing.

I'll attach the log of a client with said error below (the client the log comes from is currently provisioned by SCS fine but unable to adopt).

Regards,

Lucas

[MIGUEL_C_Intel]

Hello, LucasOIZ,

Thank you for your update on the outcome of running the EMA Configuration tool.  Please send us the EMA server log through a private message.

Look forward to your response with the log.

Regards,

Miguel C.

Intel Customer Support Technician

[LucasOIZ]

Hey @MIGUEL_C_Intel 

Since I can't find an option to send you a private message, I'll attach the logs with removed sensitive data to this post.

The client hostname in question is XXZN0043 for example.

Regards,

Lucas

[MIGUEL_C_Intel]

Hello, LucasOIZ,

The issue started after doing the BIOS update of the machines, and the PKI DNS suffix in the MEBx BIOS of the machine looks empty according to the EMA Configuration tool.

Can you confirm the status of the PKI DNS suffix in MEBx?

We can gather this by using this alternative method.

PowerShell script:  

Run the command called: CSME-DiscoverySmbiosAdvanced.ps1 (.\CSME-DiscoverySmbiosAdvanced.ps1)

Downloading the Intel® Endpoint Management Assistant (Intel® EMA) API Sample Scripts is necessary.

https://www.intel.com/content/www/us/en/download/19693/30076/intel-endpoint-management-assistant-intel-ema-api-sample-scripts.html

Open a Power Shell window as an administrator.

Go to the path where the program was installed.

Look for PowerShell>Snippets folder

Finally, run the command: .\CSME-DiscoverySmbiosAdvanced.ps1

Look forward to your response with the log.

Regards,

Miguel C.

Intel Customer Support Technician

[MIGUEL_C_Intel]

Hello, Lucas,

Adding to my previous post.  

I noted the endpoint XXZN0053 (HP ProBook 650 G4) shows the EMA software version 1.10.1.0; as per logs.  The forum was opened with the EMA version 1.9.1.0.  Did you recently update the EMA version?

Do you mind sending me a log from any working machine EMA Configuration Tool?

Look forward to your response with the log.

Regards,

Miguel C.

Intel Customer Support Technician

[LucasOIZ]

Hey @MIGUEL_C_Intel 

First; Yes we updated the EMA version in the meantime, we hoped it would fix the issue

I'll attach you the requested items:

- G9 working.png: CSME discovery of a working device (this is a different hardware-type but, it works)

- G4 not working.png: CSME discovery of the device "XXZN0043" used as a example in my last post together with the EMA Logs

- XXZN0073_System_Summary.xml: EMAConfigTool log from the working device the screenshot "G9 Working" is from.

In Addition, I'll add a CSME Discovery plus a EMAConfigTool of a Probook 650 G5 (XXZN0059) which is special because it worked to adopt from SCS after updating the BIOS, shows the issue with WSMAN when running EMAConfigTool and does report a functioning ME WMI with CSME Discovery 

I noticed that on every device not working to adopt the CSME Discovery reports "Intel(R) ME WMI" as "False" but there are also working devices which show the same result

Regards,

Lucas

[MIGUEL_C_Intel]

Hello, LucasOIZ,

I sent you a private email, look forward to hearing from you.

Regards,

Miguel C.

Intel Customer Support Technician

[LucasOIZ · Accepted solution]

@MIGUEL_C_Intel and I were able to figure out my issue;

It was the lack of the needed ME-Drivers, in case anyone runs into the same issue.

Thanks for your time, help and the pointing in the right direction.

Regards,

Lucas