EMA Server / Client with vpro won't finish configuration for PKI certificate.

Mike_Modality · ‎05-04-2023

Hello,

I'm trying to deploy vpro / ema. I have an off net server running the EMA server with an AMT certificate installed. When I install the ema agent on a device and install the necessary msh file, it connects, I can reboot the system, but it's provisioning is pending configuration.

Any help with this would be greatly appreciated.

Here is some information about the setup.

Server is Server 2022 - I have enabled older SSL protocols for testing.

Intel® AMT profile: **removed**

Operating System: Microsoft® Windows 11

Intel® EMA Agent: Win64-Service v1.10.0

Intel® ME: v9.1.45.3000 Admin Control Mode

CIRA selected: Yes

Intel® AMT setup status: Pending Configuration

Interface: **removed**

Nearby endpoints: 0

Hardware Management Capability: Intel® Active Management Technology

On the client side, I see this error when it tries to connect.

[2023-05-04 01:46:48.411 PM] \Agent\MeshManageability\agent\microstack\ILibAsyncSocket.c:505 internalSocket ERROR: 0. Last error: 0

2023-05-04 11:52:21.9499|INFO||6740|50|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=**removed**- [1] - Message:AMT Profile detected : (***removed***,5C675EE9).
2023-05-04 11:53:08.0998|WARN||6740|50|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed***- [1] - Warning:Unable to connect to Intel AMT computer for round 2, 127.0.0.1:50250
2023-05-04 11:53:08.0998|WARN||6740|50|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Warning:(Host=127.0.0.1, Computer=***removed***, Domain=, Tls=True, Endpoint=(***removed***,5C675EE9), User=SYSTEM, UserId=00000000-0000-0000-0000-000000000000)
2023-05-04 11:53:08.0998|WARN||6740|50|AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Failed PKI provisioning : (***removed***,5C675EE9).

MIGUEL_C_Intel · ‎05-05-2023

Hello, Michael,

I will gladly assist you.

The log is showing a failure while validating the provisioning:

2023-05-04 11:53:08.0998|WARN||6740|50|AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Failed PKI provisioning: (***removed***,5C675EE9).

Please confirm if the Remote access to the Endpoint is working even when you are getting the Pending Configuration message in the EMA web console.

If yes, please restart the EMA services or restart the server.

1- Do you mind giving me more details of the Certificate? Is it a self-Certificate or any authorized OEM Intel® AMT certificate?

2- The Certificate chain (Root, Intermediate, and Leaf) needs to comply with SHA256 ( 2048 bits ). Please send a picture of the Cert chain from the Certificates Path tab.

3- How did you provision the endpoint? Using the EMA agent file or manually in the MEBx BIOS.

4- Please include the EMA log from Server. The path is:

[System drive]\Program File(x86)\Intel\Platform Manager\EmaLogs

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-08-2023

Thank you for responding. I'm happy to provide any details you require.

I've gone to bat with this for over a month.

This is recent fresh install, both devices have been rebooted, remote access works. I've tried with multiple devices, for this install it has just been the one device. For vpro systems with the same older version I get the same results, I can remote into them, but it's pending configuration and CIRA does not connect.

For a newer vpro system ,I get a cert verify failure.

If needed I can join those devices again to generate the logs.

I've attached the logs for this vpro system with the one system trying to provision.

For your direct questions.

1 & 2. This is an AMT Certificate purchased as such from Sectigo / commodo

3. In all my test cases the device was provisioned using the EMA Agent. Systems were as up to date for vpro as possible, rebooted, and I also fully unprovisioned them to clear them out, and also set my network suffix to match the certificate.

4. See attachment.

SergioS_Intel · ‎05-08-2023

Hello Mike_Modality,

Thank you for the additional information.

Can you please let us know the brand and model of the systems that you are using and how many systems are you having this issue?

Best regards,

Sergio S.

Intel Customer Support Technician

Mike_Modality · ‎05-09-2023

Hello SergioS.

I'm having this problem with at least two systems I've tested. One is a lenovo M80. This M80 had the same error as the 30AH004MUS.

The system that is currently in the logs.

Manufacturer

LENOVO

Model

30AH004MUS

--This system is the one in the logs with the current error

Other system I tried in a previous EMA server installation was

Manufacturer

LENOVO

Model

11TG0020US

This newer system when it pushed the PKI certificate gave me a 'CERT_VERIFY_ERROR' when pushing the chain. My first thought that there was a problem with the certificate but the certificate provider has been less than helpful, and it does get accepted by the M80 and the other system I listed here.

So far I've actually actually go vpro to fully work on any system as yet to date, we're hoping to add it to our tools if I can make something happen.

MIGUEL_C_Intel · ‎05-09-2023

Hello, Mike_Modality,

I reviewed the logs, and only the certificate issue pops-up has seemed.

After reviewing the pictures provided and the documentation available, I noted the following. Sectigo SHA256 Certificate hash was included in systems with Intel® AMT 15 and later. Systems with older AMT versions require a different vendor Certificate. I am including the documentation.

Releases 15.0.45, 16.1, and later support the following root certificate

https://software.intel.com/sites/manageability/AMT_Implementation_and_Reference_Guide/WordDocuments/rootcertificatehashes.htm

The Certificate hash is a code included in the BIOS firmware of the machines, it validates the Certificate included in the EMA server.

To verify if the current Cert belongs to AMT, go to the Cert - Comodo AMT Cert (leaf) and validate the Enhanced Key usage matches AMT OID: 2.16.840.1.113741.1.2.3

It is possible to validate the Intel® AMT version by running:

Endpoint Management Assistant Configuration Tool

https://www.intel.com/content/www/us/en/download/19805/30485/intel-endpoint-management-assistant-configuration-tool-intel-ema-configuration-tool.html

Installation:

Double-click the .msi file and follow the prompts.

Run:

a-Open a command prompt (alternatively, you can run the tool from within Windows PowerShell*).

b-Navigate to the installation folder (default C:\Program Files (x86)\Intel\EMAConfigTool).

c-Run the command: EMAConfigTool.exe -filename XXXX --verbose

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-09-2023

Thank you for the reply.

I have previously verified it had the correct OID.

Please see below

Here is the endpoints, you can see their intel AMT version

Please see the additional log file attached, this is what happens when the modern AMT system connects. If you need me to submit other / full logs let me know.

I was thinking the same thing you were, that the certificate they gave me would only work on newer systems.

So I tried it with a newer system, and it didn't work. I received a cert verify failure which is in the log.

MIGUEL_C_Intel · ‎05-09-2023

Hello, Mike_Modality,

Thank you for your response with the log and pictures.

Taking into consideration that it is a new installation, and the Certificate is new. Do you mind doing the following:

Unconfigure the endpoints. It is possible to perform this by the following:

a- First, we need to access the EMA web console and gather the access password for each endpoint if you selected the randomize option. From the action option of each endpoint, we can gather the password.

b- Unconfigure the endpoint using Endpoint Management Assistant Configuration tool (ECT).

https://www.intel.com/content/www/us/en/download/19805/30485/intel-endpoint-management-assistant-configuration-tool-intel-ema-configuration-tool.html

c- Uninstall and delete the EMA agent file from each endpoint.

d- Finally, go to the EMA web console and stop provisioning the endpoint.

Latest Intel® Endpoint Management Assistant (Intel® EMA) 1.10.1

https://www.intel.com/content/www/us/en/download/19449/intel-endpoint-management-assistant-intel-ema.html

Before provisioning the endpoints, please send me the ECT logs from both systems (Intel® AMT version 9 and 16). Please send them as a zip file.

Installation:

Double-click the .msi file and follow the prompts.

Run:

a-Open a command prompt (alternatively, you can run the tool from within Windows PowerShell*).

b-Navigate to the installation folder (default C:\Program Files (x86)\Intel\EMAConfigTool).

c-Run the command: EMAConfigTool.exe -filename XXXX --verbose

I am including a summary of the case:

2- Endpoint

LENOVO Model 30AH004MUS

MIT-WKBNCH-SRV v9.1.45 - Provisioned

SSM-WS02

Windows 10

ME: 16.0.15.1620

AMT status: Pending Activation

Operating System: Microsoft® Windows 11

Intel® EMA Agent: Win64-Service v1.10.0

Intel® ME: v9.1.45.3000 Admin Control Mode

CIRA selected: Yes

Intel® AMT setup status: Pending Configuration

For vpro systems with the same older version I get the same results, I can remote into them, but it's pending configuration and CIRA does not connect.

---------------------------------

3- Endpoint

LENOVO Model 11TG0020US

SSM-WS02 v16.0.15 Not Provisioned

2023-05-04 11:53:08.0998|WARN||6740|50|AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Failed PKI provisioning: (***removed***,5C675EE9).

For a newer vpro system, I get a cert verify failure.

EMALog-ManageabilityServer

2023-05-09 13:48:44.1461|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - ema.modality.ca : (SSM-WS02,99B51BD7).

2023-05-09 13:48:44.2254|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - Sectigo RSA Domain Validation Secure Server CA : (SSM-WS02,99B51BD7).

2023-05-09 13:48:44.3081|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - USERTrust RSA Certification Authority : (SSM-WS02,99B51BD7).

2023-05-09 13:48:44.3831|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - AAA Certificate Services : (SSM-WS02,99B51BD7).

Excuse me for all the troubleshooting; I am trying to narrow down the issue.

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-10-2023

Thank you.

I've completed the steps you requested on the newer device running the 16 version. I have yet to complete it on the older device as I'm getting a WSman connection error, so I may just fresh that system and try it again to get you clean logs. Please see the attach file with the log for SSM-WS02 after it was unprovisioned.

MIGUEL_C_Intel · ‎05-10-2023

Hello, Mike_Modality,

I reviewed the ECT log and confirmed the Lenovo 11TG0020US is not provisioned, has no PKI DNS suffix, and it is using ME version 16.0.15.1620.

I noticed, no network is recognized (wire or wireless). Are you using a docking station?

Finally, please double-check if the machine is using the latest BIOS version. I am sending Lenovo’s website. Current BIOS: M40KT3DA

Lenovo ThinkCentre M80s Gen 3 – SFF

https://pcsupport.lenovo.com/us/en/products/desktops-and-all-in-ones/thinkcentre-m-series-desktops/thinkcentre-m80s-gen-3/downloads/ds556726-flash-bios-update-for-thinkcentre-m80t-gen-3-m80s-gen-3-m90t-gen-3-m90s-gen-3-neo-70t-gen3?category=BIOS%2FUEFI

I look forward to the pending log and answers.

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-11-2023

Hey Miguel.

I performed an bios update but that didn't seem to change anything.

It's a wired connection, no docking station as this is a SFF PC with DHCP. I'm attaching the relevant IPconfig information. If you need more of the output let me know. It shows the DNS suffix there which is odd that it doesn't show up for the PKI DNS Suffix.

I'm not sure why it shows the IP as 0.0.0.0 in the log from ema config tool, is that just the IP it's binding to or how it reads DHCP vs static? It sounds to me like the AMT is just using the standard 0.0.0.0 any interface configuration that other network applications commonly use.

MIGUEL_C_Intel · ‎05-11-2023

Hello, Mike_Modality,

Yes, you are right. It is very odd, the IP address is not recognized, and the network connection is working. Intel® AMT uses the same IP address of the machine, it does not create a dedicated connection.

Do you mind running our tool called Intel® System Support Utility for Windows and sharing the results?

https://www.intel.com/content/www/us/en/download/18377/intel-system-support-utility-for-windows.html

In addition, please open a command line window and run the command: ipconfig

Please let me know if you have a VPN, proxy, or any restrictions.

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-11-2023

I have no restrictions, no VPN, no proxy, and full access to our internal firewall which is direct to an external static. Nothing should be interfering with it's connection and I have full access to all our configurations.

I ran the command with ipconfig /all

MIGUEL_C_Intel · ‎05-11-2023

Hello, Mike_Modality,

Thank you for your quick response.

It seems the firewall is not letting the EMA server verify the endpoint. Please disable the firewall on both sides.

Note: from the previous post, the PKI DNS suffix is empty in the endpoint because we ran the ECT tool with the command reconfigure. For provisioning, it is necessary to install and run the EMA agent file again.

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-12-2023

Hey Miguel.

Unfortunately I've been running it without the firewall the entire time so it would not be related to that. I've also ensured all the necessary ports are forwarded. Communication shows activity when the devices try to provision, in the example with the 16 version, the error is cert_verify_failure, I would expect a different error message if it was a communication failure by a firewall.

The PKI suffix message stays the same on the 16 version even after provisioning the device again, where I stay in the same not activated state, and it just keeps retrying until it gives up until a reboot.

MIGUEL_C_Intel · ‎05-12-2023

Hello, Mike_Modality,

I am going to investigate internally the issue with the engineering team; please send me a new ECT log after reinstalling the EMA agent file to the endpoint with AMT 16. In addition, please send a new Server log after trying to provision this endpoint.

EMA Configuration Tool log instructions:

a-Open a command prompt (alternatively, you can run the tool from within Windows PowerShell*).

b-Navigate to the installation folder (default C:\Program Files (x86)\Intel\EMAConfigTool).

c-Run the command: EMAConfigTool.exe --verbose

EMA logs from Server

[System drive]\Program File(x86)\Intel\Platform Manager\EmaLogs

EMA log from the endpoint:

[System drive]\Program Files\Intel\EMA Agent\EMAagentlog

Regards,

Miguel C.

Intel Customer Support Technician

MIGUEL_C_Intel · ‎05-17-2023

Hello, Mike_Modality,

I hope this post finds you well.

By any chance, have you been able to work on my request?

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-17-2023

Sorry for the delay in response, it's been busy. Please see the attached logs. There was no EMA agent log folder created after install.

SSMWS02 is the ECT log, and the other EMA logs are well, the EMA logs. These were grabbed right after installing the device and the ema agent provsioning.

MIGUEL_C_Intel · ‎05-18-2023

Hello, Mike_Modality,

Thank you for providing me with the EMA server logs and the ECT log of the endpoint. We are still getting the issue; the provisioning of the endpoint is failing. This is the reason the endpoint log was not created.

The Connection-specific DNS Suffix says ema.modality.ca. Usually, we should see the IP address assigned by the Internet Service Provider or IP assigned by the company and not the URL of EMA.

Your network configuration is not allowing the certificate validation. In addition, please verify which domain was used for the certificate, it should match your company domain. As an example, for Intel it is intel.com

I am adding a summary of the errors:

EMALog-ManageabilityServer

HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Warning:Failed to push activation certificate - CERT_VERIFY_FAILED : (SSM-WS02,99B51BD7).

RequestHostBasedProvisioningEx - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Warning:Unable to go to admin mode, rolling back out of client mode : (SSM-WS02,99B51BD7).

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Connecting to Swarm Server : (SSM-WS02,99B51BD7).

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Requesting ME unprovisionning : (SSM-WS02,99B51BD7).

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Disconnecting Swarm Server : (SSM-WS02,99B51BD7).

PushCredentialsToMeshAgent - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Clearing credentials from ema agent : (SSM-WS02,99B51BD7).

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Deactivation completed : (SSM-WS02,99B51BD7).

PerformPkiSetup - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Warning: Failed Intel AMT SetupAdmin activation : (SSM-WS02,99B51BD7).

AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Failed PKI provisioning : (SSM-WS02,99B51BD7).

ECT log:

ME Version 16.0.15.1620

MESKU Intel(R) Full AMT Manageability

ME Provisioning State Not Provisioned

Is AMT Provisioned False

Is AMT Ready For Provisioning True

Micro LMS State NotPresent

IsEHBCEnabled False

ControlMode: None

PKI DNS Suffix: Not Found

I look forward to hearing from you.

Regards,

Miguel C.

Intel Customer Support Technician

Mike_Modality · ‎05-18-2023

Hello MIGUEL_C_Intel.

Yes, I'm aware that it fails to provision. That's why I posted the original issue about the PKI certificate failing to provision. I've seen the logs and watched in live with the exact failures you have listed so we're in sync with that part, I'm quite aware of which part is failing. Why it's failing is the part I'm trying to diagnose.

Systems here use Azure, so I'm free to set the domain prefix to anything I'd like. If the certificate is for ema.modality.ca, are you saying the DNS suffix should just be modality.ca? I can make that change without issue if that is the case.

I'm also not sure how the network is not allowing the certificate to provision. On older vpro systems as demonstrated, which I can get logs and do it again, the certificate gets pushed and is accepted. That's on the exact same network, physically beside the newer vpro system that fails to provision. Everything configuration-wise is the same except the version of vpro. Our firewall has no rules restricting any outbound traffic and return paths on the network these systems are being tested on. If you feel confident the network is at fault I can even test these system in an isolated DMZ network to prove it out. The only problem I have with the older vpro systems, is that the CIRA fails to connect.

Thank you.

MIGUEL_C_Intel · ‎05-18-2023

Hello, Mike_Modality,

Yes, please use the DNS suffix modality.ca. The current DNS ema.modality.ca should not be an inconvenience; we want to keep the configuration with the recommended EMA settings.

Related to CIRA connection failure with the old machine; this is a limitation of the Intel® AMT version. Intel® EMA (CIRA) requires AMT version 11.8.79 or later. It is possible to provision and access the machine with limitations.

Details in section 5 Agent Prerequisites
https://downloadmirror.intel.com/646990/Intel_EMA_Release_Notes.pdf#page=16

Regarding the Certificate issue, please send me the following:
Go to the Settings tab of the EMA web console (tenant account) and send me a picture.

Finally, for our records, please let me know the SQL version you are running, and where is installed. In the case of using Azure, please confirm if you are using Azure SQL app or did you create a VM and installed the database in it.

Look forward to your response; if there is no response to this email, I will send you a follow-up on 5/23/2023.

Regards,
Miguel C.
Intel Customer Support Technician