Intel vPro® Platform
Intel Manageability Forum for Intel® EMA, AMT, SCS & Manageability Commander
2834 Discussions

EMA Server / Client with vpro won't finish configuration for PKI certificate.

Mike_Modality
Beginner
4,801 Views

Hello,

I'm trying to deploy vpro / ema. I have an off net server running the EMA server with an AMT certificate installed. When I install the ema agent on a device and install the necessary msh file, it connects, I can reboot the system, but it's provisioning is pending configuration.

 

Any help with this would be greatly appreciated.

Here is some information about the setup.

Server is Server 2022 - I have enabled older SSL protocols for testing.

 

**removed**
 
 
     
 
 
**removed**
 
 

 

On the client side, I see this error when it tries to connect.

[2023-05-04 01:46:48.411 PM] \Agent\MeshManageability\agent\microstack\ILibAsyncSocket.c:505 internalSocket ERROR: 0. Last error: 0

 

2023-05-04 11:52:21.9499|INFO||6740|50|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=**removed**- [1] - Message:AMT Profile detected : (***removed***,5C675EE9).
2023-05-04 11:53:08.0998|WARN||6740|50|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed***- [1] - Warning:Unable to connect to Intel AMT computer for round 2, 127.0.0.1:50250
2023-05-04 11:53:08.0998|WARN||6740|50|PerformRound2Provisioning - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Warning:(Host=127.0.0.1, Computer=***removed***, Domain=, Tls=True, Endpoint=(***removed***,5C675EE9), User=SYSTEM, UserId=00000000-0000-0000-0000-000000000000)
2023-05-04 11:53:08.0998|WARN||6740|50|AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Failed PKI provisioning : (***removed***,5C675EE9).

0 Kudos
33 Replies
MIGUEL_C_Intel
Employee
3,776 Views

Hello, Michael,


I will gladly assist you.


The log is showing a failure while validating the provisioning:

2023-05-04 11:53:08.0998|WARN||6740|50|AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Failed PKI provisioning: (***removed***,5C675EE9).


Please confirm if the Remote access to the Endpoint is working even when you are getting the Pending Configuration message in the EMA web console. 

If yes, please restart the EMA services or restart the server.


1- Do you mind giving me more details of the Certificate? Is it a self-Certificate or any authorized OEM Intel® AMT certificate?

2- The Certificate chain (Root, Intermediate, and Leaf) needs to comply with SHA256 ( 2048 bits ).  Please send a picture of the Cert chain from the Certificates Path tab.

3- How did you provision the endpoint? Using the EMA agent file or manually in the MEBx BIOS.

4- Please include the EMA log from Server. The path is: 

[System drive]\Program File(x86)\Intel\Platform Manager\EmaLogs


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,729 Views

Thank you for responding. I'm happy to provide any details you require.

I've gone to bat with this for over a month. 

This is recent fresh install, both devices have been rebooted, remote access works. I've tried with multiple devices, for this install it has just been the one device. For vpro systems with the same older version I get the same results, I can remote into them, but it's pending configuration and CIRA does not connect.

For a newer vpro system ,I get a cert verify failure.

If needed I can join those devices again to generate the logs.

 

I've attached the logs for this vpro system with the one system trying to provision.

 

For your direct questions.

1 & 2.  This is an AMT Certificate purchased as such from Sectigo / commodo

Mike_Modality_0-1683564846412.png

Mike_Modality_1-1683564890531.png

 

3. In all my test cases the device was provisioned using the EMA Agent. Systems were as up to date for vpro as possible, rebooted, and I also fully unprovisioned them to clear them out, and also set my network suffix to match the certificate.

 

4. See attachment.

 

 

0 Kudos
SergioS_Intel
Moderator
3,712 Views

Hello Mike_Modality,

 

Thank you for the additional information.


Can you please let us know the brand and model of the systems that you are using and how many systems are you having this issue?

 

 Best regards,

 Sergio S.

 Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,683 Views

Hello SergioS.

 

I'm having this problem with at least two systems I've tested. One is a lenovo M80. This M80 had the same error as the 30AH004MUS.

The system that is currently in the logs.

Manufacturer
LENOVO
Model
30AH004MUS
--This system is the one in the logs with the current error
 
Other system I tried in a previous EMA server installation was 
Manufacturer
LENOVO
Model
11TG0020US
 
This newer system when it pushed the PKI certificate gave me a 'CERT_VERIFY_ERROR' when pushing the chain. My first thought that there was a problem with the certificate but the certificate provider has been less than helpful, and it does get accepted by the M80 and the other system I listed here.
 
So far I've actually actually go vpro to fully work on any system as yet to date, we're hoping to add it to our tools if I can make something happen.
0 Kudos
MIGUEL_C_Intel
Employee
3,677 Views

Hello, Mike_Modality,


I reviewed the logs, and only the certificate issue pops-up has seemed. 


After reviewing the pictures provided and the documentation available, I noted the following. Sectigo SHA256 Certificate hash was included in systems with Intel® AMT 15 and later.  Systems with older AMT versions require a different vendor Certificate. I am including the documentation. 


Releases 15.0.45, 16.1, and later support the following root certificate

https://software.intel.com/sites/manageability/AMT_Implementation_and_Reference_Guide/WordDocuments/rootcertificatehashes.htm


The Certificate hash is a code included in the BIOS firmware of the machines, it validates the Certificate included in the EMA server.


To verify if the current Cert belongs to AMT, go to the Cert - Comodo AMT Cert (leaf) and validate the Enhanced Key usage matches AMT OID: 2.16.840.1.113741.1.2.3


It is possible to validate the Intel® AMT version by running:

Endpoint Management Assistant Configuration Tool 

https://www.intel.com/content/www/us/en/download/19805/30485/intel-endpoint-management-assistant-configuration-tool-intel-ema-configuration-tool.html

 

Installation:

Double-click the .msi file and follow the prompts.

 

Run:

a-Open a command prompt (alternatively, you can run the tool from within Windows PowerShell*).

b-Navigate to the installation folder (default C:\Program Files (x86)\Intel\EMAConfigTool).

c-Run the command: EMAConfigTool.exe -filename XXXX --verbose 


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,665 Views

Thank you for the reply.

I have previously verified it had the correct OID.

Please see below

Mike_Modality_0-1683664495728.png

Here is the endpoints, you can see their intel AMT version

 

Mike_Modality_1-1683665442184.png

 

Please see the additional log file attached, this is what happens when the modern AMT system connects. If you need me to submit other / full logs let me know.

 

Mike_Modality_2-1683665532329.png

 

 

I was thinking the same thing you were, that the certificate they gave me would only work on newer systems.

So I tried it with a newer system, and it didn't work. I received a cert verify failure which is in the log.

0 Kudos
MIGUEL_C_Intel
Employee
3,652 Views

Hello, Mike_Modality,


Thank you for your response with the log and pictures.


Taking into consideration that it is a new installation, and the Certificate is new. Do you mind doing the following:


Unconfigure the endpoints. It is possible to perform this by the following:

a- First, we need to access the EMA web console and gather the access password for each endpoint if you selected the randomize option. From the action option of each endpoint, we can gather the password. 

b- Unconfigure the endpoint using Endpoint Management Assistant Configuration tool (ECT).

https://www.intel.com/content/www/us/en/download/19805/30485/intel-endpoint-management-assistant-configuration-tool-intel-ema-configuration-tool.html

c- Uninstall and delete the EMA agent file from each endpoint.

d- Finally, go to the EMA web console and stop provisioning the endpoint.


Latest Intel® Endpoint Management Assistant (Intel® EMA) 1.10.1

https://www.intel.com/content/www/us/en/download/19449/intel-endpoint-management-assistant-intel-ema.html 


Before provisioning the endpoints, please send me the ECT logs from both systems (Intel® AMT version 9 and 16). Please send them as a zip file. 


Installation:

Double-click the .msi file and follow the prompts.

 

Run:

a-Open a command prompt (alternatively, you can run the tool from within Windows PowerShell*).

b-Navigate to the installation folder (default C:\Program Files (x86)\Intel\EMAConfigTool).

c-Run the command: EMAConfigTool.exe -filename XXXX --verbose


I am including a summary of the case:

2- Endpoint

LENOVO Model 30AH004MUS

MIT-WKBNCH-SRV v9.1.45 - Provisioned

SSM-WS02

Windows 10

ME: 16.0.15.1620

AMT status: Pending Activation


Operating System: Microsoft® Windows 11

Intel® EMA Agent: Win64-Service v1.10.0

Intel® ME: v9.1.45.3000 Admin Control Mode  

CIRA selected: Yes

Intel® AMT setup status: Pending Configuration


For vpro systems with the same older version I get the same results, I can remote into them, but it's pending configuration and CIRA does not connect.

---------------------------------

3- Endpoint

LENOVO Model 11TG0020US

SSM-WS02   v16.0.15 Not Provisioned

2023-05-04 11:53:08.0998|WARN||6740|50|AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=***removed*** - [1] - Failed PKI provisioning: (***removed***,5C675EE9).


For a newer vpro system, I get a cert verify failure.


EMALog-ManageabilityServer

2023-05-09 13:48:44.1461|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - ema.modality.ca : (SSM-WS02,99B51BD7). 

2023-05-09 13:48:44.2254|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - Sectigo RSA Domain Validation Secure Server CA : (SSM-WS02,99B51BD7). 

2023-05-09 13:48:44.3081|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - USERTrust RSA Certification Authority : (SSM-WS02,99B51BD7). 

2023-05-09 13:48:44.3831|INFO||7048|34|HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Pushing activation certificate - AAA Certificate Services : (SSM-WS02,99B51BD7).


Excuse me for all the troubleshooting; I am trying to narrow down the issue.


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,619 Views

Thank you.

 

I've completed the steps you requested on the newer device running the 16 version. I have yet to complete it on the older device as I'm getting a WSman connection error, so I may just fresh that system and try it again to get you clean logs. Please see the attach file with the log for SSM-WS02 after it was unprovisioned.

0 Kudos
MIGUEL_C_Intel
Employee
3,601 Views

Hello, Mike_Modality,


I reviewed the ECT log and confirmed the Lenovo 11TG0020US is not provisioned, has no PKI DNS suffix, and it is using ME version 16.0.15.1620.


I noticed, no network is recognized (wire or wireless). Are you using a docking station?


Finally, please double-check if the machine is using the latest BIOS version. I am sending Lenovo’s website. Current BIOS: M40KT3DA


Lenovo ThinkCentre M80s Gen 3 – SFF

https://pcsupport.lenovo.com/us/en/products/desktops-and-all-in-ones/thinkcentre-m-series-desktops/thinkcentre-m80s-gen-3/downloads/ds556726-flash-bios-update-for-thinkcentre-m80t-gen-3-m80s-gen-3-m90t-gen-3-m90s-gen-3-neo-70t-gen3?category=BIOS%2FUEFI


I look forward to the pending log and answers.


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,576 Views

Hey Miguel.

I performed an bios update but that didn't seem to change anything.

It's a wired connection, no docking station as this is a SFF PC with DHCP. I'm attaching the relevant IPconfig information. If you need more of the output let me know. It shows the DNS suffix there which is odd that it doesn't show up for the PKI DNS Suffix.

I'm not sure why it shows the IP as 0.0.0.0 in the log from ema config tool, is that just the IP it's binding to or how it reads DHCP vs static? It sounds to me like the AMT is just using the standard 0.0.0.0 any interface configuration that other network applications commonly use.

0 Kudos
MIGUEL_C_Intel
Employee
3,561 Views

Hello, Mike_Modality,


Yes, you are right.  It is very odd, the IP address is not recognized, and the network connection is working.  Intel® AMT uses the same IP address of the machine, it does not create a dedicated connection.


Do you mind running our tool called Intel® System Support Utility for Windows and sharing the results?

https://www.intel.com/content/www/us/en/download/18377/intel-system-support-utility-for-windows.html


In addition, please open a command line window and run the command: ipconfig

Please let me know if you have a VPN, proxy, or any restrictions.


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,556 Views

I have no restrictions, no VPN, no proxy, and full access to our internal firewall which is direct to an external static. Nothing should be interfering with it's connection and I have full access to all our configurations.

 

I ran the command with ipconfig /all

0 Kudos
MIGUEL_C_Intel
Employee
3,537 Views

Hello, Mike_Modality,


Thank you for your quick response.


It seems the firewall is not letting the EMA server verify the endpoint. Please disable the firewall on both sides.  


Note: from the previous post, the PKI DNS suffix is empty in the endpoint because we ran the ECT tool with the command reconfigure.  For provisioning, it is necessary to install and run the EMA agent file again. 


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,514 Views

Hey Miguel.

 

Unfortunately I've been running it without the firewall the entire time so it would not be related to that. I've also ensured all the necessary ports are forwarded. Communication shows activity when the devices try to provision, in the example with the 16 version, the error is cert_verify_failure, I would expect a different error message if it was a communication failure by a firewall.

 

The PKI suffix message stays the same on the 16 version even after provisioning the device again, where I stay in the same not activated state, and it just keeps retrying until it gives up until a reboot.

0 Kudos
MIGUEL_C_Intel
Employee
3,505 Views

Hello, Mike_Modality,


I am going to investigate internally the issue with the engineering team; please send me a new ECT log after reinstalling the EMA agent file to the endpoint with AMT 16. In addition, please send a new Server log after trying to provision this endpoint.


EMA Configuration Tool log instructions:

a-Open a command prompt (alternatively, you can run the tool from within Windows PowerShell*).

b-Navigate to the installation folder (default C:\Program Files (x86)\Intel\EMAConfigTool).

c-Run the command: EMAConfigTool.exe --verbose


EMA logs from Server

[System drive]\Program File(x86)\Intel\Platform Manager\EmaLogs

 

EMA log from the endpoint:

[System drive]\Program Files\Intel\EMA Agent\EMAagentlog


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
MIGUEL_C_Intel
Employee
3,364 Views

Hello, Mike_Modality,


I hope this post finds you well.


By any chance, have you been able to work on my request?


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,354 Views

Sorry for the delay in response, it's been busy. Please see the attached logs. There was no EMA agent log folder created after install.

SSMWS02 is the ECT log, and the other EMA logs are well, the EMA logs. These were grabbed right after installing the device and the ema agent provsioning.

Mike_Modality_0-1684364571457.png

 

0 Kudos
MIGUEL_C_Intel
Employee
3,316 Views

Hello, Mike_Modality,


Thank you for providing me with the EMA server logs and the ECT log of the endpoint.  We are still getting the issue; the provisioning of the endpoint is failing. This is the reason the endpoint log was not created.


The Connection-specific DNS Suffix says ema.modality.ca. Usually, we should see the IP address assigned by the Internet Service Provider or IP assigned by the company and not the URL of EMA.  


Your network configuration is not allowing the certificate validation.  In addition, please verify which domain was used for the certificate, it should match your company domain. As an example, for Intel it is intel.com


I am adding a summary of the errors:

EMALog-ManageabilityServer

HostBasedAdminUpdate - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Warning:Failed to push activation certificate - CERT_VERIFY_FAILED : (SSM-WS02,99B51BD7). 

RequestHostBasedProvisioningEx - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Warning:Unable to go to admin mode, rolling back out of client mode : (SSM-WS02,99B51BD7). 

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Connecting to Swarm Server : (SSM-WS02,99B51BD7). 

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Requesting ME unprovisionning : (SSM-WS02,99B51BD7). 

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Disconnecting Swarm Server : (SSM-WS02,99B51BD7). 

PushCredentialsToMeshAgent - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Clearing credentials from ema agent : (SSM-WS02,99B51BD7). 

TriggerMeHbpUnprovision - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Deactivation completed : (SSM-WS02,99B51BD7). 

PerformPkiSetup - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Warning: Failed Intel AMT SetupAdmin activation : (SSM-WS02,99B51BD7). 

AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.10.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Failed PKI provisioning : (SSM-WS02,99B51BD7).


ECT log:

  ME Version    16.0.15.1620

  MESKU Intel(R) Full AMT Manageability

  ME Provisioning State Not Provisioned

  Is AMT Provisioned False

  Is AMT Ready For Provisioning True

  Micro LMS State NotPresent

  IsEHBCEnabled  False

  ControlMode:   None

  PKI DNS Suffix:   Not Found


I look forward to hearing from you.


Regards,

Miguel C.

Intel Customer Support Technician


0 Kudos
Mike_Modality
Beginner
3,310 Views

Hello MIGUEL_C_Intel.

 

Yes, I'm aware that it fails to provision. That's why I posted the original issue about the PKI certificate failing to provision. I've seen the logs and watched in live with the exact failures you have listed so we're in sync with that part, I'm quite aware of which part is failing. Why it's failing is the part I'm trying to diagnose.

 

Systems here use Azure, so I'm free to set the domain prefix to anything I'd like. If the certificate is for ema.modality.ca, are you saying the DNS suffix should just be modality.ca? I can make that change without issue if that is the case. 

 

I'm also not sure how the network is not allowing the certificate to provision. On older vpro systems as demonstrated, which I can get logs and do it again, the certificate gets pushed and is accepted. That's on the exact same network, physically beside the newer vpro system that fails to provision. Everything configuration-wise is the same except the version of vpro. Our firewall has no rules restricting any outbound traffic and return paths on the network these systems are being tested on. If you feel confident the network is at fault I can even test these system in an isolated DMZ network to prove it out.  The only problem I have with the older vpro systems, is that the CIRA fails to connect.

 

Thank you.

0 Kudos
MIGUEL_C_Intel
Employee
3,289 Views

Hello, Mike_Modality,

 

Yes, please use the DNS suffix modality.ca.  The current DNS ema.modality.ca should not be an inconvenience; we want to keep the configuration with the recommended EMA settings.

 

Related to CIRA connection failure with the old machine; this is a limitation of the Intel® AMT version. Intel® EMA (CIRA) requires AMT version 11.8.79 or later.  It is possible to provision and access the machine with limitations.


Details in section 5 Agent Prerequisites
https://downloadmirror.intel.com/646990/Intel_EMA_Release_Notes.pdf#page=16

 

Regarding the Certificate issue, please send me the following:
Go to the Settings tab of the EMA web console (tenant account) and send me a picture.

 

Finally, for our records, please let me know the SQL version you are running, and where is installed.  In the case of using Azure, please confirm if you are using Azure SQL app or did you create a VM and installed the database in it.

 

Look forward to your response; if there is no response to this email, I will send you a follow-up on 5/23/2023.

 

Regards,
Miguel C.
Intel Customer Support Technician

0 Kudos
Reply