Re: Unable to connect to a Swarm Server - Failed PKI provisioning

TomW · ‎01-29-2024

Hi,

Not sure where to go from here as I'm at a bit of a dead end having tried just about everything I could think of. I'm sure I had PKI provisioning (with CIRA) working at one point, but at the moment none of my clients are able to provision.

The two notable entries in the EMALog-ManageabilityServer.txt log are "Error:Unable to connect to a Swarm Server" and "Failed PKI provisioning". I went down the rabbit hole of assuming it was a PKI issue, but now I'm thinking there's some issue related to not being able to connect to the Swarm Server which is causing PKI provisioning to fail.

Note, in the log entries below, the windows domain has been subsituted with `.local` and the public domain has be substitured with `my.domain.local`. Intel EMA server and Intel vPro compatible certificate is using domain `intelema.my.domain.com`, and PKI DNS suffix has been configured in the BIOS of all computers as `my.domain.com`

Relevant log extract follows:

2024-01-30 12:18:36.8294|INFO||4728|23|AttemptPhase1_Pki - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Attempting phase 1 PKI provisioning : (M104,6FA481CF). 
2024-01-30 12:18:36.8294|INFO||4728|23|PerformPkiSetup - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Get Mesh information (Tenant) : (M104,6FA481CF). 
2024-01-30 12:18:36.8294|INFO||4728|23|PerformPkiSetup - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Starting PKI Setup process for endpoint: (M104,6FA481CF) ComputerName: M104.local 
2024-01-30 12:18:36.9075|INFO||4728|23|PerformPkiSetup - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Setup computer name M104.local : (M104,6FA481CF). 
2024-01-30 12:18:36.9075|INFO||4728|23|RequestHostBasedProvisioningEx - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Setup computer name M104.local : (M104,6FA481CF). 
2024-01-30 12:18:36.9075|INFO||4728|23|RequestHostBasedProvisioningEx - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Sending Agent Stop Remote Configuration Message : (M104,6FA481CF). 
2024-01-30 12:18:36.9075|INFO||4728|23|RequestHostBasedProvisioningEx - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Message:Connecting to Swarm Server : (M104,6FA481CF). 
2024-01-30 12:18:36.9232|WARN||4728|40|MessageManager_ReceivedMessageEx - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [0] - Warning:Received stop remote configuration status from: 6FA481CF, status: INVALID_PT_MODE (3) 
2024-01-30 12:18:37.0326|ERROR||4728|23|RequestHostBasedProvisioningEx - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Error:Unable to connect to a Swarm Server, user=SYSTEM : (M104,6FA481CF). 
2024-01-30 12:18:37.0326|WARN||4728|23|PerformPkiSetup - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Warning: Failed Intel AMT SetupAdmin activation : (M104,6FA481CF). 
2024-01-30 12:18:37.0326|WARN||4728|23|AttemptPhase1 - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [1] - Failed PKI provisioning : (M104,6FA481CF).

SwamServer log shows the following:

2024-01-30 12:18:32.9700|INFO||9076|26|ProcessCommand - MeshServer.MeshAgent, EMASwarmServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Message:Got 0 provisioning hash(s) from M104. Match found! DNS Suffix: my.domain.com 
2024-01-30 12:18:33.0169|INFO||9076|54|ProcessCommand - MeshServer.MeshAgent, EMASwarmServer, Version=1.12.1.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Message:Intel AMT OTP confirmed.

Running `emaagent.exe -swarmserver` on one of the clients yields the expect result of:

Intel(R) EMA Swarm server address and port are intelema.my.domain.local:8080

Checking netstat and Resource Monitor on the server, there's an established connection between the client and the swarm server. See screenshot below. So not sure why it's saying it's unable connect to sawrm server unless that's a red herring?

If somebody could assist, that'd be very much appreciated.

TomW · ‎01-29-2024

Not sure if this has anything to do with the problem I'm having, but...

I've gone into the MEBx BIOS and double-checked the PKI DNS suffix. It looked right, but when I select it to edit, I noticed some random characters at the end of the string (",%W"). See photo below. If I enter any other domain, even if it's just adding or removing a single character, save, exit and go back into it, those extra characters don't appear at the end. As soon as I re-enter my domain name exactly, those extra characters come back, though sometimes they change, e.g. ",½#" or ",$W".

I've even done a full reset of MEBx (disable and re-enable) which cleared the PKI TLS suffix. Upon re-entering, goes back to doing the same thing. Has to be some weird bug. Not sure if this is what's causing my provisioning issues or not.

I'm on Intel Management Engine Bios Extension 16.0.0.0002/Intel ME 16.1.27.2176

Happy to private message the domain name for somebody at Intel to have a look at to see if they can reproduce this issue using our domain name.

I've just tried with a laptop that has a slightly different version of Intel ME and it doesn't exhibit this bug. It also can't provision and gets the same PKI provisioning failure.

I'm happy to provide my certificate and chain to somebody if they can verify it.

Victor_G_Intel · ‎01-30-2024

Hello TomW,

Thank you for your response.

Please bear in mind that everything that you post here in the forum is publicly available for all users. We don’t recommend the use of the forum for any privately considered matters since the use of it is meant for community based support instead of privately based support, in that way both us, peers and sometimes former engineers can jump in and help. In this case we believe this situation you are experiencing can be best handle privately; therefore, we encourage you to open a private ticket with the following link.

Submit Service Request

https://supporttickets.intel.com/supportrequest?lang=en-US&productId=123804:9758

Best regards,

Victor G.

Intel Technical Support Technician

TomW · ‎01-30-2024

Thanks victor, support request has been raised. No doubt someone else will come across this same issue in the future, so once we work out what's going on, I'll post back here. Hopefully I don't forget.

TomW · ‎02-01-2024

I ended up getting to the bottom of the issue myself in the end. Writing this here in case it helps somebody else in the future.

The error "Unable to connect to a Swarm Server" seems to indicate a problem with server-side Intel EMA components talking to the Swarm Server via the admin port (8089 by default). Another thread on these forums had that same log entry here which helped me come to this conclusion: https://community.intel.com/t5/Intel-vPro-Platform/Invalid-certificate/m-p/1506964

In my case, I could see the the SwarmServer was listening on port 8089. I could also telnet to this port and establish a connection at least temporary, so there wasn't anything getting in its way. My problem stemmed from the fact that I had renamed the server that Intel EMA was hosted on. Even though I re-installed Intel EMA after the server rename, I used the same database.

By chance, I ended up coming across the following in Intel's server installation and maintenance guide which clued me onto the problem:

For Server ID, you will need to review the Intel EMA database, specifically the [dbo].[ServerSettings] table.
The correct Server ID value on this dialog will be the value of ValueInt field in the database table with Type
= 2 and for the server Name corresponding to your new additional server

Having previously looked at this ServerSettings table, I rembered seeing two entries in there; one for the old server name and one for the new server name. The new (correct) server name entry had a ValueInt of 2.

So to fix this, I had to update the Swarm Server ID from 1 to 2 on the Server Settings page of the Intel EMA web/admin portal (logged on as global administrator). I had to change this on the Swarm Server, Ajax Server and Manageability Server tabs as they each have their own config.

So even though the IP and port were correct, the Server ID needs to be correct as well, otherwise it won't be able to connect.

Everything immediately began provisioning successfuly after this. Hopefully this helps somebody else and saves them the hours and hours of troubleshooting this took me.