Re: Re:AMT Broken by KB5008102 (SysErr: DSID-031A1255, problem 22 (Invalid argument))

IanB · ‎03-04-2022

We've just noticed that our RCS server has stopped provisioning since November when we applied KB5008102.

We've seen the update to 12.2.0.170 (Intel® Setup and Configuration Software (Intel® SCS)), but that package isn't a complete package, and we're still running 12.0.0.129.

We've given the 12.2.0.170 ACUConfig.exe a try against our 12.0.0.129 server, but it fails with the same error, presumably because the 12.0.0.129 RCS service itself isn't updated and is still trying to create samaccountnames with $iME on the end.

Are we missing something? Or do we need to get the sever up to 12.2.x before putting on the 12.2.0.170 update? And if so, where do we get a 12.2.x installer from as I can't find it now it's approaching end of life.

Thanks in advance

SergioS_Intel · ‎03-04-2022

Hello IanB,

Thank you for contacting Intel Customer Support.

I understand that you are unable to Provision after Applying Microsoft* Security Update KB5008102. The change to how SCS creates the AD computer objects only seems to impact customers who select the option for Cisco* ISE to authenticate against Any Subject or Alternative Name Attributes in the Certificate (for Active Directory Only).

I will be more than glad to help you today.

There is a step-by-step guide that will help you with your problem:

https://www.intel.com/content/www/us/en/support/articles/000089661/software/manageability-products.html

I hope the information on the previous site helps you. We will be looking forward to your updates.

Best regards,

Sergio S.

Intel Customer Support Technician

IanB · ‎03-07-2022

Sergio,

Thank you for your reply.

We aren't using Cisco ISE in our environment. We're using a Microsoft CA for the certificates.

I've tried the change suggested in the article anyway, but when we remove the Host name and DNS Hostname (FQDN) as per the article we get an error telling us that it's not a valid configuration:

Just to be clear, this is the same issue that you worked on in this case: Re:AD Integration broken after KB5008102 - Page 2 - Intel Communities

Regards,

Ian.

SergioS_Intel · ‎03-07-2022

Hello IanB,

We appreciate the additional information, please allow us to check it and we will get back to you.

Best regards,

Sergio S.

Intel Customer Support Technician

SergioS_Intel · ‎03-08-2022

Hello IanB,

Thank you for waiting for updates, in order to continue troubleshooting your problem, please check the Intel® Setup and Configuration Software (Intel® SCS) - Archive version 12.2.0.170

https://www.intel.com/content/www/us/en/download/19764/30337/intel-setup-and-configuration-software-intel-scs-archive.html, and please run the update on the environment to test the provisioning on the devices.

Additionally, please let us know your environment details (how many devices do you have, how many are having the issue, your company information, for how long has the environment been running).

Looking forward to your information.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

IanB · ‎03-09-2022

Sergio,

Thanks for the reply. We've already tried the update you've highlighted. It appears to be intended for later versions of the RCS server than ours. It looks like it is a partial update for 12.2.x systems. We are on 12.0.0.129.

When we run the two installers in that package it appears to want to install them on different paths than our current 12.0.0.129 system. Also, running the acuconfig.exe that it unpacks still produces the same error as our current version of acuconfig. I assume this is because the RCS server doesn't get an update from that package, so it still tries to create records in Active Directory with SamAccountName's in the form host$iME which is now blocked by the microsoft update from November 2021 (See KB5008102—Active Directory Security Accounts Manager hardening changes (CVE-2021-42278) (microsoft.com)).

The error we see in the acuconfig log is:
Thread:12928(ERROR) : ACU Configurator, Category: Exit Source: Src\ActivatorMain.cpp : configurator::LogAndExit Line: 226: ***********Exit with code 75. Details: Failed to complete remote configuration of this Intel(R) AMT device. An Active Directory interface internal error occurred. Active directory create object function CreateDSObject failed with error -2147024865 (A device attached to the system is not functioning.: LDAP Provider: 00000523: SysErr: DSID-031A1255, problem 22 (Invalid argument), data 0). Verify the container path and privileges.

On the Active Directory DC we get:

Log Name: System
Source: Microsoft-Windows-Directory-Services-SAM
Date: 04/03/2022 11:56:00
Event ID: 16991
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: DCNAME.contoso.com
Description:
The security account manager blocked a non-administrator from creating or renaming a computer account using an invalid sAMAccountName. sAMAccountName on computer accounts must end with a single trailing $ sign.

Attempted sAMAccountName: HOSTNAME$iME
Recommended sAMAccountName: HOSTNAME$iME$

Regards,

Ian.

JoseH_Intel · ‎03-09-2022

Hello IanB,

Could you try this installer package instead: Intel® Setup and Configuration Software (Intel® SCS) - Archive ? This is the full package (75.2 MB) and contains the required files to perform a full upgrade.

Regards

Jose A.

Intel Customer Support Technician

IanB · ‎03-14-2022

Thank you Jose A,

The in-place upgrade did the trick. We're up and running again now.

Regards,

Ian.

mezzadrist · ‎04-10-2023

Hi Ian,

The Intel Setup and Configuration Software Archive has been removed from Intel's web page.

Is there another location where I could download the full version of 12.2.0.170 ?

In the future, I'll migrate to Intel EMA, but for right now I'd like to try the 12.2.0.170 patch if it's available.

mezzadrist · ‎04-10-2023

Hi Jose,

The Intel Setup and Configuration Software Archive has been removed from Intel's web page.

Is there another location where I could download the full version of 12.2.0.170 ?

In the future, I'll migrate to Intel EMA, but for right now I'd like to try the 12.2.0.170 patch if it's available.

SergioS_Intel · ‎03-14-2022

Hello IanB,

We are glad to hear that you were able to resolve your problem, please let us know if you need further assistance or if we can close this thread.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

mezzadrist · ‎04-10-2023

Hi Sergio,

The Intel Setup and Configuration Software Archive that you and Jose have referred to has been removed from Intel's web page.

Is there another location where I could download the full version of 12.2.0.170 ?

In the future, I'll migrate to Intel EMA, but for right now I'd like to try the 12.2.0.170 patch if it's available.

IanB · ‎03-15-2022

Sergio,

It was all going great until some of our scheduled maintenance jobs started overnight. Now we're getting the server service crashing out repeatedly.

Faulting application name: RCSServer.exe, version: 12.2.0.170, time stamp: 0x61e9e0d1
Faulting module name: RCSServer.exe, version: 12.2.0.170, time stamp: 0x61e9e0d1
Exception code: 0xc0000409
Fault offset: 0x00397384
Faulting process id: 0x1108
Faulting application start time: 0x01d8385bfa450f49
Faulting application path: C:\Program Files (x86)\Intel\SCS12.2\Service\RCSServer.exe
Faulting module path: C:\Program Files (x86)\Intel\SCS12.2\Service\RCSServer.exe
Report Id: 3fd2a835-a44f-11ec-813a-005056b60087
Faulting package full name:
Faulting package-relative application ID:

We can't stop the jobs running either. As soon as the service starts it attempts to carry on running them, then it crashes.
We've tried aborting, but it's now stuck as aborting. We can't edit them either.

How do we force them to stop so we can get control of the system again?

This is the tail end of an rcs.log (hostnames redacted)
2022-03-15 12:25:57: Thread:2576(DETAIL) : HOSTNAME1.ad.bangor.ac.uk, Category: AMTCommunicator Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\amt\amtinterface\wsmancommunicator.cpp : AMTInterfaceNamespace::WSMANCommunicator::CommitChanges Line: 7558: WS-Management call CommitChanges (SetupAndConfigObj.InvokeCommitChanges) ok
2022-03-15 12:25:57: Thread:6664(DETAIL) : HOSTNAME2.ad.bangor.ac.uk, Category: AMTCommunicator Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\amt\amtinterface\wsmancommunicator.cpp : AMTInterfaceNamespace::WSMANCommunicator::SetNetworkEnabled Line: 708: WS-Management call SetWiredNetworkSettings (AMT_GeneralSettings.Put) ok
2022-03-15 12:25:57: Thread:6664(DETAIL) : HOSTNAME2.ad.bangor.ac.uk, Category: Commit Changes Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\vproconfiguration\vproconfigurationinternal.cpp : vProConfigurationNamespace::vProConfigurationInternal::CommitChangesEx Line: 1553:
2022-03-15 12:25:58: Thread:6664(DETAIL) : HOSTNAME2.ad.bangor.ac.uk, Category: AMTCommunicator Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\amt\amtinterface\wsmancommunicator.cpp : AMTInterfaceNamespace::WSMANCommunicator::CommitChanges Line: 7558: WS-Management call CommitChanges (SetupAndConfigObj.InvokeCommitChanges) ok
2022-03-15 12:25:58: Thread:3136(DETAIL) : HOSTNAME3.ad.bangor.ac.uk, Category: AMTCommunicator Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\amt\amtinterface\wsmancommunicator.cpp : AMTInterfaceNamespace::WSMANCommunicator::GetAmtVersion Line: 120: Failed while calling WS-Management call GetAmtVersion (CIM_SoftwareIdentity.Get). Intel(R) AMT connection error 0xc000521d: The caller is unauthorized.
2022-03-15 12:25:58: Thread:3136(ERROR) : HOSTNAME3.ad.bangor.ac.uk, Category: AMT Interface error Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\vproconfiguration\vproconfigurationinternal.cpp : vProConfigurationNamespace::vProConfigurationInternal::TestConnection Line: 1266: Failed while calling WS-Management call GetAmtVersion (CIM_SoftwareIdentity.Get). Intel(R) AMT connection error 0xc000521d: The caller is unauthorized. , error in discover 0xc000521d
2022-03-15 12:25:58: Thread:3136(DETAIL) : HOSTNAME3.ad.bangor.ac.uk, Category: TestAllConnections params Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\vproconfiguration\vproconfigurationinternal.cpp : vProConfigurationNamespace::vProConfigurationInternal::TestAllConnections Line: 548: Connection data - Connection type: TLS-SSL, FQDN: HOSTNAME3.ad.bangor.ac.uk, IP: 147.143.41.27, UserName: admin
2022-03-15 12:25:58: Thread:3136(DETAIL) : HOSTNAME3.ad.bangor.ac.uk, Category: Test Connection Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\vproconfiguration\vproconfigurationinternal.cpp : vProConfigurationNamespace::vProConfigurationInternal::TestConnection Line: 796:
2022-03-15 12:25:58: Thread:3136(DETAIL) : HOSTNAME3.ad.bangor.ac.uk, Category: Certificate store Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\vproconfiguration\vproconfigurationinternal.cpp : vProConfigurationNamespace::vProConfigurationInternal::TestConnection Line: 1202: Valid certificates for SSL connection not found. Certificate for Mutual TLS
2022-03-15 12:25:58: Thread:3136(DETAIL) : HOSTNAME3.ad.bangor.ac.uk, Category: DiscoverAMTConnectionMode Source: c:\gitlab-runner\builds\1ok2djwe\0\bcpd\manageability\scs\scs12_2dev_klockwork\products\scs\modules\vproconfiguration\vproconfigurationinternal.cpp : vProConfigurationNamespace::vProConfigurationInternal::TestConnection Line: 1214: Connection Info-HOSTNAME3.ad.bangor.ac.uk admin TLS_CONN:

Regards,

Ian

SergioS_Intel · ‎03-15-2022

Hello IanB,

We are sorry to read about your issue.

Since you mentioned that there is a maintenance job that keep crashing the RCS, can you please let us know if there is a 3rd party tool that does this maintenance, if so can you please provide us it's name?

Also, please let us know if it is affecting all the target systems or just a few.

Looking forward to your response.

Best regards,

Sergio S.

Intel Customer Support Technician

IanB · ‎03-16-2022

Sergio,

As per the attached screenshot, these are the maintenance jobs set up inside the SCS console from the jobs tab. No third party product involved. Now that they've started they are causing the RCSServer service to crash repeatedly and we can't stop them from running. They're stuck on "aborting" and options to edit them are greyed out.

Regards,

Ian.

SergioS_Intel · ‎03-17-2022

Hello IanB,

We appreciate the additional information, please allow us to check it and we will get back to you.

Best regards,

Sergio S.

Intel Customer Support Technician

SergioS_Intel · ‎03-18-2022

Hello Ianb,

Can you please confirm that you followed the steps on Intel® Setup and Configuration Software (Intel® SCS) User Guide (https://www.intel.com/content/dam/support/us/en/documents/software/Intel_SCS_User_Guide.pdf)

section 8.7 page 190 to either abort or delete a job?

Were these maintenance jobs created and working fine prior to the update?

Best regards,

Sergio S.

Intel Customer Support Technician

IanB · ‎03-21-2022

Sergio,

Yes, that's how we aborted the jobs. However they remain at status "Aborting" and looking at the rcs.log they appear to continue to do something when the server service starts that causes the server to crash and the job to never complete the abort.

We can't delete the jobs because they're still running.

Regards,

Ian.

SergioS_Intel · ‎03-21-2022

Hello IanB,

We appreciate the additional information, please allow us to check it and we will get back to you.

Best regards,

Sergio S.

Intel Customer Support Technician

SergioS_Intel · ‎03-22-2022

Hello IanB,

Thank you for waiting for our updates.

The cause was due to a handful (37) of NULL valued records in the database and the fact that Kerberos authentication was no longer working for AMT for those computers.

The cleanup will be pretty straightforward, delete the records from the database and unconfigure AMT. The bad news is it’s going to require that you perform a full unconfiguration, which is necessary to clear the unknown randomized admin password from AMT. And since Kerberos authentication is broken and we don’t know the admin password, you’ll need to unconfigure AMT locally from the MEBx, which you would need to get into anyway because of a full unconfiguration also removes your custom configuration certificate hash from the MEBx.

This same cause is most likely also what’s causing the RCSService crashes on the production server.

These are the database queries you’ll need to execute.

Identify Problem Records

SELECT * FROM [IntelAMTQ001].[dbo].[amt] WHERE curr_admin_password is null

Delete Problem Records

DELETE FROM IntelAMTQ001.dbo.amt WHERE curr_admin_password IS NULL

We hope this information helps.

Best regards,

Sergio S.

Intel Customer Support Technician

SergioS_Intel · ‎03-28-2022

Hello IanB,

We are following your case and would like to know if the steps provided on our previous post were useful to solve your issue.

Best regards,

Sergio S.

Intel Customer Support Technician