Machines that were once provisioned are no longer provisioned

[idata]

We are currently experiencing issues with our vPro Machines that we manage through SCCM 2007 SP1. About three months ago we deployed over 300 HP DC7800 machines running AMT v.3.2.2. Most of them provisioned successfully and we were able to use Out of Band Management on them. However recently we have noticed that many machines that were once provisioned are no longer provisioned. The AMT status for the machines show "Unknown" but the oobmgmt.log shows "Device Already Provisioned". We are currently unable to use Out of Band Management on most of the machines.

Not sure where to begin troubleshooting. Need help.

Thank you,

Kristine Marcellus

[idata]

Kristine,

The first thing I would do is run meinfowin.exe on one of the clients that you believe is unprovisioned, and see what the provisioning status of the AMT firmware is. Once we know this, we can determine whether or not the problem lies somewhere within the ConfigMgr client software.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Hi Trevor,

I ran the tool on the client. Here is the information:

Intel(R) MEInfo Win Version: 2.5.0.1032

BIOS Version: 786F1 v01.26

Intel(R) AMT code versions:

Flash: 3.2.2 Netstack: 3.2.2 Apps: 3.2.2 Intel(R) AMT: 3.2.2 Sku: 14 VendorID: 8086 Build Number: 1033 Recovery Version: 3.2.2 Recovery Build Num: 1033 Legacy Mode: False

Link status: Link up

Cryptography fuse: EnabledFlash protection: EnabledLast reset reason: Power upSetup and Configuration: CompletedBIOS Mode: Post BootDedicated Mac Address: 00-22-64-a4-44-4dHost Mac Address: 00-22-64-a4-44-4cFWU Override Counter: AlwaysFWU Override Qualifier: AlwaysFW on Flash Desc Override: DisableKedron Driver Version: Not AvailableKedron HW Version: Not AvailableUNS Version: 3.2.0.1018LMS Version: 3.0.10.1018HECI Version: 3.0.30.1086

[idata]

Kristine,

It appears that, based upon the "Setup and configuration: Completed" message, that the device truly is provisioned. It would logically conclude that you would receive the "Device already provisioned" message in your oobmgmt.log.

Can you right-click on the system resource and run an AMT discovery? In the context menu, select Out of Band Management --> Discover Management Controllers. When you run this, look at the amtopmgr.log file on ConfigMgr site server, and see what messages come up.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Hi Trevor,

Attached are the messages from the amtopmgr.log.

Thanks,

Kristine

[idata]

Kristine,

If you run "Update collection membership" on a collection where this AMT system is contained, wait for the query to complete, then refresh the view, what does the AMT status field show as? You might need to go to the View menu to select the AMT status column, but I'm sure you're already familiar with this

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[William_Y_Intel]

Can you clear the CMOS battery to reset one of the devices you are having problems with and see if we can get the system to reprovision again through the SCCM agent (as done before)? This would not be the ultimate fix but I would simply like to know if these devices will re-provision again.

[idata]

Hi Trevor,

I Updated Collection Membership on the collection and did a refresh and the AMT status is now showing Detected.

Kristine

[idata]

Hi Bill,

After clearing the CMOS battery we were able to get the machine to reprovision successfully. We are now able to use Out of Band Managment tools on the machine.

Kristine

[William_Y_Intel]

I have a theory that the Digest password on AMT and SCCM OOB console got out of sync. If that did actually happen, this would be the behavior I believe. SCCM could no longer manage the device since it does not have the admin password it uses to manage the device (and this would most like change the status in SCCM to Detected). The CMOS clear sets everything back to factory default (admin disgest password would be admin, SCCM would know this, and then during provisioning it would modify it and use this new password for management). Now we need to figure out how this would happen.

[idata]

Bill,

That's kinda the direction I was going in ... but I wanted to do a little more discovery, first.

The thing is ... I think we're going to start running into this issue also. Bear with my theoretical thought process here: Since there is an entry in the ConfigMgr database for AMT that's maintained separately from the ConfigMgr resource, there is a record elsewhere that joins the ConfigMgr resource, and the AMT information (including the randomized digest password). If the ConfigMgr resource record is disjoined from that machine, and a new one is created for the same system (new resource ID, SCCM GUID, etc), then the AMT information will no longer be tied to the ConfigMgr resource.

This scenario could happen if, for example, a system is reimaged, which is (or at least can be) a somewhat common occurrence. It could also happen if the ConfigMgr client fails on a system, and needs to be removed / repaired.

In my opinion, there needs to be some sort of maintenance task that reguarly checks for "stale" AMT records, and attempts to determine if there is a matching ConfigMgr resource to join it to. Theoretically, this shouldn't be hard to do, though I don't have my ConfigMgr database in front of me to investigate right at the moment.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[William_Y_Intel]

As for reimaging the system, a process should be followed to perform a full unprovision in SCCM to clear the MEBx, remove the AMT OU Object and revoke the certificate for that system name. Then after the reimaging process, let the SCCM agent perform the normal inband agent based provisioning. Think of it as a process performed to clean out the AD for old computer objects. Now for the SCCM agent reinstall, I was not aware this would break the functionality of SCCM to AMT. This is something I will have to test and report back. Thanks.

[idata]

Bill,

I respectfully disagree.

Since AMT is an out-of-band technology, I think that refreshing an operating system on a client should not require AMT to be unprovisioned and reprovisioned. In fact, I would expect that AMT would facilitate the reimaging of a system, and not cause additional complication to the process.

The method that we are currently using to name our systems, at least at HQ, is based on the serial number of the system. Every time a system is reimaged, it would retain the same hostname, so refreshing it would not cause a hostname mismatch.

Since our system refreshes do not involve renaming a device, I would expect to not have to reprovision a device. On the other hand, if the hostname does change, I realize that it would need to be reprovisioned.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[William_Y_Intel]

I don't think you are disagreeing but merely asking for what you would "like" SCCM product to do. Unfortunately, reality today is that an unprovision is required if you - and rename the system. Good feedback for Microsoft and I'd like for you to make note of it through your Microsoft contacts. We can do the same on our side.

Intel has worked on developing tools for other ISV console to help with the syncing of names when the OS and AMT names mismatch (Reflector: http://communities.intel.com/docs/DOC-1431 http://communities.intel.com/docs/DOC-1431). But this would not help if the system was provisioned with SCCM and associated certificates are loaded in the firmware and AD Objects created for specific AMT names. Since SCCM applies a cert to each AMT device and it is specific to the name of the AMT system, we need to make sure this certificate name matches.

Today's reality is that you should perform the process step in SCCM to unprovision the AMT device before re-imaging a system. Maybe someone in the community has written some tools to help with this manual step in SCCM. And probably more appropriate for new thread since we are starting to diverge from Kristine's issue and getting more into product improvements.

[William_Y_Intel]

Kristine,

Can you try to unplug the system so it will reboot the MEBx? I beleive the problem you are seeing is related to an issue that was discovered with Kerberos. After 25 continuous days of the MEBx being on (remember that could include the OS off but MEBx still running in S5 state) and the machine stops accepting Kerberos credentials. The problem has been root caused and firmware is being worked to address. This will be released trhough your OEM. Please validate that this power cycle addresses your issue. Thanks.

[idata]

Not to re-hash this old thread, but I am seeing this exact issue in that even with reimaging a computer and using the same name (so no hostname mismatch problem) I still have to fully unprovision the computer before SCCM will discover the AMT Status as aything but "Unknown"...

Trevor are you really not having this issue with reimaging your systems?