TCB_OUT_OF_DATE despite TCB recovery

[CLauverjat]

Hello,

We are experiencing problems with the attestation failing because of a TCB_OUT_OF_DATE. We are aware that Intel has updated the TCB due to the discovery of vulnerabilities that require MCU patch (INTEL-SA-00615, INTEL-SA-00657). And we have applied the TCB Recovery procedure, but despite that the attestation still fails due to a TCB_OUT_OF_DATE

Machine description:

OS : Ubuntu 20.04 LTS

Model : DELL PowerEdge-R750xs

CPU : Intel(R) Xeon(R) Gold 6334 CPU @ 3.60GHz

We are using intel-microcode Ubuntu package to apply microcode updates.

Here is the procedure we followed:

• Upgrade Intel SGX PSW to version 2.18.101.1
• Upgrade Intel SGX DCAP to version 1.15.100.3
• Upgrade BIOS to the latest version provided by DELL : https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=g7k8g&oscode=ubt20&productcode=poweredge-r750xs&lwp=rt
• Upgrade intel-microcode package to 3.20220809.0ubuntu0.20.04.1
• We used PCKIDRetrievalTool to retrieve the Provisioning Certification Key (PCK) Certificates to attest to a new TCB. For that we followed Intel® Software Guard Extensions Data Center Attestation Primitives (Intel® SGX DCAP): A Quick Install Guide available at https://www.intel.com/content/www/us/en/developer/articles/guide/intel-software-guard-extensions-data-center-attestation-primitives-quick-install-guide.html

Here are the SGX PSW + DCAP software versions after upgrade : (output from apt list --installed)

libsgx-ae-epid/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-ae-id-enclave/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-ae-le/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-ae-pce/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-ae-qe3/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-ae-qve/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-ae-tdqe/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-aesm-ecdsa-plugin/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-aesm-epid-plugin/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-aesm-launch-plugin/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-aesm-pce-plugin/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-aesm-quote-ex-plugin/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-dcap-default-qpl/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-dcap-ql/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-dcap-quote-verify/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-enclave-common-dev/unknown,now 2.18.101.1-focal1 amd64 [installed]
libsgx-enclave-common/unknown,now 2.18.101.1-focal1 amd64 [installed]
libsgx-epid/unknown,now 2.18.101.1-focal1 amd64 [installed]
libsgx-headers/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
libsgx-launch/unknown,now 2.18.101.1-focal1 amd64 [installed]
libsgx-pce-logic/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-qe3-logic/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-quote-ex/unknown,now 2.18.101.1-focal1 amd64 [installed]
libsgx-ra-network/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-ra-uefi/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-tdx-logic/unknown,now 1.15.100.3-focal1 amd64 [installed]
libsgx-uae-service/unknown,now 2.18.101.1-focal1 amd64 [installed]
libsgx-urts/unknown,now 2.18.101.1-focal1 amd64 [installed]
sgx-aesm-service/unknown,now 2.18.101.1-focal1 amd64 [installed,automatic]
sgx-dcap-pccs/unknown,now 1.15.100.3-focal1 amd64 [installed]
sgx-pck-id-retrieval-tool/unknown,now 1.15.100.3-focal1 amd64 [installed]

Concerning the microcode here is the tool version :

intel-microcode/focal-updates,focal-security,now 3.20220809.0ubuntu0.20.04.1 amd64 [installed]

It looks like the right microcode update is applied see dmesg output below :

dmesg | grep "microcode"
[ 1.610253] microcode: sig=0x606a6, pf=0x1, revision=0xd00037b
[ 1.610401] microcode: Microcode Update Driver: v2.2.

The microcode version indicated is 0x37b

and according to your documentation https://www.intel.com/content/www/us/en/developer/topic-technology/software-security-guidance/processors-affected-consolidated-product-cpu-model.html for our processor we need (at least) 2022.2: 0x375. So it should be ok, right ?

But despite the procedure we still encounter the TCB_OUT_OF_DATE error.

At this point we were a bit lost, so we modified the QVL to extract the TCB values from the PCK certificate to get a better picture of the TCB, and we got :

certPceSvn = 11
# Tcb[i] indicates the SVN for each level of the TCB
Tcb[0] = 4
Tcb[1] = 4
Tcb[2] = 3
Tcb[3] = 3
Tcb[4] = 255
Tcb[5] = 255
Tcb[6] = 0
Tcb[7] = 0
Tcb[8] = 0
Tcb[9] = 0
Tcb[10] = 0
Tcb[11] = 0
Tcb[12] = 0
Tcb[13] = 0
Tcb[14] = 0
Tcb[15] = 0

By comparing that to the TCBInfo.json we can see that up-to-date TCB should have a PCESVN of 13 but we only have a PCESVN of 11.

Is there something we overlooked ? How can we resolve this TCB issue ?

Thank you

[KFPW_Intel]

Hi,

Thank you for your interest in Intel® SGX.

Recently there is another posting asked a similar issue regards to TCB OutOfDate. Do you see any errors that can share to us?

Based on the last similar issue, we suggested to check the version of the uCode loaded by the BIOS, which can be found in the BIOS setup menu. The latest cpu_svn[0] is 0x7.

In the meantime, we are investigating this issue with the development teams regards to the TCB Recovery mentioned, especially for Intel® Xeon® Gold 6334 Processor.

Please allow some time for us to investigate, thank you for your patience.

Regards,

Ken

[KFPW_Intel]

Hi,

Is the suggested method to check the uCode loaded by the BIOS works for you? Hope that it helps.

Furthermore you can get the SGX TCB Info via curl -v -X GET "https://api.trustedservices.intel.com/sgx/certification/v4/tcb?fmspc={}" based on the documentation that may assist you for your use cases.

Please inform us if you have any questions regards to this issue. Thank you.

Regards,

Ken

[CLauverjat · Accepted solution]

Hello,

After further investigation, we were able to solve the issue. I only had remote access to the server, so I was only able to access the BIOS via the racadm tool that allow remote control to the DELL server. Because of that I could not find the CPU revision, but I expect the revision to be the last one since we have done all relevant BIOS updates. 

By I'll cut to the chase, when I was in the (remote) BIOS settings I still found a SysSecurity.SgxFactoryReset setting that could be turned on. I did that (which resetted SGX keys), and after a restart it finally worked, the TCB_OUT_OF_DATE went away, and we know have a STATUS_TCB_SW_HARDENING_NEEDED which is to be expected ! 

I was somewhat surprised this worked, so I did some digging and I found this document https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/resources/intel-sgx-software-and-tcb-recovery-guidance.html which states :

> If the 3rd Generation Intel® Xeon® Scalable Processors, code name Icelake-SP Post Launch Release 3 (PLR3) microcode update (MCU) is loaded at reset (otherwise known as FIT-loaded¹) and the platform then executes the TCB Recovery Boot Flow², a microcode (MCheck) error occurs that causes Intel® Software Guard Extensions (Intel® SGX) to be disabled.

I don't now for sure if the problem I encountered was related to the issue described in the document because in our case the Intel SGX was never disabled... but I expect a SgxFactoryReset to inititate a Initial Platform Establishment (IPE) flow, which the document says work fine contrary to the TCB Recovery Boot Flow (which was what we were doing before).

Anyway with the caveat that we lost the sealing keys during the reset (fortunately we could do that because we did not have any important data sealed on the server), it solves the issue for us.

[KFPW_Intel]

Hi,

This is amazing! It is good to know that issue can be solved.

Thank you for sharing the solutions to us.

This thread will be marked as answered and Intel will no longer monitor this thread.

Please start a new thread if you need further help.

Regards,

Ken