Uniform processor TCB failures on production IAS.

[Dr__Greg]

Good morning to everyone, I hope the week is starting out well.

We would be interested in whether or not any other groups are seeing near universal GROUP_OUT_OF_DATE responses from the production IAS services.  It appears as if the production IAS instance had its TCB rolled last week and we haven't been able to get any hardware to properly attest since then.

The production IAS instance is currently advertising a processor SVN as follows:

08080101010100000000000000000000

The Intel hardware we are testing with and that is running the most recent firmware available is generating CPUSVN's of the following form in the SGX quotes:

0707ffffff0100000000000000000000

Something would seem to be wrong if Intel SGX hardware with the most recent firmware release can't meet the expected processor security version standards.

Dr. Greg

[Scott_R_Intel]

Hello Dr. Greg.

What Intel platforms/CPUs are you using and what BIOS versions do you have installed?

Regards.

Scott

[Dr__Greg]

Good morning Scott, I hope this note finds your day starting well, thanks for taking the time to reach out.

With respect to Intel platforms, we are currently demonstrating the problem on a NUC6i5SYB NUC and a STK2MV64CC ComputeStick.  Both platforms have bee been upgraded to the latest versions of firmware available from Intel's download site.

More specifically the BIOS version numbers from the DMI tables on the platform are as follows:

NUC6i5SYB NUC:  SYSKLi35.86A.0067.2018.0606.1229

STK2MV64CC ComputeStick:  CCSKLm5v.86A.0057.2018.0521.1920

So the firmware date on the NUC is 6/6/2018 and on the ComputeStick it is 5/21/2018.

At these firmware levels the hosts are quoting the following CPUSVN's in their reports:

NUC6i5SYB NUC:  0707ffffff0100000000000000000000

STK2MV64CC ComputeStick:  07070204ff0100000000000000000000

Intel's production IAS is reporting GROUP_OUT_OF_DATE for both hosts secondary to the CPUSVN being out of date.  The Platform Information Report returned by IAS indicates the following security version preferences:

CPUSVN: 08080101010100000000000000000000

ISVSVN: 7

The ID numbers returned by IAS for the attestation reports are as follows:

NUC6i5SYB NUC:   62532737327849248580982883810190258538

STK2MV64CC ComputeStick:  219549571466215172756280634729550381192

If you have access to the IAS backend and want to check the transactions.  Alternately I can attach the REST based IAS quote responses if it would be useful to have those available.

While it isn't an Intel platform, we have a Gigabyte Q170M-MK that is upgraded to the most recent version of firmware for that host.  The firmware is from just after the initial Spectre/Meltdown vulnerabilities so is a bit dated.  That host is generating a CPUSVN out of date attestation and has a CPUSVN recommendation identical to the other two hosts.

We had originally thought that the CPUSVN's may be unique to the host/processor platform but based on what we are seeing it may be more of a generic specification of a set of attributes the processor microcode should have.

So the problem appears to be that the production IAS is comparing the CPUSVN in the generated reports to something beyond what current production releases of firmware are at.  Obviously it would be a significant concern if a decision has been made to deprecate the above platforms as being too old.  It is already becoming apparent that SGX on platforms such as the Gigabyte boards may be problematic since maintaining valid attestations is going to be a function of how long the vendor chooses to supply firmware upgrades for a platform.

Scott, if you scan the other posts you will find a thread where we were engaging with a group that was not able to generate valid attestations with any hardware against the test IAS instance.  That group was experiencing this problem on multiple hardware platforms, all of which had been upgraded to the latest firmware.  We had decoded the platform information reports for that group and the CPUSVN's being recommended by the test IAS instance didn't make any sense.  We had asked for some additional diagnostic information but never heard back and were unable to come to a root cause determination as to what was going on.  Based on that exchange we were harboring concerns that something was awry with CPUSVN versioning information.

If what we are seeing is correct there is a lot of SGX hardware out there that is currently unable to generate valid attestations.

Hopefully the above information is useful, let us know if we can provide any additional diagnostic information.

Have a good day.

Dr. Greg

[Scott_R_Intel]

Hello again, Dr. Greg.

Thanks for the detailed information regarding your systems and failure.  New BIOS's for both of your platforms are being developed and/or validated and will be released ASAP.  I will reply again as soon as I have more information regarding the releases.

Also, thanks for the info about the other post.  I will look for it.

Scott

[Scott_R_Intel]

Hi again Dr. Greg.

There is now a BIOS update available for your STK2MV64CC compute stick that has an updated microcode.  The BIOS update for your NUC6i5SYB should be available shortly as well.

Regards.

Scott

[RDrag]

https://downloadcenter.intel.com/product/91979/Intel-Compute-Stick-STK2mv64CC shows no BIOS update for August, it's still 0058 from July 27th. To be honest, the way Intel is handling this is getting ridiculous... if Remote Attestation can suddenly stop working without anyone at Intel noticing it, then requiring a BIOS update that doesn't exist at the time to make it work again... it's not OK, really. If this was a production system somewhere in the cloud, we'd be at the mercy of the provider to update the BIOS.

[Scott_R_Intel]

Hello, Razvan.

BIOS v. 0058 for the STK2mv64CC is the latest version and has the required microcode update for the GROUP_OUT_OF_DATE issue Dr. Greg reported while using the previous BIOS version, 0057.

Regards.

Scott

[Scott_R_Intel]

Hello again, Dr. Greg.

A BIOS update (v. 0068) is now available for your Intel NUC6i5SYB NUC (SYSKLi35.86A).

https://downloadcenter.intel.com/product/89190

Regards.

Scott

[Dr__Greg]

Good morning Scott, I hope this note finds your day and week starting well.

I apologize for the delay in getting back to you but we wanted to confirm all of our findings before we reported back on the status of the BIOS updates.

We upgraded both the NUC6i5SYB and the STK2MV64CC to the latest firmware releases that Intel validated.  The platforms are now quoting the following CPUSVN's:

NUC6i5SYB: 0808ffffff0100000000000000000000

STK2MV64CC: 08080204ff0100000000000000000000

Unfortunately IAS is still reporting GROUP_OUT_OF_DATE for both platforms,  Interestingly, the platform information report that IAS is returning indicates no reason for the TCB failure.  There are no bits set in the TCB evaluation flags or the PSE evaluation flags.  We are not submitting a PSE manifest for attestation so the latter is not unexpected but the lack of TCB violation flags would seem to be problematic, given that IAS is flagging the platform TCB as invalid.

The platform information report is recommending the following CPUSVN:

08080101010100000000000000000000

So the microcode update raised the platform CPUSVN's above the recommended minimum as desired but this did not affect the GROUP_OUT_OF_DATE status assignment by IAS.

So there is some type of issue that we need to get tracked down.

The BIOS update now makes this situation very similar to what was going on in the following thread:

https://software.intel.com/en-us/forums/intel-software-guard-extensions-intel-sgx/topic/755769

We were trying to assist Yu who was seeing something similar to this on the test-IAS instance.  They had multiple platforms all upgraded to the latest available firmware.  The CPUSVN's their platforms were quoting were clearly more recent then the test-IAS recommendation but they were getting a GROUP_OUT_OF_DATE report.  In their case though the CPUSVN bit was set in the TCB evaluation flags, which was clearly wrong but different then what we are seeing with a GROUP_OUT_OF_DATE report and no indication of a TCB deficiency.

We can provide full IAS quoting and transaction logs for both platforms, including the raw REST attestation report response that includes the IAS transaction identifier, if that would be helpful in getting this tracked down.

We will look forward to your thoughts.

Have a good day.

Dr. Greg

[Scott_R_Intel]

Hello Dr. Greg.

Yes, please provide a zip of the logs at your convenience.  Also, what version of the PSW are you using currently?

Thanks.

Scott

[Dr__Greg]

Good afternoon Scott, I hope this note finds your day going well.

I'm attaching two remote attestation log files to this post,  the files are small enough that it didn't seem to make much sense to add the additional complexity of compressing them.  The attachments are named by the model number of the platform that generated them.

The files contain the output of our remote attestation unit test harness and the tests were run this afternoon.  In order to rule out that the problem may be secondary to whether or not the enclave was attesting in debug or production mode we ran the test on the STK2MV64CC platform with the test enclave signed with our production key with debugging disabled.  The log from the NUC6i55SYB was run with an enclave that was signed with a debug key.  As you will be able to tell from the logs the measurement of the enclave is identical in both cases.

The tests were run with the enclaves from the 2.2 release of the Intel PSW.  The following SVN's are being reported for the two enclaves:

libsgx_qe.signed.so: 7

libsgx_pce.signed.so: 6

In the FWIW department it seems like a strange inconsistency that the TCB evaluation flags are able to differentiate between a quoting enclave (QE) and a certification enclave (PCE) being out of date while only a single platform security version value is included in the platform information report, which appears to specify the recommended security version of the quoting enclave.  So if the PCE is out of date there doesn't appear to be a way of communicating that to the party evaluating the platform information report on a GROUP_OUT_OF_DATE condition.

Hopefully the above information is useful, please let us know if any additional informational is needed.

Have a good evening.

Dr. Greg

[Scott_R_Intel]

Hi Dr. Greg.

Did you remember to call sgx_report_attestation_status after the failed attestation attempt with the returned attestation status?  This should cause the PSW to reprovision your platform and get a new EPID key.  Most importantly, remember to call it with attestation_status != 0. The Linux remote attestation sample isv_app has it commented out, but it is recommended to always be called.

As for the QE/PCE SVN...  neither of these are returned in the platform info report.  Are you referring to the pse_isvsvn that just happens to be the same version as the QE SVN in your logs?

Regards.

Scott

[Dr__Greg]

Good morning Scott.

Thanks for the feedback, we will re-provision the EPID on the platforms and see if this corrects the issue  Given the disclosure yesterday of the L1 cache attacks all of the issues we have seen now make more sense.

Just as an aside.  We don't use the code in question (sgx_report_attestation_status) but for those that do it seems to be rather important functionality to be commented out.  We would at least recommend a comment in the code that failure to call the function will result in persistent inability of the platform to correctly attest its TCB status.

Just to clarify our understanding and possibly assist others in the future.  Based on our experiences is it correct to assume that the TCB compliance issues reported by the TCB evaluation flags (CPU, QE and PCE SVN out of date) are sufficient but not necessary for a GROUP_OUT_OF_DATE report?

We had assumed, incorrectly, that the TCB evaluation flags were conclusive with respect to articulating the rationale for why the platform TCB was not considered fully conformant.  It seems problematic, from the perspective of a verifying party, to have a mysterious fourth 'bucket' with respect to a TCB compliance fault on an attesting platform.

For example, In this case, is it safe to assume that GROUP_OUT_OF_DATE is secondary to the fact that the SVN of the provisioning (PVE) enclave has been rolled forward or is this a generic re-keying secondary to the L1 cache vulnerability issues?  Obviously if there is the possibility that the EPID private key could have been provisioned by a PVE that could theoretically leak information it would be a reasonable security response to force a mass re-keying of platforms in the field.

In response to your issue in the second paragraph above, we were referring to the 'latest_equivalent_tcb_psvn' member of the structure.  We are now assuming that this structure details the CPUSVN and the SVN of the provisioning enclave, is this correct?

We wouldn't have assumed the pse_isvsvn structure element was relevant as the SVN's for both of the PSE enclave in the 2.2 binary enclave release are 8 rather then 7.

All of this takes us back to our previous issue with respect to the clarity of the platform information report structure.  It seems less then a straight forward approach to cite deficiencies in the TCB of the QE/PCE enclaves and then to not have a bit member that specifies the PVE enclave as deficient.  It also seems odd to provide feedback on the required SVN of the PVE but then to not do so on the QE and PCE's.

Thanks again for your feedback, we will look forward to clarifications on the meaning of a GROUP_OUT_OF_DATE attestation report in the presence of no explicitly detailed deficiencies in the attesting platform's TCB.

Have a good day.

Dr. Greg

[Zavalyshyn__Igor]

Hi Scott and Dr. Greg,

Was there any update on this issue?

Thanks