Quoting Enclave not Launching

[Oktopus]

Hello,

I am running on INTEL(R) XEON(R) GOLD 6548Y+ under Ubuntu 24.04 and have a problem interfacing with the quoting enclave. I will number my questions as I pose them to make it easier for you to respond.

The AESM daemon fails to launch the QE. From journalctl -u aesmd:

systemd[1]: Starting aesmd.service - Intel(R) Architectural Enclave Service Manager...
groupadd[3039197]: group added to /etc/group: name=sgx_prv, GID=1011
groupadd[3039197]: group added to /etc/gshadow: name=sgx_prv
groupadd[3039197]: new group: name=sgx_prv, GID=1011
usermod[3039367]: add 'aesmd' to group 'sgx_prv'
usermod[3039367]: add 'aesmd' to shadow group 'sgx_prv'
usermod[3039396]: add 'aesmd' to group 'sgx'
usermod[3039396]: add 'aesmd' to shadow group 'sgx'
aesm_service[3039422]: aesm_service: warning: Turn to daemon. Use "--no-daemon" option to execute in foreground.
systemd[1]: Started aesmd.service - Intel(R) Architectural Enclave Service Manager.
aesm_service[3039432]: [error_driver2api sgx_enclave_common.cpp:280] Enclave not authorized to run, .e.g. provisioning enclave hosted in app without access rights to /dev>
aesm_service[3039432]: [load_qe ../qe_logic.cpp:721] Error, call sgx_create_enclave QE fail [load_qe], SGXError:4004.
aesm_service[3039432]: Failed to load QE3: 0x4004
aesm_service[3039432]: The server sock is 0x5ee419a48f90

(1) I assume that I should be able to launch the quoting enclave and obtain quotes from it even without subscribing to the Intel attestation service. Of course, the quotes will be meaningless without the verification collateral, but suppose I am just starting development and want to get a feel for how the system is supposed to work in the end. Am I right to assume this or does the QE need some token from Intel to be able to launch?

I can see that it was EACCES error that error_driver2api has translated in the log above. Now, in the kernel driver for Intel SGX, EACCES is returned from the ENCLS[EINIT] ioctl (sgx_encl_init) when enclave attributes are outside those explicitly allowed by the mask (linux-sgx/external/dcap_source/driver/linux/ioctl.c:532 or linux/arch/x86/kernel/cpu/sgx/ioctl.c:496). The comment there speaks of the provisioning attribute SGX_ATTR_PROVISIONKEY.
(2) Could this be the reason for the failure?
(3) Where is the quoting enclave found on the system? Is it /opt/intel/sgx-aesm-service/aesm/ (assume the default prefix for SGX, i.e., /opt/intel)? There are two SO files there which look like quoting enclaves: libsgx_qe.signed.so and libsgx_qe3.signed.so.
(4) Why are there two? What is the difference? Is one for legacy EPID-based attestation and the other for ECDSA attestation?
(5) Which one gets loaded by aesmd and how do I check that?
When checking with sgx_sign dump, I can see that libsgx_qe3.signed.so has attribute SGX_ATTR_PROVISIONKEY (0x10) set whereas libsgx_qe.signed.so does not, which makes me think libsgx_qe3.signed.so corresponds to ECDSA-based attestation as it is supposed to interface with the PCE and "the PCE will only honor requests from enclaves with ATTRIBUTES.PROVISIONKEY set to 1" as said in [1, 3.1.1.1].
(6) Could it be that is failing because this attribute is set and not explicitly allowed in the kernel driver. It seems SGX_ATTR_PROVISIONKEY is only set in sgx_ioc_enclave_provision (linux-sgx/external/dcap_source/driver/linux/ioctl.c:707). Is there an ioctl call missing in AESM daemon that would set this attribute and enable the launch?

Any tips on how to debug this further would be greatly appreciated.

I also tried running the example in SGXDataCenterAttestationPrimitives/SampleCode/QuoteGenerationSample. When running without SGX_AESM_ADDR, i.e., in in-proc mode, it times out waiting for some response from localhost:8081:

[QCNL] Info: Fetching from the local cache for: 'https://localhost:8081/sgx/certification/v4/pckcert?qeid=3D3FD702814ADA8CA24173377265D48E&cpusvn=0202191B03FF00060000000000000000&pcesvn=1000&pceid=0000'
[QCNL] Info: Cache missed.
[QCNL] Info: Data not found in local cache. Trying to fetch response from remote URL: 'https://localhost:8081/sgx/certification/v4/'.

(7) What is supposed to be listening on that port? (Nothing currently is.)
(8) Is it supposed to be the AESM daemon?
(9) Or should my own caching service, as described in DCAP documents, be running there?
(10) Or maybe the address is wrong and it can be set somehow (via a config file? where?) to point to an Intel-provided service, like the attestation service API?
(11) What is the difference between certification v1, v2, v3, and v4? v1 is EOL, so I suppose it corresponds to EPID attestation (now deprecated), but v2-v4?
(12) E.g., why is subscription optional for v4 GET PCK Certificate API?

When running the QuoteGenerationSample with SGX_AESM_ADDR=1, i.e., in out-of-proc mode, it fails at sgx_qe_get_target_info with error 0xe042, a timeout:

[APP] Step1: Call sgx_qe_get_target_info:
DEBUG: [sgx_qe_get_target_info ../sgx_dcap_ql_wrapper.cpp:399] Required pub key id size is: 32
DEBUG: [sgx_qe_get_target_info ../sgx_dcap_ql_wrapper.cpp:406] Call sgx_init_quote_ex - second with allocated pub_key_id_buffer.
Error: [sgx_qe_get_target_info ../sgx_dcap_ql_wrapper.cpp:412] Error in sgx_init_quote_ex. 0xe042
Error in sgx_qe_get_target_info. 0xe042

(13) I suppose this is because it is reaching out to the AESM daemon which has failed (?), i.e., it is running but failed to launch the enclave. Is this right?

I would greatly appreciate it if you answered all the questions and provided some next steps on how to debug and resolve this issue?

Please also point me to the relevant documentation or other resources I may have missed, as most of the documentation I found on DCAP and ECDSA attestation was very high-level and took access to the QE for granted.

Thanks in advance.

References:
[1] https://www.intel.com/content/dam/develop/external/us/en/documents/intel-sgx-support-for-third-party-attestation-801017.pdf

[Benny_Intel]

Hello,

could you please make sure the Intel SGX devices exist at /dev/sgx*?
If yes, please provide the permissions of these devices.
Additionally, please provide in the list of groups of the user "aesmd".

As you use Ubuntu 24.04, you do not need to install an Intel SGX driver - especially not the out-of-tree driver.
Can you confirm that you did not install such a driver?

The QuoteGenerationSample tries to reach out to a collateral caching server, e.g., Intel PCCS.
When installed on the same machine, the default URL for Intel PCCS is https://localhost:8081, which you see in your log.

A subscription to our attestation service (Intel® Tiber™ Trust Authority) is not necessary for quote generation. However, Intel® Provisioning Certification Service (Intel PCS) is used to receive a PCK Certificate for your machine during quote generation. Subscription to this service is optional. There is an anonymous endpoint, but it can be throttled when too many requests come in at the same time. 

Detailed steps to setup your environment can be found at https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/02/infrastructure_setup/ - for Intel TDX. The steps are pretty much the same for Intel SGX, so you can find a lot of information there.
Especially, you also need to follow one of the provided registration variants.

Best regards,
Benny

[Oktopus]

Hi Benny,

Thanks for taking the time. The following SGX devices are visible (+ symlinks in /dev/sgx/, all with maximum permissions):

crw-rw---- 1 root sgx 10, 125 Apr 7 11:25 /dev/sgx_enclave
crw-rw---- 1 root sgx_prv 10, 126 Apr 7 11:25 /dev/sgx_provision
crw-rw---- 1 root sgx 10, 124 Apr 7 11:25 /dev/sgx_vepc

User aesmd is a member of the following groups: aesmd, sgx, and sgx_prv, so, in particular, it seems it should have read and write access to all SGX devices under /dev.

I did not install out-of-tree drivers, only the userspace DCAP SDK. lsmod does not show anything SGX-related.

(1) Where can I find Intel PCCS? I see it was removed from the DCAP repo with release 1.22.

Also, all this reaching out to collateral caching service is happening as part of the sgx_qe_get_target_info call. (2) Could you explain what is exactly is happening in this call step by step? I was under the impression that it only loads the QE (or reaches out to it via the AESM daemon) and the QE responds with its sgx_target_info_t structure (its MRENCLAVE, etc.). In other words, I assumed that target info is hardware-derived and the collateral (cert chain, CRL, etc.) is fetched independently by the relying party. I see that my assumption was wrong and it's all done together by this one function, but I still do not see the bigger picture of how this flow works. Could you elaborate?

(3) Do I understand you correctly that there is an open endpoint that anyone can just reach out to at any time and obtain the attestation collateral, i.e., at least the PCK Certificate? In other words: I can have my machine reach out to this endpoint without prior registration/subscription/licensing and get a certificate for my Provisioning Enclave, which then issues certificates for the Quoting Enclaves thus establishing the trust chain? And subscription is only to get a dedicated endpoint ("preferential" treatment, if you allow me, for subscribing clients)? Is my understanding correct?

(4) Could you explain the difference between v2, v3, v4 GET PCK Certificate APIs? Are v2-v3 only supported for backward compatibility for legacy clients or is there a qualitative distinction between the different versions and a reason to use anything other than v4?

Thanks for linking the guide on TDX/SGX environment setup. I am sure it will clear up many things for me.

[Benny_Intel]

Hello,

Thank you for providing the information about /dev. This all looks good.

What (installation) steps did you execute leading to the aesmd log entry "Enclave not authorized to run, .e.g. provisioning enclave hosted in app without access rights to /dev>"?
Can you reproduce it yourself on a fresh machine?

Your other questions:
(1) Please use PCCS as provided in an older branch of the repo (e.g.,  by https://github.com/intel/SGXDataCenterAttestationPrimitives/tree/DCAP_1.21/QuoteGeneration/pccs). Our Intel TDX Enabling guide also described how to install it using this version at Ubuntu 24.04 (https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/02/infrastructure_setup/#provisioning-certificate-caching-service-pccs).

(2) Details of the DCAP library can be found in https://download.01.org/intel-sgx/latest/dcap-latest/linux/docs/Intel_SGX_ECDSA_QuoteLibReference_DCAP_API.pdf.

(3) Intel PCS is used to provide quote generation collateral, e.g., PCK Cert, and quote verification collateral, e.g., TCB Info, SGX QE Identity, Revocation list. At the moment, every collateral can be retrieved without any registration/subscription/licensing. However, PCK Cert retrieval should be used with a subscription for "preferential" treatment.

Please adhere to the terms of use of Intel PCS (https://api.portal.trustedservices.intel.com/termsAndConditions) stating collateral caching is required.

Note that "platform registration" as described at https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/02/infrastructure_setup/#platform-registration is still necessary, before a PCK Cert can be retrieved.

(4) Yes, v2-v3 APIs are only there for backward compatibility. Please use v4. A guide about the differences between v3 and v4 can be found at https://api.trustedservices.intel.com/documents/PCS_V3-V4_migration_guide.pdf.

Best regards,
Benny

[Oktopus]

Hello,

Thanks to your help, I got both in-proc and out-of-proc quote generation (and in fact quote verification as well) working. It seems the only problem was the missing PCCS service.

The enclave launch error in aesmd was unrelated. It seems that after a reboot, aesmd has successfully loaded the enclave (the error is seen only once in the system journal). As a side note, how can I confirm that? It has device /dev/sgx_enclave open, but is there a way to identify which enclaves were successfully launched by a process?

Anyway, the aesmd is live and provides service.

If it is of any interest to you, I believe aesmd was provided with the OS already. I reproduced the error log on a fresh machine. If you would like, I can run some extra steps to investigate the issue, but my original problem is resolved.

Thanks again