Determine security ramifications to protect personal data and information
114 Discussions

Empower SecretFlowCloud Privacy-Preserving LLM Platform with Intel® TDX

0 0 485

Authored by: Yunge Zhu (Intel, Xeon Customer Solution Engineer), Aihui Zhou (AntGroup, Senior Engineer), Edmund Song (Intel, Principal Engineer)

1. Introduction

Large Language Models (LLMs), are adept at handling complex tasks, marking a significant shift in the field of artificial intelligence. They are increasingly being integrated across diverse industries and are set to drive significant innovation. As the adoption of LLMs grows, so does the need for stringent privacy protections. There is a rising concern about the potential for data breaches or improper use of user interactions with these models. Furthermore, given the considerable commercial value of LLMs, there is a pressing need to protect these assets from theft, particularly when deployed in the cloud, where they are vulnerable to unauthorized access. These security challenges have hindered the advancement of LLMs.In response, SecretFlow, in collaboration with Intel® TDX, has developed the SecretFlowCloud Privacy-Preserving LLM Platform. This platform offers a comprehensive suite of secure services, including inference, fine-tuning, and pre-training of LLMs. It is designed to protect both the models and the data, facilitating secure question-and-answer interactions.

2. Motivation

Security for LLM is a critical priority, with a particular focus on the privacy of user Q&A interactions. It is essential to keep user dialogues confidential and to prevent unauthorized access by service providers. The urgency to protect LLMs as valuable commercial assets from theft is heightened, especially in cloud environments where there's a risk of unauthorized entities, including cloud service operators, misappropriating these assets. Additionally, the training data used in the pre-training or fine-tuning of LLMs in the cloud, which carries significant business and privacy implications, requires stringent protection against unauthorized use or disclosure. The existing measures have not fully met these pressing security concerns, highlighting the need for advanced solutions. This has led to the development of the SecretFlowCloud Privacy-Preserving LLM Platform, designed to address these gaps and ensure robust protection for both the models and the data involved.


3. Solution

3.1 Intel® TDX Overview

To help protect data in use and enable confidential computing, Intel has developed and contributed two innovative hardware-based security engines – Intel® Software Guard Extensions (Intel® SGX), an application-level isolation technology, and Intel® TDX, a virtualization-level isolation technology. Furthermore, Intel® TDX can also easily extend support to heterogeneous Trusted Execution Environment (TEE) usage for a more comprehensive confidential computing solution. With these two built-in security technologies, the 5th gen Intel® Xeon® Scalable processors can provide holistic confidential computing capabilities. These capabilities make it possible for CSPs to offer IaaS, PaaS and SaaS applications in a hardware-based TEE without requiring modifications to their existing applications.

  • Trusted Domain for Confidential Computing: Intel® TDX introduces a new virtual guest environment , termed "TD", through utilizing Intel® Virtual Machine Extension (Intel® VMX) and Intel® Multi-Key Total Memory Encryption (Intel® MK-TME). This TD can be isolated from other TDs, instances, as well as underlying system software components. The enforcement of these security measures is accomplished by the TDX Module operating in an enhanced security privilege mode - Secure Arbitration Mode (SEAM).
  • In-flight Memory Encryption with Outstanding Performance: Intel® TDX enables users to encrypt sensitive data in-flight through a built-in memory encryption engine in the integrated memory controller (IMC) of CPU processors. By running the applications in an Intel® TDX-based TEE, when the cloud application is processing user sensitive data, the state of the data can always be protected confidentially while offering outstanding performance compared to traditional data protection methods.
  • Easy-to-Use for Hyperscale Deployment: The “lift-and-shift” simplifies the migration of complex  systems to confidential computing. In addition, Intel® TDX also provides rich cloud operation capabilities for hyperscale deployment, such as live migration and TCB upgrades without service interruption. All these reduce the operation and maintenance costs of confidential computing and improve overall availability.


Figure 1. Confidential Computing on Intel® Xeon® Scalable Platform


3.2 Architecture of SecretFlowCloud Privacy-Preserving LLM



Figure 2. SecreFlowCloud Privacy-Preserving LLM Platform Arctechture run over Intel® TDX

SecretFlow[1], in collaboration with Intel, has developed a Privacy-Preserving solution for LLMs based on Intel® TDX, known as the SecretFlowCloud Privacy-Preserving LLM Platform. This platform aims to safeguard both data and models while providing encrypted inference, fine-tuning, and pre-training functionalities, as shown in Figure2.

  • Large Language Model Inference: Model owner may deploy LLMs onto the SecretFlowCloud for inference services, with encryption measures in place during both transmission and usage. Additionally, the queries and responses of the inference services are encrypted, allowing only the client users to decrypt and view them.
  • Large Language Model Fine-Tuning:  Data owner can upload their LLMs and datasets, which remain encrypted throughout the uploading process. The fine-tuning occurs within TDX environments to ensure the security of both the LLMs and associated datasets. The fine-tuned models can subsequently be used for inference services.
  • Large Language Model Pre-Training:  Model owners also have the option to pre-train using uploaded datasets within the TDX, thereby ensuring their security. Models resulting from this pre-training are then available for additional fine-tuning or for direct use in inference tasks.

Key modules of the solution are outlined below.

Key Management System (KMS)

The KMS oversees the encrypted keys for LLMs and datasets. It also ensures that these keys are applied according to client-specified policies and are not misused, such as restricting the use of datasets solely to fine-tuning or pre-training. To guarantee the KMS's reliability, it is hosted within a Trusted Domain with SecretFlowCloud, making the service measurements public. Before uploading LLMs or datasets, Model owners perform a remote attestation of the KMS to verify its operation within a TDX environment, in line with SecretFlowCloud’s specified service measurements.

Transport and Storage Security

Throughout the uploading process, LLMs and datasets remain encrypted, with the keys generated by the Model owner and managed by the KMS. Once within the SecretFlowCloud storage facility, the encrypted status is maintained during all processing, with keys overseen by KMS, ensuring that no party, including operational staff from SecretFlowCloud, can access the raw data.

LLM Inference

Model owners can deploy LLMs for inference services within a TDX environment. Upon initiation, the service requests the LLM keys from KMS, which remotely attests the service to ensure operation within TDX. Once authenticated, the keys are securely conveyed to the service which decrypts the LLM files solely within the TDX, shielding the model's information from theft. Utilizing cloud-native offerings, the SecretFlowCloud LLM inference service maximizes advantages such as high availability, elastic scalability, monitoring, and alerting capabilities.

Safe LLM Q&A

The inference service presents a secure Q&A interface, where users first remote-authenticate the service to ensure it is running in a TDX environment and aligns with the service measurements disclosed by SecretFlowCloud. Once attested, users can establish a secure channel, for example, by encrypting their queries using the public key of the inference service and sending them as digital envelopes. The returned answers from the LLM are encrypted by the service before being sent back, and users decrypt these locally to retrieve the plain text. Throughout this entire Q&A interaction, the dialogue between users and the LLM remains encrypted, thus protecting the users' privacy beyond the reach of SecretFlowCloud and any third parties. Users can access this LLM inference service via SecretFlowCloud’s API or through the webpage provided by SecretFlowCloud. When using the webpage, remote authentication and encryption of query content are performed locally in the browser.

LLM Fine-Tuning

Fine-tuning helps LLMs perform better on specific tasks or within certain domains. SecretFlowCloud enables clients to fine-tune LLMs using uploaded datasets (e.g., Supervised Fine-Tuning [SFT], Reinforcement Learning from Human Feedback [RLHF]), with the service deployed in a TDX environment. The fine-tuning service requests the LLM and dataset keys from the KMS, which then remotely attests the service to ensure it operates within TDX. Once authenticated, keys are securely transferred to the fine-tuning service. Throughout the fine-tuning process, LLMs and datasets are decrypted solely within the TDX, protecting them from being stolen. The fine-tuned LLMs are stored in encrypted form, with the keys managed by the KMS and deployable as an inference service.

LLM Pre-Training

SecretFlowCloud supports direct pre-training by Model owner using uploaded datasets, deployed within a TDX environment. The pre-training service requests dataset keys from the KMS, which then remotely attests the service to confirm its operation within TDX. Once authenticated, the keys are securely transferred to the pre-training service. Throughout the pre-training process, datasets are decrypted only within the TDX, ensuring their protection from theft. Pre-trained LLMs are stored in encrypted form, with keys managed by the KMS. Clients can further fine-tune these pre-trained models or directly deploy them for inference services.

As described, once LLMs and datasets leave the client's domain, their entire lifecycle, whether it be for inference, fine-tuning, or pre-training, involves encrypted transmission channels and encrypted storage with keys managed by the KMS operating within a TDX. All computational processes take place within the TDX environment, leveraging the confidentiality and integrity protection provided by TEEs, thereby ensuring LLMs and datasets remain secure against theft. Furthermore, by offering a secure LLM Q&A interface, user dialogue content privacy is thoroughly protected. Accordingly, the SecretFlowCloud Privacy-Preserving LLM Platform presents its solution to the LLM security issues highlighted in the motivation section.

4. Public Preview

We are excited to announce that SecretFlowCloud Privay-Preserving LLM based-on Intel® TDX now supports the public preview. SecretFlowCloud Privay-Preserving LLM with Intel® Trust Domain enable users to bring confidential LLM to the cloud without code changes to applications.

This article focuses on introducing the Large Language Model Showroom feature of the platform. The Showroom allows you to deploy an open-source LLM and experience the secure Q&A functionality.

Initially, we proceed to the LLM Showroom interface and select “Execute” to move to the model deployment interface.


You have the option to deploy an open-source LLM, such as Qwen-7B.


Following successful deployment, you will advance to the model Q&A interface. Before gaining access, your browser performs a remote authentication, ensuring entry into the Q&A interface is secured. Similar to other LLM Q&A interfaces, you input your question, and the LLM responds based on your query. At first glance, everything seems normal; however, the interaction between you and the LLM, including the Q&A content, is encrypted.


By clicking the “View Privacy” button (the eye icon in the above image), the interface displays the encrypted dialogue. Your questions are encrypted locally by the browser before being dispatched to the LLM, making it impossible for others to discern the content of your query. Similarly, the LLM’s responses are encrypted, ensuring only you can access the content of the answers. These processes occur locally in your browser and are automated. Without compromising the user experience, the SecretFlowCloud Large Language Model Showroom permits a similar experience to other LLM Q&A services, while significantly upholding user privacy.


You're welcome to explore more functionalities of the SecretFlowCloud Privacy-Preserving LLM Platform.

5 Summary

In addressing the privacy protection concerns of LLM interactions and the security of LLMs and datasets during inference, fine-tuning, and pre-training, SecretFlow, in assistance with Intel and the integrity and confidentiality protection of Intel® TDX, has launched the SecretFlowCloud Privacy-Preserving LLM Platform. This platform ensures the safety of data and LLMs under a one-stop-shop scheme, providing full lifecycle encryption-enabled functionalities such as inference, fine-tuning, and pre-training, without the need for clients to switch platforms. This flexibility allows clients to efficiently address the privacy protection needs of the LLM era.

In the future, SecretFlowCloud plans to introduce more confidential LLM capabilities, such as Confidential RAG (Retrieval-Augmented Generation), confidential applications of large models, and more. For instance, in comparison to existing RAG implementations, the SecretFlowCloud Privacy-Preserving LLM Platform will place greater emphasis on the security of knowledge throughout the process, including the parsing, storage, and retrieval of knowledge, to ensure that it does not leak and is only used for RAG purposes. Combined with existing secure LLM inference capabilities, this approach will better protect user privacy. Stay tuned for our future updates.


Reference links



Tags (3)
About the Author
Yunge is a Xeon Customer Solution Team engineer (DCAI China). He focuses on Confidential Computing collaborations with China CSPs and support Intel customers to enable Intel SGX and TDX technologies. He is also the maintainer of Intel opensource project CCZoo: