CN117313869B - A privacy-preserving inference method for large models based on model segmentation - Google Patents

Abstract

The present invention discloses a privacy protection reasoning method for a large model based on model segmentation, and belongs to the field of computer artificial intelligence and large model security technology. Through model segmentation: the Encoder and Decoder of the original large model are deployed on the client, and the middle part of the large model is left locally on the server; model compression: the middle layer is compressed by the server and sent to the client, and a small model with the basic functions of the original large model is formed on the client; model fine-tuning: the client fine-tunes the model through the loss function; model reasoning: the client sends the trained Encoder to the server according to the protocol, and the intermediate result is obtained by reasoning, and then combined with the Decoder on the local server to complete the training. The present invention takes into account both model performance and privacy protection, effectively prevents reconstruction attacks, and has no negative impact on the effect of the model; at the same time, it prevents the model privacy leakage of the large model and the data leakage of the user, has high computational efficiency, and does not require a large amount of computing resources on the client.

Description

A privacy-preserving inference method for large models based on model segmentation

Technical Field

The present invention relates to the fields of computer artificial intelligence and large model security technology, and in particular to a large model privacy protection reasoning method based on model segmentation.

Background technique

Privacy-preserving reasoning for large models based on model segmentation has very important applications in the fields of artificial intelligence and large model security. Model segmentation technology refers to dividing a complete neural network model into two or more sub-modules, and then processing these sub-modules separately to complete different tasks. Its core idea is to make the model easier to understand, optimize and debug through modular design. Model segmentation technology originated in the 1990s, and early work mainly used simple serial module connection methods. In the 21st century, more complex tree structures and multi-branch connections were proposed. Today, artificial intelligence large models are in a stage of rapid development. Artificial intelligence large models refer to neural network models with extremely large parameter scales, usually with billions to hundreds of billions of parameters, and obtain general language or visual capabilities through pre-training on massive data. Early large models include language models such as the GPT series and BERT and vision transformer-like visual models. In the past two years, the number of parameters has exploded, and models with tens of billions of parameters have emerged, such as GPT-3 and Switch Transformer. With the further enhancement of computing power, it is expected that the number of parameters will continue to increase rapidly.

Nowadays, with the development of deep learning and large models, model segmentation technology has been widely used in natural language processing, computer vision and other fields. Model segmentation technology reduces the difficulty of training through modular design and improves the ability of the model to adapt to new tasks. It is an important technical means to achieve transfer learning. With the continuous development of transfer learning technology, the emerging Offsite-Tuning technology adopts a privacy-preserving method, and the data owner does not need to share his sensitive data with the model owner. Traditional transfer learning methods may require data owners to share their data and pay expensive fees so that the model owner can fully fine-tune, while Offsite-Tuning reduces the need to share data by sending lightweight adapters and simulators to data owners, thereby reducing costs. For large base models, Offsite-Tuning is more computationally efficient. The data owner only needs to fine-tune the adapter locally without accessing the full model weights, so a lot of computing time and resources can be saved.

However, transfer learning also has some security issues, such as adversarial sample attacks on the source model or target model, model leakage through model extraction, and the risk of privacy leakage of source and target domain sample data. Therefore, the present invention proposes a large model privacy-preserving reasoning method based on model segmentation to solve the above problems.

Summary of the invention

The purpose of the present invention is to provide a large model privacy protection reasoning method based on model segmentation, which takes into account both model performance and privacy protection, effectively prevents reconstruction attacks, and has no negative impact on the effect of the model; at the same time, the model segmentation technology prevents the model privacy leakage of the large model and the user's data leakage, has high computational efficiency, and does not require a large amount of computing resources on the client.

To achieve the above object, the present invention adopts the following technical solutions:

A large model privacy-preserving reasoning method based on model segmentation includes the following steps:

S1. Model segmentation: The first n layers of encoders and the last n layers of decoders of the original large model are deployed on the client, and the middle part of the large model is kept locally on the server.

S2, model compression: The server compresses the intermediate layer and sends it to the client, and forms a small model with the basic functions of the original large model on the client;

S3, model fine-tuning: The client fine-tunes the model through the loss function;

S4, model reasoning: The client sends the trained first n layers of Encoder to the server according to the protocol, and then performs reasoning to obtain the intermediate results, and then combines them with the last n layers of Decoder on the local server to complete the training and obtain the final result.

Preferably, in step S1, the original data of the client is used to train n model layers without leaving the local machine, and the trained Encoder and Decoder are obtained by completing the training of the first n layers and the last n layers of the large model on the client.

Preferably, in step S2, the remaining intermediate layers of step S1 are compressed to form a simulator module for providing approximate gradient directions during the adaptation process. The simulator module is sent from the server to the client, and is combined with the Encoder and Decoder deployed on the client and the original large model to form a complete small model.

Preferably, in step S3, the small model obtained in step S2 is fine-tuned by a loss function, and the loss function L is:

Among them, _L1 is the loss function of the original large model task, _L2 is the cosine distance of _f2 , _f'2 , and f'n _-2 , _f2 is the intermediate feature, and _f'2 is the intermediate feature between fn _-2 and the original large model.

Preferably, in step S4, the Encoder is combined with the server after fine-tuning the task on the client, and is jointly trained with the rest of the large model to coordinate the model parameters between the server and the client. Finally, the server sends the trained intermediate results back to the client and combines them with the trained Encoder to refine the model and obtain the final output.

Therefore, the present invention adopts the above-mentioned large model privacy protection reasoning method based on model segmentation to achieve the following beneficial effects:

1. The present invention can protect both the model privacy on the server side and the data privacy on the user side by deploying part of the large model locally on the client and performing training.

2. By using the fine-tuning method of model compression technology locally on the client, the middle part of the large model is compressed into a simulator to assist the client in fine-tuning the encoder and decoder locally, while balancing the loss function of model performance and privacy protection, ensuring that the performance of the fine-tuning model does not decrease while preventing reconstruction attacks.

The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of the present invention.

Detailed ways

The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. The words "include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, but do not exclude other elements or objects. The terms "set", "install" and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two elements. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

Example

As shown in FIG1 , the present invention provides a large model privacy protection reasoning method based on model segmentation, comprising the following steps:

1. (Step S1) Model segmentation: The first n layers of encoders and the last n layers of decoders of the original large model are deployed on the client, and the middle part of the large model is left locally on the server.

The original large model includes the Transformer model, Seq2Seq model, etc.; the first n layers are set to w ₁ , w ₁ , etc.; the last n layers are set to w _n-1 , w _n , etc.

By completing the training of the first n layers and the last n layers on the client, we can obtain the trained Encoder and Decoder. This strategy ensures that the client data is invisible to the server during the training process, and the client's original data does not leave the local computer, but is only used to train these n key model layers. This not only protects sensitive data from being exposed during network transmission, but also reduces the risk of data leakage. Only the first n layers and the last n layers need to be trained on the client. Compared with the entire large model, the required computing resources are greatly reduced. Even on devices with limited resources, training can be easily performed without the need for a large computing cluster. This effective way of utilizing computing resources reduces the cost and complexity of the training process.

2. (Step S2) Model compression: The intermediate layer is compressed by the server and sent to the client, and a small model with the basic functions of the original large model is formed on the client.

1) The server compresses the remaining intermediate layer R of step S1 to form a simulator module E, which is used to provide approximate gradient directions during the adaptation process. This part contains the main functional information of the original model and is a fixed and untrainable part.

2) The simulator module construction process occurs on the server side, where the intermediate layer is processed by a carefully designed compression algorithm to ensure that the size of the simulator is minimized while retaining key model functional characteristics. This compression process is lossy and aims to preserve the main information of the model while reducing the size of the simulator module so that it can be efficiently transmitted on the network.

3) The simulator module obtained above is sent from the server to the client, where it is combined with the encoder and decoder deployed by the original large model to form a complete small model. This small model has sufficient performance and functions to complete specific tasks, and is also assisted by the simulator module. With the assistance of the small model, users can adjust the encoder and decoder using their own data.

3. (Step S3) Model fine-tuning: The client fine-tunes the model through the loss function.

The small model obtained in step S2 is fine-tuned using the loss function. The specific formula is:

Among them, _L1 is the loss function of the original large model task, _L2 is the cosine distance of _f2 , _f'2 , and f'n _-2 , _f2 is the intermediate feature, and _f'2 is the intermediate feature between fn _-2 and the original large model.

1) The loss function _L1 of the original large model task is usually determined according to the nature of the specific task. It can be cross entropy loss, maximum likelihood loss or other loss functions suitable for the task. The role of _L1 is to ensure that the fine-tuned small model maintains high performance on the original task, that is, the performance of the model will not decrease. This part of the loss function ensures the effectiveness of the model on the task.

2) The cosine distance _L2 measures the similarity of the model in the intermediate feature space. By minimizing the _L2 loss, reconstruction attacks (i.e., attackers try to reconstruct the original data based on intermediate features) can be prevented. The introduction of _L2 loss enhances the security of the model and ensures that the intermediate representation of the model is not easy to leak sensitive information.

4. (Step S4) Model inference: The client sends the trained first n layers of Encoder to the server according to the protocol, and then performs inference to obtain intermediate results, and then combines them with the last n layers of Decoder on the local server to complete the training and obtain the final result.

The client sends the fine-tuned trained encoder to the server, and the encoder is fine-tuned to adapt to specific application scenarios or task requirements; the server will combine with this encoder and conduct joint training with the rest of the large model. This process ensures that the model parameters between the server and the client are coordinated and consistent, thereby maintaining the performance and functionality of the model to the greatest extent.

The server sends the intermediate results after training back to the client. These intermediate results include the server's further training results of the model and possible performance improvements. The client combines these intermediate results with its own trained encoder to further refine the model and obtain the final output. Step S4 is the last part of the entire process, which aims to ensure that the model reaches the best performance level through the collaboration between the server and the client.

Therefore, the present invention adopts a large model privacy protection reasoning method based on model segmentation, which takes into account both model performance and privacy protection, effectively prevents reconstruction attacks, and has no negative impact on the model effect; at the same time, model segmentation technology prevents the model privacy leakage of large models and the data leakage of users, has high computational efficiency, and does not require a large amount of computing resources on the client.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims (3)

1. The large model privacy protection reasoning method based on model segmentation is characterized by comprising the following steps of:

s1, model segmentation: deploying the front n layers of encoders and the last n layers of encoders of the original large model at the client, and leaving the middle part of the large model at the server;

s2, model compression: the middle layer is compressed through the server side and then sent to the client side, and a small model with the basic function of the original large model is formed at the client side;

s3, fine adjustment of a model: the client side carries out fine adjustment on the small model obtained in the step S2 through a loss function;

s4, model reasoning: the client sends the trained front n layers of encodings to the server according to the protocol, then performs reasoning to obtain an intermediate result, and then completes training by combining with the last n layers of encodings of the client to obtain a final result;

in step S2, the rest intermediate layer in step S1 is compressed to form a simulator module for providing approximate gradient direction in the adapting process, the simulator module is sent to the client side from the server side, and the client side is combined with the Encoder and the Decode deployed by the original large model to form a complete small model.

2. The large model privacy preserving reasoning method based on model segmentation as set forth in claim 1, wherein: in step S1, the original data of the client is used for training 2n model layers without leaving the local area, and the trained Encoder and the Encoder are obtained by training the n layers before and the last n layers before the client completes the large model.

3. The large model privacy preserving reasoning method based on model segmentation as set forth in claim 2, wherein: in step S4, the Encoder is combined with the server after the client performs task fine tuning, and is combined with the rest of the large model for training, model parameters between the server and the client are coordinated and consistent, and finally the server sends a trained intermediate result back to the client and is combined with the last n layers of decoders for training, so that a final result is obtained.