CN117010484A - Personalized federal learning generalization method, device and application based on attention mechanism - Google Patents

Abstract

The application relates to a personalized federal learning generalization method, equipment and application based on an attention mechanism, which comprises the following steps: initializing the sharing parameters of the global model, sending the sharing parameters to a client which is pre-connected, receiving the sharing parameters and the personalized parameters of each client after local training, and updating the sharing parameters of the server based on the sharing parameters of each client; and sending the personalized parameters of the existing client and the shared parameters of the server to an untrained new client, and generating the personalized parameters at the new client by using the super network based on the attention mechanism. The new client trains with local data to update the super network parameters instead of the local model parameters. The sharing parameter part is unchanged, and personalized parameters of the new client are generated through super network learning. When the super network of the new client is constructed, the super network refers to the personalized parameters of each model at the same time so as to introduce the correlation information of the personalized parameters of the client and promote the final effect.

Description

Personalized federated learning generalization methods, equipment and applications based on attention mechanism

Technical field

The invention relates to the field of artificial intelligence technology, and in particular to a personalized federated learning generalization method, equipment and application based on an attention mechanism.

Background technique

Federated learning trains a common model by sharing parameters or gradients trained by each client's data on the premise of "data islands" (that is, data between clients does not communicate with each other and is not uploaded to the server) to protect the client's data privacy. Personalized federated learning is a commonly used federated learning method. The purpose is to retain personalized model parameters according to the different data distribution of each client and adapt to the data distribution of this client to improve the effect of the local model.

Personalized federated learning involves an important issue, that is, how to ensure the generalization of the model. Specifically, when a new client is added, especially a client with less trainable data, the effect of the new client is often difficult to guarantee. The reason is that when there is less data, the local model directly trains the overall parameters, which is prone to overfitting and reduces the model effect.

Chinese Patent Publication No. CN115600686A discloses a federated learning system based on a personalized Transformer. This application sets up a super network on the server and assigns randomly initialized embedding vectors to newly added clients, and then uses local data to train the personality of the new clients. model. However, randomly initialized trainable embedding vectors are not easy to converge. In addition, the model structure of each client lacks flexibility and is only suitable for local models with attention layers such as transformers.

Contents of the invention

The purpose of the present invention is to provide a personalized federated learning generalization method, equipment and application based on the attention mechanism in order to overcome the shortcomings of the above-mentioned existing technologies, improve the convergence of new clients and improve the training effect by easing over-fitting. .

The object of the present invention can be achieved through the following technical solutions:

One aspect of the present invention provides a personalized federated learning generalization method based on the attention mechanism, which is applied to the server and includes the following steps:

Initialize the shared parameters of the global model and send them to at least one client that has established a connection in advance, receive and store the shared parameters and personalized parameters of each client after local training, and update the shared parameters of the server based on the shared parameters of each client, multiple times Execute this step until the termination condition is reached;

Send the personalized parameters of each existing client and the shared parameters of the server to the untrained new client, use the attention mechanism-based super network to generate personalized parameters on the new client, and train the super network based on the local data of the new client. Network, complete the local update of the new client's super network personalization parameters.

As a preferred technical solution, the termination condition is that the communication round reaches a preset value.

As a preferred technical solution, the input of the super network is the personalized parameters of each existing client, and the output is the personalized parameters of the new client.

As a preferred technical solution, the attention mechanism-based super network includes:

Fully connected layer, used to generate hidden vectors;

Multiple normalization layers and multiple self-attention layers set between the normalization layers are used to generate personalized parameters of new clients based on latent vectors.

As a preferred technical solution, the shared parameters of the new client adopt the shared parameters of the server.

As a preferred technical solution, the following steps are also included:

Receive shared parameters and personalized parameters from multiple clients, including new clients after parameter initialization, and weightedly update the shared parameters of the server based on the shared parameters of each client.

As a preferred technical solution, based on the shared parameters of each client, the shared parameters of the server are updated through weighted aggregation.

Another aspect of the present invention provides a personalized federated learning generalization method based on the attention mechanism, which is applied to new untrained clients, including the following steps:

Receive personalized parameters from multiple clients that have been trained locally, as well as shared parameters from the server that have been globally summed;

Use local data training to update the parameters of the super network based on the attention mechanism. Based on the personalized parameters of multiple clients that have been locally trained, use the trained super network to generate personalized parameters of the new client, and add the server to the global The shared parameters of and are used as shared parameters of the new client;

Upload the updated personalized parameters and shared parameters to the server.

Another aspect of the present invention provides an electronic device, including: one or more processors and a memory, one or more programs are stored in the memory, and the one or more programs include a program for executing the above-mentioned based on Instructions for generalization methods for personalized federated learning of attention mechanisms.

Another aspect of the present invention provides the application of the above personalized federated learning generalization method based on the attention mechanism. For the Internet of Vehicles including a server and at least one vehicle terminal, the personalized federated learning generalization method is applied to all The server is deployed with a global model, and the vehicle-mounted terminal is deployed with a local model. The local model includes shared parameters and personalized parameters. The vehicle-mounted terminal also includes a device for generating the vehicle-mounted model when joining the Internet of Vehicles. Hypernetworks with personalized parameters.

Compared with the prior art, the present invention has the following advantages:

(1) Improve the convergence of new client training and improve the training effect: Compared with the solution of using an ordinary global average model to initialize the new client model and then directly perform local training, the present invention uses a super network based on the attention mechanism to generate new The client's personalized parameters can not only ensure the rapid convergence of the new client model, but also avoid overfitting due to lack of data in local training, and retain the generalization ability of the global model due to its wide coverage of data. Different from the existing solution of allocating embedding vectors to each client for training, the super network training input of the present invention is the personalized parameter of each trained client, which is easy to converge.

(2) Suitable for scenarios where there are multiple client model structures, with strong applicability: Unlike some existing solutions that restrict clients to adopt a certain network structure, the local model structure of each client in the present invention is not limited. , for example, it can be either CNN, transformer, or other structures. The personalized layer in the network structure serves as the output of the super network, so the client's local training can be more flexible and not restricted by computing power conditions. , In addition, the super network of this application is located on the client instead of the server. You can flexibly choose whether to use the super network according to the local situation of the client.

Description of the drawings

Figure 1 is a flow chart of the federated learning generalization method applied to the server in the embodiment;

Figure 2 is a schematic structural diagram of the hypernetwork in the embodiment;

Figure 3 is a flow chart of the federated learning generalization method applied to new clients in the embodiment;

Figure 4 is a flow chart of the parameter update process of an existing client in the embodiment;

FIG. 5 is a schematic structural diagram of the electronic device in the embodiment.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of protection of the present invention.

It should be noted that, as long as there is no conflict, the features in the following embodiments and implementation modes can be combined with each other.

Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in a process or processes in a flowchart and/or a block or blocks in a block diagram.

It should also be noted that the terms "comprises," "comprises," or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements not only includes those elements, but also includes Other elements are not expressly listed or are inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or device that includes the stated element.

Example 1

In order to solve or partially solve the problem in the existing technology that when a new client is added to federated learning, the model effect of the new client is difficult to guarantee, this embodiment provides a personalized federated learning generalization method based on the attention mechanism to Applied on the server side. This method is based on the attention mechanism of model weight similarity, and uses a super network based on the attention mechanism of model parameter correlation on the new client to aggregate and train multiple models of the original clients to obtain the model parameters of the new client.

In this embodiment, there are N clients, 1 server, and K communication rounds.

Referring to Figure 1, this method includes the following steps:

S1, randomly initializes the shared parameters of the global model of the server and client’s personalized parameters{/> , …/> };

S2, the server sends initialization parameters to each client;

S3, the client receives and updates shared parameters , perform local parameters based on local data (including/> and/> ), get{/> ,/> …/> } and {/> ,/> …/> };

S4, the client will update the local parameters { ,/> …/> } and {/> ,/> …/> }Upload to server;

S5, the server receives the parameters uploaded by each client, and compares { based on the amount of training data of each client. ,/> … } Perform weighted aggregation to obtain new /> . Jump to step S2 until the number of cycles reaches the preset communication round K;

S6, add new client Participate in training and transfer the shared parameters stored in the server/> and personalized parameters{ ,/> …/> }Transfer to new client;

S7, build a super network based on the attention mechanism on the new client to generate local model parameters, use local data to train the super network, and obtain the parameters of the local model;

S8: Transmit the parameters of the local model to the server. The server receives the model parameters of each client and performs weighted aggregation according to the amount of training data of each client.

See Figure 2 for a schematic structural diagram of a supernetwork based on the attention mechanism. The input of the model is the personalized parameters of the existing client { ,/> …/> }, the output is the new client/> Personalized parameters/> . The model includes a fully connected layer, a normalization layer 1, a self-attention layer 1, a normalization layer 2, a self-attention layer 2, and a normalization layer 3 that are connected in sequence. The fully connected layer is used to generate multiple latent vectors matching the number of existing clients based on the personalized parameters of existing clients. It should be emphasized that the type and number of layers in this embodiment can be changed. For example, multiple sets of normalization layers and self-attention layer structures can be used.

See Figure 4 for the parameter update process of an existing client, which includes the following steps:

S1, receives and updates shared parameters ;

S2, perform local parameters (including and/> ), get{/> ,/> … } and {/> ,/> …/> };

S3, the client will update the local parameters { ,/> …/> } and {/> ,/> …/> }Upload to server.

This method considers the relationship between clients, specifically introducing an attention mechanism. The input of the same super network is the personalized parameters of multiple original clients to generate personalized parameters of new clients.

In order to illustrate the advantages of this method, a federated learning server-side update algorithm is provided below as a comparison example, which specifically includes the following steps:

Step1, randomly initialize the parameters of the global model ;

Step2, send global model parameters to each client;

Step 3: The client receives global parameters and updates local parameters;

Step 4: The server receives the parameters of each client, performs weighted aggregation according to the amount of training data of each client, and jumps to step 2 until the number of cycles reaches the preset communication round K;

It can be seen that compared with using the ordinary global average model to initialize the new client model and then directly perform local training, the present invention uses a super network based on the attention mechanism to ensure the rapid convergence of the new client model and avoid local training. Among them, overfitting due to lack of data retains the generalization ability of the global model due to its extensive coverage of data. The reason is that if the initialized client model is directly trained with complete parameters, the overall client model will move towards the local optimum of the local data distribution. When there is little local data, the optimal solution will be far away The global optimal solution is very far away, thus affecting the effectiveness of the local model. However, the input of the super network is the model of other clients, so that the output model is constrained by the global training results, which can greatly improve the overfitting phenomenon and still ensure the convergence effect of the new client.

In a specific application scenario, for the Internet of Vehicles including the server and at least one vehicle terminal, the signed personalized federated learning generalization method is applied to the server. The server is deployed with a global model, and the vehicle terminal is deployed with a local model. The local model Including shared parameters and personalized parameters, the vehicle terminal also includes a super network used to generate personalized parameters when joining the Internet of Vehicles.

When the present invention constructs a super network of a new client, the super network simultaneously refers to the personalized parameters of each model, so that the correlation information of the client's personalized parameters can be introduced to improve the final effect. Unlike previous solutions, the correlation between models is not considered during the training process.

Example 2

On the basis of Embodiment 1, referring to Figure 3, this embodiment provides a personalized federated learning generalization method based on the attention mechanism to be applied to new (i.e., untrained) clients. The method includes Follow these steps:

S1, receives the personalized parameters of multiple existing client models and the shared parameters of the server through weighted aggregation

S2, update the super network parameters based on local data training to obtain local model personalized layer parameters, and use global average shared parameters for other layers. ;

S3, build a super network based on the attention mechanism, and the network input is the personalized parameter in the client model { ,/> …/> }, the output is the local model personalized layer parameters/> ;

S4, upload new parameters to the server.

In order to illustrate the advantages of this method, a federated learning client update algorithm is provided as a comparison example below, which specifically includes the following steps:

Step 31: Receive global model parameters as local model parameters, and the personalization layer retains the original parameters;

Step32, update the local model based on local data training and obtain updated local model parameters;

Step 33: Transmit the updated local parameters except the personalization layer to the server.

The present invention uses a super network based on the attention mechanism to ensure the rapid convergence of the new client model, avoids overfitting due to lack of data in local training, and retains the generalization ability of the global model due to its wide coverage of data.

Example 3

This embodiment provides an electronic device, including: one or more processors and a memory, where one or more programs are stored in the memory, and the one or more programs include: Computer program instructions for the personalized federated learning generalization method based on the attention mechanism described in Example 2.

The methods or devices described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or A combination of any of these devices.

FIG. 5 is a schematic structural diagram of an electronic device. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for services. The non-volatile memory stores instructions for executing the personalized federated learning generalization method in Embodiment 1 or 2. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it. Implement the method of data collection described in Figure 1 above. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and may also be hardware or logic device.

Memory may include non-permanent storage in computer-readable media, random access memory (RAM), and/or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer-readable media includes both persistent and non-volatile, removable and non-removable media that can be implemented by any method or technology for storage of information. Information may be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory. (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cassettes, tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

Example 4

This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, the one or more programs including: 2 Computer program instructions for the personalized federated learning generalization method based on the attention mechanism.

When using the personalized federated learning generalization method of Embodiment 1, the computer program instructions are:

S1, randomly initializes the shared parameters of the global model of the server and client’s personalized parameters{/> , …/> };

S2, the server sends initialization parameters to each client;

S3, receive the client’s updated local parameters { ,/> …/> } and {/> ,/> …/> };

S4, the server receives the parameters uploaded by each client, and compares { based on the amount of training data of each client. ,/> … } Perform weighted aggregation to obtain new /> . Jump to step S2 until the number of cycles reaches the preset communication round K;

S5, when adding a new client When participating in training, the shared parameters stored in the server/> and personalized parameters{/> ,/> …/> }Transfer to new client;

S6, build a super network based on the attention mechanism on the new client to generate local model parameters, use local data to train the super network, obtain the parameters of the local model, receive the parameters of the local model, and based on the model parameters of each client, according to each client Weighted aggregation is performed based on the amount of terminal training data.

When using the personalized federated learning generalization method of Embodiment 2, the computer program instructions are:

S1, receives the personalized parameters of multiple existing client models and the shared parameters of the server through weighted aggregation

S2, update the super network parameters based on local data training to obtain local model personalized layer parameters, and use global average shared parameters for other layers. ;

S3, build a super network based on the attention mechanism, and the network input is the personalized parameter in the client model { ,/> …/> }, the output is the local model personalized layer parameters/> ;

S4, upload new parameters to the server.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes in the flowchart and/or in a block or blocks in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

Example 5

This embodiment provides a personalized federated learning generalization system based on the attention mechanism, including N clients and 1 server.

Among them, the server is used to perform the following processes:

S1, randomly initializes the shared parameters of the global model of the server and client’s personalized parameters{/> , …/> };

S2, the server sends initialization parameters to each client;

S3, receive the client’s updated local parameters { ,/> …/> } and {/> ,/> …/> };

S4, the server receives the parameters uploaded by each client, and compares { based on the amount of training data of each client. ,/> … } Perform weighted aggregation to obtain new /> . Jump to step S2 until the number of cycles reaches the preset communication round K;

When a new client joins the system The server is also used to execute:

S5, when adding a new client When participating in training, the shared parameters stored in the server/> and personalized parameters{/> ,/> …/> }Transfer to new client;

S6, build a super network based on the attention mechanism on the new client to generate local model parameters, use local data to train the super network, obtain the parameters of the local model, receive the parameters of the local model, and based on the model parameters of each client, according to each client Weighted aggregation is performed based on the amount of terminal training data.

The client is used to perform the following processes:

S1, receives the personalized parameters of multiple existing client models and the shared parameters of the server through weighted aggregation

S2, update the super network parameters based on local data training to obtain local model personalized layer parameters, and use global average shared parameters for other layers. ;

S3, build a super network based on the attention mechanism, and the network input is the personalized parameter in the client model { ,/> …/> }, the output is the local model personalized layer parameters/> ;

S4, upload new parameters to the server.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent methods within the technical scope disclosed in the present invention. Modifications or substitutions shall be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. A personalized federated learning generalization method based on the attention mechanism, which is characterized in that it is applied to the server and includes the following steps: Initialize the shared parameters of the global model and send them to at least one client that has established a connection in advance, receive and store the shared parameters and personalized parameters of each client after local training, and update the shared parameters of the server based on the shared parameters of each client, multiple times Execute this step until the termination condition is reached; Send the personalized parameters of each existing client and the shared parameters of the server to the untrained new client, use the attention mechanism-based super network to generate personalized parameters on the new client, and train the super network based on the local data of the new client. Network, complete the local update of the new client's super network personalization parameters. 2. A personalized federated learning generalization method based on attention mechanism according to claim 1, characterized in that the termination condition is that the communication round reaches a preset value. 3. A personalized federated learning generalization method based on attention mechanism according to claim 1, characterized in that the input of the super network is the personalized parameters of each existing client, and the output is the personalized parameter of the new client. Personalization parameters. 4. A personalized federated learning generalization method based on an attention mechanism according to claim 1, characterized in that the super network based on the attention mechanism includes: Fully connected layer, used to generate hidden vectors; Multiple normalization layers and multiple self-attention layers set between the normalization layers are used to generate personalized parameters of new clients based on latent vectors. 5. A personalized federated learning generalization method based on attention mechanism according to claim 1, characterized in that the shared parameters of the new client adopt the shared parameters of the server. 6. A personalized federated learning generalization method based on an attention mechanism according to claim 1, characterized in that it further includes the following steps: Receive shared parameters and personalized parameters from multiple clients, including new clients after parameter initialization, and weightedly update the shared parameters of the server based on the shared parameters of each client. 7. A personalized federated learning generalization method based on the attention mechanism according to claim 1, characterized in that based on the shared parameters of each client, the shared parameters of the server are updated through weighted aggregation. 8. A personalized federated learning generalization method based on the attention mechanism, which is characterized in that it is applied to new untrained clients and includes the following steps: Receive personalized parameters from multiple clients that have been trained locally, as well as shared parameters from the server that have been globally summed; Use local data training to update the parameters of the super network based on the attention mechanism. Based on the personalized parameters of multiple clients that have been locally trained, use the trained super network to generate personalized parameters of the new client, and add the server to the global The shared parameters of and are used as shared parameters of the new client; Upload the updated personalized parameters and shared parameters to the server. 9. An application of the personalized federated learning generalization method based on the attention mechanism according to any one of claims 1 to 8, characterized in that, for the Internet of Vehicles including a server and at least one vehicle-mounted terminal, the personalized The federated learning generalization method is applied to the server. The server is deployed with a global model. The vehicle-mounted terminal is deployed with a local model. The local model includes shared parameters and personalized parameters. The vehicle-mounted terminal also includes: The super network of personalized parameters is generated when joining the Internet of Vehicles. 10. An electronic device, characterized in that it includes: one or more processors and a memory, one or more programs are stored in the memory, and the one or more programs include a method for executing the method of claim 1- 8. Instructions for any of the personalized federated learning generalization methods based on the attention mechanism.