CN114358286A - Mobile equipment federal learning method and system - Google Patents
Mobile equipment federal learning method and system Download PDFInfo
- Publication number
- CN114358286A CN114358286A CN202210218303.5A CN202210218303A CN114358286A CN 114358286 A CN114358286 A CN 114358286A CN 202210218303 A CN202210218303 A CN 202210218303A CN 114358286 A CN114358286 A CN 114358286A
- Authority
- CN
- China
- Prior art keywords
- training
- server
- client
- data
- global model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a mobile device federal learning method and a system, wherein the system comprises the following steps: FL server, intermediary and customer end; the method comprises the following steps: the mobile equipment is added into a federal learning training task; the FL server initializes a global model and collects local data distribution of participants; rebalancing the federated learning training task by means of data enhancement and rescheduling; sending the balanced federal learning task to a lower-level client, and training a global model by using a red fox algorithm by the client; aggregating and feeding back the trained global model to obtain a latest global model; and performing federal learning by using the latest global model so as to improve the efficiency of federal learning of the mobile equipment. The invention alleviates the global unbalance problem of the training data and improves the accuracy of the model by the steps of data enhancement and rescheduling.
Description
Technical Field
The invention relates to the field of communication, in particular to a mobile device federal learning method and a mobile device federal learning system.
Background
In practice, computing time is the bottleneck for federal learning. To control the computation time of a device, its amount of data used for training may be controlled based on lower and upper limits, which may create more data imbalance conditions in the system that may reduce the efficiency of federal learning.
Meanwhile, the data distribution on the mobile device is also unbalanced, and the accuracy of federal learning on the unbalanced distributed data set is lower than that of a balanced data set, so that the training result of the model is deviated.
In conventional machine learning systems, where the algorithm runs on a large data set that is evenly partitioned across multiple servers in the cloud, Federated Learning (FL) is typically trained from a large, non-independent, co-distributed and unbalanced data set that is generated by different UEs (user equipments) into different distributions. It is noted that the following situation may occur during the parameter update phase: while the iterative algorithm running on the FL requires very low latency and high throughput connections between computational units, the AP (access point) typically needs to transmit the weights after training over unreliable channels over a limited spectrum and a limited number of UEs for global aggregation. These factors make the problems of slow operation and fault tolerance of the FL more prominent than non-distributed machine learning training. To successfully deploy FL, new tools and new thinking are needed for model development and training, and communication costs become an important limiting factor due to the inability to directly access the raw data.
The existing achievements at present are studied on algorithm and communication respectively. In terms of algorithm, mainly reducing the overhead of the update uploading phase, making the model training more efficient, wherein typical methods include reducing the communication bandwidth by updating only UEs with significant training improvement, compressing gradient vectors by quantization, or using momentum methods in sparse update to speed up the training process, however these methods neglect the unique properties of the wireless channel, and have room for further improvement.
While in terms of communication, the related solutions can be roughly divided into two categories: the first category greatly reduces the communication overhead by enlarging the communication interval, but this approach reduces the final accuracy of the model and makes it difficult to obtain the optimal communication step size. The second type is data compression, i.e., compressing data prior to transmission. However, the compression process can be time consuming, especially when joint learning is performed on battery-sensitive (e.g., smart phones) or low-performance (e.g., netbook, gateway) devices.
Disclosure of Invention
In view of the above problems, an object of the present invention is to provide a method and a system for federal learning of a mobile device, which alleviate the global imbalance problem of training data and improve the accuracy of a model by performing the steps of data enhancement and rescheduling.
The invention provides a method suitable for remote upgrade of a lithium battery in a first aspect, which comprises the following steps:
s102: the mobile equipment is added into a federal learning training task;
s104: the FL server initializes a global model and collects local data distribution of participants;
s106: rebalancing the federated learning training task by means of data enhancement and rescheduling;
s108: sending the balanced federal learning task to a lower-level client, and training a global model by using a red fox algorithm by the client;
s110: aggregating and feeding back the trained global model to obtain a latest global model;
s112: and performing federal learning by using the latest global model so as to improve the efficiency of federal learning of the mobile equipment.
In this scheme, S104 specifically is:
in an initialization stage, the FL server firstly waits for the mobile equipment to join in an FL model training task;
the mobile device participates in training by sending its local data distribution information to the FL server;
after determining the clients to participate in the training, the FL server initializes the weights and optimizers of the neural network model and collects the local data distribution of the participants.
In this embodiment, the data enhancement process in S106 is as follows:
initializing a downsampling threshold τdAnd enhancing the threshold τaHere is provided withτaIs taudIf z-score of the data size of a class is greater than τ d, then the class is considered as a majority class, denoted Yd;
if the z-score of the data size of a class is smaller than tau a, the class is regarded as a few classes and is denoted as Ya;
when in use
When the temperature of the water is higher than the set temperature,
incorporate the set into Yd
When in use
When the temperature of the water is higher than the set temperature,
the set is incorporated into Ya
Wherein the content of the first and second substances,
for the size of the data set to be,
is the standard deviation of the data set and,
is the mean of the data set and,
z-score for the dataset;
the process is completed for the processes of Yd, Ya,
after the calculation, the server sends the parameters to the client;
then, all clients perform data expansion and downsampling in parallel;
the client operates on each sample in the dataset: if the source of the sample is in the set Yd, then determine if the amount of the source set y has been reduced to the original amount
If the sampling is satisfied, the sampling is reserved, and if the sampling is not satisfied, the sampling is returned to be empty; if the source of the sample is in the set Ya, the sample is enhanced, including operations of random shift, random rotation, random shearing and random scaling, and the size of the enhanced part is the original sample size
-1；
Once all clients have completed the enhancement and downsampling, the FL server creates an intermediary to reschedule the clients to achieve partial balancing.
In this solution, the rescheduling process in S106 is as follows:
the broker traverses the data distribution of all the unallocated clients and selects the data distribution to make the data distribution of the broker closest to the uniformly distributed clients; on the basis, a ZOA algorithm is executed, so that each medium is guaranteed to be utilized to the maximum extent;
when the broker reaches the maximum allocated client limit, the FL server will create a new broker and repeat the above process until the training task can be rescheduled for all clients.
In the scheme, the ZOA algorithm is as follows:
firstly, setting a minimum heap for a ZOA algorithm to store the cost of adding tasks to each resource;
then it initializes the lower limit of tasks allocated to resources, and if they can receive more tasks, the cost of allocating the next task to the heap increases; its main loop receives a new task by taking one of the resources with the lowest cost, allocating additional tasks to it, making the best allocation of one task at a time, updating its cost on the heap if it can still receive more tasks; after all iterations of the main loop, all tasks will be allocated to the existing resources and the ZOA algorithm is complete.
In this embodiment, the step S108 specifically includes:
when each round of communication starts, each intermediary sends a task to a lower-level client, each client uses the red fox algorithm to train for E times, executes anti-virus-throwing operation judgment of the red fox algorithm, updates the global model, and then returns the updated global model to the corresponding intermediary;
the intermediary receives the updated result and sends the updated result to the next waiting training client;
all clients under intermediary management complete one-time receiving, training and feedback; this process is cycled Em times;
finally, all intermediaries send updates of the global model to the FL server.
In the scheme, the anti-poison throwing operation judgment of the red fox algorithm specifically comprises the following steps:
firstly, selecting the number K of federal learning, defining the number of clients as m, defining the number of heuristic iterations performed on a server as Tserver and the number Ttrain of training times of the clients, and preparing databases of all the clients;
the server starts the operation of the initial global model by creating the initial global model and sends the initial global model to all the clients;
after the global model is distributed, there is a time for waiting for a result, if the number of training iterations is greater than 10, after each 10 iterations, the information of the current training state is sent to the server as information;
if the information of the previous stage of the client does not exist, the server stores the information;
in the opposite case, when this information is obtained and there is preview data, the actual loss function value and the number of poison samples are calculated;
if f2 is above 10% of the population, stopping training operations for the client and flagging as likely poisoned;
when this value of f2 is below 5% of all samples and the value of f1 is greater than 0, the current model is modified using meta-heuristics;
when the number of poisoned samples rapidly increases, the training is interrupted; this behavior implies the possibility of poisoning, which requires the training process to be interrupted in order not to significantly affect the model;
this can be expressed as:
executing a red fox algorithm when f1 is more than or equal to 0 ^ f2 and less than or equal to 0.05 marked with | D |;
when f2 is more than or equal to 0.1, marked with | D |, the training process is directly interrupted;
f1(L1,L2)=L1−L2；
f2(m1,m2)=m1−m2；
wherein L1 and L2 are values of loss functions of two adjacent iterations, m1 and m2 are numbers of samples poisoned by the two adjacent iterations, and | D | is the size of the database.
In this scheme, S110 specifically includes:
after the FL server receives the updated model sent by the intermediary, the model weight is compensated by using Taylor series expansion and a Fisher information matrix, then the FL server calculates the nererov quantity and updates the global model by using the compensated weight, and then pushes the latest global model to the intermediary.
A second aspect of the present invention provides a mobile device federal learning system, the system comprising: FL server, intermediary and customer end;
the FL server is responsible for maintaining a global model w0, deploying the model to the intermediaries, and synchronously extracting aggregation update delta wi from each intermediary by using a federal average algorithm;
the client is a mobile phone or an internet of things device for maintaining a local training data set;
according to the characteristics of data distribution, clients can be divided into three categories:
unifying the client: having training data sufficiently balanced and ready to run the FL application;
and (3) small client side: the data volume is relatively small, and the training process is difficult to participate;
deflecting the client: having enough training data, but biased towards retaining certain classes of data, resulting in local imbalances;
the intermediary has two functions: firstly, the training process for three types of clients is rearranged; secondly, the intermediary also needs to make the collected data distribution nearly uniform; the second is a unified client with sufficiently balanced training data and ready to run the FL application;
in this scenario, the system may perform the following operations:
first, the FL server initializes the global model to start training;
then, the FL server starts a new round of communication r and sends the global model to the intermediary;
next, each moderator m coordinates the training of the designated clients and calculates the update of the weights
；
Finally, the FL server collects all intermediary updates, aggregates these updates with weights
Wherein, in the step (A),
is the cumulative training limit assigned to the broker's client, n = ∑ Σ
；
The server then updates the global model to
And ends the round.
Is the starting model for the next round, i.e., round n +1 communication.
The invention discloses a method and a system for federal learning of mobile equipment, which relieve the global unbalance problem of training data and improve the accuracy of a model by seventeen percent through the steps of data enhancement and rescheduling.
The invention uses ZOA polynomial time algorithm in the task allocation part, and compared with the traditional task scheduling algorithm Fed-LBAP and the like, the ZOA can obviously reduce the time required by operation under different conditions, especially when the task quantity is large. The ZOA may be easily adjusted to maximize the number of tasks allocated to the resource.
The method utilizes the red fox algorithm to replace the traditional training method, and simultaneously, the mechanism can select different meta-heuristic algorithms according to different application scenes and characteristics, so that the method has stronger universality.
The invention provides a mechanism for cleaning data sets under the condition of data set poisoning, and compared with the traditional federal learning method, the mechanism can improve the accuracy by at least 10%.
The communication processing mode of the invention adopts the ZanOverlap method, and the communication and the calculation are parallel, so that the federal learning communication efficiency can be improved, and the invention is completely compatible with a plurality of other data compression methods and has strong usability.
The method solves the problem of dependence between training iterations by relaxing chain constraint and a data compensation mechanism, thereby relieving the problem of time consumption of gradient update caused by parallelism in ZanOverlap, and keeping the convergence speed of federal learning by a parameter optimization mode.
The system of the invention divides the client into three types according to the characteristics of data distribution, thereby fully utilizing the characteristics of different clients to train and further operate.
Drawings
FIG. 1 is a flow chart illustrating a method for remote upgrading of a lithium battery according to the present application;
fig. 2 shows a block diagram of a system suitable for remote upgrade of a lithium battery according to the present application.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
Fig. 1 shows a flowchart of a method suitable for remote upgrade of a lithium battery according to the present application.
As shown in fig. 1, the present invention provides a method for remote upgrade of a lithium battery, comprising the following steps:
s102: the mobile equipment is added into a federal learning training task;
s104: the FL server initializes a global model and collects local data distribution of participants;
s106: rebalancing the federated learning training task by means of data enhancement and rescheduling;
s108: sending the balanced federal learning task to a lower-level client, and training a global model by using a red fox algorithm by the client;
s110: aggregating and feeding back the trained global model to obtain a latest global model;
s112: and performing federal learning by using the latest global model so as to improve the efficiency of federal learning of the mobile equipment.
According to the embodiment of the present invention, S104 specifically is:
in an initialization stage, the FL server firstly waits for the mobile equipment to join in an FL model training task;
the mobile device participates in training by sending its local data distribution information to the FL server;
after determining the clients to participate in the training, the FL server initializes the weights and optimizers of the neural network model and collects the local data distribution of the participants.
According to an embodiment of the present invention, the data enhancement process in S106 is as follows:
initializing a downsampling threshold τdAnd enhancing the threshold τaWhere τ is setaIs taudIf z-score of the data size of a class is greater than τ d, then the class is considered as a majority class, denoted Yd;
if the z-score of the data size of a class is smaller than tau a, the class is regarded as a few classes and is denoted as Ya;
when in use
When the temperature of the water is higher than the set temperature,
incorporate the set into Yd
When in use
When the temperature of the water is higher than the set temperature,
the set is incorporated into Ya
Wherein the content of the first and second substances,
for the size of the data set to be,
is the standard deviation of the data set and,
is the mean of the data set and,
z-score for the dataset;
complete pairYd，Ya，
After the calculation, the server sends the parameters to the client;
then, all clients perform data expansion and downsampling in parallel;
the client operates on each sample in the dataset: if the source of the sample is in the set Yd, then determine if the amount of the source set y has been reduced to the original amount
If the sampling is satisfied, the sampling is reserved, and if the sampling is not satisfied, the sampling is returned to be empty; if the source of the sample is in the set Ya, the sample is enhanced, including operations of random shift, random rotation, random shearing and random scaling, and the size of the enhanced part is the original sample size
-1；
Once all clients have completed the enhancement and downsampling, the FL server creates an intermediary to reschedule the clients to achieve partial balancing.
According to the embodiment of the present invention, the rescheduling process in S106 is as follows:
the broker traverses the data distribution of all the unallocated clients and selects the data distribution to make the data distribution of the broker closest to the uniformly distributed clients; on the basis, a ZOA algorithm is executed, so that each medium is guaranteed to be utilized to the maximum extent;
when the broker reaches the maximum allocated client limit, the FL server will create a new broker and repeat the above process until the training task can be rescheduled for all clients.
According to the embodiment of the invention, the ZOA algorithm is as follows:
firstly, setting a minimum heap for a ZOA algorithm to store the cost of adding tasks to each resource;
then it initializes the lower limit of tasks allocated to resources, and if they can receive more tasks, the cost of allocating the next task to the heap increases; its main loop receives a new task by taking one of the resources with the lowest cost, allocating additional tasks to it, making the best allocation of one task at a time, updating its cost on the heap if it can still receive more tasks; after all iterations of the main loop, all tasks will be allocated to the existing resources and the ZOA algorithm is complete.
It should be noted that, in the ZOA algorithm, specifically:
for T ∈ N, T denotes the number of homogenous, independent, and meta-tasks, and R denotes a set of N computational resources (e.g., mobile devices). For each computation resource i ∈ R, the number of tasks it can compute has a lower and upper bound (Li ∈ N, Ui ∈ N) and its own non-decreasing cost function Ci (·): n → R ≧ 0 represents the cost for which multiple tasks are assigned. Our problem is to find a task allocation Ai e N for each resource i e R, thus minimizing the time-out Cmax, which is defined as follows:
while distributing all tasks in the resource and complying with their lower and upper bounds.
The cost functions are independent between resources, but they are all non-decreasing, in this context, we use indices i and k to represent resources and tasks, respectively.
So as to improve the efficiency of the federal learning operation in the aspect of task scheduling. The ZOA uses a dynamic programming approach to compute the optimal final allocation by iteratively finding the optimal allocation for more and more tasks. Its main idea is based on the concept of allocating the next task t +1 to a resource j that will minimize the completion time
The formula is as follows:
the algorithm first sets a minimum heap to store the cost of adding tasks to each resource. It then initializes the lower limit of tasks allocated to resources, and if they can receive more tasks, the cost of allocating the next task to the heap increases. Its main loop receives a new task by taking one of the resources with the lowest cost, allocating additional tasks to it, making the best allocation of one task at a time, updating its cost on the heap if it can still receive more tasks. After all iterations of the main loop, all tasks will be allocated to the existing resources and the algorithm is complete.
According to the embodiment of the present invention, the S108 specifically is:
when each round of communication starts, each intermediary sends a task to a lower-level client, each client uses the red fox algorithm to train for E times, executes anti-virus-throwing operation judgment of the red fox algorithm, updates the global model, and then returns the updated global model to the corresponding intermediary;
the intermediary receives the updated result and sends the updated result to the next waiting training client;
all clients under intermediary management complete one-time receiving, training and feedback; this process is cycled Em times;
finally, all intermediaries send updates of the global model to the FL server.
According to the embodiment of the invention, the anti-virus-throwing operation judgment of the red fox algorithm is specifically as follows:
firstly, selecting the number K of federal learning, defining the number of clients as m, defining the number of heuristic iterations performed on a server as Tserver and the number Ttrain of training times of the clients, and preparing databases of all the clients;
the server starts the operation of the initial global model by creating the initial global model and sends the initial global model to all the clients;
after the global model is distributed, there is a time for waiting for a result, if the number of training iterations is greater than 10, after each 10 iterations, the information of the current training state is sent to the server as information;
if the information of the previous stage of the client does not exist, the server stores the information;
in the opposite case, when this information is obtained and there is preview data, the actual loss function value and the number of poison samples are calculated;
if f2 is above 10% of the population, stopping training operations for the client and flagging as likely poisoned;
when this value of f2 is below 5% of all samples and the value of f1 is greater than 0, the current model is modified using meta-heuristics;
when the number of poisoned samples rapidly increases, the training is interrupted; this behavior implies the possibility of poisoning, which requires the training process to be interrupted in order not to significantly affect the model;
this can be expressed as:
executing a red fox algorithm when f1 is more than or equal to 0 ^ f2 and less than or equal to 0.05 marked with | D |;
when f2 is more than or equal to 0.1, marked with | D |, the training process is directly interrupted;
f1(L1,L2)=L1−L2；
f2(m1,m2)=m1−m2；
wherein L1 and L2 are values of loss functions of two adjacent iterations, m1 and m2 are numbers of samples poisoned by the two adjacent iterations, and | D | is the size of the database;
wherein f1 is an iteration loss function change difference for indicating the training direction;
the formula: f1(L1, L2) = L1-L2;
wherein L1 and L2 are the values of two adjacent double iteration loss functions, respectively.
f2 is the difference of the number of poisoning samples, which is used to indicate the possibility of database poisoning;
the formula: f2(m1, m2) = m 1-m 2;
m1 and m2 are the numbers of poisoned samples of two adjacent iterations respectively.
It should be noted that the invention uses the red fox algorithm as a training mode and establishes a corresponding anti-data poisoning mechanism.
Each client in the training process is interpreted as a specific set of elements β, L, m, | D | where β is the model used, L is the value of the loss function, m is the number of samples poisoned, and | D | is the size of the database. Upon receiving the model, the client begins training it. It is assumed that at each iteration, the server receives the information β, L, m, | D | in the form of a set, which contains information null when it receives data for the first time, but receives valid information for the second time later. In two adjacent iterations, (i.e. already β 1, L1, m1, | D | and β 2, L2, m2, | D |), the difference between the changes in the values of the loss function will indicate the potential direction of training. Moreover, differences in the number of infected samples would indicate the likelihood of database poisoning. Two fitness functions are defined using the above differences:
f1(L1,L2)=L1−L2,
f2(m1,m2)=m1−m2.
it should be noted that the red fox algorithm has the following steps:
the analytical model β i is a set of weights described as floating point numbers, and the whole model can be considered as one individual (red fox), which can be represented as a vector
Where s is the number of all weights in the model. And evaluating the individual quality through a loss function by utilizing a private database Dsever on a previous server.
Fox groups are created. Randomly selecting 5 individuals to create a fox group, and selecting the fox group with the best performance (according to a loss function) at present as the fox group
And will be
As a sixth fox, all 6 foxes then underwent distance movements to find prey. This is a global motion, which is defined as follows:
where the coefficient alpha is a scaling parameter defined within the range. When a fox is placed in a new position, it will analyze the future direction of movement of the prey. It has two options-to get close to the prey, or to leave for a better chance. This is performed by the following conditions:
if the fox chooses to leave, he does not perform a local movement. In another case, the fox will approach the target according to the following:
wherein the coefficients
Is a random value, r is the angle of view of a given fox, defined as:
where a is a random parameter.
After performing the global and possibly local movements described above, the whole fox group finds the optimal individuals on the server. This operation goes through Tserver iterations and if the best model has a small loss value, it is sent back to the staff for the rest of the training process.
According to the embodiment of the present invention, the S110 specifically is:
after the FL server receives the updated model sent by the intermediary, the model weight is compensated by using Taylor series expansion and a Fisher information matrix, then the FL server calculates the nererov quantity and updates the global model by using the compensated weight, and then pushes the latest global model to the intermediary.
It should be noted that the communication processing mode of the present invention adopts the ZanOverlap method, which can improve the communication efficiency of federal learning
ZanOverlap comprises the following steps:
1) the global model is launched on the central server and pushed to each participating client.
2) Each participating client performs local model training on respective data for E iterations.
3) When the local training is completed, the client immediately continues the next iteration of the local training, while instructing another process to push their local model to the central server.
4) When the central server receives the improved model of the client, the Taylor series expansion and the Fisher information matrix are used for compensating the weight of the model, and then the central server calculates the nerterov momentum and updates the global model by using the compensated weight.
5) The central server pushes the latest global model to the participating clients.
Compared with the traditional federal learning, the method provided by the invention is a learning framework which is optimized and improved in multiple aspects. Firstly, the fox group algorithm is used in the algorithm part, so that the optimization speed can be improved on one hand, and the data anti-virus application can be facilitated on the other hand; then, dividing the client into a server, an intermediary and a client, and further introducing a data balance mode to process data; then, a ZOA algorithm is used in the task allocation part, so that the task allocation efficiency can be improved; finally, the ZanOverlap method is adopted in the communication part, and the communication efficiency between the client and the intermediary and between the intermediary and the server is improved.
Fig. 2 illustrates a block diagram of a mobile device federal learning system of the present invention.
As shown in fig. 2, a second aspect of the present invention provides a mobile device federal learning system, where the mobile device federal learning system 2 includes: FL server 21, broker 22 and client 23;
the FL server 21 is responsible for maintaining a global model w0, deploying the model to the intermediaries, and synchronously extracting aggregation updates Δ wi from each intermediary by using a federated averaging algorithm;
the client 23 is a mobile phone or internet of things device that maintains a local training data set;
the clients 23 can be classified into three categories according to the characteristics of data distribution:
unifying the client: having training data sufficiently balanced and ready to run the FL application;
and (3) small client side: the data volume is relatively small, and the training process is difficult to participate;
deflecting the client: having enough training data, but biased towards retaining certain classes of data, resulting in local imbalances;
the broker 22 has two functions: firstly, the training process for three types of clients is rearranged; secondly, the intermediary also needs to make the collected data distribution nearly uniform; the second is a unified client with sufficiently balanced training data and ready to run the FL application;
according to an embodiment of the invention, the system may perform the following operations:
first, the FL server initializes the global model to start training;
then, the FL server starts a new round of communication r and sends the global model to the intermediary;
next, each moderator m coordinates the training of the designated clients and calculates the update of the weights
；
Finally, the FL server collects all intermediary updates, aggregates these updates with weights
Wherein, in the step (A),
is the cumulative training limit assigned to the broker's client, n = ∑ Σ
；
The server then updates the global model to
And ends the round.
Is the starting model for the next round, i.e., round n +1 communication.
The invention discloses a method and a system for federal learning of mobile equipment, which relieve the global unbalance problem of training data and improve the accuracy of a model by seventeen percent through the steps of data enhancement and rescheduling.
The invention uses ZOA polynomial time algorithm in the task allocation part, and compared with the traditional task scheduling algorithm Fed-LBAP and the like, the ZOA can obviously reduce the time required by operation under different conditions, especially when the task quantity is large. The ZOA may be easily adjusted to maximize the number of tasks allocated to the resource.
The method utilizes the red fox algorithm to replace the traditional training method, and simultaneously, the mechanism can select different meta-heuristic algorithms according to different application scenes and characteristics, so that the method has stronger universality.
The invention provides a mechanism for cleaning data sets under the condition of data set poisoning, and compared with the traditional federal learning method, the mechanism can improve the accuracy by at least 10%.
The communication processing mode of the invention adopts the ZanOverlap method, and the communication and the calculation are parallel, so that the federal learning communication efficiency can be improved, and the invention is completely compatible with a plurality of other data compression methods and has strong usability.
The method solves the problem of dependence between training iterations by relaxing chain constraint and a data compensation mechanism, thereby relieving the problem of time consumption of gradient update caused by parallelism in ZanOverlap, and keeping the convergence speed of federal learning by a parameter optimization mode.
The system of the invention divides the client into three types according to the characteristics of data distribution, thereby fully utilizing the characteristics of different clients to train and further operate.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all the functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: a mobile storage device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Alternatively, the integrated unit of the present invention may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or a part contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
Claims (10)
1. A mobile device federal learning method, comprising the steps of:
s102: the mobile equipment is added into a federal learning training task;
s104: the FL server initializes a global model and collects local data distribution of participants;
s106: rebalancing the federated learning training task by means of data enhancement and rescheduling;
s108: sending the balanced federal learning task to a lower-level client, and training a global model by using a red fox algorithm by the client;
s110: aggregating and feeding back the trained global model to obtain a latest global model;
s112: and performing federal learning by using the latest global model so as to improve the efficiency of federal learning of the mobile equipment.
2. The method for federal learning of a mobile device as claimed in claim 1, wherein S104 specifically is:
in an initialization stage, the FL server firstly waits for the mobile equipment to join in an FL model training task;
the mobile device participates in training by sending its local data distribution information to the FL server;
after determining the clients to participate in the training, the FL server initializes the weights and optimizers of the neural network model and collects the local data distribution of the participants.
3. The method for federal learning of a mobile device as claimed in claim 1 or 2, wherein the data enhancement procedure of S106 is as follows:
initializing a downsampling threshold τdAnd enhancing the threshold τaWhere τ is setaIs taudIf z-score of the data size of a class is greater than τ d, then the class is considered as a majority class, denoted Yd;
if the z-score of the data size of a class is smaller than tau a, the class is regarded as a few classes and is denoted as Ya;
when in use
When the temperature of the water is higher than the set temperature,
incorporate the set into Yd
When in use
When the temperature of the water is higher than the set temperature,
the set is incorporated into Ya
Wherein the content of the first and second substances,
for the size of the data set to be,
is the standard deviation of the data set and,
is the mean of the data set and,
z-score for the dataset;
the process is completed for the processes of Yd, Ya,
after the calculation, the server sends the parameters to the client;
then, all clients perform data expansion and downsampling in parallel;
the client operates on each sample in the dataset: if the source of the sample is in the set Yd, then determine if the amount of the source set y has been reduced to the original amount
If the sampling is satisfied, the sampling is reserved, and if the sampling is not satisfied, the sampling is returned to be empty; if the source of the sample is in the set Ya, the sample is enhanced, including operations of random shift, random rotation, random shearing and random scaling, and the size of the enhanced part is the original sample size
-1；
Once all clients have completed the enhancement and downsampling, the FL server creates an intermediary to reschedule the clients to achieve partial balancing.
4. The method according to claim 3, wherein the rescheduling process of S106 is as follows:
the broker traverses the data distribution of all the unallocated clients and selects the data distribution to make the data distribution of the broker closest to the uniformly distributed clients; on the basis, a ZOA algorithm is executed, so that each medium is guaranteed to be utilized to the maximum extent;
when the broker reaches the maximum allocated client limit, the FL server will create a new broker and repeat the above process until the training task can be rescheduled for all clients.
5. The method of claim 4, wherein the ZOA algorithm is as follows:
firstly, setting a minimum heap for a ZOA algorithm to store the cost of adding tasks to each resource;
then it initializes the lower limit of tasks allocated to resources, and if they can receive more tasks, the cost of allocating the next task to the heap increases; its main loop receives a new task by taking one of the resources with the lowest cost, allocating additional tasks to it, making the best allocation of one task at a time, updating its cost on the heap if it can still receive more tasks; after all iterations of the main loop, all tasks will be allocated to the existing resources and the ZOA algorithm is complete.
6. The method for federal learning of a mobile device as claimed in claim 1 or 5, wherein the S108 is specifically:
when each round of communication starts, each intermediary sends a task to a lower-level client, each client uses the red fox algorithm to train for E times, executes anti-virus-throwing operation judgment of the red fox algorithm, updates the global model, and then returns the updated global model to the corresponding intermediary;
the intermediary receives the updated result and sends the updated result to the next waiting training client;
all clients under intermediary management complete one-time receiving, training and feedback; this process is cycled Em times;
finally, all intermediaries send updates of the global model to the FL server.
7. The federal learning method for mobile devices as claimed in claim 6, wherein the judgment of the anti-virus-throwing operation of the red fox algorithm is specifically as follows:
firstly, selecting the number K of federal learning, defining the number of clients as m, defining the number of heuristic iterations performed on a server as Tserver and the number Ttrain of training times of the clients, and preparing databases of all the clients;
the server starts the operation of the initial global model by creating the initial global model and sends the initial global model to all the clients;
after the global model is distributed, there is a time for waiting for a result, if the number of training iterations is greater than 10, after each 10 iterations, the information of the current training state is sent to the server as information;
if the information of the previous stage of the client does not exist, the server stores the information;
in the opposite case, when this information is obtained and there is preview data, the actual loss function value and the number of poison samples are calculated;
if f2 is above 10% of the population, stopping training operations for the client and flagging as likely poisoned;
when this value of f2 is below 5% of all samples and the value of f1 is greater than 0, the current model is modified using meta-heuristics;
when the number of poisoned samples rapidly increases, the training is interrupted; this behavior implies the possibility of poisoning, which requires the training process to be interrupted in order not to significantly affect the model;
this can be expressed as:
executing a red fox algorithm when f1 is more than or equal to 0 ^ f2 and less than or equal to 0.05 marked with | D |;
when f2 is more than or equal to 0.1, marked with | D |, the training process is directly interrupted;
f1(L1,L2)=L1−L2；
f2(m1,m2)=m1−m2；
wherein L1 and L2 are values of loss functions of two adjacent iterations, m1 and m2 are numbers of samples poisoned by the two adjacent iterations, and | D | is the size of the database;
wherein f1 is an iteration loss function change difference for indicating the training direction; f2 is the difference in the number of poisoning samples, indicating the possibility of database poisoning.
8. The method for federal learning of a mobile device as claimed in claim 1 or 7, wherein the S110 is specifically:
after the FL server receives the updated model sent by the intermediary, the model weight is compensated by using Taylor series expansion and a Fisher information matrix, then the FL server calculates the nererov quantity and updates the global model by using the compensated weight, and then pushes the latest global model to the intermediary.
9. A mobile device federal learning system, the system comprising: FL server, intermediary and customer end;
the FL server is responsible for maintaining a global model w0, deploying the model to the intermediaries, and synchronously extracting aggregation update delta wi from each intermediary by using a federal average algorithm;
the client is a mobile phone or an internet of things device for maintaining a local training data set;
according to the characteristics of data distribution, clients can be divided into three categories:
unifying the client: having training data sufficiently balanced and ready to run the FL application;
and (3) small client side: the data volume is relatively small, and the training process is difficult to participate;
deflecting the client: having enough training data, but biased towards retaining certain classes of data, resulting in local imbalances;
the intermediary has two functions: firstly, the training process for three types of clients is rearranged; secondly, the intermediary also needs to make the collected data distribution nearly uniform; the second is a unified client with sufficiently balanced training data and ready to run the FL application.
10. A mobile device federal learning system as claimed in claim 9, wherein the system is operable to:
first, the FL server initializes the global model to start training;
then, the FL server starts a new round of communication r and sends the global model to the intermediary;
next, each moderator m coordinates the training of the designated clients and calculates the update of the weights
；
Finally, the FL server collects all intermediary updates, aggregates these updates with weights
Wherein, in the step (A),
is the cumulative training limit assigned to the broker's client, n = ∑ Σ
；
The server then updates the global model to
And the present round is finished,
is the starting model for the next round, i.e., round n +1 communication.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210218303.5A CN114358286A (en) | 2022-03-08 | 2022-03-08 | Mobile equipment federal learning method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210218303.5A CN114358286A (en) | 2022-03-08 | 2022-03-08 | Mobile equipment federal learning method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114358286A true CN114358286A (en) | 2022-04-15 |
Family
ID=81094299
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210218303.5A Pending CN114358286A (en) | 2022-03-08 | 2022-03-08 | Mobile equipment federal learning method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114358286A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114818996A (en) * | 2022-06-28 | 2022-07-29 | 山东大学 | Method and system for diagnosing mechanical fault based on federal domain generalization |
| CN114827289A (en) * | 2022-06-01 | 2022-07-29 | 深圳大学 | Communication compression method, system, electronic device and storage medium |
| CN115002031A (en) * | 2022-05-11 | 2022-09-02 | 大连理工大学 | Unbalanced data distribution-based federated learning network traffic classification model training method, model and classification method |
| CN115329987A (en) * | 2022-08-04 | 2022-11-11 | 苏州大学 | User selection method in federated learning system |
| CN117311993A (en) * | 2023-11-28 | 2023-12-29 | 华东交通大学 | Cloud computing load balancing method and system |
| CN117875455A (en) * | 2024-03-08 | 2024-04-12 | 南京信息工程大学 | Federal learning data poisoning defense method based on data enhancement |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113947494A (en) * | 2021-10-26 | 2022-01-18 | 山大地纬软件股份有限公司 | Method and system for detecting medical insurance fraud behavior based on federal learning |
| CN113962359A (en) * | 2021-09-30 | 2022-01-21 | 华东师范大学 | Self-balancing model training method based on federal learning |
| CN113971090A (en) * | 2021-10-21 | 2022-01-25 | 中国人民解放军国防科技大学 | Layered federal learning method and device of distributed deep neural network |
-
2022
- 2022-03-08 CN CN202210218303.5A patent/CN114358286A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113962359A (en) * | 2021-09-30 | 2022-01-21 | 华东师范大学 | Self-balancing model training method based on federal learning |
| CN113971090A (en) * | 2021-10-21 | 2022-01-25 | 中国人民解放军国防科技大学 | Layered federal learning method and device of distributed deep neural network |
| CN113947494A (en) * | 2021-10-26 | 2022-01-18 | 山大地纬软件股份有限公司 | Method and system for detecting medical insurance fraud behavior based on federal learning |
Non-Patent Citations (4)
| Title |
|---|
| DAWID POLAP 等: "Meta-heuristic as manager in federated learning approaches for image processing purposes", 《APPLIED SOFT COMPUTING》 * |
| LAERCIO L. PILLA: "Optimal Task Assignment to Heterogeneous Federated Learning Devices", 《ARXIV》 * |
| MOMING DUAN 等: "Self-Balancing Federated Learning With Global Imbalanced Data in Mobile Systems", 《IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS》 * |
| YUHAO ZHOU 等: "Communication-Efficient Federated Learning With Compensated Overlap-FedAvg", 《IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115002031A (en) * | 2022-05-11 | 2022-09-02 | 大连理工大学 | Unbalanced data distribution-based federated learning network traffic classification model training method, model and classification method |
| CN115002031B (en) * | 2022-05-11 | 2023-09-29 | 大连理工大学 | Federal learning network flow classification model training method, model and classification method based on unbalanced data distribution |
| CN114827289A (en) * | 2022-06-01 | 2022-07-29 | 深圳大学 | Communication compression method, system, electronic device and storage medium |
| CN114818996A (en) * | 2022-06-28 | 2022-07-29 | 山东大学 | Method and system for diagnosing mechanical fault based on federal domain generalization |
| CN115329987A (en) * | 2022-08-04 | 2022-11-11 | 苏州大学 | User selection method in federated learning system |
| CN117311993A (en) * | 2023-11-28 | 2023-12-29 | 华东交通大学 | Cloud computing load balancing method and system |
| CN117311993B (en) * | 2023-11-28 | 2024-03-08 | 华东交通大学 | Cloud computing load balancing method and system |
| CN117875455A (en) * | 2024-03-08 | 2024-04-12 | 南京信息工程大学 | Federal learning data poisoning defense method based on data enhancement |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114358286A (en) | Mobile equipment federal learning method and system | |
| US20220391771A1 (en) | Method, apparatus, and computer device and storage medium for distributed training of machine learning model | |
| CN112882815B (en) | Multi-user edge calculation optimization scheduling method based on deep reinforcement learning | |
| CN105704255B (en) | A kind of server load balancing method based on genetic algorithm | |
| CN113516250A (en) | Method, device and equipment for federated learning and storage medium | |
| CN111027736A (en) | Micro-service combined deployment and scheduling method under multi-objective optimization | |
| CN110362380B (en) | Network shooting range-oriented multi-objective optimization virtual machine deployment method | |
| EP4350572A1 (en) | Method, apparatus and system for generating neural network model, devices, medium and program product | |
| CN110502323B (en) | Real-time scheduling method for cloud computing tasks | |
| CN116134448A (en) | Joint machine learning using locality sensitive hashing | |
| CN114169543A (en) | Federal learning algorithm based on model obsolescence and user participation perception | |
| Li et al. | Data analytics for fog computing by distributed online learning with asynchronous update | |
| CN107040475B (en) | Resource scheduling method and device | |
| CN116384504A (en) | Federal migration learning system | |
| CN111047040A (en) | Web service combination method based on IFPA algorithm | |
| CN115794354A (en) | Cloud video transcoding task scheduling based on K-Means clustering and improved AOA algorithm | |
| CN116339932A (en) | Resource scheduling method, device and server | |
| CN114723071A (en) | Federal learning method and device based on client classification and information entropy | |
| CN114035954A (en) | Scheduling system and task scheduling system based on DDQN algorithm | |
| US20240028911A1 (en) | Efficient sampling of edge-weighted quantization for federated learning | |
| US20240028974A1 (en) | Edge-weighted quantization for federated learning | |
| US11805067B1 (en) | Dynamically generating and managing request queues for processing electronic requests via a shared processing infrastructure | |
| CN117496275B (en) | Class learning-based depth image classification network training method, electronic equipment and storage medium | |
| US20240111607A1 (en) | Similarity-based quantization selection for federated learning with heterogeneous edge devices | |
| CN117557870B (en) | Classification model training method and system based on federal learning client selection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220415 |