CN110728376B - Federal learning method and device based on tree topology structure - Google Patents
Federal learning method and device based on tree topology structure Download PDFInfo
- Publication number
- CN110728376B CN110728376B CN201911000573.3A CN201911000573A CN110728376B CN 110728376 B CN110728376 B CN 110728376B CN 201911000573 A CN201911000573 A CN 201911000573A CN 110728376 B CN110728376 B CN 110728376B
- Authority
- CN
- China
- Prior art keywords
- node
- layer
- model
- nodes
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000012549 training Methods 0.000 claims abstract description 186
- 238000005070 sampling Methods 0.000 claims abstract description 68
- 230000002776 aggregation Effects 0.000 claims abstract description 67
- 238000004220 aggregation Methods 0.000 claims abstract description 67
- 238000012545 processing Methods 0.000 claims abstract description 13
- 230000005540 biological transmission Effects 0.000 claims abstract description 10
- 238000004590 computer program Methods 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 11
- 238000006116 polymerization reaction Methods 0.000 abstract description 12
- 238000010801 machine learning Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 12
- 238000013507 mapping Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000004931 aggregating effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a federal learning method and device based on a tree topology structure, relates to the technical field of machine learning, and aims to solve the problems of overlong federal aggregation time and high network transmission pressure. The tree topology of the method comprises at least two layers of structures, each layer of structure comprises at least one node, each node corresponds to a training model, and the method comprises the following steps: if the preset triggering condition is met, sampling the nodes of the current layer of the tree topology structure; determining a training model corresponding to the current layer node after sampling processing, and performing federal aggregation on the determined training model. The method reduces the network transmission pressure of the layered tree topology structure when the federation polymerization is carried out, and accelerates the speed of the federation polymerization.
Description
Technical Field
The application relates to the technical field of machine learning, in particular to a federal learning method and device based on a tree topology structure.
Background
In order to solve the problem that the storage pressure of the server is high and the network transmission pressure is high due to the excessive number of clients, a layered tree topology is often adopted to alleviate the problem, see fig. 1, fig. 1 shows a tree topology of three layers, when the federation is triggered, according to the known federation relation, the federation is performed by each training model of the third layer to obtain each training model of the second layer, for example, the training model 131, the training model 132 and the training model corresponding to the training model 132 are subjected to federation to obtain the training model corresponding to the training model 121, and the federation of other training models can be described with reference to the description above.
However, when the current layer performs federation, the hierarchical tree topology structure must wait for the next layer federation of the current layer to complete, and as long as any one of the next layer federations is not completed, it may cause the federation of the current layer to wait indefinitely.
In summary, when federation is performed on a tree topology structure based on layering, federation time is too long, network transmission pressure is high, and success of federation cannot be ensured.
Disclosure of Invention
The embodiment of the application provides a federation learning method and a federation learning device based on a tree topology structure, which are used for at least reducing network transmission pressure of the hierarchical tree topology structure when federation polymerization is carried out and accelerating the speed of federation polymerization.
In a first aspect of the present application, a federal learning method based on a tree topology, where the tree topology includes at least two layers of structures, each layer of structure includes at least one node, and each node corresponds to a training model, including:
if the preset triggering condition is met, sampling the current layer node of the tree topology structure;
Determining a training model corresponding to the current layer node after sampling processing, and performing federal aggregation on the determined training model.
In one possible implementation, the determined training model performs federal aggregation, including:
And performing federation aggregation on the determined training models to obtain training models corresponding to at least one current parent layer node, wherein the current parent layer is the layer where the parent node of the current layer node is located.
In one possible implementation manner, the sampling processing on the current layer node of the tree topology includes:
Sampling at least one current layer node from the current layer nodes;
deleting the training model corresponding to the at least one current layer node, or marking the at least one current layer node, so that the training model corresponding to the at least one current layer node does not participate in the federation aggregation.
In one possible implementation, the preset trigger condition includes one or more of the following:
The number of the current layer nodes is larger than a first preset value;
the training model corresponding to the current layer node does not accord with the first preset federal model condition.
In one possible implementation manner, the sampling at least one current layer node from the current layer nodes includes:
randomly sampling at least one current layer node from the current layer nodes; or (b)
Sampling current layer nodes with preset node number from the current layer nodes.
In one possible implementation manner, before the sampling the current layer node with the preset number of nodes from the current layer nodes, the method further includes:
Determining the number of the nodes of the current layer, and determining the preset number of nodes corresponding to the number of the nodes of the current layer according to the preset mapping relation between the number of the nodes and the preset number of the nodes.
In one possible implementation manner, the federal model aggregation is sequentially performed to the top layer from the bottom layer of the tree topology, and before the federal model aggregation is sequentially performed to the top layer from the bottom layer of the tree topology, the method further includes:
and receiving a training model corresponding to the bottom node of the tree topology structure, wherein the training model is obtained by training a local model through local training data for the client corresponding to the bottom node.
In one possible implementation, the method further includes:
If the top layer federation model obtained by federation does not meet the second preset federation model condition, the top layer federation model is sent to a client corresponding to the bottom layer node, so that the client takes the top layer federation model as the local model to carry out next training;
And receiving a training model obtained by the client for next round training, and sequentially carrying out next round federal model aggregation to the top layer from the bottom layer of the tree topology structure.
In a second aspect of the present application, there is provided a federal learning apparatus based on a tree topology, the tree topology including at least two layers, each layer including at least one node, each node corresponding to a training model, the apparatus comprising:
The node sampling unit is used for sampling the current layer node of the tree topology structure if the preset trigger condition is met;
The federation aggregation unit is used for determining the training model corresponding to the current layer node after sampling processing and performing federation aggregation on the determined training model.
In one possible implementation manner, the federation aggregation unit is configured to perform federation aggregation on the determined training model to obtain a training model corresponding to a node of at least one current parent layer, where the current parent layer is a layer where a parent node of the current layer node is located.
In a possible implementation manner, the node sampling unit is configured to:
Sampling at least one current layer node from the current layer nodes;
deleting the training model corresponding to the at least one current layer node, or marking the at least one current layer node, so that the training model corresponding to the at least one current layer node does not participate in the federation aggregation.
In one possible implementation, the preset trigger condition includes one or more of the following:
The number of the current layer nodes is larger than a first preset value;
the training model corresponding to the current layer node does not accord with the first preset federal model condition.
In a possible implementation manner, the node sampling unit is configured to:
randomly sampling at least one current layer node from the current layer nodes; or (b)
Sampling current layer nodes with preset node number from the current layer nodes.
In a possible implementation manner, the node sampling unit is configured to:
Before sampling current layer nodes with preset node numbers from the current layer nodes, determining the node numbers of the current layer nodes, and determining preset node numbers corresponding to the node numbers of the current layer nodes according to preset mapping relations of the node numbers and the preset node numbers.
In one possible implementation manner, the federation aggregation unit is configured to sequentially perform federation model aggregation to a top layer from a bottom layer of the tree topology, where before sequentially performing federation model aggregation to the top layer from the bottom layer of the tree topology, a training model corresponding to a bottom layer node of the tree topology is received, and the training model is obtained by training a local model through local training data for a client corresponding to the bottom layer node.
In one possible implementation, the federal polymerization unit is further configured to:
If the top layer federation model obtained by federation does not meet the second preset federation model condition, the top layer federation model is sent to a client corresponding to the bottom layer node, so that the client takes the top layer federation model as the local model to carry out next training;
And receiving a training model obtained by the client for next round training, and starting the bottom layer of the tree topology structure and sequentially carrying out next round federal model aggregation to the top layer.
In a third aspect, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of the first aspect and one of the possible implementations when executing the program.
In a fourth aspect, there is provided a computer readable storage medium storing computer instructions that, when run on a computer, cause the computer to perform the method according to any one of the first aspect and one of the possible implementations.
The embodiment of the application has at least the following beneficial effects:
in the scheme of the application, when the tree topology structure is subjected to horizontal federal learning, the training model of part of nodes is discarded, and the network transmission pressure during the transmission of the training model is reduced; and because the training model of part of nodes is discarded, the time required for federal polymerization is also reduced; meanwhile, the storage pressure of the server side for federal aggregation is correspondingly reduced due to the fact that the training model of part of nodes is discarded.
On the other hand, in the training model of discarding part of the nodes, random discarding can be selected, so that federation can be performed according to the rest of the training model, and the time for obtaining the top-level federation model by the tree topology structure is reduced.
Drawings
FIG. 1 is a schematic diagram of a tree topology provided in the background of the application;
FIG. 2 is a schematic diagram of a standard federal learning process provided by an embodiment of the present application;
FIG. 3 is a flowchart of another tree topology according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a federal learning process based on a tree topology according to an embodiment of the present application;
FIG. 5 is a schematic diagram of another federal learning process based on tree topology according to an embodiment of the present application;
FIG. 6 is a block diagram of a federal learning device according to an embodiment of the present application;
fig. 7 is a block diagram of a computer device according to an embodiment of the present application.
Detailed Description
In order to better understand the technical solutions provided by the embodiments of the present application, the following detailed description will be given with reference to the accompanying drawings and specific embodiments.
In order to facilitate a better understanding of the technical solutions of the present application, the following description will refer to proper nouns/technical terms.
The client may be a terminal device such as a mobile phone, a computer, a Personal digital assistant (Personal DIGITAL ASSISTANT, PDA), an intelligent television, a vehicle-mounted device, or a network device such as a server.
Federal model learning concept: and encrypting the built machine learning model by using a technical algorithm, wherein a plurality of clients in the federal learning architecture do not need to give own data when the model is trained, training the local model according to the global federal model encrypted by parameters issued by the server and training data local to the clients, returning the local model to the server for aggregating and updating the global federal model, and issuing the updated federal model to the clients again, and repeating circularly until convergence.
As shown in fig. 2, a standard federal model learning process is given below, specifically including:
step S201, each client 21 trains a local model locally to obtain a training model;
step S202, each client 21 sends the training model obtained by training to a server;
Step S203, the server 22 receives a plurality of training models and performs federation aggregation on the plurality of models to obtain federation models;
step S204, the server sends the federation model obtained by federation aggregation to each client;
in step S205, each client receives the federation model, and updates the local model to the federation model.
However, the standard federal model learning process of fig. 2 clearly suffers from the following drawbacks:
1) When the number of clients in the federal structure is large, after a plurality of clients send training models, the storage pressure of the server is very high;
2) When the number of clients in the federal architecture is large, the pressure of network transmission is great when multiple clients send training models to the server.
The above drawbacks in the standard-based federal model training are often alleviated by using a layered tree topology, as shown in fig. 1, which shows a tree topology of three layers, and when federal model polymerization is triggered, according to a known federal polymerization relationship, federal polymerization is first performed by each training model of the third layer, so as to obtain each training model of the second layer.
However, when the current layer performs federation, the hierarchical tree topology must wait for the next layer federation of the current layer to complete, and as long as any one of the next layer federations is not complete, it may cause the federation of the current layer to wait indefinitely, as shown in fig. 3, a four-layer tree topology is provided, in fig. 3, each circle represents a node, and each node corresponds to a training model, where the federation result of the node 321 needs to wait for the federation of the node 331 and the node 332 to complete, and similarly, the federation of the node 331 needs to wait for the training models of the node 31 and the node 342 to complete, the federation of the node 332 needs to wait for the training models of the node 343 and the node 344 to complete, and as long as the federation training or model training of one of the intermediate nodes is not complete, the federation of the node 421 may wait indefinitely.
In view of this, the present application provides a method, an apparatus and a medium for lateral federal learning based on a tree topology, wherein the tree topology is known and includes at least two layers of structures, each layer of structure includes at least one node, and each node corresponds to a training model, and specifically includes: when the preset triggering condition is met, sampling the current layer node of the tree topology structure, further performing a training model corresponding to the current layer node after sampling, and performing federal aggregation on the determined training model.
Further, federation aggregation is carried out on the determined training models, so that training models corresponding to at least one node of the current parent layer are obtained, and the current parent layer is a layer of a parent node shuttle of the current layer node.
The method can be applied to a scene comprising a server and a plurality of clients, wherein a training model obtained by training a local model by the clients is used as a training model of a bottom node of the tree topology structure, and the server performs horizontal federal learning according to the known tree topology structure.
The scenario of the above method application may also include a plurality of servers and a plurality of clients, where the plurality of servers and the plurality of clients construct a tree-like network topology according to a known tree-like topology, each client serves as a bottom node of the tree-like topology, each server serves as a node other than the bottom node, and the servers of each layer perform federal aggregation on the training model of the next layer.
The method provided by the application is further described below:
in this embodiment, in the process of performing horizontal federation learning based on the tree topology structure, federation model aggregation may be sequentially performed to the top layer from the bottom layer of the tree topology structure, and when any layer is performed, if a preset trigger condition is met, sampling processing is performed on the nodes of the layer.
The preset triggering conditions include one or more of the following conditions:
the first trigger condition: the number of the nodes of the current layer is larger than a first preset value.
The first preset value may be, but is not limited to, 1 or 2, that is, when the number of nodes of the current layer model is 1, the training model corresponding to the node may be directly used as the final federal model; when the number of nodes of the current layer model is 2, the training models corresponding to the 2 nodes can be directly subjected to federation aggregation, and the federation model obtained by the federation aggregation is used as a final federation model.
The second trigger condition: the training model corresponding to the current layer node does not accord with the first preset federal model condition.
If the training models corresponding to the current layer of nodes have a first preset number of training models which do not accord with the first preset federal model conditions; the first preset federal model condition is used for measuring the prediction accuracy of the top-level federal model, so that the preset federal model condition can be set as follows: the model prediction accuracy is larger than a second preset value, and a person skilled in the art can set the preset federal model condition according to other actual requirements; the first preset number and the second preset value may be set by those skilled in the art according to actual needs.
In this embodiment, federation model aggregation is sequentially performed from the bottom layer to the top layer of the tree topology, where before federation model aggregation is sequentially performed from the bottom layer to the top layer of the tree topology, a training model corresponding to the bottom node of the tree topology should be received.
The training model corresponding to the bottom layer node can be a client corresponding to the bottom layer node, and the training model is obtained by training the local training data.
After the federation model is polymerized to the top layer by the method to obtain a top layer federation model, whether the obtained top layer federation model meets the second preset federation model condition or not can be judged, if not, the top layer federation model is sent to a client corresponding to a bottom layer node, and the client is instructed to take the top layer federation model as the local model to carry out next training; and
And receiving a training model obtained by the client for next round training, taking the received training model as a training model corresponding to the bottom node, and sequentially carrying out next round federation model aggregation on the top layer from the bottom layer of the tree topology structure until the obtained top layer federation model meets the second preset federation model condition.
The second preset federal model condition is used for measuring the prediction accuracy of the top federal model, and the second preset federal model condition and the first preset federal model condition can be set to be the same condition or can be set to be different conditions, for example, the second preset federal model condition is set as follows: the model prediction accuracy is greater than a third preset value, which may be the same as or different from the first preset value in the first preset federal model condition.
In this embodiment, the current layer node of the tree topology may be sampled by, but not limited to, the following sampling method:
sampling at least one current layer node from the current layer nodes;
deleting the training model corresponding to the sampled at least one current layer node, or marking the sampled at least one current layer node, so that the training model corresponding to the sampled at least one current layer node does not participate in federal aggregation.
Several methods of sampling at least one current tier node from the current tier nodes are provided below:
the first sampling method is as follows: at least one current layer node is randomly sampled from the current layer nodes.
In the sampling method, the current layer node is randomly sampled, so that the number of the sampled nodes is uncertain, and the specific sampled current layer node is also uncertain.
The second sampling method is as follows: sampling current layer nodes with preset node number from the current layer nodes.
In the sampling mode, the node number range of the node number of the current layer node can be determined;
determining the number of preset nodes corresponding to the number range of the nodes, sampling the current layer nodes with the number of the preset nodes from the current layer nodes, wherein the number of the sampled preset nodes is determined at the moment, and the specific current layer points sampled are not determined.
The number of the node number ranges and the specific numerical ranges are not limited too much, and can be set by those skilled in the art according to actual needs, and the following only provides an exemplary example:
If the number of nodes of the bottom node of the known tree topology is 15, three node number ranges, such as 2-6, 7-11 and 12-15, can be set, and if only one node is left, the training model corresponding to the node is determined to be the final federal model, so that the node number range does not need to contain 1.
The determining the number of the preset nodes corresponding to the number range of the nodes may include, but is not limited to, the following two methods:
the first preset node number determining mode comprises the following steps:
And determining the preset node number corresponding to the node number range of the current layer node according to the preset mapping relation of the node number range and the preset node number.
Alternatively, a person skilled in the art may obtain the appropriate preset number of nodes corresponding to different number of node ranges according to experiments and other manners, so as to construct a preset mapping relationship between the number of nodes range and the preset number of nodes; if the number of nodes is 7-11, the corresponding preset number of nodes is 3, etc.
The second mode for determining the number of preset nodes is as follows:
determining the sampling node number range corresponding to the node number range according to a preset mapping relation between the node number range and the sampling node number range; randomly determining that the number of one sampling node in the number range of the sampling nodes is the number of preset nodes corresponding to the number range of the nodes of the current layer.
Alternatively, a person skilled in the art may obtain a suitable sampling node number range corresponding to different node number ranges according to an experiment or other manners, so as to construct a preset mapping relationship between the node number range and the sampling node number range; if the number of the nodes ranges from 7 to 11, the corresponding number of the sampling nodes can range from 2 to 5, and when the number of the nodes ranges from 7 to 11 is determined, the number of the sampling nodes is determined to range from 2 to 5, and the number of the preset nodes, of which the number is the number range of the nodes of the current layer, can be randomly determined from 2, 3, 4 and 5.
Examples of 2 lateral federal learning methods based on tree topologies are given below:
example 1: randomly deleting training model corresponding to at least one node in current layer
As shown in fig. 4, the tree topology includes a node 411 at the top layer, a node 421 and a node 422 at the second layer, a node 431 to a node 434 at the third layer, a node 441 to a node 449 at the fourth layer, and a node 4501 to a node 4517 at the fifth layer, and the federal aggregation process of the tree topology is as follows:
the first step: and randomly discarding part of nodes at a fifth layer, and performing federal aggregation on the rest of training models to obtain training models corresponding to the nodes of a fourth layer.
In fig. 4, the hatched circles indicate the nodes selected for discarding, and the dotted circles indicate the nodes of the factor layer that are all discarded or marked for discarding, and the nodes of the training model cannot be obtained.
In fig. 4, training models corresponding to nodes 4501, 4503, 4504, 4508, 4511, 4512, 4513, and 4517 are discarded at the fifth layer, and therefore, since training models corresponding to nodes 4501 and 4502 are discarded, node 442 does not obtain a training model, and similarly, node 446 does not obtain a model.
And a second step of: and partially discarding the obtained training model in the fourth layer, and performing federal aggregation on the rest training models to obtain the training model corresponding to the nodes of the third layer.
As shown in fig. 4, the training models corresponding to node 444 and node 445 are discarded at the third layer, respectively, and since the training model corresponding to node 445 is discarded, the training model is not obtained at node 446, and thus the corresponding training model is not obtained at node 433.
And thirdly, partially discarding the obtained training model at the third layer, and performing federal aggregation on the rest training models to obtain the training model corresponding to the nodes of the second layer.
As shown in fig. 4, the training models corresponding to node 431 are discarded at the third level, respectively.
And fourthly, partially discarding the obtained training model in the second layer, and performing federation aggregation on the rest training models to obtain federal models corresponding to the nodes 411 of the first layer.
As shown in fig. 4, the training models corresponding to the node 422 are discarded respectively in the second layer, and only the training model corresponding to the node 421 remains after the training model corresponding to the node 422 is discarded, and the training model corresponding to the node 421 is directly used as the federal model corresponding to the node 411.
Alternatively, in the fourth step, the federal model corresponding to the node 411 may be obtained by directly federally aggregating the 2 training models of the second layer without discarding the training models.
Example 2: discarding at least one training model according to the node number range of the current layer node
The tree topology is shown in fig. 5, the top layer includes node 511, the second layer includes node 521 and node 522, the third layer includes node 531 to node 534, the fourth layer includes node 541 to node 549, and the fifth layer includes node 5501 to node 5517;
The mapping relation between the preset node number range and the sampling node number range is as follows in table 1:
TABLE 1
The federal polymerization process for the tree topology is as follows:
the first step: discarding part of nodes at the fifth layer, and performing federal aggregation on the rest of training models to obtain training models corresponding to the nodes of the fourth layer.
In fig. 5, the hatched circles indicate the nodes selected for discarding, and the dotted circles indicate the nodes of the factor layer that are all discarded or marked for discarding, and the nodes of the training model cannot be obtained.
The number of nodes of the fifth layer is 17, the nodes belong to a preset node number range from 12 to 18, the corresponding sampling node number range is from 5 to 9, at the moment, the numerical value 6 is randomly determined from the numbers 5 to 9, the nodes 5502, 5505, 5508, 5509, 5513 and 5517 selected randomly from the nodes of the fifth layer are discarded, training models corresponding to the 6 selected nodes are discarded, and the rest training models are subjected to federal polymerization to obtain training models corresponding to the nodes of the fourth layer.
And a second step of: and partially discarding the obtained training model in the fourth layer, and performing federal aggregation on the rest training models to obtain the training model corresponding to the nodes of the third layer.
The number of nodes in the fourth layer is 8, the number of the nodes belongs to a preset node number range of 7-11, the number of the corresponding sampling nodes is 3-5, at the moment, the numerical value 3 is randomly determined from the numbers 3-5, the nodes 541, 545 and 547 selected randomly from the nodes in the fourth layer are discarded, training models corresponding to the 3 selected nodes are discarded, and the rest training models are subjected to federation aggregation to obtain training models corresponding to the nodes in the third layer.
And thirdly, partially discarding the obtained training model at the third layer, and performing federal aggregation on the rest training models to obtain the training model corresponding to the nodes of the second layer.
The number of nodes of the third layer is 4, the number of the nodes belongs to a preset node number range of 3-6, the number of the corresponding sampling nodes is 1-2, at the moment, the number of 2 is randomly determined from the numbers of 1 and 2, nodes 531 and 534 selected randomly from the nodes of the third layer are discarded, training models corresponding to the 2 selected nodes are discarded, and federal aggregation is carried out on the remaining training models to obtain training models corresponding to the nodes of the second layer.
Fourth, because the number of nodes corresponding to the second layer is 2, the number of nodes discarded in the layer is 0, and the 2 training models in the layer are directly subjected to federal aggregation to obtain the training model corresponding to the node 511.
As shown in fig. 6, based on the same technical concept, the embodiment of the present application further provides a horizontal federal learning device 600, where the tree topology includes at least two layers, each layer includes at least one node, and each node corresponds to a training model, and the device includes:
a node sampling unit 601, configured to sample a current layer node of the tree topology if a preset trigger condition is met;
and the federation unit 602 is configured to determine the training model corresponding to the current layer node after the sampling processing, and perform federation aggregation on the determined training model.
Optionally, the federation aggregation unit is configured to federate aggregate the determined training models to obtain a training model corresponding to a node of at least one current parent layer, where the current parent layer is a layer where a parent node of the current layer node is located.
Optionally, the node sampling unit is configured to:
sampling at least one current layer node from the current layer nodes;
Deleting the training model corresponding to the at least one current layer node, or marking the at least one current layer node, so that the training model corresponding to the at least one current layer node does not participate in the federation aggregation.
Optionally, the preset triggering condition includes one or more of the following:
the number of the current layer nodes is larger than a first preset value;
the training model corresponding to the current layer node does not accord with the first preset federal model condition.
Optionally, the node sampling unit is configured to:
Randomly sampling at least one current layer node from the current layer nodes; or (b)
Sampling current layer nodes with preset node number from the current layer nodes.
Optionally, the node sampling unit is configured to:
Before sampling the current layer node with the preset node number from the current layer nodes, determining the node number of the current layer node, and determining the preset node number corresponding to the node number of the current layer node according to a preset mapping relation between the node number and the preset node number.
Optionally, the federation aggregation unit is configured to sequentially perform federation model aggregation to a top layer from a bottom layer of the tree topology, where before sequentially performing federation model aggregation to the top layer from the bottom layer of the tree topology, receive a training model corresponding to a bottom layer node of the tree topology, where the training model is obtained by training a local model through local training data, and the client corresponding to the bottom layer node is used as the training model.
Optionally, the above federal polymerization unit is further configured to:
if the top layer federation model obtained by federation does not meet the second preset federation model condition, the top layer federation model is sent to a client corresponding to the bottom layer node, so that the client takes the top layer federation model as the local model to carry out next training;
And receiving a training model obtained by the client for next training, starting the bottom layer of the tree topology structure, and sequentially carrying out next federal model aggregation to the top layer.
Based on the same technical concept, an embodiment of the present application further provides a computer device 700, please refer to fig. 7, which includes a processor 701 and a memory 702, wherein:
The memory 702 has stored therein a computer program;
the processor 701, when executing the computer program, implements the federal learning method based on a tree topology discussed above, and will not be repeated here.
One processor 701 is illustrated in fig. 7, but the number of processors 701 is not limited in practice.
Wherein the memory 702 may be a volatile memory (RAM), such as a random-access memory (RAM); the memory 702 may also be a non-volatile memory (non-volatile memory), such as read-only memory, flash memory (flash memory), hard disk (HARD DISK DRIVE, HDD) or solid state disk (solid-state disk) (STATE DRIVE, SSD), or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The memory 702 may be a combination of the above.
As an example, the functions of the various modules of the lateral federal learning apparatus 600 of fig. 6 may be implemented by the processor 701 of fig. 7.
Based on the same technical concept, an embodiment of the present application also provides a computer readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a federal learning method based on a tree topology as previously discussed.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (9)
1. The federal learning method based on the tree-like network topology structure is characterized in that the tree-like network topology structure comprises at least two layers of structures, each layer of structure comprises at least one node, each node corresponds to a training model, the tree-like network topology structure is constructed by a plurality of clients and a plurality of servers according to the known tree-like topology structure, the bottom nodes in the tree-like topology structure are clients, the nodes outside the bottom nodes are servers, and the servers of each layer perform federal aggregation on the training models of the next layer; the method comprises the following steps:
starting from a bottom client of the tree topology structure, sequentially performing federal model aggregation to a top server, and sampling current layer nodes of the tree topology structure if a preset trigger condition is met when the tree topology structure is processed to any layer;
Determining a training model corresponding to the current layer node after sampling treatment, and performing federal aggregation on the determined training model;
the sampling processing for the current layer node of the tree topology structure comprises the following steps:
Sampling at least one current layer node from the current layer nodes;
Deleting the training model corresponding to the at least one current layer node, or marking the at least one current layer node, so that the training model corresponding to the at least one current layer node does not participate in the federation aggregation, and reducing network transmission pressure when the training model is transmitted;
Determining a training model corresponding to the current layer node after sampling, and performing federal aggregation on the determined training model, wherein the method comprises the following steps:
Transmitting the corresponding training model to the corresponding parent node of the upper layer through the current layer nodes which remain after sampling processing, and reducing the storage pressure of the server side;
And performing federal aggregation on the received training model through the father node.
2. The method of claim 1, wherein the determined training model performs federal aggregation comprising:
And performing federation aggregation on the determined training models to obtain training models corresponding to at least one current parent layer node, wherein the current parent layer is the layer where the parent node of the current layer node is located.
3. The method of claim 1, wherein the preset trigger conditions include one or more of:
The number of the current layer nodes is larger than a first preset value;
the training model corresponding to the current layer node does not accord with the first preset federal model condition.
4. The method of claim 1, wherein the sampling at least one current tier node from the current tier nodes comprises:
randomly sampling at least one current layer node from the current layer nodes; or (b)
Sampling current layer nodes with preset node number from the current layer nodes.
5. The method of claim 1, wherein federal model aggregation is performed sequentially from a bottom level to a top level of the tree topology, wherein before the federal model aggregation is performed sequentially from the bottom level to the top level of the tree topology, further comprising:
and receiving a training model corresponding to the bottom node of the tree topology structure, wherein the training model is obtained by training a local model through local training data for the client corresponding to the bottom node.
6. The method as recited in claim 5, further comprising:
If the top layer federation model obtained by federation does not meet the second preset federation model condition, the top layer federation model is sent to a client corresponding to the bottom layer node, so that the client takes the top layer federation model as the local model to carry out next training;
And receiving a training model obtained by the client for next round training, and sequentially carrying out next round federal model aggregation to the top layer from the bottom layer of the tree topology structure.
7. The federal learning aggregation device based on the tree-like network topology structure is characterized in that the tree-like topology structure comprises at least two layers of structures, each layer of structure comprises at least one node, each node corresponds to a training model, the tree-like network topology structure is constructed by a plurality of clients and a plurality of servers according to the known tree-like topology structure, the bottom nodes in the tree-like topology structure are clients, the nodes outside the bottom nodes are servers, and the servers of each layer perform federal aggregation on the training models of the next layer; the device comprises:
The node sampling unit is used for sequentially carrying out federal model aggregation on a top-level server from a bottom-level client of the tree topology structure, and sampling the current-level nodes of the tree topology structure if a preset trigger condition is met when the tree topology structure is carried out on any level;
The federation aggregation unit is used for determining a training model corresponding to the current layer node after sampling treatment and performing federation aggregation on the determined training model;
the node sampling unit is specifically configured to:
Sampling at least one current layer node from the current layer nodes;
Deleting the training model corresponding to the at least one current layer node, or marking the at least one current layer node, so that the training model corresponding to the at least one current layer node does not participate in the federation aggregation, and reducing network transmission pressure when the training model is transmitted;
The federal polymerized units are particularly useful for:
Transmitting the corresponding training model to the corresponding parent node of the upper layer through the current layer nodes which remain after sampling processing, and reducing the storage pressure of the server side;
And performing federal aggregation on the received training model through the father node.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any of claims 1-6 when the program is executed.
9. A computer readable storage medium storing computer instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911000573.3A CN110728376B (en) | 2019-10-21 | 2019-10-21 | Federal learning method and device based on tree topology structure |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911000573.3A CN110728376B (en) | 2019-10-21 | 2019-10-21 | Federal learning method and device based on tree topology structure |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110728376A CN110728376A (en) | 2020-01-24 |
| CN110728376B true CN110728376B (en) | 2024-05-03 |
Family
ID=69220489
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911000573.3A Active CN110728376B (en) | 2019-10-21 | 2019-10-21 | Federal learning method and device based on tree topology structure |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110728376B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282933B (en) * | 2020-07-17 | 2022-03-01 | 中兴通讯股份有限公司 | Federal learning method, device and system, electronic equipment and storage medium |
| WO2022066089A1 (en) * | 2020-09-23 | 2022-03-31 | Chalmers Ventures Ab | System and method for scalable machine learning in a communication network |
| CN112288094B (en) * | 2020-10-09 | 2022-05-17 | 武汉大学 | Federal network representation learning method and system |
| CN113112307A (en) * | 2021-04-30 | 2021-07-13 | 欧冶云商股份有限公司 | Steel price prediction method, device, equipment and medium based on federal learning |
| CN113515760B (en) * | 2021-05-28 | 2024-03-15 | 平安国际智慧城市科技股份有限公司 | Horizontal federal learning method, apparatus, computer device, and storage medium |
| CN114666274A (en) * | 2022-03-17 | 2022-06-24 | 广州广电运通金融电子股份有限公司 | Federal learning method, device, system and readable medium for asynchronous mode training |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5630184A (en) * | 1992-11-07 | 1997-05-13 | International Business Machines Corporation | Method of deleting and adding nodes in a spanning tree network by collating replies from other nodes |
| CN104850447A (en) * | 2014-10-15 | 2015-08-19 | 中国太原卫星发射中心 | Data collecting method based on high-rise architecture |
| CN108616380A (en) * | 2016-12-13 | 2018-10-02 | 财团法人工业技术研究院 | The tree network restoration methods and controller of software defined network |
-
2019
- 2019-10-21 CN CN201911000573.3A patent/CN110728376B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5630184A (en) * | 1992-11-07 | 1997-05-13 | International Business Machines Corporation | Method of deleting and adding nodes in a spanning tree network by collating replies from other nodes |
| CN104850447A (en) * | 2014-10-15 | 2015-08-19 | 中国太原卫星发射中心 | Data collecting method based on high-rise architecture |
| CN108616380A (en) * | 2016-12-13 | 2018-10-02 | 财团法人工业技术研究院 | The tree network restoration methods and controller of software defined network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110728376A (en) | 2020-01-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110728376B (en) | Federal learning method and device based on tree topology structure | |
| CN110942154B (en) | Data processing method, device, equipment and storage medium based on federal learning | |
| CN110163368B (en) | Deep learning model training method, device and system based on mixed precision | |
| CN109117953B (en) | Network parameter training method and system, server, client and storage medium | |
| CN110062272A (en) | A kind of video data handling procedure and relevant apparatus | |
| CN103929353B (en) | The treating method and apparatus of instant message | |
| CN109919313B (en) | Gradient transmission method and distributed training system | |
| CN107565973B (en) | Method for realizing node-extensible Huffman coding and circuit structure | |
| CN106713495B (en) | The method for uploading and access method in IP geographical position, device and access system | |
| CN111898484A (en) | Method and device for generating model, readable storage medium and electronic equipment | |
| CN113572697A (en) | Load balancing method based on graph convolution neural network and deep reinforcement learning | |
| CN109120669B (en) | Method, medium, and system for building block chain based on P2P internet | |
| CN116016199B (en) | Information control method, system, electronic equipment and readable storage medium | |
| CN110287179A (en) | A kind of filling equipment of shortage of data attribute value, device and method | |
| CN108334553B (en) | Data screening method and device based on block chain | |
| CN107301618A (en) | Based on the GPU basis matrixs accelerated parallel and homography matrix method of estimation and system | |
| CN110750363B (en) | Computer storage management method and device, electronic equipment and storage medium | |
| CN116800671A (en) | Data transmission method, apparatus, computer device, storage medium, and program product | |
| CN108898527A (en) | A kind of traffic data fill method based on the generation model for having loss measurement | |
| CN109635940A (en) | A kind of image processing method and image processing apparatus based on convolutional neural networks | |
| CN107644055B (en) | communication guarantee capability generation method based on training data | |
| CN111461403A (en) | Vehicle path planning method and device, computer readable storage medium and terminal | |
| CN117081726B (en) | Method and device for transmitting files in blocking and grading mode | |
| CN113762526B (en) | Federal learning method, hierarchical network system, storage medium and electronic device | |
| CN116684083B (en) | Inadvertent key value storage method based on two hash functions and one-way step thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |