CN115174397A - Federal edge learning training method and system combining gradient quantization and bandwidth allocation - Google Patents
Federal edge learning training method and system combining gradient quantization and bandwidth allocation Download PDFInfo
- Publication number
- CN115174397A CN115174397A CN202210896876.3A CN202210896876A CN115174397A CN 115174397 A CN115174397 A CN 115174397A CN 202210896876 A CN202210896876 A CN 202210896876A CN 115174397 A CN115174397 A CN 115174397A
- Authority
- CN
- China
- Prior art keywords
- node
- quantization
- representing
- gradient
- edge server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013139 quantization Methods 0.000 title claims abstract description 142
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012549 training Methods 0.000 title claims abstract description 33
- 238000005457 optimization Methods 0.000 claims abstract description 19
- 238000004364 calculation method Methods 0.000 claims abstract description 13
- 238000011002 quantification Methods 0.000 claims abstract description 4
- 239000013598 vector Substances 0.000 claims description 36
- 230000005540 biological transmission Effects 0.000 claims description 21
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000002079 cooperative effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000005562 fading Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention discloses a federated edge learning training method and system combining gradient quantification and bandwidth allocation. The method comprises the following steps: the node sends the parameter information of the equipment to the edge server; the edge server calculates the channel condition of each node according to the parameter information sent by each node to obtain the channel gain of each uplink channel, solves the optimization problem according to the channel gain, the sending power, the sample number and the calculation capacity, distributes the quantization bit number and the bandwidth, and broadcasts the available quantization bit number and the global model to the nodes participating in the federal learning; the node calculates a local update gradient according to the global model and the local data, quantizes the local update gradient based on the quantization bit number, and sends the quantized local gradient to the edge server; the edge server aggregates the received gradients and updates the global model. The method can effectively relieve the influence of limited edge resources and equipment isomerism on federal learning training, relieve the communication bottleneck and improve the training efficiency.
Description
Technical Field
The invention relates to the field of distributed systems, in particular to a federated edge learning training method and system combining gradient quantification and bandwidth allocation.
Background
Federal Edge Learning (fed Edge Learning) is becoming a popular distributed privacy preserving machine Learning framework, with multiple Edge devices collaboratively training a machine Learning model with the help of an Edge server. In federal edge learning, an edge device calculates the gradient of a global model from local data and uploads the gradient to an edge server in an iterative manner for model updating. However, due to the limited nature of the shared radio spectrum and the excessive training parameters, federal edge learning is often severely impacted by communication bottlenecks. One effective way to alleviate the communication bottleneck is quantization, i.e. using fewer bits to represent the gradient. Considering gradient quantization to reduce traffic flow and reduce overall training delay is a key issue in federal edge learning. The total training delay is proportional to the number of training rounds required and the delay per round. The former is closely related to the gradient quantization scheme, i.e., the quantization level of each edge device. The latter is determined jointly by the gradient quantization scheme and the bandwidth allocation scheme. The bandwidth allocation scheme specifies how the spectrum is shared between edge devices, and is closely related to gradient quantization, e.g., when an edge device uses a higher quantization level to result in greater traffic, it needs more bandwidth to shorten its transmission time. At the same time, the delay for each iteration is determined by the slowest edge device. Therefore, it is necessary to optimize the bandwidth allocation and gradient quantization selection in conjunction with all edge devices.
Currently, some schemes for joint optimization considering bandwidth allocation and gradient quantization in different environments have been proposed. Among other things, some schemes consider a most-valued-based random quantization approach, where the quantization level at each edge server depends on the dynamic range of its local gradient. Thus, joint optimization can only be performed when all edge devices have performed their local gradients, which results in the more computationally intensive edge devices having to wait for the slower devices to begin transmission. In contrast, other efforts consider another gradient-modulo based quantization method, where the variance of the quantized gradient is only related to the quantization level, so the edge server can optimize the quantization bit number allocation scheme before each round of training begins. However, although the above methods have a good effect, the following problems are generally present in these methods: (1) Only the convergence of the model training is considered, the number of training rounds is optimized, but the delay per round is not considered. (2) All nodes use the same quantization level, resulting in the results of federal learning being suboptimal.
Disclosure of Invention
The invention aims to: the invention mainly aims to relieve the communication bottleneck in edge federated learning, overcome the defects in the prior art and provide a federated edge learning training method and system for jointly optimizing gradient quantization and bandwidth allocation.
The technical scheme is as follows: in order to achieve the above object, the technical solution of the present invention is as follows: a federated edge learning training method combining gradient quantization and bandwidth allocation comprises the following steps:
each node sends the parameter information of the equipment to the edge server through an uplink channel;
the edge server estimates the channel gain and the computing capacity of the nodes according to the parameter information uploaded by the nodes, establishes an optimization problem with the minimum iteration time and the model convergence as the target and solves the optimization problem according to the channel gain, the sending power, the sample number and the computing capacity of each node, and obtains the quantization bit number and the bandwidth distributed by each node;
the edge server broadcasts available quantization bit number and a global model to the nodes participating in the federal learning;
the node calculates a local update gradient according to the global model and the local data, and quantizes the local update gradient based on the quantization bit number;
the node sends the quantized local update gradient to an edge server;
the edge server aggregates the received gradients, updates the global model, and finishes the federal learning if the global model is converged; otherwise, the steps of federated learning are performed from scratch until the global model converges.
Further, the parameter information sent by each node to the edge server includes: CPU frequency, sample number, transmission power and equipment position, wherein the CPU frequency and the sample number are used for estimating the time required by local calculation of the node, and the node position and the transmission power are used for estimating the transmission capability of the node.
Further, the optimization problem is as follows:
(P1):minσ 2 ·t round
wherein ,
represents the convergence of the model, N represents the total number of nodes, N m Number of samples, s, representing node m m Representing the quantization level of node m, Z representing the modulo upper bound of the gradient, d representing the number of model parameters, t round The time required for the iteration is indicated,
represents the time required for the calculation of node m, q m ＝
Number of quantization bits, P, representing node m m Represents the transmission power of node m, h m Representing the channel gain of node m, b m Represents the bandwidth allocated to node m, E m Representing the energy limit of node m, N 0 A white gaussian noise representing the channel is generated,
represents the energy required by the calculation stage of the node M, M represents the number of nodes participating in training, B represents the total bandwidth of an uplink channel,
refers to the number of bits, N, used to represent the element by default in the system + Representing a set of positive integers.
Further, the node quantizing the local update gradient based on the quantization bit number includes: the node receives the quantized bit number distribution information, searches the quantized bit number distributed by the node through the node number at the front end of each quantized bit number distribution unit in the quantized bit number distribution information block, and stores the quantized bit number in the storage unit; and the node quantizes the updating gradient of the local model by using a uniform quantization method by using the quantization bit number stored in the storage unit.
Further, the uniform quantization method includes: the total quantization interval is uniformly divided to obtain sub-quantization intervals, the number of the sub-quantization intervals is quantization levels, and the relation between the quantization levels s and the quantization bit number q is as follows: s =2 q 1, when the value to be quantized is located in a certain sub-quantization interval, quantizing the value to be quantized to the left end point or the right end point of the sub-quantization interval according to a specified quantization rule.
Further, the specified quantization rule is: flattening the update gradient into vectors, the values of the codebook mapped by the components of each vector are as follows:
Q s (v i )＝‖v‖sgn(v i )ξ i (v,s),i＝1,2,…,n
wherein ,Qs (v i ) To update the value of the ith component of the gradient flattened vector v, sgn (v) i ) Symbol of the ith component of vector vNumber xi i (v, s) are independent random variables, which represent the value of the ith component of the vector v according to probability, and specifically are:
wherein, the probability p (a, s) = as-l, l is a nonnegative integer less than or equal to s.
Further, the nodes employ a triplet (| v |) 2 σ and ξ) represents the quantized update gradient and transmits the information of the triplet to the edge server, wherein the specific content of the triplet is as follows: the modulus | v | of the vector 2 A vector xi consisting of a vector component sign vector sigma and a component mapping integer, wherein xi i ＝s·ξ i (v,s)。
The invention also provides a federated learning system, which comprises an edge server and a plurality of nodes, wherein each node is used for sending parameter information of self equipment to the edge server through an uplink channel, receiving the quantization bit number and the global model broadcast and sent by the edge server, calculating a local update gradient according to the global model and local data, quantizing the local update gradient based on the quantization bit number, and sending the quantized local update gradient to the edge server;
the edge server is used for estimating the channel gain and the computing capacity of the nodes according to the parameter information uploaded by the nodes, establishing an optimization problem with the aim of minimizing iteration time and model convergence and solving the optimization problem according to the channel gain, the sending power, the sample number and the computing capacity of each node, obtaining the quantization bit number and bandwidth distributed by each node, broadcasting the available quantization bit number and a global model to the nodes participating in federal learning, receiving the local update gradient of the nodes after quantization, aggregating the received gradient, and updating the global model until the global model converges.
Compared with the prior art, the invention has the following advantages and beneficial effects: (1) The invention fully utilizes the interactivity of the edge server and the nodes learned by the federation, so that the edge server performs uniform quantization which can adaptively adjust the quantization bit number of the update gradient uploaded by the nodes by acquiring the calculation time of the nodes and the quality condition of an uplink channel, and fully explores the edge-end cooperative activity. (2) The uniform quantization method adopted by the invention is changed from the fact that the whole gradient vector needs to be transmitted into the fact that only the triple containing the gradient vector information needs to be transmitted, the data size uploaded by the nodes after quantization is obviously reduced, and the quantization variance is smaller. (3) The invention has better expansion potential, can further reduce the scale of the data uploaded after quantization by using an efficient coding mode, and can add a sparse method after quantization according to the requirement to achieve better data compression effect. (4) The invention does not need to add an additional device in the original federal learning system, effectively reduces the system layout cost, is simple and effective, and has small interaction overhead among nodes.
Drawings
FIG. 1 is a system diagram of federated learning of the present invention;
FIG. 2 is a general flow diagram of the federated learning training method of the present invention;
FIG. 3 is a block of quantization bit allocation information obtained after quantization bit allocation is performed according to the present invention;
FIG. 4 is a diagram illustrating the quantification of elements in the gradient according to the present invention;
fig. 5 is a flow chart of a specific implementation of federated learning of the present invention.
Detailed Description
The technical scheme of the invention is further explained by combining the attached drawings.
Fig. 1 is a schematic diagram of a federal learning system according to the present invention, where the federal learning system includes an edge server and a plurality of nodes, where the edge server is deployed in a base station, and each edge server includes a computing unit, a storage unit, and a sending unit. The node comprises a quantization unit, a storage unit, a training unit and a sending unit. The edge server and the nodes cooperate to complete a specified task, original data required to be used by the task is distributed in the nodes, and the edge server cannot contact the original data; the exchange of local model updating and global model updating needs to be completed between the nodes and the edge servers, the nodes update the local models by using the local original data and the global models broadcasted by the edge servers, and the edge servers update the global models by using the local models updated by the nodes.
In this embodiment, since the node is usually a portable mobile smart device such as a mobile phone and a smart watch, the battery has a limited size, and cannot provide sufficient resources to send a large-scale local model update, the federal edge learning training framework for jointly optimizing gradient quantization and bandwidth allocation provided by the present invention considers the energy limit of the device, and can be used to alleviate a communication bottleneck generated when the edge server interacts with the node in the federal learning system.
Fig. 2 is a schematic flow diagram of a federated edge learning training method combining gradient quantization and bandwidth allocation according to the present invention. The method mainly comprises the following steps:
1) Uploading parameter information by the node, and estimating the channel gain of the node by the edge server;
and the nodes participating in the federal learning send equipment parameter information to the edge server, wherein the parameters comprise information such as CPU frequency, sample number, physical position, sending power and the like of the equipment. Wherein the CPU frequency and the number of samples are used to estimate the time required for local computation, and the node location and the transmit power are used to estimate the transmission capability of the node.
The method comprises the steps that an edge server receives equipment information sent by nodes, and the edge server sequentially stores the equipment information of each node in a storage unit in a queue mode; the edge server calls the equipment information of one node in the storage unit queue each time according to
Estimating the channel gain in a computing unit, where o m Is a Rayleigh fading parameter, d m Is the distance between node m and the base station, i.e. the edge server; and copying the node numbers after channel gain estimation is finished, and obtaining channel gain information of all nodes after the channel gain units of all nodes are connected end to end, and storing the channel gain information in a storage unit.
2) The edge server uses the parameters of the nodes to solve the optimization problem, allocates the quantization bit number and bandwidth which can be used by the nodes, and broadcasts a global model and the allocation condition of the quantization bit number;
according to shannon's theorem, it can be inferred that the larger the transmission power of a node is, the larger the channel gain is, the stronger the transmission capability is, but a closed solution of the transmission capability cannot be obtained from the shannon's formula, and only estimation is possible. In the present invention, according to the formula
Estimating the time required by local calculation of the node, N in the formula m Representing the number of samples owned by a node, F m Represents the node CPU frequency and gamma represents the number of CPU cycles required to process one sample per node.
The transmission capability of the node is estimated using the channel gain and the node information (including the transmission power) in the storage unit. According to the second formula of Shannon, the transmission rate r of node m m Expressed as:
wherein ,bm Represents the bandwidth, P, allocated to node m m Representing the transmission power of node m, N 0 Representing the gaussian white noise of the channel.
And the edge server substitutes the estimated channel gain information and the local calculation time of the nodes into an optimization problem with unknown parameters and determined specific models, respectively solves the quantization bit number and bandwidth distributed by each node to obtain a quantization bit number distribution unit, obtains a quantization bit number distribution information block after head-to-tail connection, and stores and sends the quantization bit number distribution information block in a queue form. The computing unit of the edge server is responsible for solving the optimization problem and aggregating the received gradients, the storage unit is responsible for storing parameter information of the data nodes and quantized bit number distribution information blocks, and the sending unit is responsible for broadcasting data to the nodes, such as quantized bit number distribution information blocks and a global model.
Fig. 3 shows a quantization bit number allocation information block obtained after quantization bit number allocation. The node number refers to a sequence number that distinguishes different nodes in the edge server.
According to the embodiment of the invention, the optimization problem is specifically as follows:
(P1):minσ 2 ·t round
wherein ,
denotes the convergence of the model, N denotes the total number of nodes, N m Number of samples, s, representing node m m The quantization level of the node m is represented, Z represents the modulo upper bound of the gradient, and d represents the number of model parameters. t is t round Representing the time required for an iteration, P m Represents the transmission power, h, of node m m Representing the channel gain of node m, b m Represents the bandwidth allocated to node m, E m Representing the energy limit of node m, q m ＝
Represents the number of quantization bits of the node m,
represents the time required for the node m to calculate, N 0 A white gaussian noise representing the channel is generated,
representing the energy required by the computing stage of the node m, the invention assumes that the power of the computing stage of the node is fixed, so the energy required by the computing stage
And calculating time
Proportional ratio, which can be calculated according to CPU frequency and calculation time, and formula is
γ m Representing coefficients related to the system architecture, M representing the number of nodes participating in the training, B representing the total bandwidth of the uplink channel,
refers to the number of bits used by the default presentation element in the system, typically 32 bits, N + Representing a set of positive integers.
The quantization bit number and the bandwidth are unknown parameters, and the bandwidth and the quantization bit number jointly determine the time required by the node for transmitting the gradient.
3) The node performs local training for a plurality of times by using the received global model and locally stored data to obtain updated local gradient information, and then quantizes the updated local gradient according to the allocated quantization bit number;
the node receives the quantized bit number distribution information block, and finds out the quantized bit number distributed by the node through the node serial number at the front end of each quantized bit number distribution unit in the quantized bit number distribution information block; and the node quantizes the updating gradient of the local model by using a uniform quantization method by using the quantization bit number stored in the storage unit. The storage unit of the node is responsible for storing training data and a global model, the training unit is responsible for using the data to train the global model to obtain gradients, the quantization unit is responsible for quantizing the trained gradients, and the sending unit is responsible for sending the quantized gradients to the edge server.
According to the embodiment of the present invention, the sub-quantization intervals in the uniform quantization method are obtained by uniformly dividing the total quantization interval, wherein when the value to be quantized is located in a certain sub-quantization interval, the value is quantized into the left end point or the right end point of the sub-quantization interval according to a certain quantization rule, and the specific quantization rule of the uniform quantization method is as follows: flattening the updated gradient into a vector, where the gradient is a multidimensional matrix, for example, the shape of the matrix may be 3 × 4 × 5, the spread vector is 60 × 1, one element in the vector is a component, and the codebook mapped by the component of each vector has the following value rule:
Q s (v i )＝‖v‖sgn(v i )ξ i (v,s),i＝1,2,…,n
wherein ,Qs (v i ) To update the value of the ith component of the gradient flattened vector v, sgn (v) i ) Is the sign, ξ, of the ith component of vector v i (v, s) are independent random variables, which represent the value of the ith component of the vector v according to probability, and specifically are:
wherein, the probability p (a, s) = as-l, l is a nonnegative integer less than or equal to s; note that a is a form factor and the substantial factor is
In the invention, the quantization level is the number of sub-quantization intervals, and the relation between the quantization level s and the quantization bit number q is as follows: s =2 q -1, or is represented by
The value of one of the values may be determined based on the other.
After the quantization of the uniform quantization method,using triplets (| v |) 2 σ and ξ) to represent the update gradient after quantization, only the information of a triplet needs to be transmitted during transmission, and the specific content of the triplet is as follows: the modulus | v | of the vector 2 A symbol vector sigma of the vector components formed in the original order and a vector xi formed by the integers of the component mapping, wherein xi i ＝s·ξ i (v,s)，ξ i Representing the ith component of vector ξ.
A schematic diagram of the quantization method is shown in fig. 4. The figure shows that one component in a vector is quantized and the component is quantized
After operation, each component can be ensured to fall in the 0,1]Within the interval, thereby converting to [0,1]In the above quantization, the ratio of the final up-down value probabilities is the ratio of the distances from the quantization points to the up-down quantization levels. In FIG. 4, a quantization level of 4, component to vector modulo ratio
The figure is 0.6, and the specific quantization rule is shown in the above formula.
4) The node uploads the quantized local update gradient to an edge server through an independent and irrelevant uplink channel;
in order to ensure that the signals transmitted in each sub-channel do not interfere with each other, isolation zones should be set between each sub-channel, so as to ensure that the signals of each channel do not interfere with each other. Illustratively, the uplink channel uses a frequency division multiplexing channel because frequency division multiplexing requires a total frequency width greater than the sum of the individual subchannel frequencies.
5) And the edge server receives the quantized update gradient, aggregates the update gradient, and uses the aggregated gradient to update the global model on the edge server.
The invention fully utilizes the interactivity of the edge server and the nodes learned by the federation, so that the edge server performs uniform quantization which can adaptively adjust the quantization bit number of the update gradient uploaded by the nodes by acquiring the calculation time of the nodes and the quality condition of an uplink channel, and fully explores the edge-end cooperative activity. The invention considers the convergence of model training and the calculation and communication delay of each round at the same time, allocates different quantization levels for the nodes, obtains the best balance between the training round and the delay of each round, and solves the defects of the previous work.
FIG. 5 shows a specific process of federated learning in one embodiment, including the following steps: (1) The edge server selects a node participating in the federate learning iteration; (2) uploading equipment parameter information by the node; (3) If the global iteration is primary global iteration, the edge server initializes the model and broadcasts the initialized global model to the nodes participating in training; otherwise, broadcasting the updated global model obtained after the last global iteration to the nodes participating in the training; (4) The edge server estimates the node channel gain and distributes the quantization bit number which can be used by the node; (5) The edge server broadcasts the distribution condition of the quantized bit number and a global model; (6) The node obtains the global model and the quantized bit distribution number obtained by the node from the broadcast information. (7) The node performs local training for one time or a plurality of times by using the received global model and locally stored data to obtain updated local gradient information; (8) The node quantizes the updated local gradient information by using the allocated quantized bit number to obtain a quantized local update gradient for uploading; (9) The node uploads the quantized local update gradient to an edge server through an independent and irrelevant uplink channel; (10) The edge server receives the actual updating gradient disturbed by the node channel, the actual updating gradient is aggregated, the aggregated gradient is used for updating a global model on the edge server, and if the global model is converged, the federal learning is finished; otherwise, the steps of federated learning are performed from scratch until the global model converges.
According to the method, an edge server is brought into an optimization problem model according to parameter information uploaded by a node, the quantization bit number q and the bandwidth b of the node are solved and are issued to the node, the node determines the quantization grade s according to the relation between the quantization bit number and the quantization grade, then determines the quantization value according to the quantization grade s and the vector v after updating gradient flattening, and uploads the triple to the edge server based on the bandwidth b after forming the triple.
Claims (10)
1. A federated edge learning training method combining gradient quantization and bandwidth allocation is characterized by comprising the following steps:
each node sends the parameter information of the equipment to the edge server through an uplink channel;
the edge server estimates the channel gain and the computing capacity of the nodes according to the parameter information uploaded by the nodes, establishes an optimization problem with minimized iteration time and model convergence as targets according to the channel gain, the sending power, the sample number and the computing capacity of each node, and solves the optimization problem to obtain the quantization bit number and the bandwidth distributed by each node;
the edge server broadcasts available quantization bit number and a global model to nodes participating in federal learning;
the node calculates a local update gradient according to the global model and the local data, and quantizes the local update gradient based on the quantization bit number;
the node sends the quantized local update gradient to an edge server;
the edge server aggregates the received gradients, updates the global model, and if the global model is converged, the federal learning is ended; otherwise, the steps of federated learning are performed from scratch until the global model converges.
2. The method of claim 1, wherein the parameter information sent by each node to the edge server comprises: CPU frequency, sample number, transmission power and equipment position, wherein the CPU frequency and the sample number are used for estimating the time required by local calculation of the node, and the node position and the transmission power are used for estimating the transmission capability of the node.
3. The method of claim 1, wherein the optimization problem is:
(P1):minσ 2 ·t round
s.t.
wherein ,
denotes the convergence of the model, N denotes the total number of nodes, N m Number of samples, s, representing node m m Representing the quantization level of node m, Z representing the modulo upper bound of the gradient, d representing the number of model parameters, t round The time required for the iteration is indicated,
representing the time required for the node m to compute,
number of quantization bits, P, representing node m m Represents the transmission power, h, of node m m Representing the channel gain of node m, b m Represents the bandwidth allocated to node m, E m Representing the energy limit of node m, N 0 A white gaussian noise representing the channel is generated,
represents the energy required by the calculation stage of the node M, M represents the number of nodes participating in training, B represents the total bandwidth of an uplink channel,
refers to the number of bits, N, used to represent the element by default in the system + Representing a set of positive integers.
4. The method of claim 1, wherein the node quantizing the local update gradient based on the number of quantization bits comprises: the node receives the quantized bit number distribution information, searches the quantized bit number distributed by the node through the node number at the front end of each quantized bit number distribution unit in the quantized bit number distribution information block, and stores the quantized bit number in the storage unit; and the node quantizes the updating gradient of the local model by using a uniform quantization method by using the quantization bit number stored in the storage unit.
5. The method of claim 4, wherein the uniform quantization method comprises: the total quantization interval is uniformly divided to obtain sub-quantization intervals, the number of the sub-quantization intervals is quantization levels, and the relation between the quantization levels s and the quantization bit number q is as follows: s =2 q And 1, when the value to be quantized is positioned in a certain sub-quantization interval, quantizing the value into the left end point or the right end point of the sub-quantization interval according to a specified quantization rule.
6. The method of claim 5, wherein the specified quantization rule is: flattening the update gradient into vectors, the values of the codebook mapped by the components of each vector are as follows:
Q s (v i )＝‖v‖sgn(v i )ξ i (v,s),i＝1,2,…,n
wherein ,Qs (v i ) To update the value of the ith component of the gradient flattened vector v, sgn (v) i ) Is the i-th component of the vector vSymbol of (xi) i (v, s) are independent random variables, which represent the value of the ith component of the vector v according to probability, and specifically are:
wherein, the probability p (a, s) = as-l, l is a nonnegative integer less than or equal to s.
7. The method of claim 6, wherein the nodes employ a triplet (| v |) 2 σ, ξ) represents the update gradient after quantization, and information of the triplet is transmitted to an edge server, and the specific content of the triplet is as follows: the modulus | v | of the vector 2 A vector xi consisting of a vector component sign vector sigma and a component mapping integer, wherein xi i ＝s·ξ i (v,s)，ξ i Representing the ith component of vector ξ.
8. The federated learning system comprises an edge server and a plurality of nodes, and is characterized in that each node is used for sending parameter information of self equipment to the edge server through an uplink channel, receiving a quantization bit number and a global model broadcast and sent by the edge server, calculating a local update gradient according to the global model and local data, quantizing the local update gradient based on the quantization bit number, and sending the quantized local update gradient to the edge server;
the edge server is used for estimating the channel gain and the computing capacity of the nodes according to the parameter information uploaded by the nodes, establishing an optimization problem with the aim of minimizing iteration time and model convergence and solving the optimization problem according to the channel gain, the sending power, the sample number and the computing capacity of each node, obtaining the quantization bit number and bandwidth distributed by each node, broadcasting the available quantization bit number and a global model to the nodes participating in federal learning, receiving the local update gradient of the nodes after quantization, aggregating the received gradient, and updating the global model until the global model converges.
9. The federated learning system of claim 8, wherein the edge server builds the following optimization problem:
(P1):minσ 2 ·t round
wherein ,
represents the convergence of the model, N represents the total number of nodes, N m Number of samples, s, representing node m m Representing the quantization level of node m, Z representing the modulo upper bound of the gradient, d representing the number of model parameters, t round The time required for the iteration is indicated,
representing the time required for the node m to compute,
number of quantization bits, P, representing node m m Represents the transmission power, h, of node m m Representing the channel gain of node m, b m Represents the bandwidth allocated to node m, E m Representing the energy limit of node m, N 0 A white gaussian noise representative of the channel is generated,
represents the energy required by the calculation stage of the node M, M represents the number of nodes participating in training, B represents the total bandwidth of an uplink channel,
refers to the number of bits, N, used to represent the element by default in the system + Representing a set of positive integers.
10. The federated learning system of claim 8, wherein the node quantifies the update gradient of the local model using a uniform quantification method that includes: the total quantization interval is uniformly divided to obtain sub-quantization intervals, the number of the sub-quantization intervals is quantization levels, and the relation between the quantization levels s and the quantization bit number q is as follows: s =2 q And 1, when the value to be quantized is positioned in a certain sub-quantization interval, quantizing the value into the left end point or the right end point of the sub-quantization interval according to a specified quantization rule.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210896876.3A CN115174397B (en) | 2022-07-28 | 2022-07-28 | Federal edge learning training method and system combining gradient quantization and bandwidth allocation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210896876.3A CN115174397B (en) | 2022-07-28 | 2022-07-28 | Federal edge learning training method and system combining gradient quantization and bandwidth allocation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115174397A true CN115174397A (en) | 2022-10-11 |
| CN115174397B CN115174397B (en) | 2023-10-13 |
Family
ID=83497319
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210896876.3A Active CN115174397B (en) | 2022-07-28 | 2022-07-28 | Federal edge learning training method and system combining gradient quantization and bandwidth allocation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115174397B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115830709A (en) * | 2022-11-23 | 2023-03-21 | 深圳市大数据研究院 | Action recognition method based on federal edge learning, server and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021121585A1 (en) * | 2019-12-18 | 2021-06-24 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods for cascade federated learning for telecommunications network performance and related apparatus |
| CN113098806A (en) * | 2021-04-16 | 2021-07-09 | 华南理工大学 | Method for compressing cooperative channel adaptability gradient of lower end in federated learning |
| CN113315604A (en) * | 2021-05-25 | 2021-08-27 | 电子科技大学 | Adaptive gradient quantization method for federated learning |
| CN113435604A (en) * | 2021-06-16 | 2021-09-24 | 清华大学 | Method and device for optimizing federated learning |
| CN113723620A (en) * | 2020-05-25 | 2021-11-30 | 株式会社日立制作所 | Terminal scheduling method and device in wireless federal learning |
| CN114599096A (en) * | 2022-02-17 | 2022-06-07 | 南京邮电大学 | Mobile edge calculation unloading time delay optimization method and device and storage medium |
-
2022
- 2022-07-28 CN CN202210896876.3A patent/CN115174397B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021121585A1 (en) * | 2019-12-18 | 2021-06-24 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods for cascade federated learning for telecommunications network performance and related apparatus |
| CN113723620A (en) * | 2020-05-25 | 2021-11-30 | 株式会社日立制作所 | Terminal scheduling method and device in wireless federal learning |
| CN113098806A (en) * | 2021-04-16 | 2021-07-09 | 华南理工大学 | Method for compressing cooperative channel adaptability gradient of lower end in federated learning |
| CN113315604A (en) * | 2021-05-25 | 2021-08-27 | 电子科技大学 | Adaptive gradient quantization method for federated learning |
| CN113435604A (en) * | 2021-06-16 | 2021-09-24 | 清华大学 | Method and device for optimizing federated learning |
| CN114599096A (en) * | 2022-02-17 | 2022-06-07 | 南京邮电大学 | Mobile edge calculation unloading time delay optimization method and device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 屈志昊，叶保留，陈贵海，唐斌: "面向边缘计算的资源优化技术研究进展", 《大数据》, pages 17 - 28 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115830709A (en) * | 2022-11-23 | 2023-03-21 | 深圳市大数据研究院 | Action recognition method based on federal edge learning, server and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115174397B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111314889B (en) | Task unloading and resource allocation method based on mobile edge calculation in Internet of vehicles | |
| CN113098806B (en) | Method for compressing cooperative channel adaptability gradient of lower end in federated learning | |
| CN110167176B (en) | Wireless network resource allocation method based on distributed machine learning | |
| CN112105062B (en) | Mobile edge computing network energy consumption minimization strategy method under time-sensitive condition | |
| CN113139662A (en) | Global and local gradient processing method, device, equipment and medium for federal learning | |
| CN113222179A (en) | Federal learning model compression method based on model sparsification and weight quantization | |
| US9723572B2 (en) | Systems and methods for uplink power control and scheduling in a wireless network | |
| CN112084025B (en) | Improved particle swarm optimization based time delay optimization method for unloading of fog calculation task | |
| CN114142907B (en) | Channel screening optimization method and system for communication terminal equipment | |
| CN111475301A (en) | Satellite resource allocation method and device and electronic equipment | |
| CN111629380A (en) | Dynamic resource allocation method for high-concurrency multi-service industrial 5G network | |
| CN105379412A (en) | System and method for controlling multiple wireless access nodes | |
| CN110233755B (en) | Computing resource and frequency spectrum resource allocation method for fog computing in Internet of things | |
| CN109831808B (en) | Resource allocation method of hybrid power supply C-RAN based on machine learning | |
| CN111182509B (en) | Ubiquitous power Internet of things access method based on context-aware learning | |
| WO2023179010A1 (en) | User packet and resource allocation method and apparatus in noma-mec system | |
| CN112911608A (en) | Large-scale access method for edge-oriented intelligent network | |
| CN115174397B (en) | Federal edge learning training method and system combining gradient quantization and bandwidth allocation | |
| CN114051222A (en) | Wireless resource allocation and communication optimization method based on federal learning in Internet of vehicles environment | |
| CN114866545A (en) | Semi-asynchronous layered federal learning method and system based on air calculation | |
| WO2024050659A1 (en) | Federated learning lower-side cooperative channel adaptive gradient compression method | |
| CN116128070B (en) | Federal learning method based on wireless air calculation and multi-bit quantization compressed sensing | |
| CN117580063A (en) | Multi-dimensional resource collaborative management method in vehicle-to-vehicle network | |
| CN113179550B (en) | Resource allocation method for distributed wireless energy and information transmission system | |
| CN116133127A (en) | Knowledge-driven resource scheduling method in Internet of vehicles in space-sky-earth integrated scene |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |