CN113902021A - High-energy-efficiency clustering federal edge learning strategy generation method and device - Google Patents

Abstract

The invention discloses a clustering federal edge learning strategy generation method and a clustering federal edge learning strategy generation device with high energy efficiency, wherein the method comprises the following steps: s1, initializing an edge access strategy by the cloud center; s2, the edge base station solves the bandwidth resource allocation strategy of the access equipment and sends the initialization model to the access equipment; s3, calculating the precision of the received global model by the equipment, training the local model by adopting a layered migration strategy according to the global model and the local data, calculating the energy spent on uploading the local model, taking the difference value between the test precision and the energy consumption as the local profit, and uploading the local model and the local profit to the accessed edge base station; s4, the edge base station hierarchically aggregates the local model, calculates edge income by averaging local income of all access devices, and uploads the edge income to the cloud center; s5, the cloud center calculates the system profit according to the received feedback information of the edge base station, and adjusts an edge access strategy by adopting a deep reinforcement learning algorithm; and S6, repeating the above processes until convergence.

Description

An energy-efficient clustering federated edge learning strategy generation method and device

technical field

The invention relates to the technical field of data processing, in particular to a method and device for generating a clustering federated edge learning strategy with high energy efficiency.

Background technique

Data security has become a key issue facing the sustainable development of artificial intelligence technology. The traditional machine learning technology is centralized, which collects the data of the device to the processing center for centralized training, however, this may lead to the leakage of user data privacy.

Federated learning is a promising distributed machine learning architecture. With the improvement of the computing power of the device, the device can use the collected data to train the local model locally, and then only need to upload the local model to the processing center for model aggregation, avoiding the need for The raw data is uploaded directly, which greatly protects data privacy.

In real life, the data between devices may exhibit non-IID characteristics, which brings challenges for federated learning to train a unified global model. Therefore, it is very interesting to study how federated learning adapts to the data on each device. At present, some studies on personalized federated learning have been proposed.

Personalized federated learning includes federated transfer learning and federated meta-learning. Its essence is to first obtain a basic global model shared by all devices, and then fine-tune the global model on each device based on local data to adapt to personalized data features. Different personalized federated learning strategies have their own drawbacks. Federated transfer learning and federated meta-learning are only suitable for data with weak heterogeneity because they need to obtain a global model covering most of the features first, and then personalize it, so they cannot deal with the personality of a system with strong heterogeneous data. ization problem.

Multi-task federated learning is also an effective method to solve the personalization of federated learning. It calculates the correlation matrix to quantify the similarity of different device models, and then uses heterogeneous data as different learning goals to perform multi-task learning. Federated multi-task learning is only suitable for convex or biconvex problems, and it is difficult to extend to non-convex problems such as common neural networks, which has limitations. In addition, most of these personalized methods are suitable for outputting data with different labels. For example, each device only has a subset of all labels, and cannot be applied to some data with different conditional distributions and obvious clustering structure.

Cluster federated learning can effectively solve the above problems. It can capture the clustering structure between data, so as to aggregate multiple models according to the data distribution to meet the heterogeneous data characteristics between devices, which greatly improves the learning accuracy. Due to the privacy of federated learning, the data distribution on the device is unknown, which brings great challenges to clustering. It can be seen from theoretical analysis that when the distance between the learning models is smaller, the data distribution of the two is closer. Therefore, without uploading the original data, cluster federated learning mostly uses the model distance to measure the similarity of data on different devices. sex. Commonly used indicators to measure model distance are Euclidean distance and cosine distance. However, some techniques can infer data information at the device from the local model, resulting in data privacy leakage. The model nonlinear encryption method can solve this problem well, but the distance between the models after nonlinear encryption may not be proportional to the original model distance, so the use of local model distance clustering is less computationally complex, but in this case However, the similarity between the data cannot be judged by the encrypted model, which leads to the failure of the clustering method, so this is not a method that can be widely applied. In addition, most of the existing cluster federated learning only considers the statistical heterogeneity of data, ignoring the resource constraints and communication bottlenecks of the system. At the same time, these studies only consider the single base station scenario and lack the expansion to multiple base stations. For devices with limited energy, the communication overhead cannot be ignored. The spectrum resources provided by a single base station are limited. For devices with poor channel status, uploading the local model will consume a lot of device energy, thus reducing the learning performance under the training cost budget. .

Traditional federated learning requires the device to upload the local model to the cloud for aggregation through the WAN, and the battery capacity of the device is often limited. The multiple rounds of communication iterations of federated learning and the huge communication overhead in each round will consume a lot of transmission energy. Thereby reducing the learning performance for a given energy budget. Multi-access edge computing (MEC) technology is a promising distributed computing framework that can support the needs of many low-latency and low-energy applications. MEC offloads delay-sensitive and computationally intensive tasks to the edge, enabling real-time and High energy efficiency. Federated edge learning takes advantage of MEC, adding multiple base stations between the cloud and the device to further assist training, and the device uploads the local model to the edge base station for aggregation. This greatly reduces the communication overhead between the device and the cloud through the WAN transmission. In addition, through the overall coordination of edge base stations and devices, the system achieves high energy efficiency and high precision in the case of non-IID data. In the multi-base station federated learning architecture, most of them only consider the training cost, such as time and energy consumption, but do not consider the opportunities and challenges brought by statistical heterogeneity to multi-base station scenarios, and lack joint optimization research on training cost and learning performance. .

SUMMARY OF THE INVENTION

Aiming at the above-mentioned shortcomings of the prior art, the present invention, in the multi-edge base station scenario, jointly considers the data distribution and energy consumption cost of the equipment, finds the intersection of statistical heterogeneity and communication bottleneck, and designs high energy efficiency and High-precision edge access strategy and resource allocation strategy, and an energy-efficient clustering federated edge learning strategy generation method and device are proposed.

In order to achieve the above object, the present invention provides the following technical solutions:

In a first aspect, the present invention provides an energy-efficient method for generating a clustering federated edge learning strategy, comprising the following steps:

S1. The cloud center initializes the edge access policy;

S2, the edge base station uses the convex optimization method to solve the bandwidth resource allocation strategy of its access device, and sends its initialization model to the access device;

S3. The device calculates the accuracy of the received global model on the local test data set, and uses the hierarchical federated migration method to train the local model according to the global model and local training data, calculates the energy spent on uploading the local model, and compares the test accuracy and energy consumption. The difference is used as the local revenue, and then the local model and local revenue are uploaded to the connected edge base station;

S4. The edge base station aggregates the local model hierarchically, and at the same time calculates the edge revenue by averaging the local revenue of all access devices, and uploads the edge revenue to the cloud center;

S5. The cloud center calculates the system revenue according to the feedback information received from the edge base station, and uses the deep reinforcement learning algorithm to adjust the edge access strategy;

S6. Repeat the above process until convergence.

Further, in step S1, the access policy a _ij between the device and the edge base station is a binary variable, that is, if the device i communicates with the edge base station j, then a _ij =1, otherwise, a _ij =0, each device accesses an edge base station.

给定边缘接入策略时，资源分配子问题的最优带宽分配β_ij计算公式如下：Further, the convex optimization method in step S2 is specifically: for edge base station j and its access device cluster

When the edge access strategy is given, the calculation formula of the optimal bandwidth allocation β _ij for the resource allocation sub-problem is as follows:

a_ij表示设备与边缘基站的接入策略，β_ij表示分配给设备i的带宽比例。where h _ij represents the channel gain between device i and edge server j, pi represents the model upload power of device _i , N ₀ represents the power spectral density of Gaussian noise, and β _ij B _j represents device i accessing edge base station j The allocated bandwidth resources, the devices accessing the edge base station j share the public spectrum with the bandwidth B _j for communication,

a _ij represents the access policy between the device and the edge base station, and β _ij represents the bandwidth ratio allocated to device i.

进行训练，设备i的损失函数公式为：Further, the device utilizes local data according to the received global model θ _j

For training, the loss function formula of device i is:

The device uses gradient descent to update the local model ω _i with the following formula:

Among them, η is the learning step size, and η≥0;

Step S3 adopts the hierarchical federated transfer learning strategy to train the local model, and divides the neural network into a basic feature layer and a personality feature layer. The specific process of the hierarchical federated transfer learning strategy is as follows:

S301. Calculate the average learning accuracy of each edge base station after a certain round of times according to the following formula:

和个性特征层模型

上传到接入的边缘基站，低于平均精度的设备上传基础特征层模型，个性特征层模型在设备本地更新，公式如下：S302. Set the equipment basic feature layer model with higher than average precision

and personality feature layer model

Uploaded to the connected edge base station, the device with lower than average precision uploads the basic feature layer model, and the individual feature layer model is updated locally on the device. The formula is as follows:

为设备i的本地个性特征层模型，

为迁移设备集。in,

is the local personality feature layer model of device i,

For the migration device set.

S303, the edge base station aggregates the basic feature layer models of all devices and aggregates the personality feature layer models of the non-migration devices, the edge base station delivers the aggregated base layer model to all access devices, and delivers the personality layer model to the non-migration devices The device, the device performs the above update according to the received model, and so on until convergence.

Further, in step S3, the learning accuracy g _ij of the global model on the local test data set is used as an index to measure the performance of the global model on the edge base station j, and the learning performance gain G of the system is the average accuracy of all devices, as shown in the formula. :

Further, in step S3, the formula for the energy _Eij consumed by the device i uploading the local model is as follows:

T _ij is the transmission delay for device i to upload the local model to the edge base station. The formula is as follows:

S represents the size of the local model, ri _ij is the transmission rate of the model uploaded by device i, and the formula is as follows:

a_ij表示设备与边缘基站的接入策略，β_ij表示分配给设备i的带宽比例。进一步地，步骤S4中，在分层聚合之前，边缘基站聚合所有收到的本地模型，公式如下：h _ij represents the channel gain between device i and edge server j, pi represents the model upload power of device _i , N ₀ represents the power spectral density of Gaussian noise, β _ij B _j is the distribution of device i accessing edge base station j The obtained bandwidth resources, the devices accessing the edge base station j share the public spectrum with the bandwidth B _j for communication,

a _ij represents the access policy between the device and the edge base station, and β _ij represents the bandwidth ratio allocated to device i. Further, in step S4, before hierarchical aggregation, the edge base station aggregates all received local models, and the formula is as follows:

为所有接入边缘基站j的设备簇，ω_i为本地模型。in,

is all the device clusters accessing edge base station j, and ω _i is the local model.

After a certain round of training, the hierarchical federated transfer learning strategy is executed, and the edge base station aggregates the received local models hierarchically. The specific method is as follows: the edge base station aggregates the basic feature layer models of all devices to ensure the generalization performance of the model, and aggregates The personality feature layer model of non-migrated devices to eliminate the influence of non-IID data between devices, the formula is as follows:

为接入边缘基站j的设备的基础特征层全局模型，为所有设备共享，

为个性特征层全局模型，为非迁移设备共享，

为接入边缘基站j的非迁移设备簇。in,

is the global model of the basic feature layer of the device accessing the edge base station j, shared by all devices,

is a global model of the personality feature layer, shared for non-migration devices,

is a cluster of non-migrated devices accessing edge base station j.

Further, the formula of the system revenue function in step S5 is as follows:

where μ is a continuous variable, and μ∈[0,1], which is used to adjust the trade-off relationship between learning performance and transmission energy consumption, and G _max and E _max are the highest accuracy and maximum energy consumption that can be achieved by the system.

Further, in step S5, deep reinforcement learning is used to adjust the edge access strategy, and the specific process of deep reinforcement learning is:

具体细节如下：S501. Describe the edge correlation problem as a Markov process

The specific details are as follows:

在第k轮，状态定义为S(k)＝{S₁(k),S₂(k),…,S_N(k)}，每一项S_i(k)定义为：(1) Status

In the kth round, the state is defined as S(k)={S ₁ (k), S ₂ (k),...,S _N (k)}, and each term S _i (k) is defined as:

S _i (k)={A _i (k-1),β _ij (k),Δ _i (k)}

Among them, Δ _i (k) indicates whether the learning accuracy is improved compared to the k-1 round, that is, Δ _i (k)=1 means that the accuracy is improved, otherwise, Δ _i (k)=0;

在第k轮，动作为每一个设备的边缘关联策略：(2) Action

In the kth round, the action is the edge association policy for each device:

A(k)={A ₁ (k), A ₂ (k), ..., A _N (k)}

Each of them A _i (k) can be expressed as:

A _i (k)={a _ij (k)}

把奖励设置为目标函数：(3) Rewards

Set the reward as the objective function:

S502, select DQN as the basic framework, and combine dueling DQN and double DQN to optimize our algorithm, use D3QN to solve the edge access problem, the Q value function Q(S, A; θ) is approximated by a neural network with a parameter of θ , represents the mapping relationship between the environment and the action, and the output of the neural network is obtained through the Bellman equation:

Among them, S', A', θ' are the state, action and corresponding parameters of the next time slot respectively;

Two Q-networks with the same structure but different parameters are used in DQN to improve the stability of the algorithm, one is the current Q-network with the latest parameters to evaluate the value function of the current state-action, and the other is the one with the past rounds The goal of the parameter Q network, and keep the Q value unchanged for a period of time, taking the Q value of the current Q network as the input of the neural network, the goal of DQN is to minimize the difference between the two Q networks, and it is defined as The loss function of DQN:

L(θ)=E[(yQ(S, A; θ)) ² ]

S503, using the DDQN algorithm, select the action corresponding to the maximum Q value in the current Q network:

Then bring the selected action into the target Q network to calculate the Q value:

y=R(S,A)+γQ'(φ(S'), ^Amax (S';θ);θ')

S504. Use dueling DQN to optimize the network structure, and divide the network into two parts, which are the value function V(S, θ, α) related only to the state and the potential function A(S, A, α) related to both the state and the action. θ, β), where θ is the common parameter of the two networks, α is the unique parameter of the value function, β is the unique parameter of the potential function, and the Q value is the sum of these two functions:

Q(S,A,θ,α,β)=V(S,θ,α)+A(S,A,θ,β).

In a second aspect, the present invention provides an energy-efficient clustering federated edge learning strategy generation device, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that , when the computer processor executes the computer program, the above energy-efficient clustering federated edge learning strategy generation method is implemented.

Compared with the prior art, the beneficial effects of the present invention are:

1. The present invention jointly considers the actual bottleneck of implementing federated learning in reality, that is, the non-IID characteristics of device data and the communication and energy constraints of devices, while most studies only consider a single problem.

2. The present invention takes into account the communication overhead of the device, and the present invention extends the traditional federated learning of a single base station to a multi-base station scenario. Different from only solving the communication bottleneck problem in the multi-base station scenario, the present invention designs a high-precision and energy-efficient edge access strategy and resource allocation strategy by jointly considering the heterogeneity of data distribution and channel state from the perspective of system benefits.

3. In order to increase the universality of the algorithm, the present invention takes into account the data privacy problem of federated learning, especially some technologies can infer the data on the device side from the model uploaded by the device. Non-linear privacy encryption is an algorithm to further protect data privacy, Therefore, the commonly used method of model distance clustering is invalid. The invention designs deep reinforcement learning, adaptively explores edge access strategies according to edge feedback information, and protects data privacy. At the same time, in order to increase the scalability of the algorithm and reduce the complexity of the algorithm, the present invention decouples the resource allocation problem to the edge base station to solve it independently.

4. The present invention considers the situation that devices with inconsistent data distribution may access the same edge base station, and for this purpose, hierarchical migration learning is designed to further improve the learning performance. The analysis shows that the hierarchical migration strategy designed in the present invention does not consume extra energy.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only described in the present invention. For some of the embodiments, those of ordinary skill in the art can also obtain other drawings according to these drawings.

FIG. 1 is an architecture of a cluster federated edge learning system provided by an embodiment of the present invention.

Detailed ways

In order to better understand the technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

The present invention provides an energy-efficient method for generating a clustering federated edge learning strategy, comprising the following steps:

S1. The cloud center initializes the edge access policy;

S2, the edge base station uses the convex optimization method to solve the bandwidth resource allocation strategy of its access device, and sends its initialization model to the access device;

S3. The device calculates the accuracy of the received global model on the local test data set, and uses the hierarchical federated migration method to train the local model according to the global model and local training data, calculates the energy spent on uploading the local model, and compares the test accuracy with the performance. The difference in consumption is used as the local revenue, and then the local model and local revenue are uploaded to the connected edge base station;

S4. The edge base station aggregates the local model hierarchically, and at the same time calculates the edge revenue by averaging the local revenue of all access devices, and uploads the edge revenue to the cloud center;

S5. The cloud center calculates the system revenue according to the feedback information received from the edge base station, and uses the deep reinforcement learning algorithm to adjust the edge access strategy;

S6. Repeat the above process until convergence.

作为边缘基站集合，

为边缘基站数量，

为设备集合，

为设备数量。对于每一个设备

采集并储存训练数据集

其中x_in为设备i的第n个储存样本，y_in为x_in的对应标签，

为设备i的训练数据量。不同设备的训练数据采集自不同的数据源，故联邦学习的训练数据非独立同分布。Among them, the present invention considers a cluster federated edge learning architecture in a multi-base station scenario, as shown in FIG. 1 , which consists of a cloud center S, M edge base stations and N devices. in the network, with

As a set of edge base stations,

is the number of edge base stations,

for the device collection,

is the number of devices. for each device

Collect and store training datasets

where x _in is the nth storage sample of device i, y _in is the corresponding label of x _in ,

is the amount of training data for device i. The training data of different devices is collected from different data sources, so the training data of federated learning is not independent and identically distributed.

In cluster federated edge learning, the goal of the system is to learn multiple models to satisfy heterogeneous data on devices. The federated learning training process includes the following steps:

The edge base station sends the initial global model to the device;

进行训练。设备i的损失函数定义为：The device utilizes local data according to the received global model θ _j

to train. The loss function for device i is defined as:

The device updates the local model ω _i using gradient descent as follows:

Among them, η is the learning step size, and η≥0.

And upload the updated local model to the connected edge base station through the wireless link;

The edge base station aggregates all received local models as follows:

为所有接入边缘基站j的设备簇。in,

is all the device clusters accessing edge base station j.

Repeat the above process until the model converges.

The access policy a _ij between the device and the edge base station is a binary variable, that is, if the device i communicates with the edge base station j, a _ij =1; otherwise, a _ij =0. Each device can only access one edge base station, so the present invention has:

In the present invention, the learning accuracy g _ij of the global model on the local data set is used as an index to measure the performance of the global model on the edge base station j, and the learning performance gain G of the system can be regarded as the average accuracy of all devices, as shown below:

It is worth noting that the access of devices with non-IID training data to the same edge base station has a negative impact on the learning performance, so statistical heterogeneity is a key issue that cannot be ignored when designing edge access strategies.

For the uploading process of the local model, the present invention adopts an orthogonal frequency division multiple access (orthogonal frequency division multiple access, OFDMA) communication standard, which is also easily extended to other communication standard. All devices accessing edge base station j share a common spectrum with available bandwidth B _j for communication, and β _ij represents the bandwidth ratio allocated to device i. Then the present invention has:

Through the above analysis, the bandwidth resource allocated by the device i accessing the edge base station j is β _ij B _j . The transfer rate of the model uploaded by device i can be expressed as follows:

where h _ij represents the channel gain between device _i and edge server j, pi represents the model upload power of device i, and N ₀ represents the power spectral density of Gaussian noise. Let S denote the size of the local model, and the transmission delay for device i to upload the local model to the edge base station can be expressed as follows:

Then, the energy consumed by device i to upload the local model can be expressed as follows:

The present invention takes the average transmission energy consumption of all devices as the communication cost of the federated learning system. Obviously, the communication cost can also be easily extended to other resources, such as training delay. The communication cost of the system can be expressed as follows:

From the above analysis, it can be seen that both the edge access strategy and the bandwidth resource allocation strategy will affect the energy consumption of the device. Therefore, communication costs should also be considered when designing edge access strategies.

In order to save the communication cost and improve the learning accuracy, the present invention uses the system benefit to quantify the overall performance of the federated learning. The present invention defines system benefits as follows:

where μ is a continuous variable, and μ∈[0,1], which is used to adjust the trade-off between learning performance and transmission energy consumption. G _max and E _max are the highest precision and energy consumption that the system can achieve. The purpose of regularization is to mitigate the influence of the two different orders of magnitude on the policy.

The goal of the present invention is to find edge access strategies and resource allocation strategies to maximize system benefits. The optimization problem can be formulated as follows:

max P

st

In the objective function, a _ij is a binary variable, and β _ij is a continuous variable. This optimization problem can be formulated as a mixed integer nonlinear programming problem (MINLP).

Due to the privacy of federated learning, the statistical distribution of device data cannot be obtained, so it is very difficult to directly obtain the global optimal solution. At the same time, in order to prevent the original data information from the local model parameters uploaded by the device, federated learning is often used in conjunction with nonlinear privacy encryption methods. Considering this problem, and also in order to increase the universality of the proposed algorithm, the present invention uses deep reinforcement learning to adaptively explore an edge access strategy in a multi-base station scenario based on edge feedback information, and can be used without data. In the case of exchange, maximize the profit of the system in a way that preserves data privacy.

Deep reinforcement learning can transform different types of variables into the same type for unified solution by discretizing continuous variables or continuous discrete variables. However, as the number of solving variables increases, deep reinforcement learning can easily get trapped in local optimal solutions, resulting in unsatisfactory results. Therefore, the present invention decouples the original problem into two sub-problems to solve, which are: the edge association problem and the resource allocation problem under a given edge access strategy. For the edge correlation sub-problem, the present invention deploys deep reinforcement learning in the cloud to adaptively adjust the access strategy between the edge base station and the device. The resource allocation sub-problem is related to the edge access problem, so the present invention decouples the resource allocation strategy to each edge base station and solves it separately under the condition of a given edge access strategy, which reduces the complexity of the algorithm and at the same time It also increases the scalability of the algorithm.

The present invention observes that when the edge access strategy is fixed, the learning performance of the system is determined accordingly, and the optimization problem can be simplified to the problem of how to allocate communication resources to minimize uploading energy consumption. The bandwidth resources of each base station are independently determined by it and have nothing to do with other edge base stations. Therefore, the resource allocation problem of multi-edge base stations can be decomposed into M sub-problems, which are solved separately on each edge base station. For each edge base station, the following problems need to be solved:

为接入边缘基站j的设备簇，N_j为设备组

中的设备数量。in,

is the device cluster accessing edge base station j, and N _j is the device group

number of devices in .

Obviously, the above problem is convex. Because the variable β _ij is convex in the feasible region, and all constraints are affine.

The present invention uses the commonly used Karush-Kuhn-Tucker (KKT) condition to obtain the analytical solution of bandwidth allocation, and the present invention has the following theorem.

给定边缘接入策略时，资源分配子问题的最优带宽分配β_ij可以表示如下：Theorem 1: For edge base station j and its training device cluster

Given an edge access strategy, the optimal bandwidth allocation β _ij for the resource allocation sub-problem can be expressed as follows:

The proof process of Theorem 1 is as follows:

The above convex problem can be solved by the Lagrangian multiplier method, and the Lagrangian equation of the objective function of the subproblem can be expressed as follows:

where λ is the Lagrange multiplier of the constraints of the convex problem. In order to solve the Lagrange equation, the present invention calculates its KKT condition:

By solving the above equation, we can get:

Based on this, the present invention can obtain the expression of bandwidth allocation and Lagrange multiplier, then the present invention has:

Meanwhile, according to KKT conditions, the present invention has:

So we get:

Through the above formula, the present invention can solve the bandwidth resource allocation variable, which can be expressed as:

Through Theorem 1, the present invention can effectively solve the problem of communication resource allocation, and for a given edge access strategy, the present invention has the optimal bandwidth resource allocation strategy in this case, and forms a one-to-one correspondence, reducing the difficulty of solving the original problem.

具体细节如下：For the edge access problem, the traditional method needs to obtain all the information before solving it, but this is impossible due to the privacy of federated learning. Deep reinforcement learning is an algorithm that continuously explores the environment without any prior information. The present invention designs a deep reinforcement learning method capable of adaptively adjusting the edge access strategy according to the feedback information of the edge base station. The edge correlation problem can be described as a Markov process

The specific details are as follows:

在第k轮，云端只能观察到来自边缘基站对上一轮边缘接入策略的反馈信息，故本发明把状态定义为S(k)＝{S₁(k),S₂(k),…,S_N(k)},每一项S_i(k)可定义为：(1) Status

In the kth round, the cloud can only observe the feedback information from the edge base station on the edge access strategy of the previous round, so the present invention defines the state as S(k)={S ₁ (k), S ₂ (k), ...,S _N (k)}, each term S _i (k) can be defined as:

S _i (k)={A _i (k-1),β _ij (k),Δ _i (k)}

Among them, Δ _i (k) indicates whether the learning accuracy is improved compared to the k-1 round, that is, Δ _i (k)=1 means that the accuracy is improved, otherwise, Δ _i (k)=0.

在第k轮，动作为每一个设备的边缘关联策略:(2) Action

In the kth round, the action is the edge association policy for each device:

A(k)={A ₁ (k),A ₂ (k),...,A _N (k)}

Each of them A _i (k) can be expressed as:

A _i (k)={a _ij (k)}

奖励是策略的指引方向，故本发明把奖励设置为目标函数：(3) Rewards

The reward is the guiding direction of the strategy, so the present invention sets the reward as the objective function:

Since the edge base station does not know all possible subsequent states and optimal actions, the present invention uses a model-free deep reinforcement learning paradigm to update the edge access policy. At the same time, in order to deal with large state space and discrete type actions, the present invention selects Deep Q Network (DQN) as the basic framework, and combines dueling DQN and double DQN to optimize the algorithm of the present invention, and uses D3QN to solve the edge access problem.

DQN is a value-based algorithm. Its Q-value function Q(S, A; θ) is approximated by a neural network with a parameter of θ, representing the mapping relationship between the environment and actions. The output of the neural network can be obtained through the Bellman equation, The present invention has:

Among them, S', A', θ' are the state, action and corresponding parameters of the next time slot, respectively.

Two Q-networks with the same structure but different parameters are used in DQN to improve the stability of the algorithm. One is the current Q-network with up-to-date parameters to evaluate the current state-action value function. The other is a target Q network with parameters from past rounds and keeps the Q value constant for a period of time. The present invention takes the Q value of the current Q network as the input of the neural network. Obviously, the goal of DQN is to minimize the difference between two Q-networks and define it as the loss function of DQN. The present invention has:

L(θ)=E[(yQ(S,A;θ)) ² ]

In order to meet the non-IID characteristics of the data in the Markov process, DQN adopts an experience playback strategy to reduce the time correlation between samples and ensure the stability of the algorithm. However, the target values of DQN are all directly obtained by greedy methods, which lead to overestimation and large bias. To solve this problem, the present invention introduces the DDQN algorithm to avoid overestimation by decoupling the selection of target actions and the evaluation of the current state. Different from the action corresponding to the maximum Q value in selecting the target Q network in the DQN, the DDQN selects the action corresponding to the maximum Q value in the current Q network, and the present invention has:

Bring the selected action into the target Q network to calculate the Q value, then the present invention has:

y=R(S,A)+γQ'(φ(S'), ^Amax (S';θ);θ')

At the same time, in order to converge faster, the present invention uses dueling DQN to optimize the network structure, and divides the network into two parts, which are the value function V(S, θ, α) only related to the state and the value function V(S, θ, α) related to both state and action. Potential function A(S, A, θ, β), where θ is the common parameter of the two networks, α is the unique parameter of the value function, and β is the unique parameter of the potential function. The Q value can be regarded as the sum of these two functions, and the present invention has:

Q(S, A, θ, α, β) = V(S, θ, α) + A(S, A, θ, β)

Dueling DQN can better evaluate the policy, thereby speeding up the convergence of the network.

It is worth noting that the edge access strategy obtained by the cloud center directly changes the access relationship between the edge base station and the device, which in turn guides the communication resource allocation strategy on the edge base station, thereby affecting the system learning performance and device energy consumption.

Considering that due to the trade-off of energy consumption in the system, devices with different data distributions are connected to the same edge base station, the present invention utilizes the advantages of transfer learning to design a hierarchical federated transfer learning strategy. The present invention can divide the neural network into basic feature layer and individual feature layer. The basic feature layer has common features of most data, and the personality feature layer captures the unique properties of different data. The hierarchical federated transfer learning of the present invention is specifically described as follows:

(1) Identifying and migrating equipment: The present invention calculates the average learning accuracy of each edge base station after a certain round:

The present invention regards the equipment with lower than average precision as the equipment whose precision needs to be further improved. Obviously, the equipment with different data distribution in the equipment cluster of the edge base station must have lower precision than the average precision. For convenience, the present invention is collectively referred to as migration devices, and other devices are referred to as non-migration devices.

和个性特征层模型

上传到接入的边缘基站。迁移设备只上传基础特征层模型，个性特征层模型在设备本地更新，本发明有：(2) Hierarchical federated transfer learning: non-transferred devices transfer their own basic feature layer model

and personality feature layer model

Upload to the connected edge base station. The migration device only uploads the basic feature layer model, and the individual feature layer model is updated locally on the device. The present invention includes:

The edge base station aggregates the basic feature layer models of all devices to ensure the generalization performance of the model, and aggregates the individual feature layer models of non-migrating devices to eliminate the influence of non-IID devices. Then the present invention has:

(3) The edge base station delivers the aggregated base layer model to all access devices, and delivers the personality layer model to non-migrated devices. The device performs the above update again according to the received model, and so on until convergence.

The layered migration learning strategy proposed by the present invention does not consume extra energy, because the same as the traditional federated learning, the device updates each layer model during local training, the difference is that when uploading, the non-migration device needs to upload each layer One layer model, and the migration device only needs to upload the personality feature layer model to the connected edge base station, which reduces the size of the uploaded model. However, the base layer model accounts for the majority of all layers, so the present invention ignores the reduced energy consumption when calculating the energy consumption.

In summary, the present invention provides an energy-efficient method for generating a clustering federated edge learning strategy:

First, in order to achieve the purpose of high-efficiency learning of the federated system, the present invention takes the learning performance as the system gain, and the communication energy consumption as the system cost, so as to obtain the system gain function. In order to study the system revenue optimization problem in the cluster federated edge learning network, the present invention jointly considers the communication conditions and the heterogeneous characteristics of the data, achieves high energy efficiency while ensuring the learning performance, and quantifies the problem as a mixed integer non- Linear programming (MINLP) problem.

Secondly, in order to effectively solve the problem of maximizing system revenue, the present invention observes that after determining the edge access strategy, the original problem can be regarded as a resource allocation problem aiming at high energy efficiency, so it is decomposed into two sub-problems: edge access Based on the access problem and the resource allocation problem for a given edge access strategy, an effective iterative optimization algorithm is designed accordingly. For the edge access sub-problem, in order to strengthen the privacy of the federated learning data and be better applicable to the model nonlinear encryption algorithm, the present invention utilizes deep reinforcement learning to explore the edge access strategy. In the resource allocation sub-problem, in order to reduce the complexity of the algorithm, a convex optimization algorithm is used to solve the resource allocation strategy.

Finally, due to the trade-off of energy consumption, devices with different data distributions may access the same base station for common training. Considering this situation, the present invention proposes a hierarchical federated transfer learning strategy, which does not consume additional energy. , to further improve the learning accuracy.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The recorded technical solutions are modified, or some technical features thereof are equivalently replaced, but these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. an energy-efficient clustering federated edge learning strategy generation method, is characterized in that, comprises the following steps: S1. The cloud center initializes the edge access policy; S2, the edge base station uses the convex optimization method to solve the bandwidth resource allocation strategy of its access device, and sends its initialization model to the access device; S3. The device calculates the accuracy of the received global model on the local test data set, and uses the hierarchical federated migration method to train the local model according to the global model and local training data, calculates the energy spent on uploading the local model, and compares the test accuracy and energy consumption. The difference is used as the local revenue, and then the local model and local revenue are uploaded to the connected edge base station; S4. The edge base station aggregates the local model hierarchically, and at the same time calculates the edge revenue by averaging the local revenue of all access devices, and uploads the edge revenue to the cloud center; S5. The cloud center calculates the system revenue according to the feedback information received from the edge base station, and uses the deep reinforcement learning algorithm to adjust the edge access strategy; S6. Repeat the above process until convergence. 2. energy-efficient clustering federated edge learning strategy generation method according to claim 1, is characterized in that, in step S1, the access strategy a _ij of equipment and edge base station is a binary variable, namely if equipment i and edge When base station j communicates, a _ij =1, otherwise, a _ij =0, and each device accesses an edge base station. 3. The energy-efficient clustering federated edge learning strategy generation method according to claim 1, wherein the convex optimization method in step S2 is specifically: for edge base station j and its access device cluster

When the edge access strategy is given, the calculation formula of the optimal bandwidth allocation β _ij for the resource allocation sub-problem is as follows:

where h _ij represents the channel gain between device i and edge server j, pi represents the model upload power of device _i , N ₀ represents the power spectral density of Gaussian noise, and β _ij B _j represents device i accessing edge base station j The allocated bandwidth resources, the devices accessing the edge base station j share the public spectrum with the bandwidth B _j for communication,

a _ij represents the access policy between the device and the edge base station, and β _ij represents the bandwidth ratio allocated to device i. 4. The energy-efficient clustering federated edge learning strategy generation method according to claim 1, wherein in step S3, the device utilizes local data according to the received global model θ _j

For training, the loss function formula of device i is:

The device uses gradient descent to update the local model ω _i with the following formula:

Among them, η is the learning step size, and η≥0; In step S3, after a certain round of training, the local model is trained by the hierarchical federated transfer learning strategy, and the neural network is divided into a basic feature layer and a personality feature layer. The specific process of the hierarchical federated transfer learning strategy is as follows: S301. Calculate the average learning accuracy of each edge base station after a certain round of times according to the following formula:

S302. Set the equipment basic feature layer model with higher than average precision

and personality feature layer model

Uploaded to the connected edge base station, the device with lower than average precision uploads the basic feature layer model, and the individual feature layer model is updated locally on the device. The formula is as follows:

in,

is the local personality feature layer model of device i,

For the migration device set. S303, the edge base station aggregates the basic feature layer models of all devices and aggregates the personality feature layer models of the non-migration devices, the edge base station delivers the aggregated base layer model to all access devices, and delivers the personality layer model to the non-migration devices The device, the device performs the above update according to the received model, and so on until convergence. 5. The energy-efficient clustering federated edge learning strategy generation method according to claim 1, characterized in that, in step S3, the learning accuracy gij of the global model on the local test data set is used to measure the value of the edge base station _j . An indicator of global model performance, the learning performance gain G of the system is the average accuracy of all devices, as shown in the formula:

6. The energy-efficient clustering federated edge learning strategy generation method according to claim 1, characterized in that, in step S3, the energy _Eij that is consumed by the device i uploading the local model is as follows:

T _ij is the transmission delay for device i to upload the local model to the edge base station. The formula is as follows:

S represents the size of the local model, ri _ij is the transmission rate of the model uploaded by device i, and the formula is as follows:

h _ij represents the channel gain between device i and edge server j, pi represents the model upload power of device _i , N ₀ represents the power spectral density of Gaussian noise, β _ij B _j is the distribution of device i accessing edge base station j The obtained bandwidth resources, the devices accessing the edge base station j share the public spectrum with the bandwidth B _j for communication,

a _ij represents the access policy between the device and the edge base station, and β _ij represents the bandwidth ratio allocated to device i. 7. The energy-efficient clustering federated edge learning strategy generation method according to claim 1, wherein in step S4, before hierarchical aggregation, the edge base station aggregates all received local models, and the formula is as follows:

in,

is all the device clusters accessing edge base station j, and ω _i is the local model. After a certain round of training, the hierarchical federated transfer learning strategy is executed, and the edge base station aggregates the received local models hierarchically. The specific method is as follows: the edge base station aggregates the basic feature layer models of all devices to ensure the generalization performance of the model, and aggregates The personality feature layer model of non-migrated devices to eliminate the influence of non-IID data between devices, the formula is as follows:

in,

is the global model of the basic feature layer of the device accessing the edge base station j, shared by all devices,

is a global model of the personality feature layer, shared for non-migration devices,

is a cluster of non-migrated devices accessing edge base station j. 8. energy-efficient clustering federated edge learning strategy generation method according to claim 1, is characterized in that, the formula of system revenue function in step S5 is as follows:

where μ is a continuous variable, and μ∈[0, 1], which is used to adjust the trade-off relationship between learning performance and transmission energy consumption, and G _max and E _max are the highest accuracy and maximum energy consumption that can be achieved by the system. 9. energy-efficient clustering federated edge learning strategy generation method according to claim 1, is characterized in that, in step S5, adopts deep reinforcement learning to adjust edge access strategy, and the concrete process of deep reinforcement learning is: S501. Describe the edge correlation problem as a Markov process

The specific details are as follows: (1) Status

In the kth round, the state is defined as S(k)={S ₁ (k), S ₂ (k), ..., S _N (k)}, and each term S _i (k) is defined as: S _i (k)={A _i (k-1), β _ij (k), Δ _i (k)} Among them, Δ _i (k) indicates whether the learning accuracy is improved compared to the k-1 round, that is, Δ _i (k)=1 means that the accuracy is improved, otherwise, Δ _i (k)=0; (2) Action

In the kth round, the action is the edge association policy for each device: A(k)={A ₁ (k), A ₂ (k), ..., A _N (k)} Each of them A _i (k) can be expressed as: A _i (k)={a _ij (k)} (3) Rewards

Set the reward as the objective function:

S502. Select DQN as the basic framework, optimize the algorithm by combining dueling DQN and double DQN, and use D3QN to solve the edge access problem. The Q-value function Q(S, A; θ) is approximated by a neural network with a parameter of θ, representing The mapping relationship between the environment and the action, the output of the neural network is obtained through the Bellman equation:

Among them, S', A', θ' are the state, action and corresponding parameters of the next time slot respectively; Two Q-networks with the same structure but different parameters are used in DQN to improve the stability of the algorithm, one is the current Q-network with the latest parameters to evaluate the value function of the current state-action, and the other is the one with the past rounds The goal of the parameter Q network, and keep the Q value unchanged for a period of time, taking the Q value of the current Q network as the input of the neural network, the goal of DQN is to minimize the difference between the two Q networks, and it is defined as The loss function of DQN: L(θ)=E[(yQ(S, A; θ)) ² ] S503, using the DDQN algorithm, select the action corresponding to the maximum Q value in the current Q network:

Then bring the selected action into the target Q network to calculate the Q value: y=R(S,A)+γQ′(φ(S′), ^Amax (S′;θ);θ′) S504. Use dueling DQN to optimize the network structure, and divide the network into two parts, namely the value function V(S, θ, α) related only to the state and the potential function A(S, A, θ, β), where θ is the common parameter of the two networks, α is the unique parameter of the value function, β is the unique parameter of the potential function, and the Q value is the sum of these two functions: Q(S, A, θ, α, β)=V(S, θ, α)+A(S, A, θ, β). 10. An energy-efficient clustering federated edge learning strategy generation device, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that the computer processor executes the When the computer program is described, the energy-efficient clustering federated edge learning strategy generation method described in any one of claims 1-9 is realized.

Images