CN115914227A - Edge Internet of things agent resource allocation method based on deep reinforcement learning - Google Patents

Abstract

The invention discloses an edge Internet of things agent resource allocation method based on deep reinforcement learning, and relates to the technical field of Internet of things, wherein the method comprises the following steps: firstly, collecting data in an environment by a terminal device x, transmitting the data to a deep reinforcement learning network model, obtaining an optimal allocation strategy by the deep reinforcement learning network model according to the data, and finally sending the data to an edge node e for calculation according to the optimal allocation strategy to realize edge internet of things proxy resource allocation; the invention solves the problems that the edge Internet of things proxy resource allocation time is long, the performance is limited, and the prior art is not enough to support the resource optimization configuration of the complex dynamic Internet of things.

Description

A resource allocation method for edge IoT agents based on deep reinforcement learning

Technical Field

The present invention relates to the technical field of Internet of Things, and in particular to an edge Internet of Things agent resource allocation method based on deep reinforcement learning.

Background Art

The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

Reasonable resource allocation is an important guarantee for efficiently supporting the power business of edge IoT agents; the power IoT is an important part of the national industrial Internet; building an efficient, secure and reliable perception layer has become an important construction task in the power industry; however, the computing power of current power IoT devices is limited and cannot effectively achieve local large-scale fast computing tasks; the edge IoT agent, as the core device of the IoT perception layer, plays the role of connecting IoT terminals and the cloud; with the access of various data such as voice, video, and images, as well as the collection of high-frequency data and the storage of heterogeneous data, how to dynamically and adaptively deploy the tasks of IoT terminals on appropriate edge IoT agent nodes is a key issue at this stage.

At present, the key issues of edge IoT agents are mainly reflected in two aspects; first, due to the mutual dependence between IoT agents at different edges, the existing combinatorial optimization methods generally use approximate algorithms or heuristic algorithms to solve deployment solutions, which not only requires a long running time but also has limited performance; second, there are multiple edge nodes in the edge IoT agent environment, and the resource capacity of the edge server is limited; therefore, different edge nodes need to cooperate through distributed decision-making to achieve optimal resource allocation to support efficient and reliable information interaction.

The emergence of multi-layer network models provides a new solution for the optimal configuration of communication network resources; the network model is trained through a multi-layer network to achieve an accurate and efficient solution; currently, some researchers have conducted research and analysis; one solution in the existing technology is based on convolutional neural networks to achieve reasonable allocation of IoT resources and efficient interaction and coordination of edge devices with terminal data and network tasks; another solution is to use Bayesian to optimize the Q-learning network to achieve rationalization and ordering of resource allocation in the network to resist DDoS network attacks; in addition, the introduction of deep spatiotemporal residual networks effectively supports the effective load balancing of industrial IoT networks and ensures that the network achieves low-latency and highly reliable data interaction; considering the heterogeneity of network devices, the existing technology mostly uses deep learning networks to effectively match network servers and user requests to allocate the best amount of resources to user devices; but it should be noted that due to the network structure of the deep network model, it is easy to fall into the problem of mismatch between computing power and processing problems when updating and iterating the network status, which limits the computing efficiency and is insufficient to support the optimal resource configuration of complex power IoT.

Summary of the invention

The purpose of the present invention is to provide an edge Internet of Things agent resource allocation method based on deep reinforcement learning in response to the above-mentioned deficiencies in the prior art, which solves the problems of long edge Internet of Things agent resource allocation time, limited performance and the insufficiency of prior art to support resource optimization configuration of complex power Internet of Things.

The technical solution of the present invention is as follows:

A resource allocation method for edge IoT agents based on deep reinforcement learning, comprising:

Step S1: The terminal device x collects data in the environment and transmits the data to the deep reinforcement learning network model;

Step S2: according to the data, obtaining the optimal allocation strategy by the deep reinforcement learning network model;

Step S3: According to the optimal allocation strategy, the data is sent to the edge node e for calculation to realize edge IoT agent resource allocation.

Furthermore, the training method of the deep reinforcement learning network model in step S1 comprises the following steps:

Step S101: Initialize the system state s of the deep reinforcement learning network model;

Step S102: Initializing the real-time ANN and the delayed ANN of the deep reinforcement learning network model;

Step S103: Initialize the experience pool O of the deep reinforcement learning network model;

Step S104: according to the current system state s _t , using the ε-greedy strategy, select the system action a _t ;

Step S105: The environment feeds back a reward σ _t+1 and the next state s _t+1 of the system according to the system action a _t ;

Step S106: Calculate a state transition sequence Δ _t according to the current system state s _t , system action a _t , reward σ _t+1 and the next system state s _t+1 , and store the state transition sequence Δ _t in the experience pool O;

Step S107: Determine whether the storage capacity of the experience pool O reaches the preset value. If so, extract N state transition sequences from the experience pool O to train the real-time ANN and the delayed ANN to complete the training of the deep reinforcement learning network model; otherwise, update the current system state s _t to the next system state s _t+1 and return to step S104.

Furthermore, the system state s in step S101 is a local uninstallation state, which is expressed as follows:

s＝[F,M,B]

in:

F is the unloading decision vector;

M is the computing resource allocation vector;

其中，b_d为第d个MEC服务器的剩余计算资源，G_d为总计算资源，

为分配给计算资源分配向量M中每个任务的计算资源；B is the remaining computing resource vector; B = [b ₁ , b ₂ , b ₃ …b _d , …],

Where b _d is the remaining computing resources of the dth MEC server, G _d is the total computing resources,

Allocate computing resources to each task in the computing resource allocation vector M;

The expression of the system action a _t in step S104 is as follows:

a _t = [x, μ, k]

in:

x is the terminal device;

μ is the unloading plan of terminal device x;

k is the computing resource allocation scheme of terminal device x;

The calculation formula of the reward σ _t+1 in step S105 is as follows:

in:

r is the reward function;

A is the objective function value at the current time t;

A' is the objective function value when the current system state s _t takes system action a _t to the next state;

A” is the calculated value under all local unloading conditions;

The expression of the state transition sequence _Δt in step S106 is as follows:

Δ _t =(s _t ,a _t ,σ _t+1 ,s _t+1 ).

Furthermore, the training method for the real-time ANN and the delayed ANN in step S107 includes the following steps:

Step S1071: for the N state transition sequences, obtain an estimated value Q(s _t , a _t , θ) of the state-action pair and a value Q(s _t+1 , a _t+1 , θ′) of the next state according to the state transition sequence;

Step S1072: Calculate the target value y of the state-action pair according to the value Q(s _t+1 ,a _t+1 ,θ′) of the next state and the reward σ _t+1 ;

Step S1073: Calculate the loss function Loss(θ) according to the estimated value Q(s _t , a _t , θ) of the state-action pair and the target value y;

Step S1074: adjusting the parameter θ of the real-time ANN through the back propagation mechanism of the loss, and reducing the loss function Loss(θ) using the optimizer RMSprop;

Step S1075: determine whether the number of steps since the last update of the delay ANN parameter θ' is equal to the set value. If so, update the delay ANN parameter θ' and proceed to step S1077; otherwise, proceed to step S1076;

Step S1076: Determine whether the training of the N state transition sequences is completed. If so, re-extract N state transition sequences from the experience pool O and return to step S1071. Otherwise, return to step S1071.

Step S1077: testing the performance index of the deep reinforcement learning network model to obtain a test result;

Step S1078: Determine whether the test result meets the requirements. If so, the real-time ANN and delayed ANN training is completed to obtain a trained deep reinforcement learning network model; otherwise, re-extract N state transition sequences from the experience pool O and return to step S1071.

Furthermore, the calculation formula of the target value y of the state-action pair in step S1072 is as follows:

in:

为maxQ(s_t+1,a_t+1,θ')的波动系数；

is the fluctuation coefficient of maxQ(s _t+1 ,a _t+1 ,θ');

Q(s _t+1 ,a _t+1 ,θ') is the value of the next state of the system;

maxQ(s _t+1 ,a _t+1 ,θ') is the maximum value of the next state of the system;

The expression of the loss function Loss(θ) in step S1073 is as follows:

in:

N is the number of state transition sequences extracted each time;

n is the sequence number of the state transition sequence.

Further, the performance indicators of the deep reinforcement learning network model in step S1077 include: global cost and reliability;

The global cost includes delay cost c ₁ , migration cost c ₂ and load cost c ₃ .

Furthermore, the expression of the delay cost _c1 is as follows:

in:

t is the number of interactions;

X is a terminal device set;

E is the set of edge nodes;

u _x is the amount of data sent;

为当前交互时间里终端设备x与边缘节点e的部署变量；

is the deployment variable of the terminal device x and the edge node e during the current interaction time;

τ _xe is the transmission delay between terminal device x and edge node e;

The expression of the migration cost _c2 is as follows:

in:

j is the migration edge node;

为上一交互时间里终端设备x与边缘节点e的部署变量；

is the deployment variable of the terminal device x and the edge node e in the last interaction time;

为当前交互时间里终端设备x与迁移边缘节点j的部署变量；

is the deployment variable of the terminal device x and the migration edge node j in the current interaction time;

The expression of the load cost _c3 is as follows:

in:

u _x is the amount of data sent.

Furthermore, the calculation of the reliability includes the following steps:

Step A1: Store the interaction data between the terminal device x and the edge node e in a sliding window and update it in real time;

Step A2: According to the historical interaction data between the terminal device x and the edge node e, the expected value based on the Bayesian trust evaluation is used to calculate the time decay degree and resource allocation rate of the current interaction;

Step A3: Calculate the reliability T _ex (t) according to the time decay degree and resource allocation rate

Furthermore, the calculation formula of the reliability T _ex (t) is as follows:

N _ex (t) = 1 - P _ex (t)

in:

U is the amount of valid information in the sliding window;

w is the current interaction information;

为时间衰减程度；

is the degree of time decay;

H _ex (t _w ) is the resource allocation rate;

的波动系数；

The coefficient of volatility;

P _ex (t _w ) positive service satisfaction of the current interaction;

N _ex (t _w ) is the negative service satisfaction of the current interaction;

s _ex (t) is the number of successful historical interactions between terminal device x and edge node e;

f _ex (t) is the number of failed historical interactions between terminal device x and edge node e.

Furthermore, the expression of the time attenuation degree in step A2 is as follows:

in:

Δt _w is the time interval from the end of the wth interaction to the beginning of the current interaction;

The calculation formula of the resource allocation rate in step A2 is as follows:

in:

source _ex (t) is the amount of resources that edge node e can provide to terminal device x in the current time slot;

source _e (t) is the total amount of resources that edge node e can provide in the current time slot.

Compared with the prior art, the present invention has the following beneficial effects:

1. A resource allocation method for edge IoT agents based on deep reinforcement learning. The optimal allocation strategy is calculated using a deep reinforcement learning network model. According to the optimal allocation strategy, the terminal data is transmitted to the edge node e for calculation, which effectively alleviates the computing pressure of the on-site equipment and avoids the storage difficulties caused by large amounts of data in the resource allocation process. It ensures reliable and efficient information interaction of the communication network and provides better information interaction support services for the power IoT.

2. A resource allocation method for edge IoT agents based on deep reinforcement learning, in which the deep reinforcement learning network model combines the perception ability of deep learning and the decision-making ability of reinforcement learning to complement each other and support the optimal strategy solution for large amounts of data.

3. A resource allocation method for edge IoT agents based on deep reinforcement learning, in which the neural networks include real-time ANN and delay ANN. After a certain number of trainings, the parameters of the delay ANN are updated to the parameters of the real-time ANN, which ensures the timeliness of the delay ANN value function and reduces the correlation between states.

4. A resource allocation method for edge IoT agents based on deep reinforcement learning, which takes global cost and reliability as performance judgment indicators of the network model, and provides a judgment basis for the network model to seek the optimal strategy.

5. A resource allocation method for edge IoT agents based on deep reinforcement learning adopts a sliding window mechanism to update interaction information, directly discards interaction information with a long interval, reduces the computational overhead of user terminals, and the reliability calculation ensures the security of user terminals during task offloading, providing a guarantee for establishing a good interaction environment.

6. A resource allocation method for edge IoT agents based on deep reinforcement learning, which calculates the quality values of various interactions between user terminals and edge servers, prepares for reliability calculations, and provides a basis for judgment in seeking the optimal strategy for network models.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of the method of the present invention.

FIG2 is a flow chart of a method for implementing a deep reinforcement learning network model in the present invention.

FIG3 is a flow chart of the training method of the real-time ANN and the delayed ANN in the present invention.

FIG4 is a flow chart of the reliability calculation method in the present invention.

FIG5 is a schematic diagram of a sliding window in the present invention.

FIG6 is a diagram showing the structure of a deep reinforcement learning network in the present invention.

FIG. 7 shows the parameters of the deep reinforcement learning network model in an embodiment of the present invention.

FIG8 is a graph showing network performance at different learning rates for a deep reinforcement learning network model according to an embodiment of the present invention.

DETAILED DESCRIPTION

It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

Embodiment 1

Please refer to FIG1 , a method for allocating resources of edge IoT agents based on deep reinforcement learning, including:

Step S1: The terminal device x collects data in the environment and transmits the data to the deep reinforcement learning network model;

Preferably, in this embodiment, the data collected by the terminal device x is the voice, video, image and other data of the user terminal;

Preferably, in this embodiment, python3+tensorflow1.0 is used as the simulation experiment platform, the hardware conditions are Intel Core i7-5200u and 16GB of memory, and 50 terminal devices x and 5 edge nodes e are set in the simulation test environment, and the terminal devices x and edge nodes e are evenly distributed in a 15 km × 15 km grid;

Preferably, in this embodiment, the terminal device x sends a task request to the edge node e once every hour, and the edge node e determines the server that executes the task in a distributed manner; wherein the load of the terminal device x comes from a real load data set, in which, due to the tidal effect, the load of the terminal task roughly follows a 24-hour periodic distribution, but may also produce random fluctuations due to environmental factors.

Preferably, in this embodiment, FIG. 7 is a deep reinforcement learning network model parameter.

Step S2: according to the data, obtaining the optimal allocation strategy by the deep reinforcement learning network model;

Step S3: According to the optimal allocation strategy, the data is sent to the edge node e for calculation to realize edge IoT agent resource allocation.

In this embodiment, specifically, as shown in FIG2 , the training method of the deep reinforcement learning network model in step S1 includes the following steps:

Step S101: Initialize the system state s of the deep reinforcement learning network model;

Step S102: Initializing the real-time ANN and the delayed ANN of the deep reinforcement learning network model;

Step S103: Initialize the experience pool O of the deep reinforcement learning network model;

Step S104: according to the current system state s _t , using the ε-greedy strategy, select the system action a _t ;

Step S105: The environment feeds back a reward σ _t+1 and the next state of the system s _t+1 according to the system action a _t ;

Step S106: Calculate a state transition sequence Δ _t according to the current system state s _t , system action a _t , reward σ _t+1 and the next system state s _t+1 , and store the state transition sequence Δ _t in the experience pool O;

Step S107: Determine whether the storage capacity of the experience pool O reaches the preset value. If so, extract N state transition sequences from the experience pool O to train the real-time ANN and the delayed ANN to complete the training of the deep reinforcement learning network model; otherwise, update the current system state s _t to the next system state s _t+1 and return to step S104.

In this embodiment, specifically, the system state s in step S101 is a local uninstallation state, which is expressed as follows:

s＝[F,M,B]

in:

F is the unloading decision vector;

M is the computing resource allocation vector;

其中，b_d为第d个MEC服务器的剩余计算资源，G_d为总计算资源，

为分配给计算资源分配向量M中每个任务的计算资源；B is the remaining computing resource vector; B = [b ₁ , b ₂ , b ₃ …b _d , …],

Where b _d is the remaining computing resources of the dth MEC server, G _d is the total computing resources,

Allocate computing resources to each task in the computing resource allocation vector M;

The expression of the system action a _t in step S104 is as follows:

a _t = [x, μ, k]

in:

x is the terminal device;

μ is the unloading plan of terminal device x;

k is the computing resource allocation scheme of terminal device x;

The calculation formula of the reward σ _t+1 in step S105 is as follows:

in:

r is the reward function;

A is the objective function value at the current time t;

A' is the objective function value when the current system state s _t takes system action a _t to reach the next state;

A” is the calculated value under all local unloading conditions;

The expression of the state transition sequence _Δt in step S106 is as follows:

Δ _t =(s _t ,a _t ,σ _t+1 ,s _t+1 ).

In this embodiment, specifically, as shown in FIG3 , the training method for the real-time ANN and the delayed ANN in step S107 includes the following steps:

Step S1071: for the N state transition sequences, obtain an estimated value Q(s _t , a _t , θ) of the state-action pair and a value Q(s _t+1 , a _t+1 , θ′) of the next state according to the state transition sequence;

Step S1072: Calculate the target value y of the state-action pair according to the value Q(s _t+1 ,a _t+1 ,θ′) of the next state and the reward σ _t+1 ;

Step S1073: Calculate the loss function Loss(θ) according to the estimated value Q(s _t , a _t , θ) of the state-action pair and the target value y;

Step S1074: adjusting the parameter θ of the real-time ANN through the back propagation mechanism of the loss, and reducing the loss function Loss(θ) using the optimizer RMSprop;

Step S1075: determine whether the number of steps since the last update of the delay ANN parameter θ' is equal to the set value. If so, update the delay ANN parameter θ' and proceed to step S1077; otherwise, proceed to step S1076;

Step S1076: Determine whether the training of the N state transition sequences is completed. If so, re-extract N state transition sequences from the experience pool O and return to step S1071. Otherwise, return to step S1071.

Step S1077: testing the performance index of the deep reinforcement learning network model to obtain a test result;

Step S1078: Determine whether the test result meets the requirements. If so, the real-time ANN and delayed ANN training is completed to obtain a trained deep reinforcement learning network model; otherwise, re-extract N state transition sequences from the experience pool O and return to step S1071.

In this embodiment, specifically, the calculation formula of the target value y of the state-action pair in step S1072 is as follows:

in:

为maxQ(s_t+1,a_t+1,θ')的波动系数；

is the fluctuation coefficient of maxQ(s _t+1 ,a _t+1 ,θ');

Q(s _t+1 ,a _t+1 ,θ') is the value of the next state of the system;

maxQ(s _t+1 ,a _t+1 ,θ') is the maximum value of the next state of the system;

The expression of the loss function Loss(θ) in step S1073 is as follows:

in:

N is the number of state transition sequences extracted each time;

n is the sequence number of the state transition sequence.

In this embodiment, specifically, the deep reinforcement learning network model performance indicators in step S1077 include: global cost and reliability;

The global cost includes delay cost c ₁ , migration cost c ₂ and load cost c ₃ .

In this embodiment, in order to achieve efficient task processing, three factors are considered: delay cost _c1 , migration cost _c2 and load cost _c3 ; since the terminal device x needs to send the collected data to the edge node e for processing, the transmission of the data will cause a time delay; when processing a task, the edge node e can also decide whether to send the task to the migration edge node j, however, since the migration task requires the redeployment of the model, a migration cost will be incurred; since the capacity of the edge node e is limited, if too many tasks are deployed on the same edge node e, the edge node e will often be overloaded, resulting in a load cost.

In this embodiment, specifically, the expression of the delay cost _c1 is as follows:

in:

t is the number of interactions;

X is a terminal device set;

E is the set of edge nodes;

u _x is the amount of data sent;

为当前交互时间里终端设备x与边缘节点e的部署变量；

is the deployment variable of the terminal device x and the edge node e during the current interaction time;

τ _xe is the transmission delay between terminal device x and edge node e;

The expression of the migration cost _c2 is as follows:

in:

j is the migration edge node;

为上一交互时间里终端设备x与边缘节点e的部署变量；

is the deployment variable of the terminal device x and the edge node e in the last interaction time;

为当前交互时间里终端设备x与迁移边缘节点j的部署变量；

is the deployment variable of the terminal device x and the migration edge node j in the current interaction time;

The expression of the load cost _c3 is as follows:

in:

u _x is the amount of data sent.

In this embodiment, specifically, as shown in FIG4 , the calculation of the reliability includes the following steps:

Step A1: Store the interaction data between the terminal device x and the edge node e in a sliding window and update it in real time;

In this embodiment, considering that the interaction experience with a long interval is not enough to update the current reliability value in time, more attention should be paid to the most recent interaction behavior, so a sliding window mechanism is used to update the interaction information; as shown in FIG5 , when the interaction information of the next time slot arrives, the time slot record with the longest interval in the window will be discarded, and the valid interaction information will be recorded in the window, thereby reducing the calculation overhead of the user terminal;

Step A2: According to the historical interaction data between the terminal device x and the edge node e, the expected value based on the Bayesian trust evaluation is used to calculate the time decay degree and resource allocation rate of the current interaction;

表示从w次交互得到的信息到当前交互时隙的信息的衰减程度，其中Δt_w＝t-t_w，t_w是w次交互时隙的结束时间，边缘服务器在每次交互过程中所能提供的计算资源量也会影响到交互信息的更新；In this embodiment, since the reliability of the edge server is dynamically updated, the longer the historical interaction information is from the current time, the smaller the impact on the current reliability evaluation. The time decay function is defined as:

It represents the attenuation degree of the information obtained from the w-th interaction to the current interaction time slot, where _Δtw = _ttw , _tw is the end time of the w-th interaction time slot, and the amount of computing resources that the edge server can provide during each interaction will also affect the update of the interaction information;

Step A3: Calculate the reliability T _ex (t) according to the time decay degree and resource allocation rate

In this embodiment, specifically, the calculation formula of the reliability T _ex (t) is as follows:

N _ex (t) = 1 - P _ex (t)

in:

U is the amount of valid information in the sliding window;

w is the current interaction information;

为时间衰减程度；

is the degree of time attenuation;

H _ex (t _w ) is the resource allocation rate;

的波动系数；ε is

The coefficient of volatility;

P _ex (t _w ) positive service satisfaction of the current interaction;

N _ex (t _w ) is the negative service satisfaction of the current interaction;

s _ex (t) is the number of successful historical interactions between terminal device x and edge node e;

f _ex (t) is the number of failed historical interactions between terminal device x and edge node e.

In this embodiment, specifically, the expression of the time attenuation degree in step A2 is as follows:

in:

Δt _w is the time interval from the end of the wth interaction to the beginning of the current interaction;

The calculation formula of the resource allocation rate in step A2 is as follows:

in:

source _ex (t) is the amount of resources that edge node e can provide to terminal device x in the current time slot;

source _e (t) is the total amount of resources that edge node e can provide in the current time slot.

In this embodiment, a deep reinforcement learning network model is used to solve the optimal allocation strategy. As shown in Figure 6, the deep reinforcement learning network model includes two neural networks. The first neural network, called the real-time ANN, is used to calculate the estimated value Q(s _t , a _t , θ) of the current state-action pair. θ refers to the parameter of the real-time ANN, which is updated each time the estimated value of the current state is calculated; the second neural network, called the delayed ANN, is used to calculate the value Q(s _t+1 , a _t+1 , θ') of the next state. The value of the next state is used to calculate the target value y.

In this embodiment, the influence of different learning rates on the deep reinforcement learning network model was tested. As shown in Figure 8, when the learning rate factor is set to 0.01, the network loss function cannot converge effectively, and the function value has obvious oscillation. On the contrary, when the learning rate is set to 0.0001, the dispersion of the network is effectively improved, and the network converges effectively at 60 iterations, but the convergence speed is significantly slower. Obviously, when it is set to 0.0001, the resource allocation performance is the best, the loss function drops quickly, the network converges more stably, and the convergence effect is better.

The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of the description in this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims (10)

1. An edge internet of things agent resource allocation method based on deep reinforcement learning is characterized by comprising the following steps:

step S1: collecting data in the environment by a terminal device x, and transmitting the data to a deep reinforcement learning network model;

step S2: obtaining an optimal distribution strategy by a deep reinforcement learning network model according to the data;

and step S3: and sending the data to an edge node e for calculation according to the optimal allocation strategy, so as to realize the allocation of the edge Internet of things agent resources.

2. The method for allocating the proxy resource of the edge internet of things based on the deep reinforcement learning of claim 1, wherein the training method of the deep reinforcement learning network model in the step S1 comprises the following steps:

step S101: initializing a system state s of the deep reinforcement learning network model;

step S102: initializing a real-time ANN and a delay ANN of the deep reinforcement learning network model;

step S103: initializing an experience pool O of the deep reinforcement learning network model;

step S104: according to the current system state s _t Selecting a system action a by using an epsilon-greedy strategy _t ；

Step S105: acting a by the environment according to said system _t Feedback award sigma _t+1 And the next state s of the system _t+1 ；

Step S106: according to the currentSystem state s _t And system action a _t Bonus sigma _t+1 And the next state s of the system _t+1 Calculating to obtain a state transition sequence delta _t And converting the state into a sequence of delta _t Storing the data to an experience pool O;

step S107: judging whether the storage capacity of the experience pool O reaches a preset value, if so, extracting N state transition sequences from the experience pool O to train the real-time ANN and the delayed ANN, and finishing the training of the deep reinforcement learning network model; otherwise, the current system state s is set _t Updating to the next state s of the system _t+1 And returns to step S104.

3. The method for allocating the proxy resource of the edge internet of things based on the deep reinforcement learning of claim 2, wherein the system state S in the step S101 is a local uninstalling state, and the expression is as follows:

s＝[F,M,B]

wherein:

f is a unloading decision vector;

m is a calculation resource allocation vector;

b is a residual computing resource vector; b = [ B ] ₁ ,b ₂ ,b ₃ …b _d ,…]，

Wherein, b _d For the remaining computing resources of the d MEC server, G _d For the total calculation resources, is>

Allocating the computing resources of each task in the vector M for the computing resources;

the system action a in step S104 _t The expression of (a) is as follows:

a _t ＝[x,μ,k]

wherein:

x is terminal equipment;

mu is the unloading scheme of the terminal equipment x;

k is a calculation resource allocation scheme of the terminal device x;

the bonus σ in said step S105 _t+1 The calculation formula of (a) is as follows:

wherein:

r is a reward function;

a is a target function value under the current time t state;

a' is the current system state s _t Taking a System action a _t The target function value when the next state is reached;

a' is the calculated value under all partial unloads;

the state transition sequence Δ in step S106 _t The expression of (a) is as follows:

Δ _t ＝(s _t ,a _t ,σ _t+1 ,s _t+1 )。

4. the method for allocating the proxy resource of the edge internet of things based on the deep reinforcement learning of claim 3, wherein the training method for the real-time ANN and the delayed ANN in the step S107 comprises the following steps:

step S1071: for the N state transition sequences, obtaining an estimated value Q(s) of a state action pair according to the state transition sequences _t ,a _t θ) and value of the next state Q(s) _t+1 ,a _t+1 ,θ')；

Step S1072: value Q(s) according to the next state _t+1 ,a _t+1 Theta') and prize sigma _t+1 Calculating to obtain a target value y of the state action pair;

step S1073: an estimate Q(s) from the state action pair _t ,a _t Theta) and a target value y, and calculating to obtain a Loss function Loss (theta);

step S1074: adjusting a parameter theta of the real-time ANN through a Loss back propagation mechanism, and reducing a Loss function Loss (theta) by using an optimizer RMSprop;

step S1075: judging whether the step number of the parameter theta 'of the last updating delay ANN is equal to a set value or not, if so, updating the parameter theta' of the delay ANN, and entering the step S1077; otherwise, go to step S1076;

step S1076: judging whether the training of the N state transition sequences is finished, if so, extracting the N state transition sequences again from the experience pool O, returning to the step S1071, and otherwise, returning to the step S1071;

step S1077: testing the performance index of the deep reinforcement learning network model to obtain a test result;

step S1078: judging whether the test result meets the requirement, if so, finishing the real-time ANN and delayed ANN training to obtain a trained deep reinforcement learning network model; otherwise, N state transition sequences are re-extracted from the experience pool O, and the process returns to step S1071.

5. The method for allocating the edge internet of things proxy resource based on deep reinforcement learning of claim 4, wherein the target value y of the state action pair in the step S1072 is calculated by the following formula:

wherein:

is maxQ(s) _t+1 ,a _t+1 θ') of the fluctuation coefficient;

Q(s _t+1 ,a _t+1 θ') is the value of the next state of the system;

maxQ(s _t+1 ,a _t+1 θ') is the maximum value of the next state of the system;

the expression of the Loss function Loss (θ) in step S1073 is as follows:

wherein:

n is the quantity value of the state transition sequence extracted each time;

n is the sequence number of the state transition sequence.

6. The method of claim 5, wherein the deep reinforcement learning network model performance indexes in step S1077 include: global cost and reliability;

the global cost comprises a delay cost c ₁ Migration cost c ₂ And load cost c ₃ 。

7. The method for allocating the edge internet of things proxy resource based on deep reinforcement learning of claim 6, wherein the delay cost c is ₁ The expression of (c) is as follows:

wherein:

t is the number of interactions;

x is a terminal equipment set;

e is an edge node set;

u _x is the amount of data sent;

deployment variables of the terminal device x and the edge node e in the current interaction time are obtained;

τ _xe the transmission delay of the terminal device x and the edge node e;

the migration cost c ₂ The expression of (a) is as follows:

wherein:

j is a migration edge node;

deployment variables of the terminal device x and the edge node e in the last interactive time are obtained;

deployment variables of the terminal device x and the migration edge node j in the current interaction time are obtained;

the load cost c ₃ The expression of (a) is as follows:

wherein:

u _x is the amount of data sent.

8. The method for allocating the proxy resource of the edge internet of things based on the deep reinforcement learning as claimed in claim 6, wherein the calculating of the reliability comprises the following steps:

step A1: storing the interactive data of the terminal device x and the edge node e in a sliding window, and updating in real time;

step A2: calculating the time attenuation degree and the resource allocation rate of the current interaction by adopting an expected value based on Bayesian trust evaluation according to historical interaction data of the terminal device x and the edge node e;

step A3: calculating to obtain the reliability T according to the time attenuation degree and the resource allocation rate _ex (t) 。

9. The method for allocating the proxy resources of the edge internet of things based on the deep reinforcement learning of claim 8, wherein the method is characterized in thatThe reliability T _ex The calculation formula of (t) is as follows:

N _ex (t)＝1-P _ex (t)

wherein:

u is the number of effective information in the sliding window;

w is current interaction information;

is the degree of temporal decay;

H _ex (t _w ) Allocating the resource rate;

epsilon is

The fluctuation coefficient of (a);

P _ex (t _w ) Positive service satisfaction of the current interaction;

N _ex (t _w ) Negative service satisfaction for the current interaction;

s _ex (t) is the number of successful historical interactions between the terminal device x and the edge node e;

f _ex (t) is the historical number of interactions that failed between terminal device x and edge node e.

10. The method for allocating the edge internet of things proxy resource based on the deep reinforcement learning of claim 9, wherein the expression of the time attenuation degree in the step A2 is as follows:

wherein:

Δt _w a time gap from the end of the w-th interaction to the start of the current interaction;

the calculation formula of the resource allocation rate in the step A2 is as follows:

wherein:

source _ex (t) is the amount of resources that the edge node e can provide to the terminal device x in the current time slot;

source _e (t) is the total amount of resources that the edge node e can provide in the current time slot.

Images