CN111367657A - Computing resource collaborative cooperation method based on deep reinforcement learning - Google Patents

Abstract

The invention relates to a computing resource collaboration method based on deep reinforcement learning, and belongs to the field of edge computing resource allocation. The method is: deploy edge servers in a honeycomb-like area in a dense 5G area; treat each edge server as an agent, and record a sample of computing resource status and corresponding actions for a period of time; replay from experience at each time t Randomly select state samples to obtain experience tuples, and then store each experience tuple in experience replay to accumulate experience; obtain new experience tuples for the Q value at the same time, and fill in the experience tuples; iterate over the Q value and bring in Two networks, target‑net and eval‑net, are trained to obtain the optimal approximate solution for collaborative decision-making. The invention breaks the association between state samples, makes the samples independent of each other, and improves the utilization rate of computing resources when they cooperate with each other.

Description

A computing resource collaboration method based on deep reinforcement learning

technical field

The invention belongs to the field of edge computing resource allocation, and relates to a computing resource collaboration method based on deep reinforcement learning.

Background technique

Currently, the Internet of Things (IoT) is an extension of Internet technology that connects ubiquitous mobile devices (MDs) with sensors via wireless networks. The Internet of Things has been widely used in many fields, and the amount of data in mobile Internet applications has grown exponentially. In order to improve efficiency, the pursuit of low latency has become a trend. However, the data is uploaded from the terminal device to the cloud, calculated and then transmitted back to the terminal device. This traditional cloud computing technology can no longer meet people's high requirements for computing efficiency.

5G can connect countless smart devices for data sharing and interaction. In addition, 5G can provide the basic idea of IoT, expand the coverage of IoT, and connect billions of MDs to the Internet. The surge in demand for data services is posing new challenges for service providers and mobile network operators. Many applications in 5G, such as face recognition, natural language processing, etc., can be operated in the terminal. Therefore, in order to exploit computational offloading, we need to jointly manage computational offloading and related radio resource allocation, which has actually attracted the attention of many researchers. It came just in time. Edge computing (EC) enables mobile terminals to actively transfer computing tasks to nearby edge servers, so that edge artificial intelligence collaboration is also possible, in order to solve the growing demand for large-scale cluster computing and achieve efficient computing. Collaborative transfer and cooperation provide an effective method.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a computing resource cooperation method based on deep reinforcement learning.

To achieve the above object, the present invention provides the following technical solutions:

A computing resource collaboration method based on deep reinforcement learning, the method includes the following steps:

Step 1: For seamless connection, the edge servers are deployed in a cellular configuration in areas with dense 5G networks;

Step 2: Treat each edge server as an agent, and record the computing resource status and corresponding actions for a period of time as samples into experience replay;

Step 3: In order to increase the independence of samples, randomly select state samples from experience replay to obtain experience tuples at each time t, and then store each experience tuple in experience replay to accumulate experience and store;

Step 4: Then, through the target network target net and the evaluation network eval net, the Q value is iterated and the new state is obtained and put into the experience replay again, and the weight parameters are updated with the loss function, and finally the optimal approximate solution is obtained, and the cooperative edge server is obtained. optimal decision.

Optionally, in the step 1, the energy and time spent by the edge server in receiving the collaborative calculation result are ignored;

Considering the system model, N mobile users offload computing tasks to edge servers through wireless links; each user has M independent tasks to complete;

To model tasks, the cellular network shape is used to maximize the coverage utilization of edge servers; offloading decisions and server computing resource allocation for each edge task are optimized cooperatively, as well as task transmission and reception, formulated to minimize completion of computing tasks An optimization case aiming at the full utilization of energy consumption and computing resources.

Optionally, in the second step, each edge server is regarded as an agent, and the computing resource status of its CPU, task volume and energy consumption at each moment is taken as a status sample, in which the partner's cpu idle configuration file is Defined as the data of the terminal equipment, the partners calculate these data in the duration t∈[0,T], denoted as U _bit (t);

The information of collaborative edge servers with idle CPUs is as follows: The collaborative edge server CPU status information refers to the state of the CPU over time, which is recorded by defining the following collaborative CPU event space, process and epoch, where α={α ₁ , α ₂ }, represents the CPU state space sample of the cooperative edge server, α ₁ and α ₂ cooperate with the edge server from busy to idle, and then from idle to busy respectively; then the cooperative edge processor process is defined as the time instant of the co-processor event sequence :

称为一个epoch；Two consecutive events are accompanied by a long time interval T _k =s _k -s _k-1 , where

called an epoch;

The process of the CPU is a sample path of the partner CPU process that allows offline design of collaborative computing strategies. For each epoch k, let I _k denote the CPU status indicator, where the values 1 and 0 indicate idle and busy states, respectively; the server's The CPU idle configuration is as follows:

The edge collaborators have acausal knowledge of the characteristics of edge servers with no CPU idle; the collaborators are assumed to reserve a q-bit buffer for storing offloaded data before processing in the CPU;

表示组合事件空间，α₃表示新的任务状态到达边缘服务器，对应的过程是一个变量序列：Consider two forms of data arrival on the edge server; a task arrival assumes that the input L-bit arrives at time t = 0, so the event space and process of the edge server CPU follow the appeal; on the other hand, burst data arrivals form a random process; for the case of burst data arrival, use

represents the combined event space, α ₃ represents the arrival of the new task state to the edge server, and the corresponding process is a variable sequence:

让L_k表示数据到达的大小，L_k＝0表示α₁和α₂状态，L_k≠0表示α₃的状态；

此外否则到达截止日期的任务无法计算，然后输入总数据

然后将其每时刻的计算资源状态作为一个状态样本；{α ₁ , α ₂ , α ₃ ...} represent the time instants of the sequence of events; moreover, every moment

Let L _k denote the size of data arrival, L _k = 0 for α ₁ and α ₂ states, and L _k ≠ 0 for α ₃ states;

Also otherwise tasks that have reached their due date cannot be counted, then enter the total data

Then take its computing resource status at every moment as a status sample;

Actions will be selected by selecting state samples to represent how to cooperate between two different adjacent edge servers; corresponding to a specific change/movement between two different adjacent states; use variable v to represent the number of different time states v = 1,2,...NM+3N, then consider the action a(t)={a _v (t)}, the action 1×(NM+3N) depends on the choice of v, for the choice of action, there are the following actions;

When 1 ≤ v ≤ NM the corresponding action a _v (t) means how to change the task x _nm offloading decision; specifically, use:

If a _v (t) = 1, then

If a _v (t) = 0, then

是整型运算，mod(v,M)是余数运算，找到相应的服务器任务v；

is an integer operation, mod(v, M) is a remainder operation, find the corresponding server task v;

When NM+1≤v≤NM+N and NM+N+1≤v≤NM+2N, the corresponding action a _v (t) is expressed as arranging collaborative computing resources for the edge server, and the action is:

If a _v (t) = 1, then

If a _v (t) = 0, then

其中C^co为边缘服务器的CPU处理计算任务的周期数，C^co,max为CPU计算的最大周期，N^do,tot为CPU核数。where the update computing resources are

Among them, C ^co is the number of cycles that the CPU of the edge server processes computing tasks, C ^co,max is the maximum cycle of CPU computing, and N ^do,tot is the number of CPU cores.

Optionally, in the third step, the system state of any edge server is defined based on the selection of CPU and computing tasks as state samples and defined actions. In the initial stage, corresponding actions are taken for the corresponding states, and the state samples are selected. As a state at a given time, and take a specific action after the action, in order to obtain the maximum cumulative reward, that is, the Q value;

Q _π (s,a)=E _π [r _t+1 +γr _t+1 +...|A _t =a,S _t =s]

Q(s,a)←Q(s,a)+δ[r+γmax _a Q(s`,a`)-Q(s,a)]

Where Q(s, a) is the value of the action state function, the discount starts at time t, and the influence of γ decay on the Q function, the closer γ is to 1, the more influence it has on subsequent decisions, the closer γ is to 0, the more it values the current benefit value;

At this time, there will be an experience tuple table D _t =(e ₁ ,...,e _t ) to record the value of each group of state and action; then in experience replay, the experience tuple e ₁ =(s _t ,a, r _t , s _t+1 ) are not full, and randomly select state samples at each time step t in experience replay to obtain experience tuples, and then store each experience tuple in experience replay to accumulate experience.

Optionally, in the fourth step, a different method is adopted, and the experience tuple e ₁ =(s _t , a, r _t , s _t+1 ) in its Experience replay is brought into two identical neural networks Training is the target network target net and the evaluation network eval net respectively; the output value Q ^tar of the targetnet represents the decay score when the behavior a is selected under the current state sample s of the edge server, namely:

where r and s` represent the corresponding score and the corresponding next observation state when the edge server takes action a in the current state s, respectively; γ is the decay factor; a` is the action taken by the edge server in state s`, w` is the weight parameter of target net;

The output value of evalnet Q ^eval represents the score of taking action a under the current state sample s of the edge server:

Where w is the weight parameter of eval net;

The ε-greedy strategy is also used to obtain behavior a. While making decisions based on the network, it is possible to explore multiple cooperative edge servers while cooperating with a single edge server with a certain probability; constantly update the experience tuples in the experience pool, and As the input of target net and eval net, Q ^eval and Q ^tar are obtained; the difference between Q ^eval and Q ^tar is used as the loss function Loss function, and the weight parameters of the evaluation network are updated by gradient descent method; for training convergence, the target network's weight parameters The weight parameters are updated by copying the weight parameters of the evaluation network at regular intervals. The model is as follows:

where s _t and a represent the current state and current action of the edge server respectively, r represents the reward obtained by taking this action, γ is the expected discount factor, and s _t+1 represents the state of the next step in the future, w vector for deep neural network fitting;

Then, the gradient descent algorithm is used to minimize the difference between the target network output and the prediction, i.e. the main network output:

Loss=(Q ^target (s _t ,a)-Q ^pre (s _t ,a,w)) ²

Finally, the two neural networks are trained using the experience tuple, and the Q value is continuously iterated, so that the edge server will obtain an optimal approximate solution under the condition that the edge server is limited by the computing resource state. optimal strategy.

The beneficial effect of the invention is that the collected computing resource state and the corresponding selection action are put into the experience pool as samples, and then a sample is randomly selected from the experience pool for two-line neural network training. The first part of this treatment breaks the connection between the samples and makes the samples independent of each other. Fixed-Q target network: To calculate the target value of the network, an existing Q value is required. A slower update network is used to provide this Q value. This improves the stability and convergence of training to further improve the efficiency and cost of collaboration between edge servers.

Other advantages, objects, and features of the present invention will be set forth in the description that follows, and will be apparent to those skilled in the art based on a study of the following, to the extent that is taught in the practice of the present invention. The objectives and other advantages of the present invention may be realized and attained by the following description.

Description of drawings

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be preferably described in detail below with reference to the accompanying drawings, wherein:

Fig. 1 is a flow chart of a computing resource collaboration method based on deep reinforcement learning;

Figure 2 is a schematic diagram of edge server deployment;

Figure 3 is a schematic diagram of the collaboration of the demand for the optimal solution.

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments are only used to illustrate the basic idea of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

Among them, the accompanying drawings are only used for exemplary description, and represent only schematic diagrams, not physical drawings, and should not be construed as limitations of the present invention; in order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings will be omitted, The enlargement or reduction does not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the accompanying drawings may be omitted.

The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms “upper”, “lower”, “left” and “right” , "front", "rear" and other indicated orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must be It has a specific orientation, is constructed and operated in a specific orientation, so the terms describing the positional relationship in the accompanying drawings are only used for exemplary illustration, and should not be construed as a limitation of the present invention. situation to understand the specific meaning of the above terms.

The technical scheme that the present invention solves the above-mentioned technical problems is:

Please refer to FIG. 1 to FIG. 3 . The following description will be given in conjunction with the accompanying drawings. The present invention includes the following steps:

Step 1: Edge servers spend negligible effort and time in receiving collaborative computation results, as they are usually much smaller than their offload counterparts. Extending the current analysis to include overhead is simple and tedious. Considering the system model, N mobile users offload their computing tasks to edge servers through wireless links. Each user has M independent tasks to complete. To model the task, the cellular network shape is used to maximize the coverage utilization of edge servers. Then, by cooperatively optimizing the offloading decision of each edge task and the allocation of server computing resources, as well as the transmission and reception of tasks, an optimization case with the goal of minimizing the energy consumption of completing computing tasks and making full use of computing resources is formulated as shown in Figure 2. Show.

Step 2: Treat each edge server as an agent, and take its computing resource status such as CPU, task volume, energy consumption, etc. as a status sample at each moment, in which the partner's cpu idle configuration file is defined as the data of the terminal device ( in bits), the partners can compute these data for the duration t∈[0,T], which is denoted as U _bit (t). The information of the collaborative edge servers with idle CPUs is as follows: The collaborative edge server CPU status information refers to the state of the CPU over time, which can be recorded by defining the following collaborative CPU event space, process and epoch, where α={α ₁ ,α ₂ }, represents the CPU state space sample of the cooperative edge server, α ₁ and α ₂ cooperate with the edge server from busy to idle, and then from idle to busy respectively. A co-processor process can then be defined as a time instant of a sequence of co-processor events:

称为一个epoch。Two consecutive events are accompanied by a long time interval T _k =s _k -s _k-1 , where

called an epoch.

The CPU's process is a sample path that allows offline design of collaborative computing strategies, such as collaborator CPU processes. For each epoch k, let _Ik denote the CPU status indicator, where the values 1 and 0 represent idle and busy states, respectively. However, let f _h denote the constant CPU frequency of the collaborator, and let C denote the number of CPU cycles required to calculate the user's 1-bit input data. Based on the above definition, the server's CPU idle configuration is as follows:

Edge collaborators have acausal knowledge of the characteristics of edge servers with no CPU idle. Finally, assume that the collaborator reserves a q-bit buffer for storing offloaded data before processing in the CPU.

表示组合事件空间，α₁,α₂上述已经介绍，α₃表示新的任务状态到达边缘服务器，对应的过程是一个变量序列:Consider two forms of data arrival at the edge server. A task arrival assumes that the input L-bit arrives at time t=0, so the event space and process of the edge server CPU follow the appeal. On the other hand, bursts of data arrivals form a random process. For a short specification, it is useful to define a random process that combines the data arrivals of the two processes and the CPU of the edge server. Hence the combinatorial random process for burst task arrivals. For burst data arrival, use

Represents the combined event space, α ₁ , α ₂ have been introduced above, α ₃ indicates that the new task state arrives at the edge server, and the corresponding process is a variable sequence:

让L_k表示数据到达的大小，L_k＝0表示α₁和α₂状态，L_k≠0表示α₃的状态。另外，

此外否则到达截止日期的任务无法计算，然后输入总数据

然后将其每时刻的CPU、任务量等计算资源状态作为一个状态样本。{α ₁ , α ₂ , α ₃ ...} represent the time instants of the sequence of events. And, every moment

Let L _k denote the size of data arrival, L _k = 0 for the α ₁ and α ₂ states, and L _k ≠ 0 for the α ₃ state. in addition,

Also otherwise tasks that have reached their due date cannot be counted, then enter the total data

Then, its computing resource status such as CPU and task amount at each moment is taken as a status sample.

Actions are selected by selecting state samples to represent how to cooperate between two different adjacent edge servers. Specifically, corresponds to a specific change/movement between two different adjacent states. Use the variable v to represent the number of different time states v = 1, 2, ... NM + 3N, and then consider the action a(t) = {a _v (t)}, the action 1 × (NM + 3N) depends on the choice of v , for the selection of operations, there are the following actions.

When 1≤v≤NM the corresponding action a _v (t) means how to change the offloading decision of task x _nm . Specifically, use:

If a _v (t) = 1, then

If a _v (t) = 0, then

是整型运算，mod(v,M)是余数运算，然后我们可以找到相应的服务器任务v。

is an integer operation, mod(v, M) is a remainder operation, and then we can find the corresponding server task v.

When NM+1≤v≤NM+N and NM+N+1≤v≤NM+2N, the corresponding action a _v (t) is expressed as arranging collaborative computing resources for the edge server, and the action is:

If a _v (t) = 1, then

If a _v (t) = 0, then

其中C^co为边缘服务器的CPU处理计算任务的周期数，C^co,max为CPU计算的最大周期，N^do,tot为CPU核数。where the update computing resources are

Among them, C ^co is the number of cycles that the CPU of the edge server processes computing tasks, C ^co,max is the maximum cycle of CPU computing, and N ^do,tot is the number of CPU cores.

Step 3: Based on the CPU and computing tasks as state samples and defining the selection of actions, the system state of any edge server is defined. In the initial stage, corresponding actions are taken for the corresponding states, and the state samples are selected as the state samples at a given time. state, and take an action after a specific action, in order to obtain the maximum cumulative reward, which is the Q value.

Q _π (s,a)=E _π [r _t+1 +γr _t+1 +...|A _t =a,S _t =s]

Q(s,a)←Q(s,a)+δ[r+γmax _a Q(s`,a`)-Q(s,a)]

Where Q(s, a) is the value of the action state function, the discount starts at time t, and the influence of γ decay on the Q function, the closer γ is to 1, the more influence it has on subsequent decisions, the closer γ is to 0, the more it values the current benefit value.

At this time, there will be an experience tuple table D _t = (e ₁ ,...,e _t ) to record the value of each group of state and action. At this time, the experience tuple e ₁ =(s _t , a, r _t , s _t+1 ) is not full in the experience replay, and randomly selects a state sample at each time step t in the experience replay to obtain the experience tuple, and then Store each experience tuple in experience replay to accumulate experience.

Step 4: Take a different method from before, and bring the experience tuple e ₁ = (s _t , a, r _t , s _t+1 ) in its Experience replay into two identical neural networks (three convolutional layers and two A fully connected layer) are trained in targetnet (target network) and eval net (evaluation network). The output value Q ^tar of the target net represents the decay score when the behavior a is selected under the current state sample s of the edge server, namely:

where r and s` represent the corresponding score and the corresponding next observation state when the edge server takes action a in the current state s, respectively; γ is the decay factor; a` is the action taken by the edge server in state s`, w` is the weight parameter of target net.

The output value of evalnet Q ^eval represents the score of taking action a under the current state sample s of the edge server:

where w is the weight parameter of eval net.

It also adopts the ε-greedy strategy (ε gradually decreases from 1 to 0) to obtain behavior a. While making decisions based on the network, it can cooperate with a certain probability to be adjacent to a single edge server, and it is also possible to explore multiple cooperative edge servers. In order to avoid the problem of optimal solution. Here, the experience tuples in the experience pool will be continuously updated and used as the input of target net and evalnet to obtain Q ^eval and Q ^tar . Taking the difference between Q ^eval and Q ^tar as the Loss function (loss function), the weight parameters of the evaluation network are updated by the gradient descent method. For training convergence, the weight parameters of the target network are updated by copying the weight parameters of the evaluation network at regular intervals. The model is as follows:

where s _t and a represent the current state and current action of the edge server respectively, r represents the reward obtained by taking this action, γ is the expected discount factor, and s _t+1 represents the state of the next step in the future, w Vector for deep neural network fitting.

Then, use the gradient descent algorithm to minimize the difference between the target network output and the prediction:

Loss=(Q ^target (s _t ,a)-Q ^eval (s _t ,a,w)) ²

Finally, the two neural networks are trained using the experience tuple, and the Q value is continuously iterated, so that the edge server will obtain an optimal approximate solution under the condition that the edge server is limited by the computing resource state. optimal strategy.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent replacements, without departing from the spirit and scope of the technical solution, should all be included in the scope of the claims of the present invention.

Claims (5)

1. a computing resource collaboration method based on deep reinforcement learning, is characterized in that: the method comprises the following steps: Step 1: For seamless connection, the edge servers are deployed in a cellular configuration in areas with dense 5G networks; Step 2: Treat each edge server as an agent, and record the computing resource status and corresponding actions for a period of time as samples into experiencereplay; Step 3: In order to increase the independence of samples, randomly select state samples from experiencereplay to obtain experience tuples at each time t, and then store each experience tuple in experiencereplay to accumulate experience and store; Step 4: Then through the target network targetnet and the evaluation network evalnet to iterate the Q value and get the new state and put it into experiencereplay again, use the loss function to update the weight parameters, finally get the optimal approximate solution, and obtain the optimal decision for edge server collaboration . 2. A kind of computing resource collaboration method based on deep reinforcement learning according to claim 1, is characterized in that: in described step 1, the energy and time spent by edge server on receiving collaborative computing result are neglected; Considering the system model, N mobile users offload computing tasks to edge servers through wireless links; each user has M independent tasks to complete; To model tasks, the cellular network shape is used to maximize the coverage utilization of edge servers; offloading decisions and server computing resource allocation for each edge task are optimized cooperatively, as well as task transmission and reception, formulated to minimize completion of computing tasks An optimization case aiming at the full utilization of energy consumption and computing resources. 3. A kind of computing resource collaboration method based on deep reinforcement learning according to claim 1, it is characterized in that: in described step 2, regard each edge server as an agent, its CPU, task at every moment The computing resource state of the amount and energy consumption is taken as a state sample, in which the partner's cpu idle profile is defined as the data of the terminal device, and the partner computes these data for the duration t ∈ [0, T], denoted as U _bit (t); The information of collaborative edge servers with idle CPUs is as follows: The collaborative edge server CPU status information refers to the state of the CPU over time, which is recorded by defining the following collaborative CPU event space, process and epoch, where α={α ₁ , α ₂ }, represents the CPU state space sample of the cooperative edge server, α ₁ and α ₂ cooperate with the edge server from busy to idle, and then from idle to busy respectively; then the cooperative edge processor process is defined as the time instant of the co-processor event sequence :

Two consecutive events are accompanied by a long time interval T _k =s _k -s _k-1 , where

called an epoch; The process of the CPU is a sample path of the partner CPU process that allows offline design of collaborative computing strategies. For each epoch k, let I _k denote the CPU status indicator, where the values 1 and 0 indicate idle and busy states, respectively; the server's The CPU idle configuration is as follows:

The edge collaborators have acausal knowledge of the characteristics of edge servers with no CPU idle; the collaborators are assumed to reserve a q-bit buffer for storing offloaded data before processing in the CPU; Consider two forms of data arrival on the edge server; a task arrival assumes that the input L-bit arrives at time t=0, so the event space and process of the edge server CPU follow the appeal; on the other hand, burst data arrivals form a random process; for the case of burst data arrival, use

represents the combined event space, α ₃ represents the arrival of the new task state to the edge server, and the corresponding process is a variable sequence:

{α ₁ , α ₂ , α ₃ ...} represent the time instants of the sequence of events; moreover, every moment

Let L _k denote the size of data arrival, L _k = 0 for α ₁ and α ₂ states, and L _k ≠ 0 for α ₃ states;

Also otherwise tasks that have reached their due date cannot be counted, then enter the total data

Then take its computing resource status at every moment as a status sample; Actions will be selected by selecting state samples to represent how to cooperate between two different adjacent edge servers; corresponding to a specific change/movement between two different adjacent states; use variable v to represent the number of different time states v = 1,2,...NM+3N, then consider the action a(t)={a _v (t)}, the action 1×(NM+3N) depends on the choice of v, for the choice of action, there are the following actions; When 1 ≤ v ≤ NM the corresponding action a _v (t) means how to change the task x _nm offloading decision; specifically, use: If a _v (t) = 1, then

If a _v (t) = 0, then

is an integer operation, mod(v, M) is a remainder operation, find the corresponding server task v; When NM+1≤v≤NM+N and NM+N+1≤v≤NM+2N, the corresponding action a _v (t) is expressed as arranging collaborative computing resources for the edge server, and the action is: If a _v (t) = 1, then

If a _v (t) = 0, then

where the update computing resources are

Among them, C ^co is the number of cycles that the CPU of the edge server processes computing tasks, C ^co,max is the maximum cycle of CPU computing, and N ^do,tot is the number of CPU cores. 4. a kind of computing resource collaboration method based on deep reinforcement learning according to claim 1, is characterized in that: in described step 3, based on CPU and computing task as the selection of state sample and definition action, thereby define any edge The system state of the server, in the initial stage, take the corresponding action for the corresponding state, select the state sample as the state at a given time, and take a specific action after the action, in order to obtain the maximum cumulative reward, that is, the Q value; Q _π (s,a)=E _π [r _t+1 +γr _t+1 +...|A _t =a,S _t =s] Q(s,a)←Q(s,a)+δ[r+γmax _a Q(s`,a`)-Q(s,a)] Where Q(s, a) is the value of the action state function, the discount starts at time t, and the influence of γ decay on the Q function, the closer γ is to 1, the more influence it has on subsequent decisions, the closer γ is to 0, the more it values the current benefit value; At this time, there will be an experience tuple table D _t =(e ₁ ,...,e _t ) to record the value of each group of state and action; then in experience replay, the experience tuple e ₁ =(s _t ,a, r _t , s _t+1 ) are not full, and randomly select state samples at each time step t in experience replay to obtain experience tuples, and then store each experience tuple in experience replay to accumulate experience. 5. a kind of computing resource collaboration method based on deep reinforcement learning according to claim 1, is characterized in that: in described step 4, adopt different method from before, experience tuple e ₁ = (s _t , a, r _t , s _t+1 ) are brought into two identical neural networks for training, namely the target network target net and the evaluation network eval net; the output value Q ^tar of the target net represents the current state sample of the edge server The decay score when behavior a is selected under s, that is:

where r and s` represent the corresponding score and the corresponding next observation state when the edge server takes action a in the current state s, respectively; γ is the decay factor; a` is the action taken by the edge server in state s`, w` is the weight parameter of target net; The output value Q ^eval of eval net represents the score of taking action a under the current state sample s of the edge server:

Where w is the weight parameter of eval net; The ε-greedy strategy is also used to obtain behavior a. While making decisions based on the network, it is possible to explore multiple cooperative edge servers while cooperating with a single edge server with a certain probability; constantly update the experience tuples in the experience pool, and As the input of target net and eval net, Q ^eval and Q ^tar are obtained; the difference between Q ^eval and Q ^tar is used as the loss function Loss function, and the weight parameters of the evaluation network are updated by gradient descent method; for training convergence, the target network's weight parameters The weight parameters are updated by copying the weight parameters of the evaluation network at regular intervals. The model is as follows:

where s _t and a represent the current state and current action of the edge server respectively, r represents the reward obtained by taking this action, γ is the expected discount factor, and s _t+1 represents the state of the next step in the future, w vector for deep neural network fitting; Then, the gradient descent algorithm is used to minimize the difference between the target network output and the prediction, i.e. the main network output: Loss=(Q ^target (s _t ,a)-Q ^pre (s _t ,a,w)) ² Finally, the two neural networks are trained using the experience tuple, and the Q value is continuously iterated, so that the edge server will obtain an optimal approximate solution under the condition that the edge server is limited by the computing resource state. optimal strategy.

Images