CN115640131A - Unmanned aerial vehicle auxiliary computing migration method based on depth certainty strategy gradient - Google Patents

Abstract

The invention provides a calculation task unloading algorithm based on deep reinforcement learning aiming at the requirements of calculation intensive and delay sensitive mobile services. And (4) considering constraint conditions such as the flight ranges, the flight speeds, the system fairness benefits and the like of the multiple unmanned aerial vehicles, and minimizing the weighted sum of the network average calculation delay and the energy consumption of the unmanned aerial vehicles. The non-convex and NP difficult problems are converted into a partial observation Markov decision process, and the multi-agent depth certainty strategy gradient algorithm is used for unloading decision of the mobile user and optimizing the flight track of the unmanned aerial vehicle. Simulation results show that the performance of the algorithm is superior to that of a baseline algorithm in the aspects of fairness of the mobile service terminal, average system time delay, total energy consumption of multiple unmanned aerial vehicles and the like.

Description

A UAV-aided Computational Migration Method Based on Deep Deterministic Policy Gradients

technical field

The invention belongs to the field of mobile edge computing (Mobile Edge Computing, MEC), relates to a multi-UAV assisted mobile edge computing method, more specifically, relates to a strategy gradient based on multi-agent depth determination (Multi-Agent Deep Deterministic Policy Gradient, MADDPG) computational migration method.

Background technique

With the development of 5G technology, computing-intensive applications running on user equipment, such as online games, VR/AR, and telemedicine, will become more prosperous and popular. These mobile applications usually require a lot of computing resources, consume a lot of energy, and due to the limited coverage of the server, the user may lose the connection with the server while moving. The server that originally requested to offload cannot send the calculation results in time at the user's next location, which will cause a waste of server computing resources and increase the delay and energy consumption of the user's re-uploading and offloading calculation tasks. For the user's offloadable tasks, many studies will take the method of offloading them all to the MEC server for execution. However, when the number of users is large or the amount of offloaded tasks is large, the limited server computing resources will lead to task queuing, and the offloading calculation delay will increase. . Due to high mobility and flexibility, UAV (Unmanned Aerial Vehicle, UAV) can assist mobile edge computing in military and civilian fields without relying on infrastructure, especially in remote areas or natural disaster areas. When the network infrastructure is unavailable due to natural disasters or the sudden increase of mobile devices exceeds the network service capacity, drones can be used as temporary communication relay stations or edge computing platforms to enhance wireless coverage and provide computing support in areas where communication is interrupted or traffic hotspots . However, the computing resources and power of drones are limited. In order to improve the performance of the MEC system, there are many key issues that need to be solved, including security ^[8] , task offloading, energy consumption, resource allocation, and users under various channel conditions. Latency performance etc.

In the UAV MEC network, various types of variables (such as UAV trajectory, task offloading strategy, and computing resource allocation) can be optimized to achieve the desired scheduling goal. Traditional optimization methods require a large number of iterations and priori knowledge to obtain an approximate optimal solution, so it is not suitable for real-time MEC applications in dynamic environments. With the widespread application of machine learning in research, many researchers are also exploring learning-based MEC scheduling algorithms, and deep reinforcement learning has now become a research hotspot in view of the latest advances in machine learning. As the network scale grows, multi-agent deep reinforcement learning provides a distributed perspective for resource management of multi-UAV MEC networks.

The present invention proposes a UAV-assisted mobile edge computing system, which uses the computing resources provided by the UAV to provide offloading services for nearby user equipment. Solve the UAV trajectory and unloading optimization problem through the method of multi-agent deep reinforcement learning, so as to obtain a scalable and effective scheduling strategy. The terminal offloads part of the computing tasks to the UAV, while the rest of the tasks are performed locally on the terminal. System processing delay and UAV energy consumption are minimized by jointly optimizing user scheduling, task offload ratio, UAV flight angle and flight speed.

Contents of the invention

Purpose of the invention: Considering the non-convexity, high-dimensional state space and continuous action space of the problem, we propose a deep reinforcement learning algorithm based on MADDPG, which can obtain the optimal calculation unloading strategy in a dynamic environment, so as to realize Joint optimization for lowest system latency and UAV energy consumption.

Technical solution: Considering the scenario of multi-user computing task offloading at the same time, the purpose of joint optimization of system delay and UAV energy consumption is achieved through reasonable and efficient UAV path planning and offloading decisions. Considering each UAV as an agent, using distributed execution and centralized training, it selects associated users based on locally observed state information and task information obtained in each time slot. By establishing a deep reinforcement learning model, the MADDPG algorithm is used to optimize the deep reinforcement learning model. According to the optimized MADDPG model, the optimal flight trajectory and unloading strategy are obtained. The above-mentioned invention is achieved through the following technical solutions: a MADDPG-based unmanned aerial vehicle-aided calculation migration method, including the following steps:

(1) Traditional MEC servers are deployed in base stations or other fixed facilities. This time, mobile MEC servers are used to combine UAV technology with edge computing;

(2), the user equipment offloads computing tasks to the UAV side through wireless communication to reduce computing delays;

(3) Construct the UAV-assisted user unloading system model, mobile model, communication model and calculation model, and give the optimization objective function;

(4), the UAV obtains the user location set, task set and service times and channel parameter information within the observation range;

(5), using Partially Observable Markov Decision Process (POMDP) modeling, considering the flight range and safety distance of the UAV, based on the user's location and task information, jointly optimize the The flight trajectory and calculation offloading strategy of the man-machine, with the goal of minimizing system delay and UAV energy consumption while ensuring user service fairness, builds a deep reinforcement learning model;

(6), considering continuous state space and continuous action space, using MADDPG-based multi-agent deep reinforcement learning algorithm for model training of computational migration;

(7), in the execution phase, based on the state s(τ) of the current environment, the UAV uses the trained model to obtain the optimal user offloading scheme and flight trajectory;

Further, the step (3) includes the following specific steps:

和

表示。无人机以固定高度H_u飞行，设无人机执行一次飞行任务的总时长为T，总时长可被分为N个等长的时隙，时隙的集合记为

每个MD在每个时隙τ有一个计算密集型任务，任务记为S_m(τ)＝{D_m(τ)，F_m(τ)}，其中D_m(τ)表示数据比特量，F_m(τ)表示每比特所需CPU周期；(3a), establish a mobile edge computing system model for UAV-assisted user offloading. The system has M mobile user devices (Mobile Device, MD) and U UAVs equipped with MEC servers.

and

express. The UAV flies at a fixed height Hu, assuming that the total time for the _UAV to perform a flight mission is T, the total time can be divided into N time slots of equal length, and the set of time slots is recorded as

Each MD has a computationally intensive task in each time slot τ, and the task is recorded as S _m (τ) = {D _m (τ), F _m (τ)}, where D _m (τ) represents the amount of data bits, F _m (τ) represents the required CPU cycles per bit;

(3b), each UAV only provides computing offloading service for one terminal device in each time slot τ, the user only needs to calculate a small part of the task locally, and the rest is offloaded to the UAV’s auxiliary computing to reduce the computational cost Delay and energy consumption, the proportion of unloaded computation is recorded as Δ _{m, u} (τ) ∈ [0, 1]. The offloading decision variables between the UAV and the user equipment can be expressed as:

D={α _{m, u} (τ)|u∈U, m∈M, τ∈T} Expression 1

Where α _{m, u} (τ) ∈ {0, 1}, when α _{m, u} (τ) = 1, it means that the calculation task of device MD _m in time slot τ is assisted by UAV _u , Δ _{m, u} (τ)>0; when α _m,u (τ)=0, it means that the computing task is only performed locally, and Δ _m,u (τ)=0. Decision variables need to satisfy:

(3c), establish a mobile model, the mobile device will randomly move to a new position in each time slot, and the movement of each device is related to its current speed and angle. Assuming that the coordinates of MD _m in the time slot τ are marked as c _m (τ)=[x _m (τ), y _m (τ)], then the coordinates of the next time slot τ+1 can be expressed as:

Among them, d _max represents the maximum moving distance of the standby, and the moving direction and distance probability all obey the uniform distribution, ρ _{1, m} , ρ _{2, m} ~ U(0, 1), and the UAV service terminal only considers its time slot the starting position of .

(3d), the horizontal trajectory of each UAV at height Hu can also be represented by the UAV’s discrete position c _u (τ _{) at each time slot, assuming that UAV u chooses} to fly to serve MD at time slot τ _m , the flight direction is recorded as β _u (τ)∈[0, 2π], the flight speed is v _u (τ)∈[0, V _max ], the flight time is t _fly , and the energy consumed by UAV flight is :

Where μ=0.5M _u t _fly , M _u is the total mass of the UAV.

(3e), the calculation offload adopts a partial offload strategy, then the local calculation delay of MD _m in time slot τ can be expressed as:

Where f _m represents the local computing power of MD _m (the number of CPU cycles per second).

(3f), using the line-of-sight link model to simulate the actual UAV-to-ground communication, the channel gain _hm,u (τ) between the UAV and the user follows the free-space path loss model, which can be expressed as:

where _g0 is the channel power gain per meter.

(3g), the instantaneous transmission rate r _m,u (τ) between the UAV and the ground equipment is defined as:

为移动设备上传链路的发射功率，σ²代表无人机端的高斯白噪声。where B represents the channel bandwidth,

is the transmitting power of the upload link of the mobile device, and σ ² represents the Gaussian white noise at the UAV side.

The transmission data delay of associated user MD _m is:

After the calculation task transmission is completed, the UAV performs the offloading calculation task, and the delay and energy consumption of the offloading calculation are respectively:

表示无人机执行计算时的CPU功率，κ_u＝10^-27为芯片常数where f _u represents the computing power of the UAV,

Indicates the CPU power when the UAV performs calculations, κ _u = 10 ^-27 is the chip constant

(3h), since the amount of output data of various computationally intensive tasks is much smaller than the input, so the delay spent in downlink transmission can be ignored, when user MD _m completes task S _m (τ) in time slot τ The extension T _m (τ) can be expressed as:

The total energy consumption of unmanned aerial vehicle UAV _u auxiliary computing offloading in time slot τ is:

(3i), the average delay of user MD _m can be expressed as:

The average computing delay of the system can be calculated as:

(3j), in order to ensure user fairness, define the fairness index ξ _τ to measure the fairness of the service:

(3k), in summary, the following objective functions and constraints can be established:

where P＝{β _u (τ), v _u (τ)}, Z＝{α _{m, u} (τ), Δ _{m, u} (τ)}, φ _t and φ _e are weight parameters, and C1 restricts no one Each time slot of the machine only serves one user, C2 and C6 limit the flight range of the drone, C3 and C4 limit the flight speed and angle of the drone, C6 indicates that the calculation task can be partially offloaded, and C7 guarantees the fair benefit of the system. d _safe and ξ _min are the pre-set minimum safe distance and minimum fairness index between UAVs.

Further, the step (5) includes the following specific steps:

(5a), consider the multi-UAV assisted computation offloading problem as a partial observation Markov decision process, consisting of tuples {S, A, O, Pr, R}. There are usually multiple agents interacting with the environment, each agent obtains its own observation o _τ ∈ O based on the current state s _τ and makes an action a _τ ∈ A, the environment generates an immediate reward r _τ ∈ R for the action to evaluate the current action Good or bad, and enter the next state with probability Pr(S _τ+1 |S _τ , A _τ ), the new state only depends on the current state and the actions of each agent. The actions of the agent are executed based on the policy π(a _τ |o _τ ), and its goal is to learn the optimal policy to maximize the long-term cumulative reward, which can be expressed as:

where γ is the reward discount.

动作空间A为发射功率和选择的信道，表示为：(5b), specifically define the observation space, each UAV has only a limited observation range, and the radius of the observation range is set to r _obs , so only part of the state information can be observed, while the global state information and other UAV's Actions are unknown. The information that a single unmanned aerial vehicle UAV _u can observe in the time slot τ includes its own position information c _u (τ) and the current position information, task information and service times of K mobile users within the observation range

The action space A is the transmit power and the selected channel, expressed as:

o _u (τ) = {c _u (τ), k _u (τ)} Expression 18

(5c), specifically define the action space. Based on the observed information, the UAV needs to determine which user to serve in the current time slot τ and the unloading ratio Δ _{m, u} (τ), and then determine its own flight angle β _u ( τ) and flight speed v _u (τ), can be written as:

a _u (τ) = {m(τ), Δ _{m, u} (τ), β _u (τ), v _u (τ)} Expression 19

(5d), define the state space, the state of the system can be regarded as the collection of all UAV observations:

s(τ)={o _u (τ)|u∈U} Expression 20

(5e), specifically define the reward, the feedback obtained by the agent after performing the action is called the reward, which is used to judge the quality of the action and guide the agent to update its strategy. Generally speaking, the reward function corresponds to the optimization goal. The goal of this optimization is to minimize the energy consumption of the UAV and the average calculation delay of the system, which is exactly negatively correlated with the maximum reward return. Therefore, the UAV performs the action The latter reward is defined as:

r _u (τ)＝D _m (τ)·(-T ^mean (τ)-ψE _u (τ)-P _u (τ)) Expression 21

Among them, D _m (τ) ∈ [0, 1] is the attenuation coefficient, which is defined as the benefit obtained after the UAV handles the unloading task of the mobile terminal. The specific calculation is as follows:

Among them, η and β are related constants, and its function image is sigmoid-like, and the input is the cumulative service times of the current user. The more times, the larger the value, the smaller the reward, and the lower the benefit. ψ is used to numerically align UAV energy consumption and user average delay. P _u (τ) is an additional penalty item. If the UAV flies out of the field after performing an action or the distance with other UAVs is less than the safe distance, the penalty needs to be increased.

(5f), according to the established S, A, O and R, establish a deep reinforcement learning model on the basis of MADDPG, using the actor-critic framework, each agent has its own actor network and critic network, and their own target network. The Actor network is responsible for formulating strategies for the agent π(o _u (τ)|θ _u ), θ _u represents its network parameters; the critic network output estimates the optimal state-action value function as Q(s(τ), a ₁ (τ),..., a _U (τ)|w _u ), w _u represents its network parameters. The input of the critic network contains the observations and actions of all agents in a time slot, but during distributed execution, the input of the actor network only needs its own observations.

The algorithm learns the Q function and the optimal strategy at the same time. When updating the critic network, it needs to extract H groups of records from the experience pool of each agent, and splice each group at the same time to obtain H new records, which are recorded as: { s _{t, i} , a _{1, i} ,..., a _{U, i} , r _{1, i} ,..., r _{U, i} , s _{t+1, i} |i=1, 2, .., H }, use the temporal difference to train the critic network of each agent intensively, and the loss function for training the Q-value function is defined as:

where y _{u, i} can be obtained from formula (24):

和

分别代表无人机UAV_u的critic目标网络和actor目标网络，目标网络都具有滞后更新的网络参数，使训练变得更稳定。in,

and

represent the critic target network and the actor target network of the UAV _u , respectively, and the target network has network parameters that are updated with a lag, making the training more stable.

The critic network needs to reduce the loss as much as possible to approach the real Q ^* value, and the actor network uses the determined strategy gradient of the Q value for gradient ascent to update the network parameters to maximize the action value:

更新目标网络：Finally at regular intervals with an update rate of

Update target network:

Further, the step (6) includes the following specific steps:

(6a), start the environment simulation, initialize each agent actor network and critic network and their respective target network parameters;

(6b), initialize the number of training rounds;

(6c), update the user's location set, task set and service times, the location set and channel parameters of the drone;

(6d), run the actor network for each agent distribution, output the action a _u (τ) according to the observation o _u (τ), and get the immediate reward r _u (τ), and go to the next state s _{τ+ 1} , so as to obtain the training data {o _u (τ), a _u (τ), r _u (τ), o _u (τ+1)};

(6e), store the training data in their respective experience playback pools;

(6f), each agent randomly samples H training data from the experience playback pool to form a training data set;

(6g), each agent calculates the loss value L(w _u ) through the critic network and the target network to update w _u , and uses the determined strategy gradient for gradient ascent, and updates the parameter θ of the actor network through the backpropagation of the neural network _u ;

(6h), the number of training times reaches the target network update interval, and the target network parameters are updated;

(6i), judging whether convergence is satisfied, if so, the optimized deep reinforcement learning model is obtained after optimization, otherwise enter step (6c);

Further, the step (7) includes the following specific steps:

(7a), using the deep reinforcement learning model trained by the MADDPG algorithm, input the state information at a certain moment;

得到最优的迁移策略和飞行路径。(7b), output the optimal action strategy

Get the optimal migration strategy and flight path.

Beneficial effects: A large-scale multi-UAV assisted MEC network computing unloading method based on the MADDPG algorithm proposed by the present invention, in the case of satisfying the constraint conditions, can minimize the unloading by jointly optimizing the UAV unloading decision and flight trajectory. The energy consumption of man-machine and the average calculation delay of the system can be stable in a series of optimizations of continuous state space and continuous action space. Under the condition that the scenarios are similar, a deep reinforcement learning algorithm based on MADDPG proposed by the present invention has superior performance in reducing energy consumption and average task delay.

Description of drawings

Fig. 1 is a schematic structural diagram of an unmanned aerial vehicle-assisted computing offloading model provided by an embodiment of the present invention;

Fig. 2 is a schematic diagram of the POMDP decision-making process of the multi-unmanned aerial vehicle calculation migration algorithm provided by the embodiment of the present invention;

Fig. 3 is the schematic diagram of algorithm training based on MADDPG that the embodiment of the present invention provides;

FIG. 4 is a simulation result diagram of the relationship between the energy consumption and computing performance of the drone under the MADDPG algorithm provided by the embodiment of the present invention.

Detailed ways

The core idea of the present invention is: using a distributed reinforcement learning method, each UAV is regarded as an agent, and by establishing a deep reinforcement learning model, the MADDPG algorithm is used to optimize the depth reinforcement learning model. According to the optimized model, the optimal migration strategy and flight path are obtained.

The present invention is described in further detail below.

(1) Traditional MEC servers are deployed in base stations or other fixed facilities. This time, mobile MEC servers are used to combine UAV technology with edge computing;

(2), the user equipment offloads computing tasks to the UAV side through wireless communication to reduce computing delays;

(3) Construct the UAV-assisted user unloading system model, mobile model, communication model and calculation model, and give the optimization objective function;

Including the following specific steps:

和

表示。无人机以固定高度H_u飞行，设无人机执行一次飞行任务的总时长为T，总时长可被分为N个等长的时隙，时隙的集合记为

每个MD在每个时隙τ有一个计算密集型任务，任务记为S_m(τ)＝{D_m(τ)，F_m(τ)}，其中D_m(τ)表示数据比特量，F_m(τ)表示每比特所需CPU周期；(3a), establish a mobile edge computing system model for UAV-assisted user offloading. The system has M mobile user devices (Mobile Device, MD) and U UAVs equipped with MEC servers.

and

express. The UAV flies at a fixed height Hu, assuming that the total time for the _UAV to perform a flight mission is T, the total time can be divided into N time slots of equal length, and the set of time slots is recorded as

Each MD has a computationally intensive task in each time slot τ, and the task is recorded as S _m (τ) = {D _m (τ), F _m (τ)}, where D _m (τ) represents the amount of data bits, F _m (τ) represents the required CPU cycles per bit;

(3b), each UAV only provides computing offloading service for one terminal device in each time slot τ, the user only needs to calculate a small part of the task locally, and the rest is offloaded to the UAV’s auxiliary computing to reduce the computational cost Delay and energy consumption, the proportion of unloaded computation is recorded as Δ _{m, u} (τ) ∈ [0, 1]. The offloading decision variable between UAV and user equipment can be expressed as:

D={α _{m, u} (τ)|u∈U, m∈M, τ∈T} Expression 1

Where α _{m, u} (τ) ∈ {0, 1}, when α _{m, u} (τ) = 1, it means that the calculation task of device MD _m in time slot τ is assisted by UAV _u , Δ _{m, u} (τ)>0; when α _m,u (τ)=0, it means that the computing task is only performed locally, and Δ _m,u (τ)=0. Decision variables need to satisfy:

(3c), establish a mobile model, the mobile device will randomly move to a new position in each time slot, and the movement of each device is related to its current speed and angle. Assuming that the coordinates of MD _m in the time slot τ are marked as c _m (τ)=[x _m (τ), y _m (τ)], then the coordinates of the next time slot τ+1 can be expressed as:

Among them, d _max represents the maximum moving distance of the standby, and the moving direction and distance probability all obey the uniform distribution, ρ _{1, m} , ρ _{2, m} ~ U(0, 1), and the UAV service terminal only considers its time slot the starting position of .

(3d), the horizontal trajectory of each UAV at height Hu can also be represented by the UAV’s discrete position c _u (τ _{) at each time slot, assuming that UAV u chooses} to fly to serve MD at time slot τ _m , the flight direction is recorded as β _u (τ)∈[0, 2π], the flight speed is v _u (τ)∈[0, V _max ], the flight time is t _fly , and the energy consumed by UAV flight is :

Where μ=0.5M _u t _fly , M _u is the total mass of the UAV.

(3e), the calculation offload adopts a partial offload strategy, then the local calculation delay of MD _m in time slot τ can be expressed as:

Where f _m represents the local computing power of MD _m (the number of CPU cycles per second).

(3f), using the line-of-sight link model to simulate the actual UAV-to-ground communication, the channel gain _hm,u (τ) between the UAV and the user follows the free-space path loss model, which can be expressed as:

where _g0 is the channel power gain per meter.

(3g), the instantaneous transmission rate r _m,u (τ) between the UAV and the ground equipment is defined as:

为移动设备上传链路的发射功率，σ²代表无人机端的高斯白噪声。where B represents the channel bandwidth,

is the transmitting power of the upload link of the mobile device, and σ ² represents the Gaussian white noise at the UAV side.

The transmission data delay of associated user MD _m is:

After the calculation task transmission is completed, the UAV performs the offloading calculation task, and the delay and energy consumption of the offloading calculation are respectively:

表示无人机执行计算时的CPU功率，κ_u＝10^-27为芯片常数where f _u represents the computing power of the UAV,

Indicates the CPU power when the UAV performs calculations, κ _u = 10 ^-27 is the chip constant

(3h), since the amount of output data of various computationally intensive tasks is much smaller than the input, so the delay spent in downlink transmission can be ignored, when user MD _m completes task S _m (τ) in time slot τ The extension T _m (τ) can be expressed as:

The total energy consumption of unmanned aerial vehicle UAV _u auxiliary computing offloading in time slot τ is:

(3i), the average delay of user MD _m can be expressed as:

The average computing delay of the system can be calculated as:

(3j), in order to ensure user fairness, define the fairness index ξ _τ to measure the fairness of the service:

(3k), in summary, the following objective functions and constraints can be established:

where P＝{β _u (τ), v _u (τ)}, Z＝{α _{m, u} (τ), Δ _{m, u} (τ)}, φ _t and φ _e are weight parameters, and C1 restricts no one Each time slot of the machine only serves one user, C2 and C6 limit the flight range of the drone, C3 and C4 limit the flight speed and angle of the drone, C6 indicates that the calculation task can be partially offloaded, and C7 guarantees the fair benefit of the system. d _safe and ξ _min are the pre-set minimum safe distance and minimum fairness index between UAVs.

(4), the UAV obtains the user location set, task set and service times and channel parameter information within the observation range;

(5), using Partially Observable Markov Decision Process (POMDP) modeling, considering the flight range and safety distance of the UAV, based on the user's location and task information, jointly optimize the The human-machine flight trajectory and calculation offloading strategy aim to minimize system delay and UAV energy consumption while ensuring user service fairness as the goal, and build a deep reinforcement learning model, including the following specific steps:

(5a), consider the multi-UAV assisted computation offloading problem as a partial observation Markov decision process, consisting of tuples {S, A, O, Pr, R}. There are usually multiple agents interacting with the environment, each agent obtains its own observation o _τ ∈ O based on the current state s _τ and makes an action a _τ ∈ A, the environment generates an immediate reward r _τ ∈ R for the action to evaluate the current action Good or bad, and enter the next state with probability Pr(S _τ+1 |S _τ , A _τ ), the new state only depends on the current state and the actions of each agent. The actions of the agent are executed based on the policy π(a _τ |o _τ ), and its goal is to learn the optimal policy to maximize the long-term cumulative reward, which can be expressed as:

where γ is the reward discount.

动作空间A为发射功率和选择的信道，表示为：(5b), specifically define the observation space, each UAV has only a limited observation range, and the radius of the observation range is set to r _obs , so only part of the state information can be observed, while the global state information and other UAV's Actions are unknown. The information that a single unmanned aerial vehicle UAV _u can observe in the time slot τ includes its own position information c _u (τ) and the current position information, task information and service times of K mobile users within the observation range

The action space A is the transmit power and the selected channel, expressed as:

o _u (τ) = {c _u (τ), k _u (τ)} Expression 18

(5c), specifically define the action space. Based on the observed information, the UAV needs to determine which user to serve in the current time slot τ and the unloading ratio Δ _{m, u} (τ), and then determine its own flight angle β _u (τ ) and flight speed v _u (τ), can be written as:

a _u (τ) = {m(τ), Δ _{m, u} (τ), β _u (τ), v _u (τ)} Expression 19

(5d), define the state space, the state of the system can be regarded as the collection of all UAV observations:

s(τ)={o _u (τ)|u∈U} Expression 20

(5e), specifically define the reward, the feedback obtained by the agent after performing the action is called the reward, which is used to judge the quality of the action and guide the agent to update its strategy. Generally speaking, the reward function corresponds to the optimization goal. The goal of this optimization is to minimize the energy consumption of the UAV and the average calculation delay of the system, which is exactly negatively correlated with the maximum reward return. Therefore, the UAV performs the action The latter reward is defined as:

r _u (τ)＝D _m (τ)·(-T ^mean (τ)-ψE _u (τ)-P _u (τ)) Expression 21

Among them, D _m (τ) ∈ [0, 1] is the attenuation coefficient, which is defined as the benefit obtained after the UAV handles the unloading task of the mobile terminal. The specific calculation is as follows:

Among them, η and β are related constants, and its function image is sigmoid-like, and the input is the cumulative service times of the current user. The more times, the larger the value, the smaller the reward, and the lower the benefit. ψ is used to numerically align UAV energy consumption and user average delay. P _u (τ) is an additional penalty item. If the UAV flies out of the field after performing an action or the distance with other UAVs is less than the safe distance, the penalty needs to be increased.

(5f), according to the established S, A, O and R, establish a deep reinforcement learning model on the basis of MADDPG, using the actor-critic framework, each agent has its own actor network and critic network, and their own target network. The Actor network is responsible for formulating strategies for the agent π(o _u (τ)|θ _u ), θ _u represents its network parameters; the critic network output estimates the optimal state-action value function as Q(s(τ), a ₁ (τ), .., a _U (τ)|w _u ), w _u represents its network parameters. The input of the critic network contains the observations and actions of all agents in a time slot, but during distributed execution, the input of the actor network only needs its own observations.

The algorithm learns the Q function and the optimal strategy at the same time. When updating the critic network, it needs to extract H groups of records from the experience pool of each agent, and splice each group at the same time to obtain H new records, which are recorded as: { s _{t, i} , a _{1, i} ,..., a _{U, i} , r _{1, i} ,..., r _{U, i} , s _{t+1, i} |i=1, 2, .., H }, use the temporal difference to train the critic network of each agent intensively, and the loss function for training the Q-value function is defined as:

where y _{u, i} can be obtained from formula (24):

和

分别代表无人机UAV_u的critic目标网络和actor目标网络，目标网络都具有滞后更新的网络参数，使训练变得更稳定。in,

and

represent the critic target network and the actor target network of the UAV _u , respectively, and the target network has network parameters that are updated with a lag, making the training more stable.

The critic network needs to reduce the loss as much as possible to approach the real Q ^* value, and the actor network uses the determined strategy gradient of the Q value for gradient ascent to update the network parameters to maximize the action value:

更新目标网络：Finally at regular intervals with an update rate of

Update target network:

(6), considering continuous state space and continuous action space, using MADDPG-based multi-agent deep reinforcement learning algorithm for model training of computational migration, including the following specific steps:

(6a), start the environment simulation, initialize each agent actor network and critic network and their respective target network parameters;

(6b), initialize the number of training rounds;

(6c), update the user's location set, task set and service times, the location set and channel parameters of the drone;

(6d), run the actor network for each agent distribution, output the action a _u (τ) according to the observation o _u (τ), and get the immediate reward r _u (τ), and go to the next state s _{τ+ 1} , so as to obtain the training data {o _u (τ), a _u (τ), r _u (τ), o _u (τ+1)};

(6e), store the training data in their respective experience playback pools;

(6f), each agent randomly samples H training data from the experience playback pool to form a training data set;

(6g), each agent calculates the loss value L(w _u ) through the critic network and the target network to update w _u , and uses the determined strategy gradient for gradient ascent, and updates the parameter θ of the actor network through the backpropagation of the neural network _u ;

(6h), the number of training times reaches the target network update interval, and the target network parameters are updated;

(6i), judging whether convergence is satisfied, if so, the optimized deep reinforcement learning model is obtained after optimization, otherwise enter step (6c);

(7), in the execution phase, based on the state s(τ) of the current environment, the UAV uses the trained model to obtain the optimal user offloading scheme and flight trajectory;

(7a), using the deep reinforcement learning model trained by the MADDPG algorithm, input the state information at a certain moment;

得到最优的迁移策略和飞行路径。(7b), output the optimal action policy

Get the optimal migration strategy and flight path.

In Figure 1, a mobile edge computing system model is described in which UAV-assisted user offloading is described. The user offloads computing tasks to UAV-assisted computing to reduce computing delays and energy consumption.

In Figure 2, the deep reinforcement learning model of UAV-assisted MEC network is described. It can be seen that multiple UAVs act as agents to select the current optimal strategy based on the current state and obtain rewards from the environment.

In Figure 3, the training model of the actor-critic framework is described. Through centralized training and decentralized execution, the critic network can refer to the behavior of other agents during the training process, so as to better evaluate the performance of the actor network and improve the strategy. stability.

In Figure 4, the simulation results of computing performance and energy consumption of drones with different algorithms are described. Based on the MADDPG algorithm, the optimal power consumption control under different computing performances can be obtained. When the CPU frequency is 12.5GHz, the energy consumption is compared to The baseline is reduced by 29.16%, which is 8.67% lower than the stochastic policy gradient algorithm. .

The contents not described in detail in the application of the present invention belong to the prior art known to those skilled in the art.

Claims (1)

1. An unmanned aerial vehicle assisted calculation migration method based on multi-agent depth determination strategy gradient is characterized by comprising the following steps:

(1) The traditional MEC servers are deployed in a base station or other fixed facilities, a movable MEC server is adopted at this time, the unmanned aerial vehicle technology is combined with edge calculation, and user equipment unloads calculation tasks to an unmanned aerial vehicle end through wireless communication so as to reduce calculation delay;

(2) Constructing an unmanned aerial vehicle auxiliary user unloading system model, a mobile model, a communication model and a calculation model, and giving an optimization objective function;

(3) The method is characterized in that a Partially Observable Markov Decision Process (POMDP) is adopted for modeling, under the condition of considering the flight range and the safety distance of the unmanned aerial vehicle, the flight tracks and the calculation unloading strategies of the multiple unmanned aerial vehicles are jointly optimized on the basis of the position and the task information of a user, a deep reinforcement learning model is constructed by taking the aim of minimizing the system delay and the energy consumption of the unmanned aerial vehicle and simultaneously ensuring the service fairness of the user as the target, and the method comprises the following specific steps:

(3a) The problem of the multi-unmanned aerial vehicle auxiliary computation unloading is regarded as a partial observation Markov decision process and is composed of tuples { S, A, O, pr and R }; there are typically multiple agents interacting with the environment, each agent based on the current state s _τ Get self observation o _τ E.g. O and make action a _τ E.g. A, the environment generates an instant reward r for the action _τ E.g. R to evaluate the current action and with probability Pr (S) _τ+1 |S _τ ，A _τ ) Entering a next state, the new state depending only on the current state and the actions of the respective agent; actions of Agents are based on policy π (a) _τ |o _τ ) Enforcement, with the goal of learning to an optimal strategy to maximize long-term jackpot, can be expressed as:

wherein γ is a reward discount;

(3b) Specifically defining observation space, each unmanned aerial vehicle only has limited observation range, and the radius of the observation range is set as r _obs Therefore, only partial state information can be observed, and the global state information and the actions of other unmanned planes are unknown; single unmanned aerial vehicle UAV _u The information observable in the time slot τ has its own position information c _u (tau) and current location information, task information and number of services for K mobile users in observation range

The action space a is the transmit power and the selected channel, and is represented as:

o _u (τ)＝{c _u (τ)，k _u (τ)}

(3c) Specifically defining an action space, based on the observed information, the drone needs to determine which user is served at the current time slot τ and the offloading ratio Δ _m，u (tau) and determining the flight angle beta thereof _u (τ) and flight velocity v _u (τ), which can be written as:

a _u (τ)＝{m(τ)，Δ _m，u (τ)，β _u (τ)，v _u (τ)}

(3d) Defining a state space, the state of the system can be regarded as a set of all unmanned aerial vehicle observations:

s(τ)＝{o _u (τ)|u∈U}

(3e) Specifically defining rewards, wherein feedback obtained after the intelligent agent executes actions is called the rewards and is used for judging the quality of the actions and guiding the intelligent agent to update the strategy; generally speaking, the reward functions all correspond to optimization objectives, the objective of this optimization is to minimize the energy consumption of the drone and the average system computation delay, and the maximum reward return is just negatively correlated, so the reward after the drone executes actions is defined as:

r _u (τ)＝D _m (τ)·(-T ^mean (τ)-ψE _u (τ)-P _u (τ))

wherein D _m (τ)∈[0，1]For the attenuation coefficient, the benefit obtained after the unmanned aerial vehicle processes the mobile terminal unloading task is defined, and the specific calculation is as follows:

wherein eta and beta are correlation constants, the function image is of a sigmoid type, the number of times of accumulated service of the current user is input, the more times are, the larger the value is, the smaller the reward is, and the lower the benefit is; psi is used to average the drone energy consumption and usersCarrying out numerical value alignment; p _u (tau) is an additional punishment item, and if the unmanned aerial vehicle flies out of the field after executing actions or the distance between the unmanned aerial vehicle and the rest unmanned aerial vehicles is less than the safe distance, the punishment item needs to be added;

(3f) Establishing a deep reinforcement learning model on the basis of MADDPG according to the established S, A, O and R, adopting an operator-critic framework, wherein each agent has an own operator network, critic network and respective target network; the Actor network is responsible for formulating a policy pi (o) for an agent _u (τ)|θ _u )，θ _u Representing its network parameters; the critic network outputs an estimate of the optimum state-action cost function denoted as Q (s (τ), a) ₁ (τ)，...，a _U (τ)|w _u )，w _u Representing its network parameters; the input of the critic network comprises the observed values and actions of all agents in a time slot, but the input of the actor network only needs the observed values of the actor network when the distribution is executed;

the algorithm learns the Q function and the optimal strategy at the same time, when the criticc network is updated, H groups of records need to be extracted from the experience pool of each intelligent agent, and each group at the same time is spliced to obtain H new records, which are recorded as: { s _t，i ，a _1，i ，...，a _U，i ，r _1，i ，...，r _U，i ，s _t+1，i I =1, 2., H }, training the criticic network of each agent using a set of timing differences, the penalty function of the training Q-value function being defined as:

wherein y is _u，i Obtained from formula (24):

wherein,

and

respectively representing Unmanned Aerial Vehicles (UAVs) _u The critic target network and the actor target network, the target network has network parameters updated later, so that the training becomes more stable;

critic networks need to minimize losses to approximate true Q ^* The operator network updates the network parameters by using the gradient of the determination strategy of the Q value to perform gradient rise so as to maximize the action value:

finally at a fixed interval at an update rate

Updating the target network:

(4) Considering a continuous state space and a continuous action space, and performing model training of computational migration by using a multi-agent deep reinforcement learning algorithm based on MADDPG;

(5) In the execution stage, the unmanned aerial vehicle obtains an optimal user unloading scheme and a flight track by using a trained model based on the state s (tau) of the current environment.

Images