CN111786713A - A UAV network hovering position optimization method based on multi-agent deep reinforcement learning - Google Patents

Abstract

An unmanned aerial vehicle network hovering position optimization method based on multi-agent deep reinforcement learning comprises the steps of firstly, modeling a channel model, a coverage model and an energy loss model in an unmanned aerial vehicle ground communication scene; modeling a throughput maximization problem of an unmanned aerial vehicle to ground communication network into a partially observable Markov decision process; obtaining local observation information and instantaneous rewards through continuous interaction of an unmanned aerial vehicle and the environment, and carrying out centralized training based on the information to obtain a distributed strategy network; and deploying the strategy network to each unmanned aerial vehicle, wherein each unmanned aerial vehicle can obtain a moving direction and a moving distance decision based on local observation information of the unmanned aerial vehicle, adjust the hovering position and perform distributed cooperation. The invention also introduces proportional fair scheduling and energy consumption loss information of the unmanned aerial vehicle into the instantaneous reward function, thereby improving the throughput, ensuring the fairness of the unmanned aerial vehicle to the ground user service, reducing the energy consumption loss and enabling the unmanned aerial vehicle cluster to adapt to the dynamic environment.

Description

A UAV network hovering position optimization based on multi-agent deep reinforcement learning method

technical field

The invention relates to the technical field of wireless communication, in particular to a method for optimizing the hovering position of a multi-unmanned aerial vehicle network based on multi-agent deep reinforcement learning.

Background technique

In recent years, due to the high mobility, easy deployment and low cost of UAVs, UAV-based communication technology has attracted extensive attention and has become a new research hotspot in the field of wireless communication. UAV-assisted communication technology mainly has the following application scenarios: UAV as a mobile base station to provide communication coverage for areas with sparse infrastructure or post-disaster areas, UAV as a relay node for two distant places that cannot directly establish a connection Communication nodes provide wireless connectivity, UAV-based data distribution and collection. The present invention is mainly aimed at the first scenario, in which the hovering position of the UAV determines the coverage performance and throughput of the entire UAV network. The ground equipment served by the drone network may be mobile, so the drone needs to constantly adjust its hovering position to achieve optimal performance.

In 2018, Qingqing Wu et al. proposed a UAV path planning scheme for a multi-UAV-to-ground communication system in the paper "Joint Trajectory and Communication Design for Multi-UAV Enabled Wireless Networks", dividing the time into multiple cycles, and the movement trajectories of UAVs in each cycle are the same. , in each time slot, the UAV base station serves a specific ground user. This scheme models the optimization problem as a mixed integer programming problem, and solves it using block coordinate gradient descent and approximate convex optimization techniques to obtain the optimal hovering position for each time slice in the cycle, and maximize the downlink between ground users. link throughput. However, the scheme proposed in this paper is only suitable for static environments, and is carried out under the assumption that ground equipment does not have mobility, and is not suitable for scenarios where ground users are constantly moving. Chi Harold Liu et al. proposed a UAV path planning algorithm based on deep reinforcement learning in the paper "Energy-Efficient UAV Control for Effective and Fair Communication Coverage: A Deep Reinforcement Learning Approach", and trained a decision model through deep reinforcement learning. The model outputs the UAVs' next decision (moving direction, moving distance) according to the current state. The method proposed in this paper can achieve fair wireless coverage over a large area and minimize the energy consumption of UAVs. However, this method only considers the coverage performance of the UAVs network, and is fair for the coarse-grained coverage of the region rather than the fine-grained coverage of the user. Furthermore, the method is a centralized scheme that requires a controller to collect information from all UAVs at each time slot in order to make decisions.

To sum up, the UAVs path planning technology in the ground-to-ground communication network based on the UAV base station mainly has the following defects: (1) The dynamics of the environment, that is, the mobility of ground users, is not considered. (2) A centralized algorithm is used, which relies on global information and centralized control. In some large-scale scenarios, it is difficult to perform centralized control. Therefore, a distributed control strategy is required. The base station only relies on the information it obtains to make decisions. (3) The service fairness considering the user level is ignored. These shortcomings make the existing UAVs trajectory optimization methods in UAV networks unsuitable for practical communication environments.

SUMMARY OF THE INVENTION

The purpose of the present invention is to propose a method for optimizing the hovering position of multiple UAVs based on multi-agent reinforcement learning, so as to solve the above technical problems.

Technical scheme of the present invention:

A UAV network hovering position optimization method based on multi-agent deep reinforcement learning, the steps are as follows:

(1) Establish a multi-UAV ground-to-ground communication network model, which mainly includes the following 4 steps:

(1.1) Establish a scene model: establish a square target area with a side length of l, in which there are N ground users and M unmanned aerial vehicle base stations (UAV-BSs), which provide communication services for ground users . Time is divided into T identical time slots, from the previous time slot to the current time slot, the ground user may be stationary or may move, so the UAV base station needs to find a new optimal hovering position in each time slot, and After reaching the target location, the ground user is selected for data transmission service.

(1.2) Establishing an air-to-ground communication model: The present invention uses an air-to-ground channel model to model the channel between the UAV base station and the ground user. The UAV base station is easier to fly than the ground base station due to its high flying height. A line-of-sight link (LoS) is established with the ground user. In the LoS case, the path loss model between the UAV base station m and the ground user n is:

表示无人机基站m和地面用户n之间的距离，其中r_n,m表示二者的水平距离，h为无人机基站固定飞行高度。根据路径损失，信道增益可以表示为

根据信道增益，无人机基站m和地面用户n之间在时隙t的数据传输速率为：where η is the extra path loss coefficient, c is the speed of light, f _c is the subcarrier frequency, α is the path loss index,

Represents the distance between the UAV base station m and the ground user n, where r _n,m represents the horizontal distance between the two, and h is the fixed flying height of the UAV base station. According to the path loss, the channel gain can be expressed as

According to the channel gain, the data transmission rate between the UAV base station m and the ground user n in the time slot t is:

where σ is the additive white Gaussian noise, p _t is the transmit power of the UAV base station, and g _n,m (t) is the channel gain between the UAV base station m and the ground user n at time t.

(1.3) Establish coverage model: Due to hardware limitations, the coverage of each UAV base station is limited. The present invention defines the maximum tolerable path loss L _max . If the path loss between the UAV base station and the user is less than L _max at a certain moment, we consider the established connection to be reliable, otherwise, we consider that the established connection fails. Therefore, the effective coverage of each UAV base station can be defined according to the maximum tolerable path loss. The range takes the projection point of the UAV base station on the ground as the center and R _cov as the radius. According to the path loss formula, R _cov It can be expressed as:

(1.4) Establishing an energy loss model: the present invention mainly focuses on the energy loss caused by the movement of the UAV, considering the flight speed V and the flight power p _f of the UAV, the flight energy consumption of the UAV base station m in the time slot t depends on Distance flown:

分别表示无人机在水平面上x轴和y轴的位置坐标。in

Represent the position coordinates of the UAV on the x-axis and y-axis on the horizontal plane, respectively.

(2) Model the problem as a locally observable Markov decision process:

其中γ∈(0,1)为折扣系数，r_m(t)表示智能体m在t时刻的奖励；Each UAV base station is equivalent to an agent; in each time slot where the environmental state is S(t), the agent m can only obtain the local observation o _m within its own coverage, and according to the decision function u _m (o _m ), choose action a _m from action set A to maximize the discounted total expected reward

where γ∈(0,1) is the discount coefficient, and r _m (t) represents the reward of agent m at time t;

和地面用户当前状态

无人机基站状态

包括无人机当前的位置信息；地面用户状态

包括当前地面用户的位置信息。The system state set S={S(t)|S(t)=(S ^u (t), S ^g (t))}, respectively including the current state of the UAV base station

and the current state of the ground user

UAV base station status

Including the current position information of the drone; ground user status

Including the location information of the current ground user.

UAV action set A={a(t)|a(t)=(θ(t),d(t))}, in time slot t, UAV m needs to make the current local observation information after obtaining the current local observation information. Decision a _m (t), move to the next hover position, so the action set includes the flight rotation angle θ(t) and the moving distance d(t).

The system rewards r(t) in time: The goal of this paper is to maximize the throughput of the UAV network while considering user service fairness and energy consumption. Therefore, the additional throughput generated by adjusting the hovering position of the drone at each time t is a positive reward, expressed as:

ΔC(t)=C(S ^u (t+1),S ^g (t))-C(S ^u (t),S ^g (t))

where C(S ^u (t), S ^g (t)) represents the throughput generated by the network when the UAV base station state is S ^u (t) and the ground user state is S ^g (t). C(S ^u (t+1), S ^g (t)) represents the throughput generated by the network when the UAV base station state is S ^u (t+1) and the ground user state is S ^g (t). Considering the fairness of user services, if there are a large number of users in a certain area, but only one user in a certain area, the drone base station will always hover in the high-density area and ignore the low-density area in order to maximize throughput. Therefore, the present invention applies a weight w _n (t) to each user's throughput reward to achieve proportional fair scheduling. R _req represents the minimum communication rate required by the ground user, and R _n (t) represents the average communication rate of the ground user n from the initial stage to time t. When the UAV base station serves the user, R _n (t) increases, and the weight of the user will gradually decrease; if the user is not served, R _n (t) decreases, and the weight of the user increases continuously. Therefore, the reward weight in areas with sparse users will continue to increase, attracting UAV base stations to serve.

Among them, a _n,m (t) is an indicator variable. At time t, if the UAV base station m serves the ground user user n, then an _n,m (t)=1. Therefore, the fairness throughput reward is comprehensively considered. and energy loss penalty, the present invention gives the system real-time reward r(t):

Among them, α represents the weight of the energy consumption penalty. The larger the α is, the more energy consumption loss the system pays attention to in decision-making, otherwise, the more energy consumption loss is ignored.

The local observation set O(t)={o ₁ (t),...,o _M (t)}, when multiple UAV base stations work together in a large area, each UAV cannot observe the global information, Only terrestrial user information within its own coverage area can be observed. o _m (t) represents the location information of the ground users within the coverage area observed by the UAV base station m at time t.

(3) Training based on multi-agent deep reinforcement learning algorithm:

表示每个策略网络的参数，Q＝{Q₁,…,Q_M}表示M个智能体的评价网络，

表示评价网络的参数，步骤(3)包括：The invention introduces the multi-agent deep reinforcement learning algorithm MADDPG into the hovering position optimization of the UAV-to-ground communication network, adopts the architecture of centralized training and distributed execution, and uses global information during training to better guide each The gradient update of the UAV's decision function, each UAV only uses the local information observed by itself to make the next decision during execution, which is more in line with the needs of the actual scene; each agent adopts the Actor-Critic architecture The DDPG network is used for training, and the strategy network is used to fit the strategy function u(o), input the local observation o, and output the action strategy a; the evaluation network is used to fit the state-action function Q(s, a), which represents the state of the system. When it is s, the expected reward obtained by taking action a; let u={u ₁ ,...,u _M } denote the deterministic policy function of M agents,

represents the parameters of each policy network, Q={Q ₁ ,...,Q _M } represents the evaluation network of M agents,

represents the parameters of the evaluation network, and step (3) includes:

(3.1) Initialize the experience playback space, set the size of the experience playback space, initialize the parameters of each DDPG network, the number of training rounds, etc.

(3.2) Starting from the training round epoch=1, starting from time t=1.

(3.3) Obtain the local observation information o of the current UAV and the current state s of the entire system; each UAV m uses the local observation information obtained in the t time slot, and adjusts the suspension based on the ∈ greedy strategy and the DDPG network output decision information a _m stop position, and according to the path loss with the ground users, based on the greedy scheme, select W ground users with the lowest path loss for communication services, obtain the instantaneous reward r, reach the next system state s' and obtain the local observation information o'; Store (s,o,a,r,s′,o′) as a sample in the experience playback space, a={a ₁ ,...,a _M } represents the joint actions of all UAVs, o={o ₁ , ..., o _m } represents the local observation information of all UAVs, t=t+1.

(3.4) If the number of samples stored in the playback space is greater than B, go to step 3.5; otherwise, continue to collect samples and return to step 3.3.

其中Q′_m表示第m个智能体的评价网络的目标网络，u′_m表示第m个智能体的策略网络的目标网络，r^k表示第k个样本中的及时奖励，a′_m表示无人机m在系统状态s^′k下根据局部观察

所作出的决策。基于全局信息，使用梯度下降法最小化损失函数

更新该智能体的评价网络的参数：(3.5) For each agent m, randomly sample a fixed number of K samples from the experience replay space, and calculate the target value, where the kth sample (s ^k , ^ok , ^ak ,r ^k ,s ^′k ,o The target value y ^k of ^k ) can be expressed as:

where Q' _m represents the target network of the evaluation network of the _mth agent, u'm represents the target network of the policy network of the mth agent, r ^k represents the timely reward in the kth sample, and a' _m represents no The human-machine m is in the system state ^s'k according to the local observation

decisions made. Minimize the loss function using gradient descent based on global information

Update the parameters of the agent's evaluation network:

According to the evaluation network and sample information, based on the policy gradient of the sample, update the parameters of the agent's policy network:

(3.6) After a certain round interval, that is, update the target network parameters θ ^Q′ and θ ^u′ : θ ^Q′ =τθ ^Q +(1-τ)θ ^Q′ ,θ ^u′ =τθ ^u +(1-τ) θ ^u′ . When the total duration T is reached or the drone's energy is exhausted, exit the current training round, otherwise, go back to step 3.3. If the number of training rounds is reached, exit the training process, otherwise enter a new training round.

(4) Allocate the trained policy network u to each UAV, deploy the UAV to the target area, and each UAV adjusts the hovering position according to its own local observation in each time slot, and adjusts the hovering position to the ground. Users perform communication services.

Beneficial effects of the present invention: The present invention proposes a method for optimizing the hovering position of a UAV network based on multi-agent deep reinforcement learning, which models the throughput maximization problem in the UAV-to-ground communication network scenario as a locally available method. Observe the Markov decision-making process, and introduce the multi-agent deep reinforcement learning method MADDPG for centralized training and distributed execution to solve the optimization problem of UAV hovering position in dynamic environment. The method enables the UAV swarm to better adapt to the dynamic environment, and multiple UAVs can cooperate in a distributed manner without relying on a centralized controller. The present invention introduces proportional fair weight in the construction of the instant reward function. and energy consumption loss information, which ensures the fairness of user services and the low energy consumption of UAV swarms to a certain extent while improving throughput.

Description of drawings

FIG. 1 is a schematic diagram of a UAV-to-ground communication network scenario according to the present invention.

FIG. 2 is a flowchart of a method for optimizing the hovering position of a UAV network based on multi-agent deep reinforcement learning of the present invention.

FIG. 3 is a flow chart of the present invention for training a distributed strategy network for UAVs based on multi-agent deep reinforcement learning.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

A multi-agent deep reinforcement learning-based hovering position optimization method for UAV networks, applied to emergency communication recovery in areas lacking ground infrastructure or post-disaster. As shown in Figure 1, the area lacks basic communication facilities, and the UAV is used as a mobile base station for communication coverage. The ground environment is dynamic, and the ground equipment may move. The UAV base station needs to constantly adjust its hovering position , for better communication service (maximizing the throughput of the system). At the same time, the fairness of the service and the loss of energy consumption should also be considered. Some ground users cannot be ignored because of the pursuit of maximum throughput, and the energy loss caused by the movement of the UAV base station should be minimized. The process of the present invention is shown in Figure 2. First, the communication model, coverage model and energy consumption model in specific application scenarios are modeled and an optimization target is constructed; secondly, according to the optimization target and the characteristics of the multi-UAV system, The optimization problem is modeled as a locally observable Markov decision process; then, a simulation platform is used to simulate a multi-UAV-to-ground communication scenario, and samples are collected through the interaction between the UAV swarm and the environment, and the multi-agent deep reinforcement learning algorithm MADDPG is used. Conduct centralized training to get a distributed policy for each drone. Finally, the trained policy network is deployed into the UAV, and the UAV cluster is deployed to the target area, and the UAVs cooperate with each other to achieve high throughput, low energy consumption, and fair communication coverage.

Specific steps are as follows:

(1) Establish a multi-UAV ground-to-ground communication network model, which mainly includes the following 4 steps:

(1.1) Establish a scene model: establish a square target area with a side length of l, in which there are N ground users and M unmanned aerial vehicle base stations (UAV-BSs), which provide communication services for ground users . Time is divided into T identical time slots, from the previous time slot to the current time slot, the ground user may be stationary or may move, so the UAV base station needs to find a new optimal hovering position in each time slot, and After reaching the target location, the ground user is selected for data transmission service.

(1.2) Establishing an air-to-ground communication model: The present invention uses an air-to-ground channel model to model the channel between the UAV base station and the ground user. The UAV base station is easier to fly than the ground base station due to its high flying height. A line-of-sight link (LoS) is established with the ground user. In the LoS case, the path loss model between the UAV base station m and the ground user n is:

表示无人机基站m和地面用户n之间的距离，r_n,m为水平距离，h为无人机基站固定飞行高度。根据路径损失，信道增益可以表示为

根据信道增益，无人机基站m和地面用户n之间在时隙t的数据传输速率为：where η is the extra path loss coefficient, c is the speed of light, f _c is the subcarrier frequency, α is the path loss index,

Represents the distance between the UAV base station m and the ground user n, rn _,m is the horizontal distance, and h is the fixed flying height of the UAV base station. According to the path loss, the channel gain can be expressed as

According to the channel gain, the data transmission rate between the UAV base station m and the ground user n in the time slot t is:

where σ is the additive white Gaussian noise, p _t is the transmit power of the UAV base station, and g _n,m (t) is the channel gain between the UAV base station m and the ground user n at time t.

(1.3) Establish coverage model: Due to hardware limitations, the coverage of each UAV base station is limited. The present invention defines the maximum tolerable path loss L _max . If the path loss between the UAV base station and the user is less than L _max at a certain moment, we consider the established connection to be reliable, otherwise, we consider that the established connection fails. Therefore, the effective coverage of each UAV base station can be defined according to the maximum tolerable path loss. The range takes the projection point of the UAV base station on the ground as the center and R _cov as the radius. According to the path loss formula, R _cov It can be expressed as:

(1.4) Establishing an energy loss model: the present invention mainly focuses on the energy loss caused by the movement of the UAV, considering the flight speed V and the flight power p _f of the UAV, the flight energy consumption of the UAV base station m in the time slot t depends on Distance flown:

分别表示无人机在水平面上x轴和y轴的位置坐标。in

Represent the position coordinates of the UAV on the x-axis and y-axis on the horizontal plane, respectively.

(2) Model the problem as a locally observable Markov decision process:

其中γ∈(0,1)为折扣系数，r_m(t)表示智能体m在t时刻的奖励；Each UAV base station is equivalent to an agent; in each time slot where the environmental state is S(t), the agent m can only obtain the local observation o _m within its own coverage, and according to the decision function u _m (o _m ), choose action a _m from action set A to maximize the discounted total expected reward

where γ∈(0,1) is the discount coefficient, and r _m (t) represents the reward of agent m at time t;

和地面用户当前状态

无人机基站状态

包括无人机当前的位置信息；地面用户状态

包括当前地面用户的位置信息。The system state set S={S(t)|S(t)=(S ^u (t), S ^g (t))}, respectively including the current state of the UAV base station

and the current state of the ground user

UAV base station status

Including the current position information of the drone; ground user status

Including the location information of the current ground user.

UAV action set A={a(t)|a(t)=(θ(t),d(t))}, in time slot t, UAV m needs to make the current local observation information after obtaining the current local observation information. Decision a _m (t), move to the next hover position, so the action set includes the flight rotation angle θ(t) and the moving distance d(t).

The system rewards r(t) in time: The goal of this paper is to maximize the throughput of the UAV network while considering user service fairness and energy consumption. Therefore, the additional throughput generated by adjusting the hovering position of the drone at each time t is a positive reward, expressed as:

ΔC(t)=C(S ^u (t+1),S ^g (t))-C(S ^u (t),S ^g (t))

where C(S ^u (t), S ^g (t)) represents the throughput generated by the network when the UAV base station state is S ^u (t) and the ground user state is S ^g (t). C(S ^u (t+1), S ^g (t)) represents the throughput generated by the network when the UAV base station state is S ^u (t+1) and the ground user state is S ^g (t). Considering the fairness of user services, if there are a large number of users in a certain area, but only one user in a certain area, the drone base station will always hover in the high-density area and ignore the low-density area in order to maximize throughput. Therefore, the present invention applies a weight w _n (t) to each user's throughput reward to achieve proportional fair scheduling. R _req represents the minimum communication rate required by the ground user, and R _n (t) represents the average communication rate of the ground user n from the initial stage to time t. When the UAV base station serves the user, R _n (t) increases, and the weight of the user will gradually decrease; if the user is not served, R _n (t) decreases, and the weight of the user increases continuously. Therefore, the reward weight in areas with sparse users will continue to increase, attracting UAV base stations to serve.

Therefore, considering the fairness throughput reward and energy loss penalty comprehensively, the present invention gives the system real-time reward r(t)

Among them, α represents the weight of the energy consumption penalty. The larger the α is, the more energy consumption loss the system pays attention to in decision-making, otherwise, the more energy consumption loss is ignored.

The local observation set O(t)={o ₁ (t),...,o _M (t)}, when multiple UAV base stations work together in a large area, each UAV cannot observe the global information, Only terrestrial user information within its own coverage area can be observed. o _m (t) represents the location information of ground users within its coverage area observed by the UAV base station m.

(3) Training based on multi-agent deep reinforcement learning algorithm:

表示每个策略网络的参数，Q＝{Q₁,…,Q_M}表示M个智能体的评价网络，

表示评价网络的参数，如图3所示，步骤(3)包括：The invention introduces the multi-agent deep reinforcement learning algorithm MADDPG into the hovering position optimization of the UAV-to-ground communication network, adopts the architecture of centralized training and distributed execution, and uses global information during training to better guide each The gradient update of the UAV's decision function, each UAV only uses the local information observed by itself to make the next decision during execution, which is more in line with the needs of the actual scene; each agent adopts the Actor-Critic architecture The DDPG network is used for training, and the strategy network is used to fit the strategy function u(o), input the local observation o, and output the action strategy a; the evaluation network is used to fit the state-action function Q(s, a), which represents the state of the system. When it is s, the expected reward obtained by taking action a; let u={u ₁ ,...,u _M } denote the deterministic policy function of M agents,

represents the parameters of each policy network, Q={Q ₁ ,...,Q _M } represents the evaluation network of M agents,

Represents the parameters of the evaluation network, as shown in Figure 3, step (3) includes:

(3.1) Initialize the experience playback space, and set the size B of the experience playback space, initialize the parameters θ of each DDPG network, the number of training rounds P, the duration T, etc.

(3.2) Starting from the training round epoch=1, starting from time t=1.

(3.3) Obtain the local observation information o of the current UAV and the current state s of the entire system; each UAV m uses the local observation information obtained in the t time slot, and adjusts the suspension based on the ∈ greedy strategy and the DDPG network output decision information a _m stop position, and according to the path loss with the ground users, based on the greedy scheme, select W ground users with the lowest path loss for communication services, obtain the instantaneous reward r, reach the next system state s' and obtain the local observation information o'; Store (s,o,a,r,s′,o′) as a sample in the experience playback space, a={a ₁ ,...,a _M } represents the joint actions of all UAVs, o={o ₁ , ..., o _m } represents the local observation information of all UAVs, t=t+1;

(3.4) If the number of samples stored in the playback space is greater than B, go to step 3.5; otherwise, continue to collect samples and return to step 3.3.

其中Q′_m表示第m个智能体的评价网络的目标网络，u′_m表示第m个智能体的策略网络的目标网络，r^k表示第k个样本中的及时奖励，a′_m表示无人机m在系统状态s^′k下根据局部观察

所作出的决策。基于全局信息，使用梯度下降法最小化损失函数

更新该智能体的评价网络的参数：(3.5) For each agent m, randomly sample a fixed number of K samples from the experience replay space, and calculate the target value, where the kth sample (s ^k , ^ok , ^ak ,r ^k ,s ^′k ,o The target value y ^k of ^k ) can be expressed as:

where Q' _m represents the target network of the evaluation network of the _mth agent, u'm represents the target network of the policy network of the mth agent, r ^k represents the timely reward in the kth sample, and a' _m represents no The human-machine m is in the system state ^s'k according to the local observation

decisions made. Minimize the loss function using gradient descent based on global information

Update the parameters of the agent's evaluation network:

According to the evaluation network and sample information, based on the policy gradient of the sample, update the parameters of the agent's policy network:

(3.6) After a certain round interval, update the evaluation target network parameters θ ^Q′ and policy target network parameters θ ^u′ : θ ^Q′ = τθ ^Q +(1-τ)θ ^Q′ ,θ ^u′ =τθ ^u +(1 -τ)θ ^u′ . When the total duration T is reached or the drone's energy is exhausted, exit the current training round, otherwise, go back to step 3.3. If the number of training rounds is reached, exit the training process, otherwise enter a new training round.

(4) Allocate the trained policy network u to each UAV, deploy the UAV to the target area, and each UAV adjusts the hovering position according to its own local observation in each time slot, and adjusts the hovering position to the ground. Users perform communication services.

In summary:

The present invention proposes a method for optimizing the hovering position of the UAV network based on multi-agent deep reinforcement learning. By modeling the throughput maximization problem in the multi-UAV-to-ground communication scenario as a locally observable Markov decision process, and use the MADDPG algorithm to solve it, so that the UAV swarm can adapt to the dynamic environment, carry out distributed cooperation, and achieve high throughput, low energy consumption and service fairness of the network.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments. The above-mentioned embodiments and descriptions only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Various changes and modifications fall within the scope of the claimed invention. The claimed scope of the present invention is defined by the appended claims and their equivalents.

Claims (1)

1. An unmanned aerial vehicle network hovering position optimization method based on multi-agent deep reinforcement learning is characterized by comprising the following steps:

(1) establishing multi-unmanned aerial vehicle to ground communication network model

(1.1) establishing a scene model: establishing a square target area with the side length of l, wherein N ground users and M unmanned aerial vehicle base stations are arranged in the area, and the unmanned aerial vehicle base stations provide communication service for the ground users; the time is divided into T identical time slots, from the last time slot to the current time slot, the ground user may be static or may move, so the unmanned aerial vehicle base station needs to search a new optimal hovering position in each time slot and select the ground user to perform data transmission service after reaching the target position;

(1.2) establishing an air-to-ground communication model: use the air-to-ground channel model to model the channel between unmanned aerial vehicle basic station and the ground user, unmanned aerial vehicle basic station is because high flight altitude, compares in ground basic station and establishes the stadia link LoS with the ground user more easily, and under the LoS condition, the path loss model between unmanned aerial vehicle basic station m and the ground user n is:

where η denotes the excess path loss coefficient, c denotes the speed of light, f_cRepresenting the subcarrier frequency, α the path loss exponent,

representing the distance, r, between the drone base station m and the ground user n_n，mThe horizontal distance is h, and the fixed flying height of the unmanned aerial vehicle base station is h; the channel gain is expressed as

According to the channel gain, the data transmission rate between the unmanned aerial vehicle base station m and the ground user n in the time slot t is R_n，m(t)：

Where σ represents additive white Gaussian noise, p_tRepresenting the transmitted power of the drone base station, g_n，m(t) represents the channel gain between the unmanned aerial vehicle base station m and the ground user n at time t;

(1.3) establishing a coverage model: defining a maximum tolerable path loss L_maxIf the path loss between the unmanned aerial vehicle base station and the user is less than L at a certain moment_maxIf the connection is not established, the connection is established; defining the effective coverage range of each unmanned aerial vehicle base station according to the maximum tolerable path loss, wherein the range takes the projection point of the unmanned aerial vehicle base station on the ground as the circle center and takes R as the circle center_covIs radius, according to the path loss formula, R_covExpressed as:

(1.4) establishing an energy loss model: paying attention to energy loss caused by movement of the unmanned aerial vehicle, and considering the flying speed V and the flying power p of the unmanned aerial vehicle_fAnd the flight energy consumption delta e of the unmanned aerial vehicle base station m in the time slot t_m(t) distance of flight dependent:

wherein,

respectively representing the position coordinates of the unmanned aerial vehicle on the x axis and the y axis on the horizontal plane at the time t;

(2) modeling the problem as a locally observable markov decision process:

each unmanned aerial vehicle base station is equivalent to an agent; in each time slot with the environment state S (t), the agent m can only obtain local observation o within the coverage range of the agent m_mAnd according to a decision function u_m(o_m) Selecting an action a from the action set_mTo maximize the total desired reward for the discount

Where γ ∈ (0,1) is the discount coefficient, r_m(t) represents the reward of agent m at time t;

the system state set S ═ { S (t) | S (t) ═ S (S)^u(t)，S^g(t)) }, respectively containing the current state of the drone base station

And the current state of the ground user

Status of each drone base station

The current position information of the unmanned aerial vehicle is included; state of each terrestrial user

Including location information of a current ground user;

in the time slot t, the unmanned aerial vehicle m needs to make a decision a after obtaining the current local observation information_m(t), move to the next hover position, so the action set includes flight rotation angle θ (t) and movement distance d (t);

system real-time rewards r (t): the throughput of the unmanned aerial vehicle network is maximized while the user service fairness and the energy consumption are considered; thus, the extra throughput generated by adjusting the hover position of the drone at each time t is a positive reward, expressed as:

ΔC(t)＝C(S^u(t+1)，S^g(t))-C(S^u(t)，S^g(t))

wherein, C (S)^u(t)，S^g(t)) indicates that the unmanned aerial vehicle base station state is S^u(t) the ground user status is S^g(t) throughput generated by the network; c (S)^u(t+1)，S^g(t)) then indicates that the unmanned aerial vehicle base station state is S^u(t +1), the ground user state is S^g(t) throughput generated by the network; considering fairness of user service, if a certain area is gathered with a large number of users and a certain area has only a small number of users, the unmanned aerial vehicle base station always hovers in a high-density area in pursuit of maximizing throughput, and ignores the low-density area, so that a weight w is applied to the throughput reward of each user_n(t) implementing proportional fair scheduling; r_reqExpressed is the minimum communication rate requirement, R, of the terrestrial user demand_n(t) represents the average communication rate of the terrestrial user n from the beginning to the time t; when the drone base station serves this user, R_n(t) increase, the user's weight gradually becomes smaller; if the user is not served, R_n(t) increase, the user weight increasing; therefore, the reward weight of the user sparse area is continuously increased, and the unmanned aerial vehicle base station is attracted to carry out service;

wherein, a_n，m(t) is an indicator variable, at time t, if drone base station m serves a ground user n, then a_n，m(t) is 1, whereas, a_n，m(t) ═ 0; therefore, considering the fairness throughput reward and the energy loss penalty comprehensively, the system rewards r (t) in real time:

wherein alpha represents the weight occupied by the energy consumption punishment, the larger alpha is, the more the system pays attention to the energy consumption loss in the decision making process, otherwise, the more the energy consumption loss is ignored;

local observation set o (t) ═ o₁(t)，...，o_M(t), when a plurality of unmanned aerial vehicle base stations cooperatively work in a large-range area, each unmanned aerial vehicle cannot observe global information and can only observe ground user information in the coverage area of the unmanned aerial vehicle; o_m(t) the position information of the ground user in the coverage range of the unmanned aerial vehicle base station m observed at the moment t is represented;

(3) training based on a multi-agent deep reinforcement learning algorithm:

the multi-agent deep reinforcement learning algorithm MADDPG is introduced into the hovering position optimization of the unmanned aerial vehicle to the ground communication network, a centralized training and distributed execution architecture is adopted, global information is used during training, gradient updating of a decision function of each unmanned aerial vehicle is better guided, each unmanned aerial vehicle only uses local information observed by the unmanned aerial vehicle to make a next decision during execution, and the unmanned aerial vehicle is more suitable for an actual fieldThe need for a scene; each agent adopts an Actor-Critic structured DDPG network for training, the strategy network is used for fitting a strategy function u (o), inputting a local observation o and outputting an action strategy a; the evaluation network is used to fit a state-action function Q (s, a) representing the desired reward for taking action a when the system state is s; let u be { u ═ u₁，...，u_MDenotes the deterministic policy functions of M agents,

parameter representing each policy network, Q ═ Q₁，...，Q_MDenotes the evaluation network of M agents,

a parameter indicative of an evaluation network;

(3.1) initializing an experience playback space, setting the size of the experience playback space, initializing parameters of each DDPG network and training rounds;

(3.2) starting from the training round epoch-1 and starting from the time t-1;

(3.3) acquiring local observation information o of the current unmanned aerial vehicle and the current state s of the whole system, wherein each unmanned aerial vehicle m uses the local observation information obtained by the time slot t, and outputs decision information a based on an ∈ greedy strategy and a DDPG network_mAdjusting the hovering position, selecting W ground users with the lowest path loss to perform communication service based on a greedy scheme according to the path loss between the hovering position and the ground users, obtaining an instant return reward r, achieving the next system state s 'and obtaining local observation information o'; storing (s, o, a, r, s ', o') as a sample in an empirical playback space, a ═ a { (a)₁，...，a_MDenotes the joint action of all drones, o ═ o₁，...，o_mThe local observation information of all unmanned aerial vehicles is represented, and t is t + 1;

(3.4) if the number of the samples stored in the playback space is greater than B, reaching the step (3.5); otherwise, continuing to collect the sample, and returning to the step (3.3);

(3.5) for each agent m, randomly sampling data from the empirical playback spaceA constant number K of samples, calculating a target value, wherein the kth sample(s)^k，o^k，a^k，r^k，s′^k，o^k) Target value y of^kCan be expressed as:

wherein Q'_mA target network u 'representing an evaluation network of the m-th agent'_mTarget network of the policy network representing the mth agent, r^kRepresents the timely reward, a ', in the kth sample'_mRepresenting unmanned plane m in system state s'^kAccording to local observation

The decision made; minimizing loss functions using gradient descent based on global information

And updating parameters of the evaluation network of the agent:

updating parameters of the intelligent agent strategy network based on the strategy gradient of the sample according to the evaluation network and the sample information:

(3.6) updating the evaluation target network parameter theta after a certain turn interval^Q′And a policy target network parameter θ^u′：θ^Q′＝τθ^Q+(1-τ)θ^Q′，θ^u′＝τθ^u+(1-τ)θ^u′Tau ∈ (0,1) represents updating weight, when reaching total duration T or the energy of the unmanned aerial vehicle is exhausted, quitting the current training round, otherwise, returning to step (3.3), if the number of the training rounds is up, quitting the training process, otherwise, entering a new training round;

(4) and distributing the trained strategy network u to each unmanned aerial vehicle, deploying the unmanned aerial vehicles to a target area, adjusting the hovering position of each unmanned aerial vehicle in each time slot according to local observation of the unmanned aerial vehicle, and performing communication service on ground users.

Images