CN116896777A - Unmanned aerial vehicle group general sense one-body energy optimization method based on reinforcement learning - Google Patents

Abstract

The invention provides an unmanned aerial vehicle group general sense integrated energy consumption optimization method based on reinforcement learning, which comprises the following steps: firstly, utilizing the positioning perception performance of the unmanned aerial vehicle cluster to endow an initial value to a Q value function network reflecting the relation between the state and the action; judging the current state; selecting a certain working point as an initial state; thirdly, selecting a current action based on an epsilon-greedy strategy according to the current state; introducing the perception performance of the unmanned aerial vehicle and the energy consumption of the communication task into the design of a Reward rewarding function, and acquiring an actual environment rewarding value of the previous action and the next state; fifthly, updating the Q value function network by using the actual environment rewarding value of the previous step; and sixthly, setting the new state as the current state, and repeating the steps of three to six until the value in the Q value function network reaches convergence. The invention can solve the problem of larger energy consumption in the ISAC network based on the unmanned aerial vehicle group in the prior art, simultaneously ensure the real-time service performance of communication and positioning of the terminal ground user, and effectively improve the service life of the network.

Description

Synaesthesia-integrated energy consumption optimization method for UAV swarm based on reinforcement learning

Technical field

The invention belongs to the technical field of integrated communication and perception of unmanned aerial vehicles, and specifically relates to a synaesthesia integrated energy consumption optimization method for a group of unmanned aerial vehicles based on reinforcement learning.

Background technique

Next-generation wireless networks (B5G/6G) promote the continuous innovation and development of wireless technology and provide key support for many emerging applications, such as connected intelligence, connected cars, and smart cities. These applications require high-quality wireless communication connections and high-speed Perception of precision. Therefore, it is foreseeable that B5G/6G networks need to have both communication and sensing capabilities to improve spectrum resource utilization. Among them, Integrated Awareness Communication (ISAC) technology is widely considered to be one of the effective solutions to achieve this goal. In order to meet users' realistic needs for full-area collaborative coverage of synesthesia services, future wireless networks need to be deployed more in complex terrain and electromagnetic environments such as dense cities and mountainous areas. However, in the above-mentioned complex environment, existing network technologies represented by cellular networks and Global Navigation Satellite Systems (GNSS) have certain flaws, making their network performance unable to meet users' high-quality communication and perception needs.

Especially in harsh environments, such as high-rise buildings or remote mountainous areas, traditional satellite positioning services may cause poor communication and perceived service quality for ground users due to problems such as network marginalization. In this case, the drone swarm, with its advantages of high mobility and flexible deployment, is expected to make up for this shortcoming by combining corresponding communication and sensing technologies. Therefore, UAV swarm-assisted technology has received widespread attention from academia and industry in recent years. However, the current power supply method for drones is usually based on battery charging. Even if a few models of drones can be replenished with energy through solar energy or other solutions, their energy supply is relatively limited, so the drones themselves are resource-constrained. insufficient. In this case, how to reasonably allocate resources such as power and service users to reduce the energy consumption of space-based platforms becomes increasingly important. Therefore, how to minimize the energy consumption of UAVs by optimizing the power allocation and service strategies of UAVs on the basis of completing communication and sensing tasks is the core research issue of this invention. Research on this issue is expected to effectively reduce the energy consumption of space-based cluster systems while ensuring system communication and perception performance, thereby improving the service life of the synesthetic integrated network of UAV clusters.

Contents of the invention

The purpose of the present invention is to provide an integrated energy consumption optimization method for UAV swarms based on reinforcement learning to solve the existing problem of large energy consumption in the ISAC network based on UAV swarms, while ensuring the communication of terminal ground users. and positioning real-time service performance, effectively improving network service life.

The invention provides a method for optimizing the energy consumption of synaesthesia for UAV groups based on reinforcement learning. The established system environment is as follows: In a single base station covering cellular synaesthesia network, the base station coordinates are (x ₀ , y ₀ , z ₀ ), the set of UAVs is The set of ground users is/> A complete decision-making process for a drone as an intelligent agent is called a decision-making cycle. In the present invention, a complete process for the drone to complete positioning sensing and communication task offloading is a decision-making cycle. It is assumed that each time t is a decision-making cycle. cycle, the state set of all decision-making cycles is called the state space set S, expressed as: S={s ₁ , s ₂ ,..., s _t ,...}. The set of actions in all decision-making cycles is called the action space set A, expressed as: A={a ₁ , a ₂ ,..., a _t ,...}. At time t, the position coordinates of drone-m and user-l to be located are u _m (t) = (x _m (t), y _m (t)) ^T and v _l = (x _l , y _l respectively) ) ^T , and assume that the height of the UAV platform-m is a fixed value H _m . All drones in the cluster fly on preset flight trajectories that meet a certain threshold range, while providing communication and location awareness services to ground users, and each drone can be identified by the ground user terminal to be served. .

The first step is to use the positioning perception performance of the UAV cluster to assign an initial value to the Q-value function network that reflects the relationship between UAV status and action.

The second step is to determine the current status of the drone from the environment. A certain working point can be selected as the initial state of the drone.

The third step is to select the current action based on the ε-greedy strategy based on the current status of the drone.

The fourth step is to introduce the drone's perception performance and communication task energy consumption into the design of the Reward reward function, and obtain the actual environmental reward value of the previous action and the next state of the drone.

The fifth step is to update the Q-value function network using the actual environmental reward value of the previous action.

Step 6: Set the new state to the current state and repeat steps 3 to 6 until the values in the Q-value function network converge.

The above steps mainly involve the following key technical points:

(1) Initialize the Q-value function network using the positioning sensing performance of the UAV cluster

The geometric configuration between the UAV and the user on the ground to be served is the premise that affects its positioning perception performance. The position (three-dimensional) precision factor PDOP can be used to characterize this performance. The drone selects different actions in different states, so the geometric configuration between the drone and the user is different, and its ability to provide perceptual services to the user to be served is different. Assume that at time t, the subset of UAV base stations that provide positioning awareness services for user-l is S _k (t), and it is assumed that the number of UAVs in the set is M ₀ . Then calculate the position (three-dimensional) accuracy factor of the UAV subset value, which can be expressed as:

In the above formula, is the Jacobian matrix of the positioning sensing observation equation of the UAV base station subset S _k (t), then it can be further expressed as the following formula:

In the above formula, u ₁ (t) = (x ₁ (t), y ₁ (t)) ^T , (1∈s _k (t)), Respectively represent the coordinates of UAV-1, UAV-m ₀ and UAV-M ₀ in the UAV base station subset S _k (t); similarly, H ₁ , /> are respectively the fixed height values of UAV-1, UAV-m ₀ and UAV-M ₀ in the UAV base station subset S _k (t), v _l = (x _l ,y _l ) ^T Then it is the location coordinate of user-l to be located.

The method of initializing the Q-value function network using the positioning perception performance of the UAV cluster is as follows: when the UAV is in the normal working state, when a subset of UAV base stations S _k (t) is selected to provide synaesthesia services to users at time t, its The value in the corresponding Q-table grid is The remaining grid positions are assigned zero values. The positioning perception performance of the UAV cluster is used to assign an initial value to the Q-value function network, thereby providing prior information for the reinforcement learning network of the UAV agent, which further facilitates the learning of the agent.

(2) Energy consumption of communication task upload

At time t, the LoS channel power gain from the m-th drone to the l-th ground user can be expressed as:

Among them, α is the path loss coefficient of the channel, β ₀ is the unit (per meter) channel gain, d _m,l (t) is the distance from the m-th drone to the l-th ground user, and there is Furthermore, the signal-to-noise ratio SNR _m,l (t) of the link at that moment can be expressed as:

Among them, P is the constant transmit power of the space-based platform, σ ² is the noise power, Indicates whether the space-based platform provides synaesthesia services for ground users. The specific meaning is: When/> Synaesthesia services are not provided when/> Synaesthesia services are provided at all times. /> is the interference from other UAV platforms, that is, co-channel channel interference, where P _u (t) is the transmit power of other UAV platforms at time t; g _u,l (t) is the transmission power of other UAV platforms at time t LoS channel power gain at t;/> Represents a collection of drones. Then the data transmission rate R _m,l (t) of the link from the m-th drone to the l-th ground user can be expressed as:

R _m,l (t)=B·log ₂ (1+SNR _m,l (t)) (5)

Among them, B is the signal bandwidth. Furthermore, the energy consumption _Em,l (t) of the link can be expressed as:

Among them, P is the constant transmit power of the space-based platform, Indicates whether the space-based platform provides synaesthesia services for ground users. The specific meaning is: When/> Synaesthesia services are not provided when/> Synaesthesia services are provided at all times. P _m (t)∈[0,1,,5] represents the number of users served by a drone. We assume that each ground user's data upload task can only be sent to at most one drone.

(3) Action selection strategy

As a value-based reinforcement learning algorithm, the Q-learning algorithm uses iterative updates of the Q-value function to find the optimal strategy π ^* for the UAV (agent). During the running of the algorithm, the agent selects actions to execute according to the ε-greedy strategy, that is, the probability that the agent randomly selects an action is ε, and the probability of selecting the action corresponding to the maximum value in the Q-value function network is 1-ε .

When the algorithm starts to be executed, the Q-table is first initialized using the method in (1) above, and then the current state s _t is selected. For each action a in this state, there is a corresponding "state-action" value. , and expressed as Q(s _t ,a). In this case, the action in this state is selected according to the ε-greedy strategy, that is, the action corresponding to the maximum value in the Q-value function network is selected, as shown in the following formula:

After selecting the action, the agent starts to perform this action, and then enters the next state s _t+1 and obtains the reward value r(t) within the current action selection decision period. At the same time, the value of the corresponding position in the corresponding Q network is updated:

Among them, γ is the discount factor, and γ∈[0,1]. Regarding the reward value function r(t), see (4) for detailed description.

(4)Design of reward value function r(t)

In order to comprehensively ensure the communication and perception performance of the UAV swarm network, the reward function r(t) is designed to be expressed as the following formula at time t:

Among them, SNR _thr is the signal-to-noise ratio threshold of known system parameters set in advance to ensure the communication performance of the agent; PDOP _thr is the three-dimensional accuracy factor threshold of known parameters. The purpose is to ensure the perception performance of the agent. Based on this function, the synaesthesia characteristics of the UAV swarm network in the energy consumption optimization action selection process are guaranteed.

Compared with existing methods, the present invention's synaesthesia-integrated energy consumption optimization method for UAV groups based on reinforcement learning has the following advantages and beneficial effects:

(a) The present invention comprehensively considers the communication sensing performance and network energy consumption of UAV clusters, and proposes an energy-efficiency optimal strategy based on the synaesthesia integrated network of UAV clusters, which can ensure UAV system communication and perception. Under the premise of high performance, it can effectively reduce the energy consumption of UAV network, thereby improving the network service life of UAV clusters with limited resources.

(b) The present invention designs an intelligent decision-making algorithm based on reinforcement learning, so that the drone can adaptively select the number of service users and power selection according to the dynamically changing environment, and on the premise of ensuring the synaesthesia performance of the system, with the minimum The task upload traffic that consumes the most network energy is offloaded, avoiding the rigid model of traditional centralized network control and overcoming the difficulties in policy formulation caused by environmental dynamics.

Description of the drawings

In order to provide a clearer explanation of the technical principles and specific process solutions of the proposed invention, the relevant drawings involved in the embodiments will be briefly described and introduced below. Obviously, the accompanying drawings 1 to 3 described below are only for description and illustration of the embodiments. For ordinary related technical researchers in this field, such other drawings can also be obtained without creative work. .

Figure 1 is a schematic flowchart of the synaesthesia-integrated energy consumption optimization method for UAV swarms based on reinforcement learning proposed by the present invention.

Figure 2 is a schematic diagram of the learning process of the Q-value function network proposed by the present invention.

Figure 3 is a schematic diagram of a synesthesia integrated network scenario based on a drone group targeted by the present invention.

Detailed ways

The features and principles of the present invention will be further described below with reference to the drawings and examples. The examples listed are only used to explain the present invention and are not intended to limit the scope of application of the present invention.

Referring to Figure 1, considering a cellular network covered by a single base station and a network radius of 500m, the present invention proposes a synaesthetic integrated energy consumption optimization method for UAV swarms based on reinforcement learning. The parameters will be used below Indicates whether the corresponding selected user is served; P _m (t)∈[0,1,,5] indicates the number of users served by a drone; Assume that the exploration rate of the reinforcement learning network is set to ε = 0.8; the discount factor γ = 0.9; l _max =1000; communication threshold SNR _thr =2dB; positioning sensing threshold PDOP _thr =1.5; the number of UAVs used to provide positioning sensing services to each ground user is 4; and it is assumed that the data of each ground user The upload task can only be sent to at most one drone as an example to further explain and introduce in detail the specific implementation method of the overall invention method provided.

Step 1: During the operation of the system, first establish the Q-table grid, and then use the UAV cluster positioning perception performance, that is, the three-dimensional accuracy factor, to initialize the Q-value function network according to the proposed plan. Specifically, when no one When the drone is in normal working state, when a subset of UAV base stations S _k (t) is selected to provide synaesthesia services to users at time t, the value in its corresponding Q-table grid is -PDOP _{sk (t)} , and the rest The grid position is assigned a value of zero.

Step 2: Select a certain state as the initial state of the drone agent based on the current network environment.

Step 3: Based on the ε-greedy strategy, select the action to select the drone service status and the number of service users under the current state s _t selected in step 2, which is determined according to formula (7) and ε = 0.8 and the value of P _m (t). Specifically, the action corresponding to the maximum value in the Q-value function network is selected with probability 1-ε=0.2, that is/> Randomly select actions with probability ε = 0.8;

Step 4: After the action decision is completed, the UAV obtains the energy consumption of the communication upload task within the action decision period, that is, E _m,l (t) is obtained through formula (6), and Substituting the value of P _m (t), the communication threshold SNR _thr =2dB and the positioning sensing threshold PDOP _thr =1.5 into formula (9), the reward value r(t) is calculated and transferred to the next state s _t+1 ;

Step 5: Substitute the exploration rate ε = 0.8 and discount factor γ = 0.9 of the reinforcement learning network into formula (8) to obtain Q(s _t , a _t ) under the action, thereby updating the value of the Q-value function network;

Step 6: Set the new state s _t+1 as the current state, and repeat steps 3 to 6 until the values in the Q-value function network reach convergence.

When the values in the Q-value function network finally reach the convergence state through continuous updating, the Q-table can be used to guide the drone to make the best decision in the corresponding state, that is, to select the optimal number of service users and the number of service users in the corresponding state. Transmit power to obtain the optimal user communication task traffic offloading strategy, that is, the optimal energy efficiency of the UAV. The entire process of the algorithm is given below:

Q-Learning Algorithm: Obtain the optimal energy efficiency strategy for the synaesthesia integrated network of UAV clusters

Initialization for any s∈S, a∈A(s),

Use key technology (1) to assign initial value to Q-table

Initialize t=1, ε=0.8, γ=0.9,

repeat:

Initialize state s according to current environment information

Repeatedly executed in each action decision cycle t:

Select action a in state s according to ε-greedy strategy

Execute action a, use formula (9) to obtain the reward function value and enter the next state s’

Use formula (8) to update the corresponding value of Q-table

Let t=t+1, s’=s

Repeat the above steps until the maximum number of iterations l _max =1000 is reached.

The overall detailed flow chart of the features and principles of the present invention involved in the above-mentioned embodiments is shown in Figure 2, and the relevant scene diagram targeted by the present invention is shown in Figure 3.

In summary, the present invention proposes a synaesthesia-integrated energy consumption optimization method for UAV groups based on reinforcement learning, which provides prior information for the reinforcement learning network by utilizing the perception performance of UAVs, allowing the UAVs to operate faster , achieve the best target state efficiently. On the other hand, on the premise of ensuring the communication and sensing performance of the UAV cluster, the system task energy consumption can be reduced, thereby effectively extending the service life of the UAV network, which is of great significance for UAV systems with limited resources.

Claims (10)

1. A synaesthesia-integrated energy consumption optimization method for UAV swarms based on reinforcement learning, which is characterized by: including the following steps: The first step is to use the positioning perception performance of the UAV cluster to assign an initial value to the Q-value function network that reflects the relationship between UAV status and action; The second step is to determine the current state of the drone from the environment; select a certain working point as the initial state of the drone; The third step is to select the current action based on the ε-greedy strategy according to the current status of the drone; The fourth step is to introduce the UAV perception performance and communication task energy consumption into the design of the Reward reward function, and obtain the actual environmental reward value of the previous action and the next state of the UAV; The fifth step is to update the Q-value function network using the actual environmental reward value of the previous action; Step 6: Set the new state to the current state and repeat steps 3 to 6 until the values in the Q-value function network converge. 2. A synaesthetic integrated energy consumption optimization method for UAV swarms based on reinforcement learning according to claim 1, characterized in that: the initial value assigned to the Q-value function network is: set at time t, providing positioning for user-l The subset of UAV base stations for sensing service is S _k (t), and the number of UAVs in the set is M ₀ ; then calculate the position accuracy factor of the UAV subset value, expressed as: In the above formula, is the Jacobian matrix of the positioning sensing observation equation of the UAV base station subset S _k (t). 3. A synaesthesia-integrated energy consumption optimization method for UAV swarms based on reinforcement learning according to claim 2, characterized by: It is further expressed as the following formula: Among them, u ₁ (t)=(x ₁ (t), y ₁ (t)) ^T , (1∈s _k (t)), (M ₀ ∈s _k (t)) respectively represent the coordinates of UAV-1, UAV-m ₀ and UAV-M ₀ in the UAV base station subset S _k (t); similarly, H ₁ ,/> are respectively the fixed height values of UAV-1, UAV-m ₀ and UAV-M ₀ in the UAV base station subset S _k (t), v _l = (x _l ,y _l ) ^T Then it is the location coordinate of user-l to be located. 4. A synaesthesia-integrated energy consumption optimization method for UAV groups based on reinforcement learning according to claim 1 or 2, characterized in that: when the UAV is in a normal working state, the UAV base station is selected at time t When the subset S _k (t) provides synaesthesia services to users, the value in its corresponding Q-table grid is The rest of the grid positions are assigned zero values; the positioning perception performance of the UAV cluster is used to assign an initial value to the Q-value function network. 5. A method for optimizing the synaesthesia integrated energy consumption of UAV groups based on reinforcement learning according to claim 1, characterized in that: the energy consumption of the communication task is, at time t, the mth UAV to the lth UAV The LoS channel power gain of ground users is expressed as: Among them, α is the path loss coefficient of the channel, β ₀ is the unit (per meter) channel gain, d _m,l (t) is the distance from the m-th drone to the l-th ground user, and there is 6. A method for optimizing energy consumption of synaesthesia for UAV groups based on reinforcement learning according to claim 5, characterized in that: the signal-to-noise ratio SNR _m,l (t) of the link at that moment is expressed as: Among them, P is the constant transmit power of the space-based platform, σ ² is the noise power, Indicates whether the space-based platform provides synaesthesia services for ground users. The specific meaning is: When/> Synaesthesia services are not provided when/> Provide synaesthesia services;/> is the interference from other UAV platforms, that is, co-channel channel interference, where P _u (t) is the transmit power of other UAV platforms at time t; g _u,l (t) is the transmission power of other UAV platforms at time t LoS channel power gain at t;/> represents a collection of UAVs; then the data transmission rate R _m,l (t) of the link from the m-th UAV to the l-th ground user is expressed as: R _m,l (t)=B·log ₂ (1+SNR _m,l (t)) (5) Among them, B is the signal bandwidth. 7. A synaesthetic integrated energy consumption optimization method for UAV swarms based on reinforcement learning according to claim 6, characterized in that: the energy consumption of the link E _m,l (t) is expressed as: Among them, P is the constant transmit power of the space-based platform, Indicates whether the space-based platform provides synaesthesia services for ground users. The specific meaning is: When/> Synaesthesia services are not provided when/> Synaesthesia services are provided at any time; P _m (t)∈[0,1,…,5] represents the number of users served by a drone; the data upload task of each ground user can only be sent to one drone at most. machine. 8. A method for optimizing energy consumption of UAV synaesthesia based on reinforcement learning according to claim 2, characterized in that: selecting the current action based on the ε-greedy strategy is: the probability that the agent randomly selects the action is ε, the probability of selecting the action corresponding to the maximum value in the Q-value function network is 1-ε; When execution starts, first use the method in formula (1) to initialize the Q-table, and then select the current state s _t . For each action a in this state, there is a corresponding "state-action" value, And expressed as Q(s _t ,a); in this case, the action in this state is selected according to the ε-greedy strategy, that is, the action corresponding to the maximum value in the Q-value function network is selected, as shown in the following formula: 9. A synaesthesia integrated energy consumption optimization method for UAV swarms based on reinforcement learning according to claim 8, characterized in that: after selecting an action, the agent starts to perform this action, and then enters the next state s _{t+ 1} and obtain the reward value r(t) within the current action selection decision period, and at the same time update the value of the corresponding position in the corresponding Q network: Among them, γ is the discount factor, and γ∈[0,1]. 10. A method for optimizing the integrated energy consumption of UAV swarm synaesthesia based on reinforcement learning according to claim 9, characterized in that: the reward value r(t) is designed: the design is at time t, and the reward function r(t) represents is the following formula: Among them, SNR _thr is the known system parameter signal-to-noise ratio threshold set in advance to ensure the communication performance of the agent; PDOP _thr is the three-dimensional precision factor threshold of known parameters; the purpose is to ensure the perception performance of the agent; based on this function, the synaesthesia characteristics of the UAV swarm network in the energy consumption optimization action selection process are guaranteed.