CN116520887A - Adaptive adjustment method for cluster structure of hybrid multiple unmanned aerial vehicles - Google Patents

Abstract

The invention provides a self-adaptive adjustment method for a cluster structure of a hybrid multi-unmanned aerial vehicle, which is characterized in that by designing a special type of state encoder, the attribute and the capability of different dimensions of heterogeneous unmanned aerial vehicles are converged to the same dimension, the dimension of a knowledge space is reduced, then multidimensional heterogeneous data which is processed by fusion of a cluster clustering algorithm based on reinforcement learning is used, a global adjustment strategy is learned, and the cluster structure and a reconstruction network are adjusted. The technical core of the state encoder is that a set of neural network is independently trained for each type of unmanned aerial vehicle, and special treatment is carried out on the attribute and the capability of different dimensions of each type of unmanned aerial vehicle, so that the difference among heterogeneous unmanned aerial vehicles is reflected better. According to the method, the characteristics of different dimensions of each type of unmanned aerial vehicle can be converged under the mixed situation, the problem of multi-dimensional heterogeneous data fusion is effectively solved, and the stability and the efficiency of the mixed multi-unmanned aerial vehicle cluster are improved.

Description

A hybrid multi-UAV swarm structure adaptive adjustment method

Technical Field

The present invention relates to the field of unmanned aerial vehicles (UAVs), and in particular to a method for adaptively adjusting a structure of a mixed multi-UAV cluster.

Background Art

The multi-dimensional heterogeneity in the mixed multi-UAV cluster scenario makes heterogeneous UAVs have great differences in motion state and communication capabilities. These differences make the topological structure inside the cluster more likely to change dynamically, resulting in changes in the relative positions and communication costs between UAVs, causing instability in the cluster structure. Cluster structure instability refers to the frequent changes in the cluster structure caused by factors such as changes in the UAVs that make up the cluster or disconnection of communication connections. When the cluster structure is unstable in the multi-UAV cluster system, in order to optimize the overall system operation efficiency, it is necessary to make necessary cluster structure adjustments to the system to ensure the stability of the cluster structure. When facing the mixed multi-UAV cluster scenario, the cluster structure adjustment method needs to solve the problem of multi-dimensional heterogeneous data fusion, which most of the existing multi-UAV cluster structure adjustment models do not consider (for example, when calculating the similarity of the motion state of heterogeneous UAVs, only the current speed and direction of the UAV are considered, ignoring the influence of other dimensional attributes such as the acceleration of the variable-speed UAV), the state extraction is inaccurate, and it is easy to fall into the local optimal solution. If we directly consider the attributes of all different dimensions of heterogeneous drones and expand the dimensions of all drone attributes to include all different types of attributes (for example, adding acceleration attributes and maximum speed attributes with a value of 0 for fixed-speed drones), the dimension of the problem solution space will increase significantly. It can be seen that the multi-dimensional heterogeneous data fusion problem in the context of mixed multi-drone clusters has an extremely important impact on the optimization of the operating efficiency of the entire system.

The fusion of multi-dimensional heterogeneous data in the context of mixed multi-UAV clusters is to improve the flexibility of the cluster structure adjustment model, reduce the dimension of the solution space, obtain better optimization effects, and maximize the operating efficiency of the entire system. In order to achieve this optimization goal, a state encoder can be designed for each type of heterogeneous UAV in the system to aggregate its attributes of different dimensions. Each state encoder uses a specially trained neural network, combined with the historical state of the heterogeneous UAV. The attributes and capabilities of the heterogeneous UAVs in different dimensions are processed by the corresponding type of state encoder and aggregated into the observation features of the UAVs with the same dimension (for example, a certain type of UAV has two dimensions of residual energy attributes, namely, residual flight energy and residual communication energy. The state encoder of this type of UAV will aggregate them according to the consumption rate of the two energies in its historical state, that is, weighted sum the energy attributes of the two dimensions according to the appropriate weights to obtain the energy attributes in the UAV observation features).

In order to integrate the processed multi-dimensional heterogeneous features with the network structure, this method introduces a cluster clustering algorithm based on reinforcement learning. The algorithm constructs a graph attention network based on the network structure of the system, uses the self-attention mechanism to integrate the features of all heterogeneous drones, and learns the pros and cons of different cluster structure adjustment strategies through the Actor-Critic architecture model, and finally gives the optimal cluster structure adjustment strategy in the current situation.

Summary of the invention

Technical issues:

The purpose of the present invention is to propose a cluster structure adjustment method under a mixed multi-UAV cluster, which mainly solves the problem that multi-dimensional heterogeneous data is difficult to fuse under mixed scenarios, which will lead to too large algorithm solution space and frequent changes in cluster structure. The core technology of the present invention is to use the multi-dimensional information of the attributes and capabilities of heterogeneous UAVs, aggregate them into a single state attribute, reduce the dimension of the solution space, use a cluster clustering algorithm based on reinforcement learning to fuse heterogeneous UAV features and cluster structure data, learn the global cluster structure adjustment strategy, adjust the cluster structure according to the strategy and reconstruct the communication network, so as to achieve global cluster structure adjustment, and further improve the stability of the cluster structure and the efficiency of the system.

Technical solution:

In a mixed multi-UAV swarm system, each UAV undertakes a different number of tasks, and the UAVs in the same swarm are often closely connected. In order to establish a stable communication network within the swarm at a low cost, we need to select the communication UAV Cluster Head (CH) in each swarm, and the member UAVs ClusterMembers (CMs) of the swarm must establish a communication connection with the CH.

In the initial stage, the given scenario is a mixed multi-UAV cluster system with uniform distribution of UAVs. U heterogeneous UAVs with multi-dimensional heterogeneity are designed, and the types of UAVs in each cluster are evenly distributed. In view of the multi-dimensional heterogeneity of the characteristics of heterogeneous UAVs in each cluster, this scheme uses a specific type of state encoder for heterogeneous UAVs to process the state characteristics of different types of UAVs respectively, and trains a set of neural networks for each type of UAV to process its attributes and capability states with different dimensions. Finally, the attributes and capability characteristics of different dimensions are aggregated through the fully connected layer in the neural network to obtain the observations of UAVs with the same dimensions.

Each type of heterogeneous UAV has a dedicated neural network, which is trained based on the historical state of that type of UAV. It is used to combine the characteristics of the UAV over the past period of time and process the characteristics of the UAV with different current dimensions to generate observation features of the same dimension. These observation features are used to learn the global cluster structure adjustment strategy through the reinforcement learning method based on the Actor-Critic architecture. The cluster structure is adjusted according to the learned strategy and the communication network is reconstructed.

Specifically, the neural network of each type of drone uses reset gates and update gates to combine historical states and current features to update its current features, and uses fully connected layers to aggregate attributes and capabilities of different dimensions into observation features of the same dimension. These observation features are sent to the graph attention network, which processes the observations of all drones in the system through the self-attention mechanism to obtain observation features representing the entire system. The Actor network will give a strategy π _θ based on the system observation features, which represents the probability of taking each global adjustment strategy under the current system state.

According to the π _θ output by the Actor network, a global adjustment strategy is selected. The utility of the UAV is obtained by weighted summing of the state factors of the heterogeneous UAVs according to the utility weights in the adjustment strategy. Clustering is performed according to the utility and communication range. The network structure of the re-clustered mixed multi-UAV cluster is reconstructed in two layers, within the cluster and across the cluster, to complete a cluster structure adjustment. At the same time, the Critic value network evaluates the optimization effect of this cluster structure adjustment and calculates the value function of the current state and the next state. The Critic state value function and the Actor strategy parameter θ are updated through the policy gradient function to train the model.

This method solves the problem of multi-dimensional heterogeneous data fusion in a mixed multi-UAV cluster scenario, aggregates the UAV states with multi-dimensional heterogeneity into the UAV observation features of the same dimension, and reduces the dimension of the understanding space. On this basis, the reinforcement learning method is used to learn the optimal cluster structure adjustment strategy to achieve the best optimization effect.

Beneficial effects:

(1) Enhance the adaptability to mixed multi-UAV cluster application scenarios. By solving the fusion problem of multi-dimensional heterogeneous data in mixed multi-UAV clusters, the impact of the attributes and capabilities of heterogeneous UAVs in different dimensions in mixed scenarios is fully considered. This can solve the problem that the previous cluster structure adjustment methods that did not consider multi-dimensional heterogeneity have poor adaptability and are prone to falling into local optimal solutions.

(2) Reduce the dimensionality of the understanding space. Use state encoders for heterogeneous drone types to perform special processing for each type of drone, and aggregate their attributes and capability states of different dimensions into a unified dimension for all drones in the system, which greatly reduces the dimensionality of the understanding space.

(3) Improving the stability of the multi-UAV cluster structure. The adaptive cluster structure adjustment method for mixed scenarios can integrate the attributes and capabilities of heterogeneous UAVs in different dimensions with multi-dimensional heterogeneous data such as cluster structures, give the optimal cluster structure adjustment strategy, achieve dynamic stability of the cluster structure, and improve the operating efficiency of the overall system.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of the structure of a heterogeneous multi-UAV swarm system.

FIG2 is an example diagram of a partial cluster structure.

FIG. 3 is a diagram showing the main principles of the method of the present invention.

FIG. 4 is a network structure diagram of the method of the present invention.

The different shapes in Figures 1 and 2 represent different types of drones. In Figure 2, the selected CH communication drone in the cluster is responsible for communicating with all other CMs member drones in the cluster. The communication between homogeneous drones is connected by solid lines, and the communication between heterogeneous drones is connected by dotted lines.

DETAILED DESCRIPTION

The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

As shown in the figure, the present invention provides a method for adaptively adjusting a hybrid multi-UAV cluster structure.

1. Initial stage

In the initial stage, heterogeneous drones, clusters, and mixed multi-drone clusters are perceived: the heterogeneous drone perception process analyzes the type, attributes, capabilities, status, and other characteristics of each drone; the cluster perception process needs to perceive the number of clusters in the current multi-drone cluster system and the composition of heterogeneous drones in each cluster; the mixed network structure perception process needs to perceive the network structure composed of communication connections between drones within and across clusters, and also calculate the changing communication costs between heterogeneous drones. In the initial stage, we describe these three perception objects and perception processes in turn:

1.1 Heterogeneous UAV perception process: Specific analysis of each UAV. Any UAV in the kth cluster C _k

It can be represented by a triple: in Represents the heterogeneous drones in the kth cluster in the system Type of drone; Representative heterogeneous drones Attributes and capability characteristics (the embodiment of multi-dimensional heterogeneity, the dimensions of the feature vector P between heterogeneous drones are different), such as remaining energy, speed, etc.; Representing heterogeneous drones state features (the dimensions of the state feature vector v are the same and have nothing to do with the type of drone), such as drone number, current position coordinates, direction, role, etc.

1.2 Cluster perception process: In the initial stage, we first need to analyze the composition of the drone cluster. The multi-drone cluster system is composed of multiple heterogeneous drone clusters C _k , each drone cluster C _k consists of heterogeneous drones. composition,

Where _Nk is the number of drones in the cluster.

1.3 Hybrid network structure perception process: In the hybrid multi-UAV cluster system targeted by this patent, there are two types of connection relationships between heterogeneous UAVs, one is the communication connection within the cluster, and the other is the communication connection across clusters. The communication connection between heterogeneous UAVs constitutes a hybrid network structure within the system. Both connections are dynamic and will change with the adjustment of the cluster structure. Combining the intra-cluster connection and cross-cluster connection in the heterogeneous multi-UAV system, the hybrid multi-UAV cluster system (MAS) with N clusters is finally described as MAS = (n ₁ ,n ₂ ,…,n _N ,E), where n _k = <C _k ,CH _k >

represents the complete perception of the k-th cluster, C _k is the heterogeneous drone set of the k-th cluster, CH _k represents the current communicating drone of the k-th cluster, and E is the adjacency matrix representing the network structure within the heterogeneous multi-drone cluster system. Using binary

Represents the communication connection in the system. The communication cost of all communication connections when they are established and maintained at each subsequent moment is expressed as Represents the heterogeneous drones in the cluster C _k With heterogeneous drones in cluster C _l The communication cost is , where l may be equal to k. Function c will be based on the drone and The communication cost is calculated based on the types of nodes and the distance between them.

2. Model construction and problem formulation

In order to adjust the cluster structure through clustering and routing algorithms in a mixed multi-UAV cluster system, it is necessary to evaluate the status of all UAVs, which requires us to use the original input obtained in the perception phase to calculate the utility of each UAV and reflect the status of the UAV through the calculated utility value. This model designs six factors for the utility of UAVs: remaining energy percentage, degree, average distance to neighbors, average motion state similarity to neighbors, average maintenance time of communication connection, and average communication cost. The utility of UAVs is obtained by weighted summation of these six factors.

The weight of each factor will affect the calculation of utility, and thus affect the accuracy of drone status assessment. As heterogeneous drones move, the cluster structure changes continuously, and the optimal utility calculation weight that can accurately reflect the drone status in the current scenario will also change accordingly. The cluster structure adjustment method in the mixed multi-drone cluster scenario needs to consider multi-dimensional heterogeneity, integrate the characteristics of heterogeneous drones in different dimensions and multi-dimensional heterogeneous data such as cluster structure, and give the optimal cluster structure adjustment strategy in the current scenario, that is, the optimal utility calculation weight.

The optimization goal of this problem is to give a cluster structure adjustment strategy that can achieve a balance between cluster structure stability and system communication costs, and achieve optimal system operation efficiency. By determining the cluster structure adjustment strategy, we intend to maximize the system operation efficiency over a long period of time. The entire system, as an intelligent entity, connects with the environment in which the system is located according to the cluster structure adjustment strategy. The Markov decision model for this problem is given below:

State: S(t), represents the cluster structure state observation of the entire system at time interval t, including the state, distribution, communication network structure and CH communication drones of the heterogeneous drones in each cluster. First, at the beginning of each time interval t, the composition and state of the heterogeneous _drones in the kth cluster are defined as:

where S _k (t) is the number of heterogeneous drones in the kth cluster Observations at time interval t N _k (t) is the number of heterogeneous drones in the kth cluster at time t. Definition for:

At time interval t, For drones Type; The attributes and capability characteristics of heterogeneous drones, including remaining energy, speed, etc. It is the status characteristics of heterogeneous drones, including drone number, location coordinates, direction, role, etc.

Then the network structure E(t) of the system at time t is defined as an adjacency matrix.

Finally, the state S(t) of the entire system can be expressed as:

S(t)＝(S ₁ (t),S ₂ (t),…,S _k (t),…,S _N(t) (t),E(t))

Where N(t) is the number of heterogeneous drone clusters in the system at time t.

·action: Define the system's cluster structure adjustment strategy. is the strategy to be adopted by the system in time interval t, defined as:

Where λ _i (i∈[1,6]) represents the utility weight of the i-th factor of the drone, and λ _i satisfies the following constraints.

λ ₁ +λ ₂ +λ ₃ +λ ₄ +λ ₅ +λ ₆ =1

Reward function: r(t), the strategy adopted by the system at time t and state S(t) Cluster structure adjustment

The global reward can be expressed as:

λ is the weight, _rs is the cluster structure stability reward, and _rc is the negative reward caused by energy consumption.

r _s =α ₁ r _ATT +α ₂ r _CTD +α ₃ r _SMD +α ₄ r _CTC

r _c =-(β ₁ r _build +β ₂ r _maintenance +β ₃ r _isolated +β ₄ r _fly )

Among them, α and β are weights, r _ATT represents the average time that the UAVs stay in the current cluster; r _CTD represents the tightness of the network structure between the UAVs in each cluster; r _SMD represents the similarity of the motion states between the UAVs in each cluster; r _CTC represents the average time that the communication connection between the UAVs in each cluster is maintained; r _build represents the communication cost of building a new communication connection caused by the cluster structure adjustment; r _maintenance represents the maintenance cost of the communication network in the system from the previous moment to this moment; r _isolated represents the additional communication loss caused by the isolated clusters in all clusters; r _fly represents the flight energy consumption of all UAVs in the system at the current moment.

Where N is the number of clusters, _Ni is the number of drones in cluster _Ci , and mt _i,j is the jth drone in cluster _Ci The time to stay in the current cluster.

in is the UAV in cluster _Ci The distance between the communication drone CH _i in the cluster.

in is the UAV in cluster _Ci The similarity of the motion state between the communicating drone CH _i in the cluster, speed is the current speed of the drone, and θ is the direction of the drone.

in is the UAV in cluster _Ci The time the communication connection with the communicating drone CH _i in the cluster is maintained.

in is the communication cost of the communication connection calculated in 1.4, I ^t is the newly constructed communication connection at time t, and the formula is:

I ^t = E(t)-(E(t)∩E(t-1))

The communication cost required for the newly constructed communication connection of the hybrid multi-UAV cluster at time t is calculated.

The communication maintenance cost of all communication connections within the mixed multi-UAV cluster at the last moment is calculated.

r _isolated = num _isolated

Where num _isolated is the number of isolated clusters in the system that have no cross-cluster communication connection with any other clusters, that is, the number of clusters that have no communication connection in E. Isolated clusters cannot communicate directly with other clusters and need to obtain the actions and status of other clusters through the system base station. Each isolated cluster will cause additional communication loss.

in is the flight energy consumption of the jth UAV in cluster _Ci at the current moment, which is affected by the type and speed of the UAV.

On this basis, according to the time step t＝1,2,…,T of the decision model and the strategy of the system Indicates that in state S(t), the system takes action probability.

According to the definition of Markov decision, the strategy of each cluster should meet the optimization condition, that is, the strategy of the system should maximize its reward function r(t), which is:

in is the action space that the system can take, which is the set of all possible utility weights.

3. Cluster structure adjustment algorithm based on reinforcement learning

This patent uses a reinforcement learning algorithm to solve the problems specified above. First, a neural network needs to be built for the Markov decision model, and then trained through the Actor-Critic framework to update the value network parameters and strategy network parameters. In order to enable the neural network to integrate the dynamically changing multi-dimensional heterogeneous data in the mixed multi-drone cluster, this patent divides the construction of the neural network in the algorithm into two levels: a special state encoder for heterogeneous drone types and a mixed multi-drone cluster structure.

3.1 Special state encoders for heterogeneous UAV types

First, we need to process the different attributes and capabilities of heterogeneous drones. There are n types of heterogeneous drones in the system, and a network is trained separately for each type of drone to process its features. For the i-th drone in the k-th cluster at time t According to the original input system state S(t) = ( _S1 (t), _S2 (t), ..., _Sk (t), ..., _SN(t) (t), E(t)), the observations of the heterogeneous UAVs contained in _Sk (t) are Extract in Represents the type number of the heterogeneous drone.

This patent uses GRU to design this special state encoder for heterogeneous drone types. The input of the encoder is the type of heterogeneous drone Attributes and capability characteristics of heterogeneous drones in different dimensions The output is a feature vector of uniform dimension The following introduction The specific calculation process.

Remember to input the drone Type number For type, the drone stored in GRU History (hidden) status is a vector representing the drone Historical features at the last moment. When we can Add to the input data. We use two gates for type type drones: reset gate r ^type and update gate u ^type . The reset gate is used to control whether to ignore the hidden state of GRU Its calculation formula is as follows:

Type is drone The drone type number, and is the weight matrix of the input features and hidden states for r ^type in the GRU network that processes the type type drone state. is the bias vector and σ is the sigmoid function.

The update gate u ^type is used to control the GRU to update the hidden state The calculation formula is as follows:

and is the weight matrix of input features and hidden states for u ^type in the GRU network that processes the type type drone state. is the bias vector and σ is the sigmoid function.

Calculate the candidate features at the current moment based on r ^type and u ^type

in Represents the multiplication of each element of the reset gate vector and the hidden state vector, and is the input and hidden state pair in the GRU network that processes the type type drone state The weight matrix, is the bias vector, tanh represents the hyperbolic tangent function, and maps the input value to the interval [-1,1].

Finally, the processed candidate features are transformed into Map to uniform dimensions:

Where H is a nonlinear activation function, is a dimensional mapping matrix dedicated to this type of drone, which is a weight matrix with Z rows and D columns, where Z is The dimension of The dimension of is, b _z is the bias vector. We get the heterogeneous UAV The feature vector processed by a specific type of state encoder Update according to u ^type As the hidden state at the next moment.

By implementing this state encoder, we have completed the processing of the attributes and capability characteristics of heterogeneous drones in different dimensions. Feature vectors of the same dimension after processing Status characteristics of drones Stitch them together to get heterogeneous drones Observed features of the same dimension

3.2 Modeling of hybrid multi-UAV swarm structure

In each cluster, there is a CH communication drone that establishes communication connections with all other CMs member drones. There may also be communication connections between drones in different clusters. This is the composition of the communication network in the mixed multi-drone cluster, which is represented by the adjacency matrix E obtained by the perception process. To train the system as an "intelligent agent", we need to model the structure of the mixed multi-drone cluster. To this end, we need to solve the problem of integrating data such as heterogeneous drone types, feature vectors with the same dimension after processing, drone state characteristics, and communication network structure within the system.

This patent uses the Graph Attention Network (GAT) to design a hybrid multi-drone cluster structure. The input of GAT is the observation features of the same dimension after processing of each heterogeneous drone obtained in 3.1. The communication network structure characteristics E(t) of the hybrid multi-UAV swarm system are output as the observation O(t) of the entire hybrid multi-UAV swarm system after being processed by the graph attention network.

exist The number l of the UAV is obtained from the equation, and x _l (t) is the heterogeneous UAV numbered l. Observational characteristics of

The communication network structure E(t) of the system is used as the edge of the graph (i.e., the network structure feature), and the self-attention mechanism is used to perform weighted average aggregation on the observations of all heterogeneous drones to obtain the observations of the entire system.

The following are the specific implementation process and formula:

According to the communication network structure E(t) of the system at time t (i.e., the adjacency matrix representing the communication network structure of the system), e _ij (t) in the adjacency matrix E(t) is used as the edge of the graph attention network. If e _ij (t) is 1, there is a communication connection between the drones numbered i and j, otherwise it is 0. In particular, e _ii (t) is 1.

Then the attention weights between the drones in the system are calculated as follows:

Where N(t) is the number of non-isolated drones in the cluster structure at time t. W is the shared weight matrix, It is a feedforward neural network. The dot product is performed with the concatenated feature vector to obtain a real number. After the LeakeyReLU activation function, it is finally normalized to obtain the attention weight α _ij (t) at time t, which represents the relative importance of the heterogeneous UAV numbered j to the heterogeneous UAV numbered i.

The new observation feature vector is calculated based on the attention weights. The formula is:

m is the number of heterogeneous drones in the system. The weighted sum is used to obtain the new observation feature vector _Xi (t) of the heterogeneous drone numbered i. Finally, the observation features of all drones processed by the self-attention mechanism are concatenated to obtain the system observation X(t).

X(t)=(X ₁ (t)||X ₂ (t)||…||X _i (t)||…||X _m (t))

Where m is the total number of drones in the system. X(t) is processed by a fully connected layer and aggregated to obtain the final observation feature O(t) of the hybrid multi-drone cluster system.

3.3 Model Training

The reinforcement learning model is trained using the Actor-Critic architecture. The mixed multi-UAV swarm system is regarded as an intelligent agent, and the trained Actor strategy network is deployed for it. The Actor network will give the action that the system will take at the current moment, that is, the swarm structure adjustment strategy. The Critic value network is used for training to calculate the value estimate of the action.

Acotr Strategy Network

The Actor network is used to select actions based on the current system state O(t) Output a strategy The parameter θ of the strategy can be used to optimize the performance of the strategy through the gradient ascent algorithm, giving a better cluster structure adjustment strategy.

Critic Value Network

A strategy for systematically observing O(t) and Acotr network outputs in a mixed multi-UAV swarm based on neural network outputs The Critic network calculates the value function of the current state and outputs the action taken in the current state. expectations of return.

Policy gradient update

The PPO algorithm is used for training, and the policy gradient formula is given:

Where θ is the policy parameter, T represents the total system running time, r _t (θ) is the probability ratio of the updated policy to the old policy after training, and ε is set to limit the change of the probability ratio to ensure that the policy changes smoothly. is the advantage function, which is calculated based on the reward function and value function given when the decision model is constructed.

After completing the model training, this method can give the optimal cluster structure adjustment strategy according to different scenarios, balance the communication cost of the system, the status of heterogeneous drones, and the stability of the network structure to achieve the best optimization effect.

3.4 Cluster structure adjustment and network reconstruction

According to the cluster structure adjustment strategy action(t) given by the trained model, the utility value of each drone is obtained by weighted summing up the six factors of each drone. The detailed design of these six factors is given below:

Remaining energy percentage

Extract the residual energy property from the UAV observation _xi (t) obtained in 3.1 by dividing it by the energy of the initial state in _xi (0).

The degree of drones in the network structure

The speed of the drone divided by the total number of other drones in the system

The average distance between a drone and its neighbor’s drones (disutility)

N _i (t) is the set of numbers of neighboring drones at time t, and d _ij is the distance between two drones, which is calculated from the drone position coordinates in the drone observations x _i (t) and x _j (t).

The average motion state similarity between the drone and its neighboring drones

N _i (t) is the set of neighboring drone numbers at time t, speed is the drone speed attribute obtained from the drone observations x _i (t) and x _j (t) obtained in 3.1, θ is the direction within the state feature in the observation, and _Vij (t) is the similarity of the motion state between the two drones. is the average value of the similarity of the motion states of neighboring drones, and the variance is calculated to obtain the average motion state similarity factor ₄ .

Average communication connection holding time

N _i (t) is the set of numbers of neighboring drones at time t, where ct _ij (t) is the duration of the communication connection between two drones at time t.

The average communication cost between a drone and its neighboring drones

N _i (t) is the set of neighboring drone numbers at time t, cost _ij (t) is the calculation method of the communication cost in the perception process, and the corresponding drone is found by the numbers i and j of the heterogeneous drones. The communication cost between the two drones is calculated based on the type and distance of the heterogeneous drones, which reflects the multi-dimensional heterogeneity of the communication capability.

A clustering algorithm is used to cluster drones according to their utility values within the communication range and select CH communication drones to divide each cluster. Then a routing algorithm is used to build a communication network within the cluster through communication drones. All drones in the system will send out a signal with a cluster number and then receive signals from other drones. From the received signal, the drone will know the cluster where the signal source is located. For drones that can receive signals from drones in other clusters, they will be used as candidates for cross-cluster communication connections. Finally, the candidate with the lowest communication cost is selected as the cross-cluster communication connection between the two clusters and built. This completes the entire cluster structure adjustment process.

Claims (8)

1. A method for adaptively adjusting the structure of a hybrid multi-UAV cluster, characterized in that a state encoder for heterogeneous UAV types is first proposed to distinguish and process different types of UAVs, aggregate the attributes and capabilities of heterogeneous UAVs in different dimensions in a hybrid scenario, and obtain heterogeneous UAV features with the same dimension; then, a clustering algorithm based on reinforcement learning is used to fuse the processed multi-dimensional heterogeneous data, and a global adjustment strategy is given to adjust the cluster structure and reconstruct the communication network. 2. The method for adaptively adjusting the structure of a mixed multi-UAV cluster according to claim 1 is characterized in that: a mixed multi-UAV cluster system with uniform distribution of UAVs is adopted, and any UAV in the kth cluster C _k Represented by a triple: in Represents the heterogeneous drones in the kth cluster in the system Type of drone; Representative heterogeneous drones Attributes and capabilities of Representing heterogeneous drones Status characteristics; The hybrid multi-UAV cluster system consists of multiple heterogeneous UAV clusters C _k , each of which consists of heterogeneous UAVs _. composition, Where _Nk is the number of drones in the cluster; there are two types of connection relationships between heterogeneous drones in the system, one is the communication connection within the cluster, and the other is the communication connection across clusters; the communication connection between heterogeneous drones constitutes a mixed network structure within the system; both connections are dynamic and will change with the adjustment of the cluster structure; integrating the intra-cluster connection and cross-cluster connection in the heterogeneous multi-drone system, the mixed multi-drone cluster system MAS with N clusters is finally described as MAS = ( _n1 , _n2 ,…, _nN , E), where _nk = < _Ck , _CHk > represents the complete perception of the kth cluster, _Ck is the set of heterogeneous drones in the kth cluster, _CHk represents the current communication drone of the kth cluster, and E is the adjacency matrix representing the network structure within the mixed multi-drone cluster system; using the binary Represents the communication connection in the system; the communication cost of all communication connections when they are established and maintained at each subsequent moment is expressed as Represents the heterogeneous drones in the cluster C _k With heterogeneous drones in cluster C _l communication costs; Function c will be based on the drone and The communication cost is calculated based on the types of nodes and the distance between them. 3. The method for adaptively adjusting the structure of a hybrid multi-UAV swarm according to claim 2, characterized in that: The state S(t) represents the cluster structure state observation of the entire system at time interval t, including the state, distribution, communication network structure and CH communication drones of the heterogeneous drones in each cluster, which is expressed as: S(t)＝(S ₁ (t),S ₂ (t),…,S _k (t),…,S _N(t) (t),E(t)) Where N(t) is the number of heterogeneous drone clusters in the system at time t, E(t) is the communication network structure of the system at time t, and S _k (t) is the number of heterogeneous drones in the kth cluster. Observations at time interval t A collection of: action is the cluster structure adjustment strategy adopted by the system at time t, which is expressed as: Where λ _i (i∈[1,6]) represents the utility weight of the i-th factor of the drone, and λ _i satisfies the following constraints; λ ₁ +λ ₂ +λ ₃ +λ ₄ +λ ₅ +λ ₆ =1 Reward r(t) is the reward for the system taking an action at time t and state S(t) The global reward obtained by adjusting the cluster structure is expressed as: λ is the weight, _rs is the reward for cluster structure stability, and _rc is the negative reward caused by energy consumption; r _s =α ₁ r _ATT +α ₂ r _CTD +α ₃ r _SMD +α ₄ r _CTC r _c =-(β ₁ r _build +β ₂ r _maintenance +β ₃ r _isolated +β ₄ r _fly ) Among them, α and β are weights, r _ATT represents the average time that the drones stay in the current cluster; r _CTD represents the tightness of the network structure between drones in each cluster; r _SMD represents the similarity of the motion states between drones in each cluster; r _CTC represents the average time that the communication connection between drones in each cluster is maintained; r _build represents the communication cost of building a new communication connection caused by the cluster structure adjustment; r _maintenance represents the maintenance cost of the communication network in the system from the previous moment to this moment; r _isolated represents the additional communication loss caused by the isolated clusters in all clusters; r _fly represents the flight energy consumption of all drones in the system at the current moment; According to the definition of Markov decision, the strategy of each cluster should meet the optimization condition, that is, the strategy of the system should maximize its reward function r(t), which is: in is the action space that the system can take, which is the set of utility weights of all possible actions. 4. A hybrid multi-UAV cluster structure adaptive adjustment method according to claim 1, characterized in that: a state encoder for heterogeneous UAV types is designed using GRU, and the input of the state encoder is the type of the heterogeneous UAV Attributes and capability characteristics of heterogeneous drones in different dimensions The output is a feature vector of uniform dimension Remember to input the drone Type number For type, the drone stored in GRU Historical status is a vector representing the drone historical features at the last moment; in dealing with When Add to the input data; The reset gate r ^type in GRU controls whether to ignore the hidden state of GRU The formula is: Type is drone The drone type number, and is the weight matrix of the input features and hidden states for r ^type in the GRU network that processes the type type drone state. is the bias vector, σ is the sigmoid function; The update gate u ^type in GRU is used to control the GRU to update the hidden state The degree of, its formula is: in and is the weight matrix of input features and hidden states for u ^type in the GRU network that processes the type type drone state. is the bias vector and σ is the sigmoid function. 5. A hybrid multi-UAV cluster structure adaptive adjustment method according to claim 4, characterized in that: the candidate features at the current moment are calculated according to r ^type and u ^type in Represents the multiplication of each element of the reset gate vector and the hidden state vector, and is the input and hidden state pair in the GRU network that processes the type type drone state The weight matrix, is the bias vector, tanh represents the hyperbolic tangent function, which maps the input value to the interval [-1,1]; through a fully connected layer, the processed candidate features are Map to uniform dimensions: Where H is a nonlinear activation function, is a dimensional mapping matrix dedicated to this type of drone, which is a weight matrix with Z rows and D columns, where Z is The dimension of dimension, b _z is the bias vector; heterogeneous UAVs are obtained The feature vector processed by a specific type of state encoder Type of drone Feature vectors of the same dimension after processing Status characteristics of drones Put them together to get the heterogeneous drone numbered l Observed features of the same dimension 6. A hybrid multi-UAV cluster structure adaptive adjustment method according to claim 5, characterized in that: according to the communication network structure E(t) of the system at time t, that is, the adjacency matrix representing the communication network structure of the system, the e _ij (t) in the adjacency matrix E(t) is used as the edge of the graph attention network, and the attention weights between the UAVs in the system are calculated, and the formula is: Where N(t) is the number of non-isolated drones in the cluster structure at time t, W is the shared weight matrix, is a feedforward neural network, which performs dot product with the concatenated feature vector to obtain a real number, passes through the LeakeyReLU activation function, and is finally normalized to obtain the attention weight α _ij (t) at time t, which represents the relative importance of the heterogeneous UAV numbered j to the heterogeneous UAV numbered i; The new observation feature vector is calculated based on the attention weights. The formula is: m is the number of heterogeneous drones in the system; weighted summation is performed to obtain the new observation feature vector _Xi (t) of the heterogeneous drone numbered i, and finally the observation features of all drones processed by the self-attention mechanism are concatenated to obtain the system observation X(t) X(t)=(X ₁ (t)||X ₂ (t)||…||X _i (t)||…||X _m (t)) X(t) is processed by a fully connected layer and aggregated to obtain the final observation features O(t) of the hybrid multi-UAV swarm system. 7. A method for adaptively adjusting the structure of a hybrid multi-UAV swarm according to claim 6, characterized in that: an Actor-Critic architecture is used to train a reinforcement learning model, the hybrid multi-UAV swarm system is regarded as an intelligent agent, and a trained Actor strategy network is deployed for it, and the Actor network is used to select actions according to the current system state O(t) Output a strategy The parameter θ of the strategy is optimized through the gradient ascent algorithm to give a better cluster structure adjustment strategy; according to the mixed multi-UAV cluster output by the neural network, the strategy output by O(t) and Acotr network is systematically observed. The Critic network calculates the value function of the current state and outputs the action taken in the current state. The expected return is then calculated, and the advantage function is updated with the policy gradient and value function. 8. The method for adaptively adjusting the structure of a hybrid multi-UAV swarm according to claim 7, characterized in that: according to the action taken by the system That is, the utility weight of cluster structure adjustment, and the utility is calculated by weighted summing up the various factors of the drone; there are six utility factors of the drone: the remaining energy percentage, the degree of the drone in the network structure, the average distance between the drone and other drones, the average motion state similarity between the drone and the neighboring drones, the average maintenance time of the communication connection, and the average communication cost between the drone and the neighboring drones; the clustering algorithm is used to cluster the drones according to their utility values within the communication range and select the CH communication drones to divide the clusters, and then the routing algorithm is used to build the communication network within the cluster through the communication drone CH to complete the cluster structure adjustment process.