CN110377049B - Brain-computer interface-based unmanned aerial vehicle cluster formation reconfiguration control method - Google Patents

Abstract

The invention relates to the fields of data analysis, brain-computer interfaces, human-computer interaction, software development and the like, and aims to improve the control effect on the formation of unmanned aerial vehicle clusters and enable operators to obtain more convenient and efficient operation feeling. An off-line training step: initializing a motor imagery training system; training the mixed deep neural network by adopting a back propagation algorithm through comparison of the classification value of the neural network and the label value, and determining a network weight; an online control step: preprocessing, extracting signal characteristics and classifying by using a mixed deep neural network based on a deep convolutional network and a deep long-short term memory network; and generating a control command according to the output classification result, and controlling the reconstruction of the virtual unmanned aerial vehicle cluster formation. The invention is mainly applied to the design and manufacture occasions of unmanned aerial vehicles.

Description

Formation reconfiguration control method of UAV swarm formation based on brain-computer interface

technical field

The invention relates to the fields of data analysis, brain-computer interface, human-computer interaction, software development and the like, in particular to a control method for formation reconfiguration of unmanned aerial vehicles based on a brain-computer interface.

Background technique

The brain-computer interface (BCI) system provides an emerging human-computer interaction method, which can control other devices by extracting the EEG signals of the operator and detecting the effective information in it. Among the many brain-computer interface paradigms, P300, steady-state visual potential (SSVEP), and motor imagery are the most popular research fields. Among them, motor imagery is the only brain-computer interface paradigm that is spontaneous and does not require external stimulation.

Motor imagery refers to the brain only has the intention of limb movement but does not actually execute it, reflecting people's expectations of the action and the rehearsal of the real action that will occur. When imagining a specific movement scenario, the brain produces continuous EEG signals. The EEG features extracted from this signal are related to the initial thinking activity of the experimenter, so that the signal can be converted into control instructions for external equipment.

As a type of machine learning, deep learning has flourished in the fields of computer vision, speech recognition, and natural language processing. In the era of big data, a large number of motor imagery datasets can be obtained through various channels. Therefore, deep learning methods can better learn and classify motor imagery features in a large amount of EEG data. As a widely used technology, deep convolutional network (CNN) can fully mine spatial features in EEG data; deep long short-term memory network (LSTM) is a temporal recurrent neural network, which is very suitable for processing and classifying time series signals. , the use of long short-term memory network can well extract the temporal features in EEG data. Therefore, a hybrid deep neural network based on deep convolutional network and deep long-term short-term memory network is constructed to conduct supervised learning of EEG signals, and fully excavate the motor imagery features in space and time, which has good real-time and accuracy. .

In the current field of aerospace research, multi-aircraft formation and human-machine collaboration have become the current research trend, and new requirements have also been put forward for the control methods of UAVs. The traditional single-plane flight control equipment can no longer meet the control needs of the current UAV swarms, so it is urgent to develop new control methods. Introducing BCI technology into the aerospace field, UAV pilots can not only rely on traditional flight control equipment to control the position of UAV swarms, but also use ideas to reconfigure and control the formation of UAV swarms, which greatly improves the performance of operators. Ability to control drone swarms.

SUMMARY OF THE INVENTION

In order to overcome the deficiencies of the prior art, the present invention aims to propose a brain-controlled UAV swarm formation reconstruction control method, which can enable the operator to control the UAV swarm through the electroencephalogram signal based on motor imagery, and change into the desired shape. It can improve the control effect of the UAV swarm formation, so that the operator can obtain a more convenient and efficient control experience. To this end, the technical solution adopted by the present invention is that the UAV swarm formation reconstruction control method based on the brain-computer interface includes an offline training step and an online training step:

Offline training steps: S1, initialization of the motor imagery training system; S2, start the interactive interface, the interactive interface randomly displays arrows pointing up, down, left and right; S3, the operator imagines the tongue, foot, left hand, For the movement of the right hand, the EEG signal of the operator is collected through the electrode cap; S4, processing the EEG signal, including: preprocessing, signal feature extraction and using a hybrid deep neural network based on a deep convolutional network and a deep long and short-term memory network. Classification; S5. By comparing the classification value of the neural network with the label value, the back-propagation algorithm is used to train the hybrid deep neural network to determine the network weight;

Online control steps: S6. Start the virtual UAV swarm formation software and enter the UAV swarm formation control interface; S7. The operator imagines the tongue, feet, left hand, and right hand according to the desired UAV swarm formation. At the same time, the electrode cap collects the EEG signal of the operator; S8. After the EEG signal is collected, process the collected EEG signal, including: preprocessing, signal feature extraction and utilization based on deep convolutional network and depth The hybrid deep neural network of the short-term memory network is used for classification; S9, a control command is generated according to the output classification result to control the formation reconstruction of the virtual UAV swarm.

Specifically, 1) EEG signal preprocessing, including:

S10, down-sampling the EEG signal to obtain a 250Hz EEG signal; S11, perform 50Hz power frequency filtering on the collected EEG signal; S12, use a time window to segment the EEG signal time series; S13, The EEG signal is filtered by a filter bank;

2) EEG feature extraction, including:

The one-to-many-common space mode method OVR-CSP is used to extract the features of the EEG signals obtained by S13. The one-to-many-common space mode method steps include:

S14, respectively, for each type of motor imagery signal, obtain its common spatial mode filtering weight W _j relative to other signals:

Among them, C _j represents the covariance matrix of this type of motor imagery signal, E _j represents the diagonal matrix containing the eigenvalues of C _j , W _j represents the common spatial mode filtering weight of this type of motor imagery signal relative to other signals, j=1 , 2, 3, and 4 represent four types of motor imagery signals, respectively;

按顺序结合

获得

S15. Extract the first two columns and the last two columns of W _j respectively and combine them into a new matrix

Combine in order

get

S16. Perform one-to-many-common spatial mode filtering on the EEG signal obtained in S13:

Among them, X represents the EEG signal obtained in S13, and Z represents the signal after one-to-many-common spatial mode filtering;

S17, perform feature extraction on the signal Z obtained in S13;

Among them, diag( ) is the diagonal element of the matrix, and tr( ) is the trace of the matrix;

3) Using a deep convolutional neural network to perform spatial feature learning on the features obtained in S17, the method steps include:

S18. The deep convolutional network contains multiple hidden layers, each hidden layer is composed of a convolutional base layer and a pooling layer, where the convolutional layer is expressed as:

hc _l =R(conv(W _l ,x _l )+b _l )

Among them, x _l and hcl represent the input and output of the first convolutional layer _{, respectively, W l and b l represent the} weight and bias of the first convolutional layer, respectively, conv( ) represents the convolution operation, and R represents the layer activation function;

S19. After each convolutional layer is a pooling layer;

S20. Convert the output of the deep convolutional neural network into a 1-dimensional vector form;

4) using a deep long short-term memory network to perform temporal feature learning on the features obtained in S20 of multiple time windows, and the method steps include:

S21. The deep deep long-term and short-term memory network is composed of multiple long-term and short-term memory network cells in series;

S22. The long short-term memory network cell is composed of a forgetting gate, an input gate, and an output gate;

S23. The forgetting gate determines the amount of information discarded from the long-term and short-term memory network cells. The gate outputs a value from 0 to 1, where 1 represents complete retention, and 0 represents complete discard:

分别表示权值和偏置信息，σ为Sigmoid函数；Among them, hl _{l, t-1} represents the long-term and short-term memory network cell output of the previous time window, x _{l, t} represents the input of the current cell, l represents the l-th hidden layer, t represents the t-th time window, W _l ^f and

respectively represent the weight and bias information, and σ is the Sigmoid function;

S24. The input gate determines the amount of new information to be updated to the long short-term memory network cells. First, decide which information needs to be updated; second, calculate the alternative update content; finally, use the alternative update content to update the cell state:

分别表示权值和偏置信息，i_l,t表示更新信息的数量，

表示备选的更新内容，C_l,t表示当前长短期记忆网络细胞的状态；in

represent the weight and bias information, respectively, i _{l, t} represent the number of update information,

represents the alternative update content, C _{l, t} represents the current state of the long-term and short-term memory network cells;

S25. The output gate is to process the cell state of the long short-term memory network to determine the output of the cell

hl _l,t =o _l,t ×tanh(C _l,t )

分别表示权值和偏置信息。where hl _l,t is the output of the long short-term memory network cell, W _l ^o and

represent the weight and bias information, respectively.

In the offline process, it is necessary to train the weights and biases of the hybrid deep neural network, and the method steps include:

S26, using the Softmax function to calculate the probability distribution of different types of EEG signals on the output of S25 in step 4

Among them, m represents the EEG category index of the output y, and T represents the total number of EEG signal categories;

S27. Use the cross-entropy function to calculate the probability distribution distance between the predicted classification of the hybrid deep neural network and the real EEG signal label

Among them, y _p is the predicted classification result of the hybrid deep neural network, and _yl is the label value of the real EEG signal;

S28, using the back propagation algorithm to update the weights and biases of the deep neural network to reduce the value of the cross-entropy function.

In the online process, the UAV formation reconfiguration adopts a fully distributed formation reconfiguration controller to control the UAV formation:

S29. Define the formation position error expression e _Pi :

Among them, P ₀ is the position of the virtual leader unmanned helicopter, c _i , c _j are the expected formation positions of the drones i and j relative to the leader, respectively;

S30. Design the following outer ring formation controller U _1i (t), so that the formation errors e _Pi and e _Vi converge to a small neighborhood of zero within a limited time, and collisions between drones can be avoided

Among them, the velocity tracking error e _Vi and the formation reconstruction error σ _Pi are respectively expressed as:

更新规则为：adaptive gain

The update rule is:

Among them, the parameter value range is a>0, b>0, c>0, β _i >0, λ ₁ >0, λ ₂ >0, λ ₃ >0, and F _1i is the neural network learning function.

S31. The collision avoidance potential energy function between UAVs i and j is designed as:

Among them, the relative distance is defined as d _ij =||P _i -P _j ||, r _a is the safe collision avoidance radius of the UAV, 0<ε _a <1 is a very small positive number, so ln(1/ ε _a )≥1, the parameter value range is η _j >0, l ₁ >0, the update rule of ρ _a is:

The characteristics and beneficial effects of the present invention are:

The brain-controlled UAV swarm formation reconstruction technology combines the advantages of the brain-computer interface technology and the UAV swarm formation control technology, which can simplify the UAV formation control instructions, increase the human-computer interaction method, and enhance the human-level focus on the UAV formation. The invention not only effectively improves the theoretical research level of the brain-computer interaction control technology, but also lays a good theoretical and technical foundation for the research and development of the brain-UAV interactive control system in the future.

Description of drawings:

Figure 1. Flow chart of the formation control reconstruction method of brain-controlled UAV swarms.

Figure 2 Schematic diagram of the placement position of the 64-lead electrode cap.

Figure 3. Schematic diagram of visual signal stimuli.

Figure 4 Schematic diagram of the hybrid deep neural network architecture.

Figure 5 Schematic diagram of a three-layer deep convolutional neural network.

Figure 6 is a schematic diagram of a deep long-term and short-term memory network, in the figure: a. The schematic diagram of the 3-layer long-term and short-term memory network; b. The schematic diagram of the long-term and short-term memory network cells.

Figure 7 Schematic diagram of the control interface for the formation reconfiguration of the UAV swarm.

Figure 8. Reconstruction rendering of brain-controlled UAV formation.

Figure 9. The effect of the combination of brain-controlled UAV formation reconstruction and VR.

Detailed ways

The technical scheme of the present invention is as follows: a brain-controlled UAV swarm formation reconstruction control method includes an offline training step and an online training step:

Offline training steps: S1, initialization of the motor imagery training system; S2, start the interactive interface, the interactive interface randomly displays arrows pointing up, down, left and right; S3, the operator imagines the tongue, foot, left hand, For the movement of the right hand, the EEG signal of the operator is collected through the electrode cap; S4, processing the EEG signal, including: preprocessing, signal feature extraction and using a hybrid deep neural network based on a deep convolutional network and a deep long and short-term memory network. Classification; S5. By comparing the classification value of the neural network with the label value, a back-propagation algorithm is used to train the hybrid deep neural network, and the network weight is determined.

Online control steps: S6. Start the virtual UAV swarm formation software and enter the UAV swarm formation control interface; S7. The operator imagines the tongue, feet, left hand, and right hand according to the desired UAV swarm formation. At the same time, the electrode cap collects the EEG signal of the operator; S8. After the EEG signal is collected, process the collected EEG signal, including: preprocessing, signal feature extraction and utilization based on deep convolutional network and depth The hybrid deep neural network of the short-term memory network is used for classification; S9, a control command is generated according to the output classification result to control the formation reconstruction of the virtual UAV swarm.

The EEG signal preprocessing, signal feature extraction and utilization of the hybrid deep neural network technology based on the deep convolution network and the deep long short-term memory network mainly include the following steps:

1) EEG signal preprocessing, including:

S10. Perform down-sampling processing on the EEG signal to obtain an EEG signal of 250 Hz; S11. Perform a power frequency filter of 50 Hz on the collected EEG signal; S12. Use a time window to segment the EEG signal time series (recommended time The window is 0.2s); S13, use a filter bank to filter the EEG signal (recommended filter frequencies are: 4-8Hz, 6-10Hz, ..., 36-40Hz; it is recommended to use a Chebyshev type 3 filter for filter).

2) EEG feature extraction, including:

The one-to-many-common space pattern method (OVR-CSP) is used to extract the features of the EEG signals obtained in S13. The one-to-many-common space pattern method steps include:

S14, respectively, for each type of motor imagery signal, obtain its common spatial mode filtering weight W _j relative to other signals:

Among them, C _j represents the covariance matrix of this type of motor imagery signal, E _j represents the diagonal matrix containing the eigenvalues of C _j , W _j represents the common spatial mode filtering weight of this type of motor imagery signal relative to other signals, j=1 , 2, 3, and 4 represent four types of motor imagery signals, respectively;

按顺序结合

获得

S15. Extract the first two columns and the last two columns of W _j respectively and combine them into a new matrix

Combine in order

get

S16. Perform one-to-many-common spatial mode filtering on the EEG signal obtained in S13:

Among them, X represents the EEG signal obtained in S13, and Z represents the signal after one-to-many-common spatial mode filtering;

S17, perform feature extraction on the signal Z obtained in S13;

where diag(·) is the diagonal element of the matrix, and tr(·) is the trace of the matrix.

3) Using a deep convolutional neural network to perform spatial feature learning on the features obtained in S17, the method steps include:

S18. The deep convolutional network contains multiple hidden layers, each hidden layer is composed of a convolutional base layer and a pooling layer, where the convolutional layer is expressed as:

hc _l =R(conv(W _l ,x _l )+b _l )

Among them, x _l and hcl represent the input and output of the first convolutional layer _{, respectively, W l and b l represent the} weight and bias of the first convolutional layer, respectively, conv( ) represents the convolution operation, and R represents the The activation function of the layer (recommended to use the RELU function), the RELU function is expressed as:

RULA(a)=max(0,a)

S19. After each convolutional layer is a pooling layer. The pooling layer is used to compress the input features. On the one hand, the computational complexity of the network is reduced. On the other hand, the features are compressed and extracted to extract the main features (pooling). layer suggests a max pooling function).

S20. Convert the output of the deep convolutional neural network into a 1-dimensional vector form.

4) using a deep long short-term memory network to perform temporal feature learning on the features obtained in S20 of multiple time windows, and the method steps include:

S21. The deep deep long-term and short-term memory network is composed of multiple long-term and short-term memory network cells in series;

S22. The long short-term memory network cell is composed of a forgetting gate, an input gate, and an output gate;

S23. The forgetting gate determines the amount of information discarded from the long-term and short-term memory network cells. The gate outputs a value from 0 to 1, where 1 represents complete retention, and 0 represents complete discard:

分别表示权值和偏置信息，σ为Sigmoid函数。Among them, hl _{l, t-1} represents the long-term and short-term memory network cell output of the previous time window, x _{l, t} represents the input of the current cell, l represents the l-th hidden layer, t represents the t-th time window, W _l ^f and

represent the weight and bias information, respectively, and σ is the Sigmoid function.

S24. The input gate determines the amount of new information to be updated to the long short-term memory network cells. First, decide what information needs to be updated. Second, the alternative update content is calculated. Finally, the cell state is updated with the alternative update content.

分别表示权值和偏置信息，i_l,t表示更新信息的数量，

表示备选的更新内容，C_l,t表示当前长短期记忆网络细胞的状态。in

represent the weight and bias information, respectively, i _{l, t} represent the number of update information,

represents the alternative update content, and C _{l, t} represents the current state of the long short-term memory network cell.

S25. The output gate is to process the cell state of the long short-term memory network to determine the output of the cell

hl _l,t =o _l,t ×tanh(C _l,t )

分别表示权值和偏置信息。where hl _l,t is the output of the long short-term memory network cell, W _l ^o and

represent the weight and bias information, respectively.

5) In the offline process, it is necessary to train the weights and biases of the hybrid deep neural network, and the method steps include:

S26, using the Softmax function to calculate the probability distribution of different types of EEG signals on the output of S25 in step 4

Among them, m represents the EEG category index of the output y, and T represents the total number of EEG signal categories.

S27. Use the cross-entropy function to calculate the probability distribution distance between the predicted classification of the hybrid deep neural network and the real EEG signal label

Among them, y _p is the predicted classification result of the hybrid deep neural network, and _yl is the label value of the real EEG signal.

S28, using the back propagation algorithm to update the weights and biases of the deep neural network to reduce the value of the cross-entropy function.

6) In the online process, the UAV formation reconfiguration adopts a fully distributed formation reconfiguration controller to control the UAV formation.

S29. Define the formation position error expression e _Pi :

Among them, P ₀ is the position of the virtual leader unmanned helicopter, c _i , c _j are the expected formation positions of the drones i and j relative to the leader, respectively.

S30. Design the following outer ring formation controller U _1i (t), so that the formation errors e _Pi and e _Vi converge to a small neighborhood of zero within a limited time, and collisions between drones can be avoided

Among them, the velocity tracking error e _Vi and the formation reconstruction error σ _Pi are respectively expressed as:

更新规则为：adaptive gain

The update rule is:

Among them, the parameter value range is a>0, b>0, c>0, β _i >0, λ ₁ >0, λ ₂ >0, λ ₃ >0, and F _1i is the neural network learning function.

S31. The collision avoidance potential energy function between UAVs i and j is designed as:

Among them, the relative distance is defined as d _ij =||P _i -P _j ||, r _a is the safe collision avoidance radius of the UAV, 0<ε _a <1 is a very small positive number, so ln(1/ ε _a )≥1, the parameter value range is η _j >0, l ₁ >0, the update rule of ρ _a is:

Social benefits: This invention is of great significance to the research and development of brain-computer interface technology and the control method of formation reconfiguration of UAV swarms. This invention is at the international advanced level, and it can be used as a new method for UAV formation control, which in turn helps to promote the development of various interaction methods and technologies for UAVs. This technology not only effectively improves the theoretical research level of brain-computer interaction control technology, but also lays a good theoretical and technical foundation for the research and development of brain-UAV interactive control system in the future.

Economic benefits: Brain-controlled UAV swarm formation reconstruction technology combines the advantages of brain-computer interface technology and UAV swarm formation control technology, which can simplify UAV formation control instructions, increase human-computer interaction, and enhance human-to-unmanned The control ability of aircraft formation reconfiguration has high economic value and has great potential applications not only in the field of commercial performances but also in the military. The brain-controlled UAV swarm formation technology can not only provide new control ideas for the development of future UAV formation flight control systems; at the same time, it can be applied to the game field as a new interactive method, which has great economic value.

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Referring to Figure 1, the flow chart of the formation reconfiguration control method of the brain-controlled UAV swarm. The system is mainly composed of offline training system and online control system. In the offline training system, electrode caps are used to collect labeled motor imagery EEG data, and the weights and biases of the hybrid deep neural network are trained in a supervised learning manner. In the online control system, the motor imagery EEG signals of the testers are collected in real time, the EEG data is segmented by a time window of 0.2s, and the data in each time window is classified by the trained hybrid deep neural network. The output results of the n time windows are consistent (n=3 is recommended), and the UAV swarm performs the formation transformation corresponding to the result.

Referring to FIG. 2 , a schematic diagram of the placement position of the lead 64 of the electrode cap. The system needs to collect EEG signals from different parts of the experimenter. According to the basic requirements of motor imagery EEG signal analysis, at least the leads C3, C4, and Cz need to be connected. The recommended leads are: FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT7, FT8, C5 , C3, C1, Cz, C2, C4, C6, T7, T8, CP5, CP3, CP1, CP2, CP4, CP6, TP7, TP8.

See Figure 3, a schematic diagram of visual signal stimulation. In the offline training process, the EEG data of different motor imagery parts of the experimenter needs to be collected. According to the red arrows in the figure, the experimenter needs to imagine the movement of different parts of their body. The left arrow represents the imagined left hand movement, the right arrow represents the imagined right hand movement, the upper arrow represents the imagined tongue movement, and the down arrow represents the imagined foot movement.

See Figure 4, a schematic diagram of the hybrid deep neural network architecture. For the collected EEG data of the experimenter's motor imagery, the preprocessing operation is firstly performed, including: filtering the 50Hz power frequency in the signal with a notch filter; using a time window to segment the EEG data (the recommended time window is 0.2 s); use a filter bank to perform subband filtering on the EEG data of each time window (recommended filter frequencies are: 4-8Hz, 6-10Hz, ..., 36-40Hz; Chebyshev type 3 filtering is recommended filter); adopt the one-to-many-common spatial pattern method (OVR-CSP) to extract features from the filtered EEG signals for each time window and each subband. Secondly, a deep convolutional neural network (CNN) is used to learn and classify the spatial features of the preprocessed EEG signals. The spatial features are used for temporal feature learning and classification, and a classification result is output for the EEG signals of each time window.

See Figure 5, a schematic diagram of a three-layer deep convolutional neural network. In the figure, "C&P" represents convolution and pooling operations, and "reshape" represents matrix variable dimension processing. The deep convolutional neural network (CNN) is recommended to use 3 hidden layers. The recommended size of the convolution kernel of each hidden layer is: 3 × 3. It is recommended to use the Zero-padding strategy in each convolution process, that is, for each hidden layer The input of the layer is filled with 0 values around the input to ensure that the output after convolution is consistent with the input dimension. It is recommended that the pooling layer of the hidden layer adopts the maximum pooling function, and the recommended size of the pooling layer filter is: 2×2. In the last layer of the deep convolutional network, the data format is converted into a one-dimensional vector of size 1×64.

Referring to FIG. 6 , a schematic diagram of a deep long-term and short-term memory network, where a is a schematic diagram of a 3-layer long-term and short-term memory network; b is a schematic diagram of a long-term and short-term memory network cell. The EEG signal is a continuous signal, so the deep convolutional network features of each time window are used as the input of a deep long short-term memory network (LSTM). It is recommended to use 3 hidden layers for the deep long short-term memory network, and the processing process of each hidden layer is shown in Fig. 6b. For each time window, an expected classification is generated. In the offline training process, the expected classification is compared with the real label to update the weights and biases of the deep hybrid neural network; in the actual control process, three times are continuously read. The window is expected to be classified. If the classification results of the three time windows are consistent, the classification result will be output.

Referring to Figure 7, a schematic diagram of the control interface for the formation reconfiguration of the UAV swarm. The UAV swarm formation control software is made based on the Unity3D three-dimensional engine. Among them, the UAV swarm ocean flight scene is generated with the AQUAS Water tool, the ocean islands and reefs are made with the MapMagic tool, and the UGUI component is used to make the software interface for displaying no The current formation of the human-machine swarm is based on the UDP transport layer communication protocol to realize the network communication function, receive the recognition result of the EEG signal, and demonstrate the corresponding UAV swarm formation transformation demonstration. A total of four formation types can be selected on the control interface, among which, imagine the tongue movement to control the "V"-shaped formation change; imagine the foot movement to control the horizontal formation change; imagine the left hand movement to change the square formation; imagine the right hand movement to change the column formation.

Specific examples are given below:

1. System hardware and software configuration

According to the overall structure of the platform shown in Figure 1 of this section, the hardware configuration used in this example is shown in the following table:

name model computer CPU: Intel Core i7-6700K, Memory: 16G, Graphics Card: GTX1070 electrode cap Neuracle 64-lead wireless digital EEG acquisition system

The software implementation methods of this example include: the EEG analysis program is developed with MATLAB and Python; the interactive interface is based on the Unity 3d engine.

2. Experimental results

In this example, the UAV formation control experiment was carried out under the experimental platform system. Figure 8 shows the effect diagram of the formation reconstruction of the brain-controlled UAV formation. Figure 9 shows the combined effect of the brain-controlled drone formation and VR, so that the operator can enjoy an immersive experience. The idea control method for formation reconstruction of UAV swarm formation has achieved good interactive simulation effect, which verifies the feasibility of the present invention.

Claims (1)

1. An unmanned aerial vehicle cluster formation reconfiguration control method based on brain-computer interfaces is characterized by comprising an off-line training step and an on-line training step:

an off-line training step: s1, initializing a motor imagery training system; s2, starting an interactive interface, wherein the interactive interface randomly displays arrows pointing to the upper, lower, left and right directions; s3, the operator respectively imagines the movement of the tongue, the feet, the left hand and the right hand according to the direction of the arrow, and the electroencephalogram signals of the operator are collected through the electrode cap; s4, processing the electroencephalogram signals, including: preprocessing, extracting signal characteristics and classifying by using a mixed deep neural network based on a deep convolutional network and a deep long-short term memory network; s5, training the mixed deep neural network by adopting a back propagation algorithm through comparison of the classification value and the label value of the neural network, and determining a network weight;

an online control step: s6, starting virtual unmanned aerial vehicle cluster formation form software, and entering an unmanned aerial vehicle cluster formation control interface; s7, enabling operators to imagine the movements of the tongue, the feet, the left hand and the right hand respectively according to the expected unmanned aerial vehicle cluster formation, and meanwhile, collecting electroencephalogram signals of the operators by the electrode caps; s8, processing the acquired electroencephalogram signals after acquiring the electroencephalogram signals, and the processing method comprises the following steps: preprocessing, extracting signal characteristics and classifying by using a mixed deep neural network based on a deep convolutional network and a deep long-short term memory network; s9, generating a control command according to the output classification result, and controlling the reconstruction of the virtual unmanned aerial vehicle cluster formation;

specifically, 1) electroencephalogram signal preprocessing, comprising:

s10, performing down-sampling processing on the electroencephalogram signal to obtain the electroencephalogram signal of 250 Hz; s11, carrying out power frequency filtering of 50Hz on the acquired electroencephalogram signals; s12, segmenting the electroencephalogram signal time sequence by adopting a time window; s13, filtering the electroencephalogram signals by adopting a filter bank;

2) the extraction of the characteristics of the electroencephalogram signals comprises the following steps:

performing feature extraction on the electroencephalogram signal obtained by the S13 by adopting a one-to-many-public space mode method OVR-CSP, wherein the one-to-many-public space mode method comprises the following steps:

s14, respectively calculating the common spatial mode filtering weight W of each type of motor imagery signal relative to other signals_j：

Wherein, C_jCovariance matrix representing the motor imagery signal of this type, E_jRepresents that it contains C_jDiagonal array of eigenvalues, W_jRepresenting the common spatial mode filtering weight of the motor imagery signal relative to other signals, wherein j is 1,2,3 and 4 respectively represent four types of motor imagery signals;

s15, respectively extracting W_jThe first two columns and the second two columns are combined into a new matrix

Are combined in sequence

To obtain

S16, performing one-to-many common spatial mode filtering on the electroencephalogram signals obtained in the S13:

wherein, X represents the electroencephalogram signal obtained at S13, and Z represents the signal after one-to-many-common spatial mode filtering;

s17, extracting the characteristics of the signal Z obtained in S13;

wherein, diag (·) is a diagonal array element of the matrix, and tr (·) is a trace of the matrix;

3) and (3) performing spatial feature learning on the features obtained in the step S17 by adopting a deep convolutional neural network, wherein the method comprises the following steps:

s18, the deep convolutional network comprises a plurality of hidden layers, each hidden layer is composed of a volume base layer and a pooling layer, wherein the volume layer is represented as:

hc_l＝R(conv(W_l,x_l)+b_l)

wherein x is_lAnd hc and_lrespectively representing the input and output of the first convolution layer, W_lAnd b_lRespectively representing the weight and deviation of the convolution layer of the first layer, conv (-) represents convolution operation, and R represents the activation function of the layer;

s19, forming a pooling layer after each convolution layer;

s20, converting the output of the deep convolutional neural network into a 1-dimensional vector form;

4) performing time feature learning on the features obtained in S20 of a plurality of time windows by using a deep long short-term memory network, wherein the method comprises the following steps:

s21, the deep depth long and short term memory network is composed of a plurality of long and short term memory network cells which are connected in series;

s22, the long and short term memory network cell is composed of a forgetting gate, an input gate and an output gate;

s23, a forgetting gate determines the amount of information discarded from the long-short term memory network cell, the gate outputs a value of 0 to 1, 1 indicates complete retention, 0 indicates complete discard:

f_l,t＝σ(W_l ^f·[hl_l,t-1,x_l,t]+b_l ^f)

wherein hl is_l,t-1Cell output of long short term memory network representing previous time window, x_l,tRepresenting the input of the current cell, l representing the l hidden layer, t representing the t time window, W_l ^fAnd

respectively representing weight and bias information, wherein sigma is a Sigmoid function;

s24, determining the amount of new information to be updated for the long-term and short-term memory network cells by the input gate, and firstly, determining which information needs to be updated; secondly, calculating alternative updating contents; and finally, updating the cell state by adopting the alternative updating content:

wherein W_l ⁱ，W_l ^c，

Respectively representing weight and bias information, i_l,tIndicating the amount of the update information to be updated,

representing alternative updates, C_l,tRepresenting the current state of the long-short term memory network cells;

s25, the output gate is used for processing the cell state of the long-short term memory network and determining the output of the cell

hl_l,t＝o_l,t×tanh(C_l,t)

Wherein hl_l,tFor the output of long and short term memory network cells, W_l ^oAnd

respectively representing weight and bias information;

in the off-line process, the weight and deviation training of the mixed deep neural network is needed, and the method comprises the following steps:

s26, calculating probability distribution of different categories of electroencephalogram signals by adopting Softmax function to the output of the S25 in the step 4

Wherein m represents the electroencephalogram category index of the output y, and T represents the total number of electroencephalogram signal categories;

s27, calculating the probability distribution distance between the prediction classification of the mixed depth neural network and the real electroencephalogram signal label by adopting a cross entropy function

Wherein, y_pPredicting classification results for a hybrid deep neural network, y_lLabeling values for the real electroencephalogram signals;

s28, updating the weight and the deviation of the deep neural network by adopting a back propagation algorithm to reduce a cross entropy function value;

in the online process, the formation of the unmanned aerial vehicle adopts a completely distributed formation reconstruction controller to control the formation of the unmanned aerial vehicle:

s29, defining a formation position error expression e_Pi：

Wherein, P₀Position of a virtual leader unmanned helicopter, c_i,c_jExpected formation positions of unmanned planes i and j relative to leader respectively;

s30, designing an outer ring formation controller U as follows_1i(t) make the formation error e_PiAnd e_ViConverge to a very small neighborhood of zero in a limited time and avoid collisions between drones

Wherein the velocity tracking error e_ViAnd formation reconstruction error sigma_PiRespectively expressed as:

adaptive gain

The update rule is:

wherein the parameter value ranges are a is more than 0, b is more than 0, c is more than 0, beta_i＞0,λ₁＞0,λ₂＞0,λ₃＞0，F_1iLearning a function for the neural network;

the collision avoidance potential energy function between S31 and unmanned aerial vehicles i and j is designed as follows:

wherein the relative distance is defined as d_ij＝||P_i-P_j||，r_aIs the safe collision avoidance radius of the unmanned aerial vehicle, and is more than 0 and less than epsilon_a< 1 is a very small normal number, so ln (1/ε)_a) Greater than or equal to 1, parameter value rangeIs eta_j＞0,l₁＞0,ρ_aThe update rule is as follows:

Images