CN114312819A - Brain heuristic type automatic driving assistance system and method based on capsule neural network - Google Patents
Brain heuristic type automatic driving assistance system and method based on capsule neural network Download PDFInfo
- Publication number
- CN114312819A CN114312819A CN202210221486.6A CN202210221486A CN114312819A CN 114312819 A CN114312819 A CN 114312819A CN 202210221486 A CN202210221486 A CN 202210221486A CN 114312819 A CN114312819 A CN 114312819A
- Authority
- CN
- China
- Prior art keywords
- electroencephalogram
- neural network
- signals
- capsule
- brain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 239000002775 capsule Substances 0.000 title claims abstract description 121
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 65
- 210000004556 brain Anatomy 0.000 title claims abstract description 57
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000004458 analytical method Methods 0.000 claims abstract description 58
- 238000000605 extraction Methods 0.000 claims abstract description 26
- 239000000284 extract Substances 0.000 claims abstract description 16
- 238000001514 detection method Methods 0.000 claims abstract description 7
- 239000013598 vector Substances 0.000 claims description 30
- 238000007781 pre-processing Methods 0.000 claims description 29
- 238000004422 calculation algorithm Methods 0.000 claims description 27
- 238000012545 processing Methods 0.000 claims description 27
- 238000012706 support-vector machine Methods 0.000 claims description 25
- 230000006870 function Effects 0.000 claims description 19
- 238000013507 mapping Methods 0.000 claims description 16
- 238000006243 chemical reaction Methods 0.000 claims description 12
- 230000008447 perception Effects 0.000 claims description 11
- 230000033764 rhythmic process Effects 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 7
- 238000011176 pooling Methods 0.000 claims description 7
- 230000003595 spectral effect Effects 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 6
- 238000002790 cross-validation Methods 0.000 claims description 6
- 230000001419 dependent effect Effects 0.000 claims description 6
- 230000033001 locomotion Effects 0.000 claims description 6
- 238000003062 neural network model Methods 0.000 claims description 6
- 238000000513 principal component analysis Methods 0.000 claims description 6
- 238000007634 remodeling Methods 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 6
- 238000001228 spectrum Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000000354 decomposition reaction Methods 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 5
- 230000004927 fusion Effects 0.000 claims description 4
- 238000004088 simulation Methods 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 claims description 4
- 238000013145 classification model Methods 0.000 claims description 3
- 238000013499 data model Methods 0.000 claims description 3
- 210000002615 epidermis Anatomy 0.000 claims description 3
- 230000004807 localization Effects 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 230000001502 supplementing effect Effects 0.000 claims description 3
- 238000012937 correction Methods 0.000 abstract description 4
- 230000007547 defect Effects 0.000 abstract description 4
- 239000003795 chemical substances by application Substances 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 8
- 238000012549 training Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 6
- 230000018109 developmental process Effects 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000001149 cognitive effect Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 4
- 230000001953 sensory effect Effects 0.000 description 4
- 206010063385 Intellectualisation Diseases 0.000 description 3
- 230000019771 cognition Effects 0.000 description 3
- 210000000337 motor cortex Anatomy 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000004069 differentiation Effects 0.000 description 2
- 238000004070 electrodeposition Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000008450 motivation Effects 0.000 description 2
- 241001465805 Nymphalidae Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000010391 action planning Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 210000003169 central nervous system Anatomy 0.000 description 1
- 210000003710 cerebral cortex Anatomy 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000013210 evaluation model Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000001652 frontal lobe Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 210000000578 peripheral nerve Anatomy 0.000 description 1
- 230000035790 physiological processes and functions Effects 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000000284 resting effect Effects 0.000 description 1
- 210000004761 scalp Anatomy 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000000638 stimulation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Landscapes
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention discloses a brain heuristic type automatic driving auxiliary system and a method based on a capsule neural network, wherein the system comprises an electroencephalogram signal acquisition system: the system is used for acquiring electroencephalogram data of a user; the error-related negative potential analysis system uses a capsule neural network to realize the detection of the error-related negative potential ERN, extracts error-related negative potential signals in the brain of a driver according to the electroencephalogram data of a user, and outputs a mark signal; motor imagery analysis system: judging the driving intention of the brain of the user according to the electroencephalogram data of the user, and extracting instructions; an automatic navigation control system: the method is used for simulating the automatic driving scene of the vehicle, realizing the positioning, the drawing establishment and the path planning of an unknown scene, further calculating the variation of the output driving state through PID control, and controlling the updating of the navigation state of the vehicle. The invention assists the safety correction of the automatic driving system based on the feature extraction and analysis model of the electroencephalogram signal and the brain-like intelligent body model, and reduces accidents caused by complex environments or technical defects.
Description
Technical Field
The invention relates to the technical field of image processing and recognition methods, in particular to a brain heuristic type automatic driving assistance system and method based on a capsule neural network.
Background
The intelligent automobile is a strategic direction for the development of the global automobile industry and also an important component of future intelligent traffic. The development of intelligent automobiles is beneficial to accelerating the transformation and upgrading of the automobile industry, cultivating digital economy, and increasing new kinetic energy of the economy. The automatic driving technology can enhance the safety and flexibility of travel, promote the information construction of urban traffic, and meet the urgent needs of vehicle informatization and intellectualization. The intelligent level of the existing automobile is mostly L0-L3, and the existing automobile cannot reach L4 and L5 in a short time, and still has a large difference from the fully intelligent L5. Under the condition of lacking of manual intervention, the test vehicle cannot well process complicated and variable road conditions, and safety accidents are easy to occur.
A Brain-Computer Interface (BCI) is a communication device that directly connects the Brain to external devices. The brain-computer interface does not depend on the participation of peripheral nerves and muscles of the brain, and utilizes electroencephalograms (EEG) induced by the perception of the central nervous system or cognitive thinking activity to acquire post signals through an acquisition system and decode perception thinking intentions, so that brain post-movement information is converted into control signals, and the brain is communicated or controlled with external equipment. With the maturity of brain-computer interface technology, the intelligent system of the new mode that combines with it is constantly emerging, has included intelligent house, education science and technology, health medical treatment, entertainment control etc..
An effective automatic driving auxiliary intelligent system is developed based on electroencephalogram signal analysis, BCI and brain-like agent technologies are applied to the existing automatic driving field, and electroencephalogram signals play an intelligent guiding role in sudden conditions of a driving system and have important significance in development of vehicle intellectualization.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a brain heuristic type automatic driving assistance system and method based on a capsule neural network, which can assist the safety correction of the automatic driving system by fusing a brain-like intelligent body model and a feature extraction and analysis model based on electroencephalogram signals and reduce accidents caused by complex environments or technical defects.
In order to achieve the purpose, the invention designs a enbrain heuristic automatic driving assistance system based on a capsule neural network, which is characterized in that the system comprises an electroencephalogram signal acquisition system, an error-related negative potential analysis system, a motor imagery analysis system and an automatic navigation control system;
the electroencephalogram signal acquisition system comprises: the device is used for collecting electroencephalogram data of a user and sending the electroencephalogram data to an upper computer for processing;
the error-dependent negative potential analysis system: the capsule neural network is used for realizing the detection function of the error-related negative potential ERN, extracting error-related negative potential signals in the brain of a driver according to electroencephalogram data of a user, outputting a mark signal and reflecting the correctness of the current vehicle navigation state;
the motor imagery analysis system: judging the driving intention of the brain of the user according to the electroencephalogram data of the user, and extracting and classifying to generate a left or right instruction;
the automatic navigation control system: the method is realized based on an ROS simulation platform and used for simulating an automatic driving scene of a vehicle, realizing positioning, mapping and path planning of an unknown scene, receiving a mark signal output by the error-related negative potential analysis system and an instruction signal output by the motor imagery analysis system, further calculating the output driving state variable quantity through PID control, and controlling the updating of the vehicle navigation state.
Furthermore, the electroencephalogram signal acquisition system is composed of a signal acquisition unit and an electroencephalogram signal preprocessing unit, the signal acquisition unit acquires potential signals of specific points of the epidermis of the brain of a user, and the electroencephalogram signal preprocessing unit amplifies and filters the acquired electroencephalogram signals.
Furthermore, the signal acquisition unit acquires electroencephalogram signals through 32 EEG channels including FP 1-O2, signals of the channels including FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 are used for feature extraction and classification of an error correlation negative potential analysis system, and signals of the channels including C3, C4 and Cz are used for feature extraction of a motor imagery signal in a time domain, a frequency domain, a spatial domain and an energy domain.
Furthermore, the error-related negative potential analysis system comprises an ERN signal preprocessing unit and a capsule neural network unit, wherein the ERN signal preprocessing unit preprocesses electroencephalogram data, screening, re-referencing, filtering, independent principal component analysis and segmentation of electroencephalogram signals are completed, an electroencephalogram data time domain oscillogram output by the electroencephalogram signal acquisition system is converted into a time sequence of an electroencephalogram topographic map, a time window is selected, the electroencephalogram signals are converted into the electroencephalogram topographic map according to a set time interval, the capsule neural network unit classifies and extracts the electroencephalogram topographic map output by the ERN signal preprocessing unit, and two marking instructions of error reaction and no error reaction are output according to whether the ERN signals are extracted or not.
Furthermore, the network structure of the capsule neural network unit mainly comprises: 4 coiling layers, 1 average pooling layer, 1 remodeling layer, 1-2 capsule layers and 1 full-connection layer; the input of the convolutional layer is sample data, the convolutional layer preliminarily extracts local features in the electroencephalogram topological graph, the sample data are projected to different spaces as much as possible by using a convolutional kernel, the perception degree of the local features is increased, and the convolutional layer outputs a coding set obtained by projecting the sample data in different modes; the average pooling layer is used for reducing the complexity of data to be processed, further accelerating the convergence efficiency, reducing the operation overhead and improving the resistance of the network model to noise contained in the electroencephalogram topological graph; the remodeling layer is used for converting input data into a format which can be recognized by the capsule neural network; the capsule layer is presented in a capsule form and used for encapsulating information obtained after encoding the electroencephalogram topological graph, the different dimensions of the capsule encode angle and distance information among local features of the electroencephalogram topological graph, and along with the increase of the number of the capsule layers, the information encoded by the capsule is from the spatial position relation of the local features to the spatial position relation of global features, so that time sequence information of the electroencephalogram topological graph is included and encoded in the form of the spatial relation; the fully-connected layer projects the output of the capsule layer into two probability values through a matrix to be used as the prediction output of the model, and the class with larger prediction value is used as the final classification output of the capsule neural network model.
Furthermore, the motor imagery analysis system comprises a motor imagery signal preprocessing unit, a signal feature extraction unit and a signal classification and identification unit; the motor imagery signal preprocessing unit completes filtering, segmentation, baseline calibration, re-reference and independent principal component analysis of the electroencephalogram signal to obtain pure electroencephalogram data; the signal feature extraction unit is used for extracting the features of signals, decomposing the trained electroencephalogram data by utilizing a wavelet packet decomposition algorithm, extracting 8-12 Hz mu rhythms and 18-30 Hz beta rhythms, and processing corresponding frequency bands of the extracted rhythms by utilizing a common space mode CSP algorithm to extract and obtain highly differentiated feature vectors; the signal classification and identification unit receives the feature vector signals extracted by the signal feature extraction unit, classifies the signals based on an SVM vector machine, and classifies the signals to obtain three instructions of left and right directions or current situations.
Furthermore, the automatic navigation control system comprises a real-time positioning unit, a mapping unit, a path planning unit and a brain-like processing unit; the real-time positioning unit is used for performing positioning calculation through a data model according to data received by the sensor, outputting a conversion relation between a world coordinate system and a odometer coordinate system according to information input by the vision sensor, the laser radar and the gyroscope, and deducing the conversion relation between a vehicle base coordinate system and the world coordinate system to realize a positioning function; the map building unit realizes the scanning of surrounding environment information by adopting a mapping tool algorithm, builds an output environment map, and builds a probability-based two-dimensional grid map based on the visual sensor information, the gyroscope information and the odometer information; the path planning unit receives the map, the sensor data flow and the odometer information generated by the map building unit by adopting a navigation toolkit set, outputs speed and angular speed instructions to the vehicle control chassis, further controls the driving state and realizes the functions of path planning and obstacle avoidance; the brain-like processing unit receives the mark signal output by the error-related negative potential analysis system and the instruction signal output by the motor imagery analysis system, and deduces to form a complete multilayer experience map by receiving and memorizing the local acquisition information of the sensor, and the map is mapped to a real scene to correct the driving state in real time.
The invention also provides a brain heuristic automatic driving assistance method based on the capsule neural network, which is realized based on the brain heuristic automatic driving assistance system based on the capsule neural network, and the method comprises the following steps:
1) acquiring electroencephalogram data: the electroencephalogram signal acquisition system acquires electroencephalogram data of a user and sends the electroencephalogram data to an upper computer for processing; an electrode cap based on an international 10-20 system electrode placement method carries out user electroencephalogram signal acquisition through 32 EEG channels including FP 1-O2, and a band-pass filter with the frequency of 0.1-70 Hz and a 50Hz notch filter are used for removing power frequency interference and other noises so as to ensure the quality of electroencephalogram signals;
2) error-dependent negative potential analysis: the error-related negative potential analysis system utilizes signals of channels of capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 to extract and classify features, extracts error-related negative potential signals in the brain of a driver according to electroencephalogram data of the user, analyzes and judges the generation of error-related negative potentials in real time, and outputs a mark signal;
3) and (3) outputting a motion direction instruction: the motor imagery analysis system selects signals of electrodes C3, C4 and Cz channels by using a wavelet packet and common space mode fusion algorithm to perform characteristic extraction of a time domain, a frequency domain, a space domain and an energy domain of the motor imagery signals, classifies the signals by using an SVM (support vector machine), obtains three instructions of left and right or keeping the current situation by classification, and outputs instruction signals;
4) controlling vehicle navigation: the automatic navigation control system simulates an automatic driving scene of a vehicle, realizes positioning, mapping and path planning of an unknown scene, receives the sign signal output in the step 2) and the instruction signal output in the step 3), further calculates the variation of the output driving state through PID control, and controls the updating of the navigation state of the vehicle.
Preferably, the specific steps of the capsule neural network in the step 2) for feature extraction and classification include:
21) collecting electroencephalogram signals of a plurality of channels, and preprocessing original signals;
22) intercepting a signal waveform according to a fixed time interval, and calculating corresponding power spectral density;
23) normalizing and mapping the power spectral density into a color spectrum, combining the spatial localization of the channel in the brain, supplementing the values of the rest positions through a spatial interpolation algorithm to obtain a sequence of an electroencephalogram topological graph changing along with time, and completely arranging the electroencephalogram topological graph in one graph according to a time sequence to be used as sample data;
24) inputting the sample data into a capsule neural network for operation;
25) performing three iterations by using a dynamic routing algorithm and updating the weight by implementing a capsule neural network;
26) and outputting the class with the maximum capsule neural network output value as a prediction result for judging whether the error related potential is generated.
Preferably, the SVM support vector machine in step 3) trains through tens of existing feature vector sets of motor imagery on the left and right, trains an SVM binary model by using feature vectors extracted from electroencephalogram data, searches for hyper-parameters (c, g), i.e., a set of optimal penalty parameters c and RBF kernel function width, in a two-dimensional grid form through cross validation, calculates an optimal value of cross validation accuracy of each set of parameters, maximizes the classification accuracy of the SVM classification model by using the highest accuracy set (c, g) as an optimal parameter, outputs 0 representing a left instruction, outputs 1 representing a right instruction, and outputs no change instruction if not.
Motor imagery and error-related negative potentials (ERNs) are important areas of BCI research. The motor imagery is initiated by the brain actively and does not depend on any sensory stimulation, and the signal is generated through the process of the imagery, and the motor imagery comprises two signals of left and right, and is divided into three levels. When the brain performs different motor imagery activities or has performed actual movement, the area where the sensory motor cortex is activated and different rhythm brain electrical signals are changed regularly. The electrophysiological phenomenon related to the motor imagery electroencephalogram signal is the ERD/ERS phenomenon, and when a certain region of the cerebral cortex is in an activated state, the energy of signals in certain frequency bands is reduced, and the electrophysiological phenomenon is called event-related desynchronization (ERD). Conversely, when the brain is in a resting or inactive state, the energy of signals in some frequency bands is increased, which is called event-related synchronization (ERS). When a human body moves a right hand or imagines the action of the right hand, the ERD phenomenon appears on the sensory-motor cortex on the left side of the brain, and the ERS phenomenon appears on the sensory-motor cortex on the right side. When a human body moves left hand or imagines left hand movement, ERS phenomenon appears on the left sensory motor cortex of the brain, and ERD phenomenon appears on the right sensory motor cortex. The error-related negative potential is the negative phase potential shift, called ERN, that is specifically related to the error response recorded in the central area of the scalp of the human brain when the individual perceives the error response, and the classic waveform is shown in FIG. 8.
ERN reflects the process by which the human brain monitors or reflects the assessment of the overall response. The ERN has potential stability, is not influenced by physiological structure change, is generally induced to be generated 100-300 ms after individual perception error, has the wave amplitude of about 10uV, low signal-to-noise ratio and large individual difference, and the specific typical characteristics of different types of ERN are shown in the table below. ERN is mainly concentrated on 4-12 Hz, namely theta and mu frequency bands. ERN begins in adolescence and increases in early adulthood, commensurate with the development of other cognitive control abilities, reflecting how important the subject is to avoid the wrong or respond correctly.
With the maturity of brain-computer interface technology, the intelligent system of the new mode that combines with it is constantly emerging, has included intelligent house, education science and technology, health medical treatment, entertainment control etc.. In 2017, science and university news research and development have developed a BCI-based intelligent home system, and a research result published by the United states department of defense advanced research program office shows that a pilot can realize the operation of multiple airplanes and unmanned planes by wearing brain-computer interface equipment. Meanwhile, in the development process of the unmanned intelligent system, people have higher requirements on the capabilities of the unmanned intelligent system in the aspects of autonomous cognition, decision making, planning, control and the like, and a typical rational intelligent body model ABGP is provided by the Chinese academy of sciences and computing technology institute Schlanter and the like, so that the external perception and the internal state are cooperated together, the environment can be perceived, and the behavior of the people can be reasonably planned. At present, many scholars at home and abroad accumulate some research achievements on the introduction of a brain intelligence heuristic intelligent driving system. Some researchers have developed a feeling detection device for the passengers of a vehicle.
The brain-like processing unit provided by the invention carries out brain-like agent structure design with brain electricity cognition function on the basis of the ABGP agent model, has the functions of processing signals of certain EEG channels and expressing relevant EEG characteristics, and further controls output to reach a target, and the agent structure is shown in figure 9. The ABGP agent model describes internal activities as four modules, including: the system comprises a perception module, a belief module, a target module and a planning module, wherein perception is cognition of a specific external environment state, and when the environment changes along with time, the perception also changes along with the change. In the brain-like intelligent body model, the perception module is realized by using the capsule neural network model, the traditional definition rule is replaced, various parameters of the pre-trained capsule neural network model are stored in a knowledge base of the belief module as beliefs, and the results of other modules are not adjusted. The brain agents realize self behavior planning through motivation driving, and plan and select motivation driving through the maximum interest of some internal events. The electroencephalogram information acquired by the brain-like intelligent body has the characteristics of nonlinearity, robustness and hierarchy, and the attributes are directly used for forming the knowledge of the belief module, so that the brain-like characteristic perception capability of the intelligent body is stronger.
The invention develops an effective automatic driving auxiliary intelligent system based on the analysis of the electroencephalogram, applies BCI and brain-like agent technologies to the existing automatic driving field, and the electroencephalogram plays an intelligent guiding role in the sudden situation of the driving system, thereby having important significance in the development of the intellectualization of vehicles.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1) the traditional electroencephalogram signal classification method focuses on time-frequency feature extraction of signals, but the problem of dimension disaster often occurs under the condition of analyzing multi-channel data, and a large amount of valuable information is lost in the dimension reduction process. The invention intercepts the original brain waveform data according to a specific time interval, converts the data into a brain electrical topographic map sequence which changes along with time, and then projects the sequence onto a picture, thereby converting the characteristics of time domain, frequency domain and space domain in the brain waveform data into the spatial position relation of each pixel point in the picture, and then identifying and classifying objects into images, thus being capable of adopting a capsule neural network method with good effect.
2) The invention applies the capsule neural network to the classification task of the electroencephalogram signals. The capsule neural network retains positional information between objects in the image and takes into account spatial hierarchical relationships. On the basis of a data set processing mode, by utilizing the sensitivity of the network to the spatial position relation among different features in the image, the high-precision identification task can be completed by using less data sets.
3) The invention combines an automatic driving auxiliary intelligent system with brain inspiring thinking, simulates human cognitive thinking, designs a brain-like intelligent system based on an ABGP intelligent body model and a capsule neural network, establishes a high-speed channel between the intelligent system and an electroencephalogram acquisition system, improves interaction among a plurality of systems and between the systems and electroencephalogram detection, and can better select optimal action planning according to a target.
4) The invention adopts a detection control mode based on a brain-computer interface, overcomes the defects of operation delay, decision error and the like caused by switching between automatic driving and manual driving, is effectively applied to intelligent assistance of automatic driving, and can also be used for detecting the brain physiological state of a driver when the driver takes a vehicle, and analyzing and optimizing the riding experience of the driver.
The idea provided by the invention is also suitable for the recognition and classification task of the electroencephalogram signals in other scenes.
Drawings
Fig. 1 is a flow chart of a brain heuristic automatic driving assistance system based on a capsule neural network according to the present invention.
FIG. 2 is a flow chart of data processing according to an embodiment of the present invention;
FIG. 3 is a diagram of an electroencephalogram cap electrode point bitmap according to an embodiment of the present invention;
FIG. 4 is a diagram of a capsule structure according to an embodiment of the present invention;
FIG. 5 is a diagram of a capsule neural network framework according to an embodiment of the present invention;
FIG. 6(a) is a signal spectrum diagram of an unprocessed signal according to an embodiment of the present invention; FIG. 6(b) is a spectrum diagram of 8-12 Hz after wavelet packet decomposition of the signal according to the embodiment of the present invention; FIG. 6(c) is a spectrum diagram of 18-30 Hz after wavelet packet decomposition of a signal according to an embodiment of the present invention; the horizontal axis of the spectrogram represents frequency, and the vertical axis represents the logarithm of the power spectral density of a signal;
FIG. 7(a) is a schematic representation of the electroencephalogram of a subject observing a correct signature when testing negative potentials associated with errors according to the embodiment of the present invention, and FIG. 7(b) is a schematic representation of the electroencephalogram of a subject observing a wrong signature;
FIG. 8 is a typical waveform diagram of ERN;
FIG. 9 is a diagram of a brain-like intelligence system architecture in accordance with the present invention.
In the figure: A. the system comprises an electroencephalogram signal acquisition system, a B error correlation negative potential analysis system, a C motor imagery analysis system, a D automatic navigation control system, a 1 signal acquisition unit, a 2 electroencephalogram signal preprocessing unit, a 3 ERN signal preprocessing unit, a 4 capsule neural network unit, a 5 motor imagery signal preprocessing unit, a 6 signal characteristic extraction unit, a 7 signal classification and identification unit, a 8 real-time positioning unit, a 9 image building unit, a 10 path planning unit and a 11 brain-like processing unit.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.
As shown in FIG. 1, the invention provides a brain heuristic automatic driving assistance system based on a capsule neural network, which comprises an electroencephalogram signal acquisition system A, an error-related negative potential analysis system B, a motor imagery analysis system C and an automatic navigation control system D.
An electroencephalogram signal acquisition system A: the device is used for collecting electroencephalogram data of a user and sending the electroencephalogram data to an upper computer for processing;
error-dependent negative potential analysis system B: the capsule neural network is used for realizing the detection function of the error-related negative potential ERN, extracting error-related negative potential signals in the brain of a driver according to electroencephalogram data of a user, outputting a mark signal and reflecting the correctness of the current vehicle navigation state;
motor imagery analysis system C: judging the driving intention of the brain of the user according to the electroencephalogram data of the user, and extracting and classifying to generate a left or right instruction;
automatic navigation control system D: the system is used for simulating an automatic driving scene of a vehicle, realizing positioning, mapping and path planning of an unknown scene, receiving a mark signal output by the error-related negative potential analysis system B and an instruction signal output by the motor imagery analysis system C, further calculating the output driving state variable quantity through PID control, and controlling the updating of the vehicle navigation state.
The electroencephalogram signal acquisition system A is composed of a signal acquisition unit 1 and an electroencephalogram signal preprocessing unit 2, the signal acquisition unit 1 acquires potential signals of specific points of the epidermis of the brain of a user, and the electroencephalogram signal preprocessing unit 2 amplifies and filters the acquired electroencephalogram signals.
The electroencephalogram signal acquisition system A adopts an electrode cap based on an international 10-20 system electrode placement method. According to the recommendation of the International electroencephalogram society for a 10-20 system with 32 electrode positions, the names of the electrode positions are shown in figure 3, and electroencephalogram signals are acquired through 32 EEG channels of FP 1-O2. In the whole experiment process, a British public physiological data set Mahnob-HCI-TaggingDatabas is adopted for model training, then electroencephalogram data collected by the system are used for testing, and audio, video, gazing data and physiological data are recorded at the same time. And a band-pass filter with the frequency of 0.1-70 Hz and a notch filter with the frequency of 50Hz are used for removing power frequency interference and other noises so as to ensure the quality of the electroencephalogram signals. In the 32 EEG channels of the signal acquisition unit 1, signals of the electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 channels are used for the error-related negative potential analysis system B to perform feature extraction and classification, and signals of the electrodes C3, C4 and Cz channels are used for the motor imagery analysis system C to perform feature extraction of the motor imagery signals in time domain, frequency domain, spatial domain and energy domain.
The error-related negative potential analysis system B comprises an ERN signal preprocessing unit 3 and a capsule neural network unit 4, and is used for carrying out feature extraction and classification by utilizing signals of channels of capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2, and analyzing and judging the generation of error-related negative potentials in real time. The ERN signal preprocessing unit 3 preprocesses the electroencephalogram data, completes the screening, re-referencing, filtering, independent principal component analysis and segmentation of electroencephalogram signals, converts the electroencephalogram data time domain oscillogram output by the electroencephalogram signal acquisition system a into a time sequence of a electroencephalogram geogram, selects a time window, and converts the electroencephalogram signals into the electroencephalogram geogram according to a set time interval, as shown in fig. 7(a) and 7 (b); the capsule neural network unit 4 classifies and extracts the electroencephalogram topographic map output by the ERN signal preprocessing unit 3, the capsule is used as a basic computing unit to construct a capsule network, the capsule is a vector consisting of a series of neurons, the value of each neuron represents posture parameters such as scaling, direction and position, the length of the capsule represents the probability of existence of a specific object, meanwhile, the transmission among the capsules is realized through a dynamic routing algorithm, so that the low-level capsules in the network can predict the activation state of the high-level capsules, and the capsule structure is shown in fig. 4. And outputting two mark instructions of error reaction and error-free reaction according to whether the ERN signal is extracted or not.
In this embodiment, the ERN signal preprocessing unit 3 intercepts data from 230ms, converts the electroencephalogram signal into an electroencephalogram topographic map every 30ms until 950ms, acquires images at 25 moments from each time domain topographic map, arranges the images in the order of 5 × 5, and retains effective information to synthesize the electroencephalogram topographic map, as shown in fig. 7(a) and 7 (b).
In this embodiment, as shown in fig. 5, the network framework of the capsule neural network unit 4 mainly includes: 4 convolution layers, 1~2 capsule layers, 1 input layer, 1 average pooling layer, 1 remolding layer, 1 full tie layer. As shown in fig. 5, the network model constructed by python includes: the input layer is an array of 32 x 3, the number of the array changes along with the change of the number of samples, the input of the convolution layer is set as sample data, different input features are extracted through convolution operation, the first layer of convolution layer can only extract some low-level features such as edges, lines, angles and other levels, more layers of networks can extract more complex features from the low-level features in an iteration mode, and the more complex features are output as feature vectors of pictures; the average pooling layer is used for reducing the complexity of data to be processed, further accelerating the convergence efficiency, reducing the operation overhead and improving the resistance of the network model to noise contained in the electroencephalogram topological graph; the remodeling layer converts input data into a format which can be identified by the capsule neural network, namely remodeling the output of Conv2D to obtain a group of vectors of each position to form the input of a low-dimensional capsule layer; the capsule layer is presented in a capsule form, 2 vectors with 16 dimensions are output, and each vector represents a classification result; the capsule layer is used for encapsulating information obtained after encoding the electroencephalogram topological graph, the different dimensions of the capsule encode the angle and distance information among the local features of the electroencephalogram topological graph, and the information encoded by the capsule is from the spatial position relationship of the local features to the spatial position relationship of the global features along with the increase of the number of the capsule layers, so that the time sequence information of the electroencephalogram topological graph is contained and encoded in the form of the spatial relationship; the output of the capsule layer is projected into two probability values through the matrix of the full-connection layer to be used as the prediction output of the model, and the class with larger prediction value is used as the final classification output of the capsule neural network model. And the full connection layer outputs the modulus of 2 vectors as a probability value to represent the possibility of two classification results in the two classification tasks, and the final output result is obtained when the probability value is large. In addition, the number of the capsule layers and the dimension of the capsule output vector can be adjusted, so that the optimal classification effect is obtained. And (3) outputting a result with a high probability value in the full connection layer of the capsule neural network, wherein if the output is 1, the ERN is not generated, and if the output is 0, the ERN is generated, so that the individual perceives that an error occurs.
The motor imagery analysis system C selects signals of electrodes C3, C4 and Cz channels to perform feature extraction of a time domain, a frequency domain, a space domain and an energy domain of the motor imagery signals by using a wavelet packet and common spatial mode (CSP) fusion algorithm. The wavelet packet extracts the brain wave biorhythm related to the movement, and then a CSP (compact strip service) is utilized to construct a spatial filter, so that the spatial domain characteristic difference of the left and right motor imagery signals is maximized. The motor imagery analysis system C comprises a motor imagery signal preprocessing unit 5, a signal feature extraction unit 6 and a signal classification and identification unit 7; the motor imagery signal preprocessing unit 5 completes filtering, segmentation, baseline calibration, re-reference and independent principal component analysis of the electroencephalogram signal to obtain pure electroencephalogram data; the signal feature extraction unit 6 extracts the features of the signals, decomposes the trained electroencephalogram data by utilizing a wavelet packet decomposition algorithm, and extracts 8-12 Hz mu rhythm and 18-30 Hz beta rhythm signals, wherein the power spectrums are shown in FIGS. 6(a), 6(b) and 6 (c); processing the extracted corresponding frequency band of the rhythm by using a common space mode CSP algorithm to extract and obtain a highly differentiated characteristic vector; the signal classification and identification unit 7 receives the feature vector signals extracted by the signal feature extraction unit 6, classifies the signals based on an SVM vector machine, and classifies the signals to obtain three instructions of left, right or current situation.
The automatic navigation control system D simulates an automatic driving environment and technical operation based on the ROS platform. The simulated driving environment and the driving vehicle are designed and formed by an open source model. The simulation vehicle is provided with a camera, a radar and other sensor models, automatic mapping and positioning functions are carried out through a gmapping tool algorithm, and path planning and navigation functions are designed through a navigation tool algorithm. The environmental picture in the vehicle driving simulation process is transmitted to a monitor through a camera and displayed to a subject in real time. The core processing unit adopts a brain-like agent to simulate human cognitive thinking, processes information sensed by a visual sensor, a laser sensor, a gyroscope and the like, and performs comprehensive processing to correct the driving state by combining error-related negative potential signals and motor imagery signals detected by the front end to complete the function of safe auxiliary driving. The automatic navigation control system D comprises a real-time positioning unit 8, a mapping unit 9, a path planning unit 10 and a brain-like processing unit 11. The real-time positioning unit 8 is used for performing positioning calculation through a data model according to data received by the sensor, outputting a conversion relation between a world coordinate system and a odometer coordinate system according to information input by the vision sensor, the laser radar and the gyroscope, and deducing a conversion relation between a vehicle base coordinate system and the world coordinate system to realize a positioning function; the map building unit 9 is used for realizing scanning of surrounding environment information by adopting a mapping tool algorithm, building an output environment map, and building a probability-based two-dimensional grid map based on visual sensor information, gyroscope information and odometer information; the path planning unit 10 receives the map, the sensor data stream and the odometer information generated by the map building unit 9 by adopting a navigation tool algorithm, outputs speed and angular speed instructions to the vehicle control chassis, further controls the driving state and realizes the functions of path planning and obstacle avoidance; the brain-like processing unit 11 receives the sign signal output by the error-related negative potential analysis system B and the instruction signal output by the motor imagery analysis system C, and deduces and forms a complete multilayer experience map by receiving and memorizing the local acquisition information of the sensor, and maps the map to a real scene to correct the driving state in real time.
The brain-like processing unit 11 combines the received sign signal output by the error-related negative potential analysis system B and the instruction signal output by the motor imagery analysis system C based on the design idea of a brain-like agent, and infers and forms a complete multi-layer experience map by receiving and memorizing the local acquisition information of various sensors, and maps the map to a real scene to correct the driving state in real time. When the ERN flag signal is 1, the system keeps the existing state and does not make any correction, and when the ERN flag signal is 0, the ERN signal in the human brain is generated, and the flag signal is used as a higher-level instruction to control the brain processing unit 11. When the sign signal is 0, the vehicle driving system immediately decelerates, meanwhile, the brain-like processing unit 11 further receives a direction instruction output by the motor imagery analysis system C, performs intensive collection and rapid processing on environmental information of the instruction direction by combining a map and positioning, outputs PID (proportion integration differentiation) instructions for controlling four wheel rotation speed control joints and two wheel steering control joints of the vehicle through a PID (proportion integration differentiation) algorithm after judging the feasible deflection direction, and when the motor imagery analysis system C does not output the direction instruction, the brain-like processing unit 11 does not perform obstacle avoidance and correction operation on the driving state.
Based on the system, the invention also provides a brain heuristic automatic driving assistance method based on the capsule neural network, which comprises the following steps:
1) acquiring electroencephalogram data: the electroencephalogram signal acquisition system A acquires electroencephalogram data of a user and sends the electroencephalogram data to an upper computer for processing; an electrode cap based on an international 10-20 system electrode placement method carries out user electroencephalogram signal acquisition through 32 EEG channels including FP 1-O2, and a band-pass filter with the frequency of 0.1-70 Hz and a 50Hz notch filter are used for removing power frequency interference and other noises so as to ensure the quality of electroencephalogram signals;
the electroencephalogram signal acquisition system A requires a tested driver to respond to the running environment condition in the picture at any time when the tested driver faces the monitor picture of the automatic navigation control system in the signal acquisition process. The test subject can completely receive the surrounding vision of the driving of the simulated vehicle, and the electrode cap is worn to ensure the real-time acquisition and transmission of each electrode signal.
2) Error-dependent negative potential analysis: the error-related negative potential analysis system B utilizes signals of channels of the capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 to extract and classify features, extracts error-related negative potential signals in the brain of a driver according to electroencephalogram data of a user, analyzes and judges the generation of error-related negative potentials in real time, and outputs a mark signal;
3) and (3) outputting a motion direction instruction: the motor imagery analysis system C selects signals of electrodes C3, C4 and Cz channels by using a wavelet packet and common space mode fusion algorithm to perform characteristic extraction of a time domain, a frequency domain, a space domain and an energy domain of the motor imagery signals, classifies the signals by using an SVM (support vector machine), obtains three instructions of left and right or keeping the current situation by classification, and outputs instruction signals;
4) controlling vehicle navigation: the automatic navigation control system D simulates the automatic driving scene of the vehicle, realizes positioning, mapping and path planning of an unknown scene, receives the sign signal output in the step 2) and the instruction signal output in the step 3), further calculates the variation of the output driving state through PID control, and controls the updating of the navigation state of the vehicle.
The specific steps of the capsule neural network in the step 2) for feature extraction and classification comprise:
21) collecting electroencephalogram signals of a plurality of channels, and preprocessing original signals; the studied EEG signal channel is an EEG signal of 11 channels in the middle of the frontal lobe of the brain, and the preprocessing comprises 4-16Hz band-pass filtering, re-referencing, independent principal component ICA analysis and time period interception.
22) Intercepting a signal waveform according to a fixed time interval, and calculating corresponding power spectral density; the waveform signal is truncated for a time interval of 30ms and the start and stop times of the truncation are 230ms and 950ms so as to maximize the searchable time span while preserving the original sequence continuity.
23) And (3) normalizing and mapping the power spectral density into a chromatogram, combining the spatial localization of the channel in the brain, and supplementing the values of the rest positions by a spatial interpolation algorithm to obtain a sequence of the electroencephalogram topological graph changing along with time. Completely arranging the electroencephalogram topological graph in a graph according to a time sequence to be used as sample data; the formula of the spatial interpolation algorithm is as follows:
where X is the position of the point to be interpolated, a, b, … …, p represents the power value for each acquisition point, and XA, XB, XP represents the distance of the point to each acquisition point to be interpolated. The power values of the removed channels, and other unconnected electrode areas, are predicted using a spatial interpolation algorithm. And predicting other areas based on data of a small number of channels, so that the signal-to-noise ratio is improved, and the convergence speed of model training is accelerated.
24) Inputting the sample data into a capsule neural network for operation; the network structure of the capsule neural network unit 4 includes: 4 layers of convolution layers, 1 layer of average pooling layer, 1-2 capsule layers and 1 layer of full-connecting layer. The capsule structure is shown in fig. 4, and the matrix multiplication formula of the input vector in the capsule neural network is as follows:
wherein,u i is the output of the neural network of the previous layer of capsule,W ij the weight matrix is used for coding important spatial relationship or other relationship between the lower layer characteristic and the upper layer characteristic, and can be regarded as different association degrees between the upper layer neuron and the lower layer neuron of each capsule.
Is a prediction vector that represents the predicted output of the previous layer capsule attempting to predict the next layer capsule.
Wherein the capsule neural network input vector is weighted and summed S j The calculation formula is as follows:
C ij to be a coupling coefficient, ofb ij The calculation results in that,b ij calculated by the updating of the dynamic routing algorithm of the capsule network core.
25) The capsule layer uses a dynamic routing algorithm to carry out three iterations and update the weight:
wherein, VjIs the output vector of the capsule of the previous layer,
new weight values for the previous layer of capsules trying to predict the output of the next layer of capsulesb ij By mixingV jAnd
dot product and add the original weightb ij Thus obtaining the product. The weights obtained in the previous two iterationsb ij Weights for different capsules representing the encoding of information characterizing the different capsules; the last iteration is tob ij The weighting for all the different dimensions of the capsule represents the encoding of the information characterized by the different dimensions in the capsule.
Wherein, the nonlinear proper compression function in the capsule neural network is as follows:
wherein,V jis the output vector of the first capsule,S jis the input vector of the first capsule, | purpleS jThe method comprises the following steps of obtaining direction information of an input vector by using the square () as an activation function, compressing the mould of the input vector between 0 and 1, representing the probability of occurrence of a certain event, wherein the higher the value is, the higher the probability of occurrence is.
Dividing the sample data into a training set and a test set according to the proportion of 7:1, and training the generalization capability of the capsule neural network model and the test set evaluation model by using the training set.
26) And outputting the class with the maximum capsule neural network output value as a prediction result for judging whether the error related potential is generated.
Training an SVM support vector machine in the step 3) by using tens of existing motor imagery feature vector sets on the left and right, training an SVM two-class model by using feature vectors extracted from electroencephalogram data, searching hyper-parameters (c and g), namely a group of optimal punishment parameters c and RBF kernel function width, in a two-dimensional grid mode through cross validation, calculating the optimal value of the cross validation precision of each group of parameters, taking the highest precision group (c and g) as the optimal parameters to enable the classification accuracy of the SVM classification model to be maximum, outputting 0 to represent a left instruction by the SVM support vector machine, outputting 1 to represent a right instruction, and outputting no change instruction if not outputting.
It should be understood that the above description of the preferred embodiments is given for clarity and not for any purpose of limitation, and that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A enlightening-type automatic driving auxiliary system based on capsule neural network is characterized in that: the system comprises an electroencephalogram signal acquisition system (A), an error-related negative potential analysis system (B), a motor imagery analysis system (C) and an automatic navigation control system (D);
the electroencephalogram signal acquisition system (A): the device is used for collecting electroencephalogram data of a user and sending the electroencephalogram data to an upper computer for processing;
the error-dependent negative potential analysis system (B): the capsule neural network is used for realizing the detection function of the error-related negative potential ERN, extracting error-related negative potential signals in the brain of a driver according to electroencephalogram data of a user, outputting a mark signal and reflecting the correctness of the current vehicle navigation state;
the motor imagery analysis system (C): judging the driving intention of the brain of the user according to the electroencephalogram data of the user, and extracting and classifying to generate a left or right instruction;
the automatic navigation control system (D): the method is realized based on an ROS simulation platform and used for simulating an automatic driving scene of a vehicle, realizing positioning, mapping and path planning of an unknown scene, receiving a mark signal output by the error-related negative potential analysis system (B) and an instruction signal output by the motor imagery analysis system (C), further calculating the output driving state variable quantity through PID control, and controlling the updating of the vehicle navigation state.
2. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 1, wherein: the electroencephalogram signal acquisition system (A) is composed of a signal acquisition unit (1) and an electroencephalogram signal preprocessing unit (2), the signal acquisition unit (1) acquires potential signals of specific points of the epidermis of the brain of a user, and the electroencephalogram signal preprocessing unit (2) amplifies and filters the acquired electroencephalogram signals.
3. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 2, wherein: the electroencephalogram signals are acquired through 32 EEG channels including FP 1-O2 by a signal acquisition unit (1), signals of FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 channels are used for a fault correlation negative potential analysis system (B) to perform feature extraction and classification, and signals of C3, C4 and Cz channels are used for a motor imagery analysis system (C) to perform feature extraction of a time domain, a frequency domain, a spatial domain and an energy domain of the motor imagery signals.
4. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 1, wherein: the error-related negative potential analysis system B comprises an ERN signal preprocessing unit (3) and a capsule neural network unit (4), wherein the ERN signal preprocessing unit (3) preprocesses electroencephalogram data, screening, re-referencing, filtering, independent principal component analysis and segmentation of electroencephalogram signals are completed, an electroencephalogram data time domain oscillogram output by the electroencephalogram signal acquisition system A is converted into a time sequence of an electroencephalogram topographic map, a time window is selected, the electroencephalogram signals are converted into the electroencephalogram topographic map according to a set time interval, the capsule neural network unit (4) classifies and extracts the electroencephalogram topographic map output by the ERN signal preprocessing unit (3), and two marking instructions of error reaction and no error reaction are output according to whether the ERN signals are extracted or not.
5. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 4, wherein: the network structure of the capsule neural network unit (4) comprises: 4 coiling layers, 1 average pooling layer, 1 remodeling layer, 1-2 capsule layers and 1 full-connection layer; the input of the convolutional layer is sample data, the convolutional layer preliminarily extracts local features in the electroencephalogram topological graph, the sample data are projected to different spaces as much as possible by using a convolutional kernel, the perception degree of the local features is increased, and the convolutional layer outputs a coding set obtained by projecting the sample data in different modes; the average pooling layer is used for reducing the complexity of data to be processed, further accelerating the convergence efficiency, reducing the operation overhead and improving the resistance of the network model to noise contained in the electroencephalogram topological graph; the remodeling layer is used for converting input data into a format which can be recognized by the capsule neural network; the capsule layer is presented in a capsule form and used for encapsulating information obtained after encoding the electroencephalogram topological graph, the different dimensions of the capsule encode angle and distance information among local features of the electroencephalogram topological graph, and along with the increase of the number of the capsule layers, the information encoded by the capsule is from the spatial position relation of the local features to the spatial position relation of global features, so that time sequence information of the electroencephalogram topological graph is included and encoded in the form of the spatial relation; the fully-connected layer projects the output of the capsule layer into two probability values through a matrix to be used as the prediction output of the model, and the class with larger prediction value is used as the final classification output of the capsule neural network model.
6. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 1, wherein: the motor imagery analysis system (C) comprises a motor imagery signal preprocessing unit (5), a signal feature extraction unit (6) and a signal classification and identification unit (7); the motor imagery signal preprocessing unit (5) completes filtering, segmentation, baseline calibration, re-reference and independent principal component analysis of the electroencephalogram signals to obtain pure electroencephalogram data; the signal feature extraction unit (6) extracts the features of the signals, decomposes the trained electroencephalogram data by utilizing a wavelet packet decomposition algorithm, extracts 8-12 Hz mu rhythms and 18-30 Hz beta rhythms, and processes corresponding frequency bands of the extracted rhythms by utilizing a common space mode CSP algorithm to extract and obtain highly differentiated feature vectors; and the signal classification and identification unit (7) receives the feature vector signals extracted by the signal feature extraction unit (6), classifies the signals based on an SVM vector machine, and classifies the signals to obtain three instructions of left, right or current situation.
7. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 1, wherein: the automatic navigation control system (D) comprises a real-time positioning unit (8), a mapping unit (9), a path planning unit (10) and a brain-like processing unit (11); the real-time positioning unit (8) is used for performing positioning calculation through a data model according to data received by the sensor, outputting a conversion relation between a world coordinate system and a odometer coordinate system according to information input by the vision sensor, the laser radar and the gyroscope, and deducing the conversion relation between a vehicle base coordinate system and the world coordinate system to realize a positioning function; the map building unit (9) adopts a mapping tool algorithm to realize the scanning of surrounding environment information, builds an output environment map, and builds a probability-based two-dimensional grid map based on the visual sensor information, the gyroscope information and the odometer information; the path planning unit (10) receives the map, the sensor data stream and the odometer information generated by the map building unit (9) by adopting a navigation toolkit set, outputs speed and angular speed instructions to the vehicle control chassis, further controls the driving state and realizes the functions of path planning and obstacle avoidance; the brain-like processing unit (11) receives the sign signal output by the error-related negative potential analysis system B and the instruction signal output by the motor imagery analysis system (C), and deduces and forms a complete multilayer experience map by receiving and memorizing the local acquisition information of the sensor, and maps the map to a real scene to correct the driving state in real time.
8. A enlightening-typed autopilot auxiliary method based on a capsule neural network, which is realized based on the enlightening-typed autopilot auxiliary system based on the capsule neural network of any one of claims 1 to 7, and is characterized in that: the method comprises the following steps:
1) acquiring electroencephalogram data: the electroencephalogram signal acquisition system (A) acquires electroencephalogram data of a user and sends the electroencephalogram data to an upper computer for processing; an electrode cap based on an international 10-20 system electrode placement method carries out user electroencephalogram signal acquisition through 32 EEG channels including FP 1-O2, and a band-pass filter with the frequency of 0.1-70 Hz and a 50Hz notch filter are used for removing power frequency interference and other noises so as to ensure the quality of electroencephalogram signals;
2) error-dependent negative potential analysis: the error-related negative potential analysis system (B) utilizes signals of channels of capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 to extract and classify features, extracts error-related negative potential signals in the brain of a driver according to electroencephalogram data of the user, analyzes and judges the generation of error-related negative potentials in real time, and outputs a mark signal;
3) and (3) outputting a motion direction instruction: the motor imagery analysis system (C) selects signals of electrodes C3, C4 and Cz channels by using a wavelet packet and common space mode fusion algorithm to extract characteristics of a time domain, a frequency domain, a space domain and an energy domain of the motor imagery signals, classifies the signals by using an SVM (support vector machine), obtains three instructions of left and right or keeping the current situation by classification, and outputs instruction signals;
4) controlling vehicle navigation: the automatic navigation control system (D) simulates an automatic driving scene of a vehicle, realizes positioning, mapping and path planning of an unknown scene, receives the sign signal output in the step 2) and the instruction signal output in the step 3), further calculates the output driving state variation through PID control, and controls the updating of the vehicle navigation state.
9. The enbrain heuristic automatic driving assistance method based on the capsule neural network of claim 8, wherein: the specific steps of the capsule neural network in the step 2) for feature extraction and classification comprise:
21) collecting electroencephalogram signals of a plurality of channels, and preprocessing original signals;
22) intercepting a signal waveform according to a fixed time interval, and calculating corresponding power spectral density;
23) normalizing and mapping the power spectral density into a color spectrum, combining the spatial localization of the channel in the brain, supplementing the values of the rest positions through a spatial interpolation algorithm to obtain a sequence of an electroencephalogram topological graph changing along with time, and completely arranging the electroencephalogram topological graph in one graph according to a time sequence to be used as sample data;
24) inputting the sample data into a capsule neural network for operation;
25) performing three iterations by using a dynamic routing algorithm and updating the weight by implementing a capsule neural network;
26) and outputting the class with the maximum capsule neural network output value as a prediction result for judging whether the error related potential is generated.
10. The enbrain heuristic automatic driving assistance method based on the capsule neural network of claim 8, wherein: the SVM support vector machine in the step 3) is trained by tens of existing motor imagery feature vector sets on the left and right, an SVM binary model is trained by using feature vectors extracted from electroencephalogram data, and through cross validation, hyper-parameters (c, g), namely a group of optimal punishment parameters c and RBF kernel function width, are searched in a two-dimensional grid mode, the optimal value of the cross validation precision of each group of parameters is calculated, the highest precision group (c, g) is used as the optimal parameter, the classification accuracy of the SVM classification model is enabled to be maximum, the SVM support vector machine outputs 0 to represent a left instruction, outputs 1 to represent a right instruction, and does not output a change instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221486.6A CN114312819B (en) | 2022-03-09 | 2022-03-09 | Brain heuristic type automatic driving assistance system and method based on capsule neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221486.6A CN114312819B (en) | 2022-03-09 | 2022-03-09 | Brain heuristic type automatic driving assistance system and method based on capsule neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114312819A true CN114312819A (en) | 2022-04-12 |
| CN114312819B CN114312819B (en) | 2022-06-28 |
Family
ID=81033022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210221486.6A Active CN114312819B (en) | 2022-03-09 | 2022-03-09 | Brain heuristic type automatic driving assistance system and method based on capsule neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114312819B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012095905A (en) * | 2010-11-04 | 2012-05-24 | Panasonic Corp | Brain wave measurement system, method, and computer program |
| CN104083258A (en) * | 2014-06-17 | 2014-10-08 | 华南理工大学 | Intelligent wheel chair control method based on brain-computer interface and automatic driving technology |
| CN107065850A (en) * | 2016-12-12 | 2017-08-18 | 东南大学 | The four motorized wheels intelligent car systems and its control method of a kind of brain wave control |
| CN109299647A (en) * | 2018-07-24 | 2019-02-01 | 东南大学 | A kind of extraction of multitask Mental imagery brain electrical feature and mode identification method towards vehicle control |
| CN109658503A (en) * | 2018-12-29 | 2019-04-19 | 北京理工大学 | A kind of driving behavior intention detection method merging EEG signals |
| CN110584597A (en) * | 2019-07-15 | 2019-12-20 | 天津大学 | Multi-channel electroencephalogram signal monitoring method based on time-space convolutional neural network and application |
| CN111329497A (en) * | 2020-02-21 | 2020-06-26 | 华南理工大学 | Wearable fatigue driving monitoring system and method based on forehead electroencephalogram signals |
| CN111783942A (en) * | 2020-06-08 | 2020-10-16 | 北京航天自动控制研究所 | Brain cognition process simulation method based on convolution cyclic neural network |
| CN112141116A (en) * | 2019-06-26 | 2020-12-29 | 现代自动车株式会社 | Method and apparatus for controlling moving body using error monitoring |
| US20210114599A1 (en) * | 2019-10-22 | 2021-04-22 | Hyundai Motor Company | Apparatus and method for generating driver skilled driving model using error monitoring |
| CN113002556A (en) * | 2019-12-03 | 2021-06-22 | 现代自动车株式会社 | Apparatus and method for providing customized driving path of moving body using brain wave signal |
-
2022
- 2022-03-09 CN CN202210221486.6A patent/CN114312819B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012095905A (en) * | 2010-11-04 | 2012-05-24 | Panasonic Corp | Brain wave measurement system, method, and computer program |
| CN104083258A (en) * | 2014-06-17 | 2014-10-08 | 华南理工大学 | Intelligent wheel chair control method based on brain-computer interface and automatic driving technology |
| CN107065850A (en) * | 2016-12-12 | 2017-08-18 | 东南大学 | The four motorized wheels intelligent car systems and its control method of a kind of brain wave control |
| CN109299647A (en) * | 2018-07-24 | 2019-02-01 | 东南大学 | A kind of extraction of multitask Mental imagery brain electrical feature and mode identification method towards vehicle control |
| CN109658503A (en) * | 2018-12-29 | 2019-04-19 | 北京理工大学 | A kind of driving behavior intention detection method merging EEG signals |
| CN112141116A (en) * | 2019-06-26 | 2020-12-29 | 现代自动车株式会社 | Method and apparatus for controlling moving body using error monitoring |
| CN110584597A (en) * | 2019-07-15 | 2019-12-20 | 天津大学 | Multi-channel electroencephalogram signal monitoring method based on time-space convolutional neural network and application |
| US20210114599A1 (en) * | 2019-10-22 | 2021-04-22 | Hyundai Motor Company | Apparatus and method for generating driver skilled driving model using error monitoring |
| CN112784397A (en) * | 2019-10-22 | 2021-05-11 | 现代自动车株式会社 | Device and method for generating a driving model for the proficiency of a driver using error monitoring |
| CN113002556A (en) * | 2019-12-03 | 2021-06-22 | 现代自动车株式会社 | Apparatus and method for providing customized driving path of moving body using brain wave signal |
| CN111329497A (en) * | 2020-02-21 | 2020-06-26 | 华南理工大学 | Wearable fatigue driving monitoring system and method based on forehead electroencephalogram signals |
| CN111783942A (en) * | 2020-06-08 | 2020-10-16 | 北京航天自动控制研究所 | Brain cognition process simulation method based on convolution cyclic neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114312819B (en) | 2022-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Gao et al. | EEG-based spatio–temporal convolutional neural network for driver fatigue evaluation | |
| Peng et al. | 3D-STCNN: Spatiotemporal Convolutional Neural Network based on EEG 3D features for detecting driving fatigue | |
| Almogbel et al. | EEG-signals based cognitive workload detection of vehicle driver using deep learning | |
| Lu et al. | XGBoost algorithm-based monitoring model for urban driving stress: Combining driving behaviour, driving environment, and route familiarity | |
| CN110377049B (en) | Brain-computer interface-based unmanned aerial vehicle cluster formation reconfiguration control method | |
| CN108537198A (en) | A kind of analysis method of the driving habit based on artificial intelligence | |
| DE102019105363B4 (en) | Procedure for a measuring system in the vehicle for the detection and classification of objects in the area around the vehicle using a deep learning procedure with a self-learning decision maker | |
| CN113743471A (en) | Driving evaluation method and system | |
| CN107967942B (en) | Children autism spectrum disorder analysis system based on near-infrared brain imaging map features | |
| Mou et al. | Driver emotion recognition with a hybrid attentional multimodal fusion framework | |
| CN106022291A (en) | Method of detecting braking intention of driver in emergency state based on neural signal | |
| CN114312819B (en) | Brain heuristic type automatic driving assistance system and method based on capsule neural network | |
| Chen et al. | A transfer learning-based CNN deep learning model for unfavorable driving state recognition | |
| Lin et al. | Nonparametric single-trial EEG feature extraction and classification of driver's cognitive responses | |
| DE102020100996A1 (en) | Ultrasonic measuring system with a self-learning AI control of microbeamformers in the vehicle | |
| Zhang et al. | ECMER: Edge-cloud collaborative personalized multimodal emotion recognition framework in the Internet of vehicles | |
| Zhou et al. | Detection of vigilance in L3 autonomous driving based on EEG | |
| CN115202479A (en) | Man-machine co-driving simulation system of man-in-loop and application thereof | |
| CN117137498B (en) | Emergency situation detection method based on attention orientation and exercise intention electroencephalogram | |
| CN114120634A (en) | Dangerous driving behavior identification method, device, equipment and storage medium based on WiFi | |
| QU et al. | Multi-Attention Fusion Drowsy Driving Detection Model | |
| KR20220076583A (en) | Customized self-driving devices and methods based on deep learning using cognitive characteristics | |
| Nader et al. | Using BCI and EEG to process and analyze driver’s brain activity signals during VR simulation | |
| CN114504329B (en) | Human brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment | |
| Sun | Video salience detection algorithm based on visual psychology in virtual environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |