CN114312819B - Brain heuristic type automatic driving assistance system and method based on capsule neural network - Google Patents
Brain heuristic type automatic driving assistance system and method based on capsule neural network Download PDFInfo
- Publication number
- CN114312819B CN114312819B CN202210221486.6A CN202210221486A CN114312819B CN 114312819 B CN114312819 B CN 114312819B CN 202210221486 A CN202210221486 A CN 202210221486A CN 114312819 B CN114312819 B CN 114312819B
- Authority
- CN
- China
- Prior art keywords
- electroencephalogram
- signals
- brain
- neural network
- signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention discloses a brain heuristic type automatic driving assistance system and method based on a capsule neural network, wherein the system comprises an electroencephalogram signal acquisition system: the system is used for collecting electroencephalogram data of a user; the error-related negative potential analysis system uses a capsule neural network to realize the detection of error-related negative potential ERN, extracts error-related negative potential signals in the brain of a driver according to the electroencephalogram data of the user, and outputs a mark signal; the motor imagery analysis system: judging the driving intention of the brain of the user according to the electroencephalogram data of the user, and extracting instructions; an automatic navigation control system: the method is used for simulating the automatic driving scene of the vehicle, realizing the positioning, the drawing establishment and the path planning of an unknown scene, further calculating the variation of the output driving state through PID control, and controlling the updating of the navigation state of the vehicle. The invention assists the safety correction of the automatic driving system based on the feature extraction and analysis model of the electroencephalogram signal and the brain-like intelligent body model, and reduces accidents caused by complex environments or technical defects.
Description
Technical Field
The invention relates to the technical field of image processing and recognition methods, in particular to a brain heuristic type automatic driving assistance system and method based on a capsule neural network.
Background
The intelligent automobile is a strategic direction of the development of the automobile industry all over the world and is also an important component of future intelligent traffic. The development of intelligent automobiles is beneficial to accelerating transformation and upgrade of the automobile industry, cultivating digital economy, and increasing new kinetic energy of the economy. The automatic driving technology can enhance the safety and flexibility of travel, promote the information construction of urban traffic, and meet the urgent needs of vehicle informatization and intellectualization. The intelligent level of the existing automobile is mostly L0-L3, and the existing automobile cannot reach L4 and L5 levels in a short time, and still has a large difference from the fully intelligent L5 level. Under the condition of lacking of manual intervention, the test vehicle cannot well process complicated and variable road conditions, and safety accidents are easy to occur.
A Brain-Computer Interface (BCI) is a communication device that directly connects the Brain to external devices. The brain-computer interface does not depend on the participation of peripheral nerves and muscles of the brain, and utilizes electroencephalograms (EEG) induced by the perception of the central nervous system or cognitive thinking activity to acquire post signals through an acquisition system and decode perception thinking intentions, so that brain post-movement information is converted into control signals, and the brain is communicated or controlled with external equipment. With the maturity of brain-computer interface technology, the intelligent system of the new mode that combines with it is constantly emerging, has included intelligent house, education science and technology, health medical treatment, entertainment control etc..
An effective automatic driving auxiliary intelligent system is developed based on electroencephalogram signal analysis, BCI and brain-like agent technologies are applied to the existing automatic driving field, and electroencephalogram signals play an intelligent guiding role in sudden conditions of a driving system and have important significance in development of vehicle intellectualization.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides a brain heuristic type automatic driving assistance system and method based on a capsule neural network, which can assist the safety correction of the automatic driving system by fusing a brain-like intelligent body model and a feature extraction and analysis model based on electroencephalogram signals, and reduce accidents caused by complex environments or technical defects.
In order to achieve the purpose, the invention designs a brain heuristic type automatic driving assistance system based on a capsule neural network, which is characterized in that the system comprises an electroencephalogram signal acquisition system, an error-related negative potential analysis system, a motor imagery analysis system and an automatic navigation control system;
the electroencephalogram signal acquisition system comprises: the device is used for collecting electroencephalogram data of a user and sending the electroencephalogram data to an upper computer for processing;
the error-dependent negative potential analysis system: the method comprises the steps that a capsule neural network is used for achieving the function of detecting error-related negative potential ERN, error-related negative potential signals in the brain of a driver are extracted according to electroencephalogram data of a user, and a mark signal is output to reflect the correctness of the current vehicle navigation state;
The motor imagery analysis system: judging the driving intention of the brain of the user according to the electroencephalogram data of the user, and extracting, classifying and generating a left or right instruction;
the automatic navigation control system: the method is realized based on an ROS simulation platform, and is used for simulating an automatic driving scene of a vehicle, realizing positioning, drawing and path planning on an unknown scene, receiving a mark signal output by the error-related negative potential analysis system and an instruction signal output by the motor imagery analysis system, further calculating the output driving state variable quantity through PID control, and controlling the updating of the vehicle navigation state.
Furthermore, the electroencephalogram signal acquisition system is composed of a signal acquisition unit and an electroencephalogram signal preprocessing unit, wherein the signal acquisition unit acquires potential signals of specific points of the epidermis of the brain of a user, and the electroencephalogram signal preprocessing unit amplifies and filters the acquired electroencephalogram signals.
Furthermore, the signal acquisition unit acquires electroencephalogram signals through 32 EEG channels including FP 1-O2, wherein signals of channels including FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 are used for feature extraction and classification of an error correlation negative potential analysis system, and signals of channels including C3, C4 and Cz are used for feature extraction of time domain, frequency domain, spatial domain and energy domain of a motor imagery signal of the motor imagery analysis system.
Furthermore, the error-related negative potential analysis system comprises an ERN signal preprocessing unit and a capsule neural network unit, wherein the ERN signal preprocessing unit is used for preprocessing electroencephalogram data, screening, re-referencing, filtering, independent principal component analysis and segmentation of electroencephalogram signals are completed, a time domain oscillogram of the electroencephalogram data output by the electroencephalogram signal acquisition system is converted into a time sequence of an electroencephalogram topographic map, a time window is selected, the electroencephalogram signals are converted into the electroencephalogram topographic map according to a set time interval, the capsule neural network unit is used for classifying and extracting the electroencephalogram topographic map output by the ERN signal preprocessing unit, and two marking instructions of error reaction and no error reaction are output according to whether the ERN signals are extracted or not.
Furthermore, the network structure of the capsule neural network unit mainly comprises: 4 coiling layers, 1 average pooling layer, 1 remodeling layer, 1-2 capsule layers and 1 full-connection layer; the input of the convolutional layer is sample data, the convolutional layer initially extracts local features in the electroencephalogram topological graph, the sample data are projected to different spaces as much as possible by using the convolutional kernel, the perception degree of the local features is increased, and the convolutional layer is output as a coding set obtained by projecting the sample data in different modes; the average pooling layer is used for reducing the complexity of data to be processed, further accelerating the convergence efficiency, reducing the operation overhead and improving the resistance of a network model to noise contained in an electroencephalogram topological graph; the remodeling layer is used for converting input data into a format which can be recognized by the capsule neural network; the capsule layer is presented in a capsule form and used for packaging information obtained after the electroencephalogram topological graph is coded, the different dimensions of the capsule code the angle and distance information among the local features of the electroencephalogram topological graph, and the information coded by the capsule is from the spatial position relationship of the local features to the spatial position relationship of the global features along with the increase of the number of the capsule layers, so that the time sequence information of the electroencephalogram topological graph is included and coded in the form of the spatial relationship; the output of the capsule layer is projected into two probability values through the matrix of the full-connection layer to serve as the prediction output of the model, and the class with the larger prediction value serves as the final classification output of the capsule neural network model.
Furthermore, the motor imagery analysis system comprises a motor imagery signal preprocessing unit, a signal feature extraction unit and a signal classification and identification unit; the motor imagery signal preprocessing unit completes filtering, segmentation, baseline calibration, re-reference and independent principal component analysis of the electroencephalogram signal to obtain pure electroencephalogram data; the signal feature extraction unit extracts the features of the signals, decomposes the trained electroencephalogram data by using a wavelet packet decomposition algorithm, extracts 8-12 Hz mu rhythm and 18-30 Hz beta rhythm, and processes corresponding frequency bands of the extracted rhythms by using a common spatial mode CSP algorithm to extract and obtain highly differentiated feature vectors; the signal classification and identification unit receives the feature vector signals extracted by the signal feature extraction unit, classifies the signals based on an SVM (support vector machine), and classifies the signals to obtain three instructions of left and right directions or keeping the current situation.
Furthermore, the automatic navigation control system comprises a real-time positioning unit, a mapping unit, a path planning unit and a brain-like processing unit; the real-time positioning unit is used for performing positioning calculation through a data model according to data received by the sensor, outputting a conversion relation between a world coordinate system and an odometer coordinate system according to information input by the vision sensor, the laser radar and the gyroscope, and deducing a conversion relation between a vehicle base coordinate system and the world coordinate system to realize a positioning function; the mapping unit adopts a mapping tool algorithm to realize the scanning of the surrounding environment information, constructs an output environment map, and creates a probability-based two-dimensional grid map based on the visual sensor information, the gyroscope information and the odometer information; the path planning unit receives the map, the sensor data flow and the odometer information generated by the map building unit by adopting a navigation toolkit set, outputs speed and angular speed instructions to the vehicle control chassis, further controls the running state and realizes the functions of path planning and obstacle avoidance; the brain-like processing unit receives the sign signal output by the error-related negative potential analysis system and the command signal output by the motor imagery analysis system, and deduces to form a complete multi-layer experience map by receiving and memorizing the local acquisition information of the sensor, and maps the map to a real scene to correct the driving state in real time.
The invention also provides a brain heuristic automatic driving assistance method based on the capsule neural network, which is realized based on the brain heuristic automatic driving assistance system based on the capsule neural network, and the method comprises the following steps:
1) acquiring electroencephalogram data: the electroencephalogram signal acquisition system acquires electroencephalogram data of a user and sends the electroencephalogram data to the upper computer for processing; an electrode cap based on an international 10-20 system electrode placement method carries out user electroencephalogram signal acquisition through 32 EEG channels including FP 1-O2, and a band-pass filter with the frequency of 0.1-70 Hz and a 50Hz notch filter are used for removing power frequency interference and other noises so as to ensure the quality of electroencephalogram signals;
2) error-dependent negative potential analysis: the error-related negative potential analysis system utilizes signals of channels of capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 to extract and classify features, extracts error-related negative potential signals in the brain of a driver according to electroencephalogram data of the user, analyzes and judges the generation of error-related negative potentials in real time, and outputs a mark signal;
3) and (3) outputting a motion direction instruction: the motor imagery analysis system selects signals of electrodes C3, C4 and Cz channels by using a wavelet packet and common space mode fusion algorithm to perform characteristic extraction of a time domain, a frequency domain, a space domain and an energy domain of the motor imagery signals, classifies the signals by using an SVM (support vector machine), obtains three instructions of left and right or keeping the current situation by classification, and outputs instruction signals;
4) Controlling the vehicle navigation: the automatic navigation control system simulates an automatic driving scene of a vehicle, realizes positioning, mapping and path planning of an unknown scene, receives the sign signal output in the step 2) and the instruction signal output in the step 3), further calculates the output driving state variable quantity through PID control, and controls the updating of the vehicle navigation state.
Preferably, the specific steps of the capsule neural network in step 2) for feature extraction and classification include:
21) collecting electroencephalogram signals of a plurality of channels, and preprocessing original signals;
22) intercepting the signal waveform according to a fixed time interval, and calculating corresponding power spectral density;
23) normalizing and mapping the power spectral density into a color spectrum, combining the spatial localization of the channel in the brain, supplementing the values of the rest positions by a spatial interpolation algorithm to obtain a sequence of the electroencephalogram topological graph changing along with time, and completely arranging the electroencephalogram topological graph in one graph according to a time sequence to be used as sample data;
24) inputting the sample data into a capsule neural network for operation;
25) implementing a capsule neural network to perform three iterations by using a dynamic routing algorithm and updating the weight;
26) And outputting the class with the maximum capsule neural network output value as a prediction result for judging whether the error related potential is generated.
Preferably, the SVM support vector machine in step 3) trains through tens of existing feature vector sets of motor imagery on the left and right, trains an SVM binary model by using feature vectors extracted from electroencephalogram data, searches for hyper-parameters (c, g), i.e., a set of optimal penalty parameters c and RBF kernel function width, in a two-dimensional grid form through cross validation, calculates an optimal value of cross validation accuracy of each set of parameters, maximizes the classification accuracy of the SVM classification model by using the highest accuracy set (c, g) as an optimal parameter, outputs 0 representing a left instruction, outputs 1 representing a right instruction, and outputs no change instruction if not.
Motor imagery and error-related negative potentials (ERNs) are important areas of BCI research. The motor imagery is initiated by the brain actively and does not depend on any sensory stimulation, and the signal is generated through the process of the imagery, and the motor imagery comprises two signals of left and right, and is divided into three levels. When the brain performs different motor imagery activities or has performed actual movement, the area where the sensory motor cortex is activated and different rhythm brain electrical signals are changed regularly. The electrophysiological phenomenon related to the motor imagery electroencephalogram signal is the ERD/ERS phenomenon, and when a certain region of the cerebral cortex is in an activated state, the energy of signals in certain frequency bands is reduced, and the electrophysiological phenomenon is called event-related desynchronization (ERD). Conversely, when the brain is in a resting or inactive state, the energy of signals in some frequency bands is increased, which is called event-related synchronization (ERS). When a human body moves a right hand or imagines the action of the right hand, the ERD phenomenon appears on the sensory-motor cortex on the left side of the brain, and the ERS phenomenon appears on the sensory-motor cortex on the right side. When a human body moves left hand or imagines left hand movement, ERS phenomenon appears on the left sensory motor cortex of the brain, and ERD phenomenon appears on the right sensory motor cortex. The error-related negative potential is the negative phase potential shift, called ERN, that is specifically related to the error response recorded in the central area of the scalp of the human brain when the individual perceives the error response, and the classic waveform is shown in FIG. 8.
ERN reflects the process by which the human brain monitors the overall response or reflects the evaluation. The ERN has potential stability, is not influenced by physiological structure change, is generally induced to be generated 100-300 ms after individual perception error, has wave amplitude of about 10uV, low signal-to-noise ratio and large individual difference, and specific typical characteristics of different types of ERN are shown in the table below. The ERN is mainly concentrated on 4-12 Hz, namely theta and mu frequency bands. ERN begins in adolescence and increases in early adulthood, commensurate with the development of other cognitive control abilities, reflecting the subject's level of importance in avoiding errors or responding correctly.
With the maturity of brain-computer interface technology, the intelligent system of the new mode that combines with it is constantly emerging, has included intelligent house, education science and technology, health care, entertainment control etc.. In 2017, science news research and development has developed a BCI-based intelligent home system, and a research result published by the advanced research program office of the United states department of defense shows that a pilot can realize the operation of multiple airplanes and unmanned planes by wearing brain-computer interface equipment. Meanwhile, in the development process of the unmanned intelligent system, people have higher requirements on the capabilities of the unmanned intelligent system in the aspects of autonomous cognition, decision-making, planning, control and the like, and a typical rational intelligent body model ABGP is provided by a Chinese academy of sciences computational technology research institute and other researchers, so that the external perception and the internal state are coordinated together, and the unmanned intelligent system can perceive the environment and reasonably plan the self behaviors. At present, many scholars at home and abroad accumulate some research results on introducing a brain intelligence heuristic intelligent driving system. Some researchers have studied the emotion sensing detection device for the vehicle.
The brain-like processing unit provided by the invention carries out brain-like agent structure design with brain electricity cognitive function on the basis of the ABGP agent model, has the functions of processing signals of certain EEG channels and expressing relevant EEG characteristics, and further controls output to reach a target, and the agent structure is shown in figure 9. The ABGP agent model describes internal activities as four modules, including: the system comprises a perception module, a belief module, a target module and a planning module, wherein perception is cognition of specific external environment states, and when the environment changes along with time, perception also changes along with the change. In the brain-like intelligent agent model, the perception module is realized by using the capsule neural network model, the traditional definition rule is replaced, various parameters of the pre-trained capsule neural network model are stored in a knowledge base of the belief module as beliefs, and the results of other modules are not adjusted. The brain intelligence body realizes self behavior planning through motor drive, and plans and selects the motor drive through the maximum interest of some internal events. The electroencephalogram information acquired by the brain-like intelligent body has the characteristics of nonlinearity, robustness and hierarchy, and the attributes are directly used for forming the knowledge of the belief module, so that the brain-like characteristic perception capability of the intelligent body is stronger.
The invention develops an effective automatic driving auxiliary intelligent system based on the analysis of the electroencephalogram, applies BCI and brain-like agent technologies to the existing automatic driving field, and the electroencephalogram plays an intelligent guiding role in the sudden situation of the driving system, thereby having important significance in the development of the intellectualization of vehicles.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1) the traditional electroencephalogram signal classification method focuses on time-frequency feature extraction of signals, but the problem of dimension disaster often occurs under the condition of analyzing multi-channel data, and a large amount of valuable information is lost in the dimension reduction process. The invention intercepts the original brain waveform data according to a specific time interval, converts the data into a brain electrical topographic map sequence which changes along with time, and then projects the sequence onto a picture, thereby converting the characteristics of time domain, frequency domain and space domain in the brain waveform data into the spatial position relation of each pixel point in the picture, and then identifying and classifying objects into images, thus being capable of adopting a capsule neural network method with good effect.
2) The invention applies the capsule neural network to the classification task of the electroencephalogram signals. The capsule neural network retains positional information between objects in the image and takes into account spatial hierarchical relationships. On the basis of a data set processing mode, by utilizing the sensitivity of the network to the spatial position relation among different features in the image, the high-precision identification task can be completed by using less data sets.
3) The invention combines an automatic driving auxiliary intelligent system with brain inspiring thinking, simulates human cognitive thinking, designs a brain-like intelligent system based on an ABGP intelligent body model and a capsule neural network, establishes a high-speed channel between the intelligent system and an electroencephalogram acquisition system, improves interaction among a plurality of systems and between the systems and electroencephalogram detection, and can better select optimal action planning according to a target.
4) The invention adopts a detection control mode based on a brain-computer interface, overcomes the defects of operation delay, decision error and the like caused by switching between automatic driving and manual driving, is effectively applied to intelligent assistance of automatic driving, and can also be used for detecting the brain physiological state of a driver when the driver takes a vehicle, analyzing and optimizing the riding experience of the driver.
The idea provided by the invention is also suitable for the task of identifying and classifying the electroencephalogram signals in other scenes.
Drawings
Fig. 1 is a flowchart of a brain heuristic automatic driving assistance system based on a capsule neural network according to the present invention.
FIG. 2 is a flow chart of data processing according to an embodiment of the present invention;
FIG. 3 is a diagram of an electroencephalogram cap electrode point bitmap according to an embodiment of the present invention;
FIG. 4 is a diagram of a capsule structure according to an embodiment of the present invention;
FIG. 5 is a diagram of a capsule neural network framework according to an embodiment of the present invention;
FIG. 6(a) is a signal spectrum diagram of an unprocessed signal according to an embodiment of the present invention; FIG. 6(b) is a 8-12 Hz spectrum of a signal after wavelet packet decomposition according to the embodiment of the present invention; FIG. 6(c) is a spectrum diagram of 18 to 30Hz after wavelet packet decomposition of a signal according to the embodiment of the present invention; the horizontal axis of the spectrogram represents frequency, and the vertical axis represents logarithm of signal power spectral density;
FIG. 7(a) is a schematic diagram of the correct-labeled electroencephalogram topographic map of the subject when the negative potential associated with the error is tested, and FIG. 7(b) is a schematic diagram of the subject when the correct-labeled electroencephalogram topographic map of the subject is observed;
FIG. 8 is a typical waveform diagram of ERN;
FIG. 9 is a diagram of a brain-like intelligence system architecture in accordance with the present invention.
In the figure: A. the system comprises an electroencephalogram signal acquisition system, a B error correlation negative potential analysis system, a C motor imagery analysis system, a D automatic navigation control system, a 1 signal acquisition unit, a 2 electroencephalogram signal preprocessing unit, a 3 ERN signal preprocessing unit, a 4 capsule neural network unit, a 5 motor imagery signal preprocessing unit, a 6 signal characteristic extraction unit, a 7 signal classification and identification unit, a 8 real-time positioning unit, a 9 image building unit, a 10 path planning unit and a 11 brain-like processing unit.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the embodiments of the present invention are not limited thereto.
As shown in fig. 1, the invention provides a brain heuristic automatic driving assistance system based on a capsule neural network, which comprises an electroencephalogram signal acquisition system a, an error-related negative potential analysis system B, a motor imagery analysis system C and an automatic navigation control system D.
An electroencephalogram signal acquisition system A: the device is used for collecting electroencephalogram data of a user and sending the electroencephalogram data to an upper computer for processing;
error-dependent negative potential analysis system B: the method comprises the steps that a capsule neural network is used for achieving the function of detecting error-related negative potential ERN, error-related negative potential signals in the brain of a driver are extracted according to electroencephalogram data of a user, and a mark signal is output to reflect the correctness of the current vehicle navigation state;
motor imagery analysis system C: judging the driving intention of the brain of the user according to the electroencephalogram data of the user, and extracting, classifying and generating a left or right instruction;
automatic navigation control system D: the system is used for simulating an automatic driving scene of a vehicle, realizing positioning, mapping and path planning of an unknown scene, receiving a mark signal output by the error-related negative potential analysis system B and an instruction signal output by the motor imagery analysis system C, further calculating the output driving state variable quantity through PID control, and controlling the updating of the vehicle navigation state.
The electroencephalogram signal acquisition system A is composed of a signal acquisition unit 1 and an electroencephalogram signal preprocessing unit 2, wherein the signal acquisition unit 1 acquires potential signals of specific points of the epidermis of the brain of a user, and the electroencephalogram signal preprocessing unit 2 amplifies and filters the acquired electroencephalogram signals.
The electroencephalogram signal acquisition system A adopts an electrode cap based on an international 10-20 system electrode placement method. According to the recommendation of a 10-20 system with 32 electrode positions by the international electroencephalogram society, the electrode positions are named as shown in figure 3, and electroencephalogram signals are acquired through 32 EEG channels of FP 1-O2. In the whole experiment process, an English public physiological data set Mahnob-HCI-TaggingDatabas is adopted for model training, then the electroencephalogram data collected by the system are used for testing, and audio, video, staring data and physiological data are recorded at the same time. And a band-pass filter with the frequency of 0.1-70 Hz and a notch filter with the frequency of 50Hz are used for removing power frequency interference and other noises so as to ensure the quality of the electroencephalogram signals. In 32 EEG channels of the signal acquisition unit 1, signals of the channels of the electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 are used for feature extraction and classification of the error correlation negative potential analysis system B, and signals of the channels of the electrodes C3, C4 and Cz are used for feature extraction of the motor imagery signals in time domain, frequency domain, spatial domain and energy domain by the motor imagery analysis system C.
The error-related negative potential analysis system B comprises an ERN signal preprocessing unit 3 and a capsule neural network unit 4, and is used for carrying out feature extraction and classification by utilizing signals of channels of capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2, and analyzing and judging the generation of error-related negative potentials in real time. The ERN signal preprocessing unit 3 preprocesses the electroencephalogram data, screening, re-referencing, filtering, independent principal component analysis and segmentation of electroencephalogram signals are completed, an electroencephalogram data time domain oscillogram output by the electroencephalogram signal acquisition system A is converted into a time sequence of a electroencephalogram topographic map, a time window is selected, and the electroencephalogram signals are converted into the electroencephalogram topographic map according to a set time interval, as shown in fig. 7 (a) and 7 (b); the capsule neural network unit 4 classifies and extracts the electroencephalogram topographic map output by the ERN signal preprocessing unit 3, the capsule is used as a basic computing unit to construct a capsule network, the capsule is a vector composed of a series of neurons, the value of each neuron represents posture parameters such as scaling, direction and position, the length of the capsule represents the probability of existence of a specific object, meanwhile, the transmission among the capsules is realized through a dynamic routing algorithm, so that the low-layer capsule in the network can predict the activation state of the high-layer capsule, and the capsule structure is shown in fig. 4. And outputting two flag instructions of error reaction and error-free reaction according to whether the ERN signal is extracted or not.
In this embodiment, the ERN signal preprocessing unit 3 intercepts data from 230ms, converts the electroencephalogram signal into a brain electrical map every 30ms until 950ms, acquires 25 time images from each time domain waveform, arranges the time images in the order of 5 × 5, and retains valid information to synthesize the brain electrical map, as shown in fig. 7 (a) and 7 (b).
In this embodiment, as shown in fig. 5, the network framework of the capsule neural network unit 4 mainly includes: 4 convolutional layers, 1~2 capsule layers, 1 input layer, 1 average pooling layer, 1 remoulds the layer, 1 complete articulamentum. As shown in fig. 5, the network model constructed by python includes: the input layer is an array of 32 × 3, the number of the array changes along with the change of the number of samples, the input layer is set as sample data, different input features are extracted through convolution operation, the first layer of convolution layer can only extract some low-level features such as edges, lines, angles and other levels, more layers of networks can extract more complex features from the low-level features in an iteration mode, and the more complex features are output as feature vectors of pictures; the average pooling layer is used for reducing the complexity of data to be processed, further accelerating the convergence efficiency, reducing the operation overhead and improving the resistance of the network model to noise contained in the electroencephalogram topological graph; the remodeling layer converts input data into a format which can be recognized by the capsule neural network, namely remodeling the output of Conv2D to obtain a group of vectors of each position to form an input of the low-dimensional capsule layer; the capsule layer is used for presenting in a capsule form, outputting 2 vectors with 16 dimensions, and each vector represents a classification result; the capsule layer is used for encapsulating information obtained after encoding the electroencephalogram topological graph, the different dimensions of the capsule encode the angle and distance information among the local features of the electroencephalogram topological graph, and the information encoded by the capsule is from the spatial position relationship of the local features to the spatial position relationship of the global features along with the increase of the number of the capsule layers, so that the time sequence information of the electroencephalogram topological graph is contained and encoded in the form of the spatial relationship; the output of the capsule layer is projected into two probability values through the matrix by the full-connection layer to be used as the prediction output of the model, and the class with the larger prediction value is used as the final classification output of the capsule neural network model. And the full connection layer outputs the modulus of 2 vectors as a probability value to represent the possibility of two classification results in the two classification tasks, and the final output result is obtained when the probability value is large. In addition, the number of capsule layers and the dimensionality of the capsule output vector can be adjusted, so that the optimal classification effect is obtained. And (3) outputting a result with a high probability value in the full connection layer of the capsule neural network, wherein if the output is 1, the ERN is not generated, and if the output is 0, the ERN is generated, so that the individual perceives that an error occurs.
The motor imagery analysis system C uses wavelet packet and common spatial mode (CSP) fusion algorithm to select signals of electrodes C3, C4 and Cz channels to perform feature extraction of time domain, frequency domain, spatial domain and energy domain of the motor imagery signals. The wavelet packet extracts the brain wave biorhythm related to the movement, and then a CSP (compact strip service) is utilized to construct a spatial filter, so that the spatial domain characteristic difference of the left and right motor imagery signals is maximized. The motor imagery analysis system C comprises a motor imagery signal preprocessing unit 5, a signal characteristic extraction unit 6 and a signal classification and identification unit 7; the motor imagery signal preprocessing unit 5 completes filtering, segmentation, baseline calibration, re-reference and independent principal component analysis of the electroencephalogram signal to obtain pure electroencephalogram data; the signal feature extraction unit 6 extracts the features of the signals, decomposes the trained electroencephalogram data by utilizing a wavelet packet decomposition algorithm, extracts 8-12 Hz mu rhythm and 18-30 Hz beta rhythm signals, and has power spectrums as shown in FIGS. 6 (a), 6 (b) and 6 (c); processing the extracted corresponding frequency band of the rhythm by using a common space mode CSP algorithm to extract and obtain a highly differentiated characteristic vector; the signal classification and identification unit 7 receives the feature vector signals extracted by the signal feature extraction unit 6, classifies the signals based on an SVM vector machine, and classifies the signals to obtain three instructions of left, right or current situation.
The automatic navigation control system D simulates the automatic driving environment and the technical operation based on the ROS platform. The simulated driving environment and the driving vehicle are designed by an open source model. The simulation vehicle is provided with a camera, a radar and other sensor models, automatic mapping and positioning functions are carried out through a gmapping tool algorithm, and path planning and navigation functions are designed through a navigation tool algorithm. The environmental picture in the vehicle driving simulation process is transmitted to a monitor through a camera and displayed to a subject in real time. The core processing unit adopts a brain-like agent to simulate human cognitive thinking, processes information sensed by a vision sensor, a laser sensor, a gyroscope and the like, and performs comprehensive processing to correct the driving state by combining error-related negative potential signals and motor imagery signals detected by the front end to complete the function of safe auxiliary driving. The automatic navigation control system D comprises a real-time positioning unit 8, a mapping unit 9, a path planning unit 10 and a brain-like processing unit 11. The real-time positioning unit 8 is used for performing positioning calculation through a data model according to data received by the sensor, outputting a conversion relation between a world coordinate system and an odometer coordinate system according to information input by the vision sensor, the laser radar and the gyroscope, and deducing a conversion relation between a vehicle base coordinate system and the world coordinate system to realize a positioning function; the map building unit 9 adopts a gmapping tool algorithm to realize scanning of surrounding environment information, builds an output environment map, and builds a probability-based two-dimensional grid map based on visual sensor information, gyroscope information and odometer information; the path planning unit 10 receives the map, the sensor data flow and the odometer information generated by the map building unit 9 by adopting a navigation tool algorithm, outputs speed and angular speed instructions to the vehicle control chassis, further controls the running state and realizes the functions of path planning and obstacle avoidance; the brain-like processing unit 11 receives the sign signal output by the error-related negative potential analysis system B and the instruction signal output by the motor imagery analysis system C, and deduces and forms a complete multilayer experience map by receiving and memorizing the local acquisition information of the sensor, and maps the map to a real scene to correct the driving state in real time.
Based on the design idea of the brain-like agent, the brain-like processing unit 11 combines the received sign signal output by the error-related negative potential analysis system B and the instruction signal output by the motor imagery analysis system C, and deduces to form a complete multi-layer experience map by receiving and memorizing the local acquisition information of various sensors, and maps the map to a real scene to correct the driving state in real time. When the ERN flag signal is 1, the system remains in the existing state without correction, and when the ERN flag signal is 0, it represents that the ERN signal is generated in the human brain, and the flag signal is used as a higher-level instruction to control the brain processing unit 11. When the signal is 0, the vehicle driving system decelerates immediately, the brain-like processing unit 11 further receives a direction instruction output by the motor imagery analysis system C, performs intensive collection and rapid processing on environmental information of the instruction direction by combining a map and positioning, outputs a PID instruction for controlling four wheel rotation speed control joints and two wheel steering control joints of the vehicle through a PID algorithm after judging a feasible deflection direction, and when the motor imagery analysis system C does not output the direction instruction, the brain-like processing unit 11 does not perform obstacle avoidance and correction operation on the driving state.
Based on the system, the invention also provides a brain heuristic automatic driving assistance method based on the capsule neural network, which comprises the following steps:
1) acquiring electroencephalogram data: the electroencephalogram signal acquisition system A acquires electroencephalogram data of a user and sends the electroencephalogram data to an upper computer for processing; an electrode cap based on an international 10-20 system electrode placement method collects electroencephalograms of a user through 32 EEG channels including FP 1-O2, and a band-pass filter with the frequency of 0.1-70 Hz and a notch filter with the frequency of 50Hz are used for removing power frequency interference and other noises so as to ensure the quality of the electroencephalograms;
the EEG signal acquisition system A requires a tested driver to react to the running environment condition in a picture at any time when facing the monitor picture of the automatic navigation control system in the signal acquisition process. The test subject can completely receive the surrounding vision of the simulated vehicle running, and the electrode cap is worn to ensure the real-time acquisition and transmission of each electrode signal.
2) Error-dependent negative potential analysis: the error-related negative potential analysis system B utilizes signals of channels of capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 to extract features and classify, extracts error-related negative potential signals in the brain of a driver according to electroencephalogram data of a user, analyzes and judges the generation of error-related negative potentials in real time, and outputs a mark signal;
3) And (3) outputting a motion direction instruction: the motor imagery analysis system C uses a wavelet packet and common space mode fusion algorithm to select signals of electrodes C3, C4 and a Cz channel to perform time domain, frequency domain, space domain and energy domain feature extraction on the motor imagery signals, an SVM (support vector machine) is used for classification, three instructions of left, right or keeping the current situation are obtained through classification, and instruction signals are output;
4) controlling the vehicle navigation: the automatic navigation control system D simulates the automatic driving scene of the vehicle, realizes positioning, mapping and path planning of an unknown scene, receives the sign signal output in the step 2) and the instruction signal output in the step 3), further calculates the output driving state variable quantity through PID control, and controls the updating of the vehicle navigation state.
The specific steps of the capsule neural network for feature extraction and classification in the step 2) comprise:
21) collecting electroencephalogram signals of a plurality of channels, and preprocessing original signals; the studied EEG signal channel is an EEG signal of 11 channels in the middle of the frontal lobe of the brain, and the preprocessing comprises 4-16Hz band-pass filtering, re-referencing, independent principal component ICA analysis and time period interception.
22) Intercepting the signal waveform according to a fixed time interval, and calculating corresponding power spectral density; the waveform signal is truncated at 30ms intervals and the start and stop times of truncation are 230ms and 950ms, so as to maximize the searchable time span while preserving the original sequence continuity.
23) The power spectral density is normalized and mapped into a color spectrum, the values of the rest positions are supplemented through a spatial interpolation algorithm by combining the spatial positioning of the channel in the brain, and a sequence of the electroencephalogram topological graph changing along with time is obtained. Completely arranging the electroencephalogram topological graphs in one graph according to a time sequence to serve as sample data; the formula of the spatial interpolation algorithm is as follows:
where X is the position of the point to be interpolated, a, b, … …, p represents the power value for each acquisition point, and XA, XB, XP represents the distance of the point to each acquisition point to be interpolated. The power values of the removed channels, and other unconnected electrode areas, are predicted using a spatial interpolation algorithm. And predicting other areas based on data of a small number of channels, so that the signal-to-noise ratio is improved, and the convergence speed of model training is accelerated.
24) Inputting the sample data into a capsule neural network for operation; the network structure of the capsule neural network unit 4 includes: 4 layers of convolution layers, 1 layer of average pooling layer, 1-2 capsule layers and 1 layer of full-connecting layer. The capsule structure is shown in fig. 4, and the matrix multiplication formula of the input vector in the capsule neural network is as follows:
wherein the content of the first and second substances,u i is the output of the neural network of the previous layer of capsule,W ij is a weight matrix used for coding important spatial relationship between lower layer characteristics and upper layer characteristics or Other relationships, which can be viewed as different degrees of association of upper neurons to lower neurons per capsule.
Is a prediction vector that represents the predicted output of the previous layer of capsules trying to predict the next layer of capsules.
Wherein the capsule neural network input vector is weighted and summed S j The calculation formula is as follows:
C ij is a coupling coefficient ofb ij The calculation results in that,b ij calculated by the updating of the dynamic routing algorithm of the capsule network core.
25) The capsule layer uses a dynamic routing algorithm to carry out three iterations and update the weight:
wherein, VjIs the output vector of the capsule of the previous layer,
new weight values for the previous layer of capsules trying to predict the output of the next layer of capsulesb ij By mixingV jAnd
dot product and add the original weightb ij Thus obtaining the product. The weights obtained in the previous two iterationsb ij Weights for different capsules representing the encoding of information characterizing the different capsules; the last iteration is tob ij The weighting for all the different dimensions of the capsule represents the encoding of the information characterized by the different dimensions in the capsule.
Wherein, the nonlinear proper compression function in the capsule neural network is as follows:
wherein the content of the first and second substances,V jis the output vector of the first capsule,S jis the input vector of the first capsule, | purpleS jThe method comprises the following steps of obtaining direction information of an input vector by using the square () as an activation function, compressing the mould of the input vector between 0 and 1, representing the probability of occurrence of a certain event, wherein the higher the value is, the higher the probability of occurrence is.
Dividing sample data into a training set and a test set according to the proportion of 7:1, and using the training set to train the generalization capability of the capsule neural network model and the test set evaluation model.
26) And outputting the class with the maximum capsule neural network output value as a prediction result for judging whether the error related potential is generated.
In the step 3), the SVM support vector machine is trained by tens of existing left and right motor imagery feature vector sets, an SVM two-class model is trained by using feature vectors extracted from electroencephalogram data, and through cross validation, hyper-parameters (c and g), namely a group of optimal punishment parameters c and RBF kernel function width, are searched in a two-dimensional grid mode, the optimal value of cross validation accuracy of each group of parameters is calculated, the highest accuracy group (c and g) is used as the optimal parameter, so that the classification accuracy of the SVM class model is maximized, the SVM support vector machine outputs 0 to represent a left instruction, outputs 1 to represent a right instruction, and does not output a change instruction.
It should be understood that the above description of the preferred embodiments is illustrative, and not restrictive, and that various changes and modifications may be made therein by those skilled in the art without departing from the scope of the invention as defined in the appended claims.
Claims (7)
1. A enlightening-type automatic driving auxiliary system based on capsule neural network is characterized in that: the system comprises an electroencephalogram signal acquisition system A, an error-related negative potential analysis system B, a motor imagery analysis system C and an automatic navigation control system D;
the electroencephalogram signal acquisition system A: the device is used for collecting electroencephalogram data of a user and sending the electroencephalogram data to an upper computer for processing;
the error-dependent negative potential analysis system B: the method comprises the steps of realizing the detection function of error-related negative potential ERN by using a capsule neural network, extracting error-related negative potential signals in the brain of a driver according to electroencephalogram data of a user, and outputting two mark instruction signals of error reaction and no error reaction so as to reflect the correctness of the current vehicle navigation state; the error-related negative potential analysis system B comprises an ERN signal preprocessing unit (3) and a capsule neural network unit (4), wherein the ERN signal preprocessing unit (3) preprocesses electroencephalogram data, screening, re-referencing, filtering, independent principal component analysis and segmentation of the electroencephalogram data are completed, a time window is selected, the electroencephalogram data are converted into electroencephalogram terrain maps according to a set time interval, the capsule neural network unit (4) classifies and extracts the electroencephalogram terrain maps output by the ERN signal preprocessing unit (3), and two mark instruction signals of error reaction and no error reaction are output according to whether the ERN signal is extracted or not;
The motor imagery analysis system C: judging the driving intention of the brain of the user according to the electroencephalogram data of the user:
the motor imagery analysis system C comprises a motor imagery signal preprocessing unit (5), a signal feature extraction unit (6) and a signal classification and identification unit (7); the motor imagery signal preprocessing unit (5) completes filtering, segmentation, baseline calibration, re-reference and independent principal component analysis of the electroencephalogram data to obtain pure electroencephalogram data; the signal feature extraction unit (6) is used for extracting the features of signals from the electroencephalogram data so as to extract and obtain highly differentiated feature vectors; the signal classification and identification unit (7) receives the feature vector signals extracted by the signal feature extraction unit (6), classifies the signals based on an SVM vector machine, and classifies the signals to obtain three instructions of left, right or current situation;
the automatic navigation control system D: the method is realized based on an ROS simulation platform, is used for simulating an automatic driving scene of a vehicle, realizing positioning, drawing and path planning of an unknown scene, receiving two mark command signals of error reaction and no error reaction output by the error-related negative potential analysis system B and a command signal output by the motor imagery analysis system C, further calculating the output driving state variable quantity through PID control, and controlling the updating of the vehicle navigation state.
2. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 1, wherein: the electroencephalogram signal acquisition system A is composed of a signal acquisition unit (1) and an electroencephalogram signal preprocessing unit (2), the signal acquisition unit (1) acquires potential signals of specific point positions of the epidermis of the brain of a user, and the electroencephalogram signal preprocessing unit (2) amplifies and filters the acquired electroencephalogram signals.
3. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 2, wherein: the electroencephalogram signals are acquired through 32 EEG channels including FP 1-O2 by a signal acquisition unit (1), signals of channels including FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 are used for a fault-related negative potential analysis system B to perform feature extraction and classification, and signals of channels including C3, C4 and Cz are used for a motor imagery analysis system C to perform feature extraction of time domain, frequency domain, spatial domain and energy domain of motor imagery signals.
4. The brain heuristic automatic driving assistance system based on the capsule neural network as claimed in claim 1, wherein: the automatic navigation control system D comprises a real-time positioning unit (8), a mapping unit (9), a path planning unit (10) and a brain-like processing unit (11); the real-time positioning unit (8) is used for performing positioning calculation through a data model according to data received by the sensor, outputting a conversion relation between a world coordinate system and a odometer coordinate system according to information input by the vision sensor, the laser radar and the gyroscope, and deducing the conversion relation between a vehicle base coordinate system and the world coordinate system to realize a positioning function; the map building unit (9) adopts a mapping tool algorithm to realize the scanning of surrounding environment information, builds an output environment map, and builds a probability-based two-dimensional grid map based on the visual sensor information, the gyroscope information and the odometer information; the path planning unit (10) receives the map, the sensor data stream and the odometer information generated by the map building unit (9) by adopting a navigation toolkit set, outputs speed and angular speed instructions to the vehicle control chassis, further controls the driving state and realizes the functions of path planning and obstacle avoidance; the brain-like processing unit (11) receives the mark instruction signal output by the error-related negative potential analysis system B and the instruction signal output by the motor imagery analysis system C, and by receiving and memorizing the local acquisition information of the sensor, a complete multilayer experience map is formed by inference and is mapped to a real scene, and the driving state is corrected in real time.
5. A enlightening-typed autopilot auxiliary method based on a capsule neural network, which is realized based on the enlightening-typed autopilot auxiliary system based on the capsule neural network of any one of claims 1 to 4, and is characterized in that: the method comprises the following steps:
1) acquiring electroencephalogram data: the electroencephalogram signal acquisition system A acquires electroencephalogram data of a user and sends the electroencephalogram data to an upper computer for processing; an electrode cap based on an international 10-20 system electrode placement method carries out user electroencephalogram signal acquisition through 32 EEG channels including FP 1-O2, and a band-pass filter with the frequency of 0.1-70 Hz and a 50Hz notch filter are used for removing power frequency interference and other noises so as to ensure the quality of electroencephalogram signals;
2) error-dependent negative potential analysis: the error-related negative potential analysis system B utilizes electroencephalogram data of capsule neural network selection electrodes FP1, F3, F7, FC5, FC1, FP2, F7, F4, F8, FC6 and FC2 channels to carry out feature extraction and classification: the ERN signal preprocessing unit (3) preprocesses the electroencephalogram data, screening, re-referencing, filtering, independent principal component analysis and segmentation of the electroencephalogram data are completed, a time window is selected, the electroencephalogram data are converted into electroencephalogram terrain maps according to a set time interval, the capsule neural network unit (4) classifies and extracts the electroencephalogram terrain maps output by the ERN signal preprocessing unit (3), and two mark instruction signals of error reaction and no error reaction are output according to whether the ERN signal is extracted or not;
3) And (3) outputting a motion direction instruction: the motor imagery analysis system C uses a wavelet packet and common space mode fusion algorithm to select signals of electrodes C3, C4 and a CZ channel to extract characteristics of a time domain, a frequency domain, a space domain and an energy domain of the motor imagery signals, an SVM (support vector machine) is used for classification, three instructions of left, right or keeping the current situation are obtained through classification, and instruction signals are output;
4) controlling the vehicle navigation: the automatic navigation control system D simulates the automatic driving scene of the vehicle, realizes positioning, mapping and path planning of an unknown scene, receives the sign instruction signals with error reaction and error-free reaction output in the step 2) and the instruction signals output in the step 3), further calculates the variation of the output driving state through PID control, and controls the updating of the navigation state of the vehicle.
6. The brain heuristic automatic driving assistance method based on the capsule neural network as claimed in claim 5, characterized in that: the specific steps of the capsule neural network for feature extraction and classification in the step 2) comprise:
21) collecting electroencephalogram signals of a plurality of channels, and preprocessing the electroencephalogram signals;
22) intercepting a signal waveform according to a fixed time interval, and calculating corresponding power spectral density;
23) Normalizing the power spectral density and mapping the power spectral density into a color spectrum, combining the spatial positioning of the channel in the brain, supplementing the values of the rest positions by a spatial interpolation algorithm to obtain a sequence of the brain electrical topographic map changing along with time, and completely arranging the brain electrical topographic map in one map according to a time sequence to be used as sample data;
24) inputting the sample data into a capsule neural network for operation;
25) implementing a capsule neural network to perform three iterations by using a dynamic routing algorithm and updating the weight;
26) and outputting the class with the maximum capsule neural network output value as a prediction result for judging whether the error related potential is generated.
7. The brain heuristic automatic driving assistance method based on the capsule neural network as claimed in claim 5, characterized in that: the SVM support vector machine in the step 3) is trained by tens of existing motion imagery feature vector sets on the left and right, an SVM binary model is trained by using feature vectors extracted from electroencephalogram data, and through cross validation, hyper-parameters (c, g), namely a group of optimal punishment parameters c and RBF kernel function width, are searched in a two-dimensional grid mode, the optimal value of the cross validation precision of each group of parameters is calculated, the highest precision group (c, g) is used as the optimal parameter, so that the classification accuracy of the SVM classification model is maximized, the SVM support vector machine outputs 0 to represent a left instruction, outputs 1 to represent a right instruction, and does not output a no-change instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221486.6A CN114312819B (en) | 2022-03-09 | 2022-03-09 | Brain heuristic type automatic driving assistance system and method based on capsule neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221486.6A CN114312819B (en) | 2022-03-09 | 2022-03-09 | Brain heuristic type automatic driving assistance system and method based on capsule neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114312819A CN114312819A (en) | 2022-04-12 |
| CN114312819B true CN114312819B (en) | 2022-06-28 |
Family
ID=81033022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210221486.6A Active CN114312819B (en) | 2022-03-09 | 2022-03-09 | Brain heuristic type automatic driving assistance system and method based on capsule neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114312819B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012095905A (en) * | 2010-11-04 | 2012-05-24 | Panasonic Corp | Brain wave measurement system, method, and computer program |
| CN104083258A (en) * | 2014-06-17 | 2014-10-08 | 华南理工大学 | Intelligent wheel chair control method based on brain-computer interface and automatic driving technology |
| CN107065850A (en) * | 2016-12-12 | 2017-08-18 | 东南大学 | The four motorized wheels intelligent car systems and its control method of a kind of brain wave control |
| CN109299647A (en) * | 2018-07-24 | 2019-02-01 | 东南大学 | A kind of extraction of multitask Mental imagery brain electrical feature and mode identification method towards vehicle control |
| CN109658503A (en) * | 2018-12-29 | 2019-04-19 | 北京理工大学 | A kind of driving behavior intention detection method merging EEG signals |
| CN110584597A (en) * | 2019-07-15 | 2019-12-20 | 天津大学 | Multi-channel electroencephalogram signal monitoring method based on time-space convolutional neural network and application |
| CN111329497A (en) * | 2020-02-21 | 2020-06-26 | 华南理工大学 | Wearable fatigue driving monitoring system and method based on forehead electroencephalogram signals |
| CN111783942A (en) * | 2020-06-08 | 2020-10-16 | 北京航天自动控制研究所 | Brain cognition process simulation method based on convolution cyclic neural network |
| CN112141116A (en) * | 2019-06-26 | 2020-12-29 | 现代自动车株式会社 | Method and apparatus for controlling moving body using error monitoring |
| CN112784397A (en) * | 2019-10-22 | 2021-05-11 | 现代自动车株式会社 | Device and method for generating a driving model for the proficiency of a driver using error monitoring |
| CN113002556A (en) * | 2019-12-03 | 2021-06-22 | 现代自动车株式会社 | Apparatus and method for providing customized driving path of moving body using brain wave signal |
-
2022
- 2022-03-09 CN CN202210221486.6A patent/CN114312819B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012095905A (en) * | 2010-11-04 | 2012-05-24 | Panasonic Corp | Brain wave measurement system, method, and computer program |
| CN104083258A (en) * | 2014-06-17 | 2014-10-08 | 华南理工大学 | Intelligent wheel chair control method based on brain-computer interface and automatic driving technology |
| CN107065850A (en) * | 2016-12-12 | 2017-08-18 | 东南大学 | The four motorized wheels intelligent car systems and its control method of a kind of brain wave control |
| CN109299647A (en) * | 2018-07-24 | 2019-02-01 | 东南大学 | A kind of extraction of multitask Mental imagery brain electrical feature and mode identification method towards vehicle control |
| CN109658503A (en) * | 2018-12-29 | 2019-04-19 | 北京理工大学 | A kind of driving behavior intention detection method merging EEG signals |
| CN112141116A (en) * | 2019-06-26 | 2020-12-29 | 现代自动车株式会社 | Method and apparatus for controlling moving body using error monitoring |
| CN110584597A (en) * | 2019-07-15 | 2019-12-20 | 天津大学 | Multi-channel electroencephalogram signal monitoring method based on time-space convolutional neural network and application |
| CN112784397A (en) * | 2019-10-22 | 2021-05-11 | 现代自动车株式会社 | Device and method for generating a driving model for the proficiency of a driver using error monitoring |
| CN113002556A (en) * | 2019-12-03 | 2021-06-22 | 现代自动车株式会社 | Apparatus and method for providing customized driving path of moving body using brain wave signal |
| CN111329497A (en) * | 2020-02-21 | 2020-06-26 | 华南理工大学 | Wearable fatigue driving monitoring system and method based on forehead electroencephalogram signals |
| CN111783942A (en) * | 2020-06-08 | 2020-10-16 | 北京航天自动控制研究所 | Brain cognition process simulation method based on convolution cyclic neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114312819A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230154207A1 (en) | Driver fatigue detection method and system based on combining a pseudo-3d convolutional neural network and an attention mechanism | |
| Almogbel et al. | EEG-signals based cognitive workload detection of vehicle driver using deep learning | |
| CN110377049B (en) | Brain-computer interface-based unmanned aerial vehicle cluster formation reconfiguration control method | |
| CN108537198A (en) | A kind of analysis method of the driving habit based on artificial intelligence | |
| Lu et al. | XGBoost algorithm-based monitoring model for urban driving stress: Combining driving behaviour, driving environment, and route familiarity | |
| DE102019105363B4 (en) | Procedure for a measuring system in the vehicle for the detection and classification of objects in the area around the vehicle using a deep learning procedure with a self-learning decision maker | |
| CN113693613A (en) | Electroencephalogram signal classification method and device, computer equipment and storage medium | |
| WO2018171875A1 (en) | Control device, system and method for determining the perceptual load of a visual and dynamic driving scene | |
| Peng et al. | 3D-STCNN: Spatiotemporal Convolutional Neural Network based on EEG 3D features for detecting driving fatigue | |
| CN106022291A (en) | Method of detecting braking intention of driver in emergency state based on neural signal | |
| Wei et al. | Driver's mental workload classification using physiological, traffic flow and environmental factors | |
| CN114312819B (en) | Brain heuristic type automatic driving assistance system and method based on capsule neural network | |
| Mora et al. | Simplified Prediction Method for Detecting the Emergency Braking Intention Using EEG and a CNN Trained with a 2D Matrices Tensor Arrangement | |
| Chen et al. | A Transfer Learning-Based CNN Deep Learning Model for Unfavorable Driving State Recognition | |
| Shi et al. | Indoor space target searching based on EEG and EOG for UAV | |
| CN115202479A (en) | Man-machine co-driving simulation system of man-in-loop and application thereof | |
| CN114120634A (en) | Dangerous driving behavior identification method, device, equipment and storage medium based on WiFi | |
| Zhang et al. | ECMER: Edge-Cloud Collaborative Personalized Multimodal Emotion Recognition Framework in the Internet of Vehicles | |
| CN113191324A (en) | Pedestrian behavior intention prediction method based on multi-task learning | |
| QU et al. | Multi-Attention Fusion Drowsy Driving Detection Model | |
| Wu et al. | EEG-based fuzzy neural network estimator for driving performance | |
| KR102585149B1 (en) | Customized self-driving devices and methods based on deep learning using cognitive characteristics | |
| CN117137498A (en) | Emergency situation detection method based on attention orientation and exercise intention electroencephalogram | |
| Memmesheimer | On the recognition of human activities and the evaluation of its imitation by robotic systems | |
| Bano et al. | Drivers Stress Identification in Real-World Driving Tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |