CN113501005A - Method and device for assisting control of vehicle based on physiological information of driver - Google Patents
Method and device for assisting control of vehicle based on physiological information of driver Download PDFInfo
- Publication number
- CN113501005A CN113501005A CN202110923088.4A CN202110923088A CN113501005A CN 113501005 A CN113501005 A CN 113501005A CN 202110923088 A CN202110923088 A CN 202110923088A CN 113501005 A CN113501005 A CN 113501005A
- Authority
- CN
- China
- Prior art keywords
- physiological information
- driver
- intention
- signal
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W50/08—Interaction between the driver and the control system
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/0205—Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/389—Electromyography [EMG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/725—Details of waveform analysis using specific filters therefor, e.g. Kalman or adaptive filters
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
- B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2503/00—Evaluating a particular growth phase or type of persons or animals
- A61B2503/20—Workers
- A61B2503/22—Motor vehicles operators, e.g. drivers, pilots, captains
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
- B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
- B60W2040/0872—Driver physiology
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2540/00—Input parameters relating to occupants
- B60W2540/221—Physiology, e.g. weight, heartbeat, health or special needs
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Veterinary Medicine (AREA)
- Heart & Thoracic Surgery (AREA)
- Public Health (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Animal Behavior & Ethology (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Psychiatry (AREA)
- Artificial Intelligence (AREA)
- Psychology (AREA)
- Physiology (AREA)
- Signal Processing (AREA)
- Developmental Disabilities (AREA)
- Child & Adolescent Psychology (AREA)
- Social Psychology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Hospice & Palliative Care (AREA)
- Educational Technology (AREA)
- Mathematical Physics (AREA)
- Automation & Control Theory (AREA)
- Mechanical Engineering (AREA)
- Evolutionary Computation (AREA)
- Transportation (AREA)
- Fuzzy Systems (AREA)
- Cardiology (AREA)
- Human Computer Interaction (AREA)
- Pulmonology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The present invention relates to the field of assisted driving of vehicles. The invention provides a method for assisting control of a vehicle based on physiological information of a driver, which comprises the following steps: s1: acquiring first physiological information of a driver; s2: acquiring second physiological information of the driver, wherein the second physiological information is different from the first physiological information; s3: generating a control signal for the vehicle in a cross-validated manner based on the first physiological information and the second physiological information; and, S4: and outputting the control signal to control the driving operation of the vehicle. The invention also provides an apparatus for assisting in controlling a vehicle based on physiological information of a driver and a computer program product. In the invention, through comprehensively considering two different types of physiological information of the driver, the identification result can be cross-verified by means of the correlation between the two types of physiological information, thereby favorably making up the defects in the analysis process of single biological information and enhancing the stability and reliability of the system.
Description
Technical Field
The invention relates to a method for assisting in controlling a vehicle based on physiological information of a driver, an apparatus for assisting in controlling a vehicle based on physiological information of a driver, and a computer program product.
Background
Along with the continuous deepening of the vehicle intelligent process, various human-computer interaction functions carried by the vehicle make contributions to safe driving and comfort improvement of a driver. However, in certain scenarios, relying on existing auxiliary prompting and early warning measures is still insufficient to completely circumvent the accident risk for the driver. For example, when a driver is exposed to an emergency hazard, sometimes it is not time to manually control the vehicle to effectively avoid the obstacle, and although some biological information present at this time already reflects the driver's braking intention, it is often too late when he actually performs a specific operation. In addition, it is difficult for a person with insufficient physical ability or a poor driving experience to safely drive the vehicle.
Currently, a method for determining a driving intention of a vehicle user using an Electroencephalogram signal is proposed in the related art, in which a corresponding driving behavior is determined by collecting an Electroencephalogram (EEG) of the vehicle driver.
Further, a method of detecting an emergency braking intention based on an Electromyogram (EMG) is also known, and in a virtual driving environment, an intention feedback when the driver performs emergency braking can be detected from the muscle activity of the driver.
However, the above solutions still have many disadvantages, and particularly, currently, the exercise intention of the user is usually detected based on a single bioelectric signal, but although the single muscle electric signal responds to the exercise intention quickly, the myoelectric signal has large individual variability and has a significant time-varying characteristic. The single electroencephalogram signal is a comprehensive reflection of central nervous activity on cerebral cortex, is weaker and is easy to interfere, and meanwhile, the frequency of a region generated by the signal is strongly related to the type of motor imagery, so that specific human intentions cannot be accurately analyzed according to the signal. Therefore, the motor intention recognition using single physiological information has a significant disadvantage.
Under such circumstances, it is desirable to provide a driving intention recognition scheme that fuses a plurality of kinds of physiological information and supports mutual verification to assist vehicle control more reliably.
Disclosure of Invention
The present invention aims to provide a method for assisting control of a vehicle based on physiological information of a driver, an apparatus for assisting control of a vehicle based on physiological information of a driver and a computer program product, which solve at least some of the problems in the prior art.
According to a first aspect of the present invention, a method for assisting control of a vehicle based on physiological information of a driver is presented, the method comprising the steps of:
s1: acquiring first physiological information of a driver;
s2: acquiring second physiological information of the driver, wherein the second physiological information is different from the first physiological information;
s3: generating a control signal for the vehicle in a cross-validated manner based on the first physiological information and the second physiological information; and
s4: and outputting the control signal to control the driving operation of the vehicle.
The invention comprises in particular the following technical concepts: by comprehensively considering two different types of physiological information of the driver, the identification result can be cross-verified by means of the correlation between the two types of physiological information, so that the defects in the analysis process of single biological information are favorably overcome, and the stability and the reliability of the system are enhanced. Thereby, the time delay of the human intervention system in the event of an emergency situation as a whole can be reduced.
Optionally, the first physiological information includes an electroencephalogram signal of the driver, and the second physiological information includes an electromyogram signal and/or a surface electromyogram signal of the driver.
Thereby, the following technical advantages are achieved: the electroencephalogram signals are the overall reflection of the electrophysiological activity of brain nerve cells on the surface of the cerebral cortex or the scalp, contain rich intention information and have good time advance, can be used for sensing the movement intention of a human body in advance, and solve the problem of time delay of other physiological information sources. The myoelectric signal is the superposition of a plurality of muscle fiber movements on time and space, and the surface myoelectric signal is an electric signal generated on the surface layer of skin when muscles contract and is quick to respond to movement consciousness. By combining the two key information sources, the advantages of different physiological information are fully exerted, and respective defects are mutually compensated.
Optionally, the step S3 includes:
s31: fusion finding a target driving intention of the driver based on the first physiological information and the second physiological information; and
s32: generating a control signal according to a target driving intention, wherein the control signal is used for triggering vehicle operation associated with the target driving intention.
Optionally, the step S31 includes:
outputting potential driving intentions of the driver for the first physiological information and the second physiological information respectively by means of a machine learning model;
combining all potential driving intents to form a candidate intent sequence; and
and screening out the target driving intention from the candidate intention sequence according to the time correlation and/or the content correlation between the first physiological information and the second physiological information.
Thereby, the following technical advantages are achieved: firstly, potential driving intentions corresponding to two kinds of physiological information are detected independently and parallelly through a signal processing algorithm and a pattern recognition algorithm, however, the two kinds of physiological information respectively show insufficiency in an analysis process, and therefore detection results obtained at the stage may not be accurate. Then, by extracting the commonalities of the two signals in time and content in a comparative verification mode and determining a final recognition result according to the commonalities, accurate detection of driving intention is achieved.
Optionally, screening the target driving intent comprises:
assigning a confidence probability to each potential driving intent in the sequence of candidate intentions by means of a pre-trained artificial neural network; and
and outputting the potential driving intention with the highest confidence probability as the target driving intention.
Thereby, the following technical advantages are achieved: by means of the division of the confidence intervals and the distribution of the confidence degrees, the comprehensive recognition result of the combined brain-muscle movement intention can be more clearly screened out.
Optionally, the step S3 further includes: and respectively judging whether the first physiological information and the second physiological information reflect the movement intention, wherein the first physiological information and the second physiological information are used for generating the control signal only under the condition that the first physiological information and the second physiological information both reflect the movement intention.
Thereby, the following technical advantages are achieved: with the filtering condition in the form of the movement intention, it is possible to exclude the disturbance term before forming a specific vehicle control.
Optionally, the determination of the exercise intention is performed jointly for first physiological information acquired at a first time and second physiological information acquired at a second time, the second time being delayed by a predefined time period with respect to the first time.
Thereby, the following technical advantages are achieved: when the driving intention is generated in the driver's brain, excitation signals are often generated first in the cerebral cortex before control information generated by the brain is transmitted to the muscles. Therefore, time deviation usually exists between different physiological information generated by the same driving intention, and the relation of functional coupling between cortical muscles can be reflected only by the synchronous characteristics between the electroencephalogram signal and the electromyogram signal. By predefining and taking such time deviations into account in the analysis, the correspondence between different physiological signals can be tightly constrained.
Optionally, the step S3 further includes: and respectively carrying out signal preprocessing and time-frequency domain signal feature extraction on the first physiological information and the second physiological information.
Thereby, the following technical advantages are achieved: the method can effectively remove artifacts and noise in corresponding physiological information, extract useful information from the original signal, and improve the accuracy of the subsequent intention identification process.
According to a second aspect of the present invention, there is provided an apparatus for assisting control of a vehicle based on physiological information of a driver, the apparatus being configured to perform the method according to the first aspect of the present invention, the apparatus comprising:
the driver management system comprises a first acquisition module, a second acquisition module and a management module, wherein the first acquisition module is configured to be capable of acquiring first physiological information of a driver;
a second acquisition module configured to be capable of acquiring second physiological information of the driver, the second physiological information being different from the first physiological information;
an analysis module configured to be able to generate a control signal for the vehicle in a cross-validated manner based on the first physiological information and the second physiological information; and
an output module configured to be capable of outputting the control signal for controlling a driving operation of a vehicle.
According to a third aspect of the present invention, a computer program product is presented, wherein the computer program product comprises a computer program for implementing the method according to the first aspect of the present invention when executed by a computer.
Drawings
The principles, features and advantages of the present invention may be better understood by describing the invention in more detail below with reference to the accompanying drawings. The drawings comprise:
fig. 1 shows a block diagram of an apparatus for assisting in controlling a vehicle based on physiological information of a driver according to an exemplary embodiment of the present invention;
FIG. 2 illustrates a flow chart of a method of assisting in controlling a vehicle based on physiological information of a driver, according to an exemplary embodiment of the present invention;
FIG. 3 shows a flow chart of three method steps of a method of assisting control of a vehicle based on physiological information of a driver, according to an exemplary embodiment of the invention;
FIG. 4 shows a flowchart of a method of assisting in controlling a vehicle based on physiological information of a driver, according to another exemplary embodiment of the present invention; and
fig. 5 shows a schematic diagram of a recurrent neural network architecture employed in the method of the present invention to enable cross-validation of first and second physiological information.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings and exemplary embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of the invention.
Fig. 1 shows a block diagram of an apparatus for assisting in controlling a vehicle based on physiological information of a driver according to an exemplary embodiment of the present invention.
As shown in fig. 1, the apparatus 1 is used, for example, for assisting control of a vehicle based on physiological information of a driver. The device 1 comprises, for example, a first acquisition module 10, a second acquisition module 20, an analysis module 30 and an output module 40. The first acquisition module 1 is configured, for example, as a head-mounted electroencephalogram detector (e.g., an electrode cap) configured to acquire electroencephalogram signals generated at the cerebral cortex in an uninterrupted manner according to changes in electrode potentials at different channels of the driver's head. The second detection module 2 is configured, for example, as a muscle electrical signal detector, which may be arranged, for example, on different parts of the driver's body (for example, on the wrist, ankle, arm, leg, neck, etc.) as required and is configured to detect the electromyographic signals or surface electromyographic signals of the driver in real time. The electromyographic signals are a temporal and spatial superposition of the action potentials of the motor units in the plurality of muscle fibers, and for the acquisition of these signals, they can be acquired invasively, for example, by means of needle electrodes. The surface electromyogram signal is the sum effect of the electrical activity of superficial muscles and nerve trunks on the surface of the skin, can reflect the activity of the neuromuscular to a certain extent, and has the advantages of no wound and simple operation compared with the electromyogram signal in the measurement mode. After collecting the above-mentioned physiological information of the driver, these are further transmitted to the analysis module 30.
The analysis module 30 is configured to generate a control signal for the vehicle in a cross-validated manner based on the acquired brain electrical signals and muscle electrical signals. For this purpose, the analysis module 30 comprises, for example, a first signal processing unit 31, a second signal processing unit 32 and a signal fusion unit 33. The first signal processing unit 31 receives the electroencephalogram signal of the driver from the first acquisition module 1 and performs signal processing thereon, which includes preprocessing of the electroencephalogram signal, feature extraction, intensity determination, and potential driving intention finding, for example. Furthermore, the evaluation module 30 comprises a second signal processing unit 32, which second signal processing unit 32 receives the electromyographic signals and/or the surface electromyographic signals of the driver from the second acquisition module 2 and processes them there in a similar manner to the processing in the first signal processing unit 31.
Here, the first signal processing unit 31 and the second signal processing unit 32 are both connected to the signal fusion unit 33. The signal fusion unit 33 is configured, for example, as a recurrent neural network in order to receive a sequence of candidate intentions formed by a combination of potential driving intentions from the first and second signal processing units 31, 32, respectively, and to filter out the target driving intention therefrom.
The screened target driving intent is provided to the output module 40 and is sent by the output module 40 to at least one actuator of the vehicle (e.g., an accelerator pedal, a brake pedal, a steering device, etc.) to control the driving behavior of the vehicle accordingly.
Fig. 2 shows a flowchart of a method for assisting in controlling a vehicle based on physiological information of a driver according to an exemplary embodiment of the present invention. The method can be carried out, for example, using the device 1 shown in fig. 1.
Physiological information in the sense of the present invention is understood as a phenomenon of the functioning of a living organism which is manifested by differences in the basic physiological structure, age, character, etc. of an individual. In the field of driving intention recognition, physiological information includes, for example, information of different forms such as an electroencephalogram signal, a skin electrical effect, an electrocardiographic signal, an eye movement, respiration, a blood vessel pressure, and the like.
In step S1, first physiological information of the driver is acquired. In this case, the driver's electroencephalogram signal is received in an uninterrupted manner, for example by means of brain-computer interface technology.
In step S2, second physiological information of the driver, which is different from the first physiological information, is acquired. In this case, for example, a surface electromyographic signal of the driver is received in real time by means of a wrist surface patch electrode.
In step S3, a control signal for the vehicle is generated in a cross-validated manner for the first and second physiological information.
Here, for example, first, in sub-step S31 of step S3, the target driving intention of the driver is fusion-found based on the first and second physiological information. In the sense of the present invention, a target driving intention is understood, for example, as a control intention of a specific vehicle behavior exhibited by a driver before a specific driving operation is performed, which includes, for example: braking intent, acceleration intent, steering intent, merge intent, follow intent, and the like.
Next, a control signal is generated according to the target driving intention in sub-step S32 of step S3. Here, the control signal is used, for example, to trigger a vehicle operation associated with the target driving intention.
In step S4, a control signal is output for controlling the driving operation of the vehicle.
Fig. 3 shows a flowchart of three steps S31, S1, S2 of a method of assisting control of a vehicle based on physiological information of a driver according to an exemplary embodiment of the present invention. The method step S31 in fig. 2 exemplarily comprises sub-steps S301-S306.
In step S301, the first physiological information is preprocessed. This can be done, for example, by means of the first signal processing unit 31 of the evaluation module 30 in fig. 1.
Independently from step S301, the second physiological information can be preprocessed in step S301' by means of the second signal processing unit 32 of the analysis module 30.
Next, in step S302, feature extraction may be performed on the preprocessed first physiological information. Independently of the feature extraction process of the first physiological information, the feature extraction may be performed on the preprocessed second physiological information in step S302'. Thereby, for example, discrete feature vectors of the first physiological information and the second physiological information are extracted only for each frequency band of interest.
In steps S303 and S303', an intensity decision is made on the first physiological information and the second physiological information, respectively, in order to check whether the respective physiological information can express the exercise intention.
In step S304, after the strength decision is performed on the first and second physiological information, respectively, it may be further determined whether both the first physiological information and the second physiological information reflect the exercise intention.
If at least one of the two physiological information is judged not to reflect the movement intention, the steps S1 and S2 are returned to obtain the first and second physiological information again.
Only when both physiological information reflect the motor intention, a first candidate intention sequence is formed for the first physiological information and a second candidate intention sequence is formed for the second physiological information in steps S305 and S305', respectively.
Next, in step S306, a target driving intention is screened out from the candidate intention sequence according to the time correlation and/or the content correlation between the first physiological information and the second physiological information.
Fig. 4 shows a flowchart of a method for assisting in controlling a vehicle based on physiological information of a driver according to another exemplary embodiment of the present invention.
In step S401, first physiological information and second physiological information of the driver are acquired, respectively. For example, the electroencephalogram signal of the driver is received in an uninterrupted manner by means of a head-mounted electroencephalogram signal detector, and the surface electromyogram signal of the driver is received by means of a helper/ankle-type electromyogram signal detector.
In step S402, the first physiological information and the second physiological information are preprocessed. The preprocessing of the first physiological information can be carried out, for example, by means of the first signal processing unit 31 of the analysis module 30 in fig. 1. In particular, in the case where the first physiological information relates to the electroencephalogram signal of the driver, it is possible to perform baseline correction thereon, thereby fluctuating all the channel signals within the same range and significantly eliminating signal drift. In addition, the distortion part in the electroencephalogram signals can be filtered by means of a digital band-pass filter, and all channel signals can be subjected to superposition averaging, so that noise and interference influence can be effectively eliminated. Independently of the preprocessing of the electroencephalogram signal, the second physiological information can be preprocessed by means of the second signal processing unit 32 of the analysis module 30. In case the second physiological information relates to the electromyographic signal and/or the surface electromyographic signal of the driver, the electromyographic signal may be preprocessed in a similar way as the electroencephalographic signal. Since electromyographic signals have a lower complexity than electroencephalographic signals, some pre-processing measures, such as only digital band-pass filtering, may also be omitted in this step as appropriate to reduce data processing overhead and latency.
In step S403, time-frequency domain feature extraction may be performed on the preprocessed first physiological information and second physiological information. Here, the time-frequency domain signal extraction is performed on the preprocessed electroencephalogram signal by using signal processing means such as fast fourier transform, power spectrum estimation, wavelet transform, and the like. Here, different frequency bands of the electroencephalogram signal are used to represent different types of signals, for example, and by the above-described feature extraction step, a signal that can be used for the purpose analysis can be extracted only for a specific frequency band of interest. The feature extraction of the preprocessed second physiological information can be performed independently from the feature extraction process of the electroencephalogram signal. Likewise, the time-frequency domain feature extraction of the preprocessed myoelectricity can be carried out here, for example, by means of a wavelet transform. Here, for example, discrete feature vectors are extracted only for each frequency band of interest.
In step S404, intensity decisions are made on the first physiological information and the second physiological information, respectively. For example, the first physiological information is checked with respect to the motor intention by means of a first intensity decider. In some cases, a particular piece of physiological information, while containing conscious behavior, does not point to a specific motor intent. These normal mental segments, if utilized by mistake to control vehicle operation, can also create safety hazards during driving. Thus, here, for example, a first threshold determiner is used to make an intensity determination on the extracted discrete feature vector of the brain electrical signal. If the intensity value of the discrete feature vectors of the electroencephalogram signal is found to exceed the first threshold value, the electroencephalogram signal can reflect the movement intention. On the contrary, it means that the electroencephalogram signal only belongs to a normal thinking segment and therefore cannot reflect the movement intention. In addition, in step S404, the second physiological information is also checked with respect to the motor intention, for example, by means of a second intensity decider. Also, in some cases, although a weak potential difference is detected at the skin surface by the muscle electrodes, this may be due to only natural contraction of muscle fibers caused by ambient temperature, lighting conditions, fatigue, and the like, and thus these surface electromyographic signals do not point to the actual intention of exercise. In order to distinguish these interference terms, the intensity of the discrete feature vector of the electromyographic signal is compared with a second threshold value, for example, by means of a second threshold value decider, and if the intensity value of the discrete feature vector is found to exceed the second threshold value, it is indicated that the electromyographic signal reflects the movement intention. Otherwise, it means that the myoelectric signal cannot reflect the movement intention.
After the intensity decision is performed on the first and second physiological information, respectively, it may be further determined whether both the first physiological information and the second physiological information reflect the movement intention. Here, when the judgment about the exercise intention is performed, the above-described exercise intention judgment is performed commonly for, for example, the first physiological information acquired at the first time and the second physiological information acquired at the second time. Here, the second time is delayed with respect to the first time by a predefined time period. Thereby, it can be ensured that there is a correspondence between the first physiological information and the second physiological information used when analyzing the same driving intention. If at least one of the two physiological information is judged not to reflect the movement intention, the step S401 is returned to obtain the first and the second physiological information again.
If both physiological information are judged to reflect the exercise intention, a first candidate intention sequence is formed for the first physiological information and a second candidate intention sequence is formed for the second physiological information in step S405. In this case, for example, feature matrices of the first and second physiological information are respectively input into corresponding pre-trained machine learning models, and all possible potential driving intents are obtained by using the input-output relationships established by the pattern recognition algorithm. Here, the machine learning model is, for example, a support vector machine model, a KNN nearest neighbor model, a decision tree model, a CNN convolutional neural network model, or the like. The mode identification method based on the support vector machine has the following advantages: the problem of data nonlinearity is solved, and the method has an advantage for processing a high-dimensional number problem. For the first physiological information, all potential driving intentions are combined to form a first candidate sequence of intentions. For the second physiological information, all potential driving intentions are combined to form a second candidate sequence of intentions.
Next, in step S406, a target driving intention is screened out from the candidate intention sequence according to the time correlation and/or the content correlation between the first physiological information and the second physiological information. Here, a confidence probability is assigned to each potential driving intention in the first and second sequences of candidate intentions, for example, by means of a pre-trained recurrent neural network, and then the potential driving intention with the highest confidence probability is output as the target driving intention.
In step S407, a control signal is generated from the output result of the recurrent neural network, and is sent to a corresponding vehicle controller unit (e.g., an accelerator pedal, a brake pedal, a steering device, etc.) so as to trigger a vehicle operation associated with the target driving intention. After the target driving intent is obtained, it may continue to jump back to step S401 in order to continue to receive the next set of physiological information.
Fig. 5 shows a schematic diagram of a recurrent neural network architecture employed in the method of the present invention to enable cross-validation of the first and second physiological information.
Here, an input layer 401, a hidden layer 402 and an output layer 403 of the recurrent neural network are shown. X1-X3, for example, represent vectors for input layer 401, H1-H2, for example, represent vectors for hidden layer 402, and Z1-Z3 represent vectors for output layer 403.
In a specific example, at a first moment in time, the input of the recurrent neural network is a first candidate sequence of intentions extracted for the first physiological information, X1 for example representing a potential driving intention "step on the brake", X2 for example representing a potential driving intention "throttle up", X3 for example representing a potential driving intention "throttle down", such first candidate sequence of intentions having been obtained in the last module, for example. At the second time, the input of the recurrent neural network is a second candidate intention sequence obtained according to the second physiological information, and the second candidate intention sequence has partial element overlapping with the first candidate intention sequence, so that the model can analyze the current unit better and know the context relationship of time. The hidden layer contains the weight values of the electroencephalogram and electromyogram characteristic parameters, and the weight values are determined in the pre-training process of the neural network and can be adaptively adjusted in time.
It should be noted that the signals of the sources of various movements of the human body are derived from the brain, so that when a driving intention is formed in the brain of a driver, the brain electrical signals of the cerebral cortex will first generate control signals and be transmitted to the hand/foot muscle fibers by using the neuromuscular junction as a medium. The myofibroblasts undergo repeated depolarization and repolarization operations, and finally generate action potentials on the skin surface.
Because the electroencephalogram signal and the muscle potential have a temporal precedence relationship and a content coherence relationship, the first candidate intention sequence and the second candidate intention sequence can be regarded as sequence segments with temporal precedence, and the recurrent neural network can depict the relationship between the current output and the previous information of one sequence. The recurrent neural network can thus, for example, remember information at a first point in time and use this information to influence the classification of the network at a second point in time and the corresponding confidence probability assignment. As shown in fig. 4, the hidden layer at each time is determined not only by the input layer at that time but also by the hidden layer at the previous time, and therefore, the output for each monomodal physiological information is delivered to the recurrent neural network to perform the multimodal fusion association determination. The recurrent neural network thus aggregates the predicted outcome of each single modality of physiological information at each time instant. After the multi-modal fusion analysis is synthesized, the output of the recurrent neural network is the final driving intention recognition result.
By using the recurrent neural network, the relevance of the electroencephalogram signal and the myoelectric effect signal in a certain time range is effectively utilized, and more reliable network training classification is carried out by utilizing the relevance, so that different driving intentions and relative confidence probabilities can be output. Therefore, the accuracy and the robustness of motion pattern recognition are improved by utilizing the cooperative complementarity between the brain myoelectric signals
Although specific embodiments of the invention have been described herein in detail, they have been presented for purposes of illustration only and are not to be construed as limiting the scope of the invention. Various substitutions, alterations, and modifications may be devised without departing from the spirit and scope of the present invention.
Claims (10)
1. A method of assisting control of a vehicle based on physiological information of a driver, the method comprising the steps of:
s1: acquiring first physiological information of a driver;
s2: acquiring second physiological information of the driver, wherein the second physiological information is different from the first physiological information;
s3: generating a control signal for the vehicle in a cross-validated manner based on the first physiological information and the second physiological information; and
s4: and outputting the control signal to control the driving operation of the vehicle.
2. The method of claim 1, wherein the first physiological information comprises a driver's electroencephalograph signal and the second physiological information comprises a driver's electromyography signal and/or surface electromyography signal.
3. The method according to claim 1 or 2, wherein the step S3 comprises:
s31: fusion finding a target driving intention of the driver based on the first physiological information and the second physiological information; and
s32: generating a control signal according to a target driving intention, wherein the control signal is used for triggering vehicle operation associated with the target driving intention.
4. The method according to claim 3, wherein the step S31 includes:
outputting potential driving intentions of the driver for the first physiological information and the second physiological information respectively by means of a machine learning model;
combining all potential driving intents to form a candidate intent sequence; and
and screening out the target driving intention from the candidate intention sequence according to the time correlation and/or the content correlation between the first physiological information and the second physiological information.
5. The method of claim 3 or 4, wherein screening target driving intents comprises:
assigning a confidence probability to each potential driving intent in the sequence of candidate intentions by means of a pre-trained artificial neural network; and
and outputting the potential driving intention with the highest confidence probability as the target driving intention.
6. The method according to any one of claims 1 to 5, wherein the step S3 further comprises: and respectively judging whether the first physiological information and the second physiological information reflect the movement intention, wherein the first physiological information and the second physiological information are used for generating the control signal only under the condition that the first physiological information and the second physiological information both reflect the movement intention.
7. The method of claim 6, wherein the determination of the motor intent is made jointly for first physiological information acquired at a first time and second physiological information acquired at a second time, the second time being delayed with respect to the first time by a predefined time period.
8. The method according to any one of claims 1 to 7, wherein the step S3 further includes:
and respectively carrying out signal preprocessing and time-frequency domain signal feature extraction on the first physiological information and the second physiological information.
9. A device (1) for assisting in controlling a vehicle based on physiological information of a driver, the device (1) being adapted to perform a method according to any one of claims 1 to 8, the device (1) comprising:
a first acquisition module (10) configured to be able to acquire first physiological information of a driver;
a second acquisition module (20) configured to be able to acquire second physiological information of the driver, the second physiological information being different from the first physiological information;
an analysis module (30) configured to be able to generate a control signal for the vehicle in a cross-validated manner based on the first physiological information and the second physiological information; and
an output module (40) configured to be able to output the control signal for controlling a driving operation of a vehicle.
10. A computer program product, wherein the computer program product comprises a computer program for implementing the method according to any one of claims 1 to 8 when executed by a computer.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110923088.4A CN113501005A (en) | 2021-08-12 | 2021-08-12 | Method and device for assisting control of vehicle based on physiological information of driver |
DE102022002867.0A DE102022002867A1 (en) | 2021-08-12 | 2022-08-08 | Method and system for vehicle control assistance based on physiological information from the driver |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110923088.4A CN113501005A (en) | 2021-08-12 | 2021-08-12 | Method and device for assisting control of vehicle based on physiological information of driver |
Publications (1)
Publication Number | Publication Date |
---|---|
CN113501005A true CN113501005A (en) | 2021-10-15 |
Family
ID=78016069
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110923088.4A Pending CN113501005A (en) | 2021-08-12 | 2021-08-12 | Method and device for assisting control of vehicle based on physiological information of driver |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN113501005A (en) |
DE (1) | DE102022002867A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114081504A (en) * | 2021-11-23 | 2022-02-25 | 青岛理工大学 | Driving intention identification method and system based on electroencephalogram signals |
CN114670848A (en) * | 2022-03-02 | 2022-06-28 | 江苏泽景汽车电子股份有限公司 | Dangerous driving prediction method and device, terminal equipment and storage medium |
CN118220195A (en) * | 2024-05-24 | 2024-06-21 | 广汽埃安新能源汽车股份有限公司 | Vehicle control method and device based on brain electricity |
-
2021
- 2021-08-12 CN CN202110923088.4A patent/CN113501005A/en active Pending
-
2022
- 2022-08-08 DE DE102022002867.0A patent/DE102022002867A1/en active Pending
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114081504A (en) * | 2021-11-23 | 2022-02-25 | 青岛理工大学 | Driving intention identification method and system based on electroencephalogram signals |
CN114081504B (en) * | 2021-11-23 | 2024-03-01 | 青岛理工大学 | Driving intention recognition method and system based on electroencephalogram signals |
CN114670848A (en) * | 2022-03-02 | 2022-06-28 | 江苏泽景汽车电子股份有限公司 | Dangerous driving prediction method and device, terminal equipment and storage medium |
CN118220195A (en) * | 2024-05-24 | 2024-06-21 | 广汽埃安新能源汽车股份有限公司 | Vehicle control method and device based on brain electricity |
Also Published As
Publication number | Publication date |
---|---|
DE102022002867A1 (en) | 2023-02-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Lopez-Martinez et al. | Multi-task neural networks for personalized pain recognition from physiological signals | |
Wei et al. | A subject-transfer framework for obviating inter-and intra-subject variability in EEG-based drowsiness detection | |
Samuel et al. | Towards efficient decoding of multiple classes of motor imagery limb movements based on EEG spectral and time domain descriptors | |
CA3078404A1 (en) | Brain-computer interface platform and process for classification of covert speech | |
US20040267152A1 (en) | Method and system for predicting and preventing seizures | |
Han et al. | Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynchronous brain-computer interface | |
CN113501005A (en) | Method and device for assisting control of vehicle based on physiological information of driver | |
Pun et al. | Brain-computer interaction research at the Computer Vision and Multimedia Laboratory, University of Geneva | |
Dong et al. | A WPCA-based method for detecting fatigue driving from EEG-based internet of vehicles system | |
WO2024032728A1 (en) | Method and apparatus for evaluating intelligent human-computer coordination system, and storage medium | |
Zhang et al. | Inferring driver's turning direction through detection of error related brain activity | |
Placidi et al. | A fast and scalable framework for automated artifact recognition from EEG signals represented in scalp topographies of independent components | |
CN117918863A (en) | Method and system for processing brain electrical signal real-time artifacts and extracting features | |
CN111736690A (en) | Motor imagery brain-computer interface based on Bayesian network structure identification | |
Ismail et al. | An EEG based Physiological Signal for Driver Behavior Monitoring Systems: A Review | |
CN114237383B (en) | Multi-state identification method based on forehead single-lead electroencephalogram signals | |
Huang et al. | Research on artificial intelligence-based computer-assisted anesthesia intelligent monitoring and diagnostic methods in health care | |
Teixeira et al. | A new approach to emotion assessment based on biometric data | |
Joaquim et al. | Learning data representation and emotion assessment from physiological data | |
Chuysud et al. | Hybrid EEG-fEMG based Human-Machine Interface for communication and control applications | |
Vidyasagar et al. | Signal to Image Conversion and Convolutional Neural Networks for Physiological Signal Processing: A Review | |
Galego et al. | Surface electromyography and electroencephalography processing in dysarthric patients for verbal commands or speaking intention characterization | |
Katertsidis et al. | INTREPID, a biosignal-based system for the monitoring of patients with anxiety disorders | |
CN112230768B (en) | Wheelchair driven by SSMVEP-ERP-OSR hybrid brain-computer interface | |
EP4343500A1 (en) | Method and apparatus for evaluating intelligent human-computer coordination system, and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication |