DE102022002867A1 - Method and system for vehicle control assistance based on physiological information from the driver - Google Patents

Abstract

The present invention relates to the field of driver assistance for vehicles. The present invention provides a vehicle control assistance method based on physiological information of the driver, the method comprising the following steps: S1: acquisition of first physiological information of the driver; S2: acquisition of second physiological information of the driver, the second physiological information deviating from the first physiological information; S3: generating control signals for the vehicle based on the first physiological information and second physiological information by cross-validation; and S4: Output of the control signals for controlling the driving operation of the vehicle. The present invention also provides a vehicle control assistance device based on physiological information of the driver and a computer program product. In the present invention, taking two different kinds of physiological information of the driver into consideration overall, based on the correlation between the two, cross-validation of the recognition result can be made, which has a positive effect on compensating for shortcomings in the analysis of only one kind of biological information, as well as improving the stability and reliability of the system increases.

Description

The present invention relates to a method for vehicle control assistance based on physiological information of the driver, a device for vehicle control assistance based on physiological information of the driver, and a computer program product.

With the progressive intensification of the intelligent design of vehicles, vehicles are equipped with various human-machine exchange functions, which contributes to enhancing the safety and comfort of the driver. On certain occasions, however, the assistance instructions and pre-warning measures are not sufficient to fundamentally avoid the risk of an accident for the driver. For example, if the driver is aware of a sudden hazard, there is sometimes not enough time to manually control the vehicle and effectively avoid the hazard. Although there is biological information that reflects the driver's intention to brake, it is too late to actually carry out the corresponding actuation. In addition, there are difficulties in driving the vehicle safely for persons with limited mobility of extremities or lack of driving experience.

A method for determining the driving intention of the user using electroencephalogram signals is known in the prior art, the method in question determining the corresponding driving actions by detecting the electroencephalogram (EEG, Electroencephalogram) of the driver.

In addition, a method based on the electromyogram (EMG, electromyogram) for measuring an intention to brake in an emergency is already known, with which in a virtual driving environment based on the driver's muscle activities, intention feedback from the driver for an intention to brake in an emergency can be measured.

However, these solutions still have numerous shortcomings. In particular, the measurement of the user's intention to move is currently only based on one type of bioelectrical signals. While using only electromyogram signals provides a rapid response to intention to move, individual variation is relatively strong and there is significant time variability. The use of only electroencephalogram signals relies on an overall reflection of the activities of the central nervous system at the cerebral cortex, which is even weaker and more prone to noise. At the same time, there is a strong correlation between the frequency of the areas generating the relevant signals and the type of intention to move. Therefore, it is impossible to make a precise grasp of the concrete human intention based on this. Thus, there are significant shortcomings in detecting intention to move based on only one type of biological information.

Against this background, there is a need to provide a solution that aggregates multiple types of physiological information to assist each other in measuring driver's intention to move, to enable more reliable vehicle control assistance.

The object of the present invention is to provide a driver physiological information-based vehicle control assistance method, a driver physiological information-based vehicle control assistance apparatus, and a computer program product to solve at least part of the problems in the prior art.

• Schritt S1: Erfassung erster physiologischer Informationen des Fahrers;
• Schritt S2: Erfassung zweiter physiologischer Informationen des Fahrers, wobei die zweiten physiologischen Informationen von den ersten physiologischen Informationen abweichen;
• Schritt S3: Auf den ersten physiologischen Informationen und zweiten physiologischen Informationen basierende Generierung von Steuerungssignalen für das Fahrzeug durch Kreuzvalidierung; sowie
• Schritt S4: Ausgabe der Steuerungssignale zur Steuerung der Fahrbetätigung des Fahrzeugs.

According to one aspect of the present invention, a method for vehicle control assistance based on physiological information of the driver is provided, said method comprising the following steps:

• Step S1: acquisition of first physiological information of the driver;
• Step S2: acquiring second physiological information of the driver, the second physiological information being different from the first physiological information;
• Step S3: generating control signals for the vehicle based on the first physiological information and second physiological information by cross-validation; as well as
• Step S4: Output of the control signals for controlling the driving operation of the vehicle.

• Unter Gesamtberücksichtigung zweier unterschiedlicher Arten von physiologischen Informationen des Fahrers kann mit Hilfe der Korrelation zwischen beiden eine Kreuzvalidierung des Erkennungsergebnisses vorgenommen werden, was sich positiv auf die Kompensation von Mängeln bei der Analyse lediglich einer Art von biologischen Informationen auswirkt sowie Stabilität und Zuverlässigkeit des Systems steigert. Somit kann in Notfallsituationen insgesamt eine Verringerung der Zeitverzögerung menschlicher Intervention ins System erfolgen.

The present invention includes in particular the following technical concepts:

• Taking into account two different types of driver's physiological information as a whole, the correlation between the two can be used to perform cross-validation of the recognition result, which has a positive effect on compensating for deficiencies in the analysis of only one type of biological information, as well as increasing the stability and reliability of the system. Thus, in emergency situations, total a reduction in the time lag of human intervention into the system.

Provision is preferably made for the first physiological information to include the driver's electroencephalogram signals, while the second physiological information to include electromyogram signals and/or surface electromyogram signals from the driver. Hereby the following technical advantages are realized: Electroencephalogram signals are an overall reflection of the electrophysiological activities of the cerebral nerve cells on the cerebral cortex or scalp surface, which contain rich intentional information and exhibit good prematurity. There is a possibility of using it to anticipate the human body's intention to move, solving the time lag problem of other sources of physiological information. Electromyogram signals are overlaps in time and space of movements of numerous muscle fibers. Surface electromyogram signals are electrical signals that arise at the surface of the skin during muscle contraction and provide a rapid response to the intention to move. By combining these two types of critical information sources, the strength of different physiological information is brought out while each other's weaknesses are balanced.

• Schritt S31: Auf den ersten physiologischen Informationen und zweiten physiologischen Informationen basierende zusammenfassende Ermittlung der Zielfahrabsicht des Fahrers; sowie

Provision is preferably made for said step S3 to include:

• Step S31: summary determination of the driver's target driving intention based on the first physiological information and second physiological information; as well as

Step S32: Generation of control signals on the basis of the target driving intention, the control signals serving to trigger vehicle activations in connection with the target driving intention.

• Mit Hilfe eines maschinellen Lernmodells erfolgende Ausgabe der potentiellen Fahrabsichten des Fahrers bezüglich der ersten physiologischen Informationen und zweiten physiologischen Informationen;
• Kombination sämtlicher potentieller Fahrabsichten mit Bildung einer zur Auswahl stehenden Absichtsreihe; sowie
• Auswahl der Zielfahrabsicht aus der zur Auswahl stehenden Absichtsreihe anhand der zeitlichen Korrelation und/oder inhaltlichen Korrelation zwischen den ersten physiologischen Informationen und zweiten physiologischen Informationen.
• Somit werden die folgenden technischen Vorteile realisiert: Zunächst erfolgt unabhängig und parallel durch Signalverarbeitung und Algorithmen zur Mustererkennung die Messung der entsprechenden potentiellen Fahrabsichten jeweils bezüglich beider Arten von physiologischen Informationen. Allerdings weisen beide Arten von physiologischen Informationen Mängel bezüglich der Analyse auf, so dass die in diesem Stadium gemessenen Ergebnisse möglicherweise nicht exakt sind. Anschließend erfolgt mittels vergleichender Validierung die Extraktion der Gemeinsamkeiten beider Arten von Signalen in Zeit und Raum, um auf diese Weise das letztendliche Erkennungsergebnis zu bestimmen, so dass die präzise Messung der Fahrabsicht realisiert wird.

It is optionally provided that step S31 includes:

• Using a machine learning model to output the driver's potential driving intentions with regard to the first physiological information and second physiological information;
• Combination of all potential driving intentions with formation of an intention row to choose from; as well as
• Selection of the target driving intention from the list of intentions available for selection based on the temporal correlation and/or content-related correlation between the first physiological information and the second physiological information.
• The following technical advantages are thus realized: First, the corresponding potential driving intentions are measured independently and in parallel by signal processing and algorithms for pattern recognition, in each case with regard to both types of physiological information. However, both types of physiological information have deficiencies in analysis, so the results measured at this stage may not be accurate. Then, by means of comparative validation, the commonalities of both types of signals are extracted in time and space, in order to determine the final recognition result, so that the precise measurement of the driving intention is realized.

• Mit Hilfe eines vorab trainierten künstlichen neuronalen Netzes für jede potentielle Fahrabsicht in der zur Auswahl stehenden Absichtsreihe erfolgende Zuordnung einer Konfidenzwahrscheinlichkeit; sowie
• Ausgabe der potentiellen Fahrabsicht mit der höchsten Konfidenzwahrscheinlichkeit als genannte Zielfahrabsicht.
• Somit werden die folgenden technischen Vorteile realisiert: Durch eine Unterteilung in Konfidenzbereiche und Zuordnung von Konfidenzen kann eine exaktere Auswahl des Gesamterkennungsergebnisses für die Bewegungsabsicht mit Kombination von Gehirn und Muskeln vorgenommen werden.
• Optional ist vorgesehen, dass Schritt S3 außerdem umfasst: Beurteilung, ob die ersten physiologischen Informationen und zweiten physiologischen Informationen eine Bewegungsabsicht widerspiegeln, wobei lediglich im Falle, dass sowohl die ersten physiologischen Informationen als auch die zweiten physiologischen Informationen eine Bewegungsabsicht widerspiegeln, die ersten physiologischen Informationen und zweiten physiologischen Informationen zur Generierung von Steuerungssignalen verwendet werden.

It is optionally provided that the selection of the target driving intention includes:

• Assignment of a confidence probability to each potential driving intention in the series of intentions available for selection with the aid of a previously trained artificial neural network; as well as
• Output of the potential driving intention with the highest confidence probability as named target driving intention.
• The following technical advantages are thus realized: By subdividing into confidence ranges and assigning confidences, a more exact selection of the overall recognition result for the intention to move with a combination of brain and muscles can be made.
• It is optionally provided that step S3 also includes: assessing whether the first physiological information and second physiological information reflect an intention to move, only in the event that both the first physiological information and the second physiological information reflect an intention to move, the first physiological information and second physiological information for generating control signals.

The following technical advantages are thus realized: With the help of the filter conditions of the intention to move, interference can be eliminated before the vehicle control is actually carried out.

It is optionally provided that a joint assessment of the intention to move is carried out with regard to the first physiological information recorded for a first time and the second physiological information recorded for a second time, with said second time being downstream in relation to the first time according to a predefinition.

• Signalvorverarbeitung der ersten physiologischen Informationen und zweiten physiologischen Informationen sowie Extraktion von Zeit-Frequenz-Signalmerkmalen.

The following technical advantages are thus realized: When the driving intention arises in the driver's cerebrum, activation signals are usually first generated in the cerebral cortex, and only then are the control signals generated in the cerebrum transmitted to the muscles. Thus, there are usually time deviations between different physiological information items generated by one and the same driving intention. Only the synchronicity characteristics of the electroencephalogram signals and the electromyogram signals can reflect the connection of the functional coupling of cerebral cortex and muscles. Here, the time deviation is taken into account by a prior definition in the analysis, with a strict limitation of the interrelationship between different physiological information. It is optionally provided that step S3 also includes:

• Signal pre-processing of the first physiological information and second physiological information and extraction of time-frequency signal features.

Thus, the following technical advantages are realized: An effective elimination of artifacts and noise in the corresponding physiological information is made possible, with the extraction of usable signals being performed from the original signals, which increases the accuracy of the subsequent intention recognition.

• Ein erstes Erfassungsmodul, welches zur Erfassung erster physiologischer Informationen des Fahrers ausgebildet ist;
• ein zweites Erfassungsmodul, welches zur Erfassung zweiter physiologischer Informationen des Fahrers ausgebildet ist, wobei die zweiten physiologischen Informationen von den ersten physiologischen Informationen abweichen;
• ein Analysemodul, welches zur auf den ersten physiologischen Informationen und zweiten physiologischen Informationen basierenden Generierung von Steuerungssignalen für das Fahrzeug durch Kreuzvalidierung ausgebildet ist; sowie
• ein Ausgabemodul, welches zur Ausgabe der Steuerungssignale zur Steuerung der Fahrbetätigung des Fahrzeugs ausgebildet ist.

According to a second aspect of the present invention, a device for vehicle control assistance based on physiological information of the driver is provided, wherein said device is used to carry out the method according to the first aspect of the present invention and wherein the device comprises:

• A first acquisition module, which is designed to acquire first physiological information of the driver;
• a second acquisition module, which is designed to acquire second physiological information about the driver, the second physiological information differing from the first physiological information;
• an analysis module configured to generate control signals for the vehicle through cross-validation based on the first physiological information and second physiological information; as well as
• an output module, which is designed to output the control signals for controlling the driving operation of the vehicle.

According to a third aspect of the present invention there is provided a computer program product, said computer program product comprising a computer program, wherein when said computer program is executed by a computer performing the method according to the first aspect of the present invention.

• 1 zeigt die Konstruktion der auf physiologischen Informationen des Fahrers basierenden Vorrichtung zur Fahrzeugsteuerungsassistenz nach einer beispielhaften Ausführungsform der vorliegenden Erfindung.
• 2 zeigt das Ablaufschema des auf physiologischen Informationen des Fahrers basierenden Verfahrens zur Fahrzeugsteuerungsassistenz nach einer beispielhaften Ausführungsform der vorliegenden Erfindung.
• 3 zeigt das Ablaufschema von drei Verfahrensschritten des auf physiologischen Informationen des Fahrers basierenden Verfahrens zur Fahrzeugsteuerungsassistenz nach einer beispielhaften Ausführungsform der vorliegenden Erfindung.
• 4 zeigt das Ablaufschema des auf physiologischen Informationen des Fahrers basierenden Verfahrens zur Fahrzeugsteuerungsassistenz nach einer anderen beispielhaften Ausführungsform der vorliegenden Erfindung, und
• 5 zeigt die Struktur des rekurrenten neuronalen Netzes zur Realisierung der Kreuzvalidierung der ersten physiologischen Informationen und zweiten physiologischen Informationen bei dem Verfahren nach der vorliegenden Erfindung.

A detailed description of the present invention is given below with reference to the attached figures for a better understanding of the principle, features and advantages of the present invention:

• 1 12 shows the construction of the driver's physiological information-based vehicle control assistance apparatus according to an exemplary embodiment of the present invention.
• 2 12 shows the flowchart of the driver's physiological information-based vehicle control assistance method according to an exemplary embodiment of the present invention.
• 3 shows the flow chart of three method steps of the method for vehicle control assistance based on physiological information of the driver according to an exemplary embodiment of the present invention.
• 4 FIG. 12 shows the flowchart of the driver physiological information-based vehicle control assistance method according to another exemplary embodiment of the present invention, and FIG
• 5 12 shows the structure of the recurrent neural network for realizing the cross-validation of the first physiological information and the second physiological information in the method according to the present invention.

In order to clarify the technical problem to be solved by the present invention, the technical solution and positive technical effects of the present invention, the following follows a further detailed explanation of the present invention, taking into account the attached figures and several exemplary embodiments. It should be understood that the exemplary embodiments described herein are for illustrative purposes only, without implying any limitation of the scope of the present invention.

1 12 shows the construction of the driver's physiological information-based vehicle control assistance apparatus according to an exemplary embodiment of the present invention.

As in 1 shown, the device 1 is used, for example, for vehicle control assistance based on the driver's physiological information, the device 1 for example comprising a first detection module 10 , a second detection module 20 , an analysis module 30 and an output module 40 . The first acquisition module 1 is designed, for example, as an electroencephalogram signal measuring device designed to be worn on the head (e.g. an electrode cap) and is used to continuously acquire electroencephalograms generated on the cerebral cortex based on changes in the electrical potential of the electrodes on different channels of the driver's head - signals. The second detection module 2 is designed, for example, as an electromyogram signal measuring device, can be provided at different points on the driver's body (e.g. wrist, ankle, arm, leg, neck, etc.) as required, and is used for real-time measurement of electromyogram signals or Driver's surface electromyogram signals. Here, the electromyogram signals are overlaps of the electrical potentials of movement units of numerous muscle fibers in time and space. These signals can be detected, for example, with the aid of inserted needle electrodes. The surface electromyogram signals are the overall effect of the electrical activity of the upper layer of muscles and nerve trunks at the skin's surface, which to some extent may reflect the activities of nerves and muscles. Compared with the measurement of the electromyogram signals, the measurement of the surface electromyogram signals offers the advantages of non-injury and easy handling. After the driver's physiological information has been recorded, it is transmitted to the analysis module 30 .

The analysis module 30 is designed to generate control signals for the vehicle based on the detected electroencephalogram signals and electromyogram signals by cross-validation. For this purpose, the analysis module 30 comprises, for example, a first signal processing unit 31, a second signal processing unit 32 and a signal combining unit 33. The relevant first signal processing unit 31 receives the driver's electroencephalogram signals from the first detection module 1 and performs the corresponding signal processing. This signal processing includes, for example, pre-processing of the electroencephalogram signals, feature extraction, an intensity assessment and the detection of the potential driving intention. The analysis module 30 also includes a second signal processing unit 32, this second signal processing unit 32 receiving the electromyogram signals and/or surface electromyogram signals of the driver from the second detection module 2 and carrying out the corresponding signal processing. This signal processing unit can be handled in a manner similar to the first signal processing unit 31 .

In this case, both the first signal processing unit 31 and the second signal processing unit 32 are connected to the signal combining unit 33 . The relevant signal combining unit 33 is designed, for example, as a recurrent neural network in order to receive from the first and second signal processing units 31 and 32 a list of intentions available for selection formed by combining all potential driving intentions and to select the target driving intention from this. The selected target driving intention is transmitted to the output module 40 and sent by the output module 40 to at least one execution device (e.g. accelerator pedal, brake pedal, steering device, etc.) of the vehicle in order to control the driving activities of the vehicle accordingly.

2 12 shows the flowchart of the driver's physiological information-based vehicle control assistance method according to an exemplary embodiment of the present invention. For example, the device 1 can be used to carry out the method 1 be used.

In terms of the present invention, physiological information is functional phenomena of the biological mechanism, which vary according to the basic physiological structure, age, sex, and so on. When recognizing the driving intention, this physiological information includes different information such as electroencephalogram signals, elec ric skin signals, electrical heart signals, pupil movements, respiration, blood pressure, etc.

In step S1, the driver's first physiological information is recorded, for example electroencephalogram signals from the driver are continuously received using the technology of the brain-computer interface.

In step S2, second physiological information about the driver is recorded, the second physiological information deviating from the first physiological information. Here, for example, real-time reception of surface electromyogram signals from the driver takes place via electrodes glued to the surface of the wrist.

In step S3, control signals for the vehicle based on the first physiological information and second physiological information are generated by cross-validation.

In this case, for example, in sub-step S31 of step S3, the summarizing determination of the driver's target driving intention based on the first physiological information and second physiological information takes place. For the purposes of the present invention, the target driving intention is understood to mean, for example, that the driver shows a control intention before carrying out specific driving operations with regard to specific driving activities of the vehicle, which includes, for example: braking intention, intention to accelerate, intention to steer, intention to change lanes, intention to follow, etc.

In sub-step S32 of step S3, control signals are generated on the basis of the target driving intention. In this case, the relevant control signals are used, for example, to trigger vehicle activations in connection with the target driving intention mentioned.

In step S4, the mentioned control signals for controlling the driving operation of the vehicle are output.

3 shows the flow chart of three method steps S31, S1 and S2 of the method for vehicle control assistance based on physiological information of the driver according to an exemplary embodiment of the present invention. The method step S31 from 2 includes, for example, the sub-steps S301-S306.

In step S301, the first physiological information is preprocessed. This can be done, for example, by the first signal processing unit 31 of the analysis module 30 1 be performed. Independently of step S301, the second signal processing unit 32 of the analysis module 30 can carry out the pre-processing of the second physiological information in step S301′.

Subsequently, in step S302, the feature extraction of the pre-processed first physiological information can be performed. The feature extraction of the preprocessed second physiological information can be carried out in step S302′ independently of this feature extraction of the preprocessed first physiological information. Thus, for a frequency section of interest, the discrete feature vectors of first physiological information and second physiological information can be extracted.

In steps S303 and S303', the intensity judgment of the first physiological information and the second physiological information is performed to check whether or not the physiological information concerned can express an intention to move.
In step S304, after the intensity assessment of the first and second physiological information has been carried out, a further assessment can be carried out as to whether both the first physiological information and the second physiological information reflect an intention to move.
If it is judged that at least one of the two kinds of physiological information does not reflect any intention to move, it returns to steps S1 and S2 to acquire the first and second physiological information again.

Only if both types of physiological information reflect an intention to move, in steps S305 and S305', a first intention series available for selection is formed with respect to the first physiological information and a second intention series available for selection is formed with respect to the second physiological information.
Then, in step S306, the target driving intention is selected from the list of intentions available for selection based on the temporal correlation and/or content-related correlation between the first physiological information and the second physiological information.
4 12 shows the flowchart of the vehicle control assistance method based on driver physiological information according to another exemplary embodiment of the present invention.

In step S401, the driver's first physiological information and second physiological information are respectively acquired, for example, the head-worn electroencephalogram signal measuring device receives electroencephalogram signals from the driver and the wrist/ankle electromyogram signal measuring device receives surface electromyogram signals of the driver. In step S402, the first physiological information and the second physiological information are preprocessed. Here, for example, by the first signal processing unit 31 of the analysis module 30 1 the pre-processing of the first physiological information can be carried out. Specifically, if the first physiological information is present as the driver's electroencephalogram signals, it is provided that these are subjected to baseline correction so that all channel signals oscillate in one and the same range, which significantly eliminates signal shifts. In addition, the part with distortion can be filtered out of the electroencephalogram signals by means of a digital bandpass filter, so that there is an even overlap of all channel signals and noise and interference are effectively eliminated. Independent of the pre-processing of the electroencephalogram signals, the second physiological information can be pre-processed with the aid of the second signal processing unit 32 of the analysis module 30 . If the present second physiological information relates to electromyogram signals and/or surface electromyogram signals of the driver, the pre-processing of the electromyogram signals can be handled similarly as for the electroencephalogram signals. Because of the lower complexity of the electromyogram signals compared to the electroencephalogram signals, some preprocessing measures can be omitted in the relevant step. For example, only the processing using the digital band-pass filter can be performed to save the cost and time for data processing. In step S403, the feature extraction of the pre-processed first physiological information and second physiological information can be performed. This can be done, for example, using signal processing methods such as fast Fourier transformation, spectral estimation, wavelet transformation, etc., so that the time-frequency signal extraction of the pre-processed electroencephalogram signals takes place. Here, different frequency sections of the electroencephalogram signals are used, for example, to express different types of signals. With the feature extraction step, the signals to be analyzed can be extracted with respect to frequency segments of interest. The feature extraction of the pre-processed second physiological information can be carried out independently of the feature extraction of the electroencephalogram signals. In this case, the time-frequency feature extraction can also be carried out by wavelet transformation. Here, the discrete feature vectors can be extracted for each frequency section of interest.

In step S404, the intensity assessment of the first physiological information and second physiological information takes place. For example, a first intensity assessment device is used to check the first physiological information with regard to an intention to move. In certain cases, certain sections of physiological information contain conscious activities, but without referring to a specific intention to move. If these normal thought fragments are mistakenly used to control vehicle operation, it can lead to latent risks in driving. Therefore, for example, a first threshold value judgment device is provided for performing the intensity judgment of the discrete feature vectors extracted from the electroencephalogram signals. If it is determined that the intensity value of these discrete feature vectors of the electroencephalogram signals exceeds a first threshold value, this indicates that the electroencephalogram signals in question may reflect an intention to move. Otherwise, this shows that the electroencephalogram signals in question belong only to normal thought fragments and cannot reflect any intention to move. In addition, in step S404, a check of the second physiological information with regard to an intention to move can be carried out, for example by means of a second intensity assessment device. In some cases, the muscle electrodes on the skin's surface measure only slight electrical potential differences, which may simply be natural contractions of the muscle fibers caused by factors such as ambient temperature, lighting conditions, fatigue, etc. Thus, such surface electromyogram signals are not indicative of actual intention to move. A second threshold value assessment device for comparing the discrete feature vectors of the electromyogram signals with a second threshold value is provided, for example, in order to distinguish such interference influences. If the intensity value of these discrete feature vectors is determined to exceed a second threshold, this indicates that the electromyogram signals in question may reflect intention to move. Otherwise, this shows that the electromyogram signals in question cannot reflect any intention to move.

After the intensity assessment of the first and second physiological information has been carried out, a further assessment can be carried out as to whether both the first physiological information and the second physiological information reflect an intention to move. When assessing the intention to move, the joint assessment of the first physiological information recorded at a first time and the second physiological information recorded at a second time takes place, with the second time mentioned being downstream of the first time according to a predefinition. It can thus be ensured that there is a correspondence between the first physiological information and the second physiological information when analyzing one and the same driving intention. If it is judged that at least one of the two kinds of physiological information does not reflect an intention to move, it returns to step S401 to acquire the first and second physiological information again.

If it is judged that both kinds of physiological information reflect an intention to move, a first intention selectable series is formed with respect to the first physiological information and a second intention selectable series is formed with respect to the second physiological information in step S405. Here, for example, the feature arrays of the first and second physiological information are entered into a correspondingly pre-trained machine learning model in order to create an input-output relationship using algorithms for pattern recognition and to determine the potential driving intention. For example, the support vector machine model, the nearest neighbor model KNN, the decision tree model, the convoluted neural network model CNN, etc. are used as machine learning models. The recognition method based on the support vector machine model mainly has the following strengths: the problem of non-linearity of the data is solved, as well as a strength related to the problem of high-dimensional data. With regard to the first physiological information, all potential driving intentions are combined into a first series of intentions that can be selected. With regard to the second physiological information, all potential driving intentions are combined into a second series of intentions that can be selected.

Then, in step S406, the target driving intention is selected from the list of intentions available for selection based on the temporal correlation and/or content-related correlation between the first physiological information and second physiological information. Here, for example, with the aid of a pre-trained recurrent neural network, a confidence probability is assigned for each potential driving intention in the intention row available for selection, with subsequent output of the potential driving intention with the highest confidence probability as the named target driving intention.
In step S407, the control signals are generated on the basis of the output result of the recurrent neural network, with the relevant control signals being sent to the corresponding vehicle control unit (e.g. accelerator pedal, brake pedal, steering device, etc.) in order to trigger vehicle operations in connection with the target driving intention mentioned . After the target driving intention has been detected, it can be returned to step S401 to detect another group of physiological information.
5 12 shows the structure of the recurrent neural network for realizing the cross-validation of the first physiological information and the second physiological information in the method according to the present invention.

Here the input layer 401, the hidden layer 402 and the output layer 403 of the recurrent neural network are shown. For example, X1-X3 mean vectors of the input layer 401, H1-H2 vectors of the hidden layer 402 and Z1-Z3 vectors of the output layer 403.
In a specific embodiment, it is provided that the input of the relevant recurrent neural network is at first a first intention row available for selection determined from first physiological information, X1 being, for example, the potential driving intention “braking”, X2, for example, expresses the potential driving intention to “accelerate” and X3, for example, expresses the potential driving intention to “take the gas off”. The relevant first intention row available for selection has already been recorded in the previous module. At a second time, the input of the relevant recurrent neural network is a second eligible intent series determined from second physiological information, where the relevant first eligible intent series has, for example, partial overlaps with elements of the second eligible intent series, which allows for a better analysis of the current units and knowledge of the chronological sequence made possible by the model. In the hidden layer are metrics of the feature parameters of electroencephalogram and elec romyogram included. These metrics were determined when the neural network in question was pre-trained and can be adjusted in time.

It should be noted here that the signals associated with movements of the human body all come from the cerebrum, so that when the intention to drive is formed in the driver's cerebrum, the electroencephalogram signals of the cerebral cortex first generate control signals, which are transmitted through neuromuscular connections as a medium to hand/ Foot muscle fibers are transferred. Through repeated depolarization and repolarization, the muscle fiber cells ultimately generate the electrical action potential on the skin surface.
Because of the non-simultaneity of electroencephalogram signals and electromyogram signals and the relationship in terms of content, the first intention series available for selection and the second intention series available for selection can be regarded as series sections with a time sequence, with the recurrent neural network establishing a relationship between the current output of a series and the previous information. The recurrent neural network can thus, for example, remember the information at a first time and use this information to influence the assignment of the corresponding confidence probability at a second time. As in 4 As shown, the hidden layer of each time is determined not only by the input layer of the time in question, but also the hidden layer of the previous time. Thus, the output of physiological information of each individual model state is transmitted to the recurrent neural network in order to make an overall assessment with a summary of several model states. Thus, the relevant recurrent neural network collects the pre-estimation results of each individual model state at any time. After the summary analysis of several model states, the output of the relevant recurrent neural network is the final recognition result of the driving intention. By using the recurrent neural network, the correlation of electroencephalogram signals and electromyogram signals within a certain time range is effectively used, this correlation enabling more reliable network training, so that the output of different driving intentions and relative confidence probabilities is possible. Thus, the complementation between electroencephalogram signals and electromyogram signals increases the accuracy and stability of the detection of movement patterns.

The particular exemplary embodiments of the present invention described in detail are for the purpose of illustration only and should not be construed as limiting the scope of the present invention. Substitutions, modifications and rearrangements can be obtained without departing from the spirit and scope of the present invention.

Claims (10)

Method for vehicle control assistance based on physiological information of the driver, said method comprising the following steps: Step S1: acquisition of first physiological information of the driver; Step S2: acquiring second physiological information of the driver, the second physiological information being different from the first physiological information; Step S3: generating control signals for the vehicle based on the first physiological information and second physiological information by cross-validation; as well as Step S4: Output of the control signals for controlling the driving operation of the vehicle. procedure after claim 1 wherein the first physiological information comprises driver's electroencephalogram signals, while the second physiological information comprises driver's electromyogram signals and/or surface electromyogram signals. procedure after claim 1 or 2 , wherein step S3 comprises: step S31: summary determination of the driver's target driving intention based on the first physiological information and second physiological information; and step S32: generation of control signals on the basis of the target driving intention, the control signals serving to trigger vehicle activations in connection with the target driving intention. procedure after claim 3 , wherein step S31 comprises: outputting the driver's potential driving intentions with respect to the first physiological information and second physiological information using a machine learning model; Combination of all potential driving intentions with formation of an intention row to choose from; and selection of the target driving intention from the list of intentions available for selection based on the temporal correlation and/or content-related correlation between the first physiological information and the second physiological information. procedure after claim 3 or 4 , wherein the selection of the target driving intention comprises: with the aid of a previously trained artificial neural network, for each potential driving intention in the series of intentions available for selection, assignment of a confidence probability; and outputting the potential driving intention with the highest confidence probability as the named target driving intention. Method according to any of the Claims 1 - 5 , wherein step S3 further includes: judging whether the first physiological information and second physiological information reflect an intention to move, wherein only in the case that both the first physiological information and the second physiological information reflect an intention to move, the first physiological information and second physiological Information is used to generate control signals. procedure after claim 6 , With regard to the first physiological information recorded for a first time and the second physiological information recorded for a second time, a joint assessment of the intention to move is made, the second time being subsequent to the first time according to a predefinition. Method according to any of the Claims 1 - 7 , wherein step S3 further comprises: signal pre-processing of the first physiological information and second physiological information and extraction of time-frequency signal features. On physiological information of the driver based device for vehicle control assistance (1), wherein the device (1) carrying out the method according to any one of Claims 1 - 8th serves and wherein the device (1) comprises: a first acquisition module (10), which is designed to acquire first physiological information of the driver; a second acquisition module (20) which is designed to acquire second physiological information about the driver, the second physiological information differing from the first physiological information; an analysis module (30) configured to generate control signals for the vehicle through cross-validation based on the first physiological information and second physiological information; and an output module (40) which is designed to output the control signals for controlling the driving operation of the vehicle. Computer program product, wherein the computer program product comprises a computer program, wherein when the computer program is executed by a computer, performing the method according to any one of Claims 1 - 8th he follows.

Images