DE102022004819A1 - Method for determining the emotional state of vehicle occupants, driver assistance system and vehicle - Google Patents

Abstract

The invention relates to a method for determining the emotional state of vehicle occupants, wherein the vehicle occupants located in a vehicle interior (1) are detected by means of at least one detection means in the form of a camera (2.1), a microphone (2.2) and/or a haptic sensor (2.3), sensor data generated by the detection means are processed by a computing unit (3), the computing unit (3) determines emotional characteristics (EM) of the individual vehicle occupants by processing the sensor data, determines an emotional state for each vehicle occupant that is dependent on the detected emotional characteristics (EM), and recognizes a boredom reaction (LR) carried out by a vehicle occupant if at least one emotional characteristic (EM) of the vehicle occupant exceeds a person-specific and emotional characteristic-specific threshold value (SW1, SW2). The method according to the invention is characterized in that the computing unit (3) predicts for at least one vehicle occupant a future point in time at which the respective vehicle occupant is likely to carry out the next boredom reaction (LR).

Description

The invention relates to a method for determining the emotional state of vehicle occupants according to the type defined in more detail in the preamble of claim 1, a driver assistance system according to the type defined in more detail in the preamble of claim 9 and a vehicle with such a driver assistance system.

Especially on long car journeys, children quickly become bored and start arguing or otherwise vent their anger and aggression on the situation. The children's parents experience this regularly and often begin each journey expecting that another argument will soon arise. The uncertainty about when exactly the relaxed part of the car journey will end can lead to tension among the parents. This creates a need for methods and means to make car journeys more pleasant.

The EN 10 2021 005 092 A1 describes a method and a device for detecting emotional states of vehicle occupants. The corresponding vehicle has interior sensors such as cameras, microphones and haptic sensors with which the vehicle's passengers are recorded. With the help of the cameras, visual information can be evaluated and, for example, gestures, facial expressions and objects can be recognized, which allows conclusions to be drawn about the emotional state of the vehicle occupants and events in the vehicle interior to be evaluated. With the help of the microphones, acoustic information can be evaluated and, for example, the conversation of the vehicle occupants can be analyzed or noise such as baby crying can be detected. With the help of haptic sensors, vital parameters such as skin conductivity, skin temperature, pulse rate, respiratory rate and the like can be recorded. All of this information is aggregated to form an overall picture of the emotional state of the vehicle occupants. With the help of statistical methods and methods based on artificial intelligence, a computing unit detects changes in the vehicle interior and assigns an emotional progression to the individual passengers over time. If a passenger's emotion changes, the computing unit determines a reason for this change in emotion. Internal and/or external vehicle functions are controlled on this basis. For example, it can be determined that a passenger of a chauffeur service feels fear due to the chauffeur's driving style, whereupon the corresponding driver is no longer assigned to this customer.

In addition, the US 2020/0310528 A1 a vehicle system for providing information to a vehicle driver in response to the emotions of a passenger. The disclosed system monitors the emotions of the individual vehicle occupants and makes suggestions to the vehicle driver to perform an action that matches the corresponding emotion of another vehicle occupant. For example, the corresponding system suggests certain topics of conversation. The vehicle driver therefore has to provide less cognitive effort to respond to their passenger and can thus devote more of their attention to what is happening while driving.

• - Jessica Murray (2022), T-minus 10: Statistician writes formula to predict kids' backseat tantrums https://www.theguardian.com/society/2022/aug/22/statistician-writes-formulato-pred ict-kids-backseat-tantru ms
• - Claudia Rowan, PA and Nick Wood (2022) Are we there yet? Maths expert calculates formula to predict back seat tantrums https://www.nottinghampost.com/news/uk-worldnews/yet-maths-expert-calculates-formula-7492694
• - Katharina Koerth (2022) 70 Minuten bis zum nächsten Wutanfall, https://www.spiegel.de/reise/autofahren-mit-kindern-was-eltern-gegen-wutanfaelle-tunkoennen-a-5eab1e1d-e0a4-4b1d-ab7d-428777dd626c

Furthermore, the British mathematician James Hind has developed a heuristic formula for predicting tantrums in children in vehicles. Please refer to the following sources:

• - Jessica Murray (2022), T-minus 10: Statistician writes formula to predict kids' backseat tantrums https://www.theguardian.com/society/2022/aug/22/statistician-writes-formulato-pred ict-kids-backseat-tantru ms
• - Claudia Rowan, PA and Nick Wood (2022) Are we there yet? Maths expert calculates formula to predict back seat tantrums https://www.nottinghampost.com/news/uk-worldnews/yet-maths-expert-calculates-formula-7492694
• - Katharina Koerth (2022) 70 minutes until the next tantrum, https://www.spiegel.de/reise/autofahren-mit-kindern-was-eltern-gegen-wutanfaelle-tunkoennen-a-5eab1e1d-e0a4-4b1d-ab7d-428777dd626c

The recognition of emotions of people based on facial, speech or physical signals using appropriate sensors is also known from the US 2012/0130196 A1 known.

• - MacWhinney, B. (2000). The CHILDES project: The database (Vol. 2). Psychology Press.
• - Sagae, K., MacWhinney, B., & Lavie, A. (2004). Adding Syntactic Annotations to Transcripts of Parent-Child Dialogs. In LREC.
• - Grigonyte, G., & Björkenstam, K. N. (2016). Language-independent exploration of repetition and variation in longitudinal child-directed speech: a tool and resources. In Proceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition (pp. 41-50) .

Furthermore, the analysis of parent-child communication using methods of computer-assisted linguistics, also known as neuro-linguistic programming (NLP), is the subject of current research. For example, reference is made to:

• - MacWhinney, B. (2000). The CHILDES project: The database (Vol. 2). Psychology Press.
• - Sagae, K., MacWhinney, B., & Lavie, A. (2004). Adding Syntactic Annotations to Transcripts of Parent-Child Dialogues. In LREC.
• - Grigonyte, G., & Björkenstam, KN (2016). Language-independent exploration of repetition and variation in longitudinal child-directed speech: a tool and resources. In Proceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition (pp. 41-50) .

The present invention is based on the object of specifying an improved method for determining the emotional state of vehicle occupants, with the aid of which the comfort for the person driving a vehicle in which the vehicle occupants are traveling can be further increased.

According to the invention, this object is achieved by a method for determining the emotional state of vehicle occupants with the features of claim 1. Advantageous embodiments and further developments as well as a driver assistance system for carrying out the said method and a vehicle with such a driver assistance system emerge from the dependent claims.

A generic method for determining the emotional state of vehicle occupants, wherein the vehicle occupants located in a vehicle interior are detected by means of at least one detection means in the form of a camera, a microphone and/or a haptic sensor, sensor data generated by the detection means are processed by a computing unit, the computing unit determines emotional characteristics of the individual vehicle occupants by processing the sensor data, determines an emotional state for each vehicle occupant that depends on the detected emotional characteristics, and recognizes a boredom action performed by a vehicle occupant when at least one emotional characteristic of the vehicle occupant exceeds a person-specific and emotional characteristic-specific threshold value, is further developed according to the invention in that the computing unit predicts a future point in time for at least one vehicle occupant at which the respective vehicle occupant is likely to perform the next boredom action.

The method according to the invention therefore not only allows the emotions or moods of the individual vehicle occupants to be recognized, but is also able to estimate a future mood. The method is able to recognize when the mood of a vehicle occupant reaches a certain critical level, in other words when the mood changes. To do this, the computing unit predicts the point in time at which the corresponding vehicle occupant is likely to carry out the situational reaction. The point in time or the period of time until the point in time is reached can then be output in the vehicle, for example, and displayed to the vehicle occupants. This tells the person driving the vehicle how long a quiet drive is still possible. This ensures relaxation, as the person driving the vehicle does not have to constantly worry that a tense driving situation will arise, for example due to arguments, shouting or the like.

In general, such a time can be predicted for every vehicle occupant, i.e. a person sitting in the driver's seat, front passenger seat or in the back of the vehicle. The person can be of any age. This is particularly advantageous for the children of the person driving the vehicle who are travelling in the back of the vehicle. However, it could also be adults or elderly people.

The boredom reaction is an activity carried out by a vehicle occupant that can increase the stress level of other vehicle occupants. This includes, for example, loud shouting, throwing objects around in the vehicle, wild gesticulations, kicking the front seat, constantly opening and closing the window lifters or any other annoying action. In order to assess whether a boredom action is being carried out, the computing unit evaluates the respective vehicle occupant based on the emotional characteristics. Different levels of emotion-specific threshold values are defined for each person. In this way, the emotions of the vehicle occupants can be determined and determined in accordance with methods and procedures known from the state of the art. The state of mind ranges from relaxed and passive to excited and active. The emotions experienced can be positive, i.e. pleasant, or negative, i.e. unpleasant. The individual emotional characteristics can be checked using the recording tools.

Visual recording devices such as cameras can be used to record the gestures and facial expressions of the vehicle occupants. This makes it possible to see whether a certain person has a relaxed face or, for example, a face distorted with anger. If the person is lying still or sitting upright in a relaxed manner, this is an indication of relaxation. If, on the other hand, the person is constantly sliding back and forth on their vehicle seat or gesticulating wildly, this is an indication of an agitated state of mind. Individual threshold values can be set, for example how distorted a certain face must be or how wildly, i.e. with what amplitude and frequency, the person must gesticulate so that a certain person-specific and emotion-specific threshold is considered to have been exceeded. Objects in the Vehicle interiors can be detected. Image recognition algorithms can be used to determine, for example, that a baby has a pacifier in its mouth, that a child is throwing a ball at its sibling, or similar. Other visually detectable characteristics can be a person's age or gender, and other visually detectable activities include glancing at the clock, particularly taking into account the frequency of gaze per unit of time, which can determine that a person is unusually often looking at the clock, drinking coffee, looking out the window, typing on a cell phone, and the like.

Acoustic recording devices such as microphones can be used to detect noise and record conversations in the vehicle interior. Using computer-aided linguistics methods, conversations can be examined for their content, sentiment and style. Methods such as topic modeling or semantic role labeling allow different levels of granularity for the content analysis of dialogues. Sentiment analysis methods can be used to capture the attitude or mood conveyed in a dialogue. This makes it possible to determine, for example, whether a person has a positive or negative attitude or what their tone is. Style classification methods can be used to evaluate the style of the formulation. For example, a concise style or the issuing of orders indicates a tense situation, while complex explanations or lengthy explanations can be an indication of a relaxed atmosphere.

With the help of haptic sensors, physiological characteristics of vehicle occupants can be recorded and vital signs can be recorded. The term "haptic" is understood here in a broader sense and refers to any sensor that directly measures the physical characteristics of vehicle occupants. These include, for example, touch-sensitive sensors, skin conductivity sensors, temperature sensors, infrared radiation-detecting sensors and the like. For example, an increased skin temperature and/or conductivity can be an indication of stress. The same applies to detecting an increase in breathing rate or pulse rate.

Accordingly, for example, an eyelid blinking frequency, degree of facial distortion, tone of voice, content of conversation, skin conductivity, pulse rate and the like form an emotion characteristic. Each person is different, so that for each person, individual emotion characteristic-specific threshold values are set through observation, after which the occurrence of a boredom action is recognized for that person. For some people, a boredom reaction is characterized by loud shouting, for example. Accordingly, the boredom action is recognized when the volume level of the person's voice exceeds a certain loudness. For another person, the boredom reaction is characterized by, for example, a physical attack on a fellow passenger, which is recognized when the person's gestures are particularly pronounced. For this, the person moves, for example, out of a predefined body position box in the vehicle interior and/or the amplitude of the movements made by the person's body parts is particularly large.

Individual emotion characteristics can be defined in the form of time series information, aggregated parameters or as values statistically determined over a period of time.

• - in der Recheneinheit eine personenspezifische heuristische Formel implementiert ist, die Recheneinheit zumindest die erfassten Emotionsmerkmale in die heuristische Formel eingibt, woraufhin ein heuristisches Ergebnis erhalten wird, die Recheneinheit das heuristische Ergebnis mit einem gespeicherten Ergebnisschwellwert vergleicht und den künftigen Zeitpunkt der prädizierten Langeweilereaktion in Abhängigkeit des Vergleichs von heuristischem Ergebnis und Ergebnisschwellwert bestimmt; oder
• - die Recheneinheit ein personenspezifisches Maschinenlernmodell ausführt, zumindest die erfassten Emotionsmerkmale in das Maschinenlernmodell als Eingangsdaten einliest und das Maschinenlernmodell daraus den künftigen Zeitpunkt der prädizierten Langeweilereaktion bestimmt, wobei die Recheneinheit zum Trainieren des Maschinenlernmodells dem Maschinenlernmodell neben den erfassten Emotionsmerkmalen als Trainingsdaten den ermittelten Gemütszustand übergibt, sodass das Maschinenlernmodell den selbst ermittelten künftigen Zeitpunkt der prädizierten Langeweilereaktion mit dem Zeitpunkt, zu dem tatsächlich eine Langeweilereaktion erkannt wird, abgleicht.

An advantageous further development of the procedure provides that

• - a person-specific heuristic formula is implemented in the computing unit, the computing unit enters at least the recorded emotion characteristics into the heuristic formula, whereupon a heuristic result is obtained, the computing unit compares the heuristic result with a stored result threshold and determines the future time of the predicted boredom reaction depending on the comparison of the heuristic result and the result threshold; or
• - the computing unit executes a person-specific machine learning model, reads at least the recorded emotion characteristics into the machine learning model as input data and the machine learning model uses this to determine the future time of the predicted boredom reaction, whereby the computing unit, in order to train the machine learning model, passes the determined state of mind to the machine learning model in addition to the recorded emotion characteristics as training data, so that the machine learning model compares the self-determined future time of the predicted boredom reaction with the time at which a boredom reaction is actually detected.

As a result, the predicted point in time is predicted directly. It is not necessary to first determine the future course of all relevant emotion characteristics.

A heuristic formula can be formulated, for example, in analogy to the theory of James Hind mentioned at the beginning. However, different calculation models can also be used to calculate formation of the formula. In order to do justice to the character traits of individual people, individual heuristic formulas are used for the different vehicle occupants. At the start of a journey, a standard formula can be used which is individually tailored to the circumstances and character traits of each person by the computing unit under continuous observation of the vehicle occupants while the vehicle is driving. To do this, the heuristic formula and its parameters can be varied randomly (by trial and error) or systematically, for example by means of a variation calculus of the corresponding functional or by standard methods of parameter determination, such as maximizing the likelihood or minimizing the negative log likelihood. However, fixed routines can also be implemented in the computing unit, whereby certain variables or arithmetic operations of the formula are specifically changed in accordance with changes in the emotional characteristics of the vehicle occupants detected over time. Such a change pattern could, for example, consist of an increase in the skin conductivity and the pulse rate of a vehicle occupant being detected, but no gesticulation or loud shouting being detected. If such a pattern is detected, the heuristic formula is changed according to previously clearly defined rules. This leads to the prediction of an earlier or later point in time at which a boredom reaction can be expected for this person, for the behavior observed by this person.

Machine learning methods can also be used to estimate the point at which a boredom reaction is about to occur. Artificial intelligence, specific machine learning methods, such as deep neural networks, or groups of customizable decision trees can be used for this. Over the course of the journey, the respective machine learning model is trained on the characteristics of the individual passengers. A standard and pre-trained machine learning model can be used initially, or an untrained machine learning model, which then learns from scratch.

The method according to the invention can be carried out reliably if the machine learning model has been sufficiently trained. This is the case if a time difference between the predicted occurrence of a boredom reaction and the actual time at which the boredom reaction is carried out by this person is smaller than a predetermined value. This predetermined value can be arbitrarily large and can be, for example, 30 minutes, 5 minutes or even 30 seconds. For example, the machine learning model is sufficiently trained if the computing unit predicts the occurrence of a boredom reaction of a vehicle occupant in one hour and 45 minutes and the boredom reaction actually occurs after one hour and 30 minutes, i.e. 15 minutes earlier. In this case, the tolerable time difference was, for example, 25 minutes.

A further advantageous embodiment of the method according to the invention further provides that the computing unit determines at least one situation parameter taking into account sensor data generated by detection means monitoring the vehicle interior and/or the vehicle surroundings and/or information provided to the computing unit by a vehicle subsystem, and the computing unit reads the situation parameter into the heuristic formula or the machine learning model as further input data to predict the next boredom reaction. This enables an even more comprehensive assessment of the overall situation in the vehicle, which enables the computing unit to detect and also predict the occurrence of a boredom reaction even more reliably. The detection means monitoring the vehicle interior can, for example, be the aforementioned vehicle interior cameras, microphones and/or haptic sensors. The vehicle can also have external sensors with which the vehicle surroundings can be monitored. These include, for example, external cameras and/or external microphones. Such an external camera can, for example, be designed as a mono camera or stereo camera and can be oriented as desired on the vehicle. This makes it possible to detect the area to the front or rear of the vehicle or even the entire vehicle surroundings. There can also be sensors for detecting depth information such as lidars, ultrasonic sensors, radar sensors and the like. This makes it possible to detect, for example, that the vehicle is approaching the end of a traffic jam, that adverse weather conditions such as heavy snowfall or rain are prevailing, that visibility is limited due to a low sun or fog, that an overtaking vehicle is cutting into the driver's lane, that the driver is driving in a traffic jam or slow-moving traffic and the like. All of these situations can affect the stress level or mood of the vehicle occupants and thus influence the point at which a boredom reaction is imminent.

To determine situation parameters, information provided by vehicle subsystems can also be evaluated. For example, the computing unit can calculate a wheel speed, a tire slip value, a requested engine power a brake pedal position, a turn signal being activated, a setting for the vehicle's air conditioning, and the like. This enables the computing unit to assess the driving situation even more precisely and thus evaluate the influence on the emotions of the vehicle's occupants. In other words, the situation parameter(s) are properties that characterize the current driving situation.

A further advantageous embodiment of the method according to the invention further provides that the computing unit determines a boredom reaction tolerance value for the person driving the vehicle, taking into account the emotional state of the person driving the vehicle, and shifts the predicted time of the next boredom reaction forwards or backwards in time depending on the extent of the boredom reaction tolerance value. In other words, different people can have different tolerance thresholds for tolerating boredom reactions. For example, while a first person becomes agitated when the children in the back of the vehicle start talking to each other louder and faster, there are other people who remain relaxed even when the children in the back are shouting at each other and becoming violent. This is taken into account with the help of the boredom reaction tolerance value.

The computing unit monitors the person driving the vehicle and their vital parameters in particular. If the computing unit then detects a boredom reaction from a vehicle occupant according to the predefined settings and the person driving the vehicle remains relatively calm, the computing unit changes the predefined settings so that the boredom reaction is only detected when the recorded emotional characteristics of the vehicle occupant who is exhibiting the boredom reaction exceed relatively high person-specific and emotional characteristic-specific threshold values. Accordingly, it takes longer or shorter until a boredom reaction is detected on the part of the vehicle occupant.

According to a further advantageous embodiment of the method, the vehicle occupants are identified, a group composition of the vehicle occupants is determined and the computing unit takes the group composition into account to predict the next boredom reaction, in particular by adjusting a person-specific and emotion-specific threshold value, adjusting the boredom reaction tolerance value of the person driving the vehicle and/or taking the group composition into account as a further input variable for the heuristic formula or the machine learning model. Identifying the vehicle occupants has several advantages. As already mentioned, an individual model, i.e. a digital image, is generated in the computing unit for each vehicle occupant, which is used to evaluate the occurrence of a boredom reaction. The longer the individual person is observed by the computing unit, the more reliably the person's actual behavior can be learned. It is particularly advantageous if the person in question is observed not only during a single trip, but during a longer period of use of the vehicle, for example over several years. The behavior of the person in question can also change over time. By identifying individual people, the corresponding digital image of the respective person can be maintained over a relatively long period of time. Accordingly, each person is assigned a unique heuristic formula or a machine learning model specifically trained for that person.

Vehicle occupants can be identified using tried and tested methods. For example, biometric features can be recognized using the detection devices. For example, a face scan, an iris scan, a fingerprint scan, a voice pitch analysis and the like can be carried out. People can also be recognized based on the devices they carry. For example, a vehicle occupant's smartphone can establish a communication connection to the computing unit via Bluetooth, Wi-Fi or wired, and the person can be identified by recognizing the characteristic Mac address or Bluetooth address of the smartphone. A vehicle key carried by the person can also serve as an identification feature. Vehicle occupants can also log into the computing unit with an individual user name, possibly supplemented by a personal password.

The group composition, i.e. who exactly is riding in the vehicle, can have a serious influence on the occurrence of boredom reactions or the tolerance level of a boredom reaction. For example, if a laid-back father drives his children to a sporting event, the children could start arguing early on. However, if the mother, who is stricter in comparison, drives the children to the sporting event, the children could behave longer. In a group consisting of father, mother and children, the father or mother could also have a higher tolerance threshold for the occurrence of boredom reactions. because the presence of the respective spouse has a calming effect.

The influence of other factors on the time at which a boredom reaction threatens to occur, such as demographic background, can also be taken into account. This includes, for example, the age, gender, origin of the vehicle occupants and the like, so that, for example, a boredom reaction is generally expected to occur earlier in children.

A further advantageous embodiment of the method according to the invention further provides that the computing unit predicts a time of falling asleep for this vehicle occupant depending on at least the recorded emotional characteristics of this vehicle occupant and takes this into account when predicting the next boredom reaction, in particular by adjusting a person-specific and emotional characteristic-specific threshold value, adjusting the boredom reaction tolerance value of the person driving the vehicle and/or taking into account the predicted time of falling asleep as a further input variable for the heuristic formula or the machine learning model. In addition to the emotional characteristics, said situation parameters and/or other variables can also be used to determine the time of falling asleep. For example, the computing unit can estimate that a vehicle occupant will fall asleep in x minutes if a reduction in the vehicle occupant's pulse rate and breathing rate is detected, the eyes close and/or a particularly calm voice is present. An evaluation of the situation parameters allows external boundary conditions to be taken into account. For example, detected precipitation, a late hour, or a monotonous journey at a constant speed can also encourage falling asleep. The computing unit is thus able to estimate whether a particular vehicle occupant will fall asleep. Accordingly, the initially predicted time at which a boredom reaction can be expected can be pushed back.

Sleep can also affect the boredom reaction tolerance value of the person driving the vehicle. For example, if two siblings are traveling in the vehicle and one of the children falls asleep, the child who is awake should not be so loud so as not to wake the sleeping child. The person driving the vehicle will therefore be aroused by recognizing comparatively quiet exclamations from the awake child or less pronounced gesticulations from the awake child, as the person driving the vehicle does not want to endanger the sleep of the other child.

A further advantageous embodiment of the method also provides that a distraction database is kept in the computing unit, the distraction database containing information that describes the influence of various actions on the individual emotional characteristics of people, and the computing unit, after determining a predicted boredom reaction and before reaching the predicted time at which this boredom reaction is likely to be carried out, outputs at least one action contained in the distraction database as a suggestion to the person driving the vehicle to carry out, the computing unit recalculating the predicted time, in particular taking into account the influence of the action on the predicted time at which the next boredom reaction is likely to be carried out. This significantly improves the comfort for the person driving the vehicle. The computing unit thus instructs the person driving the vehicle to carry out certain activities that push back the time at which a boredom reaction is expected to be carried out. This makes it possible to drive for longer in a relaxed state.

For example, the distraction database may include the following actions: passing a whole grain or chocolate bar, playing an audio book, playing a video, taking a break from the journey, for example at a rest stop, and the like. Corresponding instructions may be issued in the vehicle in a variety of ways. For example, a written instruction may be displayed on any display device in the vehicle and/or an acoustic instruction may be played via loudspeakers in the vehicle.

A vehicle subsystem can also be automatically controlled to provide a specific functionality. For example, the vehicle's air conditioning can be automatically reset or the vehicle's infotainment system can automatically tune to a specific radio station at a specific volume.

A further advantageous embodiment of the method further provides that the heuristic formulas, trained machine learning models and/or data used to train the machine learning models, which are individually adapted to a large number of fleet vehicles and to a large number of vehicle occupants, are transmitted to a central computing device and the central computing device derives a generally valid heuristic standard formula and/or a generally valid standard machine learning model from the totality of the information received. An example of a central computing device is a cloud server, also known as a backend. The central computing device can be operated by the vehicle manufacturer, for example. The vehicles can communicate with the central computing device via mobile communications, Wi-Fi, or even via cable, for example during a maintenance interval. The data generated during the service life of the fleet vehicles can thus be aggregated and combined to form a particularly accurate prediction model. The individually adapted heuristic formulas and/or trained machine learning models can be combined directly to form new models, or a new adaptation or training process can be carried out on the central computing device based on the corresponding raw data. In this way, in addition to the already adapted heuristic formulas and/or trained machine learning models, the emotional characteristics, situation parameters, and other variables recorded during the respective journeys, as well as the mood states and boredom reactions determined from them, can also be transferred to the central computing device. Such generally valid heuristic standard formulas or generally valid standard machine learning models can then be integrated into new vehicles, which already allow a satisfactory prediction of a potentially occurring boredom reaction ex works.

A driver assistance system comprising at least one detection means, a computing unit for evaluating sensor data generated by the detection means and an output means that can be controlled by the computing unit is set up according to the invention to carry out a method described above. For this purpose, the driver assistance system is designed using hardware and software components. Thus, the execution of a corresponding program code on a processor of a corresponding computing unit causes the method steps described above to be carried out.

According to the invention, a vehicle comprises such a driver assistance system. The vehicle can be any vehicle such as a car, truck, van, bus or the like. In such a vehicle, a particularly comfortable and relaxing journey is possible for a large number of vehicle occupants.

Further advantageous embodiments of the method according to the invention for determining the emotional state of vehicle occupants, the driver assistance system and the vehicle also emerge from the exemplary embodiments which are described in more detail below with reference to the figures.

• 1 eine schematisierte Draufsicht auf ein erfindungsgemäßes Fahrzeug;
• 2 ein schematisierter Aufbau einer zur Durchführung eines erfindungsgemäßen Verfahrens zum Ermitteln des Gemütszustands von Fahrzeuginsassen durch das in 1 gezeigte Fahrzeug verwendete Recheneinheit;
• 3 ein Ablaufdiagramm des erfindungsgemäßen Verfahrens; und
• 4 ein Diagramm, zeigend den Verlauf zweier bei einem Fahrzeuginsassen beobachteten Emotionsmerkmale über die Zeit.

Showing:

• 1 a schematic plan view of a vehicle according to the invention;
• 2 a schematic structure of a device for carrying out a method according to the invention for determining the emotional state of vehicle occupants by means of the 1 computing unit used in the vehicle shown;
• 3 a flow chart of the method according to the invention; and
• 4 a diagram showing the course of two emotional characteristics observed in a vehicle occupant over time.

A in 1 The vehicle 6 according to the invention shown comprises at least one detection means of the type of a camera 2.1, a microphone 2.2 and/or a haptic sensor 2.3. With the aid of the detection means, a vehicle interior 1 of the vehicle 6 is detected. This serves to detect and observe vehicle occupants (not shown) and to 4 to recognize the emotional characteristics EM of the vehicle occupants and to determine their respective expression. For example, facial expressions, gestures and/or objects in the vehicle interior 1 can be recognized using the camera 2.1, noise, shouting or even conversations can be detected using the microphone 2.2 and the content of the conversation can be determined using speech analysis methods. Vital parameters of the vehicle occupants can be recorded using haptic sensors 2.3. In 4 As an example, a first emotion feature EM1 is shown as gestures performed by a person, realized by the amplitude of a respective body part movement, and as a second emotion feature EM2 is the volume of the person's voice.

Sensor data generated by the recording devices are evaluated by a computing unit 3. This compares the recorded emotion characteristics EM with person-specific and emotion characteristic-specific threshold values SW1, SW2, which are also in 4 qualitatively for the first emotion feature EM1 and the second emotion feature EM2. The computing unit 3 detects the occurrence of a boredom reaction LR when, depending on the configuration, one or more emotion features EM exceed their respective corresponding threshold value SW1, SW2. This is shown in 4 represented by a hatched area.

A boredom reaction (LR) is a reaction carried out by a vehicle occupant. Actions that can increase the stress level of other vehicle occupants, especially the person driving the vehicle. Examples include kicking a front seat, shouting loudly, arguments between two vehicle occupants, throwing objects around, and the like.

The method according to the invention is based on the idea of predicting a future point in time at which the next boredom reaction LR of a vehicle occupant is imminent. This information, i.e. the length of time until the boredom reaction LR is imminent, can then be issued to the vehicle occupants as information. For this purpose, the vehicle 6 comprises visual and/or acoustic output means, for example a display device 7.1 and loudspeakers 7.2. This informs the vehicle occupants how long an undisturbed journey with the vehicle 6 is likely to be possible. This reduces the stress level of the person driving the vehicle and allows the person driving the vehicle to plan the course of the rest of the journey better.

The vehicle 6 can also include external detection means in the form of, for example, a radar sensor 8.1, a lidar 8.2, a camera 8.3 and/or an ultrasonic sensor system 8.4. Using the external detection means, the vehicle 6 can detect its surroundings and assess the respective driving situation. In particular, the traffic situation around the vehicle 6 can be analyzed. The computing unit 3 can also receive information from a control unit 9 of the vehicle 6. One or more vehicle subsystems are executed on such a control unit 9 or such a control unit 9 is integrated into such a vehicle subsystem. This can be, for example, an engine control unit or a control unit 9 for providing a functionality such as a driver assistance system. The computing unit 3 thus has access to, for example, current weather service information, current traffic service information, parameters that describe a state of the vehicle, such as a requested engine torque, a current wheel speed, tire slip, a battery charge level and the like. This enables the computing unit 3 to evaluate the current driving situation particularly comprehensively and to take into account the influence on the emotional state of the vehicle occupants.

The variables that affect the mood of the vehicle occupants are read into a person-specific calculation model. For this purpose, the computing unit 3 can have or execute a person-specific heuristic formula or a person-specific machine learning model. The heuristic formula or the machine learning model determines from the monitored variables the said point in time at which the next boredom reaction LR of a respective vehicle occupant is likely to occur.

The vehicle 6 can also have a telecommunications unit 10, which enables communication with a central computing device 5 external to the vehicle. Communication can take place wirelessly, for example via mobile radio, Wi-Fi or the like. The vehicle 6 can be part of a vehicle fleet. The individual vehicles 6 of the vehicle fleet can exchange information with the central computing device 5. For example, emotion characteristics EM measured over the service life of the vehicle 6 as well as other sensor data and/or a heuristic formula or trained machine learning model tailored specifically to an individual vehicle occupant can be transmitted to the central computing device 5. This enables the central computing device 5 to derive a generally valid standard machine learning model or a generally valid heuristic standard formula. These standard formulas or standard machine learning models can, for example, be initially introduced into the vehicles 6 of the vehicle fleet after they have been manufactured, so that a sufficiently accurate prediction of the next time at which a boredom reaction LR is imminent can be made when the vehicles 6 are delivered. Over the remainder of the vehicle’s lifespan, this formula or machine learning model is then individually tailored to the respective persons using the corresponding vehicle 6.

2 shows the structure of the computing unit 3 in an enlarged detailed view. The computing unit 3 comprises a person identification module 3.1, a modelling module 3.2, a model module 3.3 and a recommendation module 3.4.

As an input variable, the computing unit 3 reads in data 201, which are provided by the internal recording means of the vehicle 6, as well as data 202, which are provided by the external recording means and the control units 9. The person identification module 3.1 reads in the data generated by the cameras 2.2 and is thus able to recognize and identify the individual people in the vehicle interior. Information generated from a visual analysis of the observed vehicle occupants is then forwarded to the modeling module 3.2 and the model module 3.3. The modeling module 3.2 then additionally reads in the data 201 and 202. The modeling module 3.2 collects and monitors the emotion characteristics EM over the observation period in a method step 210. In method step 215, the modeling module module 3.2 determines the occurrence of a boredom reaction LR. In method step 220, the modeling module 3.2 decides how the respective model used to estimate the future time of the renewed occurrence of a boredom reaction LR of the respective vehicle occupant should be further tailored to this vehicle occupant. If this is the said heuristic formula, for example, arithmetic operators can be changed or newly added or removed and the parameters taken into account, i.e. variables, can be changed. If, on the other hand, it is the said machine learning model, it is trained accordingly.

As output variable 203, the modeling module 3.2 outputs information that a boredom reaction LR is currently detected.

The model module 3.3 stores the heuristic formula or machine learning model mentioned. The model module 3.3 reads the data 201, 202 as well as the information received from the person identification module 3.1 and the modeling module 3.2 and uses this to determine a forecast for the time at which the next boredom reaction LR of a respective vehicle occupant is likely to occur. The model module 3.3 outputs this information as output variables 204.

The output variable 204 is read in by the recommendation module 3.4. The recommendation module 3.4 includes a distraction database 4 that includes information that describes the influence of various actions on the individual emotional characteristics EM of people. The goal is to issue a recommendation as to which action a vehicle occupant, for example the person driving the vehicle, can perform in order to postpone the time at which the boredom reaction LR of a vehicle occupant will occur. This enables a longer, undisturbed journey with the vehicle 6. For example, such an action can provide for giving a child their favorite cuddly toy, driving to a rest area and taking a break, playing an audio book, or the like. The computing unit 3 takes the respective preferences of the vehicle occupants into account. This makes it possible to postpone the predicted time at which a boredom reaction LR will occur particularly effectively. The recommendation module 3.4 issues a corresponding recommendation for action as output variable 205.

3 shows a flow chart of the method according to the invention, wherein the computing unit 3 has already been trained on known individuals. The method starts in a method step 301. In a method step 302, the computing unit 3 retrieves the relevant data 201 and 202 from the respective sources.

In method step 303, the computing unit 3 checks whether a clear identification of the respective recognized vehicle occupants is possible (if the computing unit 3 has not been trained, a generally valid prediction model could also be used in the following). If this is not the case, the method ends in method step 308. If this is possible, however, the computing unit 3 checks in method step 304 whether the heuristic formula or the machine learning model used to predict the future point in time at which the boredom reaction LR of the respective person will occur has been sufficiently modeled or trained.

If this is not the case, the method jumps to method step 307, where the computing unit 3 detects the occurrence of a boredom reaction LR. The emotion characteristics EM are recorded and used for the further development of the respective person-specific prediction model.

However, if the corresponding model is sufficiently accurate, the computing unit 3 predicts in method step 305 the expected point in time at which a boredom reaction LR of the respective person is imminent.

In the subsequent method step 306, the computing unit 3 can issue a recommendation to the vehicle occupants as to which actions can be carried out particularly favorably in order to postpone the estimated point in time at which the boredom reaction LR threatens to occur.

This in 4 The diagram shown has already been used in connection with 1 The diagram shows the time t on the abscissa and the emotion characteristics EM on the ordinate. In the 4 In the embodiment shown, the computing unit 3 is configured such that a boredom reaction LR is detected for the observed person when the two emotion characteristics EM1, EM2 exceed the respective threshold values SW1 and SW2.

The in 4 The situation underlying the diagram corresponds, for example, to a situation in which a bored child asks his parents several times when he will reach his destination and, due to boredom and dissatisfaction, becomes louder and louder when he asks the question again. In addition, the child begins to gesticulate. As soon as the child has reached a certain volume and with a certain intensity If the child gesticulates around in a relaxed manner, the boredom response LR is recognized.

To define the threshold values SW1, SW2, for example, the person driving the vehicle can manually give feedback to the computing unit 3 as to when the person driving the vehicle is annoyed. However, the computing unit 3 can also determine this automatically, for example by evaluating the stress level of the person driving the vehicle determined on the basis of the sensor values.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• DE 102021005092 A1 [0003]
• US 20200310528 A1 [0004]
• US 20120130196 A1 [0006]

Cited non-patent literature

• Jessica Murray (2022), T-minus 10: Statistician writes formula to predict kids' backseat tantrums https://www.theguardian.com/society/2022/aug/22/statistician-writes-formulato-pred ict-kids-backseat-tantru ms [0005]
• Claudia Rowan, PA and Nick Wood (2022) Are we there yet? Maths expert calculates formula to predict back seat tantrums https://www.nottinghampost.com/news/uk-worldnews/yet-maths-expert-calculates-formula-7492694 [0005]
• Katharina Koerth (2022) 70 minutes until the next tantrum, https://www.spiegel.de/reise/autofahren-mit-kindern-was-eltern-gegen-wutanfaelle-tunkoennen-a-5eab1e1d-e0a4-4b1d-ab7d-428777dd626c [0005]
• Grigonyte, G., & Björkenstam, K. N. (2016). Language-independent exploration of repetition and variation in longitudinal child-directed speech: a tool and resources. In Proceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition (pp. 41-50) [0007]

Claims (10)

Method for determining the emotional state of vehicle occupants, wherein the vehicle occupants located in a vehicle interior (1) are detected by means of at least one detection means in the form of a camera (2.1), a microphone (2.2) and/or a haptic sensor (2.3), sensor data generated by the detection means are processed by a computing unit (3), the computing unit (3) determines emotional characteristics (EM) of the individual vehicle occupants by processing the sensor data, determines an emotional state for each vehicle occupant that is dependent on the detected emotional characteristics (EM), and recognizes a boredom reaction (LR) carried out by a vehicle occupant if at least one emotional characteristic (EM) of the vehicle occupant exceeds a person-specific and emotional characteristic-specific threshold value (SW1, SW2), characterized in that the computing unit (3) predicts a future point in time for at least one vehicle occupant at which the respective vehicle occupant is likely to carry out the next boredom reaction (LR). Procedure according to Claim 1 , characterized in that - a person-specific heuristic formula is implemented in the computing unit (e), the computing unit (3) enters at least the detected emotion characteristics (EM) into the heuristic formula, whereupon a heuristic result is obtained, the computing unit (3) compares the heuristic result with a stored result threshold value and determines the future time of the predicted boredom reaction (LR) depending on the comparison of the heuristic result and the result threshold value; or - the computing unit (3) executes a person-specific machine learning model, reads at least the recorded emotion characteristics (EM) into the machine learning model as input data and the machine learning model determines the future time of the predicted boredom reaction (LR) therefrom, wherein the computing unit (3) for training the machine learning model passes the determined state of mind to the machine learning model in addition to the recorded emotion characteristics (EM) as training data, so that the machine learning model compares the self-determined future time of the predicted boredom reaction (LR) with the time at which a boredom reaction (LR) is actually detected. Procedure according to Claim 2 , characterized in that the computing unit (3) determines at least one situation parameter taking into account sensor data generated by detection means monitoring the vehicle interior (1) and/or the vehicle environment and/or information provided to the computing unit (3) by a vehicle subsystem and reads the situation parameter into the heuristic formula or the machine learning model as further input data for predicting the next boredom reaction (LR). Method according to one of the Claims 1 until 3 , characterized in that the computing unit (3) determines a boredom reaction tolerance value for the person driving the vehicle, taking into account the emotional state of the person driving the vehicle, and shifts the predicted time of the next boredom reaction (LR) forwards or backwards in time depending on an expression of the boredom reaction tolerance value. Method according to one of the Claims 1 until 4 , characterized in that the vehicle occupants are identified, a group composition of the vehicle occupants is determined and the computing unit (3) takes the group composition into account to predict the next boredom reaction (LR), in particular by adapting a person-specific and emotion feature-specific threshold value (SW1, SW2), adapting the boredom reaction tolerance value of the person driving the vehicle and/or taking the group composition into account as a further input variable for the heuristic formula or the machine learning model. Method according to one of the Claims 1 until 5 , characterized in that the computing unit (3) predicts a time of falling asleep for a vehicle occupant as a function of at least the recorded emotion characteristics (EM) of this vehicle occupant and takes this into account for predicting the next boredom reaction (LR), in particular by adjusting a person-specific and emotion characteristic-specific threshold value (SW1, SW2), adjusting the boredom reaction tolerance value of the person driving the vehicle and/or taking into account the predicted time of falling asleep as a further input variable for the heuristic formula or the machine learning model. Method according to one of the Claims 1 until 6 , characterized in that a distraction database (4) is kept in the computing unit (3), wherein the distraction database (4) comprises information which describes the influence of various actions on the individual emotional characteristics (EM) of persons and the computing unit (3) after determining a predicted boredom reaction (LR) and before reaching the predicted time at which this boredom reaction (LR) is likely to be carried out, outputs at least one action included in the distraction database (4) as a suggestion to be carried out to the person driving the vehicle, wherein the computing unit (3) recalculates the predicted time, particularly taking into account the influence of the action on the predicted time at which the next boredom reaction (LR) is likely to be carried out. Method according to one of the Claims 2 until 7 , characterized in that the heuristic formulas, trained machine learning models and/or data used to train the machine learning models from a large number of fleet vehicles and individually adapted to a large number of vehicle occupants are transmitted to a central computing device (5), and the central computing device (5) derives a generally valid heuristic standard formula and/or a generally valid standard machine learning model from the totality of the information received. Driver assistance system, comprising at least one detection means, a computing unit (3) for evaluating sensor data generated by the detection means and an output means controllable by the computing unit, characterized by a device for carrying out a method according to one of the Claims 1 until 8th . Vehicle (6), characterized by a driver assistance system according to Claim 9 .

Images