EP4251455A1

EP4251455A1 - An adaptive warning system and method thereof

Info

Publication number: EP4251455A1
Application number: EP21794333.1A
Authority: EP
Inventors: Christian Andreas Schenkewitz
Original assignee: Continental Automotive Technologies GmbH
Current assignee: Aumovio Germany GmbH
Priority date: 2020-11-27
Filing date: 2021-10-13
Publication date: 2023-10-04
Also published as: GB202018674D0; GB2601343B; GB2601343A; WO2022111907A1

Abstract

A warning system and method for use in a motor vehicle is disclosed. The warning system includes a warning unit for executing a first warning and a monitoring system for observing a number of occupants in a passenger cabin, to determine if there are other passengers in the passenger cabin. In response to feedback information received from the monitoring system, the warning unit is operable to execute at least a second warning, depending on the operator's reaction and whether there are no other passengers in the passenger cabin, or there are other passengers in the passenger cabin.

Description

An Adaptive Warning System and Method Thereof

Technical Field This disclosure relates to a warning system, in particular an adaptive warning system for use in a motor vehicle.

Background

Traditional warning systems in motor vehicles uses visual warnings such as tell-tale icons to notify an operator of potential hazard.As an operator's focus is usually on the road, the use of acoustic warning is introduced. However, existing acoustic warning systems are usually designed based upon generic safety mechanisms.A very common example is to use beeping sound from a buzzer as a form of warning. In another example, pre-recorded audio messages are broadcasted as a warning notice in the event to potential hazard. There are no specific re- quirements or standards for using acoustic warnings to notify operators of potential hazards.Therefore,the type of audio used is solely dependent on developer's design specification.

Although the conventional techniques explained above may serve its purpose to warn an operator, each operator may respond differently to different acousticwarningsand reaction timesmay be affected. In addition, for less experienced operators, loud acoustic warnings may cause unnecessary stress or pressure instead. In a scenario where there are multiple occupants onboard, loud acoustic warningsmay be annoying and disturb other passengers onboard the motor vehicle.

There is therefore a need to provide a warning system that overcomes, or at least ameliorates, the problem(s) described above. Furthermore, the other desirable features and charac teristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

Summary

The objective of the disclosure is solved by a warning system for use in a motor vehicle comprising:

- a warning unit for o executing a first warning;

- a monitoring system for observing o a number of occupants in a passenger cabin; wherein

- the warning unit is operable o to execute at least a second warning in response to

- feedback information received from the monitoring system, wherein the feedback information comprises:

• an operator's reaction to

• the first warning executed,

• a number of passengers in the passenger cabin, to determine if there are other passengers in the passenger cabin or a combination thereof.

The advantage of the above described aspect of this disclosure yields warning system which provides feedback information from a monitoring system for observing a number of occupants in a passenger. More in particular, this aspect of the disclosure presents an adaptive warning system that execute warning and subsequent warnings based upon feedback information from a monitoring system, for observing a number of occupants in a passenger cabin and/or an operator's reaction to a first warning executed. Specifically, the feedback information defines a condition of the warning system to trigger a next course of action by the warning unit. More specifically, this condition defines a selection of the at least second warning to be executed.

Preferred is a warning system as described above or as described above as being preferred, in which:

- the monitoring system is operable to o perform emotion recognition based upon

^■ observation of one or more occupants in the passenger cabin.

The advantage of the above aspect of this disclosure is to identify the occupants in the vehicle as the operator or the occupant, and classify the operator's reaction according to emotion recognition algorithm performed, based upon frames of images captured by the monitoring system.

Preferred is a warning system as described above or as described above as being preferred, in which the operator's reaction comprises:

- idle,

- active, or

- startled.

The advantage of the above aspect of this disclosure yields an adaptive warning system by providing feedback information as to a reaction of the operator to the first warning executed. Preferred is a warning system as described above or as described above as being preferred, in which:

- the warning unit is operable to o execute the second warning comprising a volume level different from a volume level of the first warning, in response to the feedback information received indicating

- the number of passengers in o the passenger cabin is zero, and

- the operator's reaction to the first warning ex ecuted is idle.

The advantage of the above aspect of this disclosure yields a warning system which executes at least a second warning in the format of an audio warning at a volume level different from the volume level of the first warning according to feedback in formation received,in particularwhen the operator did not react to the first warning executed and there is no passenger in the passenger cabin. Therefore, an adaptive warning system is achieved.

Preferred is a warning system as described above or as described above as being preferred, in which: - the warning unit is operable to o execute the second warning comprising a volume level different from a volume level of the first warning, in response to the feedback information received indicating - the number of passengers in o the passenger cabin is zero, and

- the operator's reaction to the first warning ex ecuted is startled. The advantage of the above aspect of this disclosure yields a warning system which executes at least a second warning in the format of an audio warning at a volume level different from the volume level of the first warning according to feedback in formation received, in particular when the operator is startled, in response to the first warning executed and there is no passenger in the passenger cabin.Therefore, an adaptive warning system is achieved.

Preferred is a warning system as described above or as described above as being preferred, in which the volume level of the second warning different from the volume level of the first warning is defined by:

- an increase in the volume of the second warning o compared to the first warning, or

- a decrease in the volume of the second warning o compared to the first warning.

The advantage of the above aspect of this disclosure defines the at least second warning executed at a volume level different from the first warning may be an increase or decrease in the volume of the second warning compared to the first warning.

- the warning unit is operable to o execute the second warning in a format dif ferent from the first warning.

The advantage of the above aspect of this disclosure allows the warning system to select and execute the at least second warning in a format different from the first warning, to ensure that the operator is informed of the warning and reacts to the warning executed. For example, the first warning may be an acoustic warning and at least a second warning may be in another format such as visual warning or tactile warning. In this way, the warning system adapts to the needs of the operator by executing warning in different format(s), to ensure that the operator is aware of any potential hazard.

Preferred is a warning system as described above or as described above as being preferred, in which the first warning comprises:

- an acoustic warning,

- a visual warning, or any combination thereof. Preferred is a warning system as described above or as described above as being preferred, in which the second warning comprises:

- an acoustic warning,

- a visual warning

- a tactile warning, or any combination thereof.

The advantage of the above aspect of this disclosure yields a warning system which executes an acoustic warning and/or a visual warning as a first warning and select a different format of warning for execution as a second warning.This ensures that even if the operator did not react to the first warning executed, at least a second warning is executed in a different format to inform the operator of potential hazard. Preferred is a warning system as described above or as described above as being preferred, further comprising:

- one or more warning source for issuing the first warning and/or the second warning. Preferred is a warning system as described above or as described above as being preferred, in which the warning source comprises:

- an audio generator for generating one or more acoustic warning; - an information display for generating one or more visual warning;

- a tactile module for generating one or more tactile warning; or any combination thereof.

The advantage of the above aspect of this disclosure defines the types of warning source that are suitable for use in the warning system disclosed herein. In particular, the warning sources provides a selection of different types of warning sources. Therefore, the warning unit ensure that the operator is informed of the warning through different mode or format of warning and reacts to the warning, through monitoring a passenger cabin of the motor vehicle and providing feedback information to the warning unit.

The objective of the disclosure is solved by a warning method for use in a motor vehicle comprising:

- executing a first warning by way of a warning unit;

- observing a passenger cabin, by way of a monitoring system; and

- executing a second warning in response to feedback information received by the monitoring system, for observing the passenger cabin. The advantage of the above described aspect of this disclosure yields warning method which provides feedback information from a monitoring system for observing a number of occupants in a passenger. Brief Description of Drawings Other objects and aspect of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings in which:

Fig. 1 shows a system block diagram of a warning system in accordance to a preferred embodiment.

Fig.2 shows a schematic diagram of a warning system in accordance to a preferred embodiment. Fig. 3 shows a flowchart of a motor vehicle warning method in accordance to a preferred embodiment.

In various embodiments described by reference to the above figures,like reference signs refer to like components in several perspective views and/or configuration.

Detailed Description of Embodiment The following detailed description ismerely exemplary in nature and is not intended to limit the invention or the application and uses of the disclosure. Furthermore, there is no intention to be bound by any theory presented in the preceding background of the disclosure or the following detailed description. It is the intent of this disclosure to present an adaptive warning system that execute warning and subsequent warnings based upon feedback information from a monitoring system, for observing a number of occupants in a passenger cabin and/or an operator's reaction to a first warning executed. Hereinafter,the term "reaction" shall refer to either a physical reaction or an emotional reaction as a result of an event that occurs. In the context used herein, a reaction of an operator may refer to behaviour such as a physical reaction to a warning executed by a warning system. For example the act of switching off an acoustic warning may be considered a physical reaction, while an emotional reaction is expressed through speech, for example vocal expression such as "what was that!?", and more in particular,facial expressionsexpressing shock,fright,anxiety or panic, indicating the operator may be distressed or feeling pressured in response to the warning executed by the warning system. .

The term "format" refers to a manner in which something is shown or arranged. Therefore, when used in the context herein, the "format" of warning may refer to a type of warning which is being executed, for example a visual warning and an acoustic warning are considered different formats.

In the context herein, the term "connect", "connected", "connection" and its grammatical variations thereof are used to define a linkage between two or more components or technical elements. The linkage may be an electrical link, a wireless communication link or bus communication between the different components or technical elements.

The term "a first", "a second" and its grammatical variations thereof are used to differentiate different components or elements described herein. For example, the term "a first warning" may refer to a type of warning and "a second warning" may refer to warning that is executed subsequent to the first warning. The terms may be used interchangeable, with no sig nificance as to a sequence of importance. Referring to Fig.1which showsa system blockdiagram of awarning system for use in a motor vehicle, in accordance to a preferred embodiment, the warning system 112 includes a warning unit 102, a monitoring system 104, and feedback information 106 which is input from the monitoring system 104 to the warning unit 102. The warning unit 102 is operable to execute a warning to inform an operator of a potential hazard. The monitoring system 104 is operable to observe an environment, for example a passenger cabin of a motor vehicle, and to determine a number of occupants in the passenger cabin. More in particular, the monitoring system 104 determines a number of passengers onboard the vehicle, to identify an operator in the passenger cabin and a number of passengers in the passenger cabin. For example, an occupant in the driver's seat shall be identified as the operator of the motor vehicle,and a number ofpassengers seating in the front passenger seat or the rear passenger seat shall be identified aspassengers in the passenger cabin. In this way, the number of passengers in the passenger cabin is identified accordingly. In addition, the monitoring system 104 is also operable to determine an operator's reaction. In an embodiment, information containing the number of occupants in the passenger cabin and/or the operator's reaction is transmitted from the monitoring system to the warning unit 102 as a form of feedback information 106. The warning unit 102 is operable to execute at least a second warning based upon feedback information 106 received. The at least second warning may serve as a reminder warning to the operator.

The warning system 112 may be connected to one or more warning source, for executing warnings in different format. By way of example, the warning system 112 may be connected to a first warning source 108, and a second warning source 110, each of the first warning source 108 and the second warning source 110 operable to execute warnings in different format. Examples of different formats of warnings may include visual warning, acoustic warning, or tactile warning. The warning unit 102 may be a control unit of the warning system 112.Advantageously, the warning unit 102 is operable to execute a first warning and at least a second warning in response to feedback information 106 received. Since the feedback information 106 comprises a number of passengers in the passenger cabin and the operator's reaction, the warning unit 102 is operable to select a second warning source to avoid disturbing other occupants onboard the motor vehicle.

Fig. 2 shows a schematic diagram of a warning system 200 in accordance to a preferred embodiment implemented in a motor vehicle 202. The warning system 200 includes a warning unit 102 for executing a first warning and at least a second warning, a monitoring system 104 for observing a number of occupants in a passenger cabin of the motor vehicle 202 and one or more warning source 108. Preferably, the monitoring system 104 is operable to observe an entire passenger cabin of the motor vehicle 202. Observation of the passenger cabin may be done by a monitoring device, for capturing images of the passenger cabin. Suitable type of monitoring device may be an imaging sensor, a thermal camera or an infrared camera. In an embodiment, the images captured by themonitoring devicemaybeprocessedby a controller or processor of the monitoring system 104 to determine a number of occupants 210 sitting in the passenger cabin when a first warning is executed. In addition, the monitoring system 104 is also operable to determine an operator's reaction. The feedback information determined by the monitoring system 104 is transmitted to the warning unit 102. Alternatively, the images captured by the monitoring system 104 is transmitted to the warning unit 102 to determine the feedback information. Ad vantageously, feedback information from the observation made by the monitoring system 104 is transmitted to the warning unit 104, thus an adaptive warning system is achieved. Feedback Information

The monitoring system 104 determines feedback information to be transmitted to the warning unit 102,and the feedback information comprises two factors.A first factor is to determine the number of occupants or passengers in the passenger cabin of the motor vehicle 202 after a first warning is executed by the warning unit 102, where the number of passengers in the passenger cabin may be defined as zero (i.e. no other occupants onboard apart from the operator) or more than one (i.e. there is at least one passenger 210 present).

A second factor is the operator's reaction to the first warning executed. The classification of operator's reactions may be executed by software or algorithm operable to perform emotion recognition to identify the different types of emotional re actions, such as vocal expressions or facial expressions, based upon consecutive frames of images captured by the monitoring device of the monitoring system 104, through observing facial features of the operator 208 identified by the monitoring system 104. The operator's reaction may be classified into three (3) different types of reactions, namely idle, active or startled.

Classifying Operator's Reaction In a first exemplary scenario,after thewarning unit 102 executes a first warning, the monitoring device of the monitoring system 104 captures images of the passenger cabin and process the images to determine amongst others, the operator's reaction. If the monitoring system 104 determines,based upon a physicalbehaviour of the operator 208 that the operator 208 did not respond to the first warning, the operator's reaction is classified as idle.By way of an example,if thewarning unit 102 executesa firstwarning in the format of a visual warning displayed on one of the information displays in the motor vehicle 202,and themonitoring system 104 determines that the operator did not look at the information display after the first warning is executed, the operator's reaction is classified as idle. Consequently, the warning unit 102 executes at least a second warning, to remind the operator 208.

In contrast to the first exemplary scenario, if, upon the execution of a visual warning as a first warning, the monitoring system 104 determines that the operator 208 looked at the information display and responded to the visual warning (for e.g. switches off hazard icons or vehicle changes lane), then the monitoring system 104 determines that the operator's reaction to the first warning is active or the operator 208 is engaged in responding to the first warning executed and classifies the operator's reaction as active accordingly.

In another scenario,the first warning executedmaybe an acoustic warning and the physical behaviour exhibited by the operator 208 indicates a sudden jump in surprise or images of the operator 208 observed by the monitoring system indicates facial features of the operator 208 to be frightened, the monitoring system 104 classifies the operator's reaction as startled.

One of the advantageous of this disclosure over known or existing warning systems is that the feedback information determined by the monitoring system considers two factors by monitoring the passenger cabin. Unlike most warning systems which focuses on monitoring the operator's reaction to solve the problem of operator's safety, by considering whether there are other occupants in the passenger cabin enables the warning unit 102 to select other format of warning the operator, such that other occupants in the passenger cabin are not disturbed. Different Formats of Warning

As shown in Fig. 2, more than one warning source may be used. Examples of suitable warning sources 108 may include an in formation display for displaying visual warning, an audio generator 110 for generating acoustic warning, a tactile module 108"'for generating a tactile warning or a combination of the aforesaid format of warning. Example of suitable types of information display may be an instrument cluster or a head-up display (HUD). The tactile module 204 may be a module operable to generate tactile warning through actuators 206 embedded in a vehicle seat 212 for providing haptic sensation or haptic feedback as a form of tactile warning, or actuators (not shown) embedded in a handle of a steering wheel 214 for providing haptic sensation or haptic feedback as a form of tactile warning. The first warning and second warning may be executed in different format, based upon the type of warning source implemented in the warning system.Advantageously, different format of warning may be selected for execution as the at least second warning,to serve asa reminderwarning to the operator,depending on the operator's reaction as well as situation within the passenger cabin. For example, if there are one or more occupants 210 in the motor vehicle 202 with the operator 208, instead of continuously executing warnings in acoustic format, the warning unit 102 is operable to change a format of the warning, for example execute a visualwarning or a tactile warning instead as a second warning, so as to avoid disturbing the other occupants in the passenger cabin. If the monitoring system 104 determines the operator's reaction is idle or startled, the warning unit 102 may select an alternative format ofwarning until a desired operator's reaction is achieved.

Further advantages of the warning system as disclosed shall be appreciated based upon the exemplary embodiments as set forth below, which describes the different combination of feedback information which may apply:

Acoustic Warning (Scenario A)

In an exemplary embodiment, the warning unit 102 executes a first warning in the form of an acoustic warning generated by the audio generator 110. The monitoring system 104 observes the passenger cabin and determines that the operator's reaction is idle and the number of occupants in the passenger cabin is zero. In response to this feedback information transmitted to the warning unit 102, the warning unit 102 executesa secondwarning.The secondwarning may be an acoustic warning with a volume level higher than the first acoustic warning, i.e. the second acoustic warning is generated at an increased volume level compared to the first acoustic warning.

Acoustic Warning (Scenario C) In an alternative embodiment, if the monitoring system 104 observes the passenger cabin and determines that the operator's reaction is startled and the number ofpassengers in thepassenger cabin is zero (i.e.the operator 208 is alone in the motor vehicle 202), the warning unit 102 may execute a second warning in a same format as the first warning. Specifically, the warning unit 102 is operable to execute a second acoustic warning, however the second acoustic warning has a volume level lower than the first acoustic warning, i.e. the second acoustic warning is generated at a decreased volume level comparted to the first acoustic warning

Acoustic Warning (Scenario B)

In another exemplary embodiment, the warning unit 102 executes a first warning in the form of an acoustic warning generated by the audio generator 110. The monitoring system 104 observes the passenger cabin and determines that the operator's reaction is idle and the number of passengers in the passenger cabin is one or more. In response to this feedback information transmitted to the warning unit 102, the warning unit 102 is operable to execute a second warning in a format different from the first warning. Instead of executing a secondwarning in a format same asthe first warning, the warning unit 102 is operable to execute a second warning either as a visual warning on an information display 108 or a tactile warning generated by a tactile module 204. In this way, the advantage of an adaptive warning system that execute warning and subsequent warnings based upon feedback information from a monitoring system, for observing a number of occupants in a passenger cabin and/or an operator's reaction to a first warning executed is achieved. Acoustic Warning (Scenario D)

In another alternative embodiment, the warning unit 102 executes a first warning in the form of an acoustic warning generated by the audio generator 110. The feedback information 106 from the monitoring system 104 indicates that the operator's reaction is startled and the number of passengers in the passenger's cabin is one or more. In response to the feedback information 106, the warning unit 102 is operable to execute a second warning in a format different from the first warning as described above. Advantageously, the other passengers in the passenger cabin will not be disturbed due to repeated execution of warnings. Neither will the driver feels stressed or pressured by the repeated execution of warnings. Acoustic Warning (Scenario E / F)

In yet another exemplary embodiment, in response to the first warning executed, feedback information 106 from the monitoring system 104 indicates that the operator's reaction is active, the warning unit 102 cease executing a second warning whether the number of passengers in the passenger cabin is indicated as zero or one or more.

As can be seen from the exemplary embodiments described above, by executing a second warning in a format different from the first warning advantageously provides an adaptive warning system, to address the needs of the operator according to the operator's reaction, and to address the needs of other occupants in the passenger cabin. Although the above exemplary embodiments provided only describes acoustic warning as a first warning executed, it shall be understood embodiments covering executing a second warning in a format different from the first warning applies to different combinations, without departing from the scope and spirit of the present disclosure. Turning now to Fig. 3, a flowchart of a motor vehicle warning method 300 in accordance to a preferred embodiment is shown. In step 302, a first warning is executed by way of a warning unit 102 of a warning system 112, 200. In a next step 304, observation of a passenger cabin of the motor vehicle is made by a monitoring system 104.As described above, the monitoring system 104 may be operable to process images captured by a monitoring device, to determine a number of occupants in the passenger cabin at 306, in particular whether there are zero passengers in the passenger cabin or at least one passenger ispresent in the passenger cabin. In the next step 308,the operator's reaction to the first warning executed at step 302 is determined.In response to the information determined at step 306 and step 308, the warning system executes a second warning, of which the format of second warning and/or volume level of the second warning compared to the first is executed based upon feedback information determined in step 306 and step 308.

As can be seen from the above description, a main advantage of the feedback information from the monitoring system is to enable the warning unit 102 to select an appropriate format of warning as the second warning. For example, if the first warning is an acoustic warning and the feedback information indicates the operator's reaction is startled, then the warning unit 102 may select an alternative format of warning, e.g.tactile warning to be executed as a second warning. Further, the first warning if an acoustic warning is selected as a choice of second warning, the warning unit 102 is operable to change a volume level of the second warning according to the feedback information received. More advantageously, if the operator's reaction is classified as startled, the warning unit 102 may select an alternative format, e.g. tactile warning to be executed as a second warning.

List of Reference Signs

100 - system block diagram 102 - warning unit 104 - monitoring system 106 - feedback information

108 - warning source (buzzer / information display) 110 - audio generator 112 - warning system

200 - schematic diagram 102 - warning unit

104 - monitoring system

106 - feedback information

108 - warning source (visual - information display) 110 - warning source (audio)

202 motor vehicle

204 tactile module

206 actuators (for tactile effect) 212 - vehicle seat 214 - steering wheel

208 operator

210 occupant / passenger 300 method flowchart

302 executing a first warning 304 observing a passenger cabin of a motor vehicle 306 number of passengers in the cabin 308 operator's reaction 310 execute a second warning

Claims

Patent claims

1. A warning system for use in a motor vehicle comprising:

- a warning unit for o executing a first warning;

- feedback information received o from the monitoring system, wherein the feedback information comprises:

- an operator's reaction to o the first warning executed,

- a number of passengers in the passenger cabin, to determine if there are other passengers in the passenger cabin or a combination thereof.

2. The warning system of claim 1, wherein

- the monitoring system is operable to o perform emotion recognition based upon

^■ observation of one or more occupants in the passenger cabin.

3. The warning system of clail 3, wherein the operator's reaction comprises:

- idle,

- active, or startled.

4. The warning system of any one of claims 1- 3, wherein

- the number of passengers in o the passenger cabin is zero, and

- the operator's reaction to the first warning ex ecuted is idle.

5. The warning system of any one of claims 1- 4, wherein

- the number of passengers in o the passenger cabin is zero, and

- the operator's reaction to the first warning ex ecuted is startled.

6. The warning system of claims 4 or 5, wherein the volume level of the second warning different from the volume level of the first warning is defined by:

7. The warning system of any one of claims 1- 6, wherein

8. The warning system of any one of claims 1- 7, wherein

- the warning unit is operable to o execute the second warning in a same format as the first warning.

9. The warning system according to any one of the preceding claims, wherein the first warning comprises:

- an acoustic warning,

- a visual warning, or any combination thereof.

10.The warning system according to any one of the preceding claims, wherein the second warning comprises:

- an acoustic warning,

- a visual warning

- a tactile warning, or any combination thereof.

11.The warning system according to any one of the preceding claims, further comprising:

- one or more warning source for issuing the first warning and/or the second warning.

12.The warning system of claim 11, wherein the warning source comprises:

- an audio generator for generating one or more acoustic warning;

- an information display for generating one or more visual warning; - a tactile module for generating one or more tactile warning; or any combination thereof.

13.A motor vehicle warning method comprising:

- executing a first warning by way of a warning unit;

- observing a passenger cabin, by way of a monitoring system; and

- executing a second warning in response to feedback information received by the monitoring system, for observing the passenger cabin.