WO2022141041A1

WO2022141041A1 - Driver monitoring method for monitoring the activity of a vehicle driver and vehicle for executing said method

Info

Publication number: WO2022141041A1
Application number: PCT/CN2020/140782
Authority: WO
Inventors: Anna Paetzold; Bilal Khan
Original assignee: Zhejiang Geely Holding Group Co., Ltd.; Ningbo Geely Automobile Research & Development Co., Ltd.
Priority date: 2020-12-29
Filing date: 2020-12-29
Publication date: 2022-07-07
Also published as: EP4271591A1; TW202241742A; CN116829415A; TWI818395B

Abstract

The present disclosure pertains to driver monitoring method for monitoring the activity of a vehicle driver, comprising the steps of estimating the fatigue level of the driver and/or a time spent waiting by the driver, establishing the driving context of the vehicle and/or the vehicle status, outputting one of a plurality of recommended actions for altering the activity of the driver based on the fatigue level, the time spent waiting, a success rate of the recommended action, the driving context and/or the vehicle status, evaluating the response of the driver to the recommended action and evaluating and updating the success rate of the recommended action. The disclosure also pertains to a vehicle comprising a controller, input means and output means for executing a corresponding method.

Description

DRIVER MONITORING METHOD FOR MONITORING THE ACTIVITY OF A VEHICLE DRIVER AND VEHICLE FOR EXECUTING SAID METHOD

Technical Field

The present invention pertains to a driver monitoring method for monitoring the activity of a vehicle driver comprising the steps of estimating the fatigue level of the driver and/or a time spent waiting by the driver, establishing the driving context of the vehicle and/or the vehicle status, outputting one of a plurality of recommended actions for altering the activity of the driver based on the fatigue level, the time spent waiting, a success rate of the recommended action, the driving context and/or the vehicle status, evaluating the response of the driver to the recommended action and evaluating and updating the success rate of the recommended action. The invention also pertains to a vehicle comprising a controller, input means and output means for executing a corresponding method.

Background Art

Current methods and corresponding systems for monitoring drivers are designed to detect the activity of a vehicle driver such that dangerous or undesired activities can be detected, reacted upon and the safety and wellbeing of the driver may be enhanced.

For example, fatigue or drowsiness of a driver or a situation, in which the driver is forced to remain idle and wait for some event may be detected and reacted upon. Evidently, the term activity is to be understood in a broad sense and may refer to behaviour indicative of a tired driver, such as erratic control movements of the vehicle or the prolonged closing of the driver’s eyes. In addition, the monitored behaviour of the driver may be indicative of a waiting situation in which the vehicle may be at rest and the driver may not be performing any significant control activities related to the vehicle movement.

Once a corresponding activity has been detected, predefined reactions may be generated in order to reduce accident risks associated with drowsy drivers or to increase the wellbeing of a waiting driver. However, the known methods are insensitive as to whether the predefined reactions actually suit the respective driver and achieve the desired results of reducing the amount of driving done by a drowsy driver or increasing the driver’s wellbeing in idle situations. This represents a problem as either the known methods will not be effective at reducing the amount of driving done by a drowsy driver and improving the driver’s wellbeing or, even worse, the driver may actually find that the predefined reactions are a nuisance to her and she may stop the execution of the method altogether as a result.

Technical Problem

The aim of the present invention is to provide an improved method and an improved vehicle, which overcome the above–outlined problems.

Technical Solution

This aim is achieved by a driver monitoring method for monitoring the activity of a vehicle driver according to claim 1 and a vehicle according to claim 10, comprising a controller, input means and output means for executing a corresponding method. Preferable embodiments of the invention are subject to the dependent claims.

According to claim 1, a driver monitoring method for monitoring the activity of a vehicle driver is provided, said method comprising the steps of

-estimating the fatigue level of the driver and/or a time spent waiting by the driver;

-establishing the driving context of the vehicle and/or the vehicle status;

-outputting one of a plurality of recommended actions for altering the activity of the driver based on the fatigue level, the time spent waiting, a success rate of the recommended action, the driving context and/or the vehicle status;

-evaluating the response of the driver to the recommended action; and

-evaluating and updating the success rate of the recommended action.

The term monitoring is to be understood in a broad sense, comprising the surveillance of the driver’s behaviour as well as suggestion for altering or enhancing the driver‘s behaviour.

The recommended actions may include changes of the set-up of the vehicle, such as its interior lighting or its seat positions. They may include the play back of certain music or the initiation of some games to be played by the driver. The recommended actions may include suggestions for activities the driver can perform, such a going for a walk, taking a nap or performing some sort of meditation. The recommended actions may be output or executed directly or the driver may be prompted to confirm a suggested action prior to its actual output or execution. Generally speaking, the output of a recommended action may comprise outputting optical and/or visual information and/or changes to the physical configuration of the vehicle.

The success rate of a given recommended action may be a function of the ratio of the number of times a driver has accepted a recommended action over the total number of times the action has been suggested to a driver.

According to the invention, a particular driver can be identified, such that the method may be adapted to each individual driver of a given vehicle. The driver identification can occur based on some face recognition steps and corresponding hardware such as a driver monitoring camera.

In case a driver uses multiple vehicles, the vehicles may be connected to each other such that the method may be executed on any vehicle driven by the same driver. Thus, the method is not limited to a single vehicle.

The presently described method may be carried out continuously while the driver is driving the vehicle. Alternatively, the execution of the method may be initiated by some predefined signal such as the occurrence of a potential driving error such as abrupt steering and/or breaking or the crossing of some road markings.

The method may be carried out in parallel, i.e. simultaneously and with respect to different suggested actions, such that more than one recommended action can be output to the driver.

According to the invention, risky driving is reduced by effectively counteracting fatigue, increasing the effectiveness of methods countering driver-drowsiness and increasing the user experience by adapting the measures to the individual driver. Furthermore, the general wellbeing of the driver may be enhanced, as the driver may be supported in enhancing potentially wearing waiting periods.

In contrast to current driver monitoring systems, this solution integrates escalation states and offers counteracting measures (e.g. modes) depending on firstly, the kind of current activity or state of the driver and her level of fatigue and, secondly, the previously established success rate of a proposed solution or recommended action. Based on these, the algorithm can learn and adapt the most effective method to counteract the individual’s fatigue and/or to improve her general wellbeing. In order to achieve this, the method may comprise a driver recognition step in which the identity of the driver is established. Hence, the method may be adapted to various drivers and their individual traits and preferences.

The fatigue level may be some score which correlates with the actual fatigue of the driver and which may be a function of fatigue indicating behaviour of the driver such as yawning, eye movement and/or erroneous driving behaviour. This behaviour may be detected by the presently described method and may be used for estimating or calculating the fatigue level.

The present invention provides a context-based recommendation algorithm in which a vehicle may perform some of the presently described remedies if a driver fatigue situation or a driver waiting situation or pattern is detected.

In a preferred embodiment of the invention, the fatigue level of the driver and/or the time spent waiting by the driver are estimated by means of a driver monitoring camera and/or by evaluating the lateral and longitudinal vehicle control performed by the driver.

In another preferred embodiment of the invention, establishing the driving context of the vehicle and/or the vehicle status comprises establishing the current weekday, daytime, driving time, weather, road type, driving history, advanced driver assistance system responses, a battery level and/or a position of the vehicle.

In another preferred embodiment of the invention, the plurality of recommended actions comprises suggesting a refresh mode and/or a rest mode. Further modes may be applied additionally or alternatively. The suggestion of a mode or of actions associated with a mode may comprise the changing of some vehicle settings, the output of some information, the prompting of the driver to confirm and initiate the output of some information and/or other method steps capable of improving the driving safety and increasing the wellbeing of the driver. The modes may be understood as a collection of various recommendable actions, wherein each mode may be associated with a particular driver fatigue level and/or driver activity.

In another preferred embodiment of the invention, the refresh mode is output at a lower fatigue level of the driver than the rest mode. If the fatigue level exceeds a threshold or fatigue is deemed probable due to the detection of e.g. a long driving time on a rural road, a corresponding counteracting measure i.e. a recommended action is proposed.

In another preferred embodiment of the invention, the rest mode comprises

-suggesting a resting area, in particular a resting area with a battery charging station;

-suggesting setting a timer for outputting an alarm, outputting an alarm based on the charging status of the battery, playing of music and/or providing a massage to signal the end of the rest mode to the driver;

-suggesting executing noise cancellation, suppression of irrelevant messages, setting of ambient light and/or temperature;

In another preferred embodiment of the invention, the refresh mode comprises

-suggesting a recreational activity.

The refresh mode may be output as a recommended action especially when the driver monitoring method has estimated that the driver has spent more than a predetermined amount of time waiting and/or not significantly controlling the vehicle. This may be the case when the driver is stuck in a traffic jam or is waiting for other passengers to come to the vehicle. Alternatively or additionally, the refresh mode may be output when the driver monitoring method has estimated a low fatigue level.

The recommended action output in the refresh mode may comprise outputting meditation instructions such that the driver may utilize the waiting time for increasing mental wellbeing. In this case, the vehicle may be set to provide a relaxing environment, for example by playing suitable music or sounds of nature, providing suitable ambient lighting and/or a massage function. Similarly, guidance to do light Yoga (e.g. Pranayama) may be output in the refresh mode.

In principle, any kind of information, which might be deemed helpful by the driver, may be output in the refresh mode. This may comprise playing e-books, enabling a game such as chess or a cards game on the vehicles computer system or suggesting a walk in a nearby area. The drivermay accept a suitable recommended action and the method ensures that the success rate of the chose recommended action is updated accordingly.

In another preferred embodiment of the invention, evaluating the response of the driver to the recommended action comprises establishing which recommended actions have been accepted by the driver and to what degree.

Accordingly, the success rate of an accepted recommended action may be increased every time the driver accepts it. Thus, the response of the driver to the recommended action is evaluated. In futures situations, the method may suggest some recommended actions more frequently, if their success rate is higher than the success rates of other actions. A recommended action which has been selected or accepted frequently by a driver and which improves the fatigue level of a driver can be stored and/or rated accordingly and can be suggested as a favorite, e.g. more frequently in future situations, especially in similar contexts and/or when similar fatigue levels are detected.

The method may monitor for how long a recommended action has been accepted or performed by the driver in order to establish to what degree the recommended action has been accepted..

In another preferred embodiment of the invention, evaluating the response of the driver to the recommended action comprises estimating the fatigue level of the driver after the recommended action has been completed.

The invention is also directed at a vehicle comprising a controller, input means and output means for executing a method according to any of claims 1 to 9. The vehicle may be designed to enable the execution of any of the presently outlined method steps, i.e. it may comprise any hardware necessary for executing the presently described method steps.

Advantageous Effects

The present invention therefore offers the benefits of a context-aware and intelligent method for monitoring a driver, the benefits including increased driving safety, improved effectiveness of counteracting methods and increased driver satisfaction because of a highly individual and adaptable monitoring method. Furthermore, the invention strengthens the perception of an intelligent vehicle.

Description of Drawings

Further features, details and advantages of the invention are disclosed in the claim set, the following description of the figure and the figure themselves.The figure shows:

Fig. 1: a flow chart of some of the steps of the method according to the invention.

Best Mode

In one method step shown in figure 1, the context of the vehicle may be checked. This may correspond to establishing the driving context of the vehicle and/or the vehicle status. In this step, the road type the vehicle is situated on may be retrieved from some accessible map data and/or from vehicle sensors such as a camera capturing the surroundings of the vehicle. Further data such as day time, traffic density, driver behaviour and/or regularity of certain driver behaviour may be obtained from vehicle sensors, from vehicle memory devices such as hard drives and/or from other sources connected to the vehicle and accessible by the present method, such as the internet and/or suitable cloud storage sources.

In another method step, the fatigue level of the driver and/or the time spent waiting by the driver may be estimated from data retrieved from a driver monitoring camera and/or from sensors surveying the vehicle’s driving characteristics.

If both, the driving context of the vehicle and the current activity of the driver have been found to indicate some undesired situation, such as a drowsy driver driving the car in an unsafe manner or a driver waiting inside the car, while the car is stationary, some recommended actions of a plurality of recommended actions may be output according to the method. The recommended actions are intended for altering the activity of the driver based on the fatigue level, the time spent waiting, a success rate of the recommended action, the driving context and/or the vehicle status.

Initially, the success rate of untried recommended actions may be set at some predetermined value, which, at first, may be identical for all recommended actions. Once a response of the driver to a recommended action has been evaluated, the success rate of the recommended action may be evaluated and updated. Hence, after performing the method for a number of times and for various different recommended actions the success rates of the recommended actions can be updated automatically to reflect on their actual effectiveness with respect to a certain driver.

As an example, the method may detect, that a drowsy driver is driving the car. This may be derived from e.g. the driver closing her eyes more regularly and for longer periods of times than average and/or from the driver performing erroneous control of the car. If the method establishes that the vehicle is driving, a rest may be suggested. The drowsiness of a driver may be detected based on direct driver behaviour, such as the driver’s eye movement. Alternatively, it may be detected from indirect data, such as when drowsiness-inducing driving context is expected, for example from the regular driving behaviour or driving trajectories detected and stored by the method or when drowsiness-inducing context is actually detected, for example if the vehicle’s present route is a rural road with low traffic density.

Alternatively, the method may detect that the car is stationary and the driver is not performing any substantial control of the car, indicating that the driver may be waiting for some event, such as the arrival of another passenger of the vehicle. In this case, other actions such as issuing Yoga instruction may be carried out.

Once the recommended action has been completed or rejected by the driver, the activity or the state of the driver may be checked again, wherein the response of the driver to the recommended action is evaluated for its effectivity. If it is found that the given recommended action is effective, e.g. the driver accepted the recommended action and her drowsiness after performing the recommended action is reduced, then the success rate of this recommended action may be increased from a previous value to a new, higher value and vice versa. Hence, the success of a proposal can be checked effectively. If it is not found that a drowsy or waiting driver is present in the vehicle, then no proposal may be output according to the method.

In effect, the algorithms comprised by the present invention are capable of retrieving data concerning the state of the driver and the current context of the vehicle. Depending on the retrieved data, the algorithms recommend suitable means or actions for improving the driver’s current state and evaluate the success of the recommended actions.

In case a similar situation i.e. a similar driver state and context of the vehicle is detected on another occasion, a previously successful action may be recommended again. The similarity of situations may refer to similar driving routes of the vehicle and/or some regular phenomena or repetitive occurrences such as repeating similar waiting times for additional passengers to come to the vehicle at certain pick-up locations. The algorithms may be designed to iteratively learn the individual preferences of a driver and the efficiencies of given suggested actions.

The present invention addresses two considerable issue with regard to vehicles driven by drivers.

On the one hand, studies show that drivers spend considerable amounts of time (in some cases ca. 75 minutes per week on average) waiting behind the steering wheels for example when picking up children from school, bringing children to clubs, waiting for other people during activities such as shopping or a doctor’s appointment. Especially in times of epidemics, it might be advisable to wait in the vehicles for e.g. a doctor’s appointment, taking disease tests or the like. The waiting time may be used by the driver for things like reading, engaging in some form of internet activity such as consuming YouTube or Facebook, listening to music, planning meals etc. However, few drivers utilize this time to actually improve their wellbeing.

On the other hand, a considerable amount of vehicle accidents is caused by drowsy drivers, which continue to control the vehicle well beyond a safe state to operate any machinery.

The present invention provides a method for detecting both, drowsiness and waiting periods of a driver such that the corresponding problems can be improved and the driver’s situation can be improved.

In a practical example, the method may detect fatigue in a driver and may suggest switching to the refresh mode, including a number of possible suggested actions associated with the refresh mode. The driver may accept the suggestion and the vehicle may take appropriate actions. After a given amount of time, e.g. after 15 minutes, the driver’s fatigue level may be estimated again and an elevated fatigue level may still be detected. In this case, it may be found that the suggested action has not been successful. As a result, the method may output suggested actions associated with the rest mode. The driver may be prompted to confirm the suggested choice. In case the driver accepts the suggestion, the method may indicate the next possible resting areas on a navigation device, such that the driver may select a suitable resting area. When suggesting a resting area, the method may consider the charge state of a vehicle battery. If the charge state is below some given value (e.g. 60%), the method may proposes an area where the vehicle can be charged. Preferably, a resting area may be suggested which can be reached within a given time (e.g.15 minutes). Once the area has been reached, the method may suggest to the driver setting a timer or considering the completion of charging for a wake-up timer. During resting, the method may offer noise cancellation and irrelevant message suppression in addition to setting the vehicle to a condition including pleasant ambient light, suitable temperature etc. Once the timer has lapsed, the method wakes up the driver slowly by e.g. playing quiet music and/or offering a massage. The method may then check the driver’s fatigue level again and update the success rate of the recommended action accordingly. Over time, the method’s algorithm establishes a driver’s preferences and characteristics from her reactions to the output recommendations, i.e. whether the driver accepts or rejects the recommendations, from the current context i.e. the weekday, daytime, weather, driving time etc., and from the success of the proposed mode i.e. the reevaluated fatigue level. Thus, future recommendations can be improved and the driver may experience a highly individualized and effective vehicle.

The invention is not limited to the presently described embodiment, but may be varied in different ways. All features defined in the claims, the description and the figure including construction details, spatial arrangements may be relevant to the invention.

Claims

Driver monitoring method for monitoring the activity of a vehicle driver, comprising the steps of

-      estimating the fatigue level of the driver and/or a time spent waiting by the driver;

-      establishing the driving context of the vehicle and/or the vehicle status;

-      outputting one of a plurality of recommended actions for altering the activity of the driver based on the fatigue level, the time spent waiting, a success rate of the recommended action, the driving context and/or the vehicle status;

-      evaluating the response of the driver to the recommended action; and

-      evaluating and updating the success rate of the recommended action.
Driver monitoring method according to claim 1, characterized in that the fatigue level of the driver and/or the time spent waiting by the driver are estimated by means of a driver monitoring camera and/or by evaluating the lateral and longitudinal vehicle control performed by the driver.
Driver monitoring method according to any of the previous claims, characterized in that establishing the driving context of the vehicle and/or the vehicle status comprises establishing the current weekday, daytime, driving time, weather, road type, driving history, advanced driver assistance system responses, a battery level and/or a position of the vehicle.
Driver monitoring method according to any of the previous claims, characterized in that the plurality of recommended actions comprises suggesting a refresh mode and/or a rest mode.
Driver monitoring method according to claim 4, characterized in that the refresh mode is output at a lower fatigue level of the driver than the rest mode.
Driver monitoring method according to claim 4 or 5, characterized in that the rest mode comprises

-      suggesting a resting area, in particular a resting area with a battery charging station;

-      suggesting setting a timer for outputting an alarm, outputting an alarm based on the charging status of the battery, playing of music and/or providing a massage to signal the end of the rest mode to the driver;

-      suggesting executing noise cancellation, suppression of irrelevant messages, setting of ambient light and/or temperature.
Driver monitoring method according to any of claims 4 to 6, characterized in that the refresh mode comprises

- suggesting a recreational activity.
Driver monitoring method according to any of the previous claims, characterized in that evaluating the response of the driver to the recommended action comprises establishing which recommended actions have been accepted by the driver and to what degree.
Driver monitoring method according to any of the previous claims, characterized in that evaluating the response of the driver to the recommended action comprises estimating the fatigue level of the driver after the recommended action has been completed.
Vehicle comprising a controller, input means and output means for executing a method according to any of claims 1 to 9.