DE102020110417A1 - Method and system for assisting a driver - Google Patents

Abstract

According to a method for assisting a driver (5) of a vehicle (1), a gaze tracking system (3) is used to determine a gaze direction (6) of the driver (5). A computing unit (4) is used to determine an intersection (7) of the viewing direction (6) with a virtual plane (8) and to track the intersection (7) during a predefined period of time, a map (9a, 9b, 9c, 9d), which represents a distribution of the position of the intersection point (7) during the period of time. The computing unit (4) is also used to determine a degree of cognitive distraction of the driver (5) based on the map (9a, 9b, 9c, 9d) and to generate or cause an output signal to inform the driver (5) depending on the particular level of cognitive distraction.

Description

The present invention relates to a method for assisting a driver of a vehicle, a gaze tracking system being used to determine a direction of gaze of the driver. The invention also relates to a corresponding driver assistance system for a vehicle and a computer program product.

Distraction of a driver of a motor vehicle can lead to serious accidents. The distraction can take various forms. For example, if the driver takes their eyes off the road it can be considered a visual distraction and if the driver takes their hands off the steering wheel it can be considered a manual distraction of the driver. In addition, there can be cognitive distraction, whereby the driver directs his mental attention away from the driving task.

Existing approaches to distraction detection include sensors that detect when the driver takes their hands off the wheel or use an eye tracker to detect when the driver is looking away from the road.

For example describes document US 6,496,117 B2 a system for monitoring a driver's attention to controlling a vehicle. The driver's gaze and facial pose are determined by means of a camera. If the driver's gaze or facial pose is not facing forward in the direction of travel, the system can issue a warning signal.

However, known approaches are unable to reliably take into account cognitive distraction, since cognitive distraction is not necessarily correlated with the fact that the driver directs his gaze to or from the roadway, or with the fact that the driver has his hands on the steering wheel or not.

It is therefore the object of the present invention to provide an improved concept for assisting a driver of a vehicle which reduces the risk of an accident due to cognitive distraction of the driver.

This problem is solved by the subject matter of the independent claims. Further designs and preferred embodiments are the subject of the dependent claims.

The improved concept is based on the idea of tracking the driver's gaze over a period of time and determining the degree of cognitive distraction based on a map that shows the distribution of an intersection of the line of sight with a virtual plane in front of the driver during the period of time.

According to the improved concept, a method for assisting a driver of a vehicle is provided. A gaze tracking system is used to determine the driver's gaze direction, and a computing unit, in particular the vehicle, is used to determine an intersection of the direction of gaze with a virtual plane and the direction of gaze and thereby the intersection point during a predefined period of time, in particular a measurement period or acquisition time span. The arithmetic unit is used to determine a map which represents a distribution of the position of the intersection point on the virtual plane during the period of time. The computing unit is used to determine a degree of cognitive distraction of the driver based on the map and to generate or cause an output signal in order to support the driver depending on the determined degree of cognitive distraction.

The gaze tracking system can include, for example, an infrared sensor or an infrared camera, with a potential position of the head or the eyes of the driver being in a field of view of the camera or the sensor. Infrared radiation, which is emitted by an infrared emitter of the gaze tracking system, is partially reflected by the driver's eyes depending on the direction of gaze. In this way, the gaze tracking system can determine the direction of gaze.

However, alternative approaches to tracking or determining the line of sight may also be applicable.

The viewing direction can be understood, for example, as a straight line that is defined by a viewing direction vector.

In order to track the point of intersection during the period of time, for example, the viewing direction is determined repeatedly at a predefined frequency, for example on the order of several tens of milliseconds, for example about 50 ms, and the point of intersection is determined for each repetition. The positions of the intersection point can be saved or cached during the period in order to determine the map.

The map can be understood as a representation, in particular as an image representation, of the point of intersection in two dimensions, in particular on the virtual plane. The card can, for example, have the form of a matrix with entries, with values of the entries of duration or Depend on the probability that the viewing direction has the corresponding point of intersection with the virtual plane during the period of time. In other words, the map shows a local distribution of gaze intensity.

For example, the map can be displayed visually as a two-dimensional heat map or color level map or grayscale map. In other words, the map can be displayed as a two-dimensional image.

Causing the output signal can include, for example, causing a further unit, for example an actuating element, a display, or a further computing unit, to generate the output signal.

The virtual plane can be understood as a geometric construction of a flat surface of predefined size. The virtual plane can also be referred to as a virtual wall or an imaginary surface.

In particular, the virtual plane is virtually at a predefined distance from the eye tracking system, for example the camera or the sensor, or a potential reference position for the head or the eyes of the driver.

According to some embodiments, a position and orientation of the virtual plane is permanently linked to a vehicle coordinate system of the vehicle or a sensor coordinate system of the gaze tracking system. In particular, the position and orientation of the virtual plane with regard to the gaze tracking system or the vehicle does not change if the direction of the driver's gaze has actually changed.

For example, the virtual plane can be positioned parallel to a transverse axis of the vehicle coordinate system or the sensor coordinate system and can be positioned at a fixed angle or predefined angle with respect to a normal axis of the vehicle coordinate system or the sensor coordinate system. The angle can be approximately zero degrees, for example, so that the virtual plane is oriented at right angles to a longitudinal axis of the vehicle coordinate system or the sensor coordinate system. The virtual plane can, for example, be positioned at a predefined distance from an origin of the vehicle coordinate system or the sensor coordinate system in a longitudinal direction.

Using a method according to the improved concept, the gaze dynamics is analyzed directly in order to identify the driver's cognitive distraction, for example in so-called daydream situations in which the driver's gaze is directed towards the road, but the gaze dynamics are not the current traffic situation or driving situation is appropriate.

By taking into account the degree of cognitive distraction, risky situations and accidents can be prevented.

Since an image-based representation of the gaze dynamics in the form of the map is used, the degree of distraction or the respective cognitive state of the driver can be determined in a particularly reliable manner, for example in comparison to a mere analysis of the viewing direction. Furthermore, due to the image-based representation, an algorithm for analyzing the map to determine the degree of cognitive distraction can require less data for training and / or generate more precise results.

By analyzing the intersection of the line of sight with a virtual plane, the movement of the driver's eyes can be regarded as being enlarged by the distance between the driver's eyes and the virtual plane. Therefore, more accurate results can be obtained, for example compared to merely tracking the driver's pupils.

According to some implementations of the method according to the improved concept, the output signal includes a warning signal to the driver.

According to some embodiments, the warning signal includes a visual signal and / or an acoustic signal and / or a haptic signal.

According to some embodiments, the output signal can be used, for example by a driver assistance system, which can contain the computing unit, in order to initiate a risk reduction measure, such as for example limiting a maximum speed of the vehicle or increasing a degree of driver assistance.

According to some embodiments, the computing unit is used to apply a classification algorithm, in particular a trained classification algorithm, to the map or to data derived based on the map in order to determine the degree of cognitive distraction.

For example, the classification algorithm can use binary classification to classify the cognitive distraction into two classes, one corresponding to distracted driving and one corresponding to neutral or undistracted driving.

However, more than two classes can be used to allow a finer response.

According to some embodiments, the computing unit is used to extract a predefined set of one or more characteristic features from the map and to apply a classification algorithm, in particular a trained classification algorithm, to the set of characteristic features.

The characteristic features can include, for example, statistical properties of the map or the distribution, such as a standard deviation or a maximum value of the distribution in a given direction on the virtual plane and so on.

The classification algorithm can also include the conversion of the predefined set of characteristic features into a form suitable for classification by the classification algorithm.

According to some implementations, the classification algorithm includes a support vector machine or an artificial neural network, for example a convolutional neural network, CNN.

These algorithms have proven to be powerful tools for image-based classification.

According to some embodiments, the processing unit is used to determine a spatial area on the virtual plane, all entries of the map corresponding to the spatial area having values that are equal to or greater than a predefined threshold value. The computing unit is used to determine the degree of distraction based on the spatial area.

The characteristic features can relate, for example, to one or more geometric features of the spatial area and / or one or more features relating to the shape of the spatial area. For example, an area or a size in a predefined direction on the virtual plane can be used as such features.

According to some embodiments, one or more further spatial areas are determined using the computing unit, all entries of the map corresponding to the respective further spatial area having respective values that are equal to or greater than a predefined respective further threshold value. The computing unit is used to determine the degree of cognitive distraction based on the further spatial areas.

The further threshold values can be different from the threshold value.

According to some embodiments, the computing unit is used to determine an image moment and / or a Hu moment based on the map. The computing unit is used to determine the degree of cognitive distraction based on the image moment and / or based on the Hu moment.

The image moment and / or the Hu moment can be viewed as features of the predefined set of characteristic features.

According to some embodiments, the computing unit is used to compare the degree of cognitive distraction with a predefined maximum distraction value and to generate the output signal as a warning signal if the degree of cognitive distraction is greater than the maximum distraction value.

According to some embodiments, the computing unit is used to track the viewing direction and thereby the point of intersection during a predefined further time span, a duration of the further time span being different from a duration of the time span. The arithmetic unit is used to determine a further map which represents a further distribution of the position of the intersection point during the further period of time. The computing unit is used to determine a further degree of cognitive distraction of the driver based on the map and to generate the output signal as a function of the determined further degree of cognitive distraction.

In general, the duration of the time period or the further time period can be selected according to a trade-off between reactivity and reliability. The shorter the duration, the higher the reactivity, but the longer the duration, the higher the reliability.

By following the direction of view and the point of intersection for different periods of time of different duration, and the respective parallel generation of the maps and the other maps, both advantages can be combined with one another and a high reactivity with high reliability can be achieved.

In particular, the further time span and the time span can overlap.

According to some embodiments, the duration of the time span is in the interval [1 s, 30 s], preferably in the interval [10 s, 20 s].

According to some embodiments, the duration of the further time span is in the interval [15 s, 120 s], preferably in the interval [30 s, 90 s].

In particular, the duration of the time span can lie in the interval [10 s, 20 s], while the duration of the further time span lies in the interval [30 s, 90 s]. For example, the period of time can be approximately 15 seconds and the further period of time can be approximately 60 seconds.

According to some embodiments, the further time span is at least twice as long as the time span, preferably the further time span is at least four times as long as the time span.

According to some embodiments, an environment sensor system of the vehicle is used to identify an object that is located in an environment of the vehicle, in particular during the time period. The computing unit is used to supplement the map with information relating to a position of the object with regard to the virtual plane, and the computing unit is used to determine the degree of cognitive distraction based on the supplemented map.

For example, the computing unit can insert a symbol that represents the object, such as a bounding box or the like.

The environment sensor system can include, for example, a further camera system, a lidar system, a radar system and / or an ultrasonic sensor system.

In particular, the environment sensor system can generate environment sensor data that represent the object, and the computing unit can supplement the map based on the environment sensor data.

The interpretation of the gaze distribution can depend on other objects in the scene, such as other vehicles, pedestrians, traffic signs and so on. The interpretation can depend on the number of objects, their respective positions and / or their type.

Thus, such implementations allow such effects to be taken into account in determining the degree of cognitive distraction. In particular, the distribution of the viewing direction, if there are further objects in the scene, can be different compared to an empty lane in front of the driver. As a result, greater reliability of the method and an improved level of security can be achieved.

According to the improved concept, a driver assistance system for a vehicle, in particular an advanced driver assistance system ADAS, is also provided. The driver assistance system includes a gaze tracking system which is configured to determine a direction of gaze of a driver of a vehicle. The driver assistance system contains a computing unit which is set up to determine an intersection of the viewing direction with a virtual plane and to track the intersection during a predefined period of time. The arithmetic unit is set up to determine a map that represents a distribution of the position of the intersection point during the period of time, to determine a degree of cognitive distraction of the driver based on the map and to generate or cause an output signal to make the driver dependent on to support the particular level of cognitive distraction.

In some embodiments, the driver assistance system can include the environment sensor system described above with regard to the method.

According to some embodiments of the driver assistance system, the gaze tracking system includes an emitter which is set up to emit infrared radiation in the direction of an eye of the driver, and an infrared sensor which is set up to detect a portion of the infrared radiation that is reflected from the driver's eye, and to generate a sensor signal as a function of the detected portion of the infrared radiation. The gaze tracking system includes a control unit which is configured to determine the gaze direction based on the sensor signal.

The control unit can be partially or completely contained by the computing unit or can be implemented separately.

Further versions of the driver assistance system in accordance with the improved concept result from the various versions of the method in accordance with the improved concept and vice versa. In particular, a driver assistance system according to the improved concept can be programmed or set up to carry out a method according to the improved concept, or the system carries out such a method.

According to the improved concept, a computer program that includes instructions is also provided. If the commands or the computer program by a driver assistance system according to are executed according to the improved concept, the commands cause the driver assistance system to carry out a method according to the improved concept.

According to the improved concept, a computer-readable storage medium that stores a computer program according to the improved concept is also provided.

Both the computer program and the computer-readable storage medium can therefore be viewed as respective computer program products that contain the commands.

Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the specified combination, but also in other combinations without departing from the scope of the invention . There are thus also embodiments of the invention to be regarded as encompassed and disclosed, which are not explicitly shown and explained in the figures, but emerge and can be generated from the explained embodiments by means of separate combinations of features. Designs and combinations of features are also to be regarded as disclosed, which therefore do not have all the features of an originally formulated independent claim. In addition, designs and combinations of features, in particular through the statements set out above, are to be regarded as disclosed which go beyond the combinations of features set forth in the back-references of the claims or differ from them.

• 1 schematisch ein Fahrzeug mit einer beispielhaften Ausführung eines Fahrerassistenzsystems gemäß dem verbesserten Konzept,
• 2 ein Flussdiagramm einer beispielhaften Ausführung eines Verfahrens gemäß dem verbesserten Konzept,
• 3 schematisch eine virtuelle Ebene zur Verwendung in beispielhaften Ausführungen gemäß dem verbesserten Konzept,
• 4 schematisch ein Bespiel einer Karte zur Verwendung in beispielhaften Ausführungen gemäß dem verbesserten Konzept,
• 5 schematisch weitere Beispiele von Karten zur Verwendung in weiteren beispielhaften Ausführungen gemäß dem verbesserten Konzept,
• 6 schematisch Schritte zur Erzeugung einer Karte zur Verwendung gemäß dem verbesserten Konzept,
• 7 schematisch Verteilungen zur Verwendung gemäß dem verbesserten Konzept,
• 8 schematisch weitere Beispiele von Karten zur Verwendung in weiteren beispielhaften Ausführungen gemäß dem verbesserten Konzept, und
• 9 weitere Beispiele von Karten zur Verwendung in weiteren beispielhaften Ausführungen gemäß dem verbesserten Konzept.

In the figures show:

• 1 schematically a vehicle with an exemplary embodiment of a driver assistance system according to the improved concept,
• 2 a flowchart of an exemplary embodiment of a method according to the improved concept,
• 3 schematically a virtual plane for use in exemplary embodiments according to the improved concept,
• 4th schematically an example of a card for use in exemplary embodiments according to the improved concept,
• 5 schematically further examples of cards for use in further exemplary embodiments according to the improved concept,
• 6th schematically steps for generating a map for use according to the improved concept,
• 7th schematic distributions for use according to the improved concept,
• 8th schematically further examples of cards for use in further exemplary embodiments according to the improved concept, and
• 9 further examples of cards for use in further exemplary embodiments according to the improved concept.

In 1 is a vehicle 1 shown schematically, which is an exemplary embodiment of a driver assistance system 2 according to the improved concept.

The driver assistance system 2 includes a gaze tracking system 3 , which is set up for this, a viewing direction 6th of a driver 5 (please refer 3 ) of the vehicle 1 to determine. It also includes the driver assistance system 2 an arithmetic unit 4th especially in combination with the eye tracking system 3 can carry out a method according to the improved concept.

Optional includes the driver assistance system 2 also an environmental sensor system 10 such as a camera system, a lidar system, a radar system and / or an ultrasonic sensor system.

The function of the driver assistance system 2 In accordance with the improved concept, the following is more precise with regard to exemplary embodiments of methods in accordance with the improved concept, and in particular with regard to 2 until 9 set out.

2 FIG. 3 shows a flow diagram of an exemplary embodiment of a method according to the improved concept.

step S0 denotes the beginning of the procedure. In step S1 determines the eye tracking system 3 a line of sight 6th of the driver 5 (please refer 3 ). Furthermore, the processing unit 4th in step S1 determines whether the line of sight is a virtual plane 8th cuts that is in front of the driver 5 located, as in 3 shown schematically.

In the example of 3 intersects the line of sight 6th the virtual plane 8th at an intersection 7th . In the event that the line of sight 6th the predefined virtual plane 8th does not cut, the procedure begins with step S0 from the front.

If on the other hand the line of sight 6th the virtual plane 8th intersects, the coordinates of the intersection point 7th on the virtual level 8th for example on a storage element of the computing unit 4th get saved. The described steps of detecting the line of sight 6th , of checking whether this corresponds to the virtual plane 8th intersects, and, optionally, saving the position of the respective intersection point 7th , are repeated periodically during a predefined time window or period.

In step S2 generates the arithmetic unit 4th a map based on the tracked line of sight and the corresponding tracked position of the intersection point 7th .

It should be noted that the procedural steps S1 and S2 can also be carried out partially parallel to each other. In particular, the time span can be a sliding time window.

To determine the map, the computing unit determines 4th a distribution or a density distribution of the viewing direction or the point of intersection 7th during the time interval. The result is a distribution map 9a that can be used as a heat map, as in 4th shown, can be viewed.

In particular, the card can 9a different colors or different gray values for different values of the densities of intersections 7th on the virtual level 8th have during the time interval.

In the steps S3 and S4 determines the arithmetic unit 4th a degree of cognitive distraction for the driver 5 based on the map 9a .

For example, during S3 the arithmetic unit 4th extract a predefined set of characteristic features. In step S4 can the arithmetic unit 4th apply a trained classification algorithm to the set of characteristic features.

As a result of the classification algorithm, the degree of cognitive distraction is obtained.

Alternatively, the computing unit 4th a classification algorithm directly on the card 9a use to determine the level of cognitive distraction.

In order to extract the characteristic features, a feature extraction module of the computing unit 4th , in particular a software module, calculate the set of features and send them to the classification algorithm, which predicts the degree of cognitive distraction of the driver. The result can be a set of information including the status of the driver, in particular his degree of cognitive distraction, in other words cognitive load. Possibly other useful information can be obtained from the map 9a can be extracted, for example the willingness or intention of the driver to change lanes.

By comparing statistical or geometric characteristics of the map 9a such as the area and standard deviation of the geometric features on the map 9a , the classification algorithm can determine if there is a high degree of cognitive distraction, such as in the map 9b exemplified in field a) or by the card 9c as in field b) of 5 shown, or whether the degree of cognitive distraction is low, as shown for example in the map 9d in field c) by 5 .

In step S5 can the arithmetic unit 4th compare the determined degree of cognitive distraction to a predetermined threshold or maximum distraction. If the cognitive level of distraction is less than the maximum distraction, the procedure starts over with S0. Otherwise an output signal in step S6 generated to the driver 5 to warn or assist by drawing attention to the cognitive distraction.

Optionally, the environmental sensor system 10 Generate sensor data that is an area around the vehicle 1 represent. The sensor data can, for example, indicate that there is another vehicle in the vicinity. Then the arithmetic unit can 4th complete the map by adding a representation 11 of the other vehicle in a supplemented map 9e is included, as in field b) of 5 displayed.

The knowledge of the presence of the further vehicle allows the classification algorithm to expect a broader heat map which, for example, should cover at least part of the further vehicle. Therefore, the representation of the further vehicle in step S4 can be used to estimate the degree of cognitive distraction.

The number of other vehicles and their positions can also influence the expected dynamics of the viewing direction and can thus be taken into account.

The following is a detailed explanation of an experimental setup for verifying the advantages of the improved concept with reference to FIG 6th until 9 described. The explanations can be understood as forming the basis of the improved concept as worked out by the inventors. Consequently, you can the following explanations relate to particularly specific and non-restrictive versions. However, individual features or combinations of features described below can also be combined with different versions of the method and / or the system for assisting a driver in accordance with the improved concept.

Specific measurements that affect the eyes, such as blinking, saccades, pupils, and fixations, are relevant for assessing cognitive load. The visual scanning behavior of an observer tends to be limited during times of increased cognitive stress, which is parallel to the fact that mental tasks lead to spatial concentration and impairment of visual detection. Based on this knowledge, instead of individually detecting and analyzing all movements affecting the eyes, an approach is described which summarizes all gaze activities. Therefore, the gaze vector of the driver's eyes is projected onto an imaginary distant surface, which is also referred to as a virtual plane. By following the change in time of this projection, an image-based representation, also referred to as a map, is created. These shapes reveal the driver's cognitive load.

In particular, a connection between short-term memory and distraction while driving is being investigated. The concepts of cognitive stress, inattention, and distraction are three different concepts. Cognitive load refers to the percentage of resources used in memory. Inattention is the fact that the driver turns his attention away from his driving task to other secondary tasks, and distraction refers to the driver being preoccupied with other tasks. Distraction leads to inattention to a certain task and this leads to a high cognitive load. Therefore, the following assumption is made: During neutral driving, the driver has enough cognitive resources to examine the environment and performs normal tasks related to driving, such as regularly checking mirrors, other vehicles, traffic signs, etc. vestibulo-ocular movements (fixations), saccades (fast, ballistic movements) and smooth transitions (slower tracking movements) can be observed. However, if the driver is distracted while driving, the driver has fewer cognitive resources left for the driving task. Thus, the gaze behavior has a shorter range. As a result, a change in eye movements is expected.

The test protocol is explained below.

The test session consisted of two driving laps on the same route. The first lap represents the starting position in which the driver naturally performed the driving task. He (all participants were male, mean age 29.4 years) is told to relax and drive carefully. This lap is called neutral driving and is important in order to learn the starting position of the gaze variations of the participants' eyes. The second lap was carried out immediately after the first on the same section of the route. During the second lap, the driver had to perform four secondary tasks aimed at cognitively overloading the driver. This lap is called distracted driving.

An important aspect of the test protocol was to simulate the driving conditions (road surface, weather, traffic jam) as similarly as possible between sessions, but also for both laps, which were carried out by a single participant. Therefore, a motorway route in France was defined as a test route for each participant. The speed limit on this freeway was constant (90 km / h) and it took 22 minutes to complete a single lap. It was driven between 10 a.m. and 5 p.m. during the day to minimize fluctuations in weather and traffic conditions.

A specialist is responsible for the test protocol, for starting the secondary tasks, for commenting on the events and for guiding the driver along the route. This specialist is referred to as a companion in the following sections.

Five drivers participated in the data collection protocol. All of them were volunteers working in the auto industry. However, they were unaware of the purpose of the driving session.

The central goal of the secondary tasks was to increase the mental workload on the driver. The literature lists the various secondary tasks such as tapping the foot (secondary task) while learning (primary task) and tasks for measuring rhythmic position or measuring detection response while driving. In an experiment based on a driving simulator, drivers had visual, manual, auditory, verbal and haptic secondary tasks to perform. Results of a gaze analysis of the eyes showed that the visual ORTs (Detection Response Tasks) were more efficient than others. In the present study However, the visual secondary task was left aside in order to keep the gaze patterns of the eyes as neutral as possible. In order to approach a more natural way of performing the experiment, attention has been directed to immersive and entertaining secondary tasks. The following four games are all designed based on the N-back task strategy. N-back tasks are cognitively distracting tasks that require participants to recall consecutive instructions. Remembering these sequential instructions leads to an increase in mental workload. Each game was designed to last four minutes. After a pause of one minute, the next game began.

The first game is called “neither yes nor no”. This game is based on the abolition of the words “yes”, “no” and their alternatives such as “yeah” or “oui”. The companion asks successive questions to force the participants to pronounce these words.

The second game is called “is in my chest”. It consists in quoting the words “is in my chest” followed by the name of an object. The participant and the companion should take turns naming all previous objects from the memory again and adding a new one to the list.

The third game is called "guess who". The participant thinks of a real or imaginary figure and the companion tries to find out the identity of the figure by asking a mobile applicator. The participant should answer the questions correctly.

The fourth game is called "the 21". The companion names one to three numbers in numerical order, for example 1, 2, 3, and the driver follows the order and names a different number of digits than the companion, for example two numbers 4th , 5 . The game continues like this. The number 21 is prohibited. Instead of 21 to say, a rule should be added to the game, for example “don't say multiples of 4”.

For data acquisition, the position of the interior parts of the vehicle, such as the interior mirror and instrument unit, is measured and displayed in a 3D world representation, as in 3 outlined. These objects are static to a reference coordinate system with the vehicle as the origin.

While driving, the driver is monitored with an infrared rear camera, which is placed in front of the instrument unit. This camera extracts the head position, the position of the eyes and their direction. The head position and line of sight of the eyes are also in 3 shown. Therefore, when the driver is looking at one of the objects that are in the scene, it can be determined.

In 3 the circle can be viewed as representing the position of the driver's head. The gaze vector of the driver's eyes is shown as a line extending from this circle. The small filled rectangles represent the interior parts of the vehicle, such as left and right mirrors, center mirrors, speed indicators of the instrument cluster, tachometer, central stack and navigation screen. The virtual plane is also called a virtual wall.

The virtual wall is a surface that is placed in front of the vehicle as if it were one of the interior parts of the vehicle. The point of intersection between this virtual wall and the gaze vector of the eyes is tracked in a given time window. Following the variations of the intersection over the area, an image-based representation is generated. This representation, which can be referred to as a heat map, consists of the collected data that is used to detect the driver's cognitive load.

The vehicle is also equipped with an RGB front camera. The positions and dimensions of the imaginary surface are set to maximize the intersection of this surface with the field of view of the camera and the area where gaze detection is available. This camera is located in the inner center of the vehicle, while the driver sits in the left front seat. Thus, the activities of the driver's gaze on the left side of the image seem to be focused on the overlays and heat maps.

The heat map is a data visualization technique used in various studies and solutions. These are often used to highlight the areas of interest to people, so we can investigate several situations on them. A heat map, as in 4th can be used for visualization and feature extraction after successive steps to create the heat map, as in the tables of 6th displayed.

The time-stamped, unprocessed intersection points in x and x between the gaze vectors of the eyes and the imaginary surface are the input of the creator of the heatmap. This data can be acquired every 50 ms when the driver looks to the imaginary plane. If the driver is not looking at the plane, the source of distraction is visual and not necessarily cognitive. For example, the driver could check his smartphone. Nevertheless, this deflection can also be detected by the present method.

Since a single point of intersection is not sufficiently informative for the problem at hand the points are cached as a sliding window, as in panel a) of 6th shown. With the aim of covering the driver's field of vision, a circle of 15 pixels is placed in the middle of the intersection points, as in field b) of 6th shown. The choice of the circle diameter that represents the gaze fixation is mainly influenced by the dimensions of the heat map, which in the present case are 640 × 400. After normalizing the circles of the field of view, this mask is used to vary the opacity of the intersection points, as in field c) of 6th displayed. A Gaussian filter is applied to the mask to reduce the noise due to gaze activity and also to reduce a detail that keeps the focus on the most examined area, as in panel d) of 6th displayed.

Feature construction is applied to the generated heat maps to reduce the data dimension. The following sets of features are extracted from each heat map based on their pixel intensities and shape.

For example, appearance features can be extracted. The pixel intensity variation of a heat map contains information about the area overlooked by the driver. The histogram is an efficient tool to visualize these data distributions, especially when the gaze activity is focused on a specific area or when it is exploring a broader area. During distracted driving, more focus is expected to be visualized at higher intensities, and during natural driving, where the driver should cover a wider area, more focus is expected to be visualized at lower intensities. Therefore, a 6-bin histogram of the number of pixels in terms of pixel intensity per heat map is generated, as in fields a) and b) in 7th displayed. 7th shows the calculated histograms of the in 4th shown heatmap. In field a), pixel intensities are shown on the abscissa and the respective number of pixels on the ordinate. In field b) the values of field a) are summed to 6 bins.

The geometric features can also be extracted. In addition to the pixel intensities that are exploited by heat map histograms, their shape must also be taken into account. The generated heat map is divided into outlines according to the difference in the pixel intensities: binary large objects (BLOBs) as in the fields of 8th shown.

Field a) from 8th represents the driver gaze activity heat map. The central areas in the BLOB represent the most pinned area and the border area is less pinned. Field b) shows a threshold value heat map, converted into a grayscale image with identified focus contours. In fields c), d), e), f), extracted contours from the threshold value heat map of field b) are shown. Each contour is defined by the pixel intensities. A binary threshold is carried out for each zone.

Furthermore, image moments can be extracted. The grayscale BLOBs are constructed from movements that contain kinetic energy and movement history. Therefore, the kinetic energy and history of each of them are examined. The following equation (1) describes the calculated features based on the image moments, which are specially weighted averages of the image intensities or a representation of the pixel distribution in the image. $${\text{M.}}_{\text{ij}}={\text{Sum}}_{\text{x}}\left[{\text{Sum}}_{\text{y}}\left[{\text{x}}^{\text{i}}{\text{y}}^{\text{i}}\text{l}\left(\text{x}\text{, y}\right)\right]\right]$$

For a given image I (x, y), i and j are integers that define the order of moments.

Moments are sensitive to the pixel position in the image, whereas the shapes are expected to be detected regardless of their positions (so-called regular moments).

Hu moments are seven special image moments that are translation invariant, rotation invariant and scale invariant. They are special combinations of the central moments designated by h1 to h7. Hu moments are extracted per BLOB.

• • Standardabweichung zu x und y
• • Schwerpunktskoordinaten (falls es mehrere Konturen gibt, ihr Mittelwert)
• • Grenzen jeder Zone (Maximum und Minimum von x und x)
• • erstes Quartil, Median und drittes Quartil auf x und y
• • Fläche der Kontur
• • Umfang der Kontur

Statistical measures can also be extracted. In order to record the information about the driver's gaze distribution over the imaginary plane, the following features are extracted as statistical measures over all BLOBs:

• • Standard deviation from x and y
• • Coordinates of the center of gravity (if there are several contours, their mean value)
• • Limits of each zone (maximum and minimum of x and x)
• • first quartile, median and third quartile on x and y
• • Area of the contour
• • Perimeter of the contour

If the driver does not look at the image and the imaginary plane for the entire given time window, the heat map could contain less relevant data. Therefore, the information on how much time was spent looking ahead is another characteristic that determines the quality of the heat map.

Finally, all extracted features are normalized by removing the mean value and scaling it to the unit variants per heat map.

A supervised binary classification algorithm based on a support vector machine is trained with extracted features. Data collected during the neutral driving has been annotated as neutral and data collected during the presence of the secondary tasks has been annotated as distracted.

The classification is validated by a stratified K-fold cross-validation technique with ten repetitions (K = 10). Stratifications seek to ensure that each convolution is representative of all straight lines in the data, which aims to ensure that each class is equally represented across the test convolution and consists in dividing the data set into patterns.

In parallel with the initial expectations, the variation in shapes obtained is visually different between neutral driving and distracted driving, as in FIG 9 displayed. These shapes occupy a wide area in neutral driving when the driver frequently checks the surroundings. However, in the presence of cognitive distraction, the area covered becomes narrower as the driver becomes more fixated on a single zone. For a heat map based on a longer observation time, a more separable visual pattern is obtained. This is due to the fact that in neutral driving, as opposed to distracted driving, the driver has more time to observe the surroundings. As a result, the distances between neutral and distracted driving patterns become more apparent over time, as in FIG 9 displayed.

9 displays heat maps for 5, 15 and 30 s for neutral driving scenarios and driving scenarios under distraction. Fields a), c), e) correspond to driving scenarios under distraction while fields b), d), f) correspond to neutral driving scenarios. Fields a) and b) correspond to 5 s, fields c) and d) correspond to 15 s and fields e) and f) correspond to 30 s.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 6496117 B2 [0004]

Claims (15)

A method for assisting a driver (5) of a vehicle (1), wherein a gaze tracking system (3) is used to determine a viewing direction (6) of the driver (5), characterized in that a computing unit (4) is used to - to determine an intersection (7) of the viewing direction (6) with a virtual plane (8) and to track the intersection (7) during a predefined period of time; - to determine a map (9a, 9b, 9c, 9d) which represents a distribution of the position of the intersection point (7) during the period of time; - to determine a degree of cognitive distraction of the driver (5) based on the map (9a, 9b, 9c, 9d); and - to generate or cause an output signal to assist the driver (5) depending on the particular degree of cognitive distraction. Procedure according to Claim 1 , characterized in that the computing unit (4) is used to apply a classification algorithm to the card (9a, 9b, 9c, 9d) in order to determine the degree of cognitive distraction. Procedure according to Claim 1 , characterized in that the computing unit (4) is used to extract a predefined set of characteristic features from the map (9a, 9b, 9c, 9d) and to apply a classification algorithm to the set of characteristic features. Method according to one of the Claims 2 or 3 , characterized in that the classification algorithm comprises a support vector machine or an artificial neural network. Method according to one of the preceding claims, characterized in that - the computing unit (4) is used to determine a spatial area on the virtual plane (8), with all entries on the map (9a, 9b, 9c, 9d), corresponding to the spatial area have respective values that are equal to or greater than a predefined threshold value; and - the computing unit (4) is used to determine the degree of cognitive distraction based on the spatial area. Method according to one of the preceding claims, characterized in that - the computing unit (4) is used to determine an image moment and / or a Hu moment based on the map (9a, 9b, 9c, 9d); and - the computing unit (4) is used to determine the degree of cognitive distraction based on the image moment and / or based on the Hu moment. Method according to one of the preceding claims, characterized in that the computing unit (4) is used to compare the degree of cognitive distraction with a predefined maximum distraction value and to generate the output signal as a warning signal if the degree of cognitive distraction is greater than the maximum distraction value. Method according to one of the preceding claims, characterized in that the computing unit (4) is used to - track the point of intersection (7) during a predefined further time period, a duration of the further time period being different from a duration of the time period; - to determine a further map (9a, 9b, 9c, 9d) which represents a further distribution of the position of the intersection point (7) during the further time period; - to determine a further degree of cognitive distraction of the driver (5) based on the map (9a, 9b, 9c, 9d); and to generate the output signal as a function of the determined further degree of cognitive distraction. Procedure according to Claim 8 , characterized in that - the duration of the time span is in the interval [1 s, 30 s], preferably in the interval [10s, 20s]; and / or the duration of the further period of time is in the interval [15 s, 120 s], preferably in the interval [30 s, 90 s]. Method according to one of the Claims 8 or 9 , characterized in that the further period of time is at least twice as long as the period of time, preferably at least four times as long as the period of time. Method according to one of the preceding claims, characterized in that the output signal includes a visual signal and / or an acoustic signal and / or a haptic signal. Method according to one of the preceding claims, characterized in that - an environment sensor system (10) is used to identify an object which is located in an environment of the vehicle (1); - The computing unit (4) is used to supplement the map (9a, 9b, 9c, 9d) with information relating to the position of the object with respect to the virtual plane (8); and - the computing unit (4) is used to determine the degree of cognitive distraction based on the supplemented map (9e). Driver assistance system for a vehicle (1), wherein the driver assistance system (2) contains a gaze tracking system (3) which is set up to determine a direction of view (6) of a driver (5) of the vehicle (1), characterized in that the driver assistance system (2) contains a computing unit (4) which is set up to - determine an intersection (7) of the viewing direction with a virtual plane (8) and to track the intersection (7) during a predetermined period of time; - to determine a map (9a, 9b, 9c, 9d) which represents a distribution of the position of the intersection point (7) during the period of time; - to determine a degree of cognitive distraction of the driver (5) based on the map (9a, 9b, 9c, 9d); and - to generate or cause an output signal to assist the driver (5) depending on the particular degree of cognitive distraction. Driver assistance system according to Claim 13 characterized in that the eye tracking system (3) - includes an emitter which is set up to emit infrared radiation in the direction of an eye of the driver (5); - Contains an infrared sensor which is set up to detect a portion of the infrared radiation that is reflected by the driver's eye (5) and to generate a sensor signal as a function of the detected portion of the infrared radiation; and includes a control unit which is configured to determine the viewing direction based on the sensor signal. Computer program product which contains commands that, when they are sent by a driver assistance system (2) according to one of the Claims 13 or 14th are executed, cause the driver assistance system to implement a method according to one of the Claims 1 until 12th perform.

Images