DE102020126701A1 - Method and device for determining the attentiveness of a driver - Google Patents

Abstract

A device for determining an alertness indicator relating to the alertness of a driver of a motor vehicle to the driving operation of the vehicle is described. The device is set up to determine a set of situation elements that are relevant to the driving operation of the vehicle on the basis of environmental data relating to the area surrounding the vehicle. Furthermore, the device is set up to determine one perception state from a plurality of different perception states for one or more situation elements from the set of situation elements on the basis of measured values relating to the viewing direction of the driver of the vehicle. The device is also set up to determine a value of the attention indicator based on the perception states of the one or more situation elements from the set of situation elements.

Description

The invention relates to a method and a corresponding device for determining the attentiveness of a driver of a vehicle.

A vehicle can have one or more driving functions that support the driver of the vehicle in driving the vehicle. This may require the driver to continue to monitor traffic and the operation of the one or more driving functions. In another example, a vehicle may be remotely controlled by a driver located outside of the vehicle. For the safety of the driving operation, a high degree of attention from the remote-control driver is typically required.

The present document deals with the technical task of determining in a precise and reliable manner the attentiveness of a driver of a vehicle in relation to the driving operation of the vehicle.

The object is solved in each case by the independent claims. Advantageous embodiments are described inter alia in the dependent claims. It is pointed out that additional features of a patent claim dependent on an independent patent claim without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim can form a separate invention independent of the combination of all features of the independent patent claim, which can be made the subject of an independent claim, a divisional application or a subsequent application. This applies equally to the technical teachings described in the description, which can form an invention independent of the features of the independent patent claims.

According to one aspect, a device for determining an alertness indicator relating to the alertness of the driver of a motor vehicle (in particular a car or a truck or a bus) to the driving operation and/or to the driving situation of the vehicle is described. The device can be set up to operate the vehicle and/or a remote control of the vehicle depending on the determined value of the alertness indicator. Safe and comfortable operation of the vehicle can thus be made possible.

The device is set up to determine a set of situation elements that are relevant to the driving operation of the vehicle on the basis of environmental data relating to the area surrounding the vehicle. The environmental data can include sensor data from one or more environmental sensors, in particular a camera, a lidar sensor, a radar sensor, an ultrasonic sensor, etc., of the vehicle. The environmental data can be analyzed using a (machine-learned) recognition unit, in particular using a classifier, in order to detect the one or more (in particular the 5 or more, or the 10 or more) situation elements which (at the current point in time n) are relevant to the operation of the vehicle.

The set of situation elements can include at least one or more situation elements for one or more objects (e.g. another road user, a traffic light, a traffic sign, etc.) in the vicinity of the vehicle that are essential for driving the vehicle (at the current point in time n) are relevant.

Alternatively or additionally, the set of situational elements can include one or more areas in the area surrounding the vehicle in which no object should be located for safe and collision-free operation of the vehicle. The one or more clearance areas may be arranged in a predefined manner relative to the vehicle. In particular, the field of view for a free space area can be known in advance. An exemplary free space area is the area of a blind spot of the vehicle, which should be observed by the driver of the vehicle (e.g. by looking over the shoulder) when the vehicle is turning, for example. A situation element can be included in the set of situation elements for such a free space area (if the free space area is relevant for the driving operation of the vehicle at the current point in time).

It is thus possible to determine a set of situational elements (which are at least partially arranged outside the vehicle) which are relevant to the driving operation of the vehicle at the current point in time n.

The device is also set up to determine one perception state from a plurality of different perception states for the one or more situation elements from the set of situation elements on the basis of measured values relating to the line of sight of the driver of the vehicle. The driver's line of sight can be determined based on sensor data from a camera directed at the driver (the sensor data also being referred to as driver data in this document). The line of sight can be the azimuth angle and/or the polar angle of the driver's gaze in a polar coordinate display the data system placed around the driver's head. The driver may be a driver located in the vehicle or a driver located outside the vehicle who remotely controls the vehicle.

The different states of perception may indicate different degrees to which the driver has perceived an element of the situation. In particular, the plurality of different perception states may include: a first perception state indicating that the driver did not recognize and/or did not see the situation element; a second perception state which indicates that the driver has recognized and/or seen the situation element, in particular only recognized and/or seen it unconsciously; and/or a third perception state which indicates that the driver has perceived the situation element, in particular consciously perceived or detected it.

A perception state can thus be determined on the basis of a large number of measured values of the driver's line of sight (which were recorded between the previous time n - 1 and the current time n) for each situation element from the set of situation elements, with the perception state being able to indicate how the driver of the vehicle has consciously perceived the respective situation element (at the current point in time).

The device is also set up to determine the value of the attention indicator (at the current point in time n) on the basis of the perception states of the one or more situation elements from the set of situation elements (at the current point in time n). The value of the attention indicator can thus be determined based on how well the driver of the vehicle has perceived the situational elements that are relevant to driving the vehicle. In this way, the driver's attentiveness can be determined in a particularly reliable and precise manner.

The device can be set up to determine the perception state of a specific situation element (from the set of situation elements), to determine the number of measured values from the plurality of measured values in relation to the driver's viewing direction that fall within the field of vision of the specific situation element. The viewing area of the specific situation element can display (all) the driver's viewing directions in which the driver's gaze is directed at the situation element. The field of view of the specific situation element can be defined by an azimuth angle range and/or by a polar angle range.

It is thus possible (in particular on the basis of the environment data) to determine for each situation element from the set of situation elements the viewing area associated with the respective situation element. The viewing areas of the situation elements can then be used to recognize whether the driver's line of sight is directed at a situation element (and thus falls within the viewing area of the situation element) or not (and therefore does not fall within the viewing area of the situation element).

The device can also be set up to select the perception state of the specific situation element from the plurality of different perception states as a function of the determined number of measured values. In particular, the first perception state can be assigned to the specific situation element (at the current point in time) if it is determined that no measured value falls within the field of view of the situation element. Alternatively or additionally, the second perception state can be assigned to the specific situation element if it is determined that at least one, in particular exactly one, measured value falls within the field of view of the situation element. Alternatively or additionally, the third perception state can be assigned to the specific situation element if it is determined that a number of measured values, in particular at least a predefined minimum number of measured values, fall within the field of view of the situation element. These conditions for assigning a perceptual state may be conditions for state transitions of a state model for determining the perceptual state (as set forth below). In this way, the states of perception of the individual situation elements can be determined in a precise manner.

The device can be set up to update the perception state for a specific situation element by means of a state model in an iterative manner at a sequence of consecutive points in time n−1, n, n+1, . . . In this case, the state model can have the plurality of different perception states as states of the state model (or of the state machine). Furthermore, the state model can include state transitions between at least some of the perception states. Furthermore, at least some of the state transitions may depend on the measured values relating to the driver's line of sight for a point in time from the sequence of points in time (as already explained above). A preferred state model is used in this document in connection with 2 described. By using a state model, the perceptions can states of the individual situation elements can be determined in a particularly precise manner.

The device can thus be set up, in an iterative manner (e.g. using a state model) starting from the perception state of the one or more situation elements from the set of situation elements at the previous point in time n-1, on the basis of the plurality of measured values with regard to the viewing direction of the driver between the previous point in time n−1 and the current point in time n, to determine the state of perception of the one or more situation elements from the set of situation elements at the current point in time n. The accuracy of the ascertained perception state can be further increased by an iterative updating of the perception state of the individual situation elements at successive points in time.

The plurality of different perceptual states may be associated with a corresponding plurality of different individual perceptual values w. Thereby, the individual perception value w for a situation element can be larger, the more the situation element is perceived by the driver (which is indicated by the perception state assigned to the situation element). The device can be set up to determine the value of the attention indicator in a particularly precise manner on the basis of the individual perception values for the one or more situation elements from the set of situation elements.

The device can be set up to determine a relevance value r for each of the one or more situation elements from the set of situation elements, which value indicates the relevance of the respective situation element for the driving operation of the vehicle. The relevance value of the individual situation elements can be provided, for example, by the (machine-learned) recognition unit, which also provides the set of relevant situation elements.

The value of the attention indicator can then be determined in a particularly precise manner on the basis of the relevance values for the one or more situation elements from the set of situation elements.

The device can be set up to determine the subset of measured values from the plurality of measured values in relation to the driver's viewing direction, which cannot be assigned to any of the situation elements from the set of situation elements. The value of the attention indicator can then also be determined on the basis of the determined subset of measured values. For this purpose, the measured values from the determined subset of measured values that cannot be assigned to any situation element can be clustered using a cluster algorithm (e.g. DBSCAN). The value of the attention indicator can then be determined in a particularly precise manner on the basis of the number o of clusters determined.

wobei K die Anzahl von Situationselementen in der Menge von Situationselementen ist, wobei w_i(n) der von dem Wahrnehmungszustand des Situationselements i abhängige individuelle Wahrnehmungswert ist, und wobei r_i(n) der Relevanzwert des Situationselements i ist, durch den die Relevanz des Situationselements i für den Fahrbetrieb des Fahrzeugs angezeigt wird. Von dem o.g. Term kann ggf. noch ein Term abgezogen werden, der von der Anzahl o von ermittelten Clustern der Teilmenge von nicht zugewiesenen Messwerten abhängt (wie weiter unten dargelegt). So kann der tatsächliche kumulierte Wahrnehmungswert in besonders präziser Weise ermittelt werdenThe device can be set up to determine an actual cumulative perception value based on the perception state of the one or more situation elements from the set of situation elements. The actual cumulative perception value can be determined based on $${\displaystyle \sum _{i=1}^K{w}_i}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right)$$

where K is the number of situation items in the set of situation items, where w _i (n) is the individual perception value dependent on the perceptual state of situation item i, and where r _i (n) is the relevance value of situation item i, by which the relevance of Situation element i is displayed for driving the vehicle. If necessary, another term can be subtracted from the above term, which depends on the number o of determined clusters of the subset of unassigned measured values (as explained further below). In this way, the actual cumulative perception value can be determined in a particularly precise manner

wobei w_max,i(n) der mit dem optimalen Wahrnehmungszustand (z.B. dem dritten Wahrnehmungszustand) des Situationselements i assoziierte (maximal mögliche) individuelle Wahrnehmungswert ist.The device can also be set up to determine an optimal cumulative perception value on the assumption that the one or more situation elements from the set of situation elements each have an optimal perception state (in particular the third perception state). The optimal cumulative perception value can be determined based on $${\displaystyle \sum _{i=1}^K{w}_{max,i}}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right)$$

where w _max,i (n) is the (maximum possible) individual perceptual value associated with the optimal perceptual state (eg, the third perceptual state) of situation element i.

The value of the attention indicator can then be determined in a particularly precise manner on the basis of the actual cumulative perception value and on the basis of the optimal cumulative perception value, in particular on the basis of the ratio of the actual cumulative perception measurement value and the optimal cumulative perception value.

According to a further aspect, a (road) motor vehicle (in particular a passenger car or a truck or a bus or a motorcycle) is described which comprises the device described in this document.

According to one aspect, a method for determining an alertness indicator relating to the alertness of a driver of a motor vehicle to the driving operation of the vehicle, in particular to a traffic situation in the area surrounding the vehicle, is described.

The method includes determining, on the basis of environmental data relating to the area surrounding the vehicle, a set of situational elements that are relevant to the driving operation of the vehicle. An example situation element is an object (e.g. another road user, a traffic sign, or a traffic light system) in the vicinity of the vehicle. The set of situational elements can be determined using a machine-learned classifier based on the environmental data.

The method also includes determining one perception state from a plurality of different perception states for the one or more situation elements from the set of situation elements on the basis of measured values relating to the viewing direction of the driver of the vehicle. The measured values in relation to the viewing direction can be determined on the basis of driver data from a camera directed at the driver. It can thus be determined on the basis of the measured values how well the driver (consciously) perceived the individual situational elements relevant to driving.

Furthermore, the method includes determining a value of the attention indicator based on the perception states of the one or more situation elements from the set of situation elements.

According to a further aspect, a software (SW) program is described. The SW program can be set up to be executed on a processor (e.g. on a control unit of a vehicle) and thereby to carry out at least one of the methods described in this document.

According to a further aspect, a storage medium is described. The storage medium can include a SW program which is set up to be executed on a processor and thereby to execute at least one of the methods described in this document.

It should be noted that the methods, devices and systems described in this document can be used both alone and in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, devices and systems described in this document can be combined with one another in a variety of ways. In particular, the features of the claims can be combined with one another in many different ways.

• 1a eine beispielhafte Verkehrssituation im Umfeld eines Fahrzeugs;
• 1b beispielhafte Komponenten eines Fahrzeugs;
• 1c eine beispielhafte Blickrichtung eines Fahrers bei einer beispielhaften Umfeldsituation;
• 2 ein beispielhaftes Zustandsmodell zur Ermittlung des Wahrnehmungszustands eines Situationselements; und
• 3 ein Ablaufdiagramm eines beispielhaften Verfahrens zur Ermittlung eines Aufmerksamkeitsindikators für die Aufmerksamkeit des Fahrers eines Fahrzeugs.

The invention is described in more detail below using exemplary embodiments. show it

• 1a an exemplary traffic situation in the vicinity of a vehicle;
• 1b exemplary components of a vehicle;
• 1c an exemplary viewing direction of a driver in an exemplary environmental situation;
• 2 an exemplary state model for determining the perception state of a situation element; and
• 3 a flowchart of an exemplary method for determining an attention indicator for the attention of the driver of a vehicle.

As explained at the outset, the present document is concerned with determining the level of alertness of a driver of a vehicle in a precise and reliable manner. In this context shows 1a an exemplary traffic situation of a vehicle 110 on a multi-lane roadway 100 with an ego lane 101, on which the vehicle 110 is driving, and with one or more neighboring lanes 102, 103. In the vicinity of the vehicle 110 there are several different other road users 120 ( in particular other vehicles, one or more pedestrians, one or more cyclists, etc.). Furthermore, one or more traffic signs, light signal systems, pedestrian crossings, lane markings, etc. can be arranged in the area surrounding vehicle 110 . These components of a road network for controlling traffic are also collectively referred to as infrastructure components 121 in this document.

1b shows exemplary components of a vehicle 110. The vehicle 110 includes one or more environment sensors 112, the turned are directed to capture environmental data (ie sensor data) in relation to the environment of the vehicle 110 .

Exemplary environment sensors 112 are an image camera, a radar sensor, a lidar sensor, an ultrasonic sensor, etc.

Furthermore, the vehicle 110 can include one or more actuators 113 for at least partially automated longitudinal and/or lateral guidance of the vehicle 110 . Exemplary actuators 113 are a drive motor, a braking device and/or a steering device.

A control unit 111 of the vehicle 110 can be set up to determine an environment model in relation to the environment of the vehicle 110 on the basis of the environment data. The environment model can, for example, describe one or more other road users 120 and/or one or more infrastructure components 121 (generally one or more objects 120, 121) in the environment of vehicle 110. The control unit 111 can be set up to operate the one or more actuators 113 of the vehicle 110 depending on the environment model, e.g. to activate a driver assistance function (such as a lane keeping assistant, a lane change assistant, adaptive cruise control, etc.) and/or an automated driving mode ( e.g. according to SAE level 3 or higher).

1c shows an exemplary driving situation in which vehicle 110 is driving longitudinally and/or laterally manually or at least partially automatically. The driver 130 of the vehicle 110 may look at different objects 120, 121 in the area surrounding the vehicle 110 while driving. The vehicle 110 may include a driver sensor 115, in particular a camera, which is set up to record sensor data (also referred to as driver data in this document ) with respect to the driver 130 to be detected. In particular, the driver data can display measured values in relation to the viewing direction 131 of the driver 130 at a large number of points in time.

The control unit 111 of the vehicle 110 can be set up to record surroundings data in relation to the surroundings (as a function of time). The driver data and the environment data can be analyzed together in order to determine which objects 120, 121 are viewed by the driver 130 while driving. It can thus be determined which objects 120, 121 are viewed by the driver 130 at different points in time. In other words, the gaze behavior of the driver 130 can be determined.

The degree of attentiveness of the driver 130 can be inferred from the gaze behavior of the driver 130 . In particular, the value of an alertness indicator for the alertness of driver 130 at a specific point in time n can be determined on the basis of the gaze behavior of driver 130 .

The control unit 111 of the vehicle 110 can be set up to determine a set of situation elements that are relevant for the driving operation of the vehicle 110 and/or for monitoring the driving operation of the vehicle 110 at a specific point in time n on the basis of the environmental data. The set of situation elements can in particular include one or more objects 120 , 121 in the area surrounding vehicle 110 .

The set of situation elements can be determined using a machine-learned recognition unit. The recognition unit can include one or more neural networks. The recognition unit can be designed to evaluate the surroundings data (in particular the image data from one or more cameras) in order to recognize one or more situational elements in the surroundings of the vehicle 110 which are (possibly) relevant to the driving operation of the vehicle 110 . In this case, the relevance, in particular a relevance value, for the driving operation can be determined for each situation element. The set of situation elements can be updated at a sequence of points in time (iteratively) on the basis of the environmental data currently recorded. In particular, it can be checked in each case which one or more situation elements enter the possible field of view of driver 130 and which one or more situation elements leave the possible field of view of driver 130 .

Furthermore, the control unit 111 of the vehicle 110 can be set up to determine a viewing area for each situation element from the set of situation elements, wherein the viewing area for a situation element indicates the one or more viewing directions 131 of the driver 130, in which the driver 130 looks in the direction of the situation element looks. In other words, the field of view for a situation element describes the set of viewing directions 131 of the driver 130 that result in the driver 130 looking in the direction of the situation element. A field of vision can be defined by an azimuth angle range and/or by a polar angle range of a spherical coordinate system that has the head of driver 130 as the origin.

The control unit 111 can also be set up, on the basis of the driver data, to calculate a large number of measured values of the line of sight 131 of the driver 130 at a large number of intermediate points in time m between the preceding point in time n−1 and to determine the current point in time n. For example, 10 or more, or 20 or more measured values of the viewing direction 131 of the driver 130 can be determined between two points in time in order to describe the viewing behavior of the driver 130 at the current point in time n.

The large number of measured values can be assigned to the different situation elements from the set of situation elements for the current point in time n. In particular, for each situation element from the set of situation elements, the number of measured values that lie in the field of view of the respective situation element can be determined. Furthermore, the subset of measured values can be determined which cannot be assigned to any situation element from the set of situation elements, i.e. which do not fall in any of the viewing areas of the situation elements from the set of situation elements. The number of measured values can thus be determined without reference to the situation. The alertness indicator at the current point in time n can then be determined based on the number of measured values per situation element and possibly based on the subset of measured values without reference to the situation.

• • ein erster Wahrnehmungszustand „Nicht-Erkannt“, d.h. das Situationselement wurde bisher nicht von dem Fahrer 130 des Fahrzeugs erkannt bzw. gesehen;
• • ein zweiter Wahrnehmungszustand „Erkannt“, d.h. das Situationselement wurde von dem Fahrer 130 des Fahrzeugs erkannt bzw. gesehen; und/oder
• • ein dritter Wahrnehmungszustand „Erfasst“, d.h. das Situationselement wurde von dem Fahrer 130 des Fahrzeugs 110 als relevantes Situationselement für die aktuelle Fahrsituation bzw. für den aktuellen Fahrbetrieb wahrgenommen.

The control unit 111 can in particular be set up to determine a perception state by the driver 130 of the vehicle 110 based on the number of measured values per situation element at the current point in time n for the individual situation elements from the set of situation elements. Exemplary states of perception are,

• • a first perception state “not recognized”, ie the situation element has not yet been recognized or seen by the driver 130 of the vehicle;
• • a second perception state “recognized”, ie the situation element was recognized or seen by the driver 130 of the vehicle; and or
• • a third perception state “detected”, ie the situation element was perceived by the driver 130 of the vehicle 110 as a relevant situation element for the current driving situation or for the current driving operation.

The perception state of the respective situation element at the current time n can thus be determined for each of the situation elements from the set of situation elements for the current time n based on the number of measured values for the respective situation element. The attention indicator can then be determined from the set of situation elements on the basis of the perception states of the individual situation elements. In particular, an actual cumulative (or total) perception value SA _actual (n) at the current point in time n can be determined on the basis of the perception states of the individual situation elements. The actual cumulative perception value SA _actual (n) can then be compared with an optimal cumulative (or total) perception value SA _optimal (n), in particular set in relation to one another, in order to determine the value of the attention indicator.

• • Wert w = 0 für den ersten Wahrnehmungszustand „Nicht-Erkannt“;
• • Wert w = 0,5 für den zweiten Wahrnehmungszustand „Erkannt“; und/oder
• • Wert w = 1 für den dritten Wahrnehmungszustand „Erfasst“.

In order to determine the actual cumulative perception value SA _actual (n), an individual perception value w can be assigned to each of the possible perception states of a situation element. A relatively high individual perception value w can indicate a relatively high degree of perception and a relatively low individual perception value w can indicate a relatively low degree of perception of the respective situation element. Exemplary individual perception values w are:

• • Value w = 0 for the first perceptual state "Unrecognized";
• • Value w = 0.5 for the second perceptual state “Recognized”; and or
• • Value w = 1 for the third perceptual state “Captured”.

The actual cumulative perception value SA _actual (n) can then be determined on the basis (the sum) of the individual perception values w of the situation elements from the set of situation elements. The optimal cumulative perception value SA _optimal (n) can be determined on the basis (the sum) of the maximum possible individual perception values w of the situation elements from the set of situation elements.

The control unit 111 can be set up (in particular on the basis of the environmental data) to determine a relevance of the individual situation elements from the set of situation elements for the current driving operation, in particular for the current traffic situation. In particular, a relevance value r can be determined for the individual situation elements, which shows the relevance of the respective situation element. A relatively low relevance value r (e.g. r=1) can indicate that the situation element has a relatively low relevance for the driving operation and may only optionally have to be perceived or detected by the driver 130 of the vehicle 110 . On the other hand, a relatively high relevance value r (e.g. r=2) can indicate that the situation element has a relatively high relevance for the driving operation and should therefore or must be perceived or detected by the driver 130 of the vehicle 110 .

wobei K die Anzahl von Situationselementen in der Menge von Situationselementen ist, wobei w_i(n) der individuelle Wahrnehmungswert des Situationselements i an dem Zeitpunkt n ist, und wobei r_i(n) der Relevanzwert des Situationselements i an dem Zeitpunkt n ist. In entsprechender Weise kann der optimale kumulierte Wahrnehmungswert SA_optimal(n) ermittelt werden, als bzw. auf Basis von $${\displaystyle \sum _{i=1}^K{w}_{max,i}}\left(n\right)\cdot r_i\left(n\right),$$

wobei w_max,i(n) der maximal mögliche individuelle Wahrnehmungswert des Situationselements i an dem Zeitpunkt n ist. Typischerweise ist w_max,i(n) = w_max ein einheitlicher (zeitlich konstanter) maximal möglicher individueller Wahrnehmungswert für alle Situationselemente (der z.B. mit dem dritten Wahrnehmungszustand assoziiert ist).The relevance, in particular the relevance value, of the individual situation elements can be taken into account when determining the attention indicator in order to increase the accuracy of the attention indicator. In particular, the actual cumulative perception value SA _actual (n) and the optimal cumulative perception value SA _optimal (n) can be determined taking into account the relevance values of the individual situation elements. The actual cumulative perception value SA _actual (n) can be determined, for example, as or on the basis of $${\displaystyle \sum _{i=1}^K{w}_i}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right),$$

where K is the number of situation items in the set of situation items, where w _i (n) is the individual perception value of situation item i at time n, and where r _i (n) is the relevance value of situation item i at time n. Correspondingly, the optimal cumulative perception value SA _optimal (n) can be determined as or on the basis of $${\displaystyle \sum _{i=1}^K{w}_{max,i}}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right),$$

where w _max,i (n) is the maximum possible individual perception value of situation element i at time n. Typically, w _max,i (n) = w _max is a uniform (constant in time) maximum possible individual perceptual value for all situational elements (e.g., associated with the third perceptual state).

As already explained above, the number of measured values of the viewing direction 131 that cannot be assigned to any situation element from the set of situation elements can also be determined for a current point in time n. The measured values not assigned to any situation element are an indicator of the inattentiveness of the driver 130 and can be taken into account (as a reducing component) when determining the alertness indicator. In particular, the measured values can be clustered without an assigned situation element using a cluster algorithm (such as DBSCAN). A maximum distance value between the measured values of a cluster and/or a minimum value of measured values per cluster can be specified for clustering. A number o of different clusters of measured values without an assigned situation element can then be determined using the cluster algorithm. An increasing number o of different clusters is an indication of increasing inattentiveness on the part of the driver 130. The number o of different clusters can be multiplied by a penalty factor γ and taken into account as a reducing component when determining the attention indicator.

der optimale kumulierte Wahrnehmungswert SA_optimal(n) kann z.B. ermittelt werden, als $$S{A}_{optimal}\left(n\right)={\displaystyle \sum _{i=1}^K{w}_{max,i}}\left(n\right)\cdot r_i\left(n\right),$$

und/oder der Aufmerksamkeitsindikator SA(n) kann z.B. ermittelt werden, als $$SA\left(n\right)=\frac{S{A}_{actual}\left(n\right)}{S{A}_{optimal}\left(n\right)}.$$

The actual cumulative perception value SA _actual (n) can be determined, for example, as $$S{A}_{act\mathrm{and}al}\left(n\right)={\displaystyle \sum _{i=1}^K{w}_i}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right)-g\cdot O,$$

the optimal cumulative perception value SA _optimal (n) can be determined, for example, as $$S{A}_{Optimal}\left(n\right)={\displaystyle \sum _{i=1}^K{w}_{max,i}}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right),$$

and/or the attention indicator SA(n) can be determined, for example, as $$SA\left(n\right)=\frac{S{A}_{act\mathrm{and}al}\left(n\right)}{S{A}_{Optimal}\left(n\right)}.$$

In the above example, the alertness indicator SA(n) is a value between 0 and 1, with 0 indicating a minimum level of alertness and 1 indicating a maximum level of alertness of the driver 130.

As already explained above, states of perception can be determined for the individual situation elements at the current point in time n. The states of perception can be updated iteratively at a sequence of points in time. In particular, the perception states of the signaling elements at the previous point in time n−1 can be updated on the basis of the measured values of the viewing direction 131 recorded between the point in time n−1 and the point in time n.

2 shows an exemplary state diagram or state model 200 for the (iterative) determination of the perception state of a situation element. The state model 200 shows the possible perception states 201, 202, 203, 204 of a situation element at a specific point in time. Furthermore, the state model 200 shows possible state transitions 211, 212, 213, 214, 215, 216 between (directly) consecutive points in time. The state transitions take place depending on the measured values 221 of the viewing direction 131 between the (directly) consecutive points in time, in particular depending on the assignment of the measured values 221 to the viewing area 220 of the situation element.

• • Wahrnehmungszustand 201: Situationselement ist nicht im möglichen Sichtbereich des Fahrers 130;
• • der erste Wahrnehmungszustand 202: Situationselement wurde nicht vom Fahrer 130 erkannt;
• • der zweite Wahrnehmungszustand 203: Situationselement wurde vom Fahrer 130 erkannt; und/oder
• • der dritte Wahrnehmungszustand 204: Situationselement wurde vom Fahrer 130 erfasst.

The possible perceptual states of state diagram 200 are,

• Perception state 201: situation element is not in the possible field of vision of driver 130;
• • the first perception state 202: situation element was not recognized by the driver 130;
• • the second perception state 203: situation element was recognized by the driver 130; and or
• • the third perception state 204: the driver 130 detected the situation element.

In particular on the basis of the surroundings data it can be determined whether the situation element has moved into the possible field of vision of the driver 130 between the previous point in time n−1 and the current point in time n. In this case, there is a state transition 211 from the perception state 201 (“not in the field of vision”) to the first perception state 202 (“not recognized”) (even then) if there is no measured value 221 in the field of vision between the two times n−1, n 220 of the situation element. The situation element can then be included in the set of situation elements for time n.

If a measured value 221 falls between the two points in time n - 1, n (at least or exactly) in the viewing area 220 of the situation element, then a state transition 212 can take place from the first perception state 202 (“not recognized”) to the second perception state 203 (“recognized”). ) take place. If several measured values 221 fall within the field of view 220 of the situation element between the two points in time n−1, n, a state transition 213 can take place from the second perception state 203 (“detected”) to the third perception state 204 (“detected”).

If it is recognized that the situation element leaves the possible field of vision of the driver 130 between the two points in time n−1, n, a state transition 216 can take place from a perception state 202, 203, 204 to the perception state 201 (“not in the field of vision”). The situation element can then be removed from the set of situation elements for time n.

A situation element that is in the possible field of vision of the driver 130 can be held in a heightened state of perception 203, 204 for a certain holding time before the situation element is transferred to a lower state of perception 202, 203 (with a reduced degree of recognition) if during the holding time, at least one measured value 221 does not fall within the field of view 220 of the situation element. In particular, the situation element can be held in the third perception state 204 (“detected”) for a first holding time and transferred to the second perception state 203 (“detected”) (state transition 214) if no measured value 221 is in the field of view within the first holding time 222 of the situation element falls. In a corresponding manner, the situation element can be held in the second perception state 203 (“recognized”) for a second holding time, and transferred to the first perception state 202 (“not recognized”) (state transition 215) if no measured value 221 within the second holding time falls within the field of view 222 of the situation element.

The respective current perception state 201, 202, 203, 204 of a situation element can thus be determined in an iterative and robust manner by means of a state model or by means of a state machine 200.

3 shows a flowchart of an exemplary (possibly computer-implemented) method 300 for determining an attention indicator in relation to the attention of a driver 130 of a motor vehicle 110 to the driving operation or to the monitoring of the driving operation of the vehicle 110. The method 300 can be carried out by a control unit 111 of the Vehicle 110 and / or be executed by a computing unit for remote control of the vehicle 110. The method 300 may be repeatedly performed at a sequence of consecutive points in time.

The method 300 includes determining 301 a set of situation elements that are relevant to the driving operation of the vehicle 110 on the basis of surroundings data (in particular based on camera data) in relation to the surroundings of the vehicle 110 . At least some of the situation elements can each be an object 120, 121 from the area surrounding vehicle 110 (e.g. another road user 120, a traffic sign 121, a traffic light 121, etc.).

Furthermore, the method 300 includes the determination 302, on the basis of measured values 221 in relation to the viewing direction 131 of the driver 130 of the vehicle 110, in each case a perception state 202, 203, 204 from a plurality of different perception states 202, 203, 204 for the one or several situation elements from the set of situation elements. In other words, a perception state 202, 203, 204 can be assigned to the individual situation elements. The different states of perception 202, 203, 204 can indicate different degrees to which the driver 130 has perceived a situation element.

The method 300 also includes determining 303 a value of the attention indicator based on the perception states 202, 203, 204 of the one or more situation elements from the set of situation elements. By determining the states of perception 202, 203, 204 for the situation elements relevant to the driving operation of the vehicle 110, the attentiveness of the driver 130 can be determined in a precise and reliable manner.

A current value SA(n) of an alertness indicator can thus be determined at a point in time n, which indicates the degree of alertness of the driver 130 of a vehicle 110 to the driving operation of the vehicle 110 . The vehicle 110, in particular a driving function of the vehicle 110, can then be operated depending on the value SA(n) of the alertness indicator. Alternatively or additionally, the vehicle 110 can be remotely controlled by a (remotely controlling) driver 130 depending on the value SA(n) of the alertness indicator. A particularly reliable and safe operation of the vehicle 110 can thus be made possible.

As explained above, the situation elements can be detected and/or determined on the basis of data surrounding the vehicle 110 . The one or more situation elements can be arranged outside of the vehicle 110 . Furthermore, one or more situation elements can be taken into account that relate to the (driving) state of the vehicle 110, such as the driving speed of the vehicle 110, the steering angle of the vehicle 110, the acceleration of the vehicle 110, etc Viewing area 220 of a state-related situation element may then be associated with a display (of the state) on a dashboard of the vehicle 110, for example. The quality of the ascertained attention indicator can be further increased by taking one or more state-related situation elements into account.

The present invention is not limited to the exemplary embodiments shown. In particular, it should be noted that the description and the figures are only intended to illustrate the principle of the proposed methods, devices and systems.

Claims (15)

Device (111) for determining an alertness indicator relating to the alertness of a driver (130) of a motor vehicle (110) to a driving operation of the vehicle (110); wherein the device (111) is set up - On the basis of environmental data relating to an environment of the vehicle (110) to determine a set of situation elements that are relevant to the driving operation of the vehicle (110); - On the basis of measured values (221) in relation to a line of sight (131) of the driver (130) of the vehicle (110), one perception state (202, 203, 204) from a plurality of different perception states (202, 203, 204) for determine one or more situation elements from the set of situation elements; wherein the different states of perception (202, 203, 204) indicate different degrees to which the driver (130) perceived an element of the situation; and - to determine a value of the attention indicator based on the perception states (202, 203, 204) of the one or more situation elements from the set of situation elements. Device (111) according to claim 1 , wherein the device (111) is set up to determine the perception state (202, 203, 204) of a situation element, - a number of measured values (221) from a large number of measured values (221) in relation to the viewing direction (131) of the driver (130) to determine which fall within a field of view (220) of the situation element; wherein the viewing area (220) of the situation element indicates viewing directions (131) of the driver (130) in which the driver's (130) view is directed to the situation element; - select the perception state (202, 203, 204) of the situation element from the plurality of different perception states (202, 203, 204) depending on the determined number of measured values (221); wherein the plurality of different perception states (202, 203, 204) comprises in particular - a first perception state (202) which indicates that the driver (130) has not recognized the situation element; - A second perception state (203) which indicates that the driver (130) recognized the situation element, in particular recognized it unconsciously; and/or - a third perception state (204) which indicates that the driver (130) perceived the situation element, in particular consciously perceived it. Device (111) according to claim 2 , wherein the device (111) is set up - to assign the first perception state (202) to the situation element if it is determined that no measured value (221) falls within the field of view (220) of the situation element; - to assign the second perception state (203) to the situation element if it is determined that at least one, in particular exactly one, Measured value (221) falls within the field of view (220) of the situation element; and/or - to assign the third perception state (204) to the situation element if it is determined that several measured values (221), in particular at least a predefined minimum number of measured values (221), fall within the field of vision (220) of the situation element. Device (111) according to any one of the preceding claims, wherein - the plurality of different perceptual states (202, 203, 204) being associated with a corresponding plurality of different individual perceptual values; and - the device (111) is set up to determine the value of the attention indicator on the basis of the individual perception values for the one or more situation elements from the set of situation elements. Device (111) according to one of the preceding claims, wherein the device (111) is set up - to determine a relevance value for the one or more situation elements from the set of situation elements, which indicates a relevance of the respective situation element for the driving operation of the vehicle (110); and - determine the value of the attention indicator based on the relevance values for the one or more situation elements from the set of situation elements. Device (111) according to one of the preceding claims, wherein the device (111) is set up - to determine a subset of measured values (221) from a plurality of measured values (221) in relation to the viewing direction (131) of the driver (130) which cannot be assigned to any of the situation elements from the set of situation elements; and - to determine the value of the attention indicator on the basis of the determined subset of measured values (221). Device (111) according to claim 6 , wherein the device (111) is set up - to cluster the measured values (221) from the determined subset of measured values (221) that cannot be assigned to any situation element, using a cluster algorithm; and - to determine the value of the attention indicator on the basis of a number of determined clusters. Device (111) according to one of the preceding claims, wherein the device (111) is set up - to determine an actual cumulative perception value on the basis of the perception state (202, 203, 204) of the one or more situation elements from the set of situation elements; - assuming that the one or more situation elements from the set of situation elements each have an optimal perception state (204), to determine an optimal cumulative perception value; and - determine the value of the attention indicator on the basis of the actual cumulative perception value and on the basis of the optimal cumulative perception value, in particular on the basis of a ratio of the actual cumulative perception value and the optimal cumulative perception value. Device (111) according to claim 8 , wherein the device (111) is set up to determine the actual cumulative perception value on the basis of $${\displaystyle \sum _{i=1}^K{w}_i}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right)$$

where - K is a number of situation items in the set of situation items; - w _i (n) is an individual perception value dependent on the perception state (202, 203, 204) of situation element i; and - r _i (n) is a relevance value of the situation element i, which indicates the relevance of the situation element i for the driving operation of the vehicle (110). Device (111) according to any one of Claims 8 until 9 , wherein the device (111) is set up to determine the optimal cumulative perception value on the basis of $${\displaystyle \sum _{i=1}^K{w}_{max,i}}\left(n\right)\cdot {\mathrm{right}}_i\left(n\right)$$

where - K is a number of situation items in the set of situation items; - w _max,i (n) is a maximum possible individual perception value associated with the optimal perception state of the situation element i; and - r _i (n) is a relevance value of the situation element i, which indicates the relevance of the situation element i for the driving operation of the vehicle (110). Device (111) according to one of the preceding claims, wherein the device (111) is set up, in an iterative manner, starting from the perception state (202, 203, 204) of the one or more situation elements from the set of situation elements at a previous current point in time, on the basis of a large number of measured values (221) in relation to the viewing direction (131) of the driver (130) between the previous point in time and a current point in time, the perception state (202, 203, 204) of the one or more situation elements from the Set of situation elements to determine at the current point in time. Device (111) according to any one of the preceding claims, wherein - the device (111) is set up to update the perception state (202, 203, 204) for a situation element by means of a state model (200) in an iterative manner at a sequence of consecutive points in time; - the state model (200) comprises the plurality of different perception states (201, 202, 203, 204); - the state model (200) comprises state transitions (211, 212, 213, 214, 215, 216) between at least some of the perceptual states (203, 204, 205); and - At least some of the state transitions (211, 212, 213, 214, 215, 216) depend on the measured values (221) in relation to the viewing direction (131) of the driver (130) for a point in time from the sequence of points in time. Device (111) according to one of the preceding claims, wherein the set of situation elements comprises one or more situation elements for one or more objects (120, 121) in the area surrounding the vehicle (110) which are relevant to the driving operation of the vehicle (110). . Device (111) according to one of the preceding claims, wherein the device (111) is set up to operate the vehicle (110) and/or a remote control of the vehicle (110) depending on the determined value of the alertness indicator. Method (300) for determining an alertness indicator relating to the alertness of a driver (130) of a motor vehicle (110) to a driving operation of the vehicle (110); the method (300) comprising - Determining (301), on the basis of environmental data relating to an environment of the vehicle (110), a set of situation elements that are relevant to the driving operation of the vehicle (110); - Determination (302), on the basis of measured values (221) in relation to a viewing direction (131) of the driver (130) of the vehicle (110), in each case a perception state (202, 203, 204) from a plurality of different perception states (202 , 203, 204) for one or more situation elements from the set of situation elements; and - determining (303) a value of the attention indicator based on the perception states (202, 203, 204) of the one or more situation elements from the set of situation elements.

Images