DE102011011221B4 - Method for detecting the attention of the driver of a motor vehicle - Google Patents

Abstract

Method for detecting the attention of the driver of a motor vehicle and for generating an activity signal for a driver assistance system, the following steps being carried out by means of biometric facial recognition by a facial biometric camera system: a) recording a driver image and determining the presence of the driver based on the driver image, b) Checking whether it is a real driver or an attempt at deception (24), c) extracting facial features of the driver from the driver image if steps a) and b) are positive, d) determining the driver's attention from the extracted features, ande ) Generating an activity signal from the driver attention determined in step d).

Description

The invention relates to a method for detecting the attention of the driver of a motor vehicle and for generating a signal for a driver assistance system according to the preamble of claim 1, as well as a use of the method in a driver assistance system according to the preamble of claim 11.

Previous algorithms for detecting driver attention, such as by looking at head rotation and eyelid closure, essentially work based on models. There are major problems with people who wear glasses and beards, who have so far argued against the introduction of such algorithms in vehicles. In addition, these algorithms do not guarantee live or deceptive detection, which is also referred to as fake detection, which is necessary for an autonomous assistance system, otherwise misuse and thus an impairment of functional safety cannot be ruled out.

The publication DE 10 2007 023 140 A1 shows a method for recognizing and identifying the driver of a motor vehicle, wherein at least one movement and / or at least one movement sequence of the driver is recorded by means of a camera system and analyzed by an image processing unit to recognize and identify the driver. The extracted movement or sequence of movements is compared with one that has already been saved. If there is a match within specified limits, the driver is then identified and settings saved for this person can be made. A biometric identification process is therefore used.

Furthermore, from the DE 10 2008 031 697 A1 a device for object identification, in particular for biometric identification, is known, which comprises an image capture unit for capturing an area in which the object to be identified is located, and an evaluation unit for processing the image data provided by the image capture unit. The device can be controlled in a first operating mode in such a way that the image data provided to the evaluation unit each have a small contrast difference in a first image area and in a second image area and the first and second image areas have a large contrast difference from one another. As a result, the first image area stands out from the second image area, forming a contour. In the first operating mode, the evaluation unit is set up to analyze the contour and, based on its course, infer the presence of the expected object or the presence of an image of the object. It is therefore checked whether it is a real object in the detection range of the image capture unit or an attempt to deceive, for example by means of photography.

The publication DE 100 24 227 A1 relates to methods and devices for electronically monitoring the level of attention of a motor vehicle driver through electronic monitoring of the eyes or eye area. In order to ensure that traffic situations caused by inattention and not just by drowsiness remain controllable, especially if the vehicle is moving in a column that moves at short distances from the vehicle in front, the speed is automatically reduced if fatigue is registered accordingly.

The publication DE 103 60 176 A1 relates to a motor vehicle with a sensor for driver monitoring, in which the sensor is mounted in the motor vehicle in a position and is suitably aligned so that the driver of the motor vehicle can be detected as unhindered as possible.

The publication DE 199 36 271 A1 relates to a device for user-specific vehicle release. A first biometric data acquisition captures user-specific biometric information as reference biometric information. The reference biometric information is transmitted via a transmission link to a receiver on the vehicle and stored in a memory. A second biometric data acquisition arranged in the vehicle records user-specific biometric information as actual biometric information. A control arrangement compares the actual biometric information with the reference biometric information and emits a release signal to an operationally necessary component if the reference biometric information matches the actual biometric information.

The publication DE 196 21 435 A1 relates to a system for monitoring a person's eyes, comprising a human interface for obtaining images of the person and providing feedback to the person, and a processor for analyzing the images. The processor converts a sequence of images into a one-dimensional signal by extracting relevant features from the images. The generated signal is then analyzed to record sleepiness. A transformation of the image sequence includes eye localization, eye tracking and the generation of an eye movement signal. The system uses the relatively high horizontal contrast density of the eye region for the determination the eye positions in a grayscale or grayscale image of a human face.

The publication DE 10 2004 044 772 A1 relates to a method for building a database for use for image-based driver identification in a motor vehicle, comprising recording and storing a plurality of two-dimensional facial images of a known driver. The current recording angle between the camera and the driver's facial normal is determined by a device that is not otherwise used for image-based driver identification and passed on to the method for building the database, and the recording of a facial image is triggered when the current recording angle reaches a certain default value.

The publication DE 10 2004 005 163 B3 relates to a device for detecting the driving ability of a driver in a vehicle, which has a lighting device for illuminating at least one eye of the driver, an image recording device for recording images of the illuminated eye, an evaluation device which is used to evaluate the images recorded by the image recording device, and one Has data storage. The lighting device illuminates at least one eye of the driver in a flash-like or intermittent manner, the evaluation device comparing measured values of the driver's pupil reaction recorded by means of the image recording device with at least one normal value of a pupil reaction stored in the data memory and, if the value falls below the normal value, the measured values of the pupil reaction are applied to a control device in this way has the effect of preventing the vehicle from being put into operation or from continuing to drive after the vehicle has stopped in the operating state.

The publication DE 10 2008 031 697 A1 relates to a device for object identification, in particular for biometric identification, which comprises an image capture unit for capturing an area in which the object to be identified is located, and an evaluation unit for processing the image data provided by the image capture unit. The device can be controlled in a first operating mode in such a way that the image data provided to the evaluation unit each have a small contrast difference in a first image area and in a second image area and the first and second image areas have a large contrast difference from one another. The first image area stands out from the second image area to form a contour. In the first operating mode, the evaluation unit is set up to analyze the contour and, based on its course, determine the presence of the expected object or the presence of an image of the object.

The invention is therefore based on the object of specifying an improved method for recognizing driver activity and its use in a driver assistance system, for example an ACC system with Stop & Go functionality (ACC: Adaptive Cruise Control or automatic distance control).

This object is achieved by a method with the features of claim 1 and by a driver assistance system with the features of claim 11. Preferred embodiments of the invention are the subject of the subclaims.

1. a) Aufnehmen eines Fahrerbildes und Bestimmen des Vorhandensein des Fahrers anhand des Fahrerbildes,
2. b) Prüfen ob es sich um einen realen Fahrer oder um einen Täuschungsversuch handelt,
3. c) Extrahieren von Gesichtsmerkmalen des Fahrers aus dem Fahrerbild, falls die Schritte a) und b) positiv sind,
4. d) Bestimmen der Fahreraufmerksamkeit aus den extrahierten Merkmalen, und
5. e) Erzeugen eines Aktivitätssignal aus der in Schritt d) bestimmten Fahreraufmerksamkeit.

The method according to the invention for detecting the attention of the driver of a motor vehicle and for generating an activity signal for a driver assistance system carries out the following steps by means of biometric facial recognition by a facial biometric camera system:

1. a) taking a driver image and determining the presence of the driver based on the driver image,
2. b) Check whether it is a real driver or an attempt to deceive,
3. c) extracting facial features of the driver from the driver image if steps a) and b) are positive,
4. d) determining driver attention from the extracted features, and
5. e) Generating an activity signal from the driver attention determined in step d).

Preferably, at the start of the journey, a frontal image of the driver is stored as the initial driver image and its extracted features for comparison purposes in order to be able to set up the method on a defined initial state. This process preferably takes place automatically and is initiated, for example, when the vehicle is unlocked, since the process of unlocking causes at least a partial power supply to the vehicle and an activation or initiation of predetermined driver assistance systems.

Preferably, step a), i.e. recording a driver image and determining the presence of a driver, is carried out by determining the facial contour of the driver in the recorded driver image.

Step b) of the method is further preferred through a live detection, a deception Connection detection and driver identification are carried out.

In particular, the live detection can be carried out by continuously checking intrinsic movements of the driver's head and/or the driver's face. Intrinsic movements are, for example, head and/or eyelid movements over a given period of time.

Deception detection can be carried out by checking the distance between the driver's head and the camera, the use of pulsed near-infrared lighting and/or the brightness distribution in the detected driver's face. The results of deception detection can also be at least partially incorporated into liveness detection.

More preferably, driver identification is carried out based on the similarity value of the initially stored driver image and the current driver image, which is also referred to as a match score.

The facial features head rotation, head position, X and Y coordinates of the driver's eyes and the degree of eye opening are preferably extracted, although the list is not exhaustive. Other suitable features such as head tilt angle or mouth features can be used.

To determine the driver activity, a biometric comparison of the extracted current facial features is preferably carried out with the stored facial features of the initial driver image.

Preferably, the driver is assessed as active and an activity signal is generated if the biometric comparison results in a frontal head direction and a degree of eye opening that corresponds to open eyes.

• i) der Fahrersitz ist belegt sowie Tür und Gurtschloss geschlossen,
• ii) der Lenkradwinkel liegt zwischen +/- 60 Grad,
• iii) die Gesichtskontur im Fahrerbild ist erkannt,
• iv) das Fahrergesicht ist „lebendig“,
• v) der Fahrerkopf ist für eine vorgegebene Zeit t1 frontal ausgerichtet, und
• vi) die Augen des Fahrers sind für eine vorgegebene Zeit t2 offen.

The driver assistance system according to the invention, which uses the method explained above, comprises a facial biometric camera system, which observes the driver, in particular while the motor vehicle is stationary in a traffic jam, and generates a driver image, and a control device for carrying out the biometric facial recognition. The driver assistance system then releases the driver's approach if the following conditions are met:

• i) the driver's seat is occupied and the door and belt buckle are closed,
• ii) the steering wheel angle is between +/- 60 degrees,
• iii) the facial contour in the driver's image is recognized,
• iv) the driver's face is "alive",
• v) the driver's head is oriented frontally for a predetermined time t1, and
• vi) the driver's eyes are open for a given time t2.

The driver assistance systems that can be considered are ACC driver assistance systems with stop & go functionality or all systems for autonomous or automated driving support that require information about the driver's condition when stationary or moving.

Further preferably, a driving release for the motor vehicle occurs when, in addition to the driver's release, the drive is released by further vehicle sensors. In other words, there is a sensor fusion of the driver's release with results from other environmental sensors such as radar, laser and/or external camera.

In addition to classic driver personalization, the described signals from the facial biometric camera system are therefore primarily used for input signals from driver assistance systems, for example in an ACC driver assistance system. The additional advantage is that an algorithm can be used multiple times on one control unit, meaning that additional software variants are no longer necessary.

Furthermore, the start-up release, which is currently after a short standstill, for example in a traffic jam, is now carried out manually by operating a lever on the steering column or the accelerator pedal, through biometric facial recognition, with the algorithm also being used for so-called liveness and deception detection, for example to detect the reading of a newspaper in a traffic jam, in which case there is no start-up clearance and thus functional safety is increased.

• 1 die Komponenten eines ACC-Stop&Go-Fahrerassistenzsystems,
• 2 ein biometrischer Erkennungsalgorithmus in schematischer Darstellung, und
• 3 ein Ablaufdiagramm der fahrerseitigen Freigabe innerhalb eines ACC-Stop&Go-Fahrerassistenzsystems.

A preferred embodiment of the invention is explained below with reference to the drawings. This shows

• 1 the components of an ACC Stop&Go driver assistance system,
• 2 a biometric recognition algorithm in a schematic representation, and
• 3 a flowchart of the driver's release within an ACC Stop&Go driver assistance system.

1 shows the components involved in an exemplary ACC driver assistance system with Stop&Go functionality, which is expanded to include camera-based, biometric driver monitoring. The manual effort should be reduced by the Drivers can be expanded towards greater automation and driving comfort. With the facial biometric camera system, the driver no longer has to carry out any manual action in the event of a traffic jam, as the system recognizes whether the driver is looking at the traffic or not. A biometric driver recognition algorithm is used.

The components of the ACC driver assistance system with Stop&Go functionality involved in making a starting decision are the HMI 1 (HMI: human-machine interface) with the components accelerator pedal 1a, steering column 1b and instrument cluster 1c, the drive 2 with the components gear 2a, deceleration 2b and acceleration 2c, the environment sensor system 3 with the components video camera 3a, for example in the windshield, radar 3b for the long range and ultrasonic sensor 3c for the close range, the biometric recognition system 4 and the approach decision 7.

The biometric recognition system 4 includes the two areas of driver parameters 5 and driver observation 6. Driver parameters 5 here are understood to mean parameters or states caused or influenced by the driver, in particular the seat occupancy 5a with the values “occupied, free”, and the belt buckle state 5b the values “closed or open”, the state of the door opening 5c with the values “open or closed” and the steering wheel angle 5d. In the driver observation area 6, a video image is recorded with a camera 6a and processed in an image processing unit 6b. The extracted or calculated signals 6c are fed to an interpretation 6d.

A decision is then made in the approach decision 7 from the corresponding signals from the involved components HMI 1, drive 2, environment sensors 3, driver parameters 5 and driver observation 6.

• - der Fahrersitz ist belegt, Tür und Gurtschloss sind geschlossen,
• - der Lenkradwinkel liegt zwischen -/+60 Grad,
• - das GBKS erkennt Gesichtskontur des Fahrers im Videobild,
• - das GBKS erkennt, dass das Gesicht des Fahrers „lebendig“ ist,
• - das GBKS erkennt, dass der Kopf des Fahrers für eine vorgegebene Zeit t1 frontal ausgerichtet ist, und
• - das GBKS erkennt, dass die Augen des Fahrers für eine vorgegebene Zeit t2 offen sind.

The facial biometric camera system, also abbreviated as GBKS, of the driver observation 6, consisting of the components mono camera, two near-infrared lighting modules and a control unit with the biometric algorithm explained below, observes the driver while the vehicle is stationary. The vehicle will only decide on automatic release if the following factors are met:

• - the driver's seat is occupied, the door and seatbelt buckle are closed,
• - the steering wheel angle is between -/+60 degrees,
• - the GBKS recognizes the driver's facial contour in the video image,
• - the GBKS recognizes that the driver's face is “alive”,
• - the GBKS detects that the driver's head is oriented frontally for a predetermined time t1, and
• - the GBKS recognizes that the driver's eyes are open for a specified time t2.

If all parameters are met, the facial biometric camera system releases the “driver” subsystem to start. A final decision for ACC Stop&Go is made via the merger with other vehicle sensors (radar, external camera, etc.).

2 shows a schematic representation of an algorithm for biometric facial recognition. Using an interior camera 11, a video image of the driver is generated and transmitted to a control device 12. A face search 13 is carried out in the control device 12. The image and the determined facial position are fed to an eye finder 14, which determines the coordinates of the driver's eyes. The determined data, ie the current image, the face position and the coordinates of the eyes, are subjected to an image quality check 15. The face is normalized in the next block and the normalized face is subjected to preprocessing 17. The desired or necessary features are then extracted from the driver's preprocessed face and combined in a pattern generator 10 to form a pattern vector, the so-called reference vector.

• - der Match Score, d.h. der Ähnlichkeitswert zwischen einem hinterlegten Frontalbild des Fahrers und dem aktuellen Gesichtbild des Fahrers, dem sogenannten Livebild,
• - die X- und Y-Koordinaten der detektierten Augen des Fahrers aus dem aktuellen Bild
• - der aktuelle Augenöffnungsgrad in Prozent, und
• - die geschätzte Kopfposition aus den geometrischen Verhältnissen im aktuellen Fahrergesicht.

The values of the algorithm used here are:

• - the match score, ie the similarity value between a stored frontal image of the driver and the current facial image of the driver, the so-called live image,
• - the X and Y coordinates of the driver's detected eyes from the current image
• - the current degree of eye opening in percent, and
• - the estimated head position from the geometric relationships in the current driver's face.

The X and Y coordinates of the driver's eyes are based, for example, on the respective eye center or eye corners in order to obtain a consistent coordinate definition.

• - der Matchscore ist abhängig von der Kopfrotation,
• - Ist der Kopf frontal, d.h. die Kopfrotation ist 0 Grad, dann ist der Match Score maximal (100%), d.h. das aktuelle Bild ist zum gespeicherten initialen Bild absolut ähnlich,
• - dreht der Fahrer den Kopf, dann sinkt der Match Score,
• - ab etwa 30 Grad hat der Match Score üblicherweise den Wert Null, mit anderen Worten bis zu diesem Winkel ist eine direkte winkel- und sektorenabhängige Erkennung möglich,
• - Erkennung der Rotationsrichtung (links/rechts) erfolgt über Symmetrie- und Richtungsberechnung der x- und y-Augenpunkte.

To detect the above head rotation, the following conditions apply:

• - the match score depends on the head rotation,
• - If the head is frontal, i.e. the head rotation is 0 degrees, then the match score is maximum (100%), i.e. the current image is absolutely similar to the saved initial image,
• - the driver turns his head, then the match score drops,
• - from around 30 degrees the match score usually has the value zero, in other words up to this angle direct angle and sector-dependent detection is possible,
• - Detection of the direction of rotation (left/right) is carried out by calculating the symmetry and direction of the x and y eye points.

The value of 30 degrees does not mean the limit of the system's functionality, but rather the value only signals that the driver is no longer looking straight ahead. The driver can still be recognized based on the value by looking at the history and a forecast is possible beyond the range.

3 shows a flowchart for determining the driver's permission to drive a driver assistance system, in particular an ACC system with stop & go functionality. At the beginning, i.e. after the driver gets in, a frontal image of the driver is recorded and saved in step 20, which is referred to as the initial driver image. Furthermore, characteristics of the current driver are extracted from the initial image and also saved.

In the next step 21, necessary conditions are checked, namely whether the driver is sitting in the vehicle, the doors are closed, the seat belt(s) are fastened, the seat is occupied, the ACC Stop&Go is activated and the vehicle is standing still in a traffic jam.

After the requirements of step 21 have been met, a current driver image, the so-called live image, is recorded in step 22. Then in step 23 it is examined whether a facial contour is present in the live image. If this is not the case, which is marked with “N”, the system goes back to step 22. If a facial contour has been recognized, it is examined in step 24 whether there is an illusion. If this is true, which is marked with “Y”, the system goes back to step 22. If there is no attempt to deceive, which is marked with “N”, then in step 25 a feature extraction from the current driver image takes place.

In step 26, a biometric comparison is made between the features determined in step 25 with the features of the initial image from step 20 to determine the head direction. If this comparison turns out to be smaller than a predetermined limit value, it is concluded in step 27 that the head direction is not frontal (output “<”) and the system goes back to step 22. In step 28, which can be reached with the output “≥”. closed to a frontal head direction, which means that the comparison was greater than or equal to a specified limit value.

The degree of eye opening is then checked in the next step 29. If this is smaller than a predetermined limit value, the driver's eyes are considered closed in step 30 and the system goes back to step 22. If the degree of eye opening is greater than or equal to the stated predetermined limit value, the driver's eyes are opened in step 31 closed. Since the driver has a frontal head position and eyes open, the driver's permission to drive is given in step 32.

Steps 23 and 24 mentioned in the flowchart make a distinction between living objects, namely the driver himself, and non-living objects, such as people in newspapers, photos, dolls, etc. This distinction is absolutely necessary. In addition, a distinction must be made as to whether it is the current driver or, for example, the passenger.

• - das gesichtsbiometrische Kamerasystem erwartet den Kopf des Fahrers in einer gewissen Entfernung vor der Kamera , dem sog. Erwartungsraum, womit einfache Foto-Täuschungen ermittelt werden können,
• - der Einsatz von gepulster Nah-Infrarot-Beleuchtung führt zur Aufdeckung von Video-Täuschungen,
• - es wird eine laufende Prüfung intrinsische Bewegungen (Kopf- und Lidbewegungen über einen Zeitraum), was zur Detektion von Abdeckung von Foto- und Puppen-Täuschungen führt,
• - mittels der Überprüfung von Helligkeitsverteilungen im erkannten Gesicht durch Abgleich zwischen bekannten Verläufen sowie Abgleich zwischen Sonnensensorrichtung und Helligkeitsverteilung im Gesicht werden Foto- und Videotäuschungen ermittelt, und
• - eine Unterscheidung zwischen Fahrer und Beifahrer erfolgt über das initial gespeichertes Frontalbild: beugt sich beispielsweise der Beifahrer in die Fahrerzone, dann würde das System einen deutlich geringeren Match Score berechnen und keine Freigabe erteilen, wodurch eine Abdeckung einer absichtlichen Fehlbedienung erreicht wird.

The deception and liveness detection of the preferred embodiment works as follows:

• - the facial biometric camera system expects the driver's head at a certain distance in front of the camera, the so-called expectation space, with which simple photo illusions can be determined,
• - the use of pulsed near-infrared lighting leads to the detection of video deceptions,
• - there will be an ongoing test of intrinsic movements (head and eyelid movements over time), leading to the detection of cover photo and doll illusions,
• - photo and video illusions are determined by checking brightness distributions in the detected face by comparing known curves as well as comparing the direction of the sun sensor and the brightness distribution in the face, and
• - A distinction between driver and front passenger is made via the initially saved frontal image: if, for example, the front passenger leans into the driver's zone, then the system would calculate a significantly lower match score and would not issue a release, thereby covering intentional incorrect operation.

A distinction between driver and front passenger can also be made via the different head rotation by evaluating the X and Y eye points.

Reference symbol list

1

HMI (Human-Machine Interface

1a

Accelerator pedal

1b

steering column

1c

Instrument cluster

2

drive

2a

corridor

2 B

delay

2c

acceleration

3

Environmental sensors

3a

Video camera front window

3b

Radar (long range)

3c

Ultrasound (close range)

4

Biometric recognition system

5

Driver parameters

5a

Seat occupancy

5b

Condition of the belt buckle

5c

State of door opening

5d

Steering wheel angle

6

Driver observation

6a

camera

6b

Image processing

6c

extracted signals

6d

interpretation

7

Approach decision

11

Interior camera

12

Control unit

13

Face search

14

Eyefinder

15

Image quality check

16

Facial normalization

17

Preparation

18

Feature extraction

19

Pattern formation

20

Storage of frontal image and feature extraction of current drivers

21

requirements

22

Live image

23

Facial contour present

24

Attempted deception

25

Feature extraction from live image

26

Biometric comparison

27

Head direction not frontal

28

Head direction frontal

29

degree of eye opening

30

eyes closed

31

Eyes open

32

Release

Claims (12)

Method for detecting the attention of the driver of a motor vehicle and for generating an activity signal for a driver assistance system, the following steps being carried out by means of biometric facial recognition by a facial biometric camera system: a) taking a driver image and determining the presence of the driver based on the driver image, b) Check whether it is a real driver or an attempt to deceive (24), c) extracting facial features of the driver from the driver image if steps a) and b) are positive, d) determining driver attention from the extracted features, and e) Generating an activity signal from the driver attention determined in step d). Procedure according to Claim 1 , characterized in that at the start of the journey a frontal image of the driver is saved (20) as the initial driver image and its extracted features for comparison purposes. Procedure according to Claim 1 or 2 , characterized in that step a) is carried out by determining the facial contour of the driver in the recorded driver image. Method according to one of the preceding claims, characterized in that step b) is carried out by live detection, deception detection and driver identification. Procedure according to Claim 4 , characterized in that the live detection is carried out by an ongoing intrinsic check Movements of the driver's head and/or the driver's face occur. Procedure according to Claim 4 , characterized in that the deception detection is carried out by checking the distance between the driver's head and the camera (6a, 11), the use of pulsed near-infrared lighting and / or the brightness distribution in the detected driver's face. Procedure according to Claim 4 , characterized in that the driver identification takes place based on the similarity value of an initially stored driver image and the current driver image. Method according to one of the preceding claims, characterized in that the facial features head rotation, head position, X and Y coordinates of the driver's eyes and the degree of eye opening (29) are extracted. Procedure according to Claim 8 , characterized in that to determine the driver activity, a biometric comparison (26) of the extracted current facial features is carried out with the stored facial features of the initial driver image. Procedure according to Claim 9 , characterized in that the driver is assessed as active and an activity signal is generated if the biometric comparison results in a frontal head direction (28) and an eye opening degree (29) for open eyes (31). Driver assistance system using the method according to one of the preceding claims with a facial biometric camera system (4), which observes the driver at least while the motor vehicle is stationary and generates a driver image, and a control device (12) for carrying out the biometric facial recognition, wherein a driver-side approach release ( 32) occurs when the following conditions are met: i) the driver's seat is occupied (5a) and the door and belt buckle (5b, 5c) are closed, ii) steering wheel angle (5d) between +/- 60 degrees, iii) facial contour in the driver's image is recognized (23), iv) Driver face is “alive”, v) Driver's head is oriented frontally for a predetermined time t (28), and vi) The driver's eyes are open for a predetermined time t (31). Driver assistance system Claim 11 , characterized in that a driving release (7) for the motor vehicle occurs when, in addition to the driver-side release (32), the drive is released by other vehicle sensors.

Images