DE102011011221A1 - Method for recognition of attention of driver of motor car and production of activity signal for adaptive cruise control stop and go driver assistance system, involves producing activity signals from determined driver attention - Google Patents

Abstract

The method involves receiving a driver image and determination of existences of a driver based on the driver image, and checking whether the driver image concerns an actual driver or an attempt of deception. Face features of the driver are extracted from the driver image if reception and checking steps are positive. Driver attention is determined from the extracted features. Activity signals are produced from the determined driver attention. A front image of the driver as an initial driver image and the extracted features are memorized for comparative purposes at start of trip. An independent claim is also included for a driver assistance system for performing a method for recognition of attention of a driver of a motor car and production of an activity signal.

Description

The invention relates to a method for detecting the attention of the driver of a motor vehicle and for generating a signal for a driver assistance system according to the preamble of claim 1, as well as a use of the method in a driver assistance system according to the preamble of claim 11.

Previous algorithms for detecting driver attention, such as regarding head rotation and eyelid closure, operate essentially based on models. There are great problems with eyeglass wearers and beards, who speak so far against introducing such algorithms in the vehicle. In addition, these algorithms do not provide for live or deceptive detection, which is also referred to as fake detection, which is necessary for an autonomous assistance system, since otherwise abuse and thus impairment of functional safety can not be ruled out.

The publication DE 10 2007 023 140 A1 shows a method for detecting and identifying the driver of a motor vehicle, wherein by means of a camera system at least one movement and / or at least one movement sequence of the driver is recorded and analyzed for recognition and identification of the driver by an image processing unit. In this case, the extracted movement or the motion sequence is compared with an already stored. If there is a match within predetermined limits then the driver is identified and stored settings for that person can be made. Therefore, a biometric identification method is used.

Furthermore, from the DE 10 2008 031 697 A1 a device for object identification, in particular for biometric identification, which comprises an image acquisition unit for detecting a region in which the object to be identified is located, and an evaluation unit for processing the image data provided by the image acquisition unit. In a first operating mode, the device can be controlled in such a way that the image data provided to the evaluation unit each have a low contrast difference in a first image area and in a second image area and the first and the second image area have a large difference in contrast to one another. As a result, the first image area is lifted off from the second image area, forming a contour. In the first operating mode, the evaluation unit is set up to analyze the contour and, based on its progression, to conclude the presence of the expected object or the presence of an image of the object. It is therefore checked whether it is a real object in the detection range of the image acquisition unit or a deception attempt, for example by means of a photograph.

The invention is therefore based on the object, an improved method for detecting the driver activity and its use in a driver assistance system, such as an ACC system with Stop & Go functionality (ACC: Adaptive Cruise Control or automatic distance control) to specify.

This object is achieved by a method having the features of claim 1 and by a driver assistance system having the features of claim 11. Preferred embodiments of the invention are subject of the dependent claims.

• a) Aufnehmen eines Fahrerbildes und Bestimmen des Vorhandensein des Fahrers anhand des Fahrerbildes,
• b) Prüfen ob es sich um einen realen Fahrer oder um einen Täuschungsversuch handelt,
• c) Extrahieren von Gesichtsmerkmalen des Fahrers aus dem Fahrerbild, falls die Schritte a) und b) positiv sind,
• d) Bestimmen der Fahreraufmerksamkeit aus den extrahierten Merkmalen, und
• e) Erzeugen eines Aktivitätssignal aus der in Schritt d) bestimmten Fahreraufmerksamkeit.

The inventive method for detecting the attention of the driver of a motor vehicle and for generating an activity signal for a driver assistance system performs the following steps by means of a biometric facial recognition by a facial biometric camera system:

• a) recording a driver image and determining the presence of the driver based on the driver image,
• b) Checking whether it is a real driver or an attempted deception,
• c) extracting facial features of the driver from the driver image if steps a) and b) are positive,
• d) determining driver attention from the extracted features, and
• e) generating an activity signal from the driver attention determined in step d).

Preferably, a frontal image of the driver as an initial driver image and its extracted features are stored at the beginning of the journey for comparison purposes, in order to be able to set the method to a defined initial state. This process preferably takes place automatically and is initiated, for example, with the unlocking of the vehicle, since the process of unlocking causes at least a partial power supply of the vehicle and an activation or initiation of predetermined driver assistance systems.

Preferably, step a), i. H. the taking of a driver image and the determination of the presence of a driver, performed by a determination of the driver's facial contour in the recorded driver image.

Further preferred is step b) of the method by a live recognition, a Deception detection and driver identification performed.

In particular, the live recognition can be carried out by a continuous check of intrinsic movements of the driver's head and / or the driver's face. Intrinsic movements are, for example, head and / or eyelid movements over a given period of time.

The deception detection can be done by checking the distance between the driver's head and camera, the use of pulsed near-infrared illumination and / or the brightness distribution in the recognized driver's face. The results of the detection of deception can also at least partially be incorporated into the live recognition.

More preferably, the driver identification is based on the similarity value of the initially stored driver image and the current driver image, which is also referred to as a match score.

Preferably, the facial features head rotation, head position, X and Y coordinates of the driver's eyes, and the eye opening degree are extracted, the enumeration being incomplete. Other suitable features such as head angle or mouth features may be used.

To determine the driver's activity, a biometric comparison of the extracted current facial features with the stored facial features of the initial driver's image is preferably performed.

Preferably, the driver is estimated to be active and an activity signal is generated when the biometric comparison gives a frontal head direction and an eye opening degree corresponding to open eyes.

• i) der Fahrersitz ist belegt sowie Tür und Gurtschloss geschlossen,
• ii) der Lenkradwinkel liegt zwischen +/–60 Grad,
• iii) die Gesichtskontur im Fahrerbild ist erkannt,
• iv) das Fahrergesicht ist ”lebendig”,
• v) der Fahrerkopf ist für eine vorgegebene Zeit t1 frontal ausgerichtet, und
• vi) die Augen des Fahrers sind für eine vorgegebene Zeit t2 offen.

The driver assistance system according to the invention, which uses the method explained above, comprises a facial biometric camera system which observes the driver in a traffic jam during the service life of the motor vehicle and generates a driver image, and a control device for carrying out the biometric face recognition. The driver assistance system then initiates a driver-side start-up release if the following conditions are met:

• i) the driver's seat is occupied and the door and buckle are closed,
• ii) the steering wheel angle is between +/- 60 degrees,
• iii) the facial contour in the driver's image is recognized,
• iv) the driver's face is "alive",
• v) the driver head is frontally aligned for a predetermined time t1, and
• vi) the driver's eyes are open for a predetermined time t2.

As a driver assistance system, ACC driver assistance systems with stop-and-go functionality or all systems for autonomous or automated driving support are considered, which require information about the driver's state when standing or driving.

Further preferably, a drive release for the motor vehicle takes place when, in addition to the driver-side release, a release of the drive takes place by further vehicle sensors. In other words, there is a sensor fusion of the driver-side release with results of other environment sensors such as radar, laser and / or outdoor camera.

The described signals of the facial biometric camera system are therefore used in addition to the classic driver personalization primarily for input signals of driver assistance systems, for example in an ACC driver assistance system. The additional advantage is the multiple use of an algorithm on a control unit, so that accounts for more software variants.

Furthermore, now the Anfahrfreigabe, which is currently after a short life, for example, in a traffic jam, manually by an operation of a lever on the steering column or the accelerator pedal, by the biometric face recognition, which additionally takes place with the algorithm so-called detection of life and deception, for example to detect the reading of a newspaper in a traffic jam, in which case no start-up clearance occurs and thus the functional safety is increased.

A preferred embodiment of the invention will be explained below with reference to the drawings. It shows

1 the components of an ACC Stop & Go driver assistance system,

2 a biometric recognition algorithm in a schematic representation, and

3 a flowchart of the driver side release within an ACC Stop & Go driver assistance system.

1 shows the involved components of an exemplary ACC driver assistance system with Stop & Go functionality, which is extended by a camera-based, biometric driver observation. This is the manual effort by the driver be extended to greater automation and ride comfort. The driver no longer has to perform any manual action with the facial biometric camera system in traffic jam, because the system recognizes whether the driver is watching the traffic or not. An algorithm of biometric driver recognition is used.

The components of the ACC driver assistance system with stop-and-go functionality involved in a start-up decision are the HMI 1 (HMI: Human machine interface) with the components accelerator pedal 1a , Steering column 1b and instrument cluster 1c , the drive 2 with the gear components 2a , Delay 2 B and acceleration 2c , the environment sensor 3 with the components video camera 3a for example in the windscreen, radar 3b for the long range and ultrasonic sensor 3c for the close range, the biometric recognition system 4 as well as the starting decision 7 ,

The biometric recognition system 4 includes the two areas driver parameters 5 as well as the driver observation 6 , Under driver parameters 5 are understood here by the driver caused or influenced parameters or states, in particular the seat occupancy 5a with the values "occupied, free", the condition buckle 5b with the values "closed or open", the state of the door opening 5c with the values "open or closed" and the steering wheel angle 5d , In the field of driver observation 6 is with a camera 6a a video image taken and in an image processing 6b processed. The extracted or calculated signals 6c become an interpretation 6d fed.

From the corresponding signals of the participating components HMI 1 , Drive 2 , Environment sensor 3 , Driver parameters 5 and driver observation 6 will then be in the starting decision 7 made a decision.

• – der Fahrersitz ist belegt, Tür und Gurtschloss sind geschlossen,
• – der Lenkradwinkel liegt zwischen –/+60 Grad,
• – das GBKS erkennt Gesichtskontur des Fahrers im Videobild,
• – das GBKS erkennt, dass das Gesicht des Fahrers ”lebendig” ist,
• – das GBKS erkennt, dass der Kopf des Fahrers für eine vorgegebene Zeit t1 frontal ausgerichtet ist, und
• – das GBKS erkennt, dass die Augen des Fahrers für eine vorgegebene Zeit t2 offen sind.

The facial biometric camera system, also abbreviated as GBKS, the driver observation 6 , consisting of the components monocamera, two near-infrared lighting modules and a control unit with the biometric algorithm explained below, observes the driver during the life of the vehicle. The vehicle only decides on automatic release if the following factors are fulfilled:

• - the driver's seat is occupied, door and buckle are closed,
• - the steering wheel angle is between - / + 60 degrees,
• - the GBKS recognizes the driver's facial contour in the video image,
• - the GBKS recognizes that the driver's face is "alive",
• - the GBKS recognizes that the head of the driver is frontal for a given time t1, and
• - the GBKS recognizes that the driver's eyes are open for a given time t2.

If all parameters are met, the face biometric camera system initiates a start-up release for the "driver" subsystem. A final decision for ACC Stop & Go will be made by merging with other vehicle sensors (radar, outdoor camera, etc.).

2 shows a schematic representation of an algorithm of biometric facial recognition. By means of an interior camera 11 a video image of the driver is generated and sent to a control device 12 transfer. In the control device 12 becomes a face search 13 carried out. The image and the determined facial position becomes an eye finder 14 fed, which determines the coordinates of the driver's eyes. The obtained data, ie the current image, the face position and the coordinates of the eyes become an image quality inspection 15 subjected. Normalization of the face occurs in the next block and the normalized face becomes preprocessing 17 subjected. Subsequently, the desired or necessary features are extracted from the preprocessed face of the driver and in a pattern creator 10 to a pattern vector, the so-called reference vector summarized.

• – der Match Score, d. h. der Ähnlichkeitswert zwischen einem hinterlegten Frontalbild
• – des Fahrers und dem aktuellen Gesichtbild des Fahrers, dem sogenannten Livebild,
• – die X- und Y-Koordinaten der detektierten Augen des Fahrers aus dem aktuellen Bild
• – der aktuelle Augenöffnungsgrad in Prozent, und
• – die geschätzte Kopfposition aus den geometrischen Verhältnissen im aktuellen Fahrergesicht.

The values of the algorithm used here are:

• - The match score, ie the similarity value between a stored frontal image
• The driver and the current face of the driver, the so-called live image,
• The X and Y coordinates of the driver's detected eyes from the current image
• - the current eye opening degree in percent, and
• - The estimated head position from the geometric relationships in the current driver's face.

The X and Y coordinates of the driver's eyes are related, for example, to the respective center of the eye or corners of the eye in order to obtain a consistent coordinate definition.

• – der Matchscore ist abhängig von der Kopfrotation,
• – Ist der Kopf frontal, d. h. die Kopfrotation ist 0 Grad, dann ist der Match Score maximal (100%), d. h. das aktuelle Bild ist zum gespeicherten initialen Bild absolut ähnlich,
• – dreht der Fahrer den Kopf, dann sinkt der Match Score,
• – ab etwa 30 Grad hat der Match Score üblicherweise den Wert Null, mit anderen Worten bis zu diesem Winkel ist eine direkte Winkel- und sektorenabhängige Erkennung möglich,
• – Erkennung der Rotationsrichtung (links/rechts) erfolgt über Symmetrie- und Richtungsberechnung der x- und y-Augenpunkte.

To recognize the above-mentioned head rotation, the following conditions apply:

• - the match score depends on the head rotation,
• - If the head is head-on, ie the head rotation is 0 degrees, then the match score is maximum (100%), ie the current image is absolutely similar to the stored initial image,
• - turns the driver's head, then the score decreases,
• - from about 30 degrees, the match score usually has the value zero, in other words up to this angle a direct angle and sector-dependent detection is possible,
• - Detection of the direction of rotation (left / right) takes place via symmetry and direction calculation of the x and y eye points.

The value of 30 degrees does not mean the limit of the functionality of the system, but the value only signals that the driver no longer looks head-on. A recognition of the driver still works on the value by means of the consideration of the history and a prognosis is possible beyond the range.

3 shows a flowchart for determining the driver side driving a driver assistance system, in particular an ACC system with stop & go functionality. At the beginning, ie after the entry of the driver, is in the step 20 a frontal image of the driver recorded and stored, which is referred to as the initial driver image. Furthermore, features of the current driver are extracted from the initial image and also stored.

In the next step 21 the necessary conditions are checked, namely whether the driver is seated in the vehicle, the doors are closed, the seat belt (s) are fastened, the seat is occupied, the ACC Stop & Go is activated and the vehicle is standing still in a traffic jam.

After the requirements of the step 21 are met in step 22 a current driver image, the so-called live image, taken. After that, in step 23 examines whether a facial contour is present in the live image. If this is not the case, which is marked with "N", then the system goes back to the step 22 back. If a facial contour has been detected then it will be in step 24 investigates if there is a deception. If this matches what is marked "J", the system goes back to step 22 , If there is no deception attempt, which is marked with "N", takes place in the step 25 a feature extraction from the current driver image.

In step 26 a biometric comparison is made between those in the step 25 determined features with the characteristics of the initial image of the step 20 for determining the head direction. If this comparison is smaller than a predetermined limit, then in step 27 concluded that the head direction is not frontal (output "<") and the system goes back to the step 22 , In the step to be reached with the output "≥" 28 is closed to a frontal head direction, which means that the comparison is greater than or equal to a predetermined limit.

Subsequently, in the next step 29 the eye opening degree checked. If this is smaller than a predetermined limit, then in step 30 the driver's eyes are considered closed and the system goes back to step 22 , If the eye opening degree is greater than or equal to said predetermined limit value, the driver's eyes are opened in step 31 closed. As the driver has a frontal head position and open eyes, is in step 32 the driver-side driving permission granted.

The step mentioned in the flowchart 23 and 24 make a distinction between living objects, namely the driver himself and non-living objects, such as persons on newspapers, photos, dolls, etc., through. This distinction is imperative. In addition, it must be distinguished whether it is the current driver or, for example, the passenger.

• – das gesichtsbiometrische Kamerasystem erwartet den Kopf des Fahrers in einer gewissen Entfernung vor der Kamera, dem sog. Erwartungsraum, womit einfache Foto-Täuschungen ermittelt werden können,
• – der Einsatz von gepulster Nah-Infrarot-Beleuchtung führt zur Aufdeckung von Video-Täuschungen,
• – es wird eine laufende Prüfung intrinsische Bewegungen (Kopf- und Lidbewegungen über einen Zeitraum), was zur Detektion von Abdeckung von Foto- und Puppen-Täuschungen führt,
• – mittels der Überprüfung von Helligkeitsverteilungen im erkannten Gesicht durch Abgleich zwischen bekannten Verläufen sowie Abgleich zwischen Sonnensensorrichtung und Helligkeitsverteilung im Gesicht werden Foto- und Videotäuschungen ermittelt, und
• – eine Unterscheidung zwischen Fahrer und Beifahrer erfolgt über das initial gespeichertes Frontalbild: beugt sich beispielsweise der Beifahrer in die Fahrerzone, dann würde das System einen deutlich geringeren Match Score berechnen und keine Freigabe erteilen, wodurch eine Abdeckung einer absichtlichen Fehlbedienung erreicht wird.

The deceptive and live detection of the preferred embodiment operates as follows:

• The facial biometric camera system expects the head of the driver at a certain distance in front of the camera, the so-called expectation space, with which simple photo illusions can be determined
• - the use of pulsed near-infrared lighting leads to the detection of video illusions,
• It is an ongoing examination intrinsic movements (head and eyelid movements over a period of time), which leads to the detection of cover of photo and doll delusions,
• - By checking brightness distributions in the recognized face by matching between known gradients and balance between the sun sensor direction and brightness distribution in the face photo and video illusions are determined, and
• - A distinction between the driver and front passenger over the initially stored frontal image: bends, for example, the passenger in the driver's area, then the system would charge a significantly lower match score and grant no clearance, which covers a deliberate misuse is achieved.

A distinction between driver and front passenger can also be made via the different head rotation by means of the evaluation of the X and Y eye points.

LIST OF REFERENCE NUMBERS

1

HMI (human-machine interface

1a

accelerator pedal

1b

pitman

1c

instrument cluster

2

drive

2a

corridor

2 B

delay

2c

acceleration

3

environment sensors

3a

Video camera windscreen

3b

Radar (far range)

3c

Ultrasound (close range)

4

Biometric recognition system

5

driver parameters

6

driver observation

7

Anfahrentscheidung

11

Interior camera

12

control unit

13

face search

14

eye finder

15

Image quality test

16

face normalization

17

preparation

18

feature extraction

19

patterning

20

Storage of frontal image and feature extraction of current drivers

21

requirements

22

Live image

23

Facial contour available

24

attempt to deceive

 25

Feature extraction from live image

26

Biometric comparison

27

Head direction not frontal

28

Head direction frontal

29

Eye opening degree

30

eyes closed

31

Eyes open

32

release

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102007023140 A1 [0003]
• DE 102008031697 A1 [0004]

Claims (12)

A method for detecting the attention of the driver of a motor vehicle and for generating an activity signal for a driver assistance system, wherein the following steps are performed by means of a biometric facial recognition by a facial biometric camera system: a) recording a driver image and determining the presence of the driver based on the driver image, b) Checking whether it is a real driver or an attempted deception, c) extracting facial features of the driver from the driver image if steps a) and b) are positive, d) determining driver attention from the extracted features, and e) generating an activity signal from the driver attention determined in step d). A method according to claim 1, characterized in that at the beginning of a frontal image of the driver as the initial driver image and its extracted features are stored for comparison purposes. A method according to claim 1 or 2, characterized in that step a) is performed by determining the facial contour of the driver in the recorded driver image. Method according to one of the preceding claims, characterized in that step b) is performed by a live detection, a deception detection and a driver identification. A method according to claim 4, characterized in that the live detection is carried out by a continuous review of intrinsic movements of the driver's head and / or the driver's face. A method according to claim 4, characterized in that the detection of deception by checking the distance between the driver's head and the camera, the use of pulsed near-infrared illumination and / or the brightness distribution in the recognized driver's face. A method according to claim 4, characterized in that the driver identification is based on the similarity value of an initially stored driver image and the current driver image. Method according to one of the preceding claims, characterized in that the facial features head rotation, head position, X and Y coordinates of the driver's eyes and the eye opening degree are extracted. A method according to claim 8, characterized in that for determining the driver activity, a biometric comparison of the extracted current facial features with the stored facial features of the initial driver image is performed. A method according to claim 9, characterized in that the driver is considered to be active and an activity signal is generated when the biometric comparison results in a head-on-head direction and an eye-opening degree for open eyes. Driver assistance system using the method according to one of the preceding claims with a facial biometric camera system, which observes the driver at least during the service life of the motor vehicle and generates a driver image, and a control unit for performing the biometric facial recognition, wherein a driver-side Anfahrfreigabe occurs when the following conditions are met : i) the driver's seat is occupied and the door and buckle are closed, ii) steering wheel angle between +/- 60 degrees, iii) facial contour in the driver's image is detected iv) driver's face is "alive", v) driver's head is frontal for a given time t, and vi) eyes of the driver are open for a given time t. Driver assistance system according to claim 11, characterized in that a drive release for the motor vehicle takes place when in addition to the driver-side release of a release of the drive is carried out by other vehicle sensors.

Images