EP1915874A2

EP1915874A2 - Method and circuit arrangement for recognising and tracking eyes of several observers in real time

Info

Publication number: EP1915874A2
Application number: EP06791307A
Authority: EP
Inventors: Alexander Schwerdtner
Original assignee: SeeReal Technologies GmbH
Current assignee: SeeReal Technologies GmbH
Priority date: 2005-08-17
Filing date: 2006-08-16
Publication date: 2008-04-30
Also published as: RU2008110044A; JP5054008B2; JP2009505247A; BRPI0616547A2; RU2408162C2; WO2007019842A3; KR20080047392A; DE112006002752A5; KR101278430B1; CA2619155A1; US7950802B2; CN101243693A; CN101243693B; US20080231805A1; WO2007019842A2

Abstract

The invention relates to a method and to a circuit arrangement for recognising and for tracking, in a contact-free manner, eye positions of several users in real time. The input data comprises a sequence of digital video frames. Said method comprises the following steps: combining a face-finder-instance which is used to examine faces, an eye-finder-instance which is used to examine eye areas, and an eye-tracker-instance which is used to recognise and track eye reference points. The aim of the invention is to convert the eye positions within a hierarchical outlet of the instance to the target, which successively restricts the dataset, which is to be processed, emerging from the dataset of the entire video frame (VF) in order to form a face target area (GZ) and subsequently an eye target area (AZ). Also, an instance or a group of instances, which run in a parallel manner, are carried out, respectively, on a calculating unit thereof.

Description

Method and circuit arrangement for recognizing and tracking eyes of several observers in real time

Field of the invention

The invention relates to a method and a circuit arrangement for a contact-free detection and tracking of eye positions or pupils of several observers in real-time mode.

The input data includes imagery as a sequence of digital video frames acquired by one or more image sensors.

It can be determined reference points of the eyes of multiple observers, without further aids, such as glasses, helmets or spots are required.

In contrast to stationary applications, for example, the monitoring of drivers and pilots, where the range of motion is particularly limited in depth and thus almost stationary, the invention serves to detect the eye positions in a large target area, allows rapid movement of the viewer and determines the coordinate the depth in a large area of for example 0.5 to 3.5 meters.

The efficient and accurate realization of eye recognition in real time is an important interface between man and machine. An important field of application of the invention resides in a device for recognizing and tracking the eye positions of observers of autostereoscopic displays. Such displays provide the viewer with a stereoscopic image impression without the need for aids such as polarization glasses. Other applications of the invention include, for example, the video holography and implementations in the field of person, face or gaze direction detection. Autostereoscopic displays, in which the display is tracked by a so-called tracking device, provide multiple viewers a large freedom of movement in a large viewer area. The error-free detection and tracking of eyes, eye positions or pupils is also an essential interface between human and machine in these areas of image presentation.

A reliable and error-free tracking device is usually not perceived by a viewer. In many applications, however, errors of the tracking system lead to undesirable side effects, which lead to poor reproduction or crosstalk, for example in the area of the 3D representation. A tracking device requires high accuracy, reliability and accuracy. The system must also be sufficiently efficient and accurate to track the significant movements, allowing the viewer maximum freedom of movement in all three spatial directions.

State of the art

Contactless tracking systems are commercially available in different variants. Simple versions usually offer a basic application software for standard operating systems and have hardware and software standardized interfaces.

The document WO 03/079902 A1, "Real-time eye-detection and tracking under various light conditions", Zhiwei Zhu Qiang Ji describes a method for non-contact detection of eyes in real time, which essentially comprises a step for eye detection and a Includes eye tracking step. Eye detection includes a combination of the active illumination method and pattern recognition. After the eyes of a viewer are first recognized, the eyes are followed, this step involving the combination and synthesis of several algorithms and techniques. Despite combination and Synthesis of various means remains the problem that larger and faster head movements in all three coordinate directions can not be tracked in real time and that the delay between delivery of the position data and image acquisition can prevent real-time processing. This concerns in particular the determination of the eye position in depth in unfavorable environmental conditions.

For example, in a vehicle, the driver's face is always within a predictable range of the dashboard. In addition, even small changes occur in vertical and horizontal directions.

In particular, the real range of motion in the depth is very small, so that usually when using a single camera, the depth position can be extrapolated with sufficient accuracy.

It is the task of the present invention in all three dimensions of a viewer space to provide a large range of motion with low computation times. In contrast to the cited prior art, it is necessary to capture the eyes in all three dimensions, including the depth. The depth should preferably cover a wide range from 0.5 to at least 3.5 meters. To determine the depth, on the one hand, several separately arranged cameras are necessary in order to be able to generate images from different directions from the target area. In addition, the detection of the eyes at a distance of up to several meters requires a very high resolution of the cameras, resulting in a large amount of data per camera and per video frame.

The problem of processing a large amount of data in real time is exacerbated when it comes to recognize multiple viewers. In particular, very compute-intensive process steps are required in order to be able to determine obscure observers by means of lighting effects, reflections or spectacle lenses. Experience shows that often the detection of a third or fourth person who is partially hidden or out of the way can only be achieved with a high, time-consuming computational effort. The required computational effort for the viewer, which is currently the worst detectable and which only with great effort, but must not affect the pursuit in real time of other viewers.

Problems with detecting eye positions mean that the incoming video frames can no longer be processed permanently in real-time mode. A maximum allowable calculation time per person and per frame can be exceeded if, for example, lenses or eyeglasses hinder the eyes or a viewer turns away quickly but only briefly from the cameras.

In view of the disadvantages of the prior art, the invention has the object to provide a method which allows to determine the eye positions of several observers, even with larger and abrupt head movements in all three coordinate directions in real time. The method is intended to detect the detection of the eye positions in a large target area, to compensate for rapid movements of the observer and to determine the coordinate of the depth in a large area. With simultaneous minimization of the error, moreover, the response time between the video recording, ie the reading of a video frame and the result delivery, ie the provision of the eye positions, should be sustainably reduced. The method should also allow for high-resolution cameras error-free results in real-time mode can be achieved.

Summary of the invention

The method is used to detect and track reference points of multiple viewer's eyes in real time. The input data includes image data as a sequence of digital video frames acquired by one or more image sensors, such as cameras. The reference points of the eyes are the positions of the pupils and / or the corner of the eye. The method comprises the interaction of a face finder instance for finding faces, subsequently and hierarchically subordinate an eye finder instance for finding areas of the eyes, and an eye tracker Instance used to detect and track eye points. The Eye Tracker instance is hierarchically subordinate to the Eye Finder instance.

The invention is based on the idea that the position determination of the eyes is implemented within a hierarchical sequence with the goal that

Search range starting from an entire video image successively restrict. The real-time behavior is realized by the hierarchical successive restriction and nesting of the search area from the complete video frame for the face finder instance to the restricted face target area for the eye finder or eye tracker instance. Furthermore, an instance or a group of instances is executed in parallel on a separate computing unit within separate processes.

The Face Finder instance searches the head or face position for each viewer in the area of an entire video frame. For each face, the instance determines a significantly smaller amount of data from the data of the entire video frame, representing the corresponding face-target area, and passes that restricted area to the Eye-Finder instance.

The eye finder instance is hierarchically subordinate to the face finder instance and only needs to process a very limited amount of data from the data of the passed face target area. The instance determines the eyes or eye positions in this data area and, in turn, defines a much smaller data volume of the face / target area than the eye / target area, whereby this limited search area is then transferred to a subsequent and hierarchically subordinate eye tracker instance.

Thereafter, the Eye Tracker Instance determines the sought reference points of the eyes in this highly constrained amount of data of the eye search area at an increased speed. With the hierarchical restriction of search scopes and the reduction in the amount of data, the Eye Tracker instance is highly effective and fast. According to the invention, in order to reduce the total delay time of the method, the instances face-finder and eye-finder / eye tracker should each be executed in parallel independently of each other within separate processes. The parallelization by assigning an instance or a group of instances to its own computing units can be implemented in several variants.

In a particularly preferred embodiment of the invention, a face-Finder instance is performed on a separate computing unit for each camera. Subsequently, each observer, who finds a face finder instance, is assigned an own arithmetic unit for the realization of an eye finder and subsequently an eye tracker instance. If a newly detected face is determined by a face finder instance, an instance of the eye finder and the eye tracker is immediately commissioned or initialized and these instances are executed on their own assigned arithmetic unit. Even for briefly lost and rediscovered faces an immediate tracking is delegated after detection of the face.

A significant advantage of the invention is that a face-Finder instance, since now the subordinate instances are executed on their own arithmetic units, is in no way blocked or obstructed. The Face Finder instance continues to search for faces in the data of the current video frame, while preserving the computational resources. Determined intermediate and partial results are transferred to a control entity for further processing / distribution, or they are taken over by partial results of the eye tracker / eye finder instances in order to be able to extrapolate the facial target areas in a positive control loop.

The immediate realization of the instances shortens the response time of the process and provides the first basis for real-time behavior. The real-time behavior is provided by the hierarchical successive restriction and nesting of the search area from the complete video frame for the Underpinned Face Finder instance to the restricted face target area for the Eye Finder or Eye Tracker instance.

Finally, real-time behavior is further underpinned and secured by the implementation of an instance or a group of instances in parallel within separate processes on a separate computing unit. For the parallelism of the instances further variants are conceivable. As mentioned above, a face finder instance and an eye finder / eye tracker instance can each be executed on a separate arithmetic unit. Furthermore, in each case a face finder / eye finder instance and an eye tracker instance can be executed on a separate arithmetic unit. An implementation of the Eye Finder instance on its own arithmetic unit also seems conceivable. However, this is an instance which requires a comparatively short computing time, so that it is advantageously allocated to a computing unit of the two computing intensive face finders or eye tracker instances.

Preferably, both the flow of the instances and their data exchange is controlled and monitored by a control entity. In particular, this instance controls the assignment of the found faces, or face target areas, to the eye finder / eye tracker instances on the individual arithmetic units. The data exchange essentially comprises the

Initialization of the instances by assigning the search areas, the exchange of partial and final results of the instances and the transfer of the resulting reference points for the eyes to an external interface. For example, the control instance updates and re-initializes the associated instances of the Eye Finder and the Eye Tracker for an already tracked face. The tax authority selects, verifies and evaluates the confidence of the found face and eye target areas. Corresponding evaluation parameters are determined by the instances in the course of the procedure and serve the control entity also for an optimal execution coordination of the instances and as well as an allocation of the existing calculation units.

The method according to the invention allows the eye positions of several observers even with larger and abrupt head movements in all three To determine coordinate directions in real time. In addition, it could be substantiated that the method can also achieve results in the real-time mode for the data volume of high-resolution camera systems.

Brief description of the figures

The following figures illustrate embodiments of the method according to the invention as part of a tracking device for an autostereoscopic display and show in Fig. 1 is a schematic representation of the nested, restricted

Search areas of the instances Face-Finder, Eye-Finder and Eye-Tracker,

2 shows a flowchart of the parallelization of the hierarchically structured instances of the method according to the invention, as well

Fig. 3 is a schematic representation of the circuit arrangement and a flowchart of the parallelization of the hierarchically structured instances of the method according to the invention.

Preferred embodiments of the invention

Fig. 1 shows the interleaved, restricted search areas of the instances of the method. As input data, image material is acquired as a sequence of digital video frames VF from a plurality of image sensors, for example a stereo infrared camera. A section of the entire video frame VF is shown schematically in the figure by the coordinate system. A first face finder instance analyzes the data of the entire video frame VF and recognizes the faces of the viewers throughout the video frame. In the figure, the data of two faces are shown. The first face on the left is obviously close to the camera, while the second right one has a higher distance to the camera.

For each detected face, the face finder instance determines from the data of the entire video frame VF a limited data area of the facial Target area GZ corresponds. The indices refer to the first face shown in the figure on the left. The determined face target area GZ now represents the restricted search area for subsequent Eye Finder instance. In this search area, the Eye Finder instance determines the eye positions and, as a result, restricts the data volume of the target area GZ to a much smaller amount of data Eye Target Range AZ equals, one. The data of the eye target area AZ with the eye positions are the input data for a subsequent eye tracker instance ET, which is now in the eye target area AZ for the current video frame and in the subsequent video frames according to the already determined motion sequence in the guided eye target area AZ finally determined reference points for the eyes as a result.

With the information of the reference points of the past video frames, according to the movement of the observer, the eye target area AZ is tracked, updated and the areas for the current and the coming frames are extrapolated. If the observer moves into the depth, a scaling of the image content may additionally be necessary.

As can be seen from the figure, the eye-target area can disintegrate into several non-contiguous subregions. As also shown in the figure, the target areas are irregular, but preferably convex, depending on the observer's head position and viewing direction. In a simple embodiment, the regions are represented by a list of parameterized geometric surfaces, such as ellipses, circles, or rectangles.

Fig. 2 builds on the last embodiment and shows a flowchart of the parallelization of the instances. The figure describes the hierarchical structuring of the instances of face finder FF, eye finder and eye tracker ET and the assignment to own calculation units R1 to R2.

In this embodiment, three arithmetic units R1 to R3 are available. A first arithmetic unit R1 is provided for the face finder instance FF. This finds the face of a first observer in the data of a video frame and thereby determines the facial target area GZ. The facial Ziθlbereich is immediately assigned its own arithmetic unit R1 for performing an eye finder EF and subsequently an eye tracker instance ET.

The figure shows the data flow of the data of the restricted target areas, ie facial target area GZ and eye target area AZ to the respective subsequent instance. An eye tracker instance ET supplies the data of the reference points of eyes to a superordinate control instance (not shown) or to an external interface. At the same time, the information of the reference points determined in the past video frames is used to track the eye target area AZ during a movement of the observer and to extrapolate for the coming frames. The data of the current eye target area as well as the areas of past frames are therefore returned to the Eye Tracker instance as shown.

The realization of the recognition and tracking of the second viewer is analogous. If there are more observers than arithmetic units, an eye-finder eye-tracker instance is preferably realized analogously for each observer, that is to say a face-target area, as independent processes running in parallel, in which case several processes naturally run on one common arithmetic unit.

3 shows a circuit arrangement and a flowchart of the parallelization of the hierarchically structured instances a parallelization of the method based on the image data of several cameras different positions For detecting and tracking the eyes, the cameras are each based on a method analogous to the above examples. A camera is thus associated with a parallelization of the instances analogous to FIGS. 1 and 2.

The left system determines from the left image data VFL (Video Frame Left) by a face finder instance FF on a first of the arithmetic unit R1 the face target area GZ1-L of the first observer. The associated Eye-Finder EF / Eye Tracker ET instances are executed on the arithmetic unit R2. For the circuit arrangement, these arithmetic units are usually implemented as CPUs or DSPs. A second group of instances on the arithmetic unit R3 is assigned to a second observer. The remaining instances and arithmetic units shown in the figure refer to the right and the associated instances or elements of the circuit arrangement, characterized by VFR (Video Frame Right) and the index "R".

One and possibly also implemented control unit, which is not shown in the figure, assumes in the process the task of controlling the individual processes and controls the data exchange. The data exchange takes place in particular within those arithmetic units which are assigned to a viewer. For example, one uses the already available information in the left, in the right, whose content is not significantly different from the left, to determine the position in the right image with a certain tolerance and to extrapolate in knowledge of. From the xy pixel position of the eye in the left, the distance of the observer, which was determined from the previous depth calculation, and the camera parameters is a

Transformation of partial results possible. For example, the data of an eye area AZ1-L found in the left field are set as input variables for the right field AZ1-R, possibly transformed, whereby it is now possible to use other algorithms or other controlling parameters than for the left-hand process.

Finally, from the and for this calculation, essentially include the resolution cameras, their pixel pitch, the focal length of the lens, the image distance of the lens to the camera, as well as the distance and orientation of the cameras. The circuit arrangement essentially comprises communicating, programmable logic modules, processors, ROMs and RAMs. Preferably, the arithmetic units are optimized and configured exclusively for the intended task, in particular for the named instances. In a further preferred embodiment, the circuit arrangement also contains independent arithmetic units for performing auxiliary processes, such as scaling, gamma correction or the like.

Claims

claims

A method for recognizing and tracking reference point positions (EP1, .., EPn) of eyes of several observers in image data of video frames (VF) acquired by at least one image sensor, characterized by

Face Finder Instance (FF) for recognizing the positions of faces in

Video frames (VF) which extracts for each face a substantially smaller portion of the entire video frame than face target area (GZ), to provide information to the face target area (GZ) at least one subsequent one

- To forward eye-Finder entity (EF) for recognizing eye positions, which extracts from each face-target area (GZ) a further substantially smaller portion of the face-target area as the eye-target area (AZ) to information about the eye-target area (AZ ) to a subsequent one

- To pass eye tracker instance (ET) for tracking the eye positions, which now define in the eye-target area (AZ) for the current video frame and in the subsequent video frames by reference point positions (EP1, .., EPn) of the eyes and information for generate a tracking device, wherein an instance or a group of instances is performed in parallel on an associated computing unit in parallel.

2. The method of claim 1, wherein the coordinates of the reference point positions (EP1, .., EPn) are horizontal and vertical position.

3. The method of claim 2, wherein starting from two-dimensional reference point positions (EP1, .., EPn) of a plurality of image sensors, the coordinates of the depth of the eyes are determined.

4. The method according to claim 3, wherein in each case a face finder instance and a combined eye finder / eye tracker instance or in each case a combined face Finder / Eye Finder instance and an Eye Tracker instance on its own arithmetic unit.

5. The method of claim 3, wherein for each detected face on at least one assigned arithmetic unit an instance of the eye finder and the eye

Trackers initialized and executed.

6. The method of claim 3, wherein the viewers and / or the resulting target positions of the eyes is assigned a scoring order, by means of which the arithmetic units are assigned.

7. The method of claim 3, wherein one or more Face Finder instances are permanently executed and reinitialize the respective associated Eye Finder / Eye Tracker instances.

8. The method according to claim 1, comprising a control entity which detects whether a face target area recognized by the face finder instance results from an already tracked or from a newly recognized face, and which the eye tracker or eye finder / Assigns eye tracker instances to the existing arithmetic units, initializes them and synchronizes the execution of all instances.

9. The method according to one or more of the preceding claims wherein image data from a plurality of image sensors are acquired and per viewer and image sensor performed an eyefinder instance, an eye tracker instance or a combined eye-finder / eye tracker instance on its own arithmetic unit is, between the observers associated instances, the resulting partial and / or final results are exchanged and processed.

10. The method of claim 9, wherein the knowledge of the position of the image sensors and the viewer and parameters of the image sensors, the partial and / or final results of the instances of an image sensor for the instances of another image sensor is transformed and adapted to its perspective.

11. The method of claim 1, wherein the reference point positions (EP1, ..., EPn) of the eyes are the position of the pupils and / or the corner of the eye.

12. Circuit arrangement for detecting and tracking

Reference point positions (EP1, .., EPn) of eyes of several observers in image data of video frames (VF) with several communicating arithmetic units (RI ₁ ..., Rn) with a face finder instance (FF), an eye finder instance (EF) and an Eye Tracker Instance (ET), where the - Face Finder Instance (FF) is used to detect the positions of faces in

Videoframes (VF) is used and extracts for each face a much smaller portion of the entire video frame than the face target area (GZ) to provide information to the face target area (GZ) to at least one subsequent eye finder (EF) for recognizing Forward eye positions, which extracts from each face-target area (GZ) a further substantially smaller portion of the face-target area as the eye-target area (AZ), information to the eye-target area (AZ) to a subsequent - Eye Tracker instance (ET ) for tracking the eye positions which now define in the eye target area (AZ) for the current video frame and in the subsequent video frames reference point positions (EP1, .., EPn) of the eyes and generate information for a tracking device, wherein an instance or a Group of instances is carried out in parallel each running on an associated arithmetic unit.

13. Circuit arrangement according to claim 12, with its own computing means for the normalization or correction of the image data for scaling, gamma correction, brightness control or the like.

14. Circuit arrangement according to claim 12, with computing means which calculates the coordinates of the depth from the determined by at least two image sensors reference points (EP1, .., EPn).

15. Circuit arrangement according to claim 12, wherein a computing unit is a CPU or DSP or the like.