WO2014169441A1

WO2014169441A1 - Method and system for eye tracking using combination of detection and motion estimation

Info

Publication number: WO2014169441A1
Application number: PCT/CN2013/074273
Authority: WO
Inventors: Wenjuan Song; Wei Zhou; Jianping Song
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2013-04-16
Filing date: 2013-04-16
Publication date: 2014-10-23
Anticipated expiration: 2015-10-16

Abstract

The present invention relates to a method for detecting eye positions. A method comprises receiving image data of a frame, determining at least one identified block in the frame, checking whether there is information on the identified block, if there is the information, calculating face position on the basis of the information using face detection algorithm, detecting eye positions using eye detection algorithm.

Description

METHOD AND SYSTEM FOR EYE TRACKING USING COMBINATION OF DETECTION AND MOTION ESTIMATION

FIELD OF THE INVENTION

The present invention relates to a method and a system for eye tracking using a combination of detection and motion estimation. More particularly, the present invention relates to eye tracking using face detection and eye detection combined with motion estimation.

BACKGROUND OF THE INVENTION

Eye tracking is the process of measuring either the point of gaze ("where we are looking") or the motion of eyes relative to the head of a user. An eye tracker is a device or system for measuring eye positions and eye movements. The eye tracker is used in research on the visual system, in psychology, in cognitive linguistics, and in product design. There are a number of methods for measuring eye movement. In this invention, we implement an eye tracker by detecting image data. The detection is divided into steps. The first step is detecting a face. Then the next step is performing eye detection on the basis of the result of the face detection as input and obtaining the eye's position. Both face detection and eye detection algorithms can use Haar features and Adaboost. Haar features are simples features based on basic computation on sub image areas. Adaboost is a simple and iterative algorithm that helps building an accurate "strong" detector by combining layers of "weak" detectors in a cascade in an initial learning step. Although Haar features and Adaboost have a much higher detected rate and faster detection result than most other detection algorithms, computation is very expensive. There are multiple scales searching, multiple location searching, and expensive image processing. That is too expensive to be suitable for mobile devices which have limited resources .

Motion estimation is the process of determining motion vectors that describe the transformation from one 2D image to another, usually from adjacent frames in a video sequence. It is an ill-posed problem as the motion is in three dimensions but the images are a projection of the 3D scene onto a 2D plane. The motion vectors may relate to the whole image (global motion estimation) or specific parts, such as rectangular blocks, arbitrary shaped patches or even per pixel. The motion vectors may be represented by a translational model or many other models that can approximate the motion of a real video camera, such as rotation and translation in all three dimensions and zoom. A Block Matching Algorithm (BMA) is a way of locating matching blocks in a sequence of digital video frames for the purpose of motion estimation. The purpose of a block Matching Algorithm is to find a matching block from a frame i in some other frame j. The Block Matching Algorithm makes use of criteria to determine whether a given block in frame j matches the search block in frame .

This invention combines detection algorithms with motion estimation algorithms to implement a one eye tracker system and makes the eye tracker system run on one limited resource (CPU, memory ...) device.

As related art, US7708407B2 discloses a method and device for eye tracking compensation to reduce motion blur arising from eye tracking characteristics. SUMMARY OF THE INVENTION

According to the present invention, when the next frame data arrives, face detection can be skipped; and the forecasted face position using motion estimation is directly used to detect eye position.

According to an aspect of the present invention, there is provided a method for detecting eye positions, comprising the steps of: receiving image data of a frame;

determining at least one identified block in the frame; checking whether there is information on the identified block; if there is the information, calculating face position on the basis of the information using move estimation algorithm; and detecting eye positions using eye detection algorithm.

According to another aspect of the present invention, there is provided a system for detecting eye positions, comprising: means for receiving image data of a frame; means for determining at least one identified block in the frame; means for checking whether there is information on the identified block; means for

calculating face position on the basis of the information using move estimation algorithm; means for detecting eye positions using eye detection algorithm.

BRIEF DESCRIPTION OF DRAWINGS

These and other aspects, features and advantages of the present invention will become apparent from the following description in connection with the accompanying drawings in which:

Figure 1A is an exemplary diagram illustrating a basic environment according to one embodiment of the present invention;

Figure IB is an exemplary diagram illustrating a detection result identified area according to one embodiment of the present invention;

Figure 2 is an exemplary diagram illustrating an eye detection process;

Figure 3 is an exemplary flow chart illustrating an eye tracker system work process;

Figure 4 is an exemplary block diagram illustrating a system according to an embodiment of the present invention .

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In the following description, various aspects of an embodiment of the present invention will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding. However, it will also be apparent to one skilled in the art that the present invention may be implemented without the specific details present herein.

The present invention is related to a method of eye tracking using face detection and eye detection combined with motion estimation to accelerate eye tracking. An eye tracker system can be run on one limited resource (CPU, memory ...) devices. For a frame data in the eye tracker system, user's eyes are located based on the detected face. However, face detection for a whole frame always consumes a lot of time. In this invention, we introduce moving estimation to replace sometimes during face

detection to accelerate the eye tracker system, and therefore, can reduce costs and accelerate tracking performance .

The present invention relates to the field of eye tracking. It is important for many applications that obtain real-life person's eye positions. For example, when a user is reading an eBook, the system can automatically adjust the content according to the user' s gaze positions. The present invention relates to the processing of images showing the face of a user as the user moves his/her head, to track the movement of the user's eyes in an accurate and reliable manner. According to the present invention, eye tracking is carried out using face detection and eye detection algorithms and combined with motion estimation algorithm to accelerate tracking performance. Figure 1A illustrates a basic environment according to one embodiment of the present invention. The user 10 is in front of a display device 12 which has a camera 11 to capture image data. Image 13 is the captured data by the camera 11 and displayed on a display device 12.

Figure IB is the same as Figure 1A except for the detection result of identified area. The rectangle 14 shows a face detection result. Two crosses 15 shows detected eye's positions.

Figure 2 illustrates eye detection process. Images (a), (b) , and (c) illustrate detecting phases in one frame image data. The image (a) is new input frame data; the image (b) shows obtaining face detection result, i.e.

face rectangle, after calculation by a face detection algorithm; the image (c) shows obtaining eye's positions according to the detected face rectangle. The images (a), (b) , and (c) comprise the entire process of one frame for the eye tracker system. Image (d) illustrates current frame face position information to forecast the next frame face position (the dashed rectangle) using moving estimate algorithm (such as a block-match algorithm and the like) . When the next frame data arrives, shown in image (e) , the face detection will be skipped and the forecasted face position as input to do eye detection will be directly used. The final detection result should be image (f) .

In the actual implementation, the forecast face position may deviate somewhat from the frame real face area. Some mechanism is necessary to guarantee the detection result. We use an algorithm to confirm the eye's positions. The certify algorithm could use statistic method to obtain a normal model of eye positions and set a threshold. When the detect result deviates the normal model, the forecast tracking data is cleared and the system restarts the face detection in the next frame.

Figure 3 is the flow chart of eye tracker system work process .

At step 301, the system determines if a new frame of image data comes. If YES, then the processing proceeds to the step 303; if NO, the processing proceeds to the end. At step 303, the system checks whether there is identified moving block information. When previous image data is stored, the system determines the result of the check to be "YES". When the current frame is the first one and there is no previous data, the system determines the result of the check to be "NO". If YES, then the processing proceeds to step 305. At step 305, the system can skip face detection and forecast face position using moving block for motion estimation. If NO at step 303, the system cannot skip face detection. At step 307, the system detects face position by face detection algorithm.

At step 309, the system detects eye positions by eye detection algorithm. At step 311, the system certificates the eye positions according to threshold. The threshold is set by a certify algorithm using statistic method.

If the system certificates eye positions at step 311, then the processing proceeds to step 313. If the system does not certificate eye positions at step 311, i.e. the detect result deviates from a normal model acquired by the certify algorithm, the result is deleted and the face detection is restarted in the next frame. If the next frame of image data does not come at step 301, the processing proceeds to END.

Figure 4 illustrates an exemplary block diagram of a system (410) according to an embodiment of the present invention. The system 410 can be a 3D TV set, computer system, tablet, portable game, smart-phone, or the like. The system 410 comprises a CPU (Central Processing Unit) 411, a camera 412, a storage 413, a display 414, and a user input module 415. A memory 416 such as RAM (Random Access Memory) may be connected to the CPU 411 as shown in Figure 4.

The camera 412 is an element for capturing the left and right images with a single lens. The CPU 411 processes the steps as explained above.

The display 414 is configured to visually present text, image, video, and any other contents to a user of the system 410. The display 414 can apply any type that is compatible with 3D contents.

The storage 413 is configured to store software programs and data for the CPU 511 to drive and operate the process as explained above.

The user input module 415 may include keys or buttons to input characters or commands, and also comprises a function for recognizing the characters or commands input with the keys or buttons. The user input module 415 can be omitted in the system depending on use application of the system.

These and other features and advantages of the present principles may be readily ascertained by one of ordinary skill in the pertinent art based on the teachings herein. It is to be understood that the teachings of the present principles may be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof. Most preferably, the teachings of the present principles are implemented as a combination of hardware and software Moreover, the software may be implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPU") , a random access memory ("RAM"), and input/output ("I/O") interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit. It is to be further understood that, because some of the constituent system components and methods depicted in the accompanying drawings are preferably implemented in software, the actual connections between the system components or the process function blocks may differ depending upon the manner in which the present principles are programmed. Given the teachings herein, one of ordinary skill in the pertinent art will be able to contemplate these and similar implementations or configurations of the present principles.

Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present principles is not limited to those precise embodiments, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present principles. All such changes and modifications are intended to be included within the scope of the present principles as set forth in the appended claims.

Claims

1. A method for detecting eye positions, comprising the steps of:

receiving image data of a frame;

determining at least one identified block in the frame ;

checking whether there is information on the

identified block;

if there is the information, calculating face position on the basis of the information using move estimation algorithm; and

detecting eye positions using eye detection

algorithm.

2. The method according to claim 1, further comprising the step of, if there is no information, detecting face position using face detection algorithm.

3. The method according to claim 1 or 2, further comprising the step of confirming eye position using predetermined threshold.

4. The method according to one of claims 1 to 3, wherein move estimation algorithm is a Block Matching Algorithm.

5. A system for detecting eye positions, comprising: means for receiving image data of a frame;

means for determining at least one identified block in the frame;

means for checking whether there is information on the identified block;

means for calculating face position on the basis of the information using move estimation algorithm;

means for detecting eye positions using eye

detection algorithm.

6. The system according to claim 5, further comprising means for, if there is no information, detecting face position using face detection algorithm.

7. The system according to claim 5 or 6, further comprising means for confirming eye position using predetermined threshold.

8. The method according to one of claims 5 to 7, wherein move estimation algorithm is a Block Matching Algorithm.