KR101286965B1 - Apparatus and method for tracking eye - Google Patents

Abstract

The present invention relates to an apparatus and a method for tracking the direction of the gaze of a user.
In accordance with another aspect of the present invention, a gaze tracking method includes: detecting a face region by receiving a face image; Designating an eye candidate region in the detected face region and detecting an eye region in the designated eye candidate region; Detecting a pupil in the detected eye region and detecting a center coordinate of the detected pupil; Calculating a motion vector of the face when no face region or eye region is detected; And controlling the position of the cursor mapped on the screen using the detected center coordinates of the pupil or the calculated motion vector.

Description

Gaze tracking device and method {APPARATUS AND METHOD FOR TRACKING EYE}

The present invention relates to a gaze tracking device and method, and more particularly to a device and a method for tracking the gaze direction of the user.

As digital technology develops today, a technology for analyzing image information and dividing it into specific regions or portions is being developed. Among these analysis technologies, face recognition technology is a technology applied to not only a digital camera but also a device for performing security technology, and has been researched and developed at various angles in various ways.

In general, face detection masks the presence or absence of a person's face in the image for any input image, and if the face is present, finds the face of each person in the image and displays its location. It can be used for criminal shot mug shot maching system, face information retrieval system and object-oriented coding system.

The eye detection system recognizes and detects the eye of the human body in the image for the purpose of realizing a personal recognition system that has been actively studied recently. That is, the purpose of the eye recognition technology is to identify the individual by using the physical characteristics unique to each person, such as fingerprint recognition, iris recognition. In particular, the fingerprint recognition technology, which has been widely used recently, is commercialized as the fingerprint recognition device is widely used in the entrances of high-end apartments or offices, and the demand of the fingerprint recognition device is expected to increase further. Because it is a diet, many people approach from time to time has its disadvantages that the utility is lowered.

Moreover, with the increasing diversity of areas requiring user authentication, such as door locks in apartments or private homes, cars, ATMs, office doors, membership clubs, etc., the eye recognition device is a conventional technology of fingerprint recognition device, IC / magnetic, etc. When combined with assistive recognition devices such as / RF card readers, various types of commercial products can be implemented depending on the security level, which requires active research.

The position of a person's gaze may be determined by considering both the direction of the face and the eyeball. However, since the direction of the eyeball is an important clue to determine the position of the gaze of the person, research has been conducted in the conventional eye tracking method to track the eye movement mainly and to correct the errors caused by facial movement. Conventional eye tracking methods include contact lenses, skin electrodes attached around the eyes, and image analysis methods using a camera vision system. In the case of using a contact lens, 'contact lens with a separate device attached to the lens surface' should be worn to attach to the corneal surface of the user. In the method using the skin electrode, several skin electrodes should be attached around the eyes.

In addition, the method using the camera vision system requires that a wearable camera is attached to the head or an expensive camera device having rotation, auto focus, and zoom functions to obtain a high resolution eye image without being affected by facial movement. There is this. As a result, all three methods have disadvantages in terms of manufacturing cost and compatibility.

Accordingly, the present invention provides a gaze tracking device and a method for tracking a gaze direction of a user by detecting a pupil in an eye region extracted by using a face image of the user.

The present invention also provides a gaze tracking device and method for tracking a gaze direction using a motion vector even when a face region or an eye region fails to be detected using a face image of a user.

In order to solve the above objects, a gaze tracking device according to an exemplary embodiment of the present disclosure may include: a face detector configured to receive a face image and detect a face region; An eye region detector for designating an eye candidate region in the detected face region and detecting an eye region in the designated eye candidate region; A pupil detector for detecting a pupil in the detected eye area and detecting a center coordinate of the detected pupil; A correction unit for calculating a motion vector of the face; And a mapping unit for controlling a position of a cursor mapped on a screen using the detected center coordinates of the pupil or the calculated motion vector.

In accordance with another aspect of the present invention, a gaze tracking method includes: detecting a face region by receiving a face image; Designating an eye candidate region in the detected face region and detecting an eye region in the designated eye candidate region; Detecting a pupil in the detected eye region and detecting a center coordinate of the detected pupil; Calculating a motion vector of the face when no face region or eye region is detected; And controlling the position of the cursor mapped on the screen using the detected center coordinates of the pupil or the calculated motion vector.

The present invention can detect the pupil in the eye region extracted by using the face image of the user to track the direction of the user's eye gaze, and even if the face region or the eye region has not been detected using the face image of the user, Can track the line of sight. In addition, the present invention can track the gaze direction of the user using the gaze without the operation by the hand.

1 is a view showing the configuration of a gaze tracking device according to an embodiment of the present invention,
FIG. 2A illustrates edge characteristics among ha wavelet features used for face detection in a gaze tracking device according to an exemplary embodiment of the present disclosure; FIG.
FIG. 2B is a diagram illustrating line characteristics among ha wavelet features used for face detection in a gaze tracking apparatus according to an exemplary embodiment of the present disclosure; FIG.
2C is a diagram illustrating a center characteristic among ha wavelet features used for face detection in a gaze tracking device according to an exemplary embodiment of the present invention;
3 is a view illustrating a face detection process by cascade classifiers in a gaze tracking apparatus according to an exemplary embodiment of the present invention;
4 is a view illustrating a facial region detection result using an Adaboost face detection technique in a gaze tracking apparatus according to an exemplary embodiment of the present invention;
5 is a diagram illustrating a process of designating an eye candidate region within a face region detected by the gaze tracking apparatus according to an exemplary embodiment of the present disclosure;
FIG. 6 is a diagram illustrating a process of detecting an eye region in a designated eye candidate region in a gaze tracking apparatus according to an exemplary embodiment of the present disclosure; FIG.
FIG. 7 is a conceptual diagram of a mapping of eye coordinates to IPTV screen coordinates in an eye tracking apparatus according to an embodiment of the present invention; FIG.
8 is a diagram illustrating a result of finding a motion vector using a hierarchical KLT feature tracker in an eye tracking apparatus according to an exemplary embodiment of the present invention;
9 is a graph comparing the time taken to reach a randomly located icon using the method according to the present invention and a normal mouse,
10 is a view showing a questionnaire result regarding interest and convenience of an eye tracking apparatus according to an embodiment of the present invention;
11 is an exemplary diagram of Internet shopping among an example of an application service in an IPTV environment using a gaze tracking device according to an embodiment of the present invention;
12 is an exemplary view of a VOD service of an example of an application service in an IPTV environment using an eye tracking apparatus according to an embodiment of the present invention;
FIG. 13 is a view illustrating an operation in a dark night environment among application service examples in an IPTV environment using a gaze tracking device according to an embodiment of the present disclosure; FIG.
14 is a diagram illustrating a gaze tracking process in an IPTV system according to an exemplary embodiment of the present invention.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, let's look at examples in which the eye tracking method is used. The gaze tracking method is divided into Skin Electrodes based method, Contact Lens based method, Head Mounted Display attachment based method, and Remote Pan & Tilting device based method. have.

The skin electrode-based method is a method of measuring an electric potential difference between a retina and a cornea by attaching an electrode around a user's eye, and calculating a gaze position based on the measured electric potential difference. The skin electrode-based method has the advantage of being able to grasp both gaze positions of both eyes, a low cost, and a simple method of use. However, the skin electrode based method has limited accuracy in the horizontal and vertical movements.

The contact lens-based method is a method of calculating a gaze position by attaching a non-slip lens to the cornea and attaching a magnetic field coil or mirror to the cornea. The contact lens based method can accurately calculate the gaze position. However, it is inconvenient to use, the eyes do not flicker freely, and the calculation range is limited.

The head mounted display attachment based method calculates the gaze position using a small camera mounted under the headband or helmet. The head mounted display attachment based method can calculate the gaze position regardless of the user's head movement. However, the camera is tilted below the user's eye level, making it insensitive to vertical movement of the eye, so it only applies to head mounted displays.

Remote Pan & Tilt device based method calculates stare position by installing pan & tilt camera and lighting around monitor. Remote pan-and-tilt device-based methods have the advantages of accurate and fast gaze position calculations and ease of application, but require more than two expensive stereo camera devices and complex algorithms to track head movements. Requires complex calibration of the liver. Then, the configuration of the eye tracking apparatus according to the exemplary embodiment of the present invention will be described in more detail with reference to FIG. 1.

1 is a diagram illustrating a configuration of a gaze tracking device according to an exemplary embodiment of the present invention.

The apparatus for tracking eye gaze according to an exemplary embodiment of the present invention may be applied to various application fields such as various types of video player, such as an IPTV system, a TV, a computer, and a game machine. In the following description, the IPTV system will be described as an example. Initially, it will be described on the assumption that the user is staring at the center point of the IPTV monitor including an image photographing device, for example, a camera, a webcam, etc., for photographing the user's face.

Referring to FIG. 1, the gaze tracking device includes a face detector 100, an eye region detector 102, a pupil detector 104, a mapping unit 106, and a correction unit 108. The face detector 100 receives a face image of a user photographed from an image photographing device connected to an IPTV. The face detector 100 detects a face region from a face image by using an Adaboost algorithm.

The adaboost algorithm is an algorithm that can accurately detect a face region when the input face image of the user is a front face. If the face image of the user is not the front face, the correction unit 108, which will be described later, uses a hierarchical KLT feature track algorithm (hereinafter referred to as a "hierarchical KLT feature tracker"). The motion vector of the face is calculated, and the mapping unit 106 extracts the center coordinates of the pupil by controlling the cursor using the calculated motion vector.

The Adaboost algorithm consists of a training phase and an execution phase, which pre-set a strong classifier consisting of a plurality of weak classifiers using a sample similar to the test object, such as a feature template. It's a step. The execution step refers to a step of determining whether or not a condition is met by applying a pre-set strong classifier to the actual inspection target.

When the face detection unit 100 extracts a face region from the input face image of the user by using the Adaboost algorithm, the face detection unit 100 divides the face image of the input user into a sub-window without using it as it is. In addition, the face detector 100 calculates a feature value by using a lower wavelet feature included in the subwindow. In this case, the performance of the Adaboost algorithm depends on the number of weak classifiers, and the accuracy of the Adaboost algorithm depends on the feature value calculation method or the window calculation method.

Since the wavelet features included in the subwindow are larger than the number of pixels, the face detector 100 calculates a feature value by focusing on a small and important feature. The lower wavelet feature may be represented by a template as shown in FIG. 2. Next, the wavelet feature used for face detection in the IPTV system according to an exemplary embodiment of the present invention will be described in more detail with reference to FIGS. 2A to 2C.

FIG. 2A illustrates edge characteristics among hawavelet features used in the gaze tracking device according to an exemplary embodiment of the present invention, and FIG. 2B illustrates line characteristics among hawavelet features used in an eye gaze tracking apparatus according to an embodiment of the present invention. FIG. 2C is a diagram illustrating a center characteristic among ha wavelet features used in an eye tracking apparatus according to an exemplary embodiment of the present invention.

Referring to Figures 2a to 2c, the wavelet feature consists of two or three black and white squares, inclined horizontally, vertically or diagonally. Each of these lower wavelet features represents facial components such as both eyes, nose, left eye, eye couple, and the like. FIG. 2A is a representation of a pupil, and FIG. 2B is a representation of various boundary components formed by the pupil and the iris.

The face detector 100 uses the calculated feature value as an input of a decision tree classifier. At this time, each classifier is called a weak classifier because it can not find the desired object individually. The decision tree classifier is used in many fields such as lady signal classification, medical diagnosis, expert system, speech recognition, etc., and is a classifier used to make complex decision processing into a simple set of decisions. The decision tree classifier can create a simple set of decisions in four steps.

First, a subset W of feature values is randomly selected using the feature values, and Decision Trees are generated from the W. The exceptions are retrieved by applying the rules generated from the non-feature values. The exception is included in the subset of feature values, and a decision tree is created using the updated subset. Through the above process, a powerful classifier is generated.

Referring to Equation 1, the variable f represents a weak classifier, and the variable n represents the number of weak classifiers and the number of stages. The variable c represents a coefficient that varies from stage to stage. At each iteration, the new weak classifier f _i is learned and accumulated.

The smaller the error value of the classifier f _i in the learning set, the larger the coefficient c _i becomes. Then, the weights of all the learning samples are updated, and in the next iteration, the role of the samples misclassified by the already made F is emphasized. If f _i is any greater than the predetermined threshold, F has a predetermined large value. In practice, however, fast processing times require as many large learning sets as there are very many weak classifiers. Next, the face detection process by the cascade classifiers will be described in more detail with reference to FIG. 3.

Referring to FIG. 3, the serial classifier is arranged in series so that the weak classifier is in front. The current search window is divided into detailed steps by each F _K classifier that determines the object to be detected. With this weak classifier in front, the computation time required for detection can be reduced. Here, each of the steps aims at a high hit-rate rather than a low false-alarm rate.

In other words, rather than reducing the error rate of finding errors that are not faces, they are taught to face without errors. In this case, it is possible to achieve good detection performance by setting the hit-rate and false-alarm rate to be achieved in each step in advance and selecting the accuracy in each step. In the present invention, using the classifier learned in this manner, when the input image is a front face, the face region is detected as shown in FIG. 4 with 99% accuracy.

The eye region detector 102 designates an area where the eye can exist, for example, an eye candidate region, as shown in FIG. 5, in the face region detected by the face detector 100 using the ratio of the face. The eye region detector 102 detects an eye region from the eye candidate region as shown in FIG. 6. 6, when the eye region is not detected as illustrated in FIG. 6, the correction unit 108 to be described later calculates a motion vector of a face using a hierarchical KLT feature tracking algorithm. The mapping unit 106 extracts the center coordinates of the correct pupil by controlling the cursor using the calculated motion vector.

Here, the ratio of the face refers to the ratio corresponding to the second area from above when the face area detected by the face detector 100 is divided into four equal parts in the vertical direction. This experiment determined that the eye candidate region was selected from 192 face images of 32 photographs, and when the face region detected from the face image was divided into four in the vertical direction, the second region included the eyes.

By determining the eye candidate region as described above, it is possible to reduce the error of falsely detecting the non-eye region with the eye. In this case, when the face area is not the front, the eye area may exist at the edge, so the search range is narrowed only in the vertical direction and the search range is not narrowed in the vertical direction.

The pupil detector 104 detects the pupil in the eye region by using an adaptive threshold estimation method. The adaptive threshold estimation method arbitrarily determines an initial threshold value T and performs binarization based on the determined threshold value. The pupil detector 104 includes a G1 group having a brightness value greater than or equal to the threshold value of the eye candidate region and a brightness value having a brightness value less than the threshold value of the eye candidate region through the binarization process. Divide into G2 group.

Then, the pupil detector 104 calculates an average value of brightness values included in the G1 group and the G2 group, and calculates a new threshold value using the calculated average value as shown in Equation 2 below.

Referring to Equation 2, μ ₁ represents an average value of the G1 group, and μ ₂ represents an average value of the G2 group. The pupil detection unit 104 repeats the above process until the amount of change in the threshold is smaller than a predetermined value.

The pupil detector 104 detects a pupil by binarizing an eye candidate region by using a feature that a gray level of an eye pupil region is lower than a surrounding region. Then, the pupil detector 104 applies a component labeling method to the detected pupils in order to reduce the effects of noise included in the eye area, for example, eyebrows.

The component naming method is an image processing method for assigning a unique code to each area separated from an image, and is used for segmentation of a binary image, feature extraction of an area, and restoration of an image. The method of naming components includes raster scan, seed connectivity, quadtree, and contour following.

That is, the pupil detector 104 tracks the contour of the pupil using the component naming method, calculates the center of gravity of the black pixel of the finally extracted pupil, and accurately detects the center coordinate of the pupil. Next, the process of mapping the center coordinates of the pupil to the logical coordinates of the IPTV screen will be described in more detail with reference to FIG. 7.

FIG. 7 is a diagram illustrating a process in which the mapping unit 106 maps center coordinates of a pupil to logical coordinates of an IPTV screen.

에 대응시킨다. 여기서, 상기 x, y는 각각 동공의 x좌표와 y좌표를 나타낸다. 그리고 상기 맵핑부(106)는 하기 <수학식 3>과 <수학식 4>를 이용하여 스크린 상에 사상되는 커서의 위치를 계산한다.
Referring to FIG. 7, the mapping unit 106 may determine a center coordinate of the pupil detected by the pupil detector 104, for example, (S _x , S _y ) as a center coordinate of the IPTV monitor.

To match. Here, x and y represent the x coordinate and the y coordinate of the pupil, respectively. The mapping unit 106 calculates the position of the cursor mapped on the screen by using Equations 3 and 4 below.

와

는 상기 (C_x, C_y)의 변화에 따라 (M_x, M_y)의 변화량을 변화시켜서 상기 모니터 상에 사상되는 커서 조절의 감도를 조정하기 위한 상수이다. 상기 (C_x, C_y)의 변화에 따라 (M_x, M_y)의 변화량을 변화시킬 때 상기 변화량이 작으면 (C_x, C_y)의 작은 변화에도 화면상의 커서를 많이 움직일 수 있다. 이런 경우, 상기 커서 이동이 민감하여 화면의 여러 픽셀을 한꺼번에 건너뛰게 될 수 있으므로 커서의 정교한 조정을 할 수 없어진다.Referring to Equation 3 and Equation 4, (C _x , C _y ) represents the center coordinates of the user pupil moved in a continuous frame, and (M _x , M _y ) represents the detection unit ( The position of the cursor mapped on the monitor when the center coordinate of the pupil detected at 104), for example, (S _x , S _y ) is mapped to the center coordinate of the IPTV monitor.

Wow

Is a constant for adjusting the sensitivity of cursor adjustment mapped on the monitor by changing the change amount of (M _x , M _y ) according to the change of (C _x , C _y ). Even small changes in the if the said amount of change is smaller when changes in the amount of change in the (M _x, M _y) in accordance with the change of _{(C x, C y) (} C x, C y) can be moved much the cursor on the display. In such a case, the cursor movement is sensitive so that several pixels on the screen can be skipped at once, so that precise adjustment of the cursor cannot be performed.

On the other hand, when the change amount of (M _x , M _y ) is changed according to the change of (C _x , C _y ), a precise adjustment is possible. However, since many face movements are required to move the cursor, the α and β values must be appropriately adjusted according to the application of the IPTV. The present invention provides a function to allow the user to set α and β according to the preference at the beginning of the system use.

In addition, the present invention minimizes the inconvenience that the user has to move a lot of the face by using the digital zoom function of the camera to enable the cursor to move a lot even with a small movement of the user. In addition, when the (C _x , C _y ) coordinates move 2 pixels or less in comparison to the previous frame of the image, the cursor is corrected by replacing (C _x , C _y ) with the center positions of both eyes detected in the previous frame. Added functionality.

When the face image of the user input to the face detector 100 is the front face, the center coordinate of the pupil may be extracted through the above process, but the face image of the user input to the face detector 100 is the side face. In this case, it is not possible to extract the center coordinate of the correct pupil through the above process.

For this reason, the correction unit 108 calculates a motion vector of a face using a hierarchical KLT feature tracking algorithm, and the mapping unit 106 uses the motion vector to generate a cursor. Extract the center coordinates of the pupil by controlling. The hierarchical KLT feature tracking algorithm is an algorithm that satisfies image brightness constancy constraints and calculates the correspondence of feature points by calculating the displacement of feature points between successive frames when the motion of the image is small. .

At this time, a sufficiently large feature point window W is required for the movement to obtain a large displacement. However, when the size of the feature point window increases, tracking performance for small displacements decreases due to operations for calculating the average of the window, and the amount of calculation also increases. For this reason, an appropriate choice of window size is required.

Therefore, in order to compromise between the size and the accuracy of the feature window, the KLT feature tracker performs hierarchical tracking to track fine movements corresponding to the feature window with large displacement. Due to the movement of the image photographing device, the feature point window moves by a certain distance between successive frames, and the shape changes slightly.

을 최소화하는 변위를 결정하는 것을 의미한다.
Referring to Equation 5, the function I represents the brightness in a specific frame, the function J represents the brightness in the next frame, the variable d represents the displacement vector, and the function η represents a difference due to a change in shape. The tracking of feature points is a mean squared error, e.g., defined in Eq.

Means to determine the displacement to minimize.

을 최소화하는 변위를 계산할 수 있다. 상기 테일러급수는 미분 가능한 어떤 함수를 다항식의 형태로 근사화할 때 사용되는 방법이다.The correction unit 108 may approximate Equation 5 using a Taylor series, and repeats the above process.

We can calculate the displacement to minimize. The Taylor series is a method used to approximate any differential function in the form of a polynomial.

Thereafter, the correction unit 108 calculates a motion vector using a hierarchical KLT feature tracking algorithm. More specifically, the correction unit 108 extracts corner points from the previous image and the current image by using a Harris corner detector. The Harris corner detector is basically a detector that extracts feature points based on a local auto-correlation function capable of measuring signal changes locally.

Then, the correction unit 108 extracts corner points corresponding to each other between the previous image and the current image. As a result, the correction unit 108 may calculate a change in corner points corresponding to each other in the current image and the previous image, that is, a motion vector.

The motion vector is a vector used to extract the remaining dynamic objects except the background from the dynamic object candidates extracted through clustering. In the present invention, the motion vector is used to extract the pupil from the eye candidate regions except for the background.

The mapping unit 106 controls the cursor as shown in FIG. 8 using the calculated motion vector. 8 is a diagram illustrating a process in which the mapping unit 106 controls a cursor through a motion vector calculated by the correction unit 108 using a hierarchical KLT feature tracking algorithm. Then, the process by which the mapping unit 106 controls the cursor through the motion vector calculated by the correction unit 108 using the hierarchical KLT feature tracking algorithm with reference to Equations 7 and 8 below. It will be described in more detail.

는 상기 KLT 특징 추적 알고리즘을 통해 계산된 모션 벡터의 x, y축 성분들을 각각 나타낸다.Referring to Equations 7 and 8, (M _x and M _y ) represent coordinates mapped on the monitor, and (M _{x prev} and M _{y prev} ) represent coordinates of the previous cursor. Also,

Represents the x and y-axis components of the motion vector calculated by the KLT feature tracking algorithm, respectively.

The adaboost algorithm may accurately extract a face region when the input face image of the user is a front face. However, when the user watches the IPTV, the rotation or the posture of the face may be changed, and thus the face region may not be extracted accurately by using the Adaboost algorithm alone.

In order to solve this problem, the correction unit 108 calculates a motion vector of the face using a hierarchical KLT feature tracking algorithm, and the mapping unit 106 uses the calculated motion vector to cursor as shown in FIG. 8. Solving the above problem was solved. Then, when the gaze tracking device of the present invention is applied to the IPTV system, the experimental results and analysis of gaze tracking will be described in more detail.

When the gaze tracking device according to the exemplary embodiment of the present invention is applied to an IPTV system, it is performed in the following environment. The computer hardware for IPTV includes an Intel (R) Core 2 (TM) Duo CPU 2.4GHz and 3GB RAM. In addition, a camera is connected to the IPTV, and the model of the camera is Logitech PN 960-00057. The monitor for IPTV is 19 inches in size and the monitor resolution is 1260 x 1024 pixels. The user is located at a distance of 80 cm from the monitor, and the camera connected to the IPTV captures a face image of the user, and the captured face image of the user is transmitted to the face detection unit 100.

와

값을 (C_x, C_y)와 (M_x, M_y)의 움직임 비가 1:4가 되도록 설정하고 진행된다.A program for eye tracking according to an embodiment of the present invention is implemented using DirectShow of DirectX 9.0c Software Development Kit (SDK) in a Microsoft Visual Studio C ++ development environment. In addition, the experiment considering the size of the menu shown on the IPTV screen

Wow

The value is set so that the movement ratio of (C _x , C _y ) and (M _x , M _y ) is 1: 4.

Next, an objective performance evaluation of the gaze tracking when the gaze tracking device according to the exemplary embodiment of the present invention is applied to the IPTV system will be described in detail with reference to FIG. 9.

FIG. 9 is a graph comparing the time taken to reach an icon randomly positioned using a general mouse and the case where the gaze tracking method according to an exemplary embodiment of the present invention is applied to an IPTV system.

Referring to Figure 9, by using the gaze tracking method according to an embodiment of the present invention using the mouse and the time it takes to position the cursor in order to the ten icon pictures randomly located on the IPTV monitor using the picture of the 10 icons This is a graph comparing the time taken to position the cursor in order. The average value of the time was carried out as described above for a total of 25 people.

When the gaze tracking method according to the embodiment of the present invention is used in the IPTV system, it takes a long time initially, but the time to reach the icon decreases with repeated use, which is almost the same as when using a mechanical mouse. Mechanical mice do not change much with time depending on the number of times. This is because most of the users are familiar with the mechanical mouse and not with the proposed algorithm.

The method according to the present invention takes an average of 64 ms to process an eye gaze position on an IPTV screen by processing one frame including an image. Through this, when the gaze tracking method according to the exemplary embodiment of the present invention is applied to an IPTV system, it can be seen that when the USB camera which acquires 15 frames per second can be used in real time. Then, the subjective performance evaluation of the eye tracking apparatus according to the embodiment of the present invention will be described in more detail with reference to FIG. 10.

In the subjective performance evaluation, a total of 25 subjects were measured for the convenience and interest of using the conventional mechanical mouse in the IPTV environment and the proposed method. Experimental results regarding convenience and interest are shown in FIG. 10. Referring to FIG. 10, it can be seen that there are many opinions that the method proposed in the present invention is of higher interest than using a mechanical mouse. On the other hand, in the survey on convenience, there were some opinions that the input method using a mouse is more convenient than the proposed method.

11 to 13 illustrate examples of using various services, such as Internet search, video-on-demand, and the like, in an IPTV environment by using the method proposed by the present invention. 11 to 13, it can be seen that the method according to the present method can be used regardless of the indoor lighting situation when watching TV. Next, the eye tracking process according to the exemplary embodiment of the present invention will be described in more detail with reference to FIG. 14.

Referring to FIG. 14, the face detector 100 receives a face image of a user photographed from an image photographing device in step 1401. In step 1402, the face detector 100 detects a face region from a face image by using an Adboost algorithm. In operation 1403, the face detector 100 determines whether the face region is successfully detected from the face image.

The determination result 1403, if the face area is successfully detected, proceeds to step 1404, and if the determination result 1403, the face area is not successfully detected, proceeds to step 1408. First, the determination result 1403, a case where the facial region is successfully detected will be described. The eye region detector 102 designates a region where the eyes may exist, for example, an eye candidate region, in the face region detected by the face detector 100 using the ratio of faces in step 1404. In operation 1405, the eye region detector 102 detects an eye region from the eye candidate region detected in operation 1404 using the Adaboost algorithm.

If the determination result 1403 does not successfully detect the face region, the correction unit 108 uses a hierarchical KLT feature tracking algorithm in step 1408 to detect the motion vector of the face. Calculate In operation 1409, the mapping unit 106 extracts an accurate pupil center coordinate by controlling a cursor using the calculated motion vector.

The hierarchical KLT feature tracking algorithm is an algorithm that satisfies image brightness constancy constraints and calculates the correspondence of feature points by calculating the displacement of feature points between successive frames when the motion of the image is small. .

At this time, a sufficiently large feature point window W is required for the movement to obtain a large displacement. However, when the size of the feature point window increases, tracking performance for small displacements decreases due to operations for calculating the average of the window, and the amount of calculation also increases. For this reason, an appropriate choice of window size is required.

Therefore, in order to compromise between the size and the accuracy of the feature window, the KLT feature tracker performs hierarchical tracking to track fine movements corresponding to the feature window with large displacement.

Next, the eye region detector 102 determines whether the eye region is successfully detected in the eye candidate region in step 1406. The determination result 1406, if the eye region is successfully detected, proceeds to step 1407, and if the determination result 1406, the eye region is not successfully detected, proceeds to step 1408. First, the determination result 1406 will be described in the case where the eye region is successfully detected.

The mapping unit 108 maps the center coordinates of the pupil detected in operation 1407 to the center coordinates of the IPTV monitor, and uses the <Equation 3> and <Equation 4> to determine the position of the cursor mapped on the screen. Calculate Here, the description of Equation 3 and Equation 4 will be omitted because it has been described above.

If the determination result 1406, the eye region is not successfully detected, the correction unit 108 in step 1408 using the hierarchical KLT feature tracking algorithm (Pyramidal Kanade Lucas Tomasi Feature Track) algorithm Calculate In operation 1409, the mapping unit 106 extracts an accurate pupil center coordinate by controlling a cursor using the calculated motion vector.

The present invention has been described above with reference to preferred embodiments thereof. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. The disclosed embodiments should, therefore, be considered in an illustrative rather than a restrictive sense. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

100: face detection unit 102: eye area detection unit
104: pupil detection unit 106: mapping unit
108: correction unit

Claims (20)

delete In the eye tracking apparatus,
A face detector for detecting a face region by receiving a face image;
An eye region detector for designating an eye candidate region in the detected face region and detecting an eye region in the designated eye candidate region;
A pupil detector for detecting a pupil in the detected eye area and detecting a center coordinate of the detected pupil;
A correction unit for calculating a motion vector of the face; And
A mapping unit for controlling the position of the cursor mapped on the screen using the detected center coordinates or the calculated motion vector.
Eye tracking device comprising a.
The method of claim 2, wherein the face detection unit,
A gaze tracking device for detecting a face region by analyzing a feature point of the received face image using an Adboost algorithm.
3. The method of claim 2,
When the face detection unit does not detect a face area in the received face image,
The gaze tracking, wherein the correction unit calculates a motion vector of a face using a hierarchical Kanade Lucas Tomasi (KLT) feature tracker, and the mapping unit controls the position of a cursor mapped on the screen using the calculated motion vector. Device.
The method of claim 2, wherein the eye area detection unit,
And dividing the detected face region into four regions in a vertical direction, and designating a second region from above among the divided regions as an eye candidate region.
3. The method of claim 2,
When the eye region detector fails to detect an eye region in the designated eye candidate region,
The gaze tracking, wherein the correction unit calculates a motion vector of a face using a hierarchical Kanade Lucas Tomasi (KLT) feature tracker, and the mapping unit controls the position of a cursor mapped on the screen using the calculated motion vector. Device.
The method of claim 2, wherein the pupil detection unit,
And detecting a pupil by binarizing the detected eye region, and detecting a center coordinate of the pupil by tracking a component of the detected pupil.
The method of claim 7, wherein the pupil detection unit,
A gaze tracking device for detecting a pupil using an adaptive threshold estimation method.
The method of claim 7, wherein the pupil detection unit,
And tracking the contour of the detected pupil to calculate the center of gravity of the extracted black pixel of the pupil, thereby detecting the center coordinate of the pupil.
The method of claim 2, wherein the correction unit,
And extracting corner points corresponding to each other from a previously detected face image and the detected face image, and calculating a change (ie, a motion vector) of the extracted corner points.
delete In the eye tracking method,
Detecting a face area by receiving a face image;
Designating an eye candidate region in the detected face region and detecting an eye region in the designated eye candidate region;
Detecting a pupil in the detected eye region and detecting a center coordinate of the detected pupil;
Calculating a motion vector of the face when no face region or eye region is detected; And
Controlling the position of the cursor mapped on the screen using the detected center coordinates or the calculated motion vector.
Eye tracking method comprising a.
The method of claim 12, wherein detecting the face area comprises:
And detecting a face region by analyzing feature points of the received face image by using an AdBoost algorithm.
13. The method of claim 12,
If the face area is not detected in the detecting of the face area,
Computing the motion vector of the face using a hierarchical Kanade Lucas Tomasi (KLT) feature tracker in calculating the motion vector, and controlling the position of the cursor on the screen using the calculated motion vector. A line of sight tracking method that controls the position of the mapped cursor.
The method of claim 12, wherein the specifying of the eye candidate region comprises:
And dividing the detected face region into four regions in a vertical direction, and designating a second region from above among the divided regions as an eye candidate region.
13. The method of claim 12,
If the eye region is not detected in the detecting of the eye region,
Computing the motion vector of the face using a hierarchical Kanade Lucas Tomasi (KLT) feature tracker in calculating the motion vector, and controlling the position of the cursor on the screen using the calculated motion vector. A line of sight tracking method that controls the position of the mapped cursor.
The method of claim 12, wherein detecting the center coordinates of the pupil,
And detecting a pupil by binarizing the detected eye region, and detecting a center coordinate of the pupil by tracking the detected component of the pupil.
The method of claim 17, wherein the detecting of the pupil comprises:
A gaze tracking method for detecting a pupil using an adaptive threshold estimation method.
The method of claim 17, wherein detecting the center coordinates of the pupil,
Tracking the contour of the detected pupil to calculate the center of gravity of the extracted black pixel of the pupil, thereby detecting the center coordinates of the pupil.
The method of claim 12, wherein calculating the motion vector comprises:
And extracting corner points corresponding to each other from a previously detected face image and the detected face image, and calculating a change (ie, a motion vector) of the extracted corner points.

Images