KR101288447B1 - Gaze tracking apparatus, display apparatus and method therof - Google Patents

Abstract

A display device is disclosed. The display apparatus includes a display unit for providing a display screen, an imaging unit for capturing an image of a user viewing a display screen, a detector for detecting information about an eyeball area of the user from the captured image, and an infrared eyeball image captured with an infrared camera. Determination unit for determining the eye gaze of the user by guessing the pupil position in the eye area using information on the eye unit and the information on the eye region detected by the storage unit, the detection unit and the display device according to the user's eye It includes a control unit for controlling the operation. Accordingly, the user's gaze on the display screen may be determined, and the display apparatus may be controlled according to the user's gaze.

Description

Gaze tracking device, display device using same and method thereof {GAZE TRACKING APPARATUS, DISPLAY APPARATUS AND METHOD THEROF}

The present invention relates to a gaze tracking device, a display device using the same, and methods thereof, and more particularly, to an apparatus for tracking the gaze of a user from a captured user image, a display device using the same, and a method thereof.

Human computer interaction (HCI) remains a high-priority research topic while mobile IT devices are popular in everyday life. In recent years, various HCI technologies, such as multi-touch interface technology and gesture recognition, have been applied to mobile IT devices such as smartphones, tablet PCs, and interactive game consoles.

In addition to these techniques, HCI, which utilizes the user's eye movements, provides convenience and high speed input, so interface technologies for detecting eye movements for manipulating a computer or analyzing the user's psychological facts are also being developed.

There are several types of devices that track the eyes of the user. Head-type or eyeglass-type eye tracking systems can provide relatively accurate gaze points. The camera attached to such a device is located close to the human pupil, so that the pupil and the pupil can be stably detected at all times. However, in the case of this type of device, the gaze point of the user may not be easily measured since the captured image may include not only the user's face but also parts unnecessary for detecting eye movement.

Therefore, most accurate eye tracking devices generally use light emitting sources near IR (Infrared Ray) to detect pupils. Pupil detection using an IR camera provides high accuracy in performing eye tracking and makes finding the pupil easier. However, the method using the IR emits energy directly to the eyes of the user and emits a lot of heat when operating, so it is not stable to use for a long time. For these reasons, embedded devices such as smart phones and tablet PCs have limited application of eye tracking systems using IR cameras, so a method of tracking a user's eyes using a conventional USB camera is required.

SUMMARY OF THE INVENTION The present invention is directed to the above-described needs, and an object of the present invention is to provide a gaze tracking device, a display device using the same, and a method for tracking the gaze of a user from a user image photographed using a conventional USB camera.

Display device according to an embodiment of the present invention for achieving the above object, the display unit for providing a display screen, an image pickup unit for picking up the image of the user viewing the display screen, the image picked up by the image pickup unit A detection unit for detecting information about an eyeball area of a user from an image, a storage unit storing infrared eyeball image information captured by an infrared camera, information on the eyeball area detected by the detection unit, and the infrared eyeball image information And a determining unit determining the eye position of the user by estimating a pupil position in an area, and a controller controlling the operation of the display apparatus according to the user's eye.

Here, the information about the eyeball area is characterized in that the iris center coordinates, the detection unit detects the face area in the captured image, detects the eyeball area in the detected face area, the iris in the detected eyeball area By detecting the region, the center coordinates of the detected iris region can be detected.

In addition, the infrared eyeball image information is characterized in that the coordinates of the center of the pupil contained in the infrared eye image, the determination unit, the neural network (Neural Network) to which the MLP algorithm is applied using the center coordinate of the pupil as learning data The apparatus may include a pupil estimating unit that estimates the pupil center coordinates of the user from the detected iris center coordinates, and a gaze determination unit that determines the eyes of the user on the display screen using the estimated pupil center coordinates.

The gaze determination unit may determine the gaze of the user on the display screen by using the following formula,

및

는 디스플레이 화면상의 사용자 시선의 x축 및 y축 좌표이고,

및

는 상기 사용자의 동공 중심 좌표 중 x축 및 y축 좌표이다.here,

And

Is the x- and y-axis coordinates of the user's line of sight on the display screen,

And

Is the x-axis and y-axis coordinates of the pupil center coordinates of the user.

The controller may display an application UI by executing an application corresponding to the application icon when the gaze of the user is located on the application icon displayed on the display screen.

The storage unit may further store the pupil center coordinates derived from the eyeball area of the user and the user gaze coordinates corresponding to the pupil center coordinates, and the gaze determination unit stores the pupil center coordinates estimated by the pupil guessing unit. When matching with the pupil center coordinates, the user's gaze on the display screen corresponding to the stored pupil center coordinates may be determined as the gaze of the user.

The storage unit may further store feature information of the face area for each of the plurality of users, and the controller may identify a user corresponding to the captured image by using the stored feature information of the face area and the gaze. The determination unit may compare the pupil center coordinates estimated by the pupil guessing unit with the pupil center coordinates corresponding to the identified user among the stored pupil center coordinates.

On the other hand, the eye tracking apparatus according to an embodiment of the present invention, the detection unit for detecting information about the eyeball area of the user from the captured user image, the storage unit storing the infrared eye image information captured by the infrared camera and the detection unit detected And a determination unit configured to determine the gaze of the user by estimating a pupil position in the eyeball area using information on the eyeball area and the infrared eyeball image information.

Here, the information about the eyeball area is characterized in that the iris center coordinates, the detection unit detects the face area in the captured image, detects the eyeball area in the detected face area, the iris in the detected eyeball area By detecting the region, the center coordinates of the detected iris region can be detected.

In addition, the infrared eyeball image information is characterized in that the coordinates of the center of the pupil included in the infrared eye image, the determination unit, the detection using a neural network applied MLP algorithm using the center coordinate of the pupil as learning data It may include a pupil guessing unit for guessing the pupil center coordinates of the user from the iris center coordinates, and a gaze determination unit for determining the gaze of the user on the display screen using the estimated pupil center coordinates.

The storage unit may further store the pupil center coordinates derived from the eyeball area of the user and the user gaze coordinates corresponding to the pupil center coordinates, and the gaze determination unit stores the pupil center coordinates estimated by the pupil guessing unit. When matching with the pupil center coordinates, the user's gaze on the display screen corresponding to the stored pupil center coordinates may be determined as the gaze of the user.

The storage unit may further store feature information of the face area for each of the plurality of users, and the controller may identify a user corresponding to the captured image by using the stored feature information of the face area and the gaze. The determination unit may compare the pupil center coordinates estimated by the pupil guessing unit with the pupil center coordinates corresponding to the identified user among the stored pupil center coordinates.

On the other hand, the display method according to an embodiment of the present invention, providing a display screen, photographing the image of the user viewing the display screen, detecting the information on the eyeball area of the user from the captured image Determining eye gaze of the user by estimating a pupil position in the eye area using information on the detected eye area and pre-stored infrared eye image information, and operation of the display apparatus according to the user's gaze It may include the step of controlling.

Here, the information about the eyeball area is characterized in that the iris center coordinates, the detecting step, detecting the face area in the captured image, detecting the eyeball area in the detected face area, The method may include detecting an iris region in the eyeball region and detecting an iris center coordinate of the detected iris region.

In addition, the infrared eyeball image information is characterized in that the coordinates of the center of the pupil contained in the infrared eye image, the step of determining, using a neural network to which the MLP algorithm is applied using the center coordinates of the pupil as learning data And estimating the pupil center coordinates of the user from the detected iris center coordinates, and determining the eyes of the user on a display screen using the estimated pupil center coordinates.

In the determining of the gaze, the gaze of the user on the display screen may be determined by using the following formula,

및

는 디스플레이 화면상의 사용자 시선의 x축 및 y축 좌표이고,

및

는 상기 사용자의 동공 중심 좌표 중 x축 및 y축 좌표이다.here,

And

Is the x- and y-axis coordinates of the user's line of sight on the display screen,

And

Is the x-axis and y-axis coordinates of the pupil center coordinates of the user.

The controlling may include displaying an application UI by executing an application corresponding to the application icon when the gaze of the user is located on the application icon displayed on the display screen.

The method may further include storing pupil center coordinates derived from the eyeball area of the user and user gaze coordinates corresponding to the pupil center coordinates, and determining the gaze of the user on the display screen may include determining the pupil center position. When the guessed pupil center coordinates are matched with the stored pupil center coordinates, the user's gaze on the display screen corresponding to the stored pupil center coordinates may be determined as the user's gaze.

Here, the method may further include storing feature information of a face area for each of the plurality of users, and the controlling may include identifying a user corresponding to the captured image by using feature information of the stored face area. The determining of the gaze of the user on the display screen may compare the estimated pupil center coordinates with pupil center coordinates corresponding to the identified user among the stored pupil center coordinates.

Meanwhile, a gaze tracking method according to an exemplary embodiment may include detecting information about an eyeball area of a user from a captured image, and using information about the detected eyeball area and a pre-stored infrared eyeball image. Can be.

The information on the eyeball area may be an iris center coordinate, and the detecting may include detecting a face area in the captured image, detecting an eyeball area in the detected face area, and detecting the detected eyeball area. The method may include detecting an iris region in the eyeball region and detecting an iris center coordinate of the detected iris region.

In addition, the infrared eyeball image information is characterized in that the coordinates of the center of the pupil contained in the infrared eye image, the determining step, using a neural network to which the MLP algorithm is applied using the center coordinates of the pupil as learning data And estimating the pupil center position of the user from the detected iris center coordinates, and determining the gaze of the user on the display screen using the estimated pupil center coordinates.

According to various embodiments of the present disclosure as described above, the gaze of the user may be tracked from an image photographed using a conventional USB camera, and the display may be manipulated using the gaze of the user.

1 is a block diagram illustrating a configuration of a display apparatus according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a gaze tracking device according to an exemplary embodiment of the present invention.
3 is a flowchart illustrating a process of detecting a center coordinate of an iris of a user from a captured user image.
4 is a flowchart illustrating a process of detecting a face region from a captured user image.
5 shows a result of detecting an eyeball area in the detected face area.
6 shows the result of detecting the tail of the eye in the detected eye area.
7 shows a graph in which the eyeball area is projected on the horizontal and vertical axes to detect the iris area and the iris center coordinates.
8A and 8B show flow charts and results of detecting pupil center coordinates in an infrared eye image.
9 shows the structure of a neural network for estimating pupil center coordinates.
10A and 10B are flowcharts illustrating an operation process of a training phase and a test phase of a neural network.
11 shows calibration to the display screen.
12A and 12B illustrate a gaze tracking result using a gaze tracking device according to an exemplary embodiment.
13 is a flowchart illustrating a display method according to an exemplary embodiment.
14 is a flowchart illustrating a gaze tracking method according to an exemplary embodiment.

Hereinafter, with reference to the accompanying drawings will be described in detail with respect to the present invention.

1 is a block diagram showing a configuration of a display device according to an embodiment of the present invention. According to FIG. 1, the display apparatus 100 includes a display 110, an imaging unit 120, a storage 130, a detector 140, a determiner 150, and a controller 160. The display device 100 may be implemented by various types of devices such as a TV, a home theater, a mobile phone, a PDA, a game machine, a notebook PC, an e-book, an MP3 player, and the like.

The display 110 may display various types of information on the display screen. In detail, the display 110 may display information previously stored in the storage 150 or various information such as a video, a photo, a document, a UI, an icon, and the like, which are provided from the outside, to the user. The display unit 110 may be implemented as a device such as a CRT, an LCD, or the like.

The imaging unit 120 captures an image of a user. In detail, the imaging unit 120 may capture an image of a user who views a display screen. The imaging unit 120 may be implemented as a photographing device embedded in various types of devices such as a TV, a mobile phone, a notebook PC, and the like. In addition, although the conventional imaging unit 120 is implemented using an IR imaging device such as an IR camera, the imaging unit 120 may be implemented as a conventional USB imaging device according to an embodiment of the present invention. Meanwhile, in the exemplary embodiment of the present invention, the imaging unit 120 is expressed as being embedded in the display apparatus 100, but the imaging unit 120 may be implemented as an external device connected to the display apparatus 100.

The storage unit 130 stores infrared eye image information captured by an infrared camera. The storage unit 130 may store various types of information to be displayed on the display unit 110. In addition, the storage 130 may store an image of the user captured by the imaging unit 120. In addition, the storage unit 130 may store various detection algorithms used in the detection unit 140 to be described later and information about the detected user eye area. In particular, user face feature information may be stored in the face detection process. The storage unit 130 may store algorithms, equations, matrices, user pupil center coordinates, and user gaze position coordinates used by the determination unit 150 to be described later.

The storage unit 130 may be configured of various memories, for example, a ROM, a flash memory, or an HDD. Meanwhile, according to an exemplary embodiment of the present disclosure, the storage unit 130 is represented as being embedded in the display apparatus 100, but the storage unit 130 is an external HDD or a memory card connected to the display apparatus 100, eg, For example, it may be implemented as a flash memory (M / S, XD, SD, etc.) or a USB memory.

The detector 140 detects information on the eyeball area of the user from the captured image. In detail, the detector 140 may sequentially detect the face area, the eye area, the iris area, and the iris center coordinates from the user image picked up by the imaging unit 120, and illustrates a method of detecting the iris center coordinates of the user. It will be described in detail with reference to 3 to 7. Here, the eyeball area may not only mean a pupil area of a dictionary meaning, but may mean an eye and an area near the eye when the face of the person is viewed.

3 is a flowchart illustrating a process of detecting a center coordinate of an iris of a user from a captured user image. The detection process according to FIG. 3 may be performed by the detector 140 of the display apparatus 100 shown in FIG. 1.

Here, the relationship between the iris center and the pupil center will be described. The person moves the eye to see the object. At this time, since the iris exists at a fixed position of the eyeball, it moves along with the eyeball movement. The pupils present in the iris are also changed by the movement of the iris muscles, so they cannot be regarded as having the same positional change with respect to eye movement. The pupil is steadily contracting and expanding by the tension of the body, and the center of the pupil is not constant with the size of the pupil. Therefore, the position of the pupil can be seen as having flexibility. The pupil exists in the central part of the iris, but the center of the pupil does not coincide with the center of the iris. In general, the center of the pupil is located from the center of the iris toward the center of the face.

According to the iris center coordinate detection process illustrated in FIG. 3, a face region is detected from a captured user image using a SM (Saliency Map) model (S310). Subsequently, the eyeball area is detected using the AdaBoost algorithm in the detected face area (S320). The iris region may be detected according to the equalization method and the Otsu threshold method in the detected eye region (S330), and the center coordinates may be detected in the detected iris region (S340).

4 is a flowchart illustrating a process of detecting a face region from a captured user image.

The selective attention model is used to detect the user's face. In detail, a bottom-up SM model may be used to detect a face of a user. The bottom-up SM model may use Retina functions and Lateral Geinculate Nucleus (LGN). Color elements R (Red), G (Green), and B (blue) are extracted using the Retina function, and can be used to detect the user's face along with the Intensity feature. The extracted skin color is filtered and Red can be the dominant contributor to filtering, and the opposite color features of Red and Green can represent the boundary features of the face and non-face areas. Using LGN and Retina functions, I (Intensity), E by performing on-center and off-center operations with different scale Gaussian pyramid images from level 0 to level ⁿ where each level is made by 2 ⁿ . Feature maps such as Edge and C (color) can be constructed. SM can be generated by the sum of three feature maps. In the SM, a relatively protruding region may be selected as a candidate region, two-dimensional data of the selected region may be generated using Otsu's critical method, and a group of divided regions may be generated using a labeling method. The selected candidate area may be used as an input of the AdaBoost (Adaptive Boosting) algorithm to detect the face area.

In addition, the detector 140 may extract face feature information by applying an incremental (2D) ² incremental two directional two dimensional PCA (PCA) technique to the detected face region. An initial column eigenvector and an eigen value are obtained from the covariance of a column data matrix of a plurality of training data, and the eigen vector and eigen value of the column data are obtained using 2DPCA. ) And select a predetermined eigenvector to select the dimension of the columnar representation space.

Here, (2D) ² PCA simultaneously executes 2DPCA in the row direction and column direction to obtain the projection matrix. Then, the feature matrix is obtained by applying the two projection matrices at the same time, so that the size of the feature matrix can be made very small while the recognition rate can be maintained at the level of 2DPCA, and thus the storage space and calculation amount can be remarkably reduced. Incremental (2D) ² PCA means a method of incrementally estimating the expression space and simultaneously executing the 2DPCA in the row direction and the column direction. This is due to the fact that the Incremental PCA is an incremental subspace algorithm that temporally models the change in the feature itself.

In detail, the detector 140 may perform an online learning after measuring the Incremental (2D) ² PCA learning threshold value using the detected face region and the identifier information input by the user (or supervisor).

Here, online learning is a method of efficiently updating an existing learning model using only new learning data, which is also called incremental / adaptive learning. Meanwhile, in the present invention, it has been described that IPCA (Incremental Principal Component Analysis) is applied. Incremental / Online / Adaptive learning, however, requires incremental subspace to model temporal changes in features such as Incremental Linear Discriminant Analysis (ILDA) and Incremental Non-negative Matrix Factorization (INMF) ) Algorithm, and classifiers for temporally modeling changes in feature distribution such as Incremental Support Vector Machine (ISVM) and Incremental Neural Network (INN).

As described above, the feature information of the user's face region displayed in the face region detection step may be stored in the storage unit 130.

The detecting of the eyeball area in the detected face area of FIG. 3 (S320) will be described in detail with reference to FIG. 5.

5 shows the result of detecting the eyeball area from the detected face area. The process of detecting the eyeball area in the detected face area is performed using the Adaboost algorithm, which is well known in the art and description thereof will be omitted.

6 shows the result of detecting the tail of the eye in the detected eye area. Detecting the tail of the eye in the detected eye area (not shown) may be performed using a simple template matching algorithm, which is well known in the art and will not be described. The reason for detecting the tail is necessary because the user's face may move in the process of photographing the user. In order to perform correction by normalizing a user's face movement, the relative distance from the center position of both eye tails to the iris center and the estimated pupil center coordinates described above may be used.

Detecting the center coordinates of the iris region and the iris in the detected eye region of FIG. 3 will be described in detail with reference to FIG. 7.

7 shows a graph in which the eyeball area is projected on the horizontal and vertical axes to detect the iris area and the iris center coordinates. The detecting of the iris region of FIG. 3 (S330) may use an image processing process including a two-dimensionalization process based on the equalization and the Otsu threshold method. Accordingly, a horizontal / vertical projection histogram can be obtained sequentially to detect the iris region in the detected eye region. The horizontal and vertical projection histograms are combined on the horizontal and vertical axes, respectively. Equation 1 shows an equation for obtaining horizontal and vertical projection histograms.

및

는 각각 x축 및 y축의 히스토그램이다. w 및 h는 각각 검출된 안구 영역의 너비 및 높이를 나타낸다. 홍채 영역은

및

에 의해 검출될 수 있다. 구체적으로, 홍채 영역은 다음 수학식 2에 의해 검출될 수 있다.here,

And

Are histograms of the x and y axes, respectively. w and h represent the width and height of the detected eye area, respectively. Iris area

And

As shown in FIG. In detail, the iris region may be detected by Equation 2 below.

,

,

및

는 각각 검출 영역의 가장 좌측 좌표, 가장 우측 좌표, 가장 위쪽 좌표 및 가장 아래쪽 좌표를 나타낸다.here,

,

,

And

Denotes the leftmost, rightmost, topmost and bottommost coordinates of the detection area, respectively.

Detecting the iris center coordinates of the detected iris region (S340) may be performed by the following equation (3).

및

는 각각 홍채 중심 좌표의 x축 좌표 및 y축 좌표를 나타낸다.here,

And

Denote x-axis coordinates and y-axis coordinates of the iris center coordinates, respectively.

Here, although the method of detecting the iris center coordinates of the user has been described with reference to FIGS. 3 to 7 using a conventional USB camera, the method of detecting the iris center coordinates shown in FIGS. The same can be applied to the captured image.

The determination unit 150 of FIG. 1 determines the eyes of the user on the display screen. To this end, the determination unit 150 may be composed of a pupil guessing unit 151 and a gaze determination unit 152.

The pupil estimator 151 estimates the pupil center coordinates of the user. Specifically, the pupil estimation unit 151 detects the detection unit 140 using a neural network to which the MLP algorithm using the center coordinates of the pupil included in the infrared eye image stored in the storage unit 130 as learning data is applied. The pupil center coordinates of the user can be estimated from the obtained iris center coordinates. Here, the center coordinates of the pupil included in the infrared eye image are different from the estimated pupil center coordinates and the pupil center coordinates stored in the storage unit 120, and the pupil center coordinates without the 'infrared' modifier is the pupil guess unit. The pupil center coordinates detected in the infrared eye image are used only for the pupil guessing process by the pupil guessing unit 151, not the pupil center coordinates used in the guessing process of 151.)

Also, preferably, the process of detecting the pupil center coordinates in the infrared eyeball image and the process of learning by the MLP algorithm using the detected pupil center coordinates as learning data may be performed in advance. Specifically, the detector 140 may detect the center coordinates of the pupil on the infrared eye image by using the infrared eye image previously stored in the storage 130, and the pupil center coordinates previously detected on the infrared eye image may be pre-recorded. It may be stored. Then, before estimating the pupil center coordinates, the pupil estimator 151 may pre-learn using the pupil center coordinates on the infrared image as learning data of the MLP algorithm.

Here, the neural network is a mathematical model that aims to represent some of the characteristics of the brain function by computer simulation. A neural network refers to a model in which artificial neurons (nodes) that form a network by synapse change change the strength of synapse through learning, and have problem solving ability. In a narrow sense, it may refer to multi-layer perceptron (MLP) using error backpropagation, but this is a misuse, and neural networks are not limited to this. Neural networks have teacher learning that is optimized for problems by the input of teacher signals (correct answers) and comparative learning that does not require teacher signals. Teacher learning is used when there is clear answer, and comparative learning is used for data clustering. As a result, in order to reduce the dimensionality of all, it is often the case that a good answer can be obtained with a comparatively small amount of calculation with respect to a problem that can not be linearly separated by data of a multidimensional quantity such as an image or statistics. Therefore, it is applied in various fields such as pattern recognition and data mining.

8A shows a flowchart for detecting pupil center coordinates in an infrared eye image. The detecting of the pupil center coordinates may include a histogram equalization step, a histogram binarization step, a dilation step, an erosion step, and a blob labeling step. The histogram equalization process may serve as a contrast adjustment of the grayscale eye image using the histogram of the image. And, in the histogram boundary process, an edge boundary using an appropriate boundary value may be applied. And mathematical morphology operations such as dilation and erosion can be used to reduce errors caused by glint and noise around the pupil. Finally, a blob labeling process is performed on the eyeball image, and the pupil area and the center coordinate of the pupil may be detected by finding an appropriate color part corresponding to the pupil. Figure 8b shows the results of finding the pupil center in the infrared eye image.

9 shows the structure of a neural network for estimating pupil center coordinates. The neural network can use one hidden layer containing seven nodes and a bias. The input vector of the neural network may consist of two-dimensional coordinates of the iris center, and the target vector may consist of two-dimensional coordinates of the pupil center obtained from the IR camera. In addition, the neural network may be divided into a training phase and a test phase.

10A is a flowchart illustrating an operation process of a training phase of a neural network.

The training phase can learn in advance about the relationship between the iris center and the pupil center before estimating the pupil center coordinates in the test phase to be described above. In the training phase, the neural network can normalize the detected iris center coordinates and the pupil center coordinates on the infrared image using the detected center coordinates of both eyes. The normalized iris center coordinates are given as input vectors, the pupil center coordinates on the normalized infrared image are given as target vectors, and the relationship between the iris center and the pupil center can be repeatedly learned until the target accuracy is satisfied.

10B is a flowchart illustrating an operation process of a test phase of a neural network.

In the test phase, the pupil center coordinates can be estimated using the iris center coordinates. The test phase normalizes the detected iris center coordinates using the eye-tail center coordinates, and the normalized iris center coordinates are used as the input vector. The neural network outputs pupil center coordinates as an output vector using previously learned data.

The gaze determination unit 152 determines the gaze of the user on the display screen. In detail, the gaze determination unit 152 may determine the gaze position of the user on the display screen through a calibration process of the pupil center coordinates estimated by the pupil estimating unit 151. For calibration, the user needs to enter the data in advance. For example, when a user gazes at a certain point on the display screen and presses an input key, a relationship with respect to the pupil position coordinates corresponding to the gaze position coordinates on the display screen may be derived. 11 shows calibration from the pupil center coordinates onto the display screen.

In order to track the gaze to the display screen using the pupil center coordinates estimated by the pupil guessing unit 151, a second order polynomial function such as Equation 4 may be considered.

내지

및

내지

는 미정 계수이다. 이러한 시스템은 12개의 미지수를 가지므로 6번 이상의 사용자 입력을 받음으로써 쉽게 풀 수 있다. 예를 들어, 칼레브레이션을 위해 9개의 점을 사용한다면, 이 시스템은 12개의 미지수 및 18개의 식으로 이루어진 과잉 결정되는 시스템이 되어 쉽게 계수들을 결정할 수 있다. 더 높은 차수의 다항식을 사용하면 정확성을 더 높일 수 있지만, 2차 다항식으로도 응시점의 칼리브레이션을 수행하는데 충분한 근사치로 수행될 수 있다.

To

And

To

Is an undetermined coefficient. Such a system has 12 unknowns, which can be easily solved by receiving more than six user inputs. For example, if we use nine points for calibration, the system becomes an overdetermined system of twelve unknowns and eighteen equations, making it easy to determine coefficients. Higher order polynomials provide better accuracy, but even quadratic polynomials can be approximated enough to perform calibration of the gaze point.

The relationship between the coefficients may be represented by a matrix as shown in Equation 5 below.

로 간단하게 나타낼 수 있다.The matrix is

Can be represented simply.

를 얻을 수 있다. 칼리브레이션 매트릭스

를 이용하여 추측된 동공 중심 좌표로부터 디스플레이 화면상의 사용자의 시선 위치를 칼리브레이션 할 수 있다. here,

Can be obtained. Calibration matrix

The gaze position of the user on the display screen may be calibrated from the estimated pupil center coordinates.

Also, when the gaze determination unit 152 compares the derived pupil center coordinates with the pupil center coordinates stored in the storage unit 130 and matches each other, the user's gaze on the display screen corresponding to the stored pupil center coordinates without the calibration process. May be determined by the user's gaze. As a result, the user's gaze position on the display screen can be more efficiently determined.

Here, when the current user is identified by the controller 160 to be described later, the gaze determination unit 152 may determine the pupil center coordinates corresponding to the identified user among the pupil center coordinates stored in the storage unit 130. The pupil center coordinates estimated at 151 can be compared. That is, when there are several users, the process of determining the gaze may be simplified by identifying the current user and comparing only the stored information corresponding to the user.

The controller 160 controls the operation of the display 110. According to an embodiment, when the user's gaze is located on an application icon displayed on a display screen, the controller 160 may execute an application corresponding to the application icon and display an application UI.

In addition, the controller 160 may identify the current user corresponding to the image captured by the imaging unit 120 using the feature information of the face regions of the users stored in the storage 130. Specifically, by comparing the feature information of the face area detected in the current user image and the feature information of each user's face area stored in the storage unit 130, a user corresponding to the feature information of the matching face area is used as the current user. You can judge.

Here, the controller 160 calculates a face recognition result by applying a gradual classifier based on the extracted face feature. Specifically, face recognition results may be calculated using the weak classifier and the strong classifier. As an example, the weak classifier may be implemented as a support vector machine (SVM), and the strong classifier may be implemented as an adaboosted SVM.

Adaboosted SVM is a classifier generated by applying Adaboost technique to SVM which is weak classifier. The Adaboost technique is an algorithm for generating a strong classifier with high detection performance through linear combination of weak classifiers. The weak classifier may be trained by combining feature information extracted from the detected face region with pre-classified recognizer information, and a strong classifier calculated by applying an adaboost may be generated by applying a weight set for each recognizer information according to the training result. In this way, the classification method using a drug classifier and a strong classifier is called a binary classification technique.

In addition, the controller 160 controls the operations of the display 110, the image capturer 120, the storage 130, the detector 140, and the determiner 150 of the display apparatus 100. In detail, the controller 160 may control the overall operation of the display apparatus 100 using various control programs and algorithms stored in the storage 130.

Accordingly, the display apparatus according to the present embodiment may determine the gaze position of the user on the display screen from the captured user image, and control the display apparatus by using the determined gaze position of the user.

2 is a block diagram illustrating a configuration of an eye gaze tracking apparatus 200 according to an exemplary embodiment. The gaze tracking device 200 is a device for determining the gaze of the user on the display screen by using the captured image of the user. The gaze tracking device 200 may include a detector 240 and a determiner 250. The detector 240 and the determiner 250 of the eye tracking apparatus 200 may correspond to the detector 140 and the determiner 150 of the display apparatus 100, respectively.

The detector 240 detects information about an eyeball area of the user from the captured image. Specifically, the detection unit 240 may sequentially detect the face area, the eye area, the iris area, and the iris center coordinates from the captured user image, and a method of detecting the iris center coordinates of the user is shown in FIGS. 3 to 7. . Here, the eyeball area may not only mean a pupil area of a dictionary meaning, but may mean an eye and an area near the eye when the face of the person is viewed. The process of detecting the iris center coordinates by the detector 240 is the same as the process of detecting the iris center coordinates by the detector 140 of the display apparatus 100.

The determination unit 250 determines the eyes of the user on the display screen. To this end, the determination unit 240 may be composed of a pupil guessing unit 251 and a gaze determination unit 252. The pupil guessing unit 251 and the gaze determination unit 252 of the determination unit 240 correspond to the pupil estimation unit 151 and the gaze determination unit 152 of the determination unit 150 of the display apparatus 100.

The pupil estimator 251 estimates the pupil center coordinates of the user. In detail, the pupil estimator 251 detects the detection unit 240 using a neural network to which the MLP algorithm using the center coordinates of the pupil included in the infrared eye image stored in the storage 230 as learning data is applied. The pupil center coordinates of the user may be estimated from the iris center coordinates, and the process of estimating the pupil center coordinates is the same as the process of estimating the pupil center coordinates by the pupil guessing unit 151 of the display apparatus 100.

The gaze determination unit 252 determines the gaze of the user on the display screen. In detail, the pupil center coordinates estimated by the pupil estimator 251 may determine the position of the user's gaze on the display screen through a calibration process. The process of determining the gaze position of the user on the display screen is the same as the process of determining the gaze position of the user on the display screen by the gaze determination unit 152 of the display apparatus 100.

Accordingly, the gaze tracking apparatus according to the present exemplary embodiment may determine the gaze position of the user on the display screen from the captured user image.

13 is a flowchart illustrating a display method according to an exemplary embodiment. According to FIG. 13, when the display method provides a display screen (S1310), an image of a user viewing the display screen is captured (S1320), and information about an eyeball area of the user is detected from the captured image (S1330). Subsequently, by using the detected information on the eyeball area and the pre-stored infrared eyeball image, the pupil position in the eyeball area is estimated to determine the user's gaze (S1340). In operation S1350, the operation of the display apparatus is controlled according to the line of sight of the user.

Referring to FIGS. 1, 3, and 13, the display method will be described in detail. The display 110 includes information previously stored in the storage 150 or an externally provided video, photo, document, UI, icon, or the like. Various information is displayed to the user (S1310). Then, the imaging unit 120 captures an image of the user viewing the display screen (S1320). The detector 140 detects information on the eyeball area of the user from the captured image (S1330). In detail, the detector 140 may sequentially detect the face area, the eyeball area, the iris area, and the iris center coordinates from the user image captured by the imaging unit 120. Referring to FIG. 3, the process of detecting the iris center coordinates of the user includes detecting a face region from the captured user image (S310), detecting an eye region from the detected face region (S320), and detecting the detected eye region. The method may include detecting an iris region (S330) and detecting an iris center coordinate of the detected iris region (S340).

 The determination unit 150 determines the gaze of the user by guessing the pupil position in the eyeball area using the information on the eyeball area detected by the detector 140 and the pre-stored infrared eyeball image (S1340). To this end, the determination unit 150 may be composed of a pupil guessing unit 151 and a gaze determination unit 152. The pupil estimator 151 uses an neural network to which the MLP algorithm using the pupil coordinates included in the infrared eye image stored in the storage 130 as learning data is applied, and the iris center detected by the detector 140. The coordinates of the pupil center of the user can be estimated from the coordinates. Here, the process of detecting the pupil center coordinates from the infrared eye image and the process of learning by the MLP algorithm using the detected pupil center coordinates as learning data may be performed in advance. The gaze determination unit 152 may determine the gaze position of the user on the display screen by calibrating the pupil center coordinates estimated by the pupil estimating unit 151. Also, when the gaze determination unit 152 compares the derived pupil center coordinates with the pupil center coordinates stored in the storage unit 130 and matches each other, the user's gaze on the display screen corresponding to the stored pupil center coordinates without the calibration process. It may be determined by the eyes of the user, and if the current user is identified by the controller 160, the pupil guessing unit to determine the pupil center coordinates corresponding to the identified user from the pupil center coordinates stored in the storage unit 130 It can be compared with the pupil center coordinates estimated at 151.

The controller 160 controls the operation of the display 110 (S1350). According to an embodiment, when the user's gaze is located on an application icon displayed on a display screen, the controller 160 may execute an application corresponding to the application icon and display an application UI. In addition, the controller 160 may identify the current user corresponding to the image captured by the imaging unit 120 using the feature information of the face regions of the users stored in the storage 130.

Therefore, the display method according to the present exemplary embodiment may determine the gaze position of the user on the display screen from the captured user image, and control the display apparatus by using the determined gaze position of the user.

14 is a flowchart illustrating a gaze tracking method according to an exemplary embodiment. According to FIG. 14, a gaze tracking method detects information on an eyeball area of a user from a captured image (S1430), and estimates a pupil position in an eyeball area using information on the detected eyeball area and a pre-stored infrared eyeball image. The eyes of the user are determined (S1440). Each step of configuring the gaze tracking method corresponds to steps S1330 and S1340 of the flowchart illustrating the display method of FIG. 13, and is performed in the same manner.

Accordingly, the gaze tracking method according to the present exemplary embodiment may determine the gaze position of the user on the display screen from the captured user image.

12A and 12B illustrate a gaze tracking result using the gaze tracking device 200 according to an exemplary embodiment.

FIG. 12A illustrates a screen obtained by dividing a display screen having a resolution of 800 * 600 into 7 * 5 areas to track the gaze using the gaze tracking apparatus 200. That is, the test on the screen having a resolution of 7 * 5. At the center of each divided area, a target for eye tracking is located. When the user looks at each target, the gaze tracking device 200 determines the user's gaze position on the display screen. Table 1 below shows the results of 10 tests on each target. Each number in Table 1 is between the position of the target on the display screen and the user's gaze position determined using the iris center coordinates, the pupil center coordinates detected by the infrared image, and the estimated pupil center coordinates by the gaze tracking device 200. The mean and the variance of the error are shown. The distance to the target was measured by Manhattan distance.

From the results shown in Table 1, the user's gaze determination using the pupil center coordinates detected by the infrared image has the highest accuracy. The user's gaze determination using the iris center coordinates has the lowest accuracy. The result of judging the user's line of sight using the estimated pupil center shows satisfactory accuracy, although less than the result using the pupil center coordinates detected by the infrared image.

Figure 12b shows the test results on the screen with a resolution of 800 * 600 for a more accurate test. Ten tests of 35 target locations each were tested. Each number in Table 2 is between the position of the target on the display screen and the user's gaze position determined using the iris center coordinates, the pupil center coordinates detected by the infrared image, and the estimated pupil center coordinates by the gaze tracking device 200. Represents the root mean square of the error. The distance to the target was measured by Manhattan distance.

As shown in Table 1, the results of Table 2 show that the test using the pupil center coordinates detected by the infrared image shows the most accurate result, and the test using the iris center coordinates shows the most inaccurate result. The results obtained with conventional USB cameras are less than the test results by pupil center coordinates detected with infrared images, but show satisfactory accuracy without any additional equipment such as IR cameras to track the user's eye.

The eye-tracking method and display method according to various embodiments of the present disclosure described above may be implemented by program codes stored in various types of recording media and executed by a CPU included in the electronic device.

Specifically, the code for performing the above-described eye tracking method and display method may include random access memory (RAM), flash memory, read only memory (ROM), erasable programmable ROM (EPROM), and electronically erasable and programmable ROM (EPEROM). ), A register, a hard disk, a removable disk, a memory card, a USB memory, a CD-ROM, and the like, may be stored in various types of recording media readable by the terminal.

Although the above has been illustrated and described with respect to the preferred embodiments of the present invention, the present invention is not limited to the above-described specific embodiments, it is usually in the art without departing from the spirit of the invention claimed in the claims. Anyone skilled in the art can make various modifications, as well as such changes are within the scope of the claims.

100: display device 110: display part
120: imaging unit 130: storage unit
140: detection unit 150: determination unit
151: pupil guessing unit 152: gaze determination unit
160: control unit 200: gaze tracking device
240 detection unit 250 determination unit
251: pupil guessing unit 252: gaze determination unit

Claims (24)

In the display device,
A display unit providing a display screen;
An imaging unit for capturing an image of a user viewing the display screen;
A detector for detecting information about an eyeball area of a user from the image picked up by the image pickup unit;
A storage unit for storing infrared eye image information to be used as learning data;
A determination unit determining the eye gaze of the user by estimating a pupil position in the eye region using information on the eye region detected by the detection unit and the stored infrared eye image information; And
And a controller configured to control an operation of the display apparatus according to the eyes of the user. The method of claim 1,
The information on the eye region is characterized in that the iris center coordinates,
Wherein:
And detecting a face region from the captured image, detecting an eye region from the detected face region, and detecting an iris region from the detected eye region, thereby detecting a center coordinate of the detected iris region. The method of claim 2,
The infrared eyeball image information is characterized in that the center coordinates of the pupil contained in the infrared eyeball image,
The determination unit,
A pupil guessing unit for estimating the pupil center coordinates of the user from the detected iris center coordinates using a neural network to which an MLP algorithm using the pupil center coordinates as learning data is applied; And
And a gaze determination unit configured to determine a gaze of a user on a display screen by using the estimated pupil center coordinates. The method of claim 3,
The gaze determination unit,
Display device, characterized in that for determining the eyes of the user on the display screen using the following equation:

here,

And

Is the x- and y-axis coordinates of the user's line of sight on the display screen,

And

Is the x-axis and y-axis coordinates of the pupil center coordinates of the user,

To

And

To

Is the unknown coefficient of the calibration matrix. 5. The method of claim 4,
The control unit,
And when the gaze of the user is located on an application icon displayed on the display screen, executing the application corresponding to the application icon to display an application UI. The method of claim 3,
Wherein,
Further storing a pupil center coordinate derived from the eyeball area of the user and a user gaze coordinate corresponding to the pupil center coordinate,
The gaze determination unit,
The display apparatus, characterized in that the user's gaze on the display screen corresponding to the stored pupil center coordinates is determined as the user's gaze when the pupil center coordinates estimated by the pupil guessing unit match with the stored pupil center coordinates. . The method according to claim 6,
Wherein,
Further stores feature information of a face area for each of the plurality of users,
The control unit,
Identify the user corresponding to the captured image by using the feature information of the stored face region,
The gaze determination unit,
And a pupil center coordinate estimated by the pupil guessing unit with a pupil center coordinate corresponding to the identified user among the stored pupil center coordinates. In the eye tracking apparatus,
A detector for detecting information about an eyeball area of a user from the captured user image;
A storage unit for storing infrared eye image information to be used as learning data; And
And a determination unit determining the eye gaze of the user by estimating a pupil position in the eye region using the information on the eye region detected by the detection unit and the stored infrared eyeball image information. 9. The method of claim 8,
The information on the eye region is characterized in that the iris center coordinates,
Wherein:
A gaze tracking device for detecting a face region in the captured image, detecting an eye region in the detected face region, detecting an iris region in the detected eye region, and detecting a center coordinate of the detected iris region . 10. The method of claim 9,
The infrared eyeball image information is characterized in that the center coordinates of the pupil contained in the infrared eyeball image,
The determination unit,
A pupil guessing unit for estimating the pupil center coordinates of the user from the detected iris center coordinates using a neural network to which an MLP algorithm using the pupil center coordinates as learning data is applied; And
A gaze determination unit configured to determine a gaze of a user on a display screen using the estimated pupil center coordinates. The method of claim 10,
Wherein,
Further storing a pupil center coordinate derived from the eyeball area of the user and a user gaze coordinate corresponding to the pupil center coordinate,
The gaze determination unit,
When the pupil center coordinates estimated by the pupil guessing unit coincide with each other by comparing the stored pupil center coordinates, a gaze tracking method of determining a user's gaze on a display screen corresponding to the stored pupil center coordinates as the gaze of the user. Device. 12. The method of claim 11,
Wherein,
Further stores feature information of a face area for each of the plurality of users,
The gaze determination unit,
Identify the user corresponding to the captured image by using the feature information of the stored face region, and determine the pupil center coordinates estimated by the pupil guessing unit with the pupil center coordinates corresponding to the identified user among the stored pupil center coordinates. A gaze tracking device, characterized in that the comparison. In the display method,
Providing a display screen;
Imaging an image of a user viewing the display screen;
Detecting information about an eyeball area of a user from the captured image;
Determining the gaze of the user by inferring a pupil position in the eyeball area using information on the detected eyeball area and pre-stored infrared eyeball image information to be used as learning data; And
And controlling an operation of the display apparatus according to the gaze of the user. The method of claim 13,
The information on the eye region is characterized in that the iris center coordinates,
Wherein the detecting comprises:
Detecting a face area in the captured image;
Detecting an eyeball area in the detected face area;
Detecting an iris region in the detected eye region; And
Detecting iris center coordinates of the detected iris region. 15. The method of claim 14,
The infrared eyeball image information is characterized in that the center coordinates of the pupil contained in the infrared eyeball image,
The determining step,
Inferring the pupil center coordinates of the user from the detected iris center coordinates using a neural network to which an MLP algorithm using the pupil center coordinates as learning data is applied; And
And determining the gaze of the user on the display screen using the estimated pupil center coordinates. 16. The method of claim 15,
The determining of the gaze may include
Display method, characterized in that to determine the eyes of the user on the display screen using the following formula:

here,

And

Is the x- and y-axis coordinates of the user's line of sight on the display screen,

And

Is the x-axis and y-axis coordinates of the pupil center coordinates of the user,

To

And

To

Is the unknown coefficient of the calibration matrix. 17. The method of claim 16,
Wherein the controlling comprises:
And when the gaze of the user is located on an application icon displayed on the display screen, executing the application corresponding to the application icon to display an application UI. 16. The method of claim 15,
Storing pupil center coordinates derived from the eyeball area of the user and user gaze coordinates corresponding to the pupil center coordinates;
Determining the eyes of the user on the display screen,
In the step of estimating the pupil center coordinates, if the estimated pupil center coordinates match with the stored pupil center coordinates, the user's gaze on the display screen corresponding to the stored pupil center coordinates is determined as the user's gaze. Display method. 19. The method of claim 18,
Storing feature information of the face area for each of the plurality of users;
Wherein the controlling comprises:
Identify the user corresponding to the captured image by using the feature information of the stored face region,
Determining the eyes of the user on the display screen,
And displaying the estimated pupil center coordinates with the pupil center coordinates corresponding to the identified user among the stored pupil center coordinates. In the eye tracking method,
Detecting information about an eyeball area of a user from the captured image; And
And determining the gaze of the user by estimating a pupil position in the eyeball area using information on the detected eyeball area and pre-stored infrared eyeball image information to be used as learning data. 21. The method of claim 20,
The information on the eye region is characterized in that the iris center coordinates,
Wherein the detecting comprises:
Detecting a face area in the captured image;
Detecting an eyeball area in the detected face area;
Detecting an iris region in the detected eye region; And
Detecting the iris center coordinates of the detected iris region. The method of claim 21,
The infrared eyeball image information is characterized in that the center coordinates of the pupil contained in the infrared eyeball image,
The determining step,
Inferring the pupil center coordinates of the user from the detected iris center coordinates using a neural network to which an MLP algorithm using the pupil center coordinates as learning data is applied; And
And determining the gaze of the user on the display screen using the estimated pupil center coordinates. The method of claim 22,
Storing pupil center coordinates derived from the eyeball area of the user and user gaze coordinates corresponding to the pupil center coordinates;
Determining the eyes of the user on the display screen,
In the step of estimating the pupil center coordinates, if the estimated pupil center coordinates match with the stored pupil center coordinates, the user's gaze on the display screen corresponding to the stored pupil center coordinates is determined as the user's gaze. Eye tracking method. 24. The method of claim 23,
Storing feature information of the face area for each of the plurality of users;
Determining the eyes of the user on the display screen,
Identify the user corresponding to the captured image by using the feature information of the stored face region, and compare the estimated pupil center coordinates with the pupil center coordinates corresponding to the identified user among the stored pupil center coordinates. Eye tracking method.

Images