KR101769177B1 - Apparatus and method for eye tracking - Google Patents

Abstract

The three-dimensional position of the user is calculated using two or more wide angle cameras, and a pan, a tilt, and a focus operation are performed using the position information and the eye area detection information of the user using the narrow- And a gaze tracking apparatus and method for calculating gaze position information from pupil center information and corneal reflection information after detecting pupil center information and corneal reflection information from an eye image obtained through operation of a coarse camera do.

Description

[0001] APPARATUS AND METHOD FOR EYE TRACKING [0002]

Embodiments of the present invention relate to a line-of-sight tracking apparatus and method for acquiring accurate line-of-sight information even when a user moves freely at a long distance.

There are two dimensional mapping function based method and three dimensional model based method to find eye position in eye tracking technology.

Some of the 3D model-based methods guarantee free movement of the user, but complex system calibration is required and the distance between the user and the screen may be less than 1m.

The method based on 2D mapping function can be used not only near distance tracking but also far distance tracking and does not require complicated system calibration but it can increase eye position error when users are out of calibration position.

In the field of distance tracking technology, it is necessary to develop a technique for reducing gaze position error due to free user movement and providing accurate gaze tracking information.

An object of the present invention is to provide an apparatus and method capable of tracking a line of sight by using a narrow angle camera and tracking a line even if the user moves away from a position where the user performs calibration.

The gaze tracking apparatus according to an embodiment of the present invention includes a calibration performing unit for performing calibration between two or more wide angle cameras, a narrow angle camera, and an image reproducing apparatus to calculate user position information for a user, A first information detecting unit for detecting eye position information on the image information, a pan for the narrow-angle camera based on the eye position information and the user position information, A second information detecting unit for detecting pupil center information on the center of the pupil from the eye image, a second information detecting unit for detecting the pupil center information of the cornea reflected from the eye of the user, A third information detecting section for detecting corneal reflected light information on the reflected light, It includes parts of gaze position calculation of calculating the eye position information from the reflected light information.

According to an aspect of the present invention, the calibration performing unit performs calibration between the at least two wide angle cameras, performs calibration between the at least two wide angle cameras and the narrow angle camera, and performs calibration between the narrow angle camera and the image playback device can do.

According to an aspect of the present invention, the first information detecting unit may detect a face area and an eye area with respect to the at least two pieces of image information.

According to an aspect of the present invention, when the user moves from a position at which the user performs calibration between two or more wide-angle cameras, the gaze position calculation unit applies the user position information to the forward motion compensation model, The gaze position information can be calculated using the user position information.

The gaze tracking apparatus according to another exemplary embodiment of the present invention includes at least two wide angle cameras for receiving image information and calculating user position information for a user through calibration, ), A tilt and focus operation, an image acquisition unit for acquiring an eye image through operation of the narrow-angle camera, a detector for detecting pupil center information and corneal reflection information from the eye image, And a gaze position calculation unit for calculating gaze position information from the pupil center information and the corneal reflection information.

The gaze tracking apparatus according to another aspect of the present invention may further include a calibration performing unit for performing calibration between the at least two wide angle cameras.

According to another aspect of the present invention, the calibration performing unit may estimate a position and a degree of rotation of another camera based on one wide-angle camera using a stereo calibration method.

According to another aspect of the present invention, the calibration performing unit may perform calibration between the at least two wide angle cameras and the narrow angle cameras.

According to another aspect of the present invention, the calibration performing unit may perform calibration between the narrow-angle camera and the image reproducing apparatus.

According to another aspect of the present invention, the calibration performing unit may estimate a difference between the center of the fan axis and the tilt axis of the narrow angle camera and the actual position of the image reproducing apparatus.

According to another aspect of the present invention, the gaze position calculation unit may calculate the gaze position information through a forward motion compensation model that corrects the gaze position according to the movement of the user.

According to another aspect of the present invention, the image acquiring unit may include a resolution adjusting unit for adjusting a resolution of the input image information, a reflected light removing unit for removing reflected light of the adjusted image information, And a pupil candidate region detecting unit for detecting a pupil candidate region by applying a filter.

According to another aspect of the present invention, the image acquisition unit includes an area selection unit for selecting two areas of different sizes in the pupil candidate area, a threshold value detection unit for detecting respective threshold values for the two areas, And comparing the two threshold values with each other to determine that there is no eyelid image in the eye image.

According to another aspect of the present invention, the image acquiring unit further includes a pupil center detecting unit that detects circular information from the pupil candidate region, and performs ellipse fitting of the detected circular information to detect the pupil center information .

According to another aspect of the present invention, the pupil center detecting unit includes a circular detecting unit for identifying a center and an outer contour of a circle by applying a circle detection method in the pupil candidate region, a position detecting unit for detecting a position A position shifting unit for shifting the selected plurality of points to a position having a largest differential value with respect to a brightness value on a straight line connecting the selected plurality of points and the center of the circle, And an elliptical fitting unit for performing elliptical fitting using a plurality of weighted points and detecting the pupil center information based on the weights.

According to another aspect of the present invention, the image obtaining unit may calculate a pan angle and a tilt angle for photographing the user based on the three-dimensional position of the user.

According to another aspect of the present invention, the image obtaining unit may include a first estimator for estimating a two-dimensional coordinate of a virtual image corresponding to the pan angle and the tilt angle, a first estimator for estimating a three- A second estimator for estimating a virtual coordinate in which the three-dimensional coordinate is projected on the virtual image by referring to a relative positional relationship, and a third estimator for estimating the pan angle and the tilt angle based on the virtual coordinates .

The gaze tracking apparatus according to another aspect of the present invention may further include an image correcting unit for correcting an error of the gaze position information caused by the pupil of the user.

A method of tracking a line of sight according to an exemplary embodiment of the present invention includes calibrating each of two or more wide-angle cameras, a narrow-angle camera, and an image reproducing apparatus to calculate user position information for a user, The method of claim 1, further comprising the steps of: receiving image information through at least one of the eye position information and the user position information; detecting eye position information with respect to the image information; detecting pupil center information on the center of the pupil from the eye image, detecting corneal reflection information on corneal reflection light reflected on the eye of the user, And calculating gaze position information from the pupil center information and the corneal reflected light information.

The gaze tracking method according to another embodiment of the present invention includes the steps of receiving image information through two or more wide angle cameras, calculating user position information for a user using the two or more wide angle cameras, Performing pan, tilt, and focus operations using user position information and eye area detection information; acquiring an eye image through operation of the narrow-angle camera; Detecting pupil center information and corneal reflection information, and calculating gaze position information from the pupil center information and the corneal reflected light information.

According to the embodiment of the present invention, it is possible to track a distance by using a narrow-angle camera and to observe a line-of-sight even if the user deviates from the position where the user performs the calibration.

1 is a block diagram illustrating a configuration of a gaze tracking apparatus according to an embodiment of the present invention.
FIG. 2 is a view showing a physical configuration of a gaze tracking apparatus according to an embodiment of the present invention.
3 is a diagram illustrating a detailed configuration of a gaze tracking apparatus according to an embodiment of the present invention.
4 is a conceptual diagram for performing a calibration method of the wide angle camera and the narrow angle camera.
5 is a conceptual diagram for performing a method of calibrating a wide angle camera and a narrow angle camera according to an aspect of the present invention.
6 is a block diagram showing the configuration of an image acquisition unit according to an aspect of the present invention.
7 is a diagram showing an example of reflected light removal according to one aspect of the present invention.
FIG. 8 shows an example in which a detection filter is applied to a pupil candidate region, and FIG. 9 shows an example of a result of detecting a pupil candidate region.
10 is a view showing an example of a large candidate region and a small candidate region in the vicinity of the pupil, and FIG. 11 is a diagram showing a result of applying the Otsu threshold method to the pupil candidate region.
12 is a block diagram showing the detailed configuration of a pupil center detecting unit according to an aspect of the present invention.
13 is a view showing the configuration of points for elliptic fitting and the results of elliptical fitting.
FIGS. 14 and 15 are diagrams showing the relationship between the screen of the image reproducing apparatus, the narrow-angle camera, and the pupil according to the movement of the user.
16 is a diagram showing the relationship between the user calibration position and the current position.
17 is a flowchart illustrating a gaze tracking method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and accompanying drawings, but the present invention is not limited to or limited by the embodiments.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The terminology used herein is a term used for appropriately expressing an embodiment of the present invention, which may vary depending on the user, the intent of the operator, or the practice of the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.

1 is a block diagram illustrating a configuration of a gaze tracking apparatus according to an embodiment of the present invention.

1, the eye tracking apparatus 100 according to an exemplary embodiment of the present invention includes a calibration execution unit 110, an image information input unit 120, a first information detection unit 130, an image acquisition unit 140, A second information detecting unit 150, a third information detecting unit 160, and a gaze position calculating unit 170.

The calibration execution unit 110 performs calibration between two or more of the wide angle camera, the narrow angle camera, and the image reproducing apparatus to calculate user position information for the user. For example, the calibration performing unit 110 may perform calibration between two or more wide angle cameras, perform calibration between two or more wide angle cameras and a narrow angle camera, and perform calibration between the narrow angle camera and the image reproducing apparatus can do.

The image information input unit 120 receives image information through two or more wide angle cameras, and the first information detection unit 130 detects eye position information on the image information. For example, the first information detection unit 130 can detect the face region and the eye region with respect to two or more pieces of image information. The image obtaining unit 140 performs pan, tilt, and focus operations on the coarse camera based on the eye position information and the user position information to obtain eye images.

The second information detecting unit 150 detects pupil center information on the pupil center from the eye image, and the third information detecting unit 160 detects the corneal reflected light information on the cornea reflected light reflected by the user's eyes.

The gaze position calculation unit 170 calculates gaze position information from the pupil center information and the corneal reflection information. For example, when the user moves from a position where calibration between two or more wide-angle cameras is performed, the gaze position calculation unit 170 applies user position information to the forward motion compensation model to correct the gaze position, Can be used to calculate gaze position information.

In addition, the gaze tracking device may further include an image correction unit 180, and the image correction unit 180 may correct the error of the line-of-sight position information due to the pupil of the user's eyes.

FIG. 2 is a view showing a physical configuration of a gaze tracking apparatus according to an embodiment of the present invention.

2, the eye tracking apparatus according to an exemplary embodiment of the present invention may include two or more wide-angle cameras 210, a narrow-angle camera 220, and two or more infrared lights 230.

The gaze tracking apparatus according to an embodiment of the present invention can calculate the three-dimensional position of the user through the calibration of the dual wide-angle camera 210 and can calculate the three-dimensional position of the user through the narrow angle camera 220, And can perform a pan-tilt-focus operation. The gaze tracking device can acquire a high-quality eye image through the above-described operation, and can calculate gaze position information by detecting pupil center information and illumination reflected light information of the acquired eye. The calculated gaze position information is provided to the image reproducing apparatus 240, and the image reproducing apparatus 240 can reproduce an image corresponding to the user's gaze.

3 is a diagram illustrating a detailed configuration of a gaze tracking apparatus according to an embodiment of the present invention.

3, the gaze tracking apparatus includes at least two wide angle cameras 310, a narrow angle camera 320, an image acquisition unit 340, a detection unit 350, and a gaze position calculation unit 360.

The two or more wide angle cameras 310 receive the image information and calculate the three-dimensional position of the user with respect to the user through the calibration. The narrow angle camera 320 uses the user position information and the eye area detection information to generate a pan, A tilt, and a focus operation.

The eye acquiring unit 340 acquires the eye image through the operation of the narrow-angle camera, the detecting unit 350 detects the pupil center information and the corneal reflection information from the eye image, and the eye-gaze position calculating unit 360 calculates eye- Eye position information is calculated from the corneal reflection information.

The gaze position calculation unit 360 can calculate gaze position information through a forward motion correction model that corrects the gaze position according to the user's motion.

The gaze tracking apparatus may further include a calibration performing unit 330 for performing calibration between the cameras and the image reproducing apparatuses.

The calibration performing unit 330 may perform calibration between the respective wide-angle cameras 310. For example, the calibration performing unit 330 may estimate the position and the degree of rotation of another camera based on one wide-angle camera using the stereo calibration method.

According to one aspect of the present invention, a gaze tracking device arranges check boards at various positions for calibration between wide angle cameras 310, photographs a check board using two wide angle cameras, Can be performed.

In addition, the calibration performing unit 330 may perform calibration between two or more wide angle cameras 310 and a narrow angle camera 320. For example, the calibration performing unit 330 may calculate a pan angle and a tilt angle for the narrow camera 320 to photograph the user based on the three-dimensional position of the user.

4 is a conceptual diagram for performing a calibration method of the wide angle camera and the narrow angle camera.

4, the gaze tracking device may perform calibration between the wide angle camera 420 and the narrow angle camera 430 to estimate position information and rotation information between the wide angle camera 420 and the narrow angle camera 430 .

According to one aspect of the present invention, in the case of calibration between the wide angle camera 420 and the coarse angle camera 430, the coarse angle camera 430 performs a pan / tilt operation to photograph all of the check boards as the user 410 moves Therefore, the coarse camera 430 must be moved.

5 is a conceptual diagram for performing a method of calibrating a wide angle camera and a narrow angle camera according to an aspect of the present invention.

Referring to FIG. 5, when the gaze tracking device gives input of the three-dimensional position of the user 510 that is found by using the wide angle camera 520, the coarse angle camera 530 detects a fan The angle and the tilt angle can be estimated.

The calibration performing unit 330 of FIG. 3 may calculate the pan angle and the tilt angle using the first estimator, the second estimator, and the third estimator.

The first estimator estimates the two-dimensional coordinates of the virtual image corresponding to the pan angle and the tilt angle, and the second estimator refers to the relative positional relationship between the three-dimensional coordinates and the two-dimensional coordinates of the three- The virtual coordinates projected on the image can be estimated. The third estimator may estimate the pan angle and the tilt angle based on the virtual coordinates.

The gaze tracking device measures the pan angle and the tilt angle of the coarse camera 530 as shown in FIG. 5 in order to determine the relationship between the three-dimensional position of the user 510 and the actually moved angle of the coarse angle camera 530, It can be converted into x, y coordinates of a dimensional image.

The gaze tracking device can make a virtual image 540 perpendicular to the reference direction and spaced apart by 1 if the direction of the pan / tilt angle is 0 degrees, respectively, as the reference direction. At this time, assuming that the movement in the x-axis in the virtual image 540 only depends on the angle of the fan, and the movement in the y-axis depends only on the tilt angle, the fan angle and tilt angle are calculated using the following equation Axis coordinate of the coordinate system 540, as shown in FIG.

The gaze tracking device can finally determine the three-dimensional coordinates of the object and the two-dimensional coordinates in the virtual image 540 using Equation 1, and calculate projection matrices between the two coordinates.

Eye-tracking device can estimate the arbitrary three-dimensional coordinate temperature may cause problems virtual coordinates on the virtual image _{(540) (x v, y} v) using the projection matrix. According to one aspect of the present invention, when the virtual coordinates are substituted into the following equation (2), the pan angle and tilt angle of the narrow-angle camera 530 for capturing the corresponding three-dimensional coordinates can be estimated.

Referring again to FIG. 3, the calibration performing unit 330 may perform calibration between the narrow-angle camera 320 and the image reproducing apparatus. For example, the calibration performing unit 330 can estimate a difference between the center of the pan axis and the tilt axis of the narrow camera 320 and the actual position of the image reproducing apparatus.

After the calibration process is completed, the gaze tracking device can find the eyes of the actual user and track the user's gaze using the coarse camera 320. [

For example, the image obtaining unit 340 detects the face region using the Haar-like based Adaboost method in the two wide angle cameras 310 for calculating the user location information, and then uses the Haar-like based Adaboost method to detect the face region Area can be detected. The gaze tracking device estimates the three-dimensional coordinates of the left eye using disparity and coordinates of the left eye in two images, and assigns the three-dimensional coordinates of the left eye to the projection matrix to estimate the pan angle and tilt angle of the coarse camera 320 .

6 is a block diagram showing the configuration of an image acquisition unit according to an aspect of the present invention.

6, an image acquisition unit 600 according to an embodiment of the present invention includes a resolution adjusting unit 610, a reflected light removing unit 620, a pupil candidate region detecting unit 630, an area selecting unit 640, 650, an image determination unit 660, and a pupil center detection unit 670.

6, the resolution adjusting unit 610 adjusts the resolution of the inputted image information, the reflected light removing unit 620 removes the reflected light of the adjusted image information, and the pupil candidate region detecting unit 630 detects The pupil candidate region can be detected by applying a predetermined filter to the image information.

7 is a diagram showing an example of reflected light removal according to one aspect of the present invention.

6 and 7, in order to detect a pupil candidate region in an eye image photographed by a narrow-angle camera, the image obtaining unit 600 may calculate an input image (a) of FIG. 7 having a resolution of 1600x1200 with a 200x100 resolution The size of the image can be reduced by the image (b) of FIG. 6A and the area of the reflected light can be removed through a grayscale morphology operation.

FIG. 8 shows an example in which a detection filter is applied to a pupil candidate region, and FIG. 9 shows an example of a result of detecting a pupil candidate region.

Referring to FIGS. 8 and 9, the gaze tracking apparatus can apply a filter as shown in FIG. 8 to detect a pupil candidate region in an image from which reflected light is removed. That is, the pupil candidate region detection unit 610 of FIG. 6 can detect the pupil candidate region by applying the filter algorithm of Equation (3) to the image information from which the reflected light is removed, and the pupil candidate regions 910 and 920 ) Can be obtained.

Where c is the center position of the filter, bi is the position of the periphery of the filter, d is the center of the filter for noise removal, Variable. ≪ / RTI >

Fig. 9 (b) shows the result when the parameter d is not applied, and Fig. 9 (c) corresponds to the result when the parameter d is applied. The gaze tracking apparatus can detect a stable pupil candidate region irrespective of whether or not the user wears glasses as shown in FIG.

When the pupil candidate region is detected, the gaze tracking apparatus can estimate whether the pupil region is covered with the eyelid in order to prevent an error in the gaze tracking result because the pupil region is covered by the eyelid.

6, the region selector 640 may select two regions having different sizes in the pupil candidate region. The threshold detector 650 detects the threshold values for the two regions, The determination unit 660 may compare the two threshold values detected and determine that there is no eyelid image in the eye image if they are the same.

10 is a view showing an example of a large candidate region and a small candidate region in the vicinity of the pupil, and FIG. 11 is a diagram showing a result of applying the Otsu threshold method to the pupil candidate region.

The gaze tracking apparatus can select regions a and b having different sizes as shown in Fig. 10 in the pupil candidate region.

The gaze tracking apparatus can obtain the results (a, b) as shown in FIG. 11 and detect the threshold values, respectively, by dividing the regions into two images (a, b) of FIG. 10 using the otsu threshold method. At this time, if the two threshold values are the same, the gaze tracking device determines that there is no eyelid in the image, and if it is different, it can be determined that the eyelid region exists.

Referring again to FIG. 6, the pupil center detecting unit 670 detects circular information from the pupil candidate region, and detects the pupil center information by performing ellipse fitting on the detected circular information.

12 is a block diagram showing the detailed configuration of a pupil center detecting unit according to an aspect of the present invention.

12, the pupil center detecting unit 1200 includes a circular detecting unit 1210, a position selecting unit 1220, a position shifting unit 1230, a weight assigning unit 1240, and an elliptic fitting unit 1250 .

The circular detection unit 1210 can discriminate between the center and the outline area of the circle by applying the circle detection method in the pupil candidate area and the position selection unit 1220 can select a plurality of points with respect to the circle outline area. The position shifting unit 1230 can move a plurality of points selected by the position having the largest differential value with respect to the brightness value on a straight line connecting the selected plurality of points and the center of the circle, A weight can be given to a plurality of moved points according to the magnitude of the differential value. The elliptic fitting unit 1250 may perform elliptical fitting using a plurality of weighted points to detect pupil center information.

For example, the pupil center detecting unit 1200 can determine the center of the circle and the outline region by applying the circular detection method to the detected pupil region, and then select 40 positions for the outline region of the circle. The gaze tracking device can draw a straight line from the above position to the center of the circle, then obtain a first derivative with respect to the brightness value on the straight line, and move the 40 points to the largest position of the derivative value.

The pupil center detecting unit 1200 can ignore the points of the portions extending over the eyelid region previously found out of the 40 points whose positions are changed and use the position first obtained in the circle detection method as it is. The weighted value is given according to the size of the first derivative value of each moving point. The larger the first derivative value, the larger the weight value can be obtained.

The pupil center detection unit 1200 can perform elliptic fitting using 40 points to which weights are applied. Since there is no concept of a weight at the time of elliptic fitting, the number of points having a high weight is increased, .

13 is a view showing the configuration of points for elliptic fitting and the results of elliptical fitting.

Referring to FIG. 13, the gaze tracking apparatus can confirm that the points for the elliptic fitting exist outside of the circle outline 1310, and as a result of the elliptic fitting, the ellipse 1320 in FIG. 13 (b) It can be confirmed that the pupil region is accurately displayed.

The user's gaze position first detects the center of the pupil and the center of the reflected light in the image captured by the coarse camera and detects a vector connecting the two positions. When the user sequentially views the six points on the screen in the user calibration step, the 2D mapping function is calculated between the vector connecting the center of the pupil and the center of the reflected light and the position of the pixel unit of six points on the actual screen do. When the user gazes at an arbitrary position on the screen, the user can determine the gaze position using the 2D mapping function, the vector between the pupil center and the center of the reflected light.

According to one aspect of the present invention, the gaze tracking device can model a shape of a real TV picture according to a user's position in a forward motion compensation model that corrects a gaze position according to a user's movement.

FIGS. 14 and 15 are diagrams showing the relationship between the screen of the image reproducing apparatus, the narrow-angle camera, and the pupil according to the movement of the user.

14 is a view illustrating a screen 1430 in which a screen 1430 and an image sensor 1420 of a coarse camera are parallel when the user is positioned in the middle of the screen 1430 of the image reproducing apparatus and the screen 1430 and the image sensor 1420 of the coarse camera And the center of the user's pupil 1410 is located in the vertical direction. At this time, if it is assumed that the infrared illumination is located at the same position as the narrow-angle camera, the eye-gaze tracking device can obtain a vector component as shown in the following Equation 4 when the user is gazing at a specific position appearing on the screen in the user calibration step.

15 shows an example of a relationship between a screen 1530 of a video reproducing apparatus, an image sensor 1520 of a coarse camera, and a user's pupil 1510 when the user moves, and the gaze tracking apparatus is shown in FIG. Vector components can be obtained.

According to one aspect of the present invention, the vector components of Equations (4) and (5) can be summarized as shown in Equation (6) below.

14 to 15 and Equations 4 to 5, the gaze tracking apparatus can calculate the relationship of the vector components as shown in Equation (7) below, and calculates the following equation (8) through the operations of Equations (6) and Can be calculated.

Referring to FIGS. 14 to 15 and Equations 4 to 5, the gaze tracking apparatus can calculate the relationship of vector components as shown in Equation (7) below, Can be calculated.

Here, the gaze tracking apparatus can estimate the three-dimensional positions of the center positions E _r and E _n of the user's pupil using the wide-angle stereo camera as shown in the following Equation (11).

)을 이용하여 하기 수학식 12와 같이 협각 카메라의 광학 중심 O에서 E_r과 E_n까지의 거리를 구할 수 있다.The gaze tracking device can find the distance between E _r and E _n at the optical center O of the coarse camera. For example, the gaze tracking device may include a projection matrix (< RTI ID = 0.0 >

), The distance from the optical center O of the coarse camera to E _r and E _n can be obtained as shown in Equation (12).

등의 성분은 사용자가 직접 줄자 등을 이용하여 측정할 수도 있다.Here, T and t denote a matrix transpose and a translation vector, respectively, and R can denote a rotation matrix. Here, D _xr , D _yr ,

Etc. may be measured by a user directly using a tape measure or the like.

와 같이 표시할 수 있으며, 팬/틸트 모듈에 의하여

만큼 회전하는 경우, 하기 수학식 13과 같이 연산 될 수 있다.In addition, the position of S _O with respect to the optical center O of the narrow-angle camera is

And can be displayed by the pan / tilt module

, The following equation (13) can be calculated.

및

의 값을 구할 수 있다.Based on Equation (13) described above, the gaze tracking apparatus calculates the gaze tracking device as shown in Equation (14)

And

Can be obtained.

Finally, the gaze tracking device can obtain the values of v _xn and v _yn as shown in the following Equation (15) based on Equations (4) to (14).

Since the gaze tracking device requires the movement of the pan / tild when the user performs the user calibration, the process of FIG. 16 can be performed to obtain the accurate gaze position.

16 is a diagram showing the relationship between the user calibration position, the reference position, and the current position.

Referring to FIG. 16, the gaze tracking apparatus can obtain the reference position 1620 at the position 1610 where the user calibration is performed, and obtain the current position 1630 based on the reference position 1620. For example, the gaze tracking device converts a PCCR (Pupil Center Corneal Reflection) vector estimated at a position 1610 where the user calibration is performed to a PCCR vector at a current position 1630, and obtains a value as shown in the following equation (16) .

Here, among the above parameters, the subscripted variables may refer to the value at the calibration position.

The gaze tracking device can provide a process of correcting the gaze tracking result in preparation for the occurrence of shaking even if the user gazes at the same position due to the shaking of the pupil and the image noise.

The gaze tracking device may further include an image correction unit, and the image correction unit may correct the error of the gaze position information due to the pupil fluctuation of the user's eyes. The image correcting unit can correct gaze position information using Equation (9) below.

For example, if the gaze tracking position detected in the current image is v ( t ) = { x , y }, and the gaze tracking position detected in the previous image is v '(t-1) Can be defined as the following equation (17). In this case, the parameter W may correspond to a parameter for determining whether the user's line of sight is moving when the distance between two line-of-sight positions is moved and using the original value without correction.

V '(t) is the corrected gaze position information, v (t) is the gaze tracking position detected in the current image, v' (t-1) is the gaze tracking position corrected in the previous image, W may correspond to a boundary value for the distance difference between the current sight position and the previous sight position.

17 is a flowchart illustrating a gaze tracking method according to an embodiment of the present invention.

Referring to FIG. 17, the gaze tracking apparatus performs calibration between the wide angle cameras 1701, 1702, and 1703 between the wide angle camera and the narrow angle cameras. In addition, the gaze tracking device may perform a user calibration (1704).

The gaze tracking device receives the image from the wide angle camera (1705), and detects the face area for each dual camera image (1706).

The gaze tracking device calculates eye area and calculates a three-dimensional position of the user (1707), and uses a coarse camera to detect a pan, a tilt, and a focus using the user's position information and eye area detection information (1408), and performs pan, tilt, and focus operations using the focus (1709).

The gaze tracking apparatus acquires an eye image through operation of the coarse camera (1710), and detects pupil center information and corneal reflection information from the eye image (1711, 1712).

The gaze tracking apparatus calculates eye position information using the eye position information and the forward movement correspondence model from the pupil center information and the corneal reflection information (1713).

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

310: Wide angle camera
320: coarse camera
330: Calibration execution unit
340:
350:
360: eye line position calculation unit

Claims (40)

A calibration performing unit for performing calibration between two or more wide angle cameras, a narrow angle camera, and an image reproducing apparatus to calculate user position information for a user;
An image information input unit receiving image information through the two or more wide angle cameras;
A first information detecting unit for detecting eye position information on the image information;
An image acquiring unit for acquiring an eye image by performing a pan operation, a tilt operation, and a focus operation on the coarse camera based on the eye position information and the user position information;
A second information detector for detecting pupil center information on the pupil center from the eye image;
A third information detecting unit for detecting corneal reflection information on corneal reflected light reflected by the eyes of the user; And
A line-of-sight position calculation unit for calculating line-of-sight position information from the pupil center information and the corneal reflection information,
Lt; / RTI >
The line-of-
When the user moves from a position where calibration between two or more wide-angle cameras has been performed, the user position information is applied to the forward motion compensation model and corrected, and then the gaze position information is calculated using the corrected user position information Eye tracking device. The method according to claim 1,
Wherein the calibration performing unit comprises:
Angle camera and performs calibration between the at least two wide-angle cameras and the narrow-angle camera, and performs calibration between the narrow-angle camera and the image reproducing device. The method according to claim 1,
Wherein the first information detecting unit comprises:
And a face region and an eye region are detected with respect to the at least two pieces of image information. delete At least two wide-angle cameras that receive image information and calculate user location information for a user through calibration;
A coarse camera that performs a pan operation, a tilt operation, and a focus operation using the user position information and the eye area detection information;
An image acquiring unit acquiring an eye image through operation of the narrow-angle camera;
A detector for detecting pupil center information and corneal reflection information from the eye image; And
A line-of-sight position calculation unit for calculating line-of-sight position information from the pupil center information and the corneal reflection information,
Lt; / RTI >
The image acquiring unit may acquire,
A resolution adjusting unit adjusting the resolution of the input image information;
A reflected light removing unit for removing reflected light of the adjusted image information; And
A pupil candidate region detection unit for detecting a pupil candidate region by applying a predetermined filter to the image information from which the reflected light is removed,
Wherein the gaze tracking device comprises: 6. The method of claim 5,
And a calibration execution unit for performing calibration between the at least two wide-
Wherein the gaze tracking device further comprises: The method according to claim 6,
Wherein the calibration performing unit comprises:
A gaze tracking device for estimating the position and degree of rotation of another camera based on a single wide-angle camera using a stereo calibration method. The method according to claim 6,
Wherein the calibration performing unit comprises:
Wherein the calibration is performed between the at least two wide angle cameras and the at least one narrow angle camera. The method according to claim 6,
Wherein the calibration performing unit comprises:
Wherein the calibration is performed between the coarse angle camera and the image reproducing device. 10. The method of claim 9,
Wherein the calibration performing unit comprises:
And estimates a difference between the center of the fan axis and the tilt axis on which the narrow angle camera moves and the actual position of the image reproducing apparatus. delete 6. The method of claim 5,
Wherein the pupil candidate region detecting unit
Wherein the pupil candidate region is detected by applying the filter algorithm of Equation (1) to the image information from which the reflected light is removed.
[Equation 1]

Where c is the center position of the filter, bi is the position of the periphery of the filter, d is the noise removal, Corresponds to the variable for. 6. The method of claim 5,
The image acquiring unit may acquire,
An area selection unit for selecting two areas having different sizes in the pupil candidate area;
A threshold detector for detecting a threshold value for each of the two areas; And
And comparing the detected two threshold values to determine that there is no eyelid image in the eye image,
Wherein the gaze tracking device further comprises: 6. The method of claim 5,
The image acquiring unit may acquire,
A pupil center detecting unit which detects circular information from the pupil candidate region and performs ellipse fitting of the detected circular information to detect the pupil center information,
Wherein the gaze tracking device further comprises: 15. The method of claim 14,
The pupil center detecting unit,
A circular detection unit for determining a center of the circle and an outline region by applying a circle detection method in the pupil candidate region;
A position selection unit for selecting a plurality of points with respect to the circle outline area;
A position shifting unit for shifting the selected plurality of points to a position where a derivative with respect to a brightness value is largest on a straight line connecting the selected plurality of points and the center of the circle;
A weighting unit for weighting the moved plurality of points according to the magnitude of the differential value; And
An elliptical fitting unit for performing elliptical fitting using the plurality of weighted points and detecting the pupil center information,
Wherein the gaze tracking device comprises: 6. The method of claim 5,
The image acquiring unit may acquire,
Wherein the coarse angle camera calculates a pan angle and a tilt angle for capturing the user based on the three-dimensional position of the user. 17. The method of claim 16,
The image acquiring unit may acquire,
A first estimator for estimating a two-dimensional coordinate of a virtual image corresponding to the pan angle and the tilt angle;
A second estimator estimating a virtual coordinate in which the three-dimensional coordinate is projected on the virtual image, with reference to a relative positional relationship between the three-dimensional coordinate of the three-dimensional position and the two-dimensional coordinate; And
A third estimating unit estimating the pan angle and the tilt angle based on the virtual coordinates,
Wherein the gaze tracking device comprises: 6. The method of claim 5,
An image correcting unit for correcting an error of the gaze position information due to a pupil of the user's eyes,
Wherein the gaze tracking device further comprises: 19. The method of claim 18,
Wherein the image correction unit comprises:
Wherein the line-of-sight position information is corrected using the following equation (2).
&Quot; (2) "

Where v '(t) is the corrected gaze position information, v (t) is a gaze tracking position detected in the current image, v' (t-1) is a gaze tracking position corrected in the previous image, W corresponds to the boundary value for the difference in distance between the current visual position and the previous visual position). 6. The method of claim 5,
The line-of-
And calculates the gaze position information through a forward motion compensation model that corrects the gaze position according to the movement of the user. Performing calibration between two or more wide-angle cameras, a coarse-angle camera, and a video playback device to calculate user location information for a user;
Receiving image information through the at least two wide angle cameras;
Detecting eye position information on the image information;
Performing pan, tilt, and focus operations on the coarse camera based on the eye position information and the user position information to obtain an eye image;
Detecting pupil center information on the pupil center from the eye image;
Detecting corneal reflection information on corneal reflected light reflected by the eye of the user; And
Calculating gaze position information from the pupil center information and the corneal reflected light information
Lt; / RTI >
Wherein the step of calculating the gaze position information comprises:
Applying the user position information to the forward motion compensation model when the user moves from a position at which the user performs calibration between two or more wide angle cameras;
Calculating the gaze position information using the corrected user position information
/ RTI > 22. The method of claim 21,
Wherein the calculating the user location information comprises:
Performing a calibration between the at least two wide-angle cameras;
Performing calibration between the at least two wide-angle cameras and the narrow-angle cameras;
Performing a calibration between the coarse camera and the image reproducing device
/ RTI > 22. The method of claim 21,
Wherein the step of detecting the pupil center information comprises:
Detecting a face region and an eye region with respect to the at least two pieces of image information;
/ RTI > delete Receiving image information through two or more wide angle cameras;
Calculating user location information for a user using the at least two wide angle cameras;
Performing pan, tilt, and focus operations using the user location information and eye area detection information using a coarse camera;
Acquiring an eye image through operation of the narrow-angle camera;
Detecting pupil center information and corneal reflection information from the eye image; And
Calculating gaze position information from the pupil center information and the corneal reflected light information
Lt; / RTI >
Wherein the acquiring of the eye image comprises:
Adjusting a resolution of the input image information;
Removing reflected light of the adjusted image information; And
Detecting a pupil candidate region by applying a predetermined filter to the image information from which the reflected light is removed
/ RTI > 26. The method of claim 25,
Performing a calibration between each of the wide-angle cameras
Wherein the gaze tracking method further comprises: 27. The method of claim 26,
A step of estimating the position and degree of rotation of another camera based on one wide-angle camera using a stereo calibration method
Wherein the gaze tracking method further comprises: 26. The method of claim 25,
Performing calibration between the at least two wide angle cameras and the narrow angle camera
Wherein the gaze tracking method further comprises: 26. The method of claim 25,
Performing a calibration between the coarse camera and the image reproducing device
Wherein the gaze tracking method further comprises: 30. The method of claim 29,
Estimating a difference between the center of the fan axis and the tilt axis on which the narrow angle camera moves and the actual position of the image reproducing apparatus
Wherein the gaze tracking method further comprises: delete 26. The method of claim 25,
Wherein the step of detecting the pupil candidate region comprises:
Wherein the pupil candidate region is detected by applying the filter algorithm of Equation (3) to the image information from which the reflected light is removed.
&Quot; (3) "

Where c is the center position of the filter, bi is the position of the periphery of the filter, d is the noise removal, Corresponds to the variable for. 26. The method of claim 25,
Wherein the acquiring of the eye image comprises:
Selecting two regions of different sizes in the pupil candidate region;
Detecting respective threshold values for the two regions; And
Comparing the detected two threshold values to determine that there is no eyelid image in the eye image
Wherein the gaze tracking method further comprises: 26. The method of claim 25,
Wherein the acquiring of the eye image comprises:
Detecting circular information from the pupil candidate region; And
Detecting the pupil center information by performing ellipse fitting of the detected circular information;
Wherein the gaze tracking method further comprises: 35. The method of claim 34,
Wherein the acquiring of the eye image comprises:
Identifying a center and an outline region of a circle by applying a circle detection method in the pupil candidate region;
Selecting a plurality of points for the outline area of the circle;
Moving the selected plurality of points to a position having a largest differential value with respect to a brightness value on a straight line connecting the selected plurality of points and the center of the circle;
Assigning weights to the shifted points according to the magnitude of the differential value; And
Performing elliptical fitting using the plurality of weighted points to detect the pupil center information
Wherein the gaze tracking method further comprises: 26. The method of claim 25,
Calculating a pan angle and a tilt angle for photographing the user based on the three-dimensional position of the user;
Wherein the gaze tracking method further comprises: 37. The method of claim 36,
Estimating two-dimensional coordinates of a virtual image corresponding to the pan angle and the tilt angle;
Estimating a virtual coordinate in which the three-dimensional coordinate is projected on the virtual image with reference to a relative positional relationship between the three-dimensional coordinate of the three-dimensional position and the two-dimensional coordinate; And
Estimating the pan angle and the tilt angle based on the virtual coordinates
Wherein the gaze tracking method further comprises: 26. The method of claim 25,
Correcting an error of the line-of-sight position information due to the pupil fluctuation of the user
Wherein the gaze tracking method further comprises: 39. The method of claim 38,
Wherein the step of correcting the error of the gaze position information comprises:
Wherein the gaze position information is corrected using the following equation (4).
&Quot; (4) "

Where v '(t) is the corrected gaze position information, v (t) is a gaze tracking position detected in the current image, v' (t-1) is a gaze tracking position corrected in the previous image, W corresponds to the boundary value for the difference in distance between the current visual position and the previous visual position). 26. The method of claim 25,
Calculating the gaze position information through a forward motion compensation model that corrects the gaze position in accordance with the movement of the user
Wherein the gaze tracking method further comprises:

Images