KR101819164B1 - Apparatus and method for gaze tracking - Google Patents

Abstract

The present invention relates to an apparatus and a method for gaze tracking. An aspect of the present invention provides the apparatus for gaze tracking, comprising: an image acquiring unit acquiring a face image of a user; a face feature point extraction unit extracting face feature points from the acquired face image; a first gaze information tracking unit extracting feature points of the eyes and the mouth from the extracted face image and calculating a face plane and a normal vector of the face plane based thereon to track a face information based gaze vector; a second gaze information tracking unit extracting eye boundary feature points from the extracted face feature points and calculating the rotational center of an eyeball and the center of an iris based thereon to track an eye information based gaze vector; a gaze information integration unit integrating the tracked first and second gaze information; and a gaze tracking performing unit performing the gaze tracking by using the integrated gaze information. The present invention can perform more rapid gaze tracking which is tolerant of a three-dimensional position and natural movement of the face.

Description

[0001] APPARATUS AND METHOD FOR GAZE TRACKING [0002]

The present invention relates to a gaze tracking apparatus and method, and more particularly, to a gaze tracking apparatus and method for tracking a user's gaze by integrating a user's face-based gaze information and eye-based gaze information based on a 3D image sensor.

Eye-gaze tracking is a study that grasps the location the user is looking at. Conventionally, gaze position tracking directly calculates the gaze position on the monitor from the inputted 2D facial image without grasping the 3D position and movement of the face when the user looks at a point on the monitor. In such a case, the three-dimensional distance between the monitor and the user's face must be fixed, and only a limited range of face movements is allowed.

In order to solve these problems, it is necessary to grasp not only the position of the line of sight on the plane but also the way of gazing at an object far away from the user's eye, thereby determining the position of the line of sight which can be represented by (X, Y, Z) Dimensional eye tracking studies were conducted.

This conventional method of locating the eye position includes a method of calculating the gaze position by measuring the potential difference between the retina and the cornea by attaching electrodes around the eyes, a method of calculating the gaze position by attaching a small camera under the head band or the helmet, There is a method of calculating a gazing position by installing a camera and a camera having a zoom function and a camera capable of rotating the camera.

However, this method requires a separate device such as a camera and illumination for eye tracking, and not only the processing speed is lowered due to the complexity of eye tracking calculation, but also it is easy to miss the accuracy of eye tracking due to the user's motion.

The present invention provides a quick and accurate gaze tracking apparatus and method by integrating a user's face-based gaze information and eye-based gaze information based on a 3D image sensor and tracking a user's gaze.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood from the following description.

According to an aspect of the present invention, a gaze tracking apparatus is provided.

The eye tracking apparatus according to an embodiment of the present invention includes an image obtaining unit for obtaining a face image of a user, a facial feature point extracting unit for extracting facial feature points from the obtained facial image, A first eye information tracking unit for tracking face information based eye vectors by calculating a normal vector of face planes and face planes based on the extracted eye plane feature points, extracting eye boundary feature points from extracted face feature points, A second eye information tracking unit for tracking an eye information based eyeball vector by calculating the center of the iris and a center of sight, a eye information integrator for integrating the first eyeball information and the second eyeball information, A line-of-sight tracking apparatus including a line-of-sight tracking performing unit may be provided.

According to another aspect of the present invention, a gaze tracking method is provided.

The eye tracking method according to an embodiment of the present invention includes the steps of acquiring a face image of a user, extracting feature points of the face from the obtained face image, extracting feature points of eyes and mouth from the extracted face feature points, Calculating face planes based on feature points of the eyes and mouth, calculating normal vectors of the calculated face planes, tracking the first eye vector based on the face information using the calculated normal vectors, Calculating the center of rotation of the eye based on the extracted eye boundary features, calculating the center of the iris based on the extracted eye boundary features, calculating the center of rotation of the eye, Tracking an eye information based second line vector using the center of the iris, integrating the traced first and second line of sight vectors, And a line-of-sight tracking unit for performing line-of-sight tracking using a vector.

The present invention integrates user's face information based eye information and eye based eye information based on a 3D image sensor to track a user's gaze, thereby enabling quicker and more accurate tracking of three-dimensional position and natural motion (rotation and movement) This is possible.

In addition, since the present invention does not require a calibration process by the initial user, it is easy to use.

1 to 3 are views illustrating a gaze tracking apparatus according to an embodiment of the present invention.
4 to 12 are views for explaining a gaze tracking method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Also, when a part is referred to as "including " an element, it is to be understood that it may include other elements as well, without departing from the other elements unless specifically stated otherwise.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings.

1 to 3 are views illustrating a gaze tracking apparatus according to an embodiment of the present invention.

1, the gaze tracking apparatus 100 includes a face image obtaining unit 110, a facial feature point extracting unit 120, a first gaze information tracing unit 130, a second gaze information tracing unit 140, A sight line information integrating unit 150 for integrating the first and second line of sight information, and a line-of-sight tracking performing unit 160.

The face image obtaining unit 110 obtains a user's face image from the image photographed by the 3D sensor.

The facial feature point extracting unit 120 acquires three-dimensional information of the facial feature points on the obtained facial image.

The first line of sight information tracking unit 130 extracts the minutiae of the eyes and mouth from the facial feature points extracted by the facial feature point extracting unit 120 and calculates the normal vectors of the face planes and the face planes based on the extracted minutiae points, ≪ / RTI >

The second line-of-sight information tracking unit 140 extracts the eye boundary feature points from the facial feature points extracted by the facial feature point extracting unit 120, calculates the center of rotation of the eyeball and the center of the iris, do.

The gaze information integrating unit 150 integrates the first gaze information and the second gaze information. At this time, the eye-gaze information integrating unit 150 adds weights to the respective gaze information in consideration of the first gaze information tracked in the relatively high-resolution face image and the second gaze information tracked in the low-resolution eye image.

The line-of-sight tracking performing unit 160 performs line-of-sight tracking using the line-of-sight information integrated in the line-of-sight information integrating unit 150.

2, the first eye-gaze information tracking unit 130 of the eye-gaze tracking apparatus 100 according to an embodiment of the present invention includes an eye and mouth feature point extracting unit 131, a face-plane calculating unit 132, Plane normal vector calculation unit 133 and a face information-based eye-line vector tracking unit 134.

The eye and mouth feature point extracting unit 131 extracts feature points of eyes and mouth from the obtained three-dimensional information of the face feature points.

The face plane calculation unit 132 calculates the face planes passing through the feature points using the extracted eye and mouth feature points.

The normal vector calculation unit 133 of the face plane calculates a normal vector passing through the face plane calculated by the face plane calculation unit 132.

The face information based eyeball vector tracking unit 134 tracks the normal vector calculated by the face vector normal vector calculation unit 133 as the first eyeball information.

3, the second gaze information tracking unit 140 of the gaze tracking apparatus 100 according to an embodiment of the present invention includes an eye boundary feature point extracting unit 141, an eyeball rotation center calculating unit 142, An iris center calculation unit 143, and an eye information based eye vector tracking unit 144.

The eye boundary feature point extracting unit 141 extracts feature points of the eye boundary from the three-dimensional information of the facial feature points acquired by the facial feature point extracting unit 120.

The eye rotation center calculation unit 142 calculates the rotation center of the eye using the eye boundary feature point extracted by the eye boundary feature point extraction unit 141 and least-mean-squares (LMS).

The iris center calculation unit 143 calculates the center coordinates of the 3D iris using the face image acquired by the RGB camera and the face image acquired by the depth sensor in the face image acquisition unit 110.

The eye information based eyeball vector tracking unit 144 calculates eyeball vectors that connect the rotation center coordinates of the eye calculated by the rotation center calculation unit 142 of the eyeball and the center coordinates of the iris calculated by the center calculation unit 143 Track with second line of sight information.

The gaze information integrating unit 150 integrates the first gaze information and the second gaze information. At this time, the eye-gaze information integrating unit 150 adds weights to the respective gaze information in consideration of the first gaze information tracked in the relatively high-resolution face image and the second gaze information tracked in the low-resolution eye image.

The line-of-sight tracking performing unit 160 performs line-of-sight tracking using the line-of-sight information integrated in the line-of-sight information integrating unit 150.

4 to 12 are views for explaining a gaze tracking method according to an embodiment of the present invention.

Referring to FIG. 4, a gaze tracking method 400 according to an exemplary embodiment of the present invention includes a step S405 of obtaining a user's face image, a step S410 of extracting feature points of a face from the obtained face image, A step S420 of calculating a face vector based on the extracted feature points of eyes and mouths S420, a step S425 of calculating a normal vector of the calculated face planes S425 A step S430 of tracking the first eye vector based on the face information using the calculated normal vector, a step S435 of extracting eye boundary feature points from the extracted facial feature points, Calculating the center of rotation based on the extracted eye boundary feature points (S445), calculating the eye information based second eye vector using the center of rotation of the eye and the center of the iris Tracking stage (S450), using a step (S455), the integrated line of sight vector to integrate the tracking first and second line of sight vector and a gaze detection execution unit (S460) for performing eye tracking.

In step S405, the gaze tracking method 400 acquires a user's face image using an RGB camera and a depth sensor.

Referring to FIG. 5, the eye tracking method 400 according to an exemplary embodiment of the present invention acquires a face image of a user using an RGB camera and a depth sensor when a user gazes at a point on a monitor separated by a predetermined distance .

In step S410, the gaze tracking method 400 extracts feature points of the face from the face image obtained in step S405. Here, the facial feature points include arbitrary pixel coordinates (x value, y value) obtained by the RGB camera and arbitrary z values obtained by the depth sensor.

Referring to FIG. 6, the eye tracking method 400 according to an exemplary embodiment may extract 1,436 facial feature points using the HD face model provided by Kinect v2.

In step S415, the gaze tracking method 400 according to an exemplary embodiment extracts feature points of eyes and mouth from the facial feature points extracted in step S410.

Referring to FIG. 7, the eye tracking method 400 according to an exemplary embodiment extracts 8 feature points (4 eyes on the right eye, 4 eyes on the left eye) and 6 feature points around the eyes, as in 710, The minutiae points of the eyes and mouth can be set as the average coordinates of the extracted minutiae.

In step S420, the gaze tracking method 400 calculates a face plane based on the eye and mouth feature points extracted in step S415.

Referring again to FIG. 7, the eye tracking method 400 according to an exemplary embodiment may calculate a face plane passing through three feature points 721, 722, and 723 using eye and mouth feature points extracted in step S420 .

In step S425, the gaze tracking method 400 calculates a normal vector passing through the face plane calculated in step S420.

In step S430, the gaze tracking method 400 tracks the first gaze vector based on the face information using the normal vector calculated in step S425.

Referring to FIG. 8, the line-of-sight tracking method 400 according to an exemplary embodiment may track a normal vector 812 perpendicular to the face plane 811 as first line-of-sight information.

In step S435, the gaze tracking method 400 extracts eye boundary feature points from the extracted facial feature points.

Referring to FIG. 9, the eye tracking method 400 according to an exemplary embodiment may extract eye boundary feature points, such as 911 to 914.

In step S440, the gaze tracking method 400 calculates the center of rotation of the eyeball using Equation 1 based on the extracted eye boundary feature points.

X _c, Y _c, Z _c in equation (1) Is the rotation center coordinate of the eyeball, and X _ic , Y _ic , Z _ic is the coordinate of one of the eye boundary feature points in the virtual eye structure.

In general, the center of rotation of the eyeball is located 13.5mm from the surface of the cornea. In addition, when one light is placed on the front face of the human eye, illumination reflection light is generated on the four optical surfaces of the eye (inside and outside of the cornea, inside and outside of the lens). Since the position of the illumination reflected light changes depending on the thickness of the lens depending on how far the user looks at the position of the user, the distance between the illuminated reflected light depends on the depth information from the user's eye to the gazing position.

Therefore, based on the structural characteristics of the human eye, it is possible to track the three-dimensional sight line position (left, right, up and down, and depth direction eye line positions) including the depth information.

Referring to FIG. 10, the eye tracking method 400 according to an exemplary embodiment includes a virtual eyeball structure including a radius of 13.5 mm and four eyeball feature points 911 to 914, such as 1000, The center can be calculated.

In step S445, the gaze tracking method 400 calculates the center of the iris based on the extracted eye boundary feature points. The line-of-sight tracking method 400 according to an embodiment can calculate the three-dimensional center coordinates of the iris by using the face image acquired by the RGB camera and the face image acquired by the depth sensor.

In step S450, the gaze tracking method 400 tracks the gaze-based second gaze vector using Equation 2 based on the center of rotation of the eye and the center of the iris calculated in steps S444 to S445.

는 안구의 회전 중심 좌표이고,

는 홍채 중심 좌표이다. In Equation 2,

Is the rotation center coordinate of the eyeball,

Is the iris center coordinate.

Referring to FIG. 11, a line-of-sight tracking method 400 according to an exemplary embodiment detects an eye region in a face image acquired by an RGB camera, such as a 1101 image, and binarizes the 1101 image as in the 1102 image. In the binarized image 1102, morphological operations such as 1103 to 1106 are performed to detect an iris region from which noise of an image has been removed. Then, the center of the rectangle including the detected iris region as the 1107 image is calculated, and the center of the rectangle calculated as the 1108 image can be set as the center coordinates (x value and y value) of the iris.

In addition, the gaze tracking method 400 according to an embodiment uses the depth face image acquired by the depth sensor to calculate a z-value that coincides with the iris center coordinates (x value and y value) of the image 1108 in Fig. 11 And the center coordinates 1109 of the three-dimensional iris can be set.

In step S455, the line-of-sight tracking method 400 integrates the first and second line-of-sight vectors traced in steps S430 and S450 using Equation (3).

및

는 가중치이다.In Equation (3), V denotes an integrated gaze information vector, V _F is a face information based first line information vector, V _E is an eye information based second line information vector,

And

Is the weight.

In step S460, the line-of-sight tracking method 400 performs line-of-sight tracking using the integrated line-of-sight vector in step S455.

Referring to FIG. 12, the line-of-sight tracking method 400 according to an exemplary embodiment may track the line-of-sight position 1204 of the user using the integrated line-of-sight information 1203.

The gaze tracking method according to various embodiments of the present invention may be implemented in the form of a program command that can be executed through various means such as a server. Further, a program and an application for executing the gaze tracking method according to the present invention may be installed in a computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on a computer-readable medium may be those specially designed and constructed for the present invention or may be available to those skilled in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: eye tracking device
110: facial image acquiring unit
120: facial feature point extracting unit
130: First line of sight information tracking unit
140: second line of sight information tracking unit
150: eye information integration unit
160: eye tracking execution unit

Claims (15)

An image obtaining unit for obtaining a face image including three-dimensional information;
A facial feature point extracting unit for extracting a plurality of facial feature points from the acquired facial image;
A first eye information tracking method for extracting feature points of both eyes and mouth from the plurality of facial feature points and calculating normal vectors of the face planes and face planes based on the extracted feature points of eyes and mouth, part;
Extracting eye boundary feature points from the plurality of facial feature points, calculating rotation center coordinates of the eye based on the extracted eye boundary feature points, calculating iris center coordinates in the eye region including the eye boundary feature points, A second eye information tracking unit for tracking an eye information based eyeball vector using the center coordinates and the iris center coordinates;
A line-of-sight information integration unit for combining the first line-of-sight information and the second line-of-sight information by applying a weight to the tracked first line of sight information and second line of sight information according to the resolution of the face image and the eye region; And
And a line-of-sight tracking unit for performing line-of-sight tracking using the integrated line-of-sight vector.
The apparatus according to claim 1, wherein the first eye-gaze information tracking unit
Eye and mouth feature point extracting units for extracting feature points of both eyes and mouth from the extracted three-dimensional information of the facial feature points;
A face plane calculation unit for calculating a face plane passing through the extracted both eyes and the mouth feature points;
A normal vector calculation unit for calculating a normal vector passing through the face plane; And
And a face information based eye vector tracking unit for tracking a user's eye using a normal vector of the face plane.
3. The apparatus of claim 2, wherein the eye and mouth feature extraction unit
Extracting a plurality of feature points located in the vicinity of the eyes and the mouth from the extracted three-dimensional information of the facial feature points, and setting the positions of the both eyes and mouth feature points as average coordinate values of the extracted feature points.
The apparatus according to claim 1, wherein the second eye-gaze information tracking unit
An eye boundary feature point extraction unit for extracting feature points of an eye boundary from the extracted three-dimensional information of the facial feature points;
An eyeball rotation center calculation unit for calculating the rotation center coordinates of the eyeball using the extracted eye boundary feature points;
An iris center calculating unit for calculating the center coordinates of the iris using the extracted eye boundary feature points; And
And an eye information based eyeball vector tracking unit for tracking a user's eye line by using the rotation center coordinates of the eyeball and the center coordinates of the iris.
5. The apparatus according to claim 4, wherein the rotation center calculation unit
Calculating a center of rotation of the eye using a virtual eye structure having a radius of 13.5 mm and including the eye boundary feature points.
5. The apparatus according to claim 4, wherein the rotation center calculation unit
And calculating the center of rotation of the eye using the eye-boundary feature point and least-mean-squares (LMS).
5. The apparatus according to claim 4, wherein the center calculation unit
An eye region is detected in the face image acquired by the RGB camera, the iris region is detected by morphological operations of the detected eye region, the center coordinates of the rectangle including the iris region are calculated, dimensional iris using a face image acquired by a depth sensor.
delete A method for tracking a user's gaze in a gaze tracking device,
Obtaining a face image including three-dimensional information;
Extracting a plurality of facial feature points from the obtained facial image;
Extracting feature points of both eyes and mouth from the extracted facial feature points;
Calculating face planes based on the extracted feature points of eyes and mouth;
Calculating a normal vector of the face plane;
Tracking the first eye vector based on the face information using the normal vector;
Extracting eye boundary feature points from the extracted facial feature points;
Calculating a rotation center of the eye based on the eye boundary feature point;
Detecting an eye region in the face image and calculating the center of the iris through image processing;
Tracking an eye information based second eye vector using the center of rotation of the eyeball and the center of the iris;
Integrating the first gaze vector and the second gaze vector by applying a weight to the first gaze vector and the second gaze vector according to the resolution of the face image and the eye region; And
And performing line-of-sight tracking using the integrated line-of-sight vector.
10. The method according to claim 9, wherein extracting feature points of both eyes and mouth from the extracted facial feature points comprises:
Extracting a plurality of feature points located in the vicinity of both eyes and mouth from the extracted three-dimensional information of the facial feature points, and setting positions of the both eyes and mouth feature points as average coordinate values of the extracted feature points.
10. The method of claim 9, wherein calculating the center of rotation of the eye based on the extracted eye boundary feature points comprises:
And calculating the center of rotation of the eye using the eye-boundary feature point and least-mean-squares (LMS).
delete 10. The method of claim 9, wherein calculating the center of the iris based on the eye boundary feature point comprises:
An eye region is detected in the face image acquired by the RGB camera and the iris region is detected by morphological operations in the detected eye region to calculate the center coordinates of the rectangle including the iris region, Dimensional iris using a facial image acquired by the facial image acquired by the facial image acquiring unit.
delete A computer program recorded on a computer-readable recording medium for executing a gaze tracking method according to any one of claims 9 to 11 and claim 13.

Images