KR20140014868A - Gaze tracking apparatus and method - Google Patents

Abstract

A method of tracking a line of sight according to an embodiment of the present invention comprises the steps of: acquiring a depth image of a subject; acquiring position information of an eyeball region included in the subject from the acquired depth image; and tracking the line of sight of a user by recognizing a pupil included in the eyeball region based on the acquired position information. [Reference numerals] (AA) Start; (BB) End; (S210) Obtain a three-dimensional distance map; (S220) Obtain X, Y, and Z coordinate information relative to a sight position; (S230) Obtain an enlarged image in which a pupil area is enlarged by using the X, Y, and Z coordinate; (S240) Recognize the pupil from the enlarged image; (S250) Track a user's sight by using a recognized pupil distance

Description

Gaze tracking device and gaze tracking method {Gaze Tracking Apparatus and Method}

An embodiment relates to a gaze tracking device and a gaze tracking method thereof.

Recently, viewers' advertising effect measurement based on user's eye tracking, driver driving behavior analysis and drowsiness prevention, menu relocation through web page consumption behavior analysis, game control using gaze interaction, iris information based user authentication, exercise and professionalism Placement of shelves through technical training programs and consumption analysis is being conducted.

PTL 1 relates to a computer vision-based eye tracking method.

This method is configured before using the look-up table for the feature values for the user's face and eyes. In the actual eye tracking, after measuring the face direction and the center point of the iris, the eye direction is calculated by comparing the characteristic value with the measured value. The gaze direction is calculated by combining the measured face direction coordinate system and the iris center coordinate system. The face orientation is measured by using a T-stick reference model to find the exact angle, and the center of the iris is obtained by determining the center of gravity of the iris color by analyzing the color distribution of the iris. Find the longest horizontal line in the elliptical iris edge and determine the center point of that horizontal line as the center point of the iris.

However, this method has a limitation to consider and collect the characteristics of a large number of characteristics in order to increase the reliability in constructing the lookup table by measuring the characteristic values of various faces and eyes in advance, and the iris area is covered by the eyelids. If you lose, your eye tracking accuracy may drop.

The prior art disclosed in Patent Literature 2 grasps a user's face as a feature surface by a plurality of feature points, and tracks the gaze by recognizing the direction of the user's face by translation and rotation of the feature surface. As a result, a highly reliable and accurate gaze tracking system can be provided at a price low enough for general use. However, it is not an intuitive gaze tracking interface because only the feature points of the face are used and eye rotation is not taken into consideration. In order to move the cursor to a desired position, the head must be continuously moved.

The non-patent paper proposes a method of estimating a three-dimensional position of an eyeball using a stereo camera device composed of two cameras and three infrared lights, and obtaining a gaze vector through the gaze vector.

In order to perform this method, the calibration process between two cameras must be preceded for the configuration of the stereo camera, and the positions between the three lights must be set correctly.

This method has a problem in that it is expensive to build a plurality of cameras and lighting devices.

Patent Document 1: KR 10-0311605

Patent Document 2: KR 10-0325365

Non-Patent Document: S. W. Shih and J. Liu, "A Novel Approach to 3-D Gaze Tracking Using Stereo Cameras," IEEE Transactions on Systems, Man and Cybernetics, Part B, vol. 34, no. 1, pp. 234-245, Feb. 2004

The embodiment provides a gaze tracking device and a gaze tracking method for tracking a gaze of a user using a time of flight (TOF) camera.

The embodiment provides a gaze tracking device and a gaze tracking method thereof, which may track an eye of a user using an array lens.

In addition, the embodiment provides a gaze tracking device and a gaze tracking method thereof capable of performing gaze tracking by using a position difference between right and left pupils.

It is to be understood that the technical objectives to be achieved by the embodiments are not limited to the technical matters mentioned above and that other technical subjects not mentioned are apparent to those skilled in the art to which the embodiments proposed from the following description belong, It can be understood.

According to an exemplary embodiment, a gaze tracking method may include: obtaining a depth image of a subject; Acquiring position information of an eyeball area included in the subject from the acquired depth image; And recognizing a pupil included in the eyeball area based on the acquired position information to track the eyes of the user.

In addition, the apparatus for tracking eye gaze according to the embodiment may include: a first camera configured to acquire a color image and a depth image of a subject; An image processor for signal processing an image acquired through the first camera; And a control unit for acquiring position information of the eyeball area included in the subject based on the obtained color image and the depth image, and recognizing a pupil included in the eyeball area based on the acquired position information. Include.

According to an embodiment, a stereoscopic depth map is constructed using a time of flight (TOF) camera or an array camera and eye gaze tracking is performed using the configured stereoscopic depth map, thereby reducing the cost for constructing the eye tracking apparatus. Can provide various functions according to eye tracking.

1 is a diagram illustrating a gaze tracking device according to an exemplary embodiment.
2 is a diagram illustrating a configuration of a first camera according to a first embodiment.
FIG. 3 is a view for explaining an image acquisition method of the first camera shown in FIG. 2.
4 and 5 are views illustrating a configuration of the first camera according to the second embodiment.
6 and 7 provide a gaze tracking method of a gaze tracking device according to an exemplary embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which: FIG. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

In order to clearly illustrate the present invention in the drawings, thicknesses are enlarged in order to clearly illustrate various layers and regions, and parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification .

1 is a diagram illustrating a gaze tracking device according to an exemplary embodiment.

Referring to FIG. 1, the eye tracking apparatus includes a first camera 110, a second camera 120, an image processor 130, a storage 140, and a controller 150.

The first camera 110 acquires an image by photographing a subject located in front and obtains distance information from the acquired image.

The distance information refers to a depth map in which contrast is changed in proportion to the distance between the first camera 110 and the subject.

The depth map determines the distance between the eye tracking apparatus and the subject and expresses the brightness of each subject. That is, according to the depth map, a subject located closer to the gaze tracking device (obviously, the first camera) is expressed at a higher contrast level (brighter) than a subject located far away.

Using the depth map, the distance from the gaze tracking device to the subject may be tracked based on brightness information of each subject represented in the depth map.

The depth map may be generally obtained using a plurality of cameras (stereo cameras).

However, in an embodiment, a depth map representing a distance to the subject may be obtained using a TOF camera using a time of flight (TOF) principle, or the depth map may be obtained using an array camera.

A depth map acquisition method using the TOF camera and the array camera will be described later.

The second camera 120 acquires an enlarged image of a specific area of the subject located in front of the second camera 120.

In this case, the second camera 120 may be implemented as a zoom lens, and acquires an enlarged image of the specific area according to a predetermined zoom ratio.

The specific area may be an area where a user's gaze is located among the subjects. In other words, the specific area may be an area where the user's eyeball is located.

The specific zoom ratio may correspond to distance information corresponding to the eyeball of the user. That is, when the distance to the eyeball increases, the zoom ratio increases in proportion to the eyeball. Accordingly, the zoom ratio for acquiring the enlarged image may be set according to the distance information.

The image processor 130 processes and outputs an image acquired through the first camera 110 and the second camera 120.

The storage unit 130 may store a program for processing and controlling each signal in the controller 150, or may store a signal-processed video, audio, or data signal.

In addition, the storage unit 130 may perform a function for temporarily storing an image, audio, or data signal input from the outside.

The storage unit 130 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory) RAM, ROM (EEPROM, etc.), and the like.

The controller 150 controls the overall operation of the eye tracking apparatus.

For example, the controller 150 controls the first camera 110 to acquire a depth map indicating the distance to the subject located in front of the first camera 110.

In addition, the controller 150 recognizes the user located in front of the acquired depth map, and thereby obtains coordinate information on the eyeball position of the user.

To this end, the controller 150 recognizes a user's face from an image acquired through the first camera 110.

The user face recognition may be performed based on a pattern feature of a face component (eg, mouth, eyes, nose, forehead, etc.) of the user.

When the face is recognized, the controller 150 obtains X coordinate and Y coordinate information on the position of the eyeball, based on a feature point of the user's eyes (obviously, the eyeball) from the recognized face.

In addition, the controller 150 confirms the contrast of the position (eye position) corresponding to the X coordinate and Y coordinate information from the acquired depth map, and obtains the Z coordinate information of the eye position based on the confirmed contrast. do.

The X and Y coordinate information may be coordinate information about a vertical line and a horizontal line with respect to the eyeball position of the user, and the Z coordinate information may be distance information about the eyeball position of the user.

As described above, when the coordinate information about the eyeball position is obtained, the controller 150 controls the second camera 120 to obtain an enlarged image in which the eyeball area of the user is enlarged.

To this end, the controller 150 sets a condition for acquiring an enlarged image by the second camera 120 based on the obtained X, Y, and Z coordinate information.

That is, the controller 150 sets an area to be enlarged through the second camera 120 using the X and Y coordinate information.

Thereafter, the controller 150 sets an enlargement ratio (zoom ratio) for the set enlarged region by using the Z information. That is, the controller 150 increases the enlargement ratio when the distance to the eyeball by the Z information is far, and decreases the enlargement ratio when the distance to the eyeball is close.

The second camera 120 obtains an enlarged image obtained by enlarging an image of an area corresponding to the X and Y coordinate information at a zoom ratio corresponding to the Z coordinate information under the control of the controller 150.

When the enlarged image is obtained through the second camera 120, the controller 150 analyzes the obtained enlarged image to recognize an eyeball.

In addition, the controller 150 tracks the gaze of the user by using the position difference between the left pupil and the right pupil according to the recognition of the pupil.

In this case, the pupil recognition may be performed by an image processing algorithm performed by the image processor 130.

The image processor 130 drives an image processing algorithm, a circular detection algorithm, and the like on the obtained enlarged image to obtain a pupil of a user. That is, the image processor 130 detects the user pupil area from the image corresponding to the user eye position acquired through the second camera 120 and drives a circular detection algorithm to determine the center point of the pupil from the detected pupil area. Acquire.

The circular detection algorithm moves a circular detection template composed of an inner circle and an outer circle to the pupil area, detects the largest gray level difference between the inner circle and the outer circle of the template, and detects the center point of the detected area. Can be obtained as a center point.

In addition, the image processor 130 may acquire the pupil center point by further performing a local binarization technique together with the circular detection algorithm.

As such, the image processor 130 performs local binarization together with a circular detection algorithm in order to accurately acquire the center point of the pupil, determines the dark portion of the binarized region as the pupil region, and determines the center of gravity of the dark region. It can be judged as the actual center point of the pupil.

Thereafter, the controller 150 tracks the gaze of the user based on the distance between the center point of the left pupil and the center point of the right pupil acquired through the image processor 130.

Since the user's eye tracking technology using the distance between the right pupil and the left pupil is already known in the art to which the present invention pertains, a detailed description thereof will be omitted.

When the user's eyes are tracked, the controller 150 measures viewer advertisement effect based on the tracked eyes, prevents driver's driving behavior and drowsiness, rearranges menus through web page consumption behavior analysis, and game using eye interaction. It provides control, control of iris information based user authentication, exercise and technical training program, and display rack placement through consumption analysis.

As described above, according to the embodiment, the depth map acquired using one TOF camera or the array camera is used to acquire location information about the pupil of the user, and the acquisition condition of the enlarged image is obtained using the acquired location information. Set it.

Thereafter, an enlarged image of the user pupil is obtained using the set acquisition condition, the user pupil is recognized from the obtained enlarged image, and the user's eyes are tracked.

According to the embodiment, as described above, by constructing a three-dimensional depth map using a time of flight (TOF) camera or an array camera, and performing eye tracking using the configured three-dimensional depth map, for the construction of the eye tracking apparatus It can provide various functions according to eye tracking while reducing cost.

2 is a diagram illustrating a configuration of a first camera according to a first embodiment.

According to the first embodiment, the first camera 110 is composed of a TOF camera.

Referring to FIG. 2, the TOF camera includes a lens module, an RGB camera, and a D camera.

The TOF camera may acquire a color image and a depth image. In general, the color image may be an image of a subject located in the front, and the depth image may be an image of a depth map expressing the distance from the subject in contrast.

That is, the color image is a collection of visible light reflected from a subject and represented, and means a viewable image of a human eye or a general camera. The color image may be expressed by dividing the R channel, the G channel, and the B channel for each pixel.

In addition, the depth image is generated by emitting electromagnetic waves other than visible light to a subject and calculating flight times of electromagnetic waves reflected from the subject. For example, the electromagnetic wave is emitted to the subject, the infrared rays reflected by the subject are collected, and at the same time, the arrival time for each pixel of the infrared is measured to calculate each flight time.

Then, depth image information for each pixel may be generated according to the calculated flight time.

Referring to FIG. 2, the RGB sensor collects visible light to generate a color image, emits infrared rays through an infrared red illuminator, and infrared rays reflected from a subject through an infrared red sensor. The depth image may be generated by collecting the.

In this case, the RGB sensor may be represented by an RGB camera, and the color image may be represented by being divided into an R channel, a G channel, and a B channel.

In addition, the D camera as an infrared red sensor may generate a depth image using the arrival time according to the reflection of the infrared light.

FIG. 3 is a view for explaining an image acquisition method of the first camera shown in FIG. 2.

Referring to FIG. 3, in order to generate the depth map (depth image), an electromagnetic wave emitter emits a specific electromagnetic wave, and the emitted electromagnetic wave may be reflected and collected by the lens after reaching the subject.

In this case, in order to generate a depth map, the arrival time of the reflected electromagnetic wave per pixel is measured and the flight time is calculated accordingly.

Accordingly, a depth image (depth map) of the subject located in front may be obtained using the flight time calculated for each pixel.

4 and 5 are views illustrating a configuration of the first camera according to the second embodiment.

4 and 5, the first camera may be configured as an array camera.

An array camera refers to a camera in which a plurality of lenses are arranged in one package.

The plurality of lenses of the array camera acquire images of the subjects spaced apart by a predetermined distance h.

In this case, different images are captured according to the distance from the subject. In addition, this may be defined as a delta (Δ) representing a distance between the start point and the end point of the image on the left and right sides, according to the alignment of the specific point with respect to the image taken by each lens.

4 and 5 illustrate changes in delta values from images acquired from subjects located at different distances.

4 defines a delta value corresponding to a subject located at a first position, and FIG. 5 defines a delta value corresponding to a subject located at a second position.

4 and 5, it can be seen that the delta value is appreciated as the distance h from the subject increases.

Accordingly, distance information with respect to the subject may be obtained by using a plurality of images acquired through the array camera, and a depth map may be generated by using the acquired distance information.

As described above, the first camera 110 may be configured as a TOF camera or an array camera, thereby generating a depth map of the distance to the subject.

6 and 7 provide a gaze tracking method of a gaze tracking device according to an exemplary embodiment.

6 and 7, the apparatus for tracking eyes, the TOF camera or the array camera acquires a subject image, and obtains a depth map indicating a distance from the subject from the acquired subject image (step 110).

That is, the TOF camera sends out electromagnetic waves, calculates the arrival time for each pixel accordingly, and generates the depth map.

In addition, the array camera may generate the depth map according to a change in a delta value for an image acquired through a plurality of lenses.

Thereafter, the controller 150 obtains location information on the gaze (eye) of the user from the generated depth map (step 120).

That is, the controller 150 causes the face recognition algorithm to be performed through the image processor 130 to detect the facial feature points of the user, and thus X, Y, and Z corresponding to the feature points of the eye from the facial feature points. Acquire coordinate information.

In operation 130, the controller 150 acquires an enlarged image of an area corresponding to the eyeball position of the user using the obtained X, Y, and Z information. Prior to this, the controller 150 sets a condition for acquiring the enlarged image by using the X, Y, and Z information, and accordingly, the second camera 120 acquires the enlarged image according to the set condition. .

Thereafter, the controller 150 recognizes the pupil of the user from the obtained enlarged image (step 140).

The controller 150 checks the distance between the left pupil and the right pupil with respect to the recognized pupil, and tracks the gaze of the user based on the determined distance (step 150).

Referring to FIG. 7, more specifically, the controller 150 allows a three-dimensional distance map (depth map) to be acquired through the first camera 110 (step 210).

Thereafter, the controller 150 tracks the eyeball of the user in the acquired depth map, and thereby obtains X, Y, and Z coordinate information about the eyeball position (step 220).

The X and Y coordinate information may be obtained from a generally obtained color image, and the Z coordinate information may be obtained from the depth map.

When the X, Y and Z coordinate information is obtained, the controller 150 sets an image enlargement condition of the second camera 120 using the X, Y and Z coordinate information. That is, an enlargement area is set using the X and Y coordinate information, and an enlargement ratio is set using the Z information.

The second camera 120 acquires an enlarged image in which the eyeball part of the user is enlarged using the image enlargement condition set through the controller 150 (operation 230).

In operation 240, the controller 150 recognizes the pupil from the acquired enlarged image. More specifically, the controller 150 recognizes the left pupil and the right pupil from the enlarged image, respectively.

Thereafter, the controller 150 tracks the eyes of the user by using the distance between the left pupil and the right pupil (step 250).

In the meantime, the gaze of the plurality of users may be tracked.

Accordingly, when a plurality of users are located in front of the eye tracking apparatus, the controller 150 performs eye tracking for each of the plurality of located users.

In addition, when the gaze tracking is performed, the gaze information tracked for each of the plurality of users is displayed to provide information about which part of each of the plurality of users is currently watching.

To this end, the controller 150 obtains eyeball position information for a plurality of users located in the distance map, and enlarges an image corresponding to each eyeball position by the second camera based on the obtained eyeball position information. To be acquired.

In addition, the controller 150 tracks the gaze of each user from each of the obtained enlarged images.

Meanwhile, in order to provide various functions according to the gaze tracking, the gaze of one user should be tracked. In this case, when the gaze of the plurality of users is tracked, various functions as described above may be effectively provided.

On the other hand, in general, the viewer watching the display device is located at the closest distance from the display device.

Accordingly, the controller 150 obtains only the eyeball position information of the user located at the closest distance from the acquired depth map, and accordingly performs eye tracking using the acquired eyeball position information.

According to an embodiment, a stereoscopic depth map is constructed using a time of flight (TOF) camera or an array camera and eye gaze tracking is performed using the configured stereoscopic depth map, thereby reducing the cost for constructing the eye tracking apparatus. Can provide various functions according to eye tracking.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

110: First camera
120: Second camera
130:
140:
150:

Claims (22)

Obtaining a depth image of the subject;
Acquiring position information of an eyeball area included in the subject from the acquired depth image; And
And a gaze tracking method of recognizing a pupil included in the eyeball area based on the acquired position information. The method of claim 1,
Using the acquired position information, acquiring an enlarged image in which the pupil area is enlarged;
The eye tracking,
A gaze tracking method performed based on the obtained enlarged image. The method of claim 1,
Wherein the acquiring of the depth image comprises:
Emitting electromagnetic waves to the subject;
Collecting the electromagnetic waves reflected by the subject;
Calculating a flight time of the reflected electromagnetic waves. The method of claim 1,
Wherein the acquiring of the depth image comprises:
Acquiring a plurality of images of the subject using an array camera;
And calculating a delta value that changes according to the distance of the subject from the obtained plurality of images. 3. The method of claim 2,
And obtaining a color image of the subject. 6. The method of claim 5,
Acquiring the location information for the eyeball area,
Recognizing a region where an eyeball of a user is located in the acquired color image;
Acquiring an X coordinate and a Y coordinate with respect to the recognized eye area;
And obtaining a Z coordinate corresponding to a depth of an area corresponding to the X coordinate and the Y coordinate in the acquired depth image. The method according to claim 6,
Acquiring the location information for the eyeball area,
Obtaining location information of an eyeball area for each of the plurality of users when the plurality of users exist in the acquired color image,
Acquiring the enlarged image,
And using the position information, acquiring an enlarged image of an eyeball area for each of the plurality of users. The method according to claim 6,
Acquiring the location information for the eyeball area,
And when the plurality of users exist in the acquired color image, acquiring location information about an eyeball area of the user located at the closest distance among the plurality of users. The method according to claim 6,
Setting a condition for acquiring the enlarged image;
The step of setting the condition includes:
Setting an enlarged area using the X coordinate and the Y coordinate;
And setting an enlargement ratio for the enlarged area using the Z coordinate. The method of claim 9,
Setting the enlargement ratio,
And using the Z coordinate, increasing the magnification ratio as the distance from the user increases. The method of claim 1,
Tracking the eyes of the user,
Recognizing a left pupil and a right pupil from the enlarged image;
And tracking the gaze of the user by using the recognized difference between the left pupil and the right pupil. A first camera acquiring a color image and a depth image of the subject;
An image processor for signal processing an image acquired through the first camera; And
And a controller configured to acquire location information of an eyeball area included in the subject based on the obtained color image and depth image, and to track a user's gaze by recognizing a pupil included in the eyeball area based on the acquired location information. Eye tracking device. 13. The method of claim 12,
The first camera,
And a time of flight (TOF) camera that emits electromagnetic waves to the subject, collects the electromagnetic waves reflected by the subject, and calculates a flight time of the reflected electromagnetic waves to obtain the depth image. 13. The method of claim 12,
The first camera,
And an array camera for acquiring a plurality of images of the subject, and obtaining the depth image by calculating a delta value that changes according to the distance of the subject from the acquired plurality of images. 13. The method of claim 12,
And a second camera configured to acquire an enlarged image obtained by enlarging a specific region of the subject at a preset magnification ratio. 16. The method of claim 15,
The control unit,
A gaze tracking device configured to set an acquisition condition of the magnified image and recognize a pupil included in the eyeball area by using the magnified image acquired according to the acquisition condition. 17. The method of claim 16,
The control unit,
Recognizing the area where the user's eyeball is located in the obtained color image to obtain the X coordinate and Y coordinate for the eyeball area,
And a Z-coordinate corresponding to a depth of an area corresponding to the X and Y coordinates, from the depth image. 18. The method of claim 17,
The control unit,
When a plurality of users exist in the acquired color image, the position information of the eyeball area for each of the plurality of users is obtained.
The second camera,
And a gaze tracking device to obtain an enlarged image of an eyeball area for each of the plurality of users using the plurality of location information. 18. The method of claim 17,
The control unit,
And a plurality of users when the plurality of users are present in the obtained color image, obtaining position information on an eyeball area of a user located at the closest distance among the plurality of users. 18. The method of claim 17,
The control unit,
And an enlarged area to be enlarged through the second camera using the X coordinate and the Y coordinate of the eyeball area, and an enlargement ratio of the enlarged area using the Z coordinate. 21. The method of claim 20,
The enlargement ratio is,
And a gaze tracking device that increases in proportion to the distance from the user according to the Z coordinate. 13. The method of claim 12,
The control unit,
A gaze tracking device for recognizing a left pupil and a right pupil from the enlarged image, and tracking the user's gaze by using the recognized distance difference between the left pupil and the right pupil.

Images