KR101961266B1 - Gaze Tracking Apparatus and Method - Google Patents

Abstract

According to an embodiment of the present invention, there is provided a line-of-sight tracking method comprising: obtaining a depth image of a subject; Acquiring positional information on an eyeball region included in the subject from the obtained depth image; And recognizing the pupil included in the eye region based on the acquired position information and tracking the user's gaze.

Description

[0001] Gaze Tracking Apparatus and Method [0002]

An embodiment relates to a gaze tracking apparatus and a gaze tracking method thereof.

Recently, it has been proposed to measure the viewer's advertisement effect based on the user's gaze tracking, analyze driver's driving behavior and prevent drowsiness, rearrange the menu through analysis of web page consumption behavior, control game using eye interaction, And placement of shelves through analysis of technical training programs and consumption behavior.

Patent Document 1 relates to a computer vision-based gaze tracking method.

This method constitutes a look-up table for the user ' s face and eye characteristic values. In actual line-of-sight tracing, the gaze direction is calculated by measuring the face direction and the center point of the iris, and then comparing the characteristic value with the measured value. The gaze direction is calculated by combining the measured face direction coordinate system and the iris center coordinate system. The direction of the face is measured by finding a correct angle by wearing a T-stick type reference model. The center of the iris is obtained by finding the center of gravity of the iris color by analyzing the color distribution of the iris, Finds the longest horizontal line in the elliptical iris edge, and determines the center point of the horizontal line as the center point of the iris.

However, in this method, in order to increase the reliability in constructing the look-up table by measuring characteristic values of various faces and eyes in advance, there is a restriction to collect and collect characteristics of a number of characteristics in advance. The eye tracking accuracy may be lowered.

The conventional technique disclosed in Patent Document 2 tracks the user's face by recognizing the face of the user as a feature surface by a plurality of feature points and recognizing the direction of the user's face by translating and rotating the feature surface. As a result, highly reliable and accurate pointing and tracking systems can be provided at a price that is affordable for general use. However, since only the feature points of the face are used and the rotation of the eyeball is not considered, it is not an intuitive eye tracking interface. In order to move the cursor to the desired position, the head should be moved continuously.

Non-patent papers propose a method of estimating the 3D position of eyeballs using a stereo camera device composed of two cameras and three infrared lights, and obtaining a gaze vector through this to obtain eye gaze position on a two-dimensional plane screen.

In order to perform this method, the calibration process between two cameras must precede the stereo camera configuration, and the position between the three lights must be accurately set.

This method has a problem that it is expensive to construct a large number of cameras and lighting devices.

Patent Document 1: KR 10-0311605

Patent Document 2: KR 10-0325365

Non-Patent Documents: S. W. Shih and J. Liu, "A Novel Approach to 3-D Gaze Tracking Using Stereo Cameras," IEEE Transactions on Systems, Man and Cybernetics, Part B, vol. 34, no. 1, pp. 234-245, Feb. 2004

In an embodiment, a gaze tracking apparatus and a gaze tracking method for tracking a user's gaze using a TOF (Time of Flight) camera are provided.

Embodiments provide a line-of-sight tracking apparatus and a line-of-sight tracking method that can track a user's line of sight using an array lens.

In addition, the embodiment provides a line-of-sight tracking apparatus capable of performing line-of-sight tracking using the difference between the positions of right and left pupils, and a line-of-sight tracking method therefor.

It is to be understood that the technical objectives to be achieved by the embodiments are not limited to the technical matters mentioned above and that other technical subjects not mentioned are apparent to those skilled in the art to which the embodiments proposed from the following description belong, It can be understood.

According to an embodiment of the present invention, there is provided a line-of-sight tracking method comprising: obtaining a depth image of a subject; Acquiring positional information on an eyeball region included in the subject from the obtained depth image; And recognizing the pupil included in the eye region based on the acquired position information and tracking the user's gaze.

According to another aspect of the present invention, there is provided a gaze tracking apparatus comprising: a first camera for acquiring a color image and a depth image of a subject; An image processor for signal processing an image obtained through the first camera; And a controller for acquiring positional information of an eyeball region included in the subject based on the obtained color image and depth image and recognizing pupils included in the eyeball region based on the obtained positional information, .

According to the embodiment, a stereoscopic depth map is constructed using a TOF (Time of Flight) camera or an array camera, and eye tracking is performed using the stereoscopic depth map, thereby reducing the cost of constructing the eye tracking device It is possible to provide various functions according to eye tracking.

1 is a view illustrating a gaze tracking apparatus according to an embodiment of the present invention.
2 is a view showing a configuration of a first camera according to the first embodiment.
3 is a view for explaining an image acquisition method of the first camera shown in FIG.
4 and 5 are views showing a configuration of a first camera according to the second embodiment.
FIGS. 6 and 7 provide a gaze tracking method of the gaze tracking apparatus according to the embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which: FIG. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

Throughout the specification, when an element is referred to as " comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise.

In order to clearly illustrate the present invention in the drawings, thicknesses are enlarged in order to clearly illustrate various layers and regions, and parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification .

1 is a view illustrating a gaze tracking apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the gaze tracking apparatus includes a first camera 110, a second camera 120, an image processing unit 130, a storage unit 140, and a control unit 150.

The first camera 110 captures an image of a subject positioned in front of the subject, and acquires distance information from the captured image.

The distance information indicates a Depth Map in which the lightness and darkness change in proportion to the distance between the first camera 110 and the subject.

The depth map is obtained by determining the distance between the gaze tracking device and the subject and expressing the distance by the brightness of each subject. That is, according to the depth map, the subject located close to the gaze tracking device (specifically, the first camera) is expressed with a higher contrast level (brighter) than the subject located far away.

Using the depth map, the distance from the gaze tracking device to the subject can be tracked based on the brightness information of each subject represented in the depth map.

The depth map can generally be obtained using a plurality of cameras (stereo cameras).

However, in the embodiment, a depth map indicating the distance to the subject is obtained using a TOF camera using a TOF (Time Of Flight) principle, or the depth map is acquired using an array camera.

A depth map acquisition method using the TOF camera and the array camera will be described later.

The second camera 120 acquires an enlarged image of a specific area of a subject positioned ahead of the subject.

At this time, the second camera 120 may be implemented as a zoom lens, and acquires an enlarged image of the specific area according to a preset specific zoom ratio.

The specific area may be an area where the user's gaze is located in the subject. In other words, the specific region may be an area where the user's eyeball is located.

In addition, the specific zoom ratio may correspond to distance information corresponding to the eyeball of the user. That is, when the distance to the eyeball increases, the zoom ratio is increased in proportion to the zoom ratio, and accordingly, a zoom ratio for acquiring the enlarged image may be set according to the distance information.

The image processor 130 processes the images obtained through the first camera 110 and the second camera 120 and outputs the processed images.

The storage unit 130 may store a program for each signal processing and control in the control unit 150, and may also store signal-processed video, audio, or data signals.

Also, the storage unit 130 may perform a function for temporarily storing video, audio, or data signals input from the outside.

The storage unit 130 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory) RAM, ROM (EEPROM, etc.), and the like.

The controller 150 controls the overall operation of the gaze tracking device.

For example, the control unit 150 controls the first camera 110 to acquire a depth map indicating a distance to the subject positioned in front of the subject.

In addition, the controller 150 recognizes the user located ahead based on the obtained depth map, and acquires coordinate information on the eye position of the user accordingly.

For this, the control unit 150 recognizes the user's face from the image acquired through the first camera 110.

The user face recognition may be performed based on a pattern feature of a user's face component (e.g., mouth, eye, nose, forehead, etc.).

When the face is recognized, the control unit 150 acquires X-coordinate and Y-coordinate information of the position of the eyeball based on the minutiae point of the user's eye (specifically eyeball) from the recognized face.

In addition, the controller 150 confirms the contrast of the position (eye position) corresponding to the X coordinate and the Y coordinate information from the obtained depth map, and obtains the Z coordinate information of the eye position based on the checked contrast do.

The X and Y coordinate information may be coordinate information of a vertical line and a horizontal line of the eyeball position of the user, and the Z coordinate information may be distance information of the eyeball position of the user.

As described above, when the coordinate information on the eye position is obtained, the controller 150 controls the second camera 120 to obtain an enlarged image obtained by enlarging the eye region of the user.

For this, the control unit 150 sets the acquisition condition of the enlarged image by the second camera 120 based on the obtained X, Y, and Z coordinate information.

That is, the controller 150 sets an area to be enlarged through the second camera 120 using the X and Y coordinate information.

Then, the control unit 150 sets an enlargement ratio (zoom ratio) for the set enlarged area using the Z information. That is, the controller 150 increases the enlargement ratio when the distance to the eyeball by the Z information is long, and decreases the enlargement ratio when the distance to the eyeball is close.

The second camera 120 acquires an enlarged image obtained by enlarging the image of the area corresponding to the X and Y coordinate information to a zoom ratio corresponding to the Z coordinate information under the control of the controller 150. [

When the enlarged image is obtained through the second camera 120, the controller 150 analyzes the enlarged image and recognizes the eye (pupil).

In addition, the control unit 150 tracks the user's gaze using the difference in position of the left pupil and the right pupil according to the recognition of the pupil.

At this time, the pupil recognition may be performed by an image processing algorithm performed by the image processing unit 130.

The image processing unit 130 drives the image processing algorithm and the circular detection algorithm on the acquired enlarged image to acquire the pupil of the user. That is, the image processor 130 detects the user's pupil region from the image corresponding to the user's eye position acquired through the second camera 120, and drives the circular detection algorithm to calculate the pupil center point from the detected pupil region .

The circular detection algorithm is a method of detecting a portion having the largest gray level sum of an inner circle and an outer circle of a template by moving a circular detection template composed of an inner circle and an outer circle to a pupil region and detecting a center point of the detected region Can be obtained as a center point.

Also, the image processing unit 130 may perform the local binarization technique in addition to the circular detection algorithm to acquire the pupil center point.

In this way, in order to accurately obtain the center point of the pupil, the image processing unit 130 performs local binarization in addition to the circular detection algorithm, determines the dark part of the binarized area as a pupil area, It can be judged as a substantial center point of the pupil.

The control unit 150 tracks the user's gaze based on the distance between the center point of the left pupil and the center point of the right pupil obtained through the image processing unit 130. [

The user's gaze tracking technology using the distances of the right pupil and the left pupil is a technique well known in the art to which the present invention belongs, and a detailed description thereof will be omitted.

When the user's line of sight is tracked, the control unit 150 determines whether the user's gaze is tracked based on the tracked line of sight, such as viewer advertisement effect measurement, driver's driving behavior analysis and drowsiness prevention, menu rearrangement through analysis of web page consumption behavior, Control, iris information based user authentication, exercise and expertise training programs, and disposition of shelves through consumption behavior analysis.

As described above, in the embodiment, position information on the pupil of the user is acquired with a depth map acquired using one TOF camera or array camera, and the acquiring condition of the enlarged image is obtained using the acquired position information Setting.

Thereafter, an enlarged image of the user's pupil is acquired using the set acquisition condition, a user's pupil is recognized from the acquired enlarged image, and the user's gaze is tracked.

According to the embodiment as described above, a stereoscopic depth map is constructed using a TOF (Time of Flight) camera or an array camera, and eye tracking is performed using the stereoscopic depth map, It can provide various functions according to eye tracking while reducing costs.

2 is a view showing a configuration of a first camera according to the first embodiment.

According to the first embodiment, the first camera 110 is configured as a TOF camera.

Referring to FIG. 2, the TOF camera includes a lens module, an RGB camera, and a D camera.

The TOF camera can acquire a color image and a depth image. The color image may be an image of a subject located in front of the subject, and the depth image may be an image of a depth map expressed in terms of a distance from the subject.

That is, the color image refers to an image obtained by collecting visible light reflected from a subject and viewed by a human eye or a general camera. The color image can be expressed by dividing into R channel, G channel and B channel for each pixel.

Further, the depth image is generated by emitting an electromagnetic wave, not visible light, to the subject and calculating the flying time of the electromagnetic wave reflected from the subject. For example, the electromagnetic wave is emitted to the subject, the infrared rays reflected by the subject are collected, and at the same time, the arrival time of each infrared pixel is measured to calculate each flight time.

Then, the depth image information for each pixel can be generated according to the calculated each flight time.

Referring to FIG. 2, an RGB sensor collects visible light to generate a color image, emits infrared rays through an infrared ray illuminator, transmits infrared rays reflected from a subject through an infrared ray sensor, To generate a depth image.

At this time, the RGB sensor may be represented by an RGB camera, and the color image may be represented by an R channel, a G channel, and a B channel.

Also, as an Infra Red sensor, a D camera can generate a depth image using arrival time due to reflection of infrared rays.

3 is a view for explaining an image acquisition method of the first camera shown in FIG.

Referring to FIG. 3, in order to generate the depth map, a certain electromagnetic wave is emitted from the electromagnetic wave radiator, and the emitted electromagnetic wave is reflected after the object reaches the object and is collected again into the lens.

In this case, to generate a depth map, the arrival time of the reflected electromagnetic wave for each pixel is measured, and the flight time is calculated accordingly.

Accordingly, a depth image (depth map) for a subject located in front of the subject can be obtained by using the calculated flying time for each pixel.

4 and 5 are views showing a configuration of a first camera according to the second embodiment.

Referring to FIGS. 4 and 5, the first camera may be configured as an array camera.

The array camera means a camera in which a plurality of lenses are arranged in the form of a single package.

The plurality of lenses provided in the array camera acquire images for an object separated by a predetermined distance h, respectively.

At this time, different images are captured according to the distance from the object. In addition, it can be defined as a delta (?) Indicating the distance between the start point of the left and right images and the end point of the image taken by each lens centered on a specific point.

Figures 4 and 5 show delta value changes from images obtained from subjects located at different distances.

Fig. 4 defines a delta value corresponding to the object located at the first position, and Fig. 5 defines the delta value corresponding to the object located at the second position.

Referring to FIGS. 4 and 5, it can be seen that the delta value is audited as the distance h from the subject increases.

Accordingly, distance information with respect to the subject can be obtained using a plurality of images acquired through the array camera, and a depth map can be generated using the obtained distance information.

As described above, the first camera 110 may include a TOF camera or an array camera, thereby generating a depth map of the distance to the subject.

FIGS. 6 and 7 provide a gaze tracking method of the gaze tracking apparatus according to the embodiment.

Referring to FIGS. 6 and 7, the TOF camera or the array camera acquires a subject image and acquires a depth map indicating a distance from the acquired subject image to the subject (Step 110).

That is, the TOF camera sends an electromagnetic wave, calculates the arrival time for each pixel accordingly, and generates the depth map.

Also, the array camera can generate the depth map according to the change of the delta value of the image obtained through the plurality of lenses.

In step 120, the controller 150 obtains the position information of the user's eyeball (eyeball) from the generated depth map,

That is, the controller 150 causes the facial recognition algorithm to be performed through the image processor 130 to detect facial feature points of the user, thereby detecting X, Y, and Z corresponding to the feature points of the eye from the facial feature points, Coordinate information is obtained.

In operation 130, the control unit 150 acquires an enlarged image of an area corresponding to the user's eye position using the obtained X, Y, and Z information. The control unit 150 sets conditions for acquiring the enlarged image using the X, Y, and Z information, and the second camera 120 accordingly acquires the enlarged image according to the set conditions .

Thereafter, the controller 150 recognizes the pupil of the user from the acquired magnified image (step 140)

Then, the controller 150 confirms the distance between the left pupil and the right pupil of the recognized pupil, and tracks the user's gaze based on the determined distance (step 150).

Referring to FIG. 7, in step 210, the control unit 150 acquires a stereoscopic distance map (depth map) through the first camera 110.

The control unit 150 tracks the user's eyeball in the acquired depth map, and acquires X, Y, and Z coordinate information of the eyeball according to the obtained depth map.

The X, Y coordinate information can be generally obtained from the obtained color image, and the Z coordinate information can be obtained from the depth map.

When the X, Y, and Z coordinate information is obtained, the controller 150 sets the image enlargement condition of the second camera 120 using the X, Y, and Z coordinate information. That is, an enlarged area is set using the X, Y coordinate information, and an enlargement ratio is set using the Z information.

The second camera 120 acquires an enlarged image obtained by enlarging the eyeball portion of the user using the image enlargement condition set through the controller 150 (Step 230).

Thereafter, the controller 150 recognizes the pupil from the acquired enlarged image (operation 240). More specifically, the control unit 150 recognizes the left pupil and the right pupil from the enlarged image, respectively.

In step 250, the controller 150 tracks the user's gaze using the distance between the left pupil and the right pupil.

On the other hand, in the above, the eyes of a plurality of users can be tracked.

Accordingly, when a plurality of users are positioned in front of the gaze tracking device, the controller 150 performs gaze tracking for each of the plurality of users located.

In addition, when the line-of-sight tracking is performed, the line-of-sight information tracked for each of the plurality of users is displayed, and information on which part of the current screen is viewed by each of the plurality of users is provided.

For this purpose, the controller 150 acquires the eye position information for a plurality of users located in the distance map, and based on the acquired eye position information, the enlarged image corresponding to each eye position by the second camera .

In addition, the control unit 150 tracks the gaze for each user from the obtained enlarged images.

Meanwhile, in order to provide various functions according to the line-of-sight tracking, the gaze of one user must be tracked. At this time, if the eyes of the plurality of users are tracked, the various functions as described above can be effectively provided.

On the other hand, in general, the viewer who is watching the display device is located at the closest distance from the display device.

Accordingly, the controller 150 acquires eye position information of the user located at the closest distance from the obtained depth map, and performs gaze tracking using the obtained eye position information.

According to the embodiment, a stereoscopic depth map is constructed using a TOF (Time of Flight) camera or an array camera, and eye tracking is performed using the stereoscopic depth map, thereby reducing the cost of constructing the eye tracking device It is possible to provide various functions according to eye tracking.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

110: First camera
120: Second camera
130:
140:
150:

Claims (22)

Acquiring a color image frame in which a subject is photographed, and a depth image frame, which is distance information between the subject and the camera;
Performing a face recognition algorithm on the color image frame to detect a feature point including an eye region of the face of the subject;
Obtaining information on the eye region;
Obtaining an image frame in which the eyeball region is enlarged using information on the eyeball region;
Detecting a pupil region by performing a binarization technique on an image frame in which the eyeball region is enlarged; And
Performing a circular detection algorithm on the pupil region to obtain a center position of the pupil;
And tracking the gaze of the subject using the center position of the pupil,
Wherein the binarization technique binarizes an image frame in which the eyeball region is enlarged, determines a dark portion of the binarized region as the pupil region,
Wherein the circular detection algorithm moves a circular detection template composed of an inner circle and an outer circle to the pupil region of the image frame in which the eyeball region is enlarged and detects a portion having a largest difference in gray level sum of the inner circle and the outer circle of the template And acquires the center point of the detected area as the center point of the pupil. The method according to claim 1,
The information of the eyeball region
Position information calculated as an X coordinate value and a Y coordinate value in the color image frame,
And depth information calculated as Z coordinate values obtained based on the contrast level in the depth image. 3. The method of claim 2,
The contrast level
The distance between the camera and the subject is represented by brightness,
Determining a distance from the camera to the subject and expressing the proximity of the subject in a higher contrast level than the subject located far away from the camera. 3. The method of claim 2,
The step of acquiring an enlarged image frame of the eye region
Setting an enlarged area using the position information,
And setting an enlargement ratio for an image frame in which the eyeball region is enlarged using the depth information. 5. The method of claim 4,
The enlargement ratio,
Wherein the Z-coordinate is set to be proportional to a distance between the camera and the subject. The method according to claim 1,
And acquiring an image frame in which the eyeball region is enlarged for each of the plurality of subjects when the plurality of the subjects are present in the obtained color image frame, and tracking a line of sight for each of the plurality of subjects. A color image frame photographing a subject;
A first camera for acquiring a depth image frame which is distance information between the subject and the camera;
A second camera for acquiring an enlarged image frame;
An image processor for signal processing the color image frame; And
Controlling the second camera and the image processing unit to acquire the enlarged image frame in which the eyeball region of the subject is enlarged,
And a controller for recognizing the pupil included in the eye region and tracking the gaze of the subject
Wherein the image processing unit performs a face recognition algorithm such that a feature point including an eye of the subject face is detected in the color image frame, performs a binarization technique to obtain the pupil area in the enlarged image frame, Performing a circular detection algorithm to obtain a center point of the pupil,
Wherein the binarization technique binarizes an image frame in which the eyeball region is enlarged, determines a dark portion of the binarized region as the pupil region,
Wherein the circular detection algorithm moves a circular detection template composed of an inner circle and an outer circle to the pupil region of the image frame in which the eyeball region is enlarged and detects a portion having a largest difference in gray level sum of the inner circle and the outer circle of the template And acquires a center point of the detected area as a center point of the pupil. 8. The method of claim 7,
The control unit
Position information calculated as an X coordinate value and a Y coordinate value in the color image frame,
And depth information calculated as a Z coordinate value obtained based on the contrast level in the depth image. 9. The method of claim 8,
The contrast level
The distance between the first camera and the subject is represented by brightness,
Wherein the distance from the first camera to the subject is determined, and the subject located close to the subject is expressed with a higher contrast level than a subject located far away from the first camera. 9. The method of claim 8,
Wherein,
Setting an enlarged area using the position information,
And controls the second camera by setting an enlargement ratio for the enlarged area using the depth information. 11. The method of claim 10,
The enlargement ratio,
And increases in proportion to a distance between the first camera and the subject corresponding to the Z coordinate. 8. The method of claim 7,
Wherein the first camera comprises:
And an array camera for acquiring a plurality of the color image frames for the subject and calculating the delta value varying from the plurality of acquired color image frames according to the distance of the subject to obtain the depth image frame, Wherein the delta value decreases as the distance from the subject increases and the delta value increases as the distance from the subject decreases. 8. The method of claim 7,
Wherein,
And acquires an image frame in which the eyeball region is enlarged for each of the plurality of subjects when the plurality of subjects are present in the color image frame and tracks a line of sight of each of the plurality of subjects. delete delete delete delete delete delete delete delete delete

Images