KR101628542B1 - Eye Tracking Apparatus and Method - Google Patents

Abstract

The present invention relates to a line-of-sight tracking apparatus and method, and more particularly, to a line-of-sight tracing apparatus and method for detecting eye information of a user using two or more cameras, Extracting displacement information of a previous image acquired through the remaining camera corresponding to the face information from a look-up table; calculating a line-of-sight vector using the displacement information; .

Description

Eye Tracking Apparatus and Method

The present invention relates to a line-of-sight tracking method, and more particularly, to a line-of-sight tracking method for calculating a line-of-sight vector using previous image information when any one of two or more cameras is blocked by an obstacle, And methods.

Generally, the gaze tracking device tracks the gaze using images acquired through two cameras. Thus, when any one of the two cameras is blocked by an obstacle such as a driver's hand or a steering wheel or the like, the eye tracking can not be performed when the user can not detect the eye information.

In particular, conventionally, a stereo camera camera and a camera projection matrix are required to calculate a line-of-sight vector. Since the eye vector is calculated through the correlation between the two cameras, the eye vector can not be calculated if the eye image of the user can not be acquired through any one of the two cameras.

SUMMARY OF THE INVENTION The present invention has been conceived to solve the above-described problems of the conventional art, and it is an object of the present invention to provide a method and a system for calculating eyeball vectors by using previous image information when any one of two or more cameras is obstructed by an obstacle, Eye gaze tracking apparatus and method.

According to an aspect of the present invention, there is provided a method of tracking a line of sight of a user using two or more cameras, The method comprising the steps of: detecting eye information in the current image; calculating face information in the current image; extracting displacement information of a previous image acquired through the remaining camera corresponding to the face information from a lookup table; And calculating a line-of-sight vector using the line-of-sight vector.

The face information may include at least one of face size, eye size, nose size, and mouth size.

The displacement information is depth information in the eye region.

The eye vector calculation step may further include calculating world coordinates of the eye information of the previous image using the displacement information, calculating eye coordinates of the eyes using the world coordinates of the eye information and eye information detected from the current image, And a step of calculating a difference between the first and second values.

Further, the eye information may include at least one of a pupil and an illumination reflex point.

Meanwhile, the gaze tracking apparatus according to an embodiment of the present invention may include a stereo camera configured by two or more cameras to acquire an image of a user, and a lookup table composed of information on previous images acquired through the two or more cameras And a control unit for calculating a line-of-sight vector using a current image and a previous image acquired through a remaining camera in which no blur occurs when a blurring phenomenon occurs in any one of the cameras of the two or more cameras .

In addition, the blurring phenomenon is a state in which the user can not acquire an image as the camera lens is covered by the obstacle.

The lookup table may include face information and displacement information in the eye region.

The control unit may calculate face information from the current image and extract displacement information of a previous image acquired through the remaining camera corresponding to the face information from the lookup table.

The controller calculates the world coordinates of the eye information of the previous image using the displacement information and calculates the eye vector using the world coordinates of the eye information of the previous image and the eye information detected in the current image .

The gaze tracking apparatus according to the present invention can calculate the gaze vector using the previous image information when any one of the two or more cameras is obstructed by the obstacle and the eye information of the user can not be detected. Therefore, the eye-gaze tracking apparatus according to the present invention enables continuous tracking of a gaze even when a blur occurs in one camera due to an obstacle such as a steering wheel.

In addition, the line-of-sight tracking method according to the present invention combines the driver's seating characteristics and the position information of the line-of-sight tracking device in the vehicle, thereby enabling the line-of-sight tracking specific to the vehicle.

In addition, the gaze tracking device can track a user's gaze using only one camera when generating an eye information lookup table for only a specific user. Therefore, a hardware saving effect can be expected.

1 is a block diagram of a gaze tracking apparatus according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a gaze tracking method according to an embodiment of the present invention; FIG.
FIG. 3 is a diagram for explaining a lookup table generation process shown in FIG. 2; FIG.
4 is a flowchart illustrating a gaze tracking method according to another embodiment of the present invention.
FIG. 5 is a view for explaining a process of extracting previous image information in FIG. 4; FIG.

The terms "comprises", "comprising", "having", and the like are used herein to mean that a component can be implanted unless otherwise specifically stated, Quot; element ".

Also, the terms " part, "" module, " and" module ", as used herein, refer to a unit that processes at least one function or operation and may be implemented as hardware or software or a combination of hardware and software . It is also to be understood that the articles "a", "an", "an" and "the" Can be used.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The eye-gaze tracking apparatus and method proposed in the present specification can be applied to a line-of-sight tracking apparatus and a line-of-sight tracking apparatus in a vehicle in which a driver's body moves only with a seat in a vehicle, It is pointed out that the distance can be calculated. Accordingly, the present invention detects eye information (a pupil and / or an illumination reflex point) from an image acquired through two or more cameras, calculates a face size in each image and / or a size of a specific portion in the face, ≪ / RTI > Further, the present invention generates a look up table (LUT) using the face size and / or the size of a specific portion in the face and the in-eye displacement. The lookup table is used to check previous image (frame) information when a blurring occurs in any one of two or more cameras.

1 is a block diagram of a gaze tracking apparatus according to an embodiment of the present invention.

1, the eye-gaze tracking apparatus 100 includes a stereo camera 110, an illumination unit 120, a memory 130, and a control unit 140. As shown in FIG.

The stereo camera 110 includes a first camera 111 and a second camera 112. In the present embodiment, the gaze tracking apparatus 100 includes two cameras. However, the present invention is not limited to this, and three or more cameras may be provided.

The stereo camera 110 simultaneously acquires an image of the same subject (user) through the first camera 111 and the second camera 112. The cameras 111 and 112 may be a CCD (charge coupled device) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a charge priming device (CPD) And an image sensor such as an infrared image sensor or the like.

The illumination unit 120 emits light (e.g., infrared rays) toward a subject (user). The illumination unit 120 may be implemented with one or more infrared light emitting devices (IR LEDs).

The memory 130 stores various programs and various data for performing eye tracking. The memory 130 stores a lookup table generated by using the information of each image (frame) acquired through the stereo camera 110. Here, the image information includes face information including at least one of face size, eye size, mouth size, and nose size, and disparity (depth information) in the eye area. In the present embodiment, it is described that the intra-eye area displacement information is used. However, the present invention is not limited to this, and a representative value of a whole image or a pupil area may be used.

The control unit 140 controls the display unit 110 to display a gaze vector using the previous image (frame) information when a blurring occurs in any one of the cameras 111 and 112 of the stereo camera 110 . In other words, when the lens of the first camera 111 or the second camera 112 is blocked by an obstacle such as a handle or a user's hand, the controller 140 controls the camera 111 or 112 The line-of-sight vector is calculated using the previously obtained image information.

The control unit 140 includes an eye detection module 141, a pupil detection module 143, an illumination reflection point detection module 145, and a gaze vector calculation module 147.

The eye detection module 141 detects face regions from the respective images output from the two cameras 111 and 112. At this time, the eye detection module 141 uses face detection algorithms such as adaboost, support vector machine (SVM), and neural network. Then, the eye detection module 141 may calculate the size of the detected face area.

The eye detection module 141 detects the eye region in the detected face region. The eye detection module 141 uses eye detection algorithms such as an adaboost, adaptive template matching, CAMShift, and a rapid eye detection method.

Further, the eye detection module 141 calculates the size of the detected eye region and stores the calculated size in the memory 130.

The pupil detection module 143 detects the pupil region in the eye region. At this time, the pupil detection module 143 detects a pupil region using a pupil detection algorithm such as a circle fitting and an ellipse fitting.

The illumination reflection point detection module 145 extracts illumination reflection points from the pupil region. Here, the illumination reflection point detection module 145 detects an illumination reflection point in the pupil area using algorithms such as template matching, circle fitting, and elliptic fitting.

The line-of-sight vector computing module 147 computes a line-of-sight vector (3D vector) using at least one of the pupil and the illumination reflection points detected from the image obtained through the cameras 111 and 112. At this time, the line of sight vector calculation module 147 uses camera information such as a camera projection matrix.

The line-of-sight vector computing module 147 receives two images of the same subject simultaneously captured through the two cameras 111 and 112 in a section where no blur occurs, and calculates a displacement map for each image using the camera correlation disparity map) and stores the displacement information (depth information) in the eye region together with the face information to generate a lookup table. The line-of-sight vector calculation module 147 generates image information for a predetermined image (frame) or more and registers the image information in the look-up table. For example, the lookup table includes image information of at least five frames.

If the eye vector calculation module 147 can not detect the eye information (pupil and / or illumination reflection point) from the image obtained through the camera 111 or 112 of either of the two cameras 111 and 112, And calculates a line-of-sight vector using the previous image information acquired through the camera 111 or 112. For example, when a blind phenomenon occurs in which the lens of any one of the cameras 111 and 112 is held by an obstacle such as a handle or a user's hand, And calculates a line-of-sight vector using the previous image and the current image acquired through the image processing unit (111 or 112).

At this time, the line-of-sight vector calculation module 147 calculates face information from the image obtained through the cameras 111 and 112 without occurrence of the blurring, and reads the displacement information corresponding to the face information calculated from the look-up table. Then, the line-of-sight vector computing module 147 computes the line-of-sight vector using the eye information detected from the image obtained through the camera displacement information and the camera 111 or 112 capable of detecting eye information. The eye information may be the center point coordinates of the pupil and / or the illumination reflex point.

FIG. 2 is a flowchart illustrating a gaze tracking method according to an embodiment of the present invention. FIG. 3 is a diagram for explaining a lookup table generation process shown in FIG. The present embodiment will be described taking as an example a case where no blurring occurs in the two cameras 111 and 112.

First, the control unit 140 acquires an image through the first camera 111 and the second camera 112, respectively (S101). That is, the first camera 111 and the second camera 112 simultaneously photograph the same subject (e.g., a user).

The control unit 140 detects a face region in each image acquired through the first camera 111 and the second camera 112, and detects an eye region in the detected face region (S103).

The control unit 140 detects a pupil region in the eye region using a pupil detection algorithm such as a circle fitting and an elliptic fitting (S105).

The control unit 140 detects an illumination reflection point in the pupil region (S107).

The control unit 140 calculates the line-of-sight vector using the illumination reflection point (S109). For example, the control unit 140 calculates a gaze vector as a direction perpendicular to the center of the illumination reflection point.

The control unit 140 calculates face information and intra-eye area displacement information from each image acquired through the first camera 111 and the second camera 112 and uses the calculated face information and in-eye area displacement information to perform lookup And creates a table (S111).

3, the first camera 111 and the second camera 112 acquire images (Nth LImg, Nth RImg) of the user U at the same time as the gaze tracking apparatus 100. Then, the gaze tracking device 100 calculates the face size and the depth value in the eye area from each image. Here, the face size includes the length and the width of the face. The gaze tracking apparatus 100 generates a look-up table (Left Img LUT) with the face size and depth value extracted from the image acquired through the first camera 111, and extracts the face extracted from the image acquired through the second camera 112 Creates a look-up table (Right Img LUT) with size and depth values. In addition, the gaze tracking device 100 extracts the face size and depth value for each image acquired through the two cameras 111 and 112, and registers the extracted face size and depth value in the lookup table.

FIG. 4 is a flowchart illustrating a gaze tracking method according to another embodiment of the present invention. FIG. 5 illustrates a process of extracting previous image information in FIG. In this embodiment, a case where a lens of one of the cameras 111 and 112 is blocked by an obstacle such as a handle or a user's hand will be described as an example. In this case, eye information (pupil and illumination reflex points) can not be detected through the camera 111 or 112 where the blurring has occurred.

First, the control unit 140 acquires an image through the camera 111 or 112 that has not undergone the blurring, and detects eye information (pupil and / or illumination reflection point) from the acquired image (S121). The control unit 140 detects the eye information through the pupil and illumination reflection point detection process shown in FIG.

The control unit 140 calculates face information from the current image acquired through the camera 111 or 112 that has not caused the blur (S123). Here, the face information includes at least one of face size, eye size, mouth size, and nose size, and the size includes length and width information.

The control unit 140 extracts the displacement information of the previous image corresponding to the calculated face information from the look-up table (S125). Here, the displacement information means depth information.

The controller 140 calculates the world coordinates of the eye information in the previous image using the displacement information (S127).

The control unit 140 calculates a line-of-sight vector using the world coordinates of the eye information in the previous image and the eye information detected in the current image (S129).

For example, as shown in FIG. 5, the first camera 111 may acquire an image of the user U, but the second camera 112 may acquire the image of the user U by obstructing the obstacle O The image can not be acquired. In this case, the gaze tracking device 100 calculates the face size (width, length) of the user U from the current image LImg acquired through the first camera 111. [ The gaze tracking apparatus 100 has a face size of 3 and a length of 4 in a look-up table (Left Img LUT) in which information about an image acquired through the first camera 111 is stored, ), The depth value 4 is extracted. The gaze tracking device 100 calculates a gaze vector (3D vector) using the extracted depth value.

The embodiments described above are those in which the elements and features of the present invention are combined in a predetermined form. Each component or feature shall be considered optional unless otherwise expressly stated. Each component or feature may be implemented in a form that is not combined with other components or features. It is also possible to construct embodiments of the present invention by combining some of the elements and / or features. The order of the operations described in the embodiments of the present invention may be changed. Some configurations or features of certain embodiments may be included in other embodiments, or may be replaced with corresponding configurations or features of other embodiments. It is clear that the claims that are not expressly cited in the claims may be combined to form an embodiment or be included in a new claim by an amendment after the application.

Embodiments in accordance with the present invention may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, an embodiment of the present invention may include one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) field programmable gate arrays, processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, a procedure, a function, or the like which performs the functions or operations described above. The software code can be stored in a memory unit and driven by the processor. The memory unit is located inside or outside the processor, and can exchange data with the processor by various known means.

It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit of the invention. Accordingly, the foregoing detailed description is to be considered in all respects illustrative and not restrictive. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

100: eye tracking device
110: Stereo camera
111: First camera
112: Second camera
120:
130: memory
140:
141: eye detection module
143: pupil detection module
145: Illumination Reflection Point Detection Module
147: eye vector operation module

Claims (10)

Detecting eye information in a current image acquired through a remaining camera when the eye information of the user can not be detected through any one of the cameras during the eye tracking using the two or more cameras;
Calculating face information from the current image;
Extracting displacement information of a previous image acquired through the remaining camera corresponding to the face information from a lookup table;
And calculating a line-of-sight vector using the displacement information. The method according to claim 1,
Wherein the face information comprises:
Face size, eye size, nose size, mouth size, and the like. The method according to claim 1,
The displacement information,
Wherein the depth information is depth information in an eye area. The method according to claim 1,
The line-of-
Calculating world coordinates of eye information of the previous image using the displacement information,
And calculating a line-of-sight vector using the world coordinates of the eye information and the eye information detected in the current image. 5. The method of claim 4,
The eye information includes:
A pupil, and an illumination reflex point. A stereo camera which is composed of two or more cameras and acquires a user's image,
A memory for storing a lookup table including information on previous images acquired through the two or more cameras;
And a control unit for calculating a line-of-sight vector using a current image and a previous image acquired through a remaining camera in which no occlusion phenomenon occurs when a blurring phenomenon occurs in any one of the cameras of the two or more cameras, Device. The method according to claim 6,
The above-
Wherein the camera is in a state in which the user can not acquire an image as the camera lens is blocked by an obstacle. The method according to claim 6,
The look-
Face information, and intra-eye area displacement information. The method according to claim 6,
Wherein,
Wherein the face information is calculated from the current image and the displacement information of the previous image acquired through the remaining camera corresponding to the face information is extracted from the lookup table. 10. The method of claim 9,
Wherein,
Calculating world coordinates of the eye information of the previous image by using the displacement information, and calculating eye coordinates by using the world coordinates of the eye information of the previous image and the eye information detected by the current image, Device.

Images