KR101601548B1 - In-Vehicle Gaze Tracker Having Movable Light and Method for Detecting Glint thereof - Google Patents

Abstract

The present invention relates to a gaze tracking apparatus having a movable light, and a method for detecting a light reflection point thereof. The gaze tracking apparatus comprises: a camera installed to face a user in a vehicle so as to acquire an image of the user; a light for irradiating light to the user; a moving structure structurally combined with the light, and moving the light in a predetermined movement orbit; and an image processing unit for detecting a light reflection point from the image by predicting the light reflection point due to the light based on prior information of the moving structure.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a gaze tracking device having moving lights,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight tracking technique, and more particularly, to a line-of-sight tracking apparatus having moving lights for detecting a line of sight by using illumination moving in a predetermined trajectory, .

In general, a gaze tracker detects a pupil region in an image acquired through a camera, detects an illumination reflex point (glint) in the pupil region, calculates a gaze vector using the detected illumination reflex point, and performs gaze tracking .

Particularly, in the conventional eye tracker, an image is binarized using a threshold technique based on a specific brightness value, in view of the fact that the illumination reflection point in the infrared image has a remarkably bright value, Is detected as a reflection point.

However, in the case of using one light source, such a conventional technique uses a predefined value of the distance from the illumination reflection point to the center of eye eye, and therefore, correction is necessary since it has different characteristics for each driver.

In addition, since the prior art uses one point light source, it is highly likely to be misunderstood because it is difficult to distinguish it from other reflection points when detecting an illumination reflection point.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above problems of the conventional art, and it is an object of the present invention to provide a gaze tracking device having moving lights for detecting an illumination reflex point using illumination moving in a predetermined orbit, .

According to an aspect of the present invention, there is provided a line-of-sight tracking apparatus having moving lights, including at least one camera installed to face a user in a vehicle and acquiring an image of the user, A moving structure structured and configured to irradiate light toward the user; a moving structure structurally coupled to the illumination and moving the illumination to a predetermined movement orbit; And an image processing unit for detecting the illumination reflection point in the image.

The moving trajectory may be a predetermined trajectory such as a circular trajectory, an elliptical trajectory, a rectangular trajectory, and a linear reciprocating trajectory.

The image processing unit may detect a pupil region in the image, detect a candidate reflex point in the pupil region, calculate a movement trajectory of the illumination, calculate a moving distance average of the candidate reflex points, Detecting a region of interest in the pupil region using a moving distance average of candidate reflections, and detecting an illumination reflex point within the detected region of interest.

The image processing unit generates an illumination reflection point template based on the dictionary information of the moving structure, and detects reflection points in the image through template matching using the illumination reflection point template.

The image processing unit converts the image into a binary image, detects an outline on the binary image, detects a candidate reflection point out of the outlines detected through fitting, and selects one of the candidate reflection points based on the dictionary information of the moving structure And the reflection point is determined to be an illumination reflection point.

Further, the image processing unit uses the center point of the illumination reflection point as a representative value.

Further, the fitting includes a circular fitting and an elliptic fitting.

According to another aspect of the present invention, there is provided a method for detecting an illumination reflex point of a gaze tracking device having moving light, comprising the steps of: acquiring an image through a camera; detecting a pupil area in the image; Predicting the generation of the illumination reflection point based on the advance information of the moving structure moving to the orbit, and detecting the illumination reflection point in the pupil area through prediction of the illumination reflection point.

The illumination reflection point detection step may include detecting a candidate reflection point in the pupil region, calculating a movement trajectory of the illumination using the coordinate value of the candidate reflection point, and calculating a moving distance average of the candidate reflection point Wherein the method further comprises the steps of: designating a region of interest in the pupil region using a movement trajectory of the illumination and an average travel distance of the candidate refractory point; and detecting an illumination reflex point in the region of interest do.

The illumination reflection point detection step may detect the illumination reflection point most similar to the illumination reflection point template in the pupil area through template matching using the illumination reflection point template generated based on the advance information of the moving structure .

The illumination reflection point detection step may include a step of converting the image detected in the pupil region into a binary image, a step of detecting an outline on the binary image, a step of detecting a candidate reflection point of the outline through fitting, Selecting one candidate refracting point from the candidate refracting points based on the advance information of the moving structure; and determining the selected candidate refracting point as an illumination refracting point.

Further, the advance information of the moving structure includes a moving locus, a moving speed, and a turning radius.

The eye-gaze tracking apparatus according to the present invention allows one light source (illumination) to move to a predetermined orbit, and thus the speed and shape of the rotation of the illumination reflex point can be known in advance by the advance information owing to the characteristic that the light source moves to a predetermined orbit, Information can be used to easily detect an illumination reflection point.

In addition, according to the present invention, it is not necessary to further implement additional hardware when attaching illumination to a rotating steering wheel or the like in a vehicle, thereby preventing an increase in production cost.

Further, according to the present invention, it is possible to achieve the same effect as using a plurality of light sources as one light source.

Further, according to the present invention, since the moving structure for moving the illumination to the predetermined orbit is provided, the movement of the illumination can be predicted using the advance information of the moving structure. Therefore, the eye-gaze tracking apparatus of the present invention can accurately detect the illumination reflection point by predicting the shape (shape) of the reflection point using the motion of the illumination.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram illustrating a gaze tracking device with moving light in accordance with one embodiment of the present invention. FIG.
FIG. 2 is an exemplary view showing an illumination refracting point in an image when the moving structure is moving at a high speed according to an embodiment of the present invention; FIG.
3 is an exemplary diagram illustrating an illumination refocus template associated with an embodiment of the present invention;
4 is a flowchart illustrating a method of detecting an illumination reflex point of a gaze tracking apparatus according to an embodiment of the present invention.
5 is a flowchart illustrating a method of detecting an illumination reflex point of a gaze tracking apparatus according to another embodiment of the present invention.
6 is a flowchart illustrating a method of detecting an illumination reflex point of a gaze tracking apparatus according to another embodiment of the present invention.

The terms "comprises", "comprising", "having", and the like are used herein to mean that a component can be implanted unless otherwise specifically stated, Quot; element ".

Also, the terms " part, "" module, " and" module ", as used herein, refer to a unit that processes at least one function or operation and may be implemented as hardware or software or a combination of hardware and software . It is also to be understood that the articles "a", "an", "an" and "the" Can be used.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present invention relates to a line-of-sight tracking apparatus which attaches an illumination (for example, an infrared light source) to a moving structure moving by a predetermined trajectory (rotation, linear reciprocation, etc.) The shape of the illumination reflection point generated in the image can be predicted according to the movement of the illumination. The line-of-sight tracking apparatus according to the present invention can accurately detect an illumination reflex point using an illumination reflex point prediction model capable of predicting the shape of an illumination reflex point according to the movement of illumination.

1 is a block diagram illustrating a gaze tracking apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the eye tracking apparatus includes a camera 110, an illumination unit 120, a moving structure 130, a memory 140, and an image processing unit 150.

The camera 110 is installed to face the user's face in front of the vehicle and photographs the user's image (e.g., a snow image or a face image). For example, the camera 110 may be mounted on a windshield, a cluster, a steering wheel, or the like in a vehicle.

The camera 110 may be implemented by a stereo camera or an infrared camera including an infrared camera and an infrared filter. The camera 110 may be composed of one or more image sensors. In this case, the image sensor may be a CCD (charge coupled device) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a charge priming device (CPD) image sensor, a CID And may be implemented as an image sensor of any one of the sensors.

The illumination 120 serves to illuminate the light towards the user. The illumination 120 is comprised of one or more light sources that emit infrared light. The light source may be an infrared light emitting device or an infrared lamp.

For example, the camera 110 is installed so as not to move in the center of the steering wheel, and the illumination 120 can be attached to the rim of the steering wheel. Alternatively, the light 120 may be installed on a structure that moves at a constant speed among the in-vehicle accessories.

The moving structure 130 serves to move the illumination 120 attached to one side to a predetermined orbit (movement path). The moving structure 130 causes the illumination 120 to move (travel) around any one of various orbits such as an elliptical orbit, a circular orbit, a rectangular orbit, and a linear reciprocating orbit around the camera 110. In this embodiment, the moving structure 130 rotates 360 degrees by way of example, but the rotation angle may be changed by 60 degrees or 90 degrees.

In the memory 140, an illumination reflection point (glint) template generated in the image according to the movement (movement) pattern of the illumination 120 is stored. The memory 140 stores a reflection point prediction model.

The image processing unit 150 detects the eye region in the image acquired through the camera 110 and detects the pupil region in the eye region. Then, the image processing unit 150 detects the illumination reflection points (reflection points) in the pupil region using the reflection point prediction model.

Since the image processing unit 150 differs in the method of detecting the reflection points according to the moving speed of the moving structure 130, the image processing unit 150 will be described separately.

First, a process (process) of detecting a reflection point by the image processing unit 150 when the movement speed of the moving structure 130 is low will be described.

,

), (

,

), …, (

,

)를 검출한다.The image processor 150 detects a candidate spot having a brightness value that is equal to or greater than a critical brightness value in the pupil region. At this time, the image processing unit 150 calculates the pixel coordinates of the candidate reflection point detected from each image (frame) acquired through the camera 110

,

), (

,

), ... , (

,

).

The image processing unit 150 calculates the variables a, b, and r by substituting the detected coordinate values of the candidate reflection points into the equation (1) of a circle having a radius r centered on the coordinate points (a, b).

,

)의 각도를 의미한다.The image processor 150 obtains an average of the movement distances between the detected candidate reflections. That is, the image processor 150 calculates the distance between the candidate reflection point detected in the current image and the candidate reflection point detected in the previous image. At this time, the image processing unit 150 calculates the movement angle [theta] between the start angle alpha and the candidate reflection point in the following [Equation 2]. The start angle α is the position of the candidate reflections detected from the first image based on the x axis in the coordinate plane

,

).

The image processing unit 150 predicts the position of the detected reflection point using [Equation 3].

Here, t = n + 1.

이다. 여기서, R은 관심영역 범위 지정 값이다.The image processing unit 150 calculates a region of interest (ROI) in which the reflection point is likely to be located from the average of the original equation and the moving distance. Interest area

to be. Here, R is a region-of-interest range designation value.

The image processor 150 designates a region of interest in the image acquired through the camera 110 and detects a reflection point within the region of interest.

The image processing unit 150 may perform a preprocessing step before detecting a reflection point and may correct an image acquired through the camera 110 using a warping method, an affine transform, and a position transform function.

Next, when the moving structure 130 is moving at a high speed, two reflection point detection processes will be described. If the moving structure 130 is moving at a high speed, the glint g is not represented by the exposure time but represented by a locus as shown in FIG.

The first explains how to detect reflections using template matching.

The image processing unit 150 generates an illumination reflection point template as shown in FIG. 3 based on the pre-information about the moving structure 130 that moves the illumination 120, and stores the generated illumination reflection point template in the memory 140. At this time, the template may be created through experiments beforehand. The advance information includes the moving speed and moving pattern of the moving structure 130, the turning radius, and the like.

The image processing unit 150 detects a reflection point through template matching in the eye region using a template specified in advance. That is, the image processing unit 150 detects a reflection point most similar to the template in the eye image. The image processing unit 150 performs template matching through stereo matching such as a sum of absolute difference (SAD) and a sum of squared difference (SSD).

In addition, the image processing unit 150 generates an image pyramid by enlarging or reducing the eye image at a predetermined ratio, and then detects a reflection point by template matching for each image size.

The second explains a method of detecting the reflection points using the advance information of the moving structure 130. FIG.

The image processing unit 150 converts the image (e.g., eye image) acquired through the camera 110 into a binary image using a thresholding technique. That is, the image processing unit 150 changes the pixels having brightness values higher than the threshold value set in the image to white "1" and the pixels lower than the threshold value to black "0".

The image processing unit 150 detects an outline (contour) on the binary image.

The image processor 150 detects a candidate reflex point (candidate reflex point locus) out of the outlines detected through the ellipse fitting and the circle fitting.

The image processing unit 150 detects candidate reflection points similar to the width and height ratio and the radius size of the movement pattern of the illumination 120 by the moving structure 130 among the candidate reflection points detected through each fitting.

The image processing unit 150 calculates the center value of the detected candidate reflections. That is, the image processing unit 150 calculates the center coordinate point (a, b) of [Equation 1] as the center value of the detected candidate reflection point. The image processing unit 150 uses the center point of the detected candidate reflection point as a representative value of the illumination reflection point.

4 is a flow chart illustrating a method of detecting an illumination reflex point of a gaze tracking device with moving illumination in accordance with an embodiment of the present invention. The present embodiment will be described taking as an example a case where the moving structure 130 rotates in a circular path and the moving speed is slow.

The image processing unit 150 acquires an image through the camera 110 and detects a pupil region in the acquired image (S101, S102).

The image processing unit 150 detects a candidate reflex point having a brightness value equal to or higher than the critical brightness value in the detected pupil region (S103). At this time, the image processor 150 detects a coordinate value of a pixel having a brightness value greater than or equal to a threshold brightness value.

The image processing unit 150 calculates a movement trajectory of the illumination 120 by the moving structure 130 using the coordinate values of the detected candidate refracting points (S104). The image processing unit 150 obtains a circle equation using the coordinate values of the detected candidate reflections. The equation of the circle means the movement trajectory of the illumination 120.

The image processing unit 150 calculates the moving distance average of the candidate reflections (S105).

The image processing unit 150 calculates an area of interest in which the illumination reflection points are likely to be located from the moving trajectory of the illumination 120 and the moving distance average of the candidate reflection points (S106).

The image processing unit 150 designates a region of interest in the image and detects an illumination reflection point located in the region of interest (S107). At this time, the image processing unit 150 may detect a reflection point using a warping method, an affine transform, and a position transform function in addition to the outline detection method.

5 is a flow chart illustrating a method of detecting an illumination reflex point of a gaze tracking device with moving illumination in accordance with another embodiment of the present invention. The present embodiment explains that the traveling speed of the moving structure 130 is fast and the reflection points are detected through template matching.

First, the image processing unit 150 generates an illumination reflection point template based on the dictionary information of the moving structure 130 (S111). The image processor 150 generates an illumination reflex point generated in the image as a template according to the movement of the illumination 120 by the moving structure 130. The advance information includes the moving speed and moving pattern of the moving structure 130, the turning radius, and the like.

Then, the image processing unit 150 acquires the eye image through the camera 110 (S112).

The image processing unit 150 detects reflection points in the eye image through template matching using the designated template (S113). In other words, the image processing unit 150 detects an illumination reflection point most similar to the template specified in the eye image.

6 is a flow chart illustrating a method of detecting an illumination reflex point of a gaze tracking device with moving illumination in accordance with another embodiment of the present invention. This embodiment describes a method of detecting a reflection point by using advance information of the moving body 130 when the moving structure 130 is moving at a high speed.

The image processing unit 150 of the gaze tracking apparatus acquires an image (e.g., eye image) through the camera 110 (S121).

The image processing unit 150 converts the image into a binary image (S122).

The image processing unit 150 detects an outline on the binary image (S123).

The image processing unit 150 detects a candidate reflection point out of the contour lines detected through the elliptic fitting and the circle fitting (S124).

The image processing unit 150 selects candidate reflection points similar to the width and height ratio and the radius size of the movement trajectory (movement pattern) of the illumination 120 by the moving structure 130 among the detected candidate reflection points (S125).

The image processing unit 150 determines the selected candidate reflection point as an illumination reflection point (S126). Here, the image processing unit 150 calculates the center value of the selected candidate reflection point, and determines the calculated center value as the illumination reflection point.

In the above-described embodiment, the illumination 120 is rotated in a circular track, but it may be implemented to move in a linear reciprocating motion. At this time, the image processing unit 150 can detect the illumination reflection point by inputting the straight line equation and the length of the straight line in advance.

The embodiments described above are those in which the elements and features of the present invention are combined in a predetermined form. Each component or feature shall be considered optional unless otherwise expressly stated. Each component or feature may be implemented in a form that is not combined with other components or features. It is also possible to construct embodiments of the present invention by combining some of the elements and / or features. The order of the operations described in the embodiments of the present invention may be changed. Some configurations or features of certain embodiments may be included in other embodiments, or may be replaced with corresponding configurations or features of other embodiments. It is clear that the claims that are not expressly cited in the claims may be combined to form an embodiment or be included in a new claim by an amendment after the application.

Embodiments in accordance with the present invention may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, an embodiment of the present invention may include one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) field programmable gate arrays, processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, a procedure, a function, or the like which performs the functions or operations described above. The software code can be stored in a memory unit and driven by the processor. The memory unit is located inside or outside the processor, and can exchange data with the processor by various known means.

It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit of the invention. Accordingly, the foregoing detailed description is to be considered in all respects illustrative and not restrictive. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

110: camera
120: Lighting
130: moving structure
140: Memory
150:

Claims (12)

At least one camera installed to face the user in the vehicle and acquiring the image of the user,
At least one light source for illuminating the user with light,
A moving structure structurally coupled to the illumination and configured to move the illumination to a predetermined trajectory,
And an image processing unit for predicting an illumination reflection point by the illumination based on the advance information of the moving structure and detecting the illumination reflection point in the image. The method according to claim 1,
The movement trajectory includes:
Wherein the predetermined trajectory is a predetermined trajectory such as a circular trajectory, an elliptical trajectory, a rectangular trajectory, and a straight reciprocating trajectory. The method according to claim 1,
Wherein the image processing unit comprises:
A moving trajectory of the illumination is calculated by detecting a pupil region in the image, a candidate reflex point is detected in the pupil region, an average of the moving distances of the candidate reflex points is calculated, and an average of the moving trajectory and the moving distance of the candidate reflex points Is used to detect a region of interest in the pupil region and to detect an illumination reflex point within the detected region of interest. The method according to claim 1,
Wherein the image processing unit comprises:
Wherein an illumination reflex point template is generated based on advance information of the moving structure and a reflex point is detected in the image through template matching using the illumination refocus point template. The method according to claim 1,
Wherein the image processing unit comprises:
Detecting an outline of an area having the same pixel value on the binary image, detecting a candidate reflex point out of the outlines detected through fitting, and calculating, based on the preliminary information of the moving structure, one of the candidate reflex points To determine an illumination reflex point. ≪ RTI ID = 0.0 > 8. < / RTI > 6. The method of claim 5,
Wherein the image processing unit comprises:
Wherein a center point of the illumination reflection point is used as a representative value. 6. The method of claim 5,
Wherein the fitting comprises a circular fitting and an elliptical fitting. Acquiring an image through a camera;
Detecting a pupil region in the image;
Predicting the generation of an illumination reflection point based on prior information of a moving structure that moves the illumination to a predetermined movement trajectory and detecting an illumination reflection point in the pupil area through illumination reflection point prediction. (EN) METHOD FOR DETECTION OF ILLUMINATION REFLECTIVE POSITION OF LINE - 9. The method of claim 8,
Wherein the illumination reflection point detection step comprises:
Detecting a candidate reflex point in the pupil region;
Calculating a movement trajectory of the illumination using the coordinate value of the candidate reflection point;
Calculating a moving distance average of the candidate reflections;
Designating a region of interest in the pupil region using a movement trajectory of the illumination and an average travel distance of the candidate refractory point;
And detecting an illumination reflex point within the region of interest. ≪ Desc / Clms Page number 20 > 9. The method of claim 8,
Wherein the illumination reflection point detection step comprises:
Wherein the illumination tracking unit detects the illumination reflection point most similar to the illumination reflection point template in the pupil area through template matching using the illumination reflection point template generated based on the advance information of the moving structure The method comprising the steps of: 9. The method of claim 8,
Wherein the illumination reflection point detection step comprises:
Converting the detected pupil region into a binary image,
Detecting an outline of an area having the same pixel value on the binary image;
Detecting a candidate reflex point among the outlines through fitting;
Selecting one of the candidate reflex points based on the advance information of the moving structure;
And determining the selected candidate reflections as the illumination reflections. ≪ RTI ID = 0.0 > 11. < / RTI > 9. The method of claim 8,
The advance information of the moving structure includes:
Wherein the movement trajectory, the moving speed, and the turning radius are included in the moving trajectory.

Images