KR101706992B1 - Apparatus and method for tracking gaze, recording medium for performing the method - Google Patents

Abstract

Disclosed is a gaze tracking apparatus including: an imaging unit acquiring a surrounding image in either a wide angle mode or a narrow angle mode, each of the wide angle mode and the narrow angle mode being an imaging mode; and a control unit detecting a face region, an eye region, and a nose region of a person shown in the surrounding image by comparing the surrounding image acquired by the imaging unit to a learning image with respect to the face region, the eye region, and the nose region stored in advance, detecting the center of a pupil region by detecting an iris region from the eye region and separating the pupil region from the iris region, detecting an ocular center depending on the angle of the persons facial pose by calculating the angle of the persons facial pose from position information related to the eye region and the nose region in the face region, and detecting the direction of the persons gaze by connecting the center of the pupil region to the ocular center.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gaze tracking apparatus and method,

The present invention relates to a gaze tracking apparatus and method, and a recording medium for performing the same, and more particularly, to a gaze tracking apparatus and method for tracking a gaze of a person from an image acquired through a PTZ camera, .

As the use of smart devices becomes more active, the development of technologies for controlling smart devices by non-contact methods such as gesture recognition, speech recognition, and gaze recognition has been continuously carried out. Particularly, in the technique of controlling the smart device in a non-contact manner through the visual recognition, a technique of acquiring a face image of a person and tracking a line of sight from a face image is a core technology.

As such eye tracking technology, generally, two camera modules such as a wide-angle camera and a narrow-angle camera are used, a human's position is detected with a wide-angle camera, and a high-resolution eye image is acquired with a narrow- .

However, in the above case, as the distance between the camera and the line-of-sight tracking object increases, a larger number of cameras are required to obtain a high-resolution image for tracking the line of sight. In such a case, .

Conventionally, the motion of the iris or the pupil center is calculated from the high-resolution eye image and the direction of the human eye is predicted according to the motion of the iris or the pupil center. However, There is a problem that the accuracy of the eye tracking result is somewhat deteriorated.

One aspect of the present invention relates to a gaze tracking apparatus and method, and a recording medium for performing the gaze tracking apparatus and method, and more particularly, to a gaze tracking apparatus and method for tracking a person's gaze by detecting an ocular center from a person's pose angle A recording medium is provided.

According to an aspect of the present invention, there is provided a gaze tracking apparatus, comprising: a photographing unit for obtaining a surrounding image in one photographing mode of a wide angle mode or a narrow angle mode; Eye region and a nose region appearing in the peripheral image, detecting an iris region from the eye region, separating the pupil region from the iris region, detecting the center of the pupil region, Calculating an angle of a face pose of the person from the position information of the eye region and the nose region in the face region to detect an eyeball center according to an angle of the face pose of the person, And a control unit for detecting the gaze direction of the person.

The control unit may further include an output unit that sets a plurality of calibration points by the control unit and outputs a point at a point corresponding to the direction of the human eye according to the control of the control unit, And outputting a point to a point corresponding to the line of sight of the user, and performing calibration of the line of sight of the person by comparing the point output to the output unit with the plurality of calibration points.

Also, the controller may generate the face region, the eye region, and the nose region classifier using MCT and Adaboost techniques from a previously stored learning image, and add the face region, eye region, And a face element detecting unit for detecting a face region, an eye region, and a nose region of a person appearing in the peripheral image by applying a region and a nose region classifier.

The control unit may control the photographing mode of the photographing unit to be a wide angle mode to obtain the peripheral image, and when the face region, the eye region, and the nose region of the person appearing in the surrounding image photographed in the wide angle mode are detected, Mode or the tilt control of the photographing unit so that the face region can be detected at the center of the image photographed in the photographing mode, Mode, which is a mode of operation of the camera.

The controller may detect an iris region by matching a variable template to an image of the eye region, apply a K-means algorithm to separate the pupil region from the iris region, And a pupil center detecting unit for detecting the center of the region.

The controller may further include a face pose detecting unit for calculating an angle of a face pose of the person in a pitch, roll, and yaw using the position information of the eye region and the nose region, Further comprising an eye center detecting unit for detecting the eye center by applying a Robert edge detector to the image of the region, performing labeling to detect an eye corner point, and using the angle of the eye corner point and the face pose of the person .

According to another aspect of the present invention, there is provided a gaze tracking method comprising: acquiring a peripheral image and comparing it with a learning image of a previously stored face area, an eye area, and a nose area, Detecting an iris region from the eye region, separating the pupil region from the iris region and detecting the center thereof, detecting the center of the eye region and the nose region from the position information of the eye region and the nose region in the face region, The center of the eyeball is detected according to the angle of the face pose of the person by calculating the angle of the pose, and the sight line direction of the person is detected by connecting the center of the pupil region and the center of the eyeball.

The method may further include setting a plurality of calibration points and performing calibration with respect to the gaze direction of the person by comparing the plurality of calibration points with points corresponding to the gaze direction of the person.

In order to detect a face region, an eye region, and a nose region of a person appearing in the peripheral image, a face region, an eye region, and a nose region classifier are generated using MCT and Adaboost technique from a previously stored learning image Eye region, a nose region, and a nose region appearing in the peripheral image by applying the face region, the eye region, and the nose region classifier to the peripheral image.

The detection of the iris region from the eye region, the separation of the pupil region in the iris region and the detection of the center thereof may be performed by detecting an iris region by matching a variable template to the image of the eye region, To separate the pupil region from the iris region, and to detect the center of gravity of the pupil region as the center of the pupil.

It is preferable that the angle of the face pose of the person is calculated from the position information of the eye region and the nose region in the face region and the eye center corresponding to the angle of the face pose of the person is detected, The angle of the face pose of the person in the pitch, the roll and the yaw is calculated by using the position information of the area, the Robert edge detector is applied to the image of the eye area, and the labeling is performed Detecting the eye corner point, and detecting the eye center using the angle of the eye corner point and the face pose of the person.

In addition, it may be a computer-readable recording medium on which a computer program is recorded, for performing the gaze tracking method.

According to one aspect of the present invention, the eye center is detected from the face pose angle of a person, and the eye line is traced by reflecting the center of the eye, thereby improving the accuracy of eye tracking even when the distance between the camera and the eye tracking object is distant .

1 is a control block diagram of a gaze tracking apparatus according to an embodiment of the present invention.
2 is a detailed block diagram of the control unit shown in FIG.
FIG. 3 is a view for explaining a face element detecting method according to an embodiment of the present invention.
4 is a view for explaining a PTZ (Pan-Tilt-Zoom) control method of a photographing unit according to an embodiment of the present invention.
5 is a diagram showing an example of a pinhole camera model.
6 and 7 are views for explaining a PTZ (Pan-Tilt-Zoom) control method of the photographing unit according to an embodiment of the present invention.
8 is a diagram for explaining an iris region detection method according to an embodiment of the present invention.
9 is a view for explaining a pupil center detection method according to an embodiment of the present invention.
10 and 11 are views for explaining a method of detecting a face pose angle according to an embodiment of the present invention.
12 is a view for explaining an eye corner point detecting method according to an embodiment of the present invention.
13 is a diagram for explaining an eye center detection method according to an embodiment of the present invention.
FIG. 14 is a diagram illustrating an eye model for calculating a gaze direction according to an embodiment of the present invention.
15 is a diagram illustrating an example of an output unit according to an embodiment of the present invention.
16 is a flowchart for explaining a line-of-sight tracking method according to an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a control block diagram of a gaze tracking apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a gaze tracking apparatus 1 according to an embodiment of the present invention may include a photographing unit 100, a controller 200, an output unit 300, and a memory unit 400. The gaze tracking device 1 according to an embodiment of the present invention may be implemented by, or implemented by, more components than the components shown in FIG. Hereinafter, the above-mentioned constituent elements will be described in detail.

The photographing unit 100 may acquire a peripheral image, and preferably acquires a peripheral image for detecting a face region of a person staring at specific contents. For this, the photographing unit 100 may be implemented as a single PTZ (Pan-Tilt-Zoom) camera operating in one of the wide angle mode and the narrow angle mode.

Specifically, the photographing unit 100 may perform a zoom operation according to a focal length of a lens set by the controller 200 to obtain an enlarged or reduced peripheral image. In other words, the photographing unit 100 may operate according to one of the wide-angle mode or the narrow-angle mode according to the control of the control unit 200 to be described later to acquire a peripheral image. Here, the wide-angle mode means a photographing mode in which the lens focal distance of the camera is set to a predetermined reference value or less to acquire a peripheral image of a relatively wide field of view, and the narrow-angle mode means that the lens focal distance of the camera is equal to or greater than a predetermined reference value The lens focal length in the wide angle mode is set to 24 to 50 mm and the lens focal length in the narrow angle mode is set to 80 to 120 mm .

In addition, the photographing unit 100 may perform a pan operation for moving left and right or a tilt operation for moving up and down under the control of the controller 200, and may acquire a peripheral image.

As described above, the photographing unit 100 first operates in a wide-angle mode to detect a person in the vicinity, acquires a peripheral image of a wide field of view, and when a face region of a person is detected in an image acquired by the control unit 200 Pan, or tilt operation, the user can switch to the narrow-angle mode to acquire an image of a high-resolution enlarged face region. A detailed description thereof will be given later.

The control unit 200 can control the photographing unit 100 to acquire an image of a face region and track a line of sight using the image of the face region acquired from the photographing unit 100. [ This will be described with reference to FIG. 2 to FIG.

FIG. 2 is a detailed block diagram of the control unit shown in FIG. 1, and FIGS. 3 to 14 are views for explaining a method of tracking a line of sight from an image of a face region in the control unit shown in FIG.

2, the controller 200 includes a face element detector 210, a photographing controller 220, a pupil center detector 230, a face pose detector 240, an eyeball center detector 250, (260). The control unit 200 may be implemented by, or implemented by, more components than the components shown in FIG. Hereinafter, the above-mentioned constituent elements will be described in detail.

The face element detecting unit 210 can detect face components of a face region of a person, an eye region, and a nose region when a person appears in a surrounding image acquired by the photographing unit 100. [ At this time, the photographing unit 100 may be set in the wide angle mode by the photographing control unit 220, which will be described later, so that the photographing unit 100 can acquire a peripheral image of a wide field of view.

3, the facial feature detecting unit 210 may detect a face region, an eye region, and a nose region of a person from a surrounding image acquired by the photographing unit 100. [ To this end, the facial feature detection unit 210 may perform a MCT (Modified Census Transform) conversion preprocessing process on the training image, and then generate a face region, an eye region, and a nose region detection classifier using an Adaboost technique. Here, performing the MCT transformation preprocessing process on the learning image is for robustness against illumination change, and the Adaboost technique is a type of template matching, and the feature values can be classified. In addition, the learning image may be face, eye, and nose images of various postures or shapes stored in advance in the memory unit 400. For example, a plurality of faces and nose images that can be classified into left, front, And may include a plurality of eye images which can be classified into open eyes and eye eyes.

As described above, the facial feature detection unit 210 generates a face region, an eye region, and a nose region detection classifier through an MCT and an Adaboost technique. The face region, the eye region, By applying the nose region detection classifier, the face region, the eye region, and the nose region of a person included in the image can be detected.

Meanwhile, the eye-gaze tracking apparatus 1 according to an embodiment of the present invention detects the center of the pupil and the center of the eye from the face image of a person, and uses the detected eye center to track a person's gaze. At this time, Since the eye region and the nose region included in the detected face region and the face region are detected from the peripheral image acquired in the state in which the photographing section 100 is in the wide angle mode, the resolution is low and it is difficult to accurately detect the pupil center and the eyeball center Therefore, a high-resolution enlarged image may be required for the detected face region.

The photographing control unit 220 controls the pan or tilt operation of the photographing unit 100 so as to acquire a higher resolution image of the face region detected by the face element detecting unit 210 , The photographing mode of the photographing unit 100 can be switched.

For example, referring to FIG. 4, the photographing control unit 220 determines whether the face region detected by the face element detecting unit 210 is located at the center of the image captured by the photographing unit 100 as shown in FIG. 4 (b) The pan or tilt operation of the photographing unit 100 is controlled so that the photographing mode of the photographing unit 100 is switched to the narrowed mode as shown in FIG. Images can be acquired. At this time, the photographing control unit 220 calculates the distance between the face region detected by the photographing unit 100 and the face element detecting unit 210, and calculates the distance between the pan and tilt of the photographing unit 100 according to the calculated distance Tilt angle, or a zoom factor of the image pickup unit 100 to control the photographing unit 100.

Specifically, the photographing control unit 220 can calculate the distance between the face region detected by the photographing unit 100 and the face element detecting unit 210 using a pinhole camera model. Referring to FIG. 5, the pinhole camera model is used to calculate the distance between a camera and an object using the property of a triangle. The pinhole camera model includes a focal length, a height of an object on a sensor, ) And the height of the actual object (Object) can be expressed by the following Equation (1).

In Equation 1, SObj _h is the height of the object on the sensor, RObj _h is the height of the actual object, f is the focal length, and z is the distance between the camera and the object.

Here, the height SObj _h of the object on the sensor can be obtained by using the following equation (2).

In Equation 2, SObj _h denotes the height of the object on the sensor, S _h denotes the height of the sensor, IObj _h denotes the height of the object displayed on the image, and I _h denotes the height of the image.

Finally, Equation (3) for calculating the distance between the camera and the object can be derived by applying Equation (2) to Equation (1).

In Equation 3 z is the distance between the camera and the object, and f is the focal length, I _h is the height, RObj _h of an image refers to the height of the actual object, and, S _h is the height of the sensor, IObj _h is image The height of the object shown in FIG.

Therefore, the shooting control unit 220 can calculate the distance between the face region detected by the shooting unit 100 and the face element detecting unit 210 using Equation (3), and calculates the distance between the shooting unit 100 A pan angle, a tilt angle, or a zoom factor of the image pickup unit 100 to control the photographing unit 100. At this time, the view angle information according to the distance calculated by the photographing control unit 220, that is, the view angle information according to the zoom position of the photographing unit 100, may be stored in the memory unit 400 in advance.

6, the photographing control unit 220 may calculate the pan angle of the photographing unit 100 using Equation (4) below and calculate the angle of the photographing unit 100 using Equation (5) 100) can be calculated.

In Equation (4), p denotes the pan angle of the photographing unit 100,? _W denotes the horizontal angle of view, W denotes the number of horizontal pixels of the image, u denotes the number of pixels Position.

In Equation (5), t denotes a tilt angle of the photographing unit 100,? _H denotes a vertical angle of view, H denotes the number of vertical pixels of the image, Position.

As described above, the photographing control unit 220 calculates the pan and tilt angles of the photographing unit 100 using equations (4) and (5), and calculates the pan and tilt angles of the photographing unit 100 The center (u, v) of the face region can be located at the center (I _cx , I _cy ) of the image acquired by the photographing section 100 by controlling the tilt operation.

Further, the photographing control unit 220 can calculate a zoom multiple of the photographing unit 100 using the following equation (6).

In Equation (6), z denotes a zoom multiple of the photographing unit 100, H denotes the number of vertical pixels of the image, Obj _h denotes the size of the face region, and w denotes a weight .

In this manner, the photographing control unit 220 calculates a zoom multiple of the photographing unit 100 using Equation (6), and thereby controls the zooming operation of the photographing unit 100, that is, The image of the enlarged face area of high resolution can be obtained by controlling the shooting section 100 of the mode in the narrow-angle mode. At this time, the photographing control unit 220 controls the photographing unit 100 to pan or tilt so that the face region can be positioned at the center of the image captured by the photographing unit 100, The image of the enlarged face region of high resolution can be obtained. In this regard, referring to FIG. 7, a person may be included in an image acquired in a state in which the photographing mode of the photographing unit 100 is in the wide angle mode, as shown in FIG. 7 (a) Since the photographing unit 100 according to the embodiment is implemented by a single PTZ camera operating in one of the wide angle mode and the narrow angle mode, When the photographing mode is switched to the narrow-angle mode without controlling the pan or tilt operation in a state in which the photographing mode is the wide-angle mode, enlargement of the surrounding background rather than enlarged image of the face region of the person included in the image And acquires the acquired image. 7 (b), the photographing control unit 220 may pan or tilt the photographing unit 100 so that the face region may be positioned at the center of the image captured by the photographing unit 100. [ The image of the enlarged face area of high resolution can be obtained by switching the photographing mode of the photographing section 100 to the narrow-angle mode.

The pupil center detecting unit 230 can detect the center of the pupil from the image of the enlarged face region at the high resolution. For this, the pupil center detecting unit 230 may detect the pupil center in the iris region after first detecting the iris from the image of the enlarged face region of high resolution.

Specifically, referring to FIG. 8, the pupil center detecting unit 230 generates a Valley Map for an eye region included in an image of a high-resolution enlarged face region, and then performs binarization using an arbitrary threshold value . Then, the pupil center detecting unit 230 performs template matching on the binarized image to detect the iris region as shown in FIG. 8 (e). At this time, the pupil center detecting unit 230 can detect the iris region by applying a variable template to the binarized image.

Here, the variable template is a method proposed by J. Y. Deng and F. Lai, wherein a small window is arranged at the edge of a circular template to calculate energy, move it in a direction to minimize energy, and adjust its size. At this time, the energy function can be expressed by Equation (7) below.

In Equation (7), A _circle denotes the number of pixels corresponding to the template region.

When the energy for the window region is calculated using Equation (7), the size of the window region can be changed using Equation (8) below, and the window region can be moved using Equation (9).

In Equation (8), f _resize denotes a size change coefficient of the window region, and f _w denotes a window region.

In Equation 9 fmove _{x, y} fmove refers to the transfer coefficient of the window area, and f _w is meant the window area.

On the other hand, the window area can be normalized using the following equation (10), and the value can be normalized to (-255 / 2, 255/2).

9, the pupil center detecting unit 230 can detect the pupil center in the detected iris region. The pupil center detecting unit 230 can detect the pupil center by generating a Valley Map for the iris region and performing clustering. At this time, the pupil center detecting unit 230 can separate the iris region and the pupil region by applying a K-means clustering algorithm to the valley map generated for the iris region as two clusters. The pupil center detecting unit 230 can detect the center of gravity of the separated pupil region as the pupil center.

On the other hand, the face pose detection unit 240 can detect a face pose from the position information of the eye region and the nose region in the enlarged face region of high resolution. Referring to FIG. 10, the face pose detection unit 240 detects a pitch, which is an angle at which the face rotates about the y axis, a roll that is an angle at which the face rotates about the z axis, It is possible to calculate the angle of the face pose defined by the yaw that is the rotation angle.

Specifically, referring to FIG. 11A, the face pose detector 240 may first calculate a pitch using Equation (11) below. The pitch can be calculated using the distance (D) between the two eye regions and the ratio of the nose length (B). At this time, the ratio (B) of the nose length can be calculated by multiplying the distance between the two eye regions by 0.618.

In Equation (11), D represents the distance between two eye regions, and dy _p represents a derivative value of the nose tip at the nose length.

11 (b), the face pose detecting unit 240 can calculate a roll using the following equation (12). The roll can be calculated using the position information of the two eye regions.

In Equation (12), dy _r means a derivative value with respect to the y axis of two eye region positions, and dx _r means a derivative value with respect to the x axis of two eye region positions.

11 (c), the face pose detector 240 can calculate Yaw using the following equation (13). The yaw can be calculated using the center point between the two eye regions and the position information of the nose end. Here, the position information of the nose tip can be detected from the learning information about the position information of the nose tip previously stored.

It means a differential value of the position of the nose end relative to the center point between eyes region is _r dx in equation (13), and dy _y means a differential value of the vertical distance between the position of the eyes and nose area.

On the other hand, the eyeball center detecting unit 250 can detect the center of the eye from the image of the enlarged face region of high resolution. To this end, the eyeball center detecting unit 250 first detects an eye corner point in the eye region included in the image of the enlarged face region of high resolution, and then calculates the angle of the face pose calculated by the eye corner point and the face pose detecting unit 240 The center of the eye can finally be detected.

Specifically, referring to FIG. 12, the eyeball center detecting unit 250 may detect an eyelid by applying a Roberts edge detector to an eye region included in an image of a high-resolution enlarged face region. Here, since the Roberts edge detector has a high calculation speed and is sensitive to noise, the eye center detector 250 applies a Gaussian kernel to the image of the eye region, removes noise, and then applies a Roberts edge detector, Similarly, the edge of the eyelid can be detected. In addition, the eyeball center detecting unit 250 may apply the labeling to the image in which the edge of the eyelid is detected and remove the small-sized block to obtain an image as shown in FIG. 12 (d). Among the labeled blocks, The end points of the block having the longest horizontal length estimated as the eye corner point can be detected as shown in (e) of FIG.

Then, the eye center detecting unit 250 can detect the center of the eye according to the detected eye corner point and the face pose angle.

Specifically, referring to Fig. 13, E1 and E2 denote eye corner points, M denotes a center point of E1 and E2, and P denotes a pupil center. At this time, the adult eyeball size is generally spherical with a diameter of about 24 mm. Accordingly, the eyeball center detecting unit 250 can detect the three-dimensional coordinates indicating the center of the eye using the angle of the face pose as shown in Equation (14) below.

In Equation 14, x ', y', and z 'denote the three-dimensional coordinates of the eye center, t _x , t _y , and t _{z denote} the center of the eye (M) X, y and z denote roll, yaw, and pitch angles, respectively, of the center of gravity M of the eye corners, and α, β and γ denote the roll, yaw and pitch angles, respectively.

On the other hand, the gaze direction detecting unit 260 can detect the gaze direction from the center of the pupil detected by the pupil center detecting unit 230 and the center of the eye detected by the eye center detecting unit 250.

Specifically, the gaze direction detecting unit 260 detects the gaze direction from the geometric model of the eye, to which the center of the pupil detected by the pupil center detecting unit 230 and the center of the eye detected by the eye center detecting unit 250 are applied, Can be detected. That is, the gaze direction detecting unit 260 can detect a line passing through the center of eye (O) and the pupil center (p) as a gaze direction.

The gaze direction detecting unit 260 sets a plurality of calibration points in an output unit 300 to be described later, outputs a point at a position corresponding to the gaze direction vector, and compares it with a plurality of calibration points, Direction can be calibrated.

Specifically, a person appearing in the peripheral image acquired by the photographing unit 100 may be in a state of gazing at a specific content. Here, the specific content may be content output from the output unit 300, or may be content output through an output module separate from the output unit 300.

For example, referring to FIG. 15, the gaze direction detecting unit 260 may set a plurality of calibration points in the output unit 300 at predetermined intervals. In FIG. 15 (a), 15 calibration points are set as an example. The gaze direction detecting unit 260 may calculate the correction coefficient using the following expression (15) or (16) so as to output the point corresponding to the gaze direction vector to the output unit 300 as shown in FIG. 15 (b) . 15B, a line of sight of a person appearing in a peripheral image acquired by the photographing unit 100 is divided into four corner portions (calibration point: 1 (1)) of the contents output through the output unit 300 or a separate output module, , 5, 11, 15) and the central part (calibration point: 8) for a certain period of time.

In equation 15 and 16, X _screen and Y _screen refers to the x, y coordinates on the output unit 300, and, N means a calibration point number, X and Y are the gaze direction vector, α _i and β _i is Quot;

Referring again to FIG. 1, a plurality of calibration points according to a predetermined interval are set by an output unit 300 under the control of the controller 200, and a point corresponding to a human's gaze direction vector can be output. At this time, the output unit 300 does not output the specific content as described above, or outputs the specific content through the separate output module without outputting the specific content from the output unit 300, Can output a point corresponding to a gaze direction vector of a person staring at a separate output module. The output unit 300 may be implemented as various types of display means such as a liquid crystal display (LCD), an organic light emitting display panel (OLED), an electrophoretic display panel (EDD), or a touch screen.

In addition, the memory unit 400 may store a program for processing and controlling the gaze tracking device 1, and may perform a function for temporarily storing input / output data. In particular, as described above, the memory unit 400 may store a learning image for generating the face region, the eye region, and the nose region detection classifier, and the face region, the eye region, and the nose region The detected image of the face region, the eye region, and the nose region can be updated with the learning image and stored.

In addition, the memory unit 400 may store angle-of-view information according to the zoom position of the photographing unit 100.

Hereinafter, a gaze tracking method according to an embodiment of the present invention will be described with reference to FIG.

16 is a flowchart for explaining a line-of-sight tracking method according to an embodiment of the present invention.

The gaze tracking method according to an embodiment of the present invention can be performed in substantially the same configuration as the gaze tracking apparatus 1 shown in Fig. Therefore, the same components as those of the gaze tracking device 1 of FIG. 1 are denoted by the same reference numerals, and repeated descriptions are omitted.

Referring to FIG. 16, a surrounding image can be acquired from the photographing unit 100 in the wide angle mode (500). The photographing unit 100 may be set in a wide angle mode under the control of the controller 200 to acquire a peripheral image of a wide field of view.

In addition, a face region, an eye region, and a nose region of a person can be detected from the acquired image (510). The control unit 200 may apply the face region, the eye region, and the nose region classifier to the image acquired by the photographing unit 100. Accordingly, when a person appears in the image acquired by the photographing unit 100, Can detect the face region, the eye region, and the nose region.

In addition, the photographing unit 100 may be controlled to pan or tilt (520) so that the detected face region may be located at the center of the image acquired by the photographing unit 100. The control unit 200 calculates the distance between the photographing unit 100 and the face region of the detected person and calculates a pan or tilt angle of the photographing unit 100 using the calculated distance The camera 100 can be controlled to pan or tilt so that the detected face area can be located at the center of the image acquired by the photographing unit 100. [

In addition, the image of the enlarged face region of high resolution can be acquired by switching the photographing mode of the photographing unit 100 to the narrow-angle mode (530). The control unit 200 calculates a zoom multiple of the photographing unit 100 using the distance between the photographing unit 100 and the detected face region of the person and displays the zoom ratio of the photographing unit 100 according to the zooming factor, The image of the enlarged face region of higher resolution can be obtained.

Thereafter, the iris region is detected from the image of the enlarged face region of high resolution, and the center of the pupil is detected in the iris region (540). The controller 200 can detect the iris region by matching the variable template to the eye region included in the image of the enlarged face region of high resolution. Then, the controller 200 can detect the center of gravity of the pupil region as a pupil center by separating the iris region and the pupil region by applying a K-means algorithm within the detected iris region.

Further, a face pose can be detected from an image of a high-resolution enlarged face region (550). The controller 200 can calculate angles of face pose of pitch, roll, and yaw from the position information of the eye region and the nose region in the image of the enlarged face region of high resolution.

In addition, an eye corner point can be detected from an image of a high-resolution enlarged face region, and an eye center according to an angle of an eye corner point and a face pose can be detected (560). The control unit 200 can detect the eye corner point by applying the Robert edge detector to the eye region included in the image of the enlarged face region of high resolution and performing labeling, and calculate the angle of the eye corner point and the face pose by using the above- Equation 15 can be applied to detect the center of the eye.

In addition, the eye direction can be detected by connecting the pupil center and the eyeball center (570). The controller 200 can generate a geometric model of the eye to which the center of the pupil and the center of the eye are applied, and detect a line passing through the center of the eye and the center of the pupil in the eye direction.

Finally, a calibration for the viewing direction may be performed (580). The control unit 200 may set a plurality of calibration points on the output unit 300 and perform calibration on the gaze direction of the person by comparing a plurality of calibration points with a point corresponding to a direction of a human's gaze have.

Such a gaze tracking method may be implemented in an application or may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the computer-readable recording medium may be ones that are specially designed and configured for the present invention and are known and available to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. It will be possible.

1: eye tracking device
100:
200:
300:
400:

Claims (12)

A photographing unit for acquiring a surrounding image in one of a wide angle mode and a narrow angle mode; And
Eye region and a nose region are compared with a learning image of a face region, a snow region, and a nose region stored in advance in the peripheral region, and a face region, an eye region, and a nose region appearing in the peripheral image are detected, Detecting the center of the pupil region in the iris region, calculating the angle of the face pose of the person from the position information of the eye region and the nose region in the face region, And a control unit for detecting a gaze direction of the person by connecting the center of the pupil region and the center of the eyeball,
Further comprising an output unit for setting a plurality of calibration points by the control unit and outputting a point at a point corresponding to the gaze direction of the person under the control of the control unit,
Wherein,
A step of outputting a point at a point corresponding to a direction of the human eye in the output unit is repeatedly performed and a calibration for the gaze direction of the person is performed by a method of comparing points output from the output unit and the plurality of calibration points Of the subject. delete The method according to claim 1,
Wherein,
Eye region and a nose region classifier using MCT and Adaboost techniques from a pre-stored learning image, and generates the face region, eye region, and nose region classifier on a peripheral image acquired by the photographing unit Eye region and a nose region of a person appearing in the peripheral image by applying the face region detecting unit to the face region detecting unit. The method according to claim 1,
Wherein,
When a face region, an eye region, and a nose region of a person appearing in a peripheral image photographed in the wide angle mode are detected by controlling the photographing mode of the photographing unit to be a wide angle mode, A photographing control unit for controlling the photographing unit to pan or tilt so that the face region can be detected from the center and for switching the photographing mode of the photographing unit to a narrow- Wherein the gaze tracking device comprises: The method according to claim 1,
Wherein,
Detecting an iris region by matching a variable template to an image of the eye region, separating the pupil region from the iris region by applying a K-means algorithm, and detecting a center of gravity of the pupil region as a center of the pupil region And a pupil center detecting unit. 6. The method of claim 5,
Wherein,
A facial pose detecting unit for calculating an angle of the facial pose of the person in a pitch, a roll, and a yaw using the position information of the eye region and the nose region; And
An eye center detecting unit for detecting the eye center by detecting the eye corner point by applying a Robert edge detector to the image of the eye region, performing labeling, and using the angle of the eye corner point and the face pose of the person Includes an eye tracking device. Eye area and a nose area of a person appearing in the surrounding image by comparing with a learning image of a previously stored face area,
Detecting an iris region from the eye region, separating the pupil region from the iris region,
Calculating an angle of the face pose of the person from the position information of the eye region and the nose region in the face region to detect an eyeball center according to an angle of the face pose of the person,
Detecting the gaze direction of the person by connecting the center of the pupil region and the center of the eyeball,
A plurality of calibration points are set,
And performing a calibration for the gaze direction of the person by comparing the plurality of calibration points with a point corresponding to the gaze direction of the person. delete 8. The method of claim 7,
The detection of the face region, the eye region and the nose region of the person appearing in the peripheral image,
Eye region and a nose region classifier using MCT and Adaboost technique from pre-stored learning images, applying the face region, eye region, and nose region classifier to the peripheral image, Eye region and a nose region of a person appearing. 8. The method of claim 7,
Detecting the iris region from the eye region, separating the pupil region from the iris region, and detecting the center thereof,
Detecting an iris region by matching a variable template to an image of the eye region, separating the pupil region from the iris region by applying a K-means algorithm, and detecting the center of gravity of the pupil region as the pupil center Eye tracking method. 11. The method of claim 10,
Calculating an angle of the face pose of the person from the position information of the eye region and the nose region in the face region and detecting an eyeball center according to an angle of the face pose of the person,
Calculating an angle of a person's face pose of pitch, roll, and yaw using the position information of the eye region and the nose region,
Wherein a gaze point is detected by applying a Robert edge detector to the image of the eye region, performing labeling to detect an eye corner point, and detecting the center of the eye using an angle of the eye corner point and a face pose of the person . 12. A computer-readable recording medium on which a computer program is recorded, for performing the method for tracking a line of sight according to any one of claims 7, 9, 11,

Images