KR20000056563A - gaze detection system - Google Patents

Abstract

PURPOSE: A detection system for gaze position is provided to enhance the accuracy of a gaze track by recognizing the face of user to the characteristic surface by the plurality of characteristic point regardless of face movement. CONSTITUTION: A detection system for gaze position is composed of a camera(CAM), an image board, a controller and a computer(COMP). The camera(CAM) obtains a face image of user(U) in predetermined time interval. The image board converts the obtained face image to the digital signal. The controller extracts the plurality of characteristic point and the characteristic surface by the digital signal, forms the characteristic surface by the plurality of characteristic point and detects the gaze position(P) of user(U) by calculating the space movement of characteristic surface according to time. The characteristic surface is composed of two eyes(f1,f2) and the center of lip(f3).

Description

Gaze detection system

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gaze detection system, and more particularly to a gaze position tracking system by recognition of the movement of a face.

According to the computerized GUI, image interfaces such as touch pads, light pens, and touch screens are widely used. In particular, the mouse constitutes a basic accessory of all computers.

However, such a mouse requires the use of one hand of a user in a limited space, and is very inconvenient when using a keyboard with two hands such as word processing, and requires physical force for the click. If you use the mouse continuously for a long time, such as web surfing), pain in the fingers, wrists, elbows, shoulders, etc., and in severe cases neuralgia symptoms such as mouse elbows similar to tennis elbows.

Accordingly, gaze tracking or gaze tracking systems have been studied in which a cursor is moved to a gaze position of a user without using a physical force such as a mouse and manipulated by blinking eyes.

What is currently being developed and used as such a gaze tracking system is a system of tracking the gaze using the movement of the pupil as shown in FIG.

This method installs an infrared detector K 'including an infrared light source and a camera on a monitor M of a computer C0MP, projects infrared rays into the pupil of the user U, and then the difference in the diffracted state of the reflected light. By tracking the eye L of the user U, the gaze position P on the monitor M, which is the observation surface F, is detected.

The conventional infrared-type eye tracking system has failed to be universally distributed except for some special purposes, for the following reasons.

First of all, this system needs to know the position of the eyes in the entire face of the user (U), accurately project infrared rays to the eyes, and then detect the reflections and diffracted light. .

The next problem is that the basic principle of this system is based on the condition that when the user U moves the gaze L, that is, the gaze position P on the monitor M, his face is not moved from the previous state but only the pupil. do. In reality, the user does not work with the face fixed for a long time, but moves the face continuously for work or conversation, so the system misses the user's gaze (L) or detects the gaze position (P). Impossible cases are frequent and error rates are high.

In addition, infrared rays are harmful to the pupils of the user U because the infrared rays are hot wires. Therefore, the recognition rate is further lowered because the intensity of the detection light cannot be increased. The risk of eye fatigue and eye diseases such as glaucoma and glaucoma in the long run is many.

In addition, this eye tracking system can track the gaze position (P) precisely when the eyes (L) direction of both eyes match, but there is a problem that the optical axis of both eyes of the general user (U) does not exactly coincide even if not eyesight. .

Even if a conventional pupil tracking system can solve all of the above problems, there remains a spatial topologically crucial problem.

In other words, the pupil tracking system detects the light reflected from the pupil in front of the eyeball and the light reflected from the retina of the back of the eyeball, and calculates the pupil's line of sight L as a fine light difference therebetween. The error is bound to be quite large.

Moreover, since the gaze L calculated in this way is extended to the monitor M to calculate the gaze position P, the error is inevitably amplified. For example, if the depth of the eyeball of the user U is 3cm and the distance between the monitor M and the space is 45cm, the gaze effect is calculated by extending the space vector 3cm in length by 45cm. This error can be amplified by (3 + 45) / 3 times, that is, 16 times, further lowering the reliability of the system.

In view of such a conventional problem, an object of the present invention is to enable accurate tracking of the gaze position irrespective of the movement of the face without using a harmful and expensive infrared detector, as well as a highly efficient gaze position with a significantly lower inherent error in configuration. It provides a tracking system at a cost low enough for general use.

1 is a schematic perspective view showing a conventional eye tracking system;

2 is a block diagram showing the mechanical configuration of the system of the present invention;

3 (A) to (E) is a sequential view showing a feature point extraction process of the system of the present invention,

4 (A) to (C) are sequential drawings showing the relationship between the feature plane and the line of sight by the principle of the present invention, that is, the feature point,

5 (A)-(C) are diagrams illustrating translation and rotation of a feature surface,

6 is a schematic perspective view of a staring position tracking system according to the present invention;

7 (A) and (B) are block diagrams showing the principle of operation of the system of FIG. 6;

8 is a schematic view showing a system with a binocular camera,

9 (A) to (E) are sequential diagrams illustrating a process of recognizing an open / closed state of an eye for manipulation by blinking eyes.

Explanation of symbols on the main parts of the drawings

U: User F: Observation Surface

P: Staring position 1: On the observation surface

f1 to f7: feature points f1, f2: feature points representing both eyes

f3: feature point representing the center of the mouth

f4, f5: feature points representing both ends of the mouth

f6, f7: feature points representing the nose (both nostrils)

Stare position tracking system according to the present invention for achieving the above object is

Face image acquisition means for acquiring a face image of the user as a digital image signal;

And a control unit for extracting a plurality of feature points of the face from the acquired digital video signal to form a feature surface from the plurality of feature points, and calculating the gaze position on the observation plane by calculating the temporal movement of the feature surface. It is done.

According to this configuration, the gaze position can be tracked only by the image of the face without troublesome infrared detection, and since the gaze position is tracked by the relative position of the face, the gaze position tracking system having a small intrinsic error can be inexpensively configured. .

As a result, the present invention can be widely used as a GUI system that replaces or supplements the conventional mouse, and can be used for special purposes such as process control by replacing or supplementing the conventional infrared tracking system.

Example

Such specific features and advantages of the present invention will become more apparent from the following description of the preferred embodiments with reference to the accompanying drawings.

2, the system of the present invention extracts a feature point and a feature plane from a camera CAM for acquiring a face image of the user U, an image board IMG for converting the acquired image into a digital signal, and this digital signal. The control unit CTR detects the gaze position P of the user U, and the controller A uses the gaze position P information detected by the control unit CTR.

In this embodiment, the face image acquisition means is composed of a camera (CAM) and an image board (IMG) on the assumption of an analog camera, but in the case of a digital camera, the output signal is a digital image signal, so a separate image board (IMG) is used. Becomes unnecessary.

On the other hand, the control unit (A) is shown as a computer (COMP) and its monitor (M), but if necessary, it can be configured as a control device such as manufacturing facilities or medical facilities, all the GUI interface, such as a mouse can be used It includes a device. In the case of the computer COMP, the observation plane F at which the gaze position P is to be traced is the front of the monitor M. FIG.

In the above description, the camera CAM is sufficient for an image acquisition camera used in a general camcorder or computer COMP, and the image board IMG is also common.

Since a video capture board is sufficient and the recent multimedia computers are shipped with a digital camera attached, the following description will mainly focus on the description of the control unit (CTR).

3, a process of extracting feature points from a face image acquired first is described.

In FIG. 3A, a user's face image is continuously acquired at a predetermined time interval (t1, t2, t3 ...) through a camera CAM and an image board IMG, and the acquired image includes a background. As shown in FIG. 3 (B), only the skin color region is extracted to remove the background and only the face image is extracted. This process filters only the I signal corresponding to the skin color in the YIQ signal, which is a general color signal method, and thus the detailed description will be unnecessary.

If there is a flesh object in the background, since it is a face image, it may be mistaken. Preferably, the difference image of the previous time interval t1 and the current time interval t2 is extracted (FIG. 3 (C)). By combining AND, the face region is extracted as shown in FIG.

Next, when the histogram of the signal corresponding to the face region is analyzed in FIG. 3E through smoothing and binarization, and filtered based on a predetermined level, the feature points of the face are extracted as shown in FIG. 3F. The above description also differs in algorithm, but since it is the basis of digital recognition, a detailed description is unnecessary. In particular, the sequential extraction of eye, nose, and mouth suitable for the present invention may refer to the corresponding part of the present patent application 98-35438. .

The feature points that can be extracted are those where the skin region and the color or brightness of the face are large, both eyes, in particular, the pupils (f1, f2), the mouth end points (f4, f5), and the two nostrils (f6). f7) (see FIG. 4 (A)).

In the subsequent continuous extraction process, as shown in FIG. 3 (G), the search area W is set around the first extracted feature points f1 to f7, and the search and extraction are started in the search area W. FIG. More accurate and faster feature points f1 to f7 can be extracted.

By this process, the face image of the user U as shown in FIG. 4A may be represented as a set of feature points f1 to f7 as shown in FIG. 4B.

At this time, when both eyes f1 and f2 are connected between the end points f4 and f5 of the mouth, a trapezoidal polygon is formed. As shown in FIG. 4C, a plane connecting the feature points f1 to f7 is shown. A feature surface 1 for numerically displaying the face area of the user U is formed, and

The normal drawn at the center point between the two eyes f1 and f2 on the feature surface 1 constitutes the gaze of the user U.

Here, the feature surface 1 may be configured as a trapezoid between the eyes f1 and f2 and the mouth end points f4 and f5, but for convenience of operation, the center between the mouth end points f4 and f5, that is, the center of the mouth is defined. It is preferable to set the characteristic point f3 which is shown, and to recognize the triangle which connects this center of gravity f3 and both eyes f1 and f2 as the feature surface 1.

Meanwhile, the extracted nostrils f6 and f7 have not been used for the formation of the feature surface 1, but may be usefully used for setting the search area W of FIG. 3 (G). It can also be used for recognition.

FIG. 5 illustrates a process of recognizing a face of the user U and a movement of the gaze L according to the feature surface 1.

First, when the user U moves the face up, down, left, and right, the feature surface 1 translates along the X and Y axes without changing its size. As a result, the line of sight L also translates along the X and Y axes so that the gaze position P on the observation surface F is also translated.

Next, when the user U rotates the face from side to side, the feature surface 1 rotates about the Y axis, and thus the line of sight L changes to a sine value of the rotation angle along the X axis direction.

When the user U rotates his / her face up and down, the feature surface 1 rotates around the X-axis, whereby the line of sight L is also changed to a sine value of the rotation angle along the Y-axis direction.

Therefore, by calculating the translation and rotation of the feature surface 1 at the next point in time at which the cursor (stare position P) on the observation plane F is at a predetermined position, the values on the X and Y axes to which the cursor is to be moved can be calculated. The cursor is then able to move to the next gaze position (P).

This is represented by the feature surface 1 by installing the camera CAM on the observation surface F (the front of the monitor M), that is, fixing the relative position with the observation surface F, as shown in FIG. The direction of the eye L is tracked by recognizing the direction of the face of the user 1, and the gaze position P is calculated by considering the distance between the observation plane F and the feature plane 1.

Here, the distance between the observation plane F and the feature plane 1, that is, the distance between the user U and the monitor M, cannot be fixed, and the feature plane 1 acquired by the camera CAM is a projection plane in the Z-axis direction. Thus, the translation and rotation of the feature plane 1 calculated therefrom can only detect a relative change relative to the virtual reference plane or the previous plane.

Accordingly, if necessary, the calibration of the gaze position P is required. As shown in FIG. 7A, the user U faces the center of the viewing surface F and the computer COMP. ), The translation or rotation in the up, down, left, and right directions (X, Y) of the user's (U) face as shown in FIG. 7 (B) is calculated relative to this reference position, and the distance also indicates the reference position. It can be computed centrally. Various methods can be used for the 3D motion evaluation method.

On the other hand, the user U may move the face back and forth (translation in the Z-axis direction) or rotate the face with respect to the observation surface F such as the monitor M. If necessary, calculation or rotation in the Z-axis direction may be required. It may be included in the operation of (1), but since it is a simple geometric operation, it does not cause much complexity.

The calculated value corresponds to the gaze position P when the user U changes the gaze L by moving only the face without moving the body.

However, as is well known, the posture of using the computer for each user U is very diverse, whereby it is easy to cause an error in determining the gaze position P.

Accordingly, it is desirable to use artificial intelligence based on neural network theory in order to learn the difference in postures with large deviations for each individual and to determine the gaze position P based on the low error rate.

Since the present invention relates to the gaze position (P) tracking system itself, a detailed description of this neural network theory and the like is omitted.

The above tracking is a two-dimensional method of tracking the gaze position P by calculating and determining the change in the projection image of the feature surface 1 in the Z-axis direction using a single camera CAM. Since it is a two-dimensional system, an error is easy to be included in grasping the distance between the observation surface F and the feature surface 1.

As described above, FIG. 8 illustrates an embodiment in which three-dimensional tracking is possible by two cameras CAM1 and CAM2, that is, a binocular camera. This three-dimensional system can determine the distance between the observation surface (F) and the feature plane (1), and by comparing the two feature planes recognized by the two cameras (CAM1, CAM2) with each other to determine the movement of the three-dimensional feature surface (1) It is possible to more accurately track the line of sight (L) and gaze position (P).

In the above description, the feature surface 1 is set between the two eyes f1 and f2 and the center of the mouth f3, whereby the grasp of the line of sight L and the gaze position P is determined by the relative position of each other. It is assumed that there is no change, in this case, the mouth is fixed to the face so that the user U turns his face without turning his eyes when he moves his gaze (L).

For example, if the user U lowers the pupil at a predetermined gaze position P, the upper line of the feature surface 1 moves downward, so that the controller CTR considers the face rotated downward and moves the cursor. Since the movement direction of the cursor is the same, but the movement amount is different, there is a fear that the cursor position does not exactly match the gaze position P.

This problem can not be maintained by the user (U) continues to turn the eyes only, automatic correction is made over time, so there is no problem in the general case, but when urgent and accurate operation such as control of the process is required It may cause problems.

In order to solve this problem, there may be a method using a conventional pupil tracking system by infrared detection. More preferably, however, additional feature points f6 and f7, such as nostrils, can be used to track pupil movement.

That is, since the relative position between the mouth f3 and the two nostrils f6 and f7 does not change, first, the feature surface 1 is formed by the binoculars f1 and f3 and the mouth f3, and the direction of the eye L is determined. First track the position, and then correct the gaze position (P) by checking the relative position between the mouth (f3) and the nostrils (f6, f7) in the feature surface (1) to determine the excessive movement of the pupil .

Then, the system of the present invention implements a system that can perform pupil tracking in parallel without using harmful and expensive infrared detectors.

In summary, the feature points constituting the upper side of the feature surface 1 are both eyes f1 and f2, and the feature points constituting the lower or lower vertex of the feature surface 1 are the mouth f3 fixed to the face. Or f4 and f5) or noses f6 and f7, and the other fixed feature that does not constitute a lower or lower vertex is used to determine whether the eyes f1 and f2 are moved.

In the configuration of the feature surface 1, the mouths f3 or f4 and f5 are preferred to the noses f6 and f7, which are relative to the distance to the observation surface F as the feature surface 1 increases in size. This is because the size is larger and the magnification of the error is smaller, which results in more accurate tracking.

As described above, according to the present invention, since the precise tracking of the gaze position P is possible, the device is very inexpensive, and thus can be used as a GUI interface device to replace a mouse of a general computer (COMP).

FIG. 9 illustrates a command method by blinking an eye to replace a command by clicking of a mouse when the system of the present invention replaces a mouse. At this time, in order to distinguish between the natural eye blink and the command, the present invention corresponds to the blinking of only one eye, that is, the left eye only in the drawing, that is, the wink corresponding to one click of the mouse.

In FIG. 9 (A), a search area is set around both eyes as shown in FIG. 3 (F) to recognize an open / closed state of the eyes, and black areas are extracted for both eyes in FIG. 9 (B). Next, the digital signal of the black region extracted in FIG. 9 (C) is filtered to remove noise of makeup or lighting around the eyebrows or eyes, and then the width W1, Compute W2) and height (H1, H2) and calculate the aspect ratio (H1 / W1, H2 / W2) and compare them for both eyes. When the aspect ratio difference between both eyes becomes larger than a predetermined threshold, in Fig. 9E, it is determined that the eye (left eye) of the side having the low aspect ratio is closed and the eye (right eye) of the high side is opened. It will be recognized as a click.

On the other hand, if the aspect ratio difference between both eyes is less than or equal to the threshold value, since the user may recognize that the eyes are open or natural blinking, the control unit CTR does not recognize the click command.

As described above, according to the present invention, a very simple and inexpensive system configuration enables accurate and rapid tracking of the gaze position, and may be used in place of a conventional computer mouse as well as a conventional infrared system.

In addition, the system of the present invention can be used not only for the GUI interface of the computer but also for any system requiring tracking of the gaze or gaze of the user.

For example, it can be used for the interface or treatment of people with general paralysis who can only move the face, and in the case of one-sided neglected patients who can only observe one side of the monitor due to different degree of eye movement or bilateral brain dysfunction. In this regard, the present invention can be used directly for their interface or treatment, and in some cases linked to the virtual reality system for the treatment. Can be used.

In addition, it can be used for the treatment of amnesia and executives who have difficulty in regressing memory or making a follow-up plan or for supporting daily life.

The system of the present invention can be used for the purpose of such medical examination and treatment, and can be effectively used in various control devices such as process control, and its ripple effect is very high.

Claims (5)

In the gaze position tracking system for tracking the gaze of the user to determine the gaze position on the observation surface, Face image acquisition means for acquiring a face image of the user as a digital image signal; A control unit which extracts a plurality of feature points of a face from the acquired digital video signal, forms a feature surface from the plurality of feature points, calculates the gaze position on the observation plane by calculating a spatial movement of the feature plane over time; Staring position tracking system characterized in that it comprises. The method of claim 1, A gaze tracking system according to claim 1, wherein the feature point of the face image includes at least one feature point representing both eyes and at least one of a mouth or a nose. The method according to claim 1 or 2, After calculating the gaze position by configuring the feature surface with the feature points of both eyes and mouth, Staring position tracking system, characterized in that to determine the movement of the eye by comparing the relative position of the feature point of the nose and the feature point of the mouth. The method of claim 1, The camera constitutes a binocular camera with two, Each feature surface is composed of images acquired by the cameras, A gaze tracking system, characterized by tracking the gaze position by comparing these two feature surfaces. The method of claim 1, And a user's command is inputted by selectively blinking only one of the eyes when the gaze position reaches a predetermined position.