JP2648198B2 - Gaze direction detection method - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a method for detecting a gaze (eye movement) direction of a person without contacting the person.

(Prior art) As a field to study the internal condition such as the degree of eye fatigue of people engaged in office equipment, electronic equipment, and other terminals, or as an auxiliary means of communication for disabled vehicles with limited limbs This is performed using a method of detecting a gaze direction of a person.

That is, conventionally, as a method of tracking and processing the movement of human eyes, there is a method of performing measurement by attaching a part of a measuring device to a head or the like. In addition, when following the eye movements, since part or all of the human body is fixed, the degree of freedom of the original movement is restricted, and only unnatural eye movement data is obtained. Was.

In FIG. 5, after the distance between the head 1a of a person (hereinafter, referred to as a subject) and the eye movement detector 2 is fixed at a certain interval, a measurement light beam is emitted from the eye movement detector 2 to the eye 1b. Then, the light beam reflected from the eyeball 1b is taken into the eyeball movement detector 2 again, subjected to data processing, and subjected to the eyeball 1b of the subject.
Is detected. This is because the eye movement detector 2 attached to the head 1a and the eyeball
There is a disadvantage that the relative position to 1b is shifted from the initial setting position.

FIG. 6 shows a state in which the subject's head 1a is fixed with a plurality of metal fittings 3, and a measurement light beam is emitted from the eyeball movement detector 2 to the eyeball 1b in the same manner as in FIG. Is again taken into the eye movement detector 2 and subjected to data processing so that a fine movement of the eyeball 1b of the subject can be detected. this is,
Although more accurate data can be detected compared to Fig. 5,
There is a disadvantage that it is not possible to obtain more natural data with a wide range of motion.

FIG. 7 shows that the contact lens 4 with an electromagnetic induction coil is attached to the cornea 1c of the eyeball 1b of the subject, an alternating magnetic field is generated around the head 1a, and the movement of the eyeball 1b guides the contact lens 4 to the coil. Eye movement is detected from a change in AC potential. This is excellent in real-time property, but has the disadvantage that natural data cannot be obtained because the movement of eye movement is directly restricted by the contact lens 4.

As described above, in the conventional gaze direction detecting method, it is necessary to attach some member to the subject, and there is a problem that natural eye movement is restricted.

(Object of the Invention) The present invention solves the above-mentioned problems of the conventional method, and detects the eye movement characteristics under a more natural movement of the subject, so that the gaze direction can be detected without contact with the subject. The purpose is to provide a detection method.

(Constitution of the invention) (Difference between features of the invention and conventional technology) In order to achieve the above object, the present invention provides an imaging system including a plurality of half mirrors arranged in parallel and a plurality of optical paths formed by the half mirrors. When a person image is captured, the characteristic pattern is projected onto the person image, the specific pattern is recognized by the imaging system, the eyeball position of the person image is detected, and the reflection of the specific pattern on the pupil is performed. The most main feature is to perform image processing on the gaze direction based on the position and shape of the pattern.

The difference from the prior art is that the gaze direction can be detected by following the natural movement of the eye without contacting the subject.

(Example) The basic configuration for carrying out the method of the present invention is as follows. In order to detect the direction in which the subject is facing, a predetermined pattern is irradiated from an infrared element arranged in front of the subject, and The reflection pattern is captured by an imaging camera via an infrared transmission filter arranged on an optical path for optically capturing the reflection pattern in parallel, and the gaze direction is detected by image processing from the position and shape of the reflection pattern.

Here, the reflection image by infrared irradiation on the head is only the eyeball, reflection images from other than the head, for example, glasses, necklaces and the like, the reflectance is different,
Due to the difference in size, it is easily distinguished from the reflected image from the eyeball.

The pattern to be irradiated is generally black or close to black. An object having a chromaticity near black absorbs infrared light having a longer wavelength than visible light. Therefore, if infrared rays are emitted toward the eyeball, the light passes through the pupil, is reflected from the back of the eyeball, and passes through the pupil again. The pupil can be extracted because the light reflected from the inside of the eyeball and the light reflected from the surface thereof have different luminances.

FIG. 1 shows a configuration diagram of an optical system for carrying out the method of the present invention. In the figure, 5 is an infrared element for emitting a specified pattern, 6 is an infrared element for extracting eyeballs, 7a, 7b, 7c
... are half mirrors for splitting the optical path 8A into a plurality of optical paths 8a, 8b, 8c... 8n, 9 is a total reflection mirror forming the optical path 8n, and 10a is an infrared ray installed on the branch optical path 8a. The filter selects a reflection image (reflection pattern) of the specific pattern on the eyeball 1b of the subject irradiated from the infrared element 5 that emits the specified pattern. Reference numeral 10b denotes an infrared filter installed on the branch optical path 8b, and selects a reflection image from the eyeball 1b of the subject irradiated by the eyeball clipping infrared element 6. .. 11n are imaging lenses provided in the branch optical paths 8a, 8b, 8c.
Reference numerals b, 12c... 12n denote imaging cameras provided on the respective branch optical paths, and reference numeral 13 denotes a processing system for performing arithmetic processing on images from the plurality of imaging cameras 12a to 12n.

Next, the operation will be described with reference to the specific pattern 5a shown in FIG. 2, the separated image shown in FIG. 3, and each example diagram showing a representative gaze direction of the subject shown in FIG.

Usually, when a wavelength in the visible light or ultraviolet range is irradiated, it is almost reflected on the surface of the eyeball, so that it is difficult to determine the moving direction of the eyeball. Further, although the contour of the eye can be extracted, it is difficult to detect even the direction in which the user is looking.

Therefore, in the present embodiment, an infrared element 5 for generating the specified pattern is arranged on the optical device side as shown in FIG.
The degree of distortion of the characteristic pattern 5a (FIG. 2) projected on the surface of the eye is captured as one piece of image information. At this time, the irradiation wavelength is infrared rays, but is different from that of the eyeball extracting infrared element 6. The reason for choosing this infrared ray is to avoid recognizing light from outside the area where the subject is gazing.

First, the subject's head 1a is arranged in front of the half mirror 7a, and the subject is irradiated with the infrared elements 5 and 6;
As shown in FIG. 2, a specific pattern 5a from the infrared element 5 is projected on the surface of the eyeball 1b. In addition, in FIG.
1d shows an iris and a pupil (hatched portion) in the eyeball 1b.

The subject's information (image) is from optical path 8A, half mirror 7a
And this optical path 8A is composed of n-1 half mirrors 7a, 7b,
7c and one total reflection mirror 9 are guided to n imaging lenses 11a, 11b, 11c... 11n via branch optical paths 8a, 8b, 8c. In the branch optical paths 8a and 8b, specific image information is selected by the infrared filters 10a and 10b. That is, only the reflection pattern 5b of the specific pattern 5a emitted from the infrared element 5 is extracted as shown in FIG. In addition, from the reflection image irradiated from the infrared element 6, only the pupil 1d of the subject is extracted as shown in FIG. 3 (2). The image (face) of the subject's head 1a via the total reflection mirror 9
Is taken out as shown in FIG. 3 (3). The third
Although not shown in the figure, visible images are taken out from the branch optical paths 8c... 8n-1.
.. 12n are input to the arithmetic processing system 13. 3 (1) and 3 (2), the temporary broken line of the head 1a is drawn for easy understanding, and is not captured by the imaging cameras 12a and 12b.

FIG. 4 is a detailed view of FIG. 3 (1). In FIGS. 4 (1) to (5), 5b is a reflection pattern of the specific pattern 5a projected by the infrared element 5, and 1e is a nose of the subject. And the other symbols represent the eyeball 1b, iris and pupil 1d shown in FIG. Here, FIG. 4 (1) shows the reflection pattern 5b of the characteristic pattern 5a on the eyeball 1b, the eyeball 1b, and the like when the subject gazes at the front in the direction of the half mirror 7a, and FIG. Specific pattern when subject gazes rightward
5 shows the relationship between the reflection pattern 5b of 5a and the relative positions of the eyeball 1b and the like. Similarly, FIG. 4 (3) shows a case where the subject is gazing leftward, FIG. 4 (4) shows a case where gazing upward, and FIG. 4 (5).
Respectively, showing that the position of the eyeball 1b and the degree of distortion of the reflection pattern 5b are different.

In order to make it easier to perform the alignment between the images separated by the imaging cameras 12a to 12n in this manner,
Only one optical path 8n captures an image to the imaging camera 12n immediately thereafter.

Next, processing of the image obtained by the optical processing is performed as follows.

First, noise is removed from an image input from each optical path. The two images obtained via the infrared filters each include an image having a particularly large luminance value. The place where the luminance value is large is the reflected image from the eyeball. One image is a specific pattern and the other is
It is a reflection image from a pupil. The first and second derivatives are subjected to arithmetic processing for these two reflected images to perform edge detection. The degree of distortion of the specific pattern can be understood by comparing the initially registered specific pattern with these edge images. The amount and direction of movement of the eyeball can be determined from the degree of distortion.

The image that does not pass through the filter is used as a reference image for performing position correction of the superposition of the eyeball and the specific pattern on the eyeball obtained through the filter.

From these combined image data, the gaze direction of the person is recognized by superimposing these three images.

The gaze direction of the subject is determined in real time by the method described above.

In the present embodiment, it goes without saying that the infrared element 5 can be changed depending on the resolution of the imaging camera. When a low-resolution imaging camera was used, detection was possible by forming a specific pattern with a plurality of elements in a square with a side of 2 m at a position 2 m ahead of the subject. When a high-resolution imaging camera was used, the detection was possible by forming a specific pattern in a 50 cm square arranged on the 2 m of the subject.

Further, the infrared element 6 is not always necessary,
Cutting out the gaze direction from the images shown in FIGS. 1A and 1B can be performed with low accuracy. In addition, the operation could be confirmed not only with the square vertex arrangement shape described in the embodiment but also with a more complicated figure such as a regular hexagonal vertex arrangement shape, a circle, a character shape, and a random shape.

As described above, since one optical path is split into the image captured by the half mirror, the alignment between the images can be easily performed. Therefore, the process of performing the arithmetic sum in the arithmetic processing system 13 can be performed. It can be performed at high speed with a simple configuration, and can be easily incorporated. As described above, since the preprocessing is optically performed instead of performing the separation based on the difference in luminance on the image data, the gazing point can be recognized with high accuracy.

(Effects of the Invention) As described above, the present invention irradiates a subject with a specific pattern in a non-contact manner with the subject, and obtains a reflection image of the specific pattern and an eyeball and a nose from a relatively vague image. Since the preprocessing for recognizing and selecting relative positions such as is performed, more effective image data can be obtained, and the gaze direction of the subject can be detected more accurately.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an optical system for performing the method of the present invention,
FIG. 2 is a view showing an example of a specific pattern irradiated on the eyeball of the subject, FIG. 3 is a view showing each example of the separated image, and FIG. 4 is a representative gaze direction of the test subject and a specific pattern projected on the eyeball of the test subject FIG. 5 to FIG. 7 are diagrams showing each example of a conventional gaze direction detecting method. 1a ... subject's head, 1b ... eyeball, 1c ... cornea, 1d ...
Iris and pupil, 1e …… nose, 5,6 …… infrared element, 5a ……
Specific pattern, 5b …… Reflection pattern, 7a, 7b, 7c ……
... half mirror, 8A ... optical path, 8a, 8b, 8c ... 8n ... branch optical path, 9 ... total reflection mirror, 10a, 10b ... infrared filter, 11a, 11b, 11c ... 11n ... imaging lens , 12a, 12b, 12
c: 12n: imaging camera, 13: arithmetic processing system.

Claims (1)

(57) [Claims] When a plurality of half mirrors are arranged in parallel and a person image is taken into an image pickup system including a plurality of optical paths formed by the half mirrors, a specific pattern is projected on the person image and the specified pattern is projected. Is recognized by the imaging system,
A gaze direction detection method, comprising: detecting an eyeball position of the person image; and performing image processing of a gaze direction based on a position and a shape of the reflection pattern of the specific pattern on the pupil.