JP2023006630A - Detection device and control method therefor - Google Patents

Abstract

To enable accurate detection of a gaze point regardless of a gaze position of an observer.SOLUTION: A CPU 3 detects a point of gaze of an observer from eyeball images of both eyes of the observer acquired from eyeball image capturing elements 17. The CPU 3 uses left-eye line-of-sight information and right-eye line-of-sight information weighted according to a directions of the line of sight of the observer to detect the point of gaze.SELECTED DRAWING: Figure 6

Description

The present invention relates to detection of an observer's gaze point.

2. Description of the Related Art In recent years, progress has been made in the automation and intelligentization of head-mounted or eyeglass-type imaging display devices having line-of-sight detection means such as VR and AR. As disclosed in Patent Document 1, the gaze point of the observer is detected, and if an object for privacy protection exists in the image displayed on the display means, processing such as mosaic is performed, or the clarity of the non-display area is improved. has been proposed.

In addition, Patent Document 2 discloses a technique for detecting a point of gaze of an observer and changing the number of image gradations between the detection area and its peripheral area to reduce the speed of image processing without impairing real-time performance. is proposed.

JP 2019-124849 JP 2001-223941

Both Patent Literature 1 and Patent Literature provide users with useful processing by detecting an observer's gaze point. However, neither of these cases considers the characteristics of eye movement according to the direction in which the observer gazes. For this reason, depending on the position where the observer gazes, it may lead to erroneous detection of the gaze point.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a detection device and a control method thereof that can accurately detect a gaze point regardless of an observer's gaze position.

In order to achieve the above object, the present invention has an imaging device for acquiring an eyeball image of an observer who observes a display device, and a detection means for detecting the gaze point of the observer from the eyeball images of both eyes of the observer. The detection means is characterized in that it is weighted according to the direction of the line of sight of the observer, and is configured to use the line of sight information of the left eye and the line of sight information of the right eye to detect the gaze point.

According to the present invention, it is possible to accurately detect the gaze point regardless of the observer's gaze position.

FIG. 1 is a schematic external view of a head-mounted line-of-sight detection device to which the first embodiment is applied; 1 is a block diagram of a line-of-sight detection device to which the first embodiment is applied; FIG. Principle explanatory drawing of the line-of-sight detection method in the first embodiment Schematic diagram of an eyeball image projected on the eyeball imaging device 17 Gaze detection flow Explanatory diagram of resolution in the line-of-sight direction in the first embodiment Explanatory diagram of the eyeball rotation angle in the line-of-sight direction in the first and second embodiments Explanatory diagram of the eyeball rotation angle in the line-of-sight direction in the first and second embodiments

Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.

<Description of configuration>
FIG. 3 shows the configuration of a head-mounted line-of-sight detection device having means for independently detecting left and right line-of-sight directions. Note that the left and right eyes are also referred to as both eyes.

FIG. 1(a) is a front perspective view, and FIG. 1(b) is a back perspective view.

As shown in the front perspective view of FIG. 1( a ), the head-mounted line-of-sight detection device comprises a head-mounted unit 101 and a controller 100 .

A connection cable 102 connects the head-mounted unit 101 and the controller 100 . The connection cable 102 is detachable. Also, the connection cable 102 has an audio connector, and has a right earphone, a left earphone, and a microphone that constitute stereo headphones (not shown).

1A is a photographing lens, and 202 is a photometry circuit.

205 is an illumination light source drive circuit, 13a and 13b are light sources such as light emitting diodes that emit infrared light that is insensitive to the user, and each light source illuminates the user's eyeball. A part of the illumination light reflected by the eyeball is condensed on the eyeball imaging device 17 .

An image pickup device 2 is arranged on a planned imaging plane of the photographing lens 1A.

FIG. 2 is a block diagram showing the electrical configuration of the controller 100 in the head-mounted line-of-sight detection device having the configuration described above. A central processing unit 3 (hereinafter referred to as CPU) of a microcomputer built in the controller 100 controls the line-of-sight detection circuit 201 , the LPF 207 , the memory section 4 and the display section 41 .

A memory unit 4 attached to the CPU 3 has a function of storing video signals from the eyeball imaging device 17, a function of storing sight line correction data for correcting individual differences in line of sight, and correction data for correcting the reliability of left and right eyeballs. there is

The line-of-sight detection circuit 201 A/D-converts the output of the eyeball image formed from the eyeball imaging element 17 by the A/D converter 201 and transmits this image information to the CPU 3 via the LPF 207 . . The CPU 3 extracts each feature point of the eyeball image required for line-of-sight detection according to a predetermined algorithm, which will be described later, and further calculates the user's line of sight from the position of each feature point.

The photometry circuit 202 amplifies the luminance signal output corresponding to the brightness of the object field based on the signal obtained from the image sensor 2 which also serves as a photometry sensor, and then logarithmically compresses and A/D converts it. to the CPU 3 as field luminance information.

In the head-mounted line-of-sight detection device, information related to the subject being gazed at may be displayed as a UI on the display unit 208 in the head-mounted unit according to the line-of-sight detection result (also referred to as line-of-sight information). .

The head-mounted line-of-sight detection device may be of a video transmission type, or may have a VR (virtual reality) or AR (augmented reality) configuration for viewing captured images and videos.

In addition, the operation member α (touch panel compatible liquid crystal, etc.), the operation member β (lever type operation member, etc.), and the operation member γ (button type cross key, etc.) indicated by 41 to 43 in the controller 100 are used for various controls of the line-of-sight detection device. Assume that a member that performs

The operation signal is transmitted to the CPU 3 via the connection cable 102 .

When adjusting the volume of the right earphone and the left earphone that constitute the stereo headphones, the controller 100 controls them.

Further, the controller 100 may issue an instruction to finely adjust the display position of the UI information (index or the like) related to the subject displayed on the display unit 208 in the head-mounted unit.

FIG. 3 is a diagram for explaining the principle of the line detection method, and corresponds to a summary diagram of the optical system for detecting the line of sight in the configuration of FIG. 1 described above.

In FIG. 3, reference numerals 13a and 13b denote light sources such as light-emitting diodes that emit infrared light imperceptible to the user, and each light source illuminates the eyeball of the user. Part of the illumination light reflected by the eyeball is condensed by the light receiving lens 16 onto the eyeball imaging device 17 .

Here, as shown in FIG. 3, the head-mounted visual line detection device has two eyeball imaging elements and acquires left and right eyeball images.

FIG. 4A is a schematic diagram of an eyeball image projected onto the eyeball imaging element 17, and FIG. 4B is a CCD output intensity chart in the eyeball imaging element 17. FIG.

That is, in the head-mounted line-of-sight detection device, this configuration is mounted on each of the left and right sides.

The line-of-sight detection means will be described below with reference to FIGS.

<Description of gaze detection operation>
FIG. 5 shows a schematic flow routine of line-of-sight detection. In FIG. 5, when the line-of-sight detection routine starts, the light sources 13a and 13b emit infrared light toward the user's eyeball 14 in S001. The eyeball image of the user illuminated by the infrared light is formed on the eyeball imaging element 17 through the light receiving lens 16, photoelectrically converted by the eyeball imaging element 17, and the eyeball image can be processed as an electric signal. becomes.

In step S002, the eyeball image signal obtained from the eyeball imaging element 17 as described above is sent to the CPU3.

In step S003, the coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b shown in FIG. 5 and the pupil center c are obtained from the eyeball image signal information obtained in S002. The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the eyeball 14 of the user. At this time, the corneal reflection images Pd and Pe formed by part of the infrared light reflected on the surface of the cornea 142 are condensed by the light receiving lens 16 and formed on the eyeball imaging element 17 (point Pd' in the drawing). , Pe'). Similarly, the light beams from the ends a and b of the pupil 141 are also imaged on the eyeball imaging device 17 . 4, FIG. 6A shows an example of a reflected image obtained from the eyeball imaging device 17, and FIG. 7B shows an example of luminance information obtained from the eyeball imaging device 17 in the area α of the above example image. indicates As shown, the horizontal direction is the X axis and the vertical direction is the Y axis. At this time, Xd and Xe are the coordinates in the X-axis direction (horizontal direction) of the images Pd' and Pe' formed by the corneal reflection images of the light sources 13a and 13b. Let Xa and Xb be the coordinates in the X-axis direction of the images a' and b' formed by the light beams from the ends a and b of the pupil 14b. In the luminance information example of (b), extremely high levels of luminance are obtained at the positions Xd and Xe corresponding to the images Pd' and Pe' formed by the corneal reflection images of the light sources 13a and 13b. An area between coordinates Xa and Xb, which corresponds to the area of the pupil 141, has an extremely low level of luminance, except for the positions Xd and Xe. On the other hand, in a region having an X coordinate value lower than Xa and a region having an X coordinate value higher than Xb, which corresponds to the region of the iris 143 outside the pupil 141, a value intermediate between the two luminance levels is obtained. From the luminance level variation information with respect to the X-coordinate position, the X-coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images of the light sources 13a and 13b and the X-coordinates of the pupil end images a' and b' are obtained. Xa and Xb can be obtained. When the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 16 is small, the coordinate Xc of the point corresponding to the pupil center c (assumed to be c′) formed on the eyeball imaging device 17 is It can be expressed as Xc≈(Xa+Xb)/2. From the above, the X coordinate of c' corresponding to the center of the pupil formed on the eyeball imaging element 17 and the coordinates of the corneal reflection images Pd' and Pe' of the light sources 13a and 13b could be estimated.

Furthermore, in step S004, the imaging magnification β of the eyeball image is calculated. β is a magnification determined by the position of the eyeball 14 with respect to the light receiving lens 16, and can be obtained substantially as a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

Further, in step S005, since the X coordinate of the midpoint of the corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of the cornea 142 substantially match, the standard distance between the center of curvature O of the cornea 142 and the center c of the pupil 141 is determined. Assuming that the physical distance is Oc, the rotation angle θ _X of the optical axis of the eyeball 14 in the ZX plane is
β*Oc*SINθ _X ≈ {(Xd+Xe)/2}-Xc
can be obtained from the relational expression of 3 and 4 show an example of calculating the rotation angle _θX when the user's eyeball 14 rotates within a plane perpendicular to the Y-axis. The calculation method of the rotation angle θy in the case of rotation within a plane perpendicular to the is also the same.

When the rotation angles θx and θy of the optical axis of the user's eyeball 14 are calculated in the previous step, θx and θy are used in step S006 to determine the position of the user's line of sight (at which the user is gazing) on the display element 10. The position of the point (hereinafter referred to as the gaze point) is obtained. Assuming that the gaze point position is the coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display element 10,
Hx_L=m×(Ax×θx+Bx)×nLx
Hx_R=m×(Ax×θx+Bx)×nRx
Hy_L=m×(Ay×θy+By)×nLy
Hy_R=m×(Ay×θy+By)×nRy
can be calculated as At this time, the coefficient m is a constant determined by the configuration of the finder optical system of the camera or the optical system of the head-mounted line-of-sight detection device, and the rotation angles θx and θy correspond to the center c of the pupil 141 on the display element 10. It is a conversion coefficient that converts to the position coordinates to be used. Assume that the coefficient m is determined in advance and stored in the memory unit 4 . Further, Ax, Bx, Ay, and By are line-of-sight correction coefficients for correcting individual differences in the line-of-sight of the user, which are obtained by performing a calibration work described later and stored in the memory unit 4 before the start of the line-of-sight detection routine. shall be memorized.

Further, the memory unit 4 stores correction coefficients (nLx, nLy, nRx, nRy) for correcting the reliability of the left and right eyes when determining the gaze point.

Comprehensive fixation points Hx and Hy are calculated from these Hx_L, Hx_R, Hy_L and Hy_R.

Here, as shown in FIG. 10, in the head-mounted line-of-sight detection device, when the gaze point on the display screen is near the center of the display screen, the left eye rotation angle L−θx and the right eye rotation angle R -θx has almost the same value (L-θx=R-θx).

Also, as shown in FIG. 8, when the gaze point is on the left end of the display screen, the relationship between the rotation angle L-θx of the left eye and the rotation angle R-θx of the right eye is:
R-θx>>L-θx
becomes.

The rotation angle L-θx of the left eye and the rotation angle R-θ of the right eye are used as the reliability of gaze point detection, and the reliability is corrected using nLx, nLy, nRx, and nRy. Then, the user's gaze point is calculated by integrating the reliability degrees.

As shown in Fig. 6, the human eye has the characteristic that the closer the point of gaze is to the center, the higher the resolution and the better the ability to accept information. .

In addition, when the human eye looks at the edge of the display screen, the eyeball tends to tremble regardless of one's will, and there is a possibility of erroneous detection when detecting the line of sight.

In this case, the reliability correction coefficients for the left and right eyeballs are selected so as to increase the weighting of the sight line detection result for the left eye, and the total gaze point is calculated.

Specifically, for example, the cutoff frequency in filtering the left eye performed by the LPF 207 is increased with respect to the output from the imaging element 17 for eyeballs when the eyeball image is formed.

In other words, the cutoff frequency during filtering for the right eye is decreased.

Also, when the effect is on the left eye, the cutoff frequency is further reduced during filtering for the right eye.

Also, as shown in FIG. 8, when the gaze point is on the right end of the display screen, the relationship between the rotation angle L-θx of the left eye and the rotation angle R-θx of the right eye is as follows.
L-θx>>R-θx
becomes.

In this case, the correction coefficients for the right and left eyeballs are selected so as to increase the reliability of the sight line detection result for the right eye, and the total gaze point is calculated.

That is, the point of gaze is determined by increasing the reliability of the eye closer to the edge so that the eye closer to the edge is emphasized.

A reliability correction coefficient is similarly selected for the Y direction. Specifically, for example, the cutoff frequency in the filtering process performed by the LPF 207 is made smaller for the left eye with respect to the output from the eyeball imaging device 17 when the eyeball image is formed.

Specifically, for example, the cutoff frequency in filtering the right eye performed by the LPF 207 is increased with respect to the output from the image sensor 17 for eyeballs formed by forming an eyeball image. In other words, the cutoff frequency during filtering for the left eye is decreased.

In addition, when the effect is on the right eye, the cutoff frequency is further reduced during the filtering process for the left eye.

By doing so, it is possible to reduce the effect of fine trembling of the eyeball movement, suppress the effect of the eye that makes it difficult to grasp the sense of distance and stereoscopic effect, and suppress the deterioration of the accuracy of sight line detection.

After the coordinates (Hx, Hy) of the center c of the pupil 141 on the display element 10 are calculated as described above, the coordinates are stored in the memory unit 4 in step S007, and the line-of-sight detection routine ends.

Although the method for obtaining the point-of-regard coordinates on the display element using the corneal reflection images of the light sources 13a and 13b has been described above, it is not limited to this, and any method for obtaining the eyeball rotation angle from the captured eyeball image may be used. However, the present invention can be applied.

In the first embodiment described above, the cutoff frequency of the LPF 207 for the image information of the left and right eyeball images is changed according to the rotation angles of the left and right eyeballs, thereby suppressing the deterioration of the sight line detection accuracy.

In this embodiment, the LPF 207 has a moving average filter structure, the rotation angles of the left and right eyeballs are detected in the same manner as in the first embodiment, and the number of moving average samples (the number of taps) is changed according to the rotation angle. .

For example, when the gaze point is close to the left edge of the display screen as shown in FIG. 8, the relationship between the rotation angle L−θx of the left eye and the rotation angle R−θx of the right eye is
R-θx>>L-θx
becomes.

At this time, the number of samples (tap number) of the LPF (moving average filter) 207 for the right eye is increased, and the number of samples (tap number) of the LPF (moving average filter) 207 for the left eye is decreased.

By doing so, it is possible to suppress the effect of trembling of the eyeballs regardless of one's will when looking at the edge of the display screen, and to suppress erroneous detection of the line of sight.

In addition, simply increasing the number of samples (the number of taps) increases the amount of calculation, resulting in deterioration of real-time performance. Therefore, by adaptively switching according to the angle of the eyeball, it is possible to suppress the effect of trembling of the eyeball regardless of one's intention when looking at the edge of the display screen, and to suppress false detection of the line of sight. .

<Others>
In the above-described embodiment, an example in which the present invention is implemented in a digital camera has been described, but the present invention may be applied to any device as long as it performs line-of-sight detection. For example, it is also possible to implement in a head mounted display, a smart phone, a PC, or the like.

Also, in the operations described using the flowcharts in the above embodiments, it is possible to change the order of the steps to be executed as appropriate so as to achieve the same purpose.

The present invention can also be configured to supply a program implementing one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium. . It can also be realized by processing in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

3 CPUs
17 Imaging device for eyeball

Claims (6)

an imaging device that acquires an eyeball image of an observer who observes the display device;
having a detection means for detecting the gaze point of the observer from the eyeball images of both eyes of the observer;
The detecting device is characterized in that the detecting means uses the sight line information of the left eye and the sight line information of the right eye with weighting according to the direction of the sight line of the observer to detect the gaze point. The detecting means weights based on the eyeball angle of the left eye and the eyeball angle of the right eye, and uses the sight line information of the left eye and the sight line information of the right eye used for detecting the sight point to determine the gaze point of the observer. 2. The detection device according to claim 1, for detecting. The detecting means may set the cutoff frequency of the filter for the line-of-sight information corresponding to the eyeball having the larger eyeball angle among the two eyes to be lower than the cutoff frequency of the filter for the line-of-sight information corresponding to the eyeball having the smaller eyeball angle. 3. A detection device according to claim 2. The detecting means reduces the number of samples of the moving average filter for the line-of-sight information corresponding to the eyeball with the larger eyeball angle than the number of samples of the moving average filter for the line-of-sight information corresponding to the eyeball with the smaller eyeball angle. 4. The detection device according to any one of claims 1 to 3, characterized in that: 5. The detection device according to any one of claims 1 to 4, wherein said display element displays an index corresponding to the gaze point detected by said detection means. A control method for a detection device having an imaging device that acquires an eyeball image of an observer observing a display device, comprising:
a detection step of detecting the gaze point of the observer from the eyeball images of both eyes of the observer;
In the detection step, the sight line information of the left eye and the sight line information of the right eye are weighted according to the direction of the sight line of the observer, and the sight line information is used to detect the gaze point.

Images