JP2022165239A - Imaging apparatus and control method for the same, and program - Google Patents

Abstract

To provide an imaging apparatus that can improve accuracy of detecting a user's line-of-sight position and a control method for the same, and a program.SOLUTION: An imaging apparatus 1 displays a through image on an internal finder 10, and comprises: an image pickup device for an eye 17 that picks up an image of the eyeball 14 of a user who looks into the finder 10 to generate eye image data; and a line of sight detection circuit 201 that detects a position of the user's line of sight fixed on the through image on the finder 10 on the basis of, the eye image data. When the user moves, with an operating member 42, the detected line-of-sight position (position C) displayed on the finder 10 to a position B and subsequently depresses a release button 5 to perform photographing with the position B as a focus position, the position B is collected as a correct answer position and is used for learning in creating a reasoner that estimates the user's line-of-sight position with the eye image data as input data.SELECTED DRAWING: Figure 9

Description

The present invention relates to an image pickup apparatus, its control method, and a program, and more particularly to an image pickup apparatus, its control method, and its program, which perform focus control based on information on a detected line-of-sight position.

In recent years, imaging devices have become increasingly automated and intelligent, making it possible to recognize the user's intended subject and perform focus control based on information about the user's line of sight when looking through the viewfinder, without having to manually enter the subject's position. An imaging device has been proposed. In this case, when the imaging device detects the line-of-sight position of the user, a deviation occurs between the line-of-sight position intended by the user and the position of the user's line of sight recognized by the imaging device, and the user's intended subject may be focused. Sometimes you can't.

On the other hand, an index is displayed in the finder before photographing, an instruction is given to the user to gaze at the index, and in the gaze state, the user's gaze position is detected, and the amount of deviation from the index position is detected. Execute calibration. After that, a technique is known in which, at the time of photographing, by correcting the line-of-sight position of the user recognized by the imaging apparatus by the amount of the detected deviation, the corrected line-of-sight position is closer to the user's intention. (See Patent Document 1, for example).

In addition, the detection accuracy of the line-of-sight position is determined, and the display objects are displayed sparsely at locations where the determined detection accuracy is low, and the display objects are densely displayed at locations where the line-of-sight detection accuracy is high. There is known a technique for preventing selection of an unintended line-of-sight position by the user (see, for example, Patent Document 2).

Japanese Patent Application Laid-Open No. 2004-008323 JP 2015-152938 A

However, in the conventional technology disclosed in Patent Document 1, when conditions such as the pupil diameter and distance of the user's eyes looking through the viewfinder are different between when shooting and when performing calibration, the detection accuracy of the line-of-sight position decreases. If calibration is performed again every time the detection accuracy of the line-of-sight position drops, the burden on the user increases.

As described in Patent Document 2, it is possible to prevent erroneous detection of the line-of-sight position by enlarging the focal frame at a location with poor detection accuracy, but if the focal frame becomes large, the user's desired Focus control may not be possible.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging apparatus, a control method thereof, and a program capable of improving detection accuracy of a user's line-of-sight position.

An imaging apparatus according to claim 1 of the present invention is an imaging apparatus for displaying a through image on an internal finder, comprising generating means for imaging an eyeball of a user looking through the finder to generate eye image data; a line-of-sight detecting means for acquiring image data and detecting a line-of-sight position of a user focused on the through-the-lens image of the finder based on the acquired eye image data; display control means for displaying on the finder movably to another position; collection means for collecting the other position as a correct position when there is a second user operation to determine the other position as the focus position; wherein the correct position is used for learning to create a reasoner for estimating the line-of-sight position using the eye image data acquired by the line-of-sight detection means as input data.

ADVANTAGE OF THE INVENTION According to this invention, the detection accuracy of a user's gaze position can be improved.

1 is a diagram showing an outline of an internal configuration of an imaging device according to Example 1; FIG. It is a figure which shows the external appearance of an imaging device. 3 is a block diagram showing an electrical configuration built in the imaging device; FIG. FIG. 4 is a view showing the viewfinder's inner field of view when the viewfinder in FIG. 3 operates; It is a figure for demonstrating the principle of a line-of-sight detection method. FIG. 4 is a diagram for explaining a method of detecting coordinates corresponding to a corneal reflection image and a pupil center from eye image data; 9 is a flowchart of line-of-sight detection processing; 4A and 4B are diagrams showing an example of eye image data at the time of calibration and at the time of photographing, which are determined to have low reliability by the line-of-sight detection reliability determination circuit in FIG. 3; FIG. 10 is a flow chart of processing for collecting correct data used for learning when creating an inference device; FIG. 6 is a flowchart of focus processing during shooting; FIG. 11 is a flowchart of a process of determining a reliability reduction factor of the first estimated gaze point position according to the second embodiment; FIG. FIG. 12 is a diagram for explaining a method of generating differential eye image data in step S1104 of FIG. 11; 1 is a schematic diagram showing an example of the overall configuration of a CNN according to Example 1; FIG. FIG. 14 is a schematic diagram showing an example of a partial configuration of the CNN of FIG. 13;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the present invention according to the claims, and not all combinations of features described in the embodiments are essential for the solution of the present invention. .

(Example 1)
1 to 10, 13, and 14, a learning method and an inference method that are executed to improve the detection accuracy of the line-of-sight position in the imaging device 1 according to the first embodiment of the present invention will be described below.

The configuration of the imaging device 1 will be described with reference to FIGS. 1 to 3. FIG.

2A is a front perspective view, FIG. 2B is a rear perspective view, and FIG. 2C is an operation member 42 of FIG. 2B. It is a figure for explaining.

In this embodiment, the imaging apparatus 1 is composed of a camera housing 1B and a photographing lens 1A detachably attached thereto.

A release button 5 is provided on the front surface of the camera housing 1B as shown in FIG. 2(a).

The release button 5 is an operation member that receives an imaging operation from the user.

Further, as shown in FIG. 2(b), an eyepiece window 6 and operation members 41 to 43 are provided on the rear surface of the camera housing 1B.

The eyepiece window 6 is a window for the user to look into an image for visual recognition displayed on a finder 10 (described later with reference to FIG. 1) included inside the camera housing 1B.

The operation member 41 is a touch panel compatible liquid crystal, the operation member 42 is a lever type operation member, and the operation member 43 is a button type cross key. In this embodiment, the operation members 41 to 43 used for camera operation such as movement control by manual operation of an estimated position of gaze, which will be described later, are provided on the camera housing 1B, but the present invention is not limited to this. For example, other operation members such as an electronic dial may be provided in addition to or instead of the operation members 41 to 43 on the camera housing 1B.

FIG. 1 is a cross-sectional view of the camera housing B cut along the YZ plane formed by the Y-axis and Z-axis shown in FIG. In FIG. 1, the same components as those in FIG. 2 are denoted by the same reference numerals.

In FIG. 1, a photographing lens 1A is detachably attached to a camera housing 1B. In this embodiment, only two lenses 101 and 102 are shown as the lenses inside the photographing lens 1A for the sake of convenience, but it is well known that the photographing lens 1A actually comprises a larger number of lenses.

The camera housing section 1B contains an image sensor 2, a CPU 3, a memory section 4, a viewfinder 10, a viewfinder drive circuit 11, an eyepiece lens 12, light sources 13a to 13b, a light splitter 15, a light receiving lens 16, and an eye sensor. An imaging element 17 is provided.

The image pickup device 2 is arranged on a planned imaging plane of the photographing lens 1A and picks up an image. The imaging device 2 also serves as a photometric sensor.

The CPU 3 is a central processing unit of a microcomputer that controls the imaging apparatus 1 as a whole.

The memory unit 4 records images captured by the image sensor 2 . The memory unit 4 also has a function of storing image signals from the image pickup device 2 and the eye image pickup device 17, and stores line-of-sight correction data for correcting individual differences in line of sight, which will be described later.

The finder 10 is composed of a liquid crystal or the like for displaying an image (through image) captured by the imaging device 2 .

A finder drive circuit 11 is a circuit that drives the finder 10 .

The eyepiece lens 12 is a lens for the user to look into and observe a visual recognition image displayed on the finder 10 through the eyepiece window 6 (FIG. 2).

The light sources 13a-13b are light sources composed of infrared light emitting diodes for illuminating the user's eyeball 14 in order to detect the user's line of sight direction, and are arranged around the eyepiece window 6 (FIG. 2). Corneal reflection images (Purkinje images) Pd and Pe (FIG. 5) of the light sources 13a to 13b are formed on the eyeball 14 by lighting the light sources 13a to 13b. In this state, light from the eyeball 14 passes through the eyepiece 12, is reflected by the light splitter 15, and is received by the light-receiving lens 16. The image sensor 17 (generating means) has a photoelectric element array such as CMOS arranged two-dimensionally. An eye image including an eyeball image is formed thereon to generate eye image data. The light-receiving lens 16 positions the pupil of the user's eyeball 14 and the eye imaging device 17 in a conjugate imaging relationship. A line-of-sight detection circuit 201 (line-of-sight detection means: FIG. 3) detects the line-of-sight direction (focusing on the image for visual recognition) from the position of the corneal reflection image in the eyeball image formed on the eye imaging element 17 by a predetermined algorithm to be described later. The viewpoint of the user who escapes (hereinafter referred to as the first estimated gaze point position) is detected.

The light splitter 15 reflects the light that has passed through the eyepiece lens 12 , forms an image on the eye imaging device 17 via the light receiving lens 16 , transmits the light from the viewfinder 10 , and allows the user to display the image on the viewfinder 10 . It is configured so that the visual recognition image to be displayed can be viewed.

The photographing lens 1A includes a diaphragm 111, a diaphragm driving device 112, a lens driving motor 113, a lens driving member 114 including a driving gear, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118.

A photocoupler 115 detects the rotation of a pulse plate 116 interlocked with a lens drive member 114 and transmits it to a focus adjustment circuit 118 .

The focus adjustment circuit 118 drives the lens drive motor 113 by a predetermined amount based on the information from the photocoupler 115 and the lens drive amount information from the camera housing 1B, and moves the photographing lens 1A to the in-focus position.

A mount contact 117 is an interface between the camera housing 1B and the photographing lens 1A, and has a known configuration. Signals are transmitted through the mount contact 117 between the camera housing 1B and the photographing lens 1A. The CPU 3 of the camera housing portion 1B obtains the type information and optical information of the photographing lens 1A to determine the focusable range of the photographing lens 1A attached to the camera housing portion 1B.

FIG. 3 is a block diagram showing an electrical configuration built into the imaging device 1. As shown in FIG. In FIG. 3, the same numbers are assigned to the same components as in FIGS.

The camera housing 1B includes a line-of-sight detection circuit 201, a photometry circuit 202, an automatic focus detection circuit 203, a signal input circuit 204, a finder drive circuit 11, a light source drive circuit 205, a line-of-sight detection reliability determination circuit 31, and a communication circuit 32. , and they are connected to the CPU 3 respectively. The photographic lens 1A also includes a focus adjustment circuit 118 and an aperture control circuit 206 included in the aperture drive device 112 (FIG. 1), which are connected to the CPU 3 of the camera housing 1B via mount contacts 117, respectively. signal transmission.

The line-of-sight detection circuit 201 A/D-converts the eye image data formed and output on the eye imaging device 17 and transmits the eye image data to the CPU 3 . The CPU 3 extracts each feature point of the eye image necessary for detecting the line of sight from the eye image data according to a predetermined algorithm, which will be described later. gazing point position).

The photometry circuit 202 amplifies the luminance signal output corresponding to the brightness of the object scene based on the signal obtained from the image pickup device 2 which also serves as a photometry sensor, and then logarithmically compresses and A/D converts it. It is sent to the CPU 3 as luminance information.

The autofocus detection circuit 203 A/D-converts signal voltages from a plurality of pixels included in the image sensor 2 and used for phase difference detection, and sends them to the CPU 3 . The CPU 3 calculates the distance to the object corresponding to each focus detection point from the signal voltages from the plurality of pixels. This is a well-known technique known as imaging plane phase difference AF. In this embodiment, there are 180 focus detection points on the imaging surface of the finder 10, as shown in the finder field image (viewing image) shown in FIG.

The signal input circuit 204 is connected to switches SW ₁ and SW ₂ (not shown). The switch SW1 is turned on by the _first stroke of the release button 5 (FIG. 2(a)) to start photometry, distance measurement, line-of-sight detection, and the like of the imaging device 1. FIG. The switch SW2 is turned on by the _second stroke of the release button 5 to start the release operation. Signals from the switches SW ₁ and SW ₂ are input to the signal input circuit 204 and transmitted to the CPU 3 .

A line-of-sight detection reliability determination circuit 31 (reliability determination means) determines the reliability of the first estimated gaze point position calculated by the CPU 3 . This determination is performed based on the difference between two pieces of eye image data, ie, eye image data acquired during calibration, which will be described later, and eye image data acquired during photographing. Specifically, the differences here are the difference in the pupil diameter, the difference in the number of corneal reflection images, and the difference in the penetration of external light detected from each of the two eye image data. More specifically, the pupil edge is calculated by the line-of-sight detection method described later with reference to FIGS. Reliability is determined to be low. This is because the pupil ends are connected to estimate the pupil 141 (FIG. 5) of the user's eyeball 14, and the more pupil ends that can be extracted, the higher the estimation accuracy. Alternatively, the degree of reliability may be determined based on how much the pupil 141 calculated by connecting pupil ends is distorted with respect to the circle. As another method, it may be determined that the reliability is high in the vicinity of the indicator that the user is gazing at during calibration, which will be described later, and the reliability is low as the distance from the indicator increases. When transmitting the user's line-of-sight position information calculated by the line-of-sight detection circuit 201 to the CPU 3 , the line-of-sight detection reliability determination circuit 31 transmits the reliability of the line-of-sight position information to the CPU 3 .

Under the control of the CPU 3, the communication circuit 32 communicates with a PC (not shown) on the server via a network (not shown) such as a LAN or the Internet.

Further, the operation members 41 to 43 described above are configured so that their operation signals are transmitted to the CPU 3, and movement control and the like of the first estimated gazing point position to be described later are performed by manual operation accordingly.

FIG. 4 is a view showing the viewfinder's internal field of view, showing a state in which the viewfinder 10 operates (a state in which an image for visual recognition is displayed).

As shown in FIG. 4, the viewfinder field includes a field mask 300, a focus detection area 400, 180 distance measuring point indices 4001 to 4180, and the like.

Each of the ranging point indices 4001 to 4180 is displayed on the through image (live view image) displayed on the viewfinder 10 so as to be displayed at a position corresponding to one of the plurality of focus detection points on the imaging surface of the viewfinder 10. It is superimposed. Further, among the range-finding point indices 4001 to 4180, the index that matches the position A, which is the current first estimated gazing point position, is highlighted in the finder 10. FIG.

Next, a line-of-sight detection method by the imaging device 1 will be described with reference to FIGS. 5 to 7. FIG.

FIG. 5 is a diagram for explaining the principle of the line-of-sight detection method, and is a schematic diagram of an optical system for performing line-of-sight detection.

In FIG. 5, the light sources 13a and 13b are light sources such as light-emitting diodes that emit infrared light that is imperceptible to the user. Light 14. Part of the illumination light emitted from the light sources 13 a and 13 b and reflected by the eyeball 14 is collected by the light receiving lens 16 onto the eye imaging device 17 .

FIG. 6A is a schematic diagram of an eye image captured by the eye image sensor 17 (an eye image projected on the eye image sensor 17), and FIG. FIG. 4 is a diagram showing output intensity of an element array;

FIG. 7 is a flowchart of line-of-sight detection processing. This process is executed by the CPU 3 reading out a program recorded in a ROM (not shown in FIG. 3).

In FIG. 7, when the line-of-sight detection process starts, the CPU 3 emits infrared light from the light sources 13a and 13b toward the eyeball 14 of the user in step S701. An image of the user's eye illuminated by the infrared light is formed on the eye imaging device 17 through the light receiving lens 16 and photoelectrically converted by the eye imaging device 17 . Thereby, an electric signal (eye image data) of a processable eye image is obtained.

In step S<b>702 , the CPU 3 acquires the eye image data obtained from the eye image sensor 17 as described above from the eye image sensor 17 .

In step S703, the CPU 3 detects coordinates corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image data obtained in step S702.

The infrared light emitted by the light sources 13a, 13b illuminates the cornea 142 of the eyeball 14 of the user. At this time, the corneal reflection images Pd and Pe formed by part of the infrared light reflected on the surface of the cornea 142 are condensed by the light receiving lens 16 and imaged on the eye imaging device 17, resulting in corneal reflection. Images Pd' and Pe' are obtained. Similarly, light beams from the ends a and b of the pupil 141 are also imaged on the eye imaging device 17 to form pupil end images a' and b'.

FIG. 6(b) shows luminance information (luminance distribution) of the region α in the eye image of FIG. 6(a). In FIG. 6(b), the horizontal direction of the eye image is the X axis and the vertical direction is the Y axis, and the luminance distribution in the X axis direction is shown. In this embodiment, the X-axis (horizontal) coordinates of the corneal reflection images Pd' and Pe' are Xd and Xe, and the X-axis coordinates of the pupil edge images a' and b' are Xa and Xb. As shown in FIG. 6B, extremely high levels of brightness are obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In a range larger than the coordinate Xa and smaller than the coordinate Xb, which corresponds to the area of the pupil 141 (the area of the pupil image 141' obtained by forming an image of the light flux from the pupil 141 on the eye imaging device 17), the coordinates Xd and Xe Extremely low levels of luminance are obtained except for On the other hand, in the area of the iris 143 outside the pupil 141 (the area of the iris image 143' outside the pupil image 141' obtained by forming an image of the light flux from the iris 143), the brightness is intermediate between the above two types. is obtained. Specifically, a luminance intermediate between the above two types of luminance is obtained in an area where the X coordinate (coordinate in the X-axis direction) is smaller than the coordinate Xa and in an area where the X coordinate is larger than the coordinate Xb.

From the luminance distribution shown in FIG. 6B, the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates with extremely low brightness can be obtained as the coordinates of the pupil edge images a' and b'. . Further, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16 is small, the pupil center image c′ ( The coordinate Xc of the center of the pupil image 141′ can be expressed as Xc≈(Xa+Xb)/2. That is, the X coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the X coordinates of the corneal reflection images Pd' and Pe' and the X coordinate of the pupil center image c' can be estimated.

Returning to FIG. 7, in step S704, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is a magnification determined by the position of the eyeball 14 with respect to the light receiving lens 16, and can be obtained as a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

In step S<b>705 , CPU 3 calculates the rotation angle of the optical axis of eyeball 14 with respect to the optical axis of light receiving lens 16 . The X coordinate of the midpoint between the corneal reflection image Pd and the corneal reflection image Pe and the X coordinate of the center of curvature O of the cornea 142 substantially match. Therefore, if the standard distance between the center of curvature O of the cornea 142 and the center c of the pupil 141 is Oc, then the rotation angle _θX of the eyeball 14 in the ZX plane (the plane perpendicular to the Y-axis) is given by: It can be calculated by the formula 1 below. The rotation angle θy of the eyeball 14 within the ZY plane (the plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating the rotation angle θx.

β×Oc×SINθ _X ≈{(Xd+Xe)/2}−Xc (Formula 1)
In step S<b>706 , the CPU 3 acquires correction coefficients (coefficient m and line-of-sight correction coefficients Ax, Bx, Ay, By) from the memory unit 4 . The coefficient m is a constant determined by the configuration of the finder optical system (light receiving lens 16, etc.) of the imaging device 1, and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c in the image for visual recognition. It is determined and stored in the memory section 4 . Further, the line-of-sight correction coefficients Ax, Bx, Ay, and By are parameters for correcting individual differences in eyeballs, are obtained by performing calibration work described later, and are stored in the memory unit 4 before the start of this process. ing.

In step S<b>707 , the CPU 3 instructs the line-of-sight detection circuit 201 to calculate the position of the user's viewpoint focused on the image for visual recognition displayed on the finder 10 (first estimated gazing point position). Specifically, the line-of-sight detection circuit 201 calculates the first estimated gazing point position using the rotation angles θx and θy of the eyeball 14 calculated in step S705 and the correction coefficient data obtained in step S706. Assuming that the coordinates (Hx, Hy) of the first estimated gazing point position are the coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the first estimated gazing point position are calculated by the following equations 2 and 3. can.

Hx=m×(Ax×θx+Bx) (Formula 2)
Hy=m×(Ay×θy+By) (Formula 3)
In step S708, the CPU 3 stores the coordinates (Hx, Hy) of the first estimated gazing point position calculated in step S706 in the memory unit 4, and ends this processing.

As described above, in the line-of-sight detection processing of this embodiment, the rotation angles θx and θy of the eyeball 14 and the correction coefficients (coefficient m and line-of-sight correction coefficients Ax, Bx, Ay, By) was used to calculate the first estimated gaze point position.

However, due to factors such as individual differences in the shape of human eyeballs, the first estimated gaze point position may not be estimated with high accuracy. Specifically, unless the values of the line-of-sight correction coefficients Ax, Ay, Bx, and By are adjusted to values suitable for the user, as shown in FIG. and position C, which is the first estimated gaze point position calculated in step S707. In FIG. 4B, the user is gazing at the person at position B, but the imaging device 1 incorrectly estimates that the user is gazing at the background at position C, which is the first estimated gazing point position. This results in a state in which appropriate focus detection and adjustment cannot be performed.

Therefore, the CPU 3 (calibration means) performs calibration work before the image pickup apparatus 1 performs image pickup (focus detection), acquires line-of-sight correction coefficients Ax, Ay, Bx, and By suitable for the user, and stores them in the memory unit. Store in 4.

Conventionally, calibration work is performed by highlighting a plurality of indices D1 to D5 at different positions as shown in FIG. there is Then, line-of-sight detection processing is performed when gazing at each visual target, and line-of-sight correction coefficients Ax, Ay, Bx, A technique for obtaining By is known as a known technique. As long as the position to be viewed by the user is suggested, the position may be emphasized by changing the brightness or color instead of displaying the index.

However, as described above, the accuracy of line-of-sight detection is degraded depending on the difference in conditions at the time of photographing and at the time of calibration. For example, this is the case when external light enters, or when the distance of the user's eyes looking into the finder 10 differs between when shooting and when performing calibration.

FIG. 8 is a diagram showing an example of eye image data at the time of calibration and photographing, which is determined to have low reliability by the line-of-sight detection reliability determination circuit 31. In FIG. FIG. 8A shows an example of eye image data acquired from the line-of-sight detection circuit 201 during calibration, and FIG. 8B shows an example of eye image data acquired from the line-of-sight detection circuit 201 during photographing. Here, in FIG. 8B, the position of the user's eyes is farther from the eyepiece window 6 during shooting than during calibration, and the size of the eyes detected from the eye image data is smaller. is shown. For example, when the image pickup apparatus 1 is set horizontally with respect to the optical axis direction for calibration, and then the image pickup apparatus 1 is turned downward with respect to the optical axis direction to photograph a flower blooming on the ground. Eye image data such as that shown in 8 may be obtained.

In such a case, the reliability of the first estimated gaze point position output from the line-of-sight detection reliability determination circuit 31 is low. A second estimated point-of-regard position is estimated by the reasoner. Here, CNN is an abbreviation for convolutional neural network, which is often used especially when performing image recognition.

In this embodiment, the CPU 3 performs calculations in the reasoner using CNN. A basic configuration of CNN will be described with reference to FIGS. 13 and 14. FIG.

FIG. 13 shows the basic configuration of a CNN for estimating the second estimated gazing point position from the eye image data output from the line-of-sight detection circuit 201 to the CPU 3. As shown in FIG.

In the flow of processing, the left end is the input and the processing proceeds to the right. The CNN has a set of two layers called a feature detection layer (S layer) and a feature integration layer (C layer), which are hierarchically configured.

In CNN, first, the following features are detected in the S layer based on the features detected in the previous layer. In addition, the features detected in the S layer are integrated in the C layer and sent to the next layer as the detection result in that layer.

The S layer consists of feature detection cell planes, and detects different features for each feature detection cell plane. Also, the C layer consists of a feature integration cell plane, and pools the detection results of the feature detection cell plane in the preceding stage. Hereinafter, the feature detection cell plane and the feature integration cell plane will be collectively referred to as feature planes when there is no particular need to distinguish them. In this embodiment, the output layer, which is the final stage layer, is composed only of the S layer without using the C layer.

Details of feature detection processing in the feature detection cell plane and feature integration processing in the feature integration cell plane will be described with reference to FIG. The feature detection cell surface is composed of a plurality of feature detection neurons, and the feature detection neurons are connected to the C layer of the prestage layer in a predetermined structure. The feature-integrating cell surface is composed of a plurality of feature-integrating neurons, and the feature-integrating neurons are connected to the S-layer of the same layer in a predetermined structure. In the _M - ^th cell plane of the L-th layer S layer shown in FIG. In the th cell plane, the output value of the feature integration neuron at the position (ξ, ζ) is expressed as y _M ^LC (ξ, ζ). At that time, if the coupling coefficients of the neurons are w _M ^LS (n, u, v) and w _M ^LC (u, v), each output value can be expressed as in Equations 4 and 5 below.

f in Equation 4 is an activation function, and may be any sigmoid function such as a logistic function or a hyperbolic tangent function. u _M ^LS (ξ, ζ) in Equation 4 is the internal state of the feature detection neuron at the position (ξ, ζ) on the M-th cell plane of the L-th layer S layer. On the other hand, in Equation 5, a simple linear sum is taken without using an activation function, so u _M ^LC (ξ, ζ) is equal to the output value y _M ^LC (ξ, ζ) calculated by Equation (5). Also, y _n ^L−1C (ξ+u, ζ+v) in Equation 4 and y _M ^LS (ξ+u, ζ+v) in Equation 5 are called the output value of the connection destination of the feature detection neuron and the output value of the connection destination of the feature integration neuron, respectively.

ξ, ζ, u, v, and n in Equations 4 and 5 will be explained.

Position (ξ, ζ) corresponds to position coordinates in the input image. For example, when the output value y _M ^LS (ξ, ζ) calculated by Equation 4 is a high output value, it is detected at the pixel position (ξ, ζ) of the input image in the L-th layer S-layer M-th cell plane. It means that the feature is likely to be present.

Also, n in Formula 4 means the n-th cell surface of the C layer of the L−1 layer, and is called the integration target feature number. Basically, sum-of-products operations are performed for all cell planes present in the L-1 layer C layer.

(u, v) are the relative position coordinates of the coupling coefficient, and the sum-of-products operation is performed in a finite range (u, v) according to the size of the feature to be detected. Such a finite range of (u, v) is called a receptive field. The size of the receptive field is hereinafter referred to as the size of the receptive field, and is represented by the number of horizontal pixels×the number of vertical pixels in the combined range.

Also, in Equation 4, when L=1, that is, in the first S layer, y _n ^L−1C (ξ+u, ζ+v) in Equation 4 becomes the input image y ^in_image (ξ+u, ζ+v). By the way, the distribution of neurons and pixels is discrete, and the connection destination feature number is also discrete, so ξ, ζ, u, v, and n are not continuous variables but discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which have a finite range.

w _M ^LS (n, u, v) in Equation 4 is a coupling coefficient distribution for detecting a predetermined feature, and by adjusting this to an appropriate value, it is possible to detect the predetermined feature. become. This adjustment of the coupling coefficient distribution is learning, and in constructing the CNN, various test patterns are presented so that the output value y _M ^LS (ξ, ζ) calculated by Equation 4 becomes an appropriate output value. Second, the coupling coefficient is adjusted by repeatedly and gradually correcting the coupling coefficient.

Next, w _M ^LC (u, v) in Equation 5 uses a two-dimensional Gaussian function and can be expressed as in Equation 6 below.

Again, since (u, v) has a finite range, the finite range is called the receptive field, and the size of the range is called the receptive field size, as in the description of the feature detection neuron. This receptive field size may be set to an appropriate value according to the size of the M-th feature of the L-th layer S layer. σ in Equation 6 is a feature size factor, which may be set to an appropriate constant according to the size of the receptive field. Specifically, it is preferable to set the value so that the outermost value of the receptive field can be regarded as approximately zero. The CNN of this embodiment is configured to estimate the second estimated point-of-regard position in the S layer, which is the final layer, by performing the above-described calculations in each layer.

Correct data (correct position) is important when creating an inference device that outputs a second estimated gaze point position as an inference result using eye image data of the user looking through the finder 10 as input data. how to define If the reliability of the first estimated gazing point position calculated by the line-of-sight detection reliability determination circuit 31 is high, the correct position may be used as the first estimated gazing point position. If the point-of-regard position is set as the correct position, the accuracy rate by learning does not increase. Therefore, in this embodiment, when the reliability of the first estimated gaze point position is low, other information obtained from the imaging device 1 is collected as correct data.

FIG. 9 is a flow chart of processing for collecting correct data used for learning when creating a reasoner. This process is executed by the CPU 3 reading out a program recorded in a ROM (not shown in FIG. 3).

In step S901, CPU 3 monitors whether the user is looking through finder 10 or not. This can be determined, for example, by whether or not the image data output from the line-of-sight detection circuit 201 is eye image data. The method is not particularly limited to this as long as it is possible to monitor whether or not the user is looking through the viewfinder 10. An optical sensor (not shown) provided around the eyepiece lens 12 is used to detect eye contact with the eyepiece lens 12. Presence/absence may be detected. If it is determined that the user is looking through the finder 10, the process proceeds to step S902.

In step S<b>902 , the CPU 3 calculates the first estimated gazing point position from the eye image data output from the line-of-sight detection circuit 201 and acquires the reliability output from the line-of-sight detection reliability determination circuit 31 . After that, the process proceeds to step S903.

In step S903, when the reliability output from the line-of-sight detection reliability determination circuit 31 is high, the CPU 3 proceeds to step S904. If the reliability is low, the process proceeds to step S905.

In step S904, CPU 3 completes this process after collecting the first estimated gazing point position as the correct position.

In step S905, CPU 3 determines whether or not there is a subject near the first estimated gazing point position. In this determination, a person may be detected as a subject, or eyes may be detected. As a result of this determination, if the subject is detected near the first estimated gazing point position, the process proceeds to step S906. Otherwise, the process proceeds to step S907.

In step S906, CPU 3 (collecting means) collects the coordinates of the subject near the first estimated gaze point position as the correct position, and then terminates this process.

In step S907, the CPU 3 (display control means) moves the first estimated gazing point position (hereinafter referred to as position C in FIG. 4B in this process) to another position (hereinafter referred to as 4(b), position B) is highlighted in the finder 10 so as to be movable. Here, the first user operation refers to manual operation by the user using any one of the operation members 41-43. After that, after the position C highlighted in the finder 10 has been moved to the position B by the first user operation, the CPU 3 determines whether an image is captured with the position B as the focus position by the imaging device 1 (release button 5 is pressed (second user operation)). Only when such shooting has been performed, the process proceeds to step S908. Note that the second user operation may be a user operation for determining another position on the finder 10 selected by the user as the focus position, and may be a user operation other than pressing the release button 5 .

In step S908, the CPU 3 (collecting means) collects the coordinates of the focus position (other position) of the captured image as the correct position, and then terminates this process.

The CPU 3 transmits the eye image data (input data) collected in the process of FIG. 9 and the correct position (correct data) at that time to a server (not shown) using the communication circuit 32 via a network such as a LAN or the Internet. to the PC of The PC on the server performs CNN machine learning using these data, and transmits a “reasoner” generated as a learning result to the imaging device 1 . The imaging device 1 may have a high-performance GPU, and the GPU (or cooperation with the CPU 3) may perform the machine learning of the CNN.

Next, a description will be given of how to use an inference device generated by performing CNN machine learning on a PC on the server.

FIG. 10 is a flowchart of focus processing during shooting. This process is executed by the CPU 3 reading out a program recorded in a ROM (not shown in FIG. 3).

In FIG. 10, the CPU 3 first performs steps S901 to S903. Since these processes have already been described in the description of FIG. 9, redundant description will be omitted.

In step S903, CPU 3 proceeds to step S1004 if the reliability of the first estimated gaze point position is high, and proceeds to step S1005 if the reliability is low.

In step S1004, CPU 3 (first focusing means) determines that the first estimated position of the point of gaze is the focus position desired by the user, and performs focusing based on the first estimated position of the point of gaze. This processing is realized by operating the automatic focus detection circuit 203 and the focus adjustment circuit 118 according to instructions from the CPU 3 . Specifically, first, the automatic focus detection circuit 203 calculates the distance to the subject corresponding to the focus detection point that matches the first estimated gazing point position. After that, the focus adjustment circuit 118 drives the lens driving motor 113 by a predetermined amount based on this information to move the photographing lens 1A to the in-focus position. After that, this process is terminated.

In step S1005, the CPU 3 inputs the eye image data output from the line-of-sight detection circuit 201 in step S902 to the inference unit transmitted from the PC on the server, and estimates the second estimated gazing point position. In this embodiment, the CPU 3 estimates the second estimated gaze point position, but the present invention is not limited to this. For example, a PC on the server may estimate the second estimated point-of-regard position. In this case, the CPU 3 transmits the eye image data output from the line-of-sight detection circuit 201 in step S902 to the PC on the server, and the PC on the server uses the inference device to estimate the second estimated gaze point position. , outputs the inference result to the CPU 3 of the imaging device 1 . After that, the process proceeds to step S1006.

In step S1006, when CPU 3 determines that the reliability of the second estimated gaze point position estimated in step S1005 is high, the process proceeds to step S1007. If it is determined that the reliability is low, the process proceeds to step S1008. In the inference unit of this embodiment, the likelihood is calculated for each of the 180 focus detection points. Therefore, the focus detection point for which the highest likelihood is calculated is set as the second estimated gazing point position. Also, when the highest likelihood value is equal to or greater than the threshold, it is determined that the reliability of the second estimated point-of-regard position is high.

In step S1007, CPU 3 (second focus means) determines that the second estimated position of the point of interest is the focus position desired by the user, and performs focus based on the second estimated position of the point of interest. This processing is realized by operating the automatic focus detection circuit 203 and the focus adjustment circuit 118 according to instructions from the CPU 3 . After that, this process is terminated.

In step S1008, the CPU 3 determines which of the first estimated gazing point position and the second estimated gazing point position is more reliable. If it is determined that the reliability of the first estimated point-of-regard position is higher, the process proceeds to step S1009. If it is determined that the reliability of the second estimated point-of-regard position is higher, the process proceeds to step S1010. move on.

In step S1009, the CPU 3 determines that the focus position desired by the user is near the first estimated position of the point of gaze, detects a subject near the first estimated position of the point of gaze, and places the detected subject as the focus point. Focus as After that, this process is terminated.

In step S1010, CPU 3 determines that the focus position desired by the user is in the vicinity of the second estimated position of the point of gaze, performs subject detection in the vicinity of the second estimated position of the point of gaze, and places the detected subject as the focus point. Focus as After that, this process is terminated.

In addition, even if the machine learning of CNN on the PC on the server progresses and the reliability of the first estimated gaze point position is high, the reliability of the second estimated gaze point position estimated by the inference device is equal to or higher than this. In this case, the inference unit may always be used for focus processing during shooting. Specifically, steps S902, S903, S1004, S1008, and S1009 in FIG. 10 are not required, and if YES in step S901, the process proceeds directly to step S1005, and if NO in step S1006, the process proceeds directly to step S1010. Thereby, the focus position is determined based only on the second estimated gaze point position.

In this embodiment, a case has been described in which the user's eyes looking through the viewfinder 10 at the time of photographing are positioned farther from the eyepiece window 6 than at the time of calibration, but the present invention is not limited to this. In other words, this embodiment can be applied when focus processing is performed under various conditions in which the reliability output by the line-of-sight detection reliability determination circuit 31 is lowered.

As described above, in this embodiment, by defining the correct position using the information of the imaging device 1 at the time of learning, significant learning can be performed even when the accuracy of line-of-sight detection is degraded. Then, by switching the line-of-sight detection circuit 201 and the inference device according to the reliability of the first and second estimated gaze point positions, the accuracy of line-of-sight detection can be improved even in shooting situations where there is a difference from the time of calibration. can be improved.

(Example 2)
11 and 12, a method of collecting input data during optimal learning when the reliability of line-of-sight detection is low under various conditions will be described according to the second embodiment of the present invention. In addition, in the present embodiment, the same numbers are assigned to the same configurations as in the first embodiment, and redundant explanations are omitted.

As described in the first embodiment, the decrease in the reliability of the first estimated gaze point position becomes conspicuous when eye information differs between calibration and photographing. However, there are a number of possible factors for this, such as the difference in the distance between the user's eye looking through the viewfinder 10 and the eyepiece lens 12, and the difference in light entering. Therefore, in this embodiment, the CPU 3 (reliability reduction factor determination means) determines such multiple reliability reduction factors, and collects input data during learning of the optimal neural network according to each factor.

FIG. 11 is a flowchart of the process of determining the reliability reduction factor of the first estimated gazing point position according to the present embodiment. This process is executed by the CPU 3 reading out a program recorded in a ROM (not shown in FIG. 3).

In FIG. 11, first, when the imaging apparatus 1 takes an image, the CPU 3 performs steps S901 to S903. Since these processes have already been described in the description of FIG. 9, redundant description will be omitted.

In step S903, if the reliability of the first estimated gazing point position is low, the CPU 3 proceeds to step S1101, and if the reliability is high, the processing ends.

In step S1101, the CPU 3 determines whether or not the size of the eye is different from that at the time of calibration, that is, whether or not the distance between the user's eye looking through the finder 10 and the eyepiece 12 is different and the reliability is lowered. do. The distance between the eyepiece 12 and the user's eyes can be calculated based on the size of the current eye size relative to the eye size at the time of calibration. For example, when the pupil is detected by the line-of-sight detection method as shown in FIGS. It is possible to guess how far away from the time of calibration. It should be noted that the method is not particularly limited to this as long as it can determine whether or not the size of the eye is different from that at the time of calibration. The distance between 12 and the user's eyes may be calculated. If it is determined to be the same as during calibration, the process advances to step S1103, and if it is determined to be different from that during calibration, the process advances to step S1102.

In step S1102, the CPU 3 stores information indicating that the distance between the viewfinder 10 and the photographer (user's eyes) is different in the memory unit 4 as a reliability lowering factor. After that, the process proceeds to step S1103.

In step S1103, CPU 3 determines whether or not the luminance is equal to or higher than a predetermined value. This can be determined by confirming the brightness of the eye image data of the user looking through the viewfinder 10 acquired by the eye imaging device 17 . If the brightness is equal to or higher than the predetermined value, it is determined that external light has entered, and the process proceeds to step S1104. If the brightness is less than the predetermined value, it is determined that no outside light has entered, and this processing ends.

In step S1104, the CPU 3 stores in the memory unit 4 information indicating that external light enters as a reliability lowering factor. In addition, difference eye image data of FIG. 12C, which will be described later, is generated as input data. After that, this process is terminated.

In general, learning using a large amount of input data is necessary to improve the accuracy of inference by an inference machine. Therefore, when the amount of input data is small, CNN often uses a technique called padding, in which the original input data is transformed to increase the amount of data. There are various methods for this padding, such as increasing noise, scaling the image, masking parts, and inverting the image. can make things worse. This is because the padding data during learning may be data of poor quality such as data that cannot actually exist or data that cannot be inferred by an inference device. In this embodiment, it is possible to perform appropriate padding according to the acquired original input data by storing the factor of the decrease in reliability according to the process of FIG. 11 .

The CPU 3 uses the communication circuit 32 to transmit the eye image data collected in the process of FIG. 9, the correct position (correct data) at that time, and the reliability reduction factor acquired in the process of FIG. to the PC on the server via The PC on the server performs CNN machine learning using these data, and transmits a “reasoner” generated as a learning result to the imaging device 1 .

When the PC on the server receives the information that the distance between the viewfinder 10 and the photographer is different as a reliability lowering factor, the eye image data (original input data) at the time of imaging is transmitted from the imaging device 1. Then, the expanded/reduced data is used as the padding data at the time of learning. More specifically, the PC on the server also acquires the eye image data during calibration from the imaging device 1, and detects the size of the pupil diameter from each of the eye image data during calibration and during photography. Inflated data is created by enlarging or reducing the original input data so that the size of the pupil diameter detected from the inflated data is within the range of these detected pupil diameter sizes. By creating the inflated data in this way, it is possible to suppress the generation of inflated data of poor quality and perform optimal inflation. In addition, it is possible to prevent the generation of inflated data, which is impossible in reality, such as masking a part of the user's pupil image.

On the other hand, if the reliability is lowered due to the entry of external light, depending on the external light conditions, the upper part of the eye image captured by the eye image sensor 17 may be crushed white, and the lower part may be crushed white. . The former is caused, for example, by sunlight entering during shooting outside in the daytime, and the latter is caused by the reflection of sunlight from snow on a ski resort. In addition, sunlight may enter from the side of the eyepiece 12 depending on the shooting posture of the imaging device 1 . That is, the portion of the eye image where the white crushing occurs varies depending on the external light conditions.

Even if the eye image data in which such white saturation occurs is used as input data for learning, the accuracy of inference by the inference device does not easily improve if the external light conditions are different. Therefore, in this embodiment, in such a case, differential eye image data generated by the method shown in FIG. 12 is collected as input data.

The eye image data of FIG. 12(a) is the eye image data output from the eye image pickup device 17, and external light 1200 enters the upper portion thereof. At the time of this imaging, the light sources 13a and 13b are turned on, and corneal reflection images 1201a to 1201c are formed on the user's cornea (eyeball).

The eye image data in FIG. 12(b) is the eye image data output by the eye image pickup element 17, and external light 1200 enters the upper portion thereof, as in FIG. 12(a). However, at the time of this imaging, the light sources 13a and 13b are turned off, and the corneal reflection images 1201a to 1201c are not formed on the user's cornea (eyeball). The eye image shown in FIG. 12(b) is slightly darker overall than the eye image shown in FIG. 12(a) because the light sources 13a and 13b are turned off.

The difference eye image data of FIG. 12(c) is difference data obtained by taking the difference between FIG. 12(a) and FIG. 12(b), and the outside light 1200 entering the upper part of the eye image is removed.

In this way, when the reliability lowering factor is the information that external light enters, the PC on the server receives as input data eye image data that forms a corneal reflection image as shown in FIG. Instead, it receives differential eye image data as shown in FIG. 12(c). Therefore, it is possible to perform learning in which the external light conditions are separated to some extent. However, since the difference eye image data of FIG. 12C is an image obtained by taking a difference with respect to strong external light, the edge around the pupil is blurred and the detection accuracy of sight line detection is deteriorated. The PC on the server performs CNN machine learning using the differential eye image data and the inflated data created based on it, so that it is possible to create a reasoner that is superior to the edge blur around the pupil. Inference is possible even if is changed. In this case, data obtained by masking a portion of the pupillary boundary of the differential eye image data transmitted from the imaging device 1 or data obtained by increasing the noise around the pupil are created and used as inflated data during learning.

As described above, in the present embodiment, when the reliability of the first estimated gazing point position is lowered in the collection of input data during learning, the CPU 3 puts a flag indicating the factor of the lowered reliability in the input data. to the PC on the server. As a result, the PC on the server can create appropriate inflated data based on the input data, and can improve the accuracy of the reasoner with a small number of times of learning.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

1 imaging device 3 CPU
REFERENCE SIGNS LIST 10 finder 13a-13b light source 17 eye imaging element 31 line-of-sight detection reliability determination circuit 41-43 operating member 118 focus adjustment circuit 201 line-of-sight detection circuit 203 automatic focus detection circuit

Claims (13)

An imaging device that displays a through image in an internal viewfinder,
generating means for generating eye image data by capturing an eyeball of the user looking through the viewfinder;
sight line detection means for obtaining the eye image data and detecting the position of the user's sight line focused on the through image of the viewfinder based on the obtained eye image data;
display control means for displaying the detected line-of-sight position on the finder so as to be movable to another position by a first user operation;
collecting means for collecting the other position as a correct position when there is a second user operation to determine the other position as the focus position;
The imaging apparatus, wherein the correct position is used for learning to create a reasoner for estimating the line-of-sight position using the eye image data acquired by the line-of-sight detection means as input data. 2. The imaging apparatus according to claim 1, wherein, when a subject is present near the detected line-of-sight position, the collection means collects the position of the subject as the correct position. Reliability determination means for determining reliability of the detected line-of-sight position;
when the reliability is high, a first focusing means for focusing the imaging device using the detected line-of-sight position as the focus position;
3. The imaging according to claim 1, further comprising second focusing means for focusing the imaging device using the line-of-sight position estimated by the inference unit as the focus position when the reliability is low. Device. a calibration means for obtaining the eye image data before the sight line position is detected by the sight line detection means, and correcting individual differences in eyeballs based on the obtained eye image data;
The reliability determination means determines the reliability based on a difference between two eye image data, the eye image data acquired by the calibration means and the eye image data acquired by the line-of-sight detection means. 4. The imaging apparatus according to claim 3, characterized by: 5. The imaging apparatus according to claim 4, wherein the difference is a difference in pupil diameter detected from each of the two eye image data. 6. The imaging apparatus according to claim 5, wherein the difference is a difference in external light detected from each of the two eye image data. further comprising a light source for illuminating the eye of the user;
7. The imaging according to claim 6, wherein the difference is a difference in the number of corneal reflection images formed on the eyeball of the user by lighting of the light source, which is detected from each of the two eye image data. Device. When the reliability determination means determines that the reliability has decreased due to the difference in the external light, the eye image data of the eyeball of the user in which the corneal reflection image is formed by turning on the light source, and the light source. The learning is performed by using, as the input data instead of the eye image data acquired by the line-of-sight detecting means, difference data between the eye image data of the eyeball of the user in which the corneal reflection image is not formed due to the light being turned off. 8. The imaging apparatus of claim 7, wherein: further comprising reliability reduction factor determination means for determining a reliability reduction factor when the reliability determination means determines that the reliability is low,
9. The image pickup apparatus according to claim 8, wherein padded data corresponding to said reliability lowering factor is created during said learning. If the reliability lowering factor is that the distance between the viewfinder and the user is different, the size of the pupil diameter is detected from each of the two eye image data, and the size of the detected pupil diameter is calculated. 10. The imaging apparatus according to claim 9, wherein said input data is enlarged/reduced so that the size of the pupil diameter detected from said inflated data falls within a range to create said inflated data. The padding is performed by masking a portion of the pupillary boundary of the differential data as the input data or creating data with increased noise around the pupil when the reliability lowering factor is the intrusion of external light. 10. The image pickup apparatus according to claim 9, wherein the image pickup apparatus is performed by: A control method for an imaging device that displays a through image in an internal finder, comprising:
a generation step of capturing an eyeball of a user looking through the finder and generating eye image data;
a line-of-sight detection step of acquiring the eye image data and detecting a position of the user's line of sight focused on the through image of the finder based on the acquired eye image data;
a display control step of displaying the detected line-of-sight position on the finder so as to be movable to another position by a first user operation;
a collecting step of collecting the other position as a correct position when there is a second user operation to determine the other position as the focus position;
The control method, wherein the correct position is used for learning for creating an inference device for estimating the line-of-sight position using the eye image data acquired in the line-of-sight detection step as input data. A computer-executable program that causes a computer to function as each means of the imaging apparatus according to any one of claims 1 to 11.

Images