JP2022066266A - Subject recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automate a gaze position calibration operation and to improve convenience for a user while accurately recognizing a subject in a subject recognition device having a line-of-sight detection function.

SOLUTION: A subject recognition device includes: imaging means; display means; line-of-sight position detecting means of a photographer; optical flow calculation means; line-of-sight movement vector measuring means for measuring a movement amount and a movement direction in which the line-of-sight position of the photographer moves at predetermined time; movement vector coincidence degree determination means for comparing an optical flow with the line-of-sight movement vector and determining a degree of coincidence between the optical flow and the line-of-sight movement vector; and subject area determination means for extracting an image area having the optical flow and determining the area to be a subject area when the determination means determines that the two match. A detection result of the line-of-sight position is corrected by using position coordinates of the determined subject area.

SELECTED DRAWING: Figure 9

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a subject recognition device having a line-of-sight detection device.

In recent years, the automation and intelligentization of cameras have progressed, and as shown in Patent Document 1, the photographer's intended subject is recognized based on the information of the photographer's line-of-sight position looking into the finder without manually inputting the subject position. However, a camera control device capable of performing focus control has been proposed.

Further, when the camera detects the line-of-sight position of the photographer as described above, a deviation occurs between the line-of-sight position intended by the photographer and the line-of-sight position of the photographer recognized by the camera, and the subject intended by the photographer. It may not be possible to focus.

On the other hand, in Patent Document 1, an index is displayed in the finder before shooting, an instruction is given to the user to gaze at the index, the line-of-sight position of the user is detected in the gaze state, and the index position is obtained. Work to detect the amount of deviation from. A calibration technique that makes it possible to detect the line-of-sight position as intended by the user by correcting the line-of-sight detection position of the photographer recognized by the camera at the time of shooting by the detected deviation amount is described. ..

Further, in Patent Document 2, when the vehicle is driven, the movement of the driver's gaze point position within a unit time is detected, and the movement direction / amount thereof is the movement direction / movement amount of the external landscape seen from the driver's seat. It shows an inattentive driving detection technique that determines that the vehicle is not looking ahead when it matches for a certain period of time or longer.

Japanese Unexamined Patent Publication No. 2004-8323 Japanese Unexamined Patent Publication No. 2010-39933

In Patent Document 1, when there is a deviation between the gazing point position intended by the photographer and the gazing point position detected by the camera, the correction is performed by calibration.

However, if the surrounding environment such as brightness or the posture of the photographer changes, the gazing point position detected by the camera changes, so calibration work must be performed frequently every time the situation changes, which is complicated. There's a problem.

Further, in Patent Document 2, paying attention to the moving direction and amount of the gazing point per predetermined time, the user is in a state of gazing at the front and driving, or a view in which the line of sight flows from the front to the side. It is judged whether it is in the state of inattentive driving that moves following.

However, there is no mention of detecting the deviation between the gaze point position intended by the user and the gaze point position of the user recognized by the camera, and the calibration work for correcting the deviation. Therefore, there is a problem that the deviation of the gazing point position remains unresolved and the camera cannot accurately recognize the subject intended by the user.

The present invention has been made in view of the above problems, and in a subject recognition device having a line-of-sight detection device, the subject to be gazed by the photographer is accurately recognized, and the calibration operation of the gaze position is automated. The purpose is to improve the convenience of the photographer.

In order to achieve the above object, the subject recognition device according to the present invention is
Imaging means and
Display means for displaying captured images and
A line-of-sight position detecting means for detecting the line-of-sight position of the photographer,
An optical flow calculation means that calculates an optical flow based on an image acquired by the image pickup means, and an optical flow calculation means.
A line-of-sight movement vector measuring means for measuring a movement amount and a movement direction in which the photographer's line-of-sight position measured by the line-of-sight position detecting means moves at a predetermined time.
A movement vector matching degree determining means that compares the optical flow calculated by the optical flow calculating means with the line-of-sight movement vector acquired by the line-of-sight movement vector acquiring means and determines the degree of coincidence between the two.
When the determination means determines that the two match a predetermined degree, the image region having the optical flow is extracted, and the subject region specifying means for identifying the region as the subject region is used.
By using the position coordinates of the specified subject area, the line-of-sight position detection result correction means for correcting the line-of-sight position detection result, and the line-of-sight position detection result correction means.
It is characterized by having.

According to the present invention, in a subject recognition device having a line-of-sight detection device, it is possible to accurately recognize the subject to be gazed by the photographer and to automate the calibration operation of the gaze position to improve the convenience of the photographer. Can be done.

Schematic diagram of the configuration of the image pickup apparatus to which the first embodiment of the present invention is applied. Block diagram of an image pickup apparatus to which the first embodiment of the present invention is applied. Explanatory drawing which shows the field of view in the finder in 1st Embodiment of this invention Principle explanatory diagram of the line-of-sight detection method in the first embodiment of the present invention Schematic diagram of the eyeball image projected on the image sensor 17 for the eyeball Outline flow routine for line-of-sight detection Subject detection operation explanatory diagram Optical flow acquisition method explanatory diagram Flow of shooting operation / automatic calibration operation in the first embodiment of the present invention

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

<Explanation of configuration>
FIG. 1 is a schematic view of a digital still camera according to the present invention.

Reference numeral 1 is a photographing lens, and in the present embodiment, two lenses 1a and 1b are shown for convenience, but it is well known that the lens is actually composed of a larger number of lenses. Reference numeral 2 is an image pickup device, which is arranged on the planned image plane of the photographing lens 1 of the digital still camera 1. The digital still camera 1 includes a CPU 3 that controls the entire camera and a memory unit 4 that records an image captured by the image sensor 2. Further, a display element 10 composed of a liquid crystal display or the like for displaying an captured image, a display element drive circuit 11 for driving the display element, and an eyepiece 12 for observing a subject image displayed on the display element 10 are provided. Have been placed.

13a to 13b are light sources for illuminating the photographer's eyeball 14 for detecting the line-of-sight direction from the relationship between the reflected elephant and the pupil due to the corneal reflex of the light source conventionally used for single-lens reflex cameras and the like, and infrared rays. It consists of a light emitting diode and is arranged around the eyepiece 12. The illuminated eyeball image and the image due to the corneal reflex of the light sources 13a to 13b pass through the eyepiece lens 12, are reflected by the light divider 15, and are used for the eyeball in which the photoelectric element train of the CCD building is two-dimensionally arranged by the light receiving lens 16. An image is formed on the image pickup device 17. The light receiving lens 16 positions the pupil of the photographer's eyeball 14 and the image sensor 17 for the eyeball in a conjugate imaging relationship. The line-of-sight direction is detected by a predetermined algorithm described later from the positional relationship between the eyeball formed on the image sensor 17 for the eyeball and the image due to the corneal reflection of the light sources 13a to 13b.

111 is a diaphragm provided in the photographing lens 1, 112 is a diaphragm drive device, 113 is a lens drive motor, 114 is a lens drive member including a drive gear, and 115 is a photocoupler, which is a pulse plate 116 linked to the lens drive member 114. The rotation of the lens is detected and transmitted to the lens focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 by a predetermined amount based on this information and the information on the lens drive amount from the camera side, and moves the photographing lens 1a to the in-focus position. Reference numeral 117 is a mount contact that serves as an interface between a known camera and a lens.

FIG. 2 is a block diagram showing an electrical configuration built into the digital still camera of the previous term configuration, and the same ones as those in FIG. 1 are numbered the same.

The central processing unit 3 (hereinafter referred to as CPU 3) of the microcomputer built in the camera body includes a line-of-sight detection circuit 201, a photometric circuit 202, an automatic focus detection circuit 203, a signal input circuit 204, a display element drive circuit 11, and an illumination light source. The drive circuit 205 is connected. Further, a signal is transmitted to the focus adjustment circuit 118 arranged in the photographing lens and the aperture control circuit 206 included in the above-mentioned aperture drive device 112 via the mount contact 117 shown in FIG. The memory unit 4 attached to the CPU 3 has a function of storing an image pickup signal from the image pickup element 2 and the image pickup element 17 for an eyeball, and a function of storing line-of-sight correction data for correcting individual differences in the line of sight, which will be described later.

The line-of-sight detection circuit 201 A / D-converts the output obtained by forming an eyeball image from the eyeball image sensor 17 (CCD-EYE), and transmits this image information to the CPU 3. The CPU 3 extracts each feature point of the eyeball image required for line-of-sight detection according to a predetermined algorithm described later, and further calculates the line-of-sight of the photographer from the position of each feature point.

The photometric circuit 202 amplifies the luminance signal output corresponding to the brightness of the image field based on the signal obtained from the image sensor 2 that also serves as a photometric sensor, and then performs logarithmic compression and A / D conversion to obtain the image field. It is sent to the CPU 3 as brightness information.

The automatic focus detection circuit 203 A / D-converts signal voltages from a plurality of pixels used for phase difference detection included in the CCD in the image pickup device 2 and sends them to the CPU 3.

The CPU 3 calculates the distance from the signals of the plurality of pixels to the subject corresponding to each focus detection point. This is a known technique known as image plane phase detection AF. In this embodiment, as an example, it is assumed that there are 180 focus detection points at positions on the imaging surface corresponding to the locations shown in the field image in the finder of FIG.

The signal input circuit 204 is turned on by the first stroke of the release button (not shown), and is turned on by the switch ₁ for starting the camera's metering, distance measurement, line-of-sight detection operation, etc., and the second stroke of the release button. Then, SW2, which is a switch for starting the release operation, is connected. The signal is input to the signal input circuit 204 and transmitted to the CPU 3.

FIG. 3 is a view showing the inside of the finder field of view, and shows a state in which the display device 10 is operated.

In FIG. 3, 300 is a field mask, 400 is a focal point detection region, and 4001 to 4180 are 180 images superimposed on a through image shown on the display element 10 at positions corresponding to a plurality of focal detection points on the imaging surface. Indicates a range-finding point optotype. Further, among those indexes, the index corresponding to the current focusing position is highlighted as in the focusing point A in the figure.

FIG. 4 is an explanatory diagram of the principle of the line-of-sight detection method, and corresponds to a schematic diagram of an optical system for performing line-of-sight detection in FIG. 1 described above.

In FIG. 4, 13a and 13b are light sources such as light emitting diodes that emit infrared light insensitive to the observer, and each light source is arranged substantially symmetrically with respect to the optical axis of the light receiving lens 16 and is the eyeball of the observer. It illuminates 14. A part of the illumination light reflected by the eyeball 14 is focused on the image pickup element 17 for the eyeball by the light receiving lens 16.

FIG. 5A is a schematic view of an eyeball image projected on the eyeball image sensor 17, and FIG. 5B is an output intensity diagram of the CCD in the eyeball image sensor 17. FIG. 6 shows a schematic flow routine for line-of-sight detection.

Hereinafter, the line-of-sight detection means will be described with reference to FIGS. 4 to 6.

<Explanation of line-of-sight detection operation>
In FIG. 6, when the line-of-sight detection routine is started, in S001, the light sources 13a and 13b radiate infrared light toward the observer's eyeball 14. The observer's eyeball image illuminated by the infrared light is imaged through a light receiving lens 16 on the eyeball image sensor 17, photoelectric conversion is performed by the eyeball image sensor 17, and the eyeball image can be processed as an electric signal. It becomes.

In step S002, the eyeball image signal obtained from the eyeball image sensor 17 as described above is sent to the CPU 3.

In step S003, the coordinates of the points corresponding to the corneal reflex images Pd, Pe and the pupil center c of the light sources 13a and 13b shown in FIG. 4 are obtained from the information of the eyeball image signal obtained in S002. The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the observer's eyeball 14, and the corneal reflex images Pd and Pe formed by a part of the infrared light reflected on the surface of the cornea 142 at this time are The light is collected by the light receiving lens 16 and imaged on the image pickup element 17 for the eyeball (points Pd', Pe'in the figure). Similarly, the luminous flux from the ends a and b of the pupil 141 is also imaged on the image sensor 17 for the eyeball.

5A shows an example of an image of a reflected image obtained from the image sensor 17 for an eyeball in FIG. 5A, and FIG. 5B shows an example of luminance information obtained from the image sensor 17 for an eyeball in the region α of the image example. Is shown.

As shown in the figure, the horizontal direction is the X axis and the vertical direction is the Y axis. At this time, the coordinates in the X-axis direction (horizontal direction) of the images Pd ′ and Pe ′ in which the corneal reflection images of the light sources 13a and 13b are formed are defined as Xd and Xe. Further, let Xa and Xb be the coordinates in the X-axis direction of the images a'and b'in which the light fluxes from the ends a and b of the pupil 14b are imaged. In the luminance information example of (b), an extremely strong level of luminance is obtained at the positions Xd and Xe corresponding to the images Pd'and Pe'in which the corneal reflection images of the light sources 13a and 13b are formed. The region between the coordinates Xa and Xb, which corresponds to the region of the pupil 141, has an extremely low level of brightness except for the positions of Xd and Xe.

On the other hand, in the region having the X coordinate value lower than Xa and the region having the X coordinate value higher than Xb, which corresponds to the region of the iris 143 outside the pupil 141, the value between the two luminance levels is intermediate. Is obtained. From the variation information of the luminance level with respect to the X coordinate position, the X coordinates Xd and Xe of the images Pd'and Pe'imaged by the corneal reflex images of the light sources 13a and 13b and the X coordinates of the pupil end images a'and b'. Xa and Xb can be obtained. Further, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16 is small, the coordinates Xc of the portion (referred to as c') corresponding to the pupil center c imaged on the image pickup device 17 for the eyeball are set. It can be expressed as Xc≈ (Xa + Xb) / 2. From the above, it was possible to estimate the X coordinate of c'corresponding to the center of the pupil imaged on the image sensor 17 for the eyeball and the coordinates of the corneal reflection images Pd'and Pe'of the light sources 13a and 13b.

Further, in step S004, the image magnification β of the eyeball image is calculated. β is a magnification determined by the position of the eyeball 14 with respect to the light receiving lens 16, and can be substantially obtained as a function of the distance (Xd—Xe) between the corneal reflex images Pd ′ and Pe ′.

Further, in step S005, since the X coordinate of the midpoint of the corneal reflection images Pd and Pe and the X coordinate of the curvature center O of the cornea 142 substantially match, the standard up to the curvature center O of the cornea 142 and the center c of the pupil 141. The angle of rotation θ _X in the ZX plane of the optical axis of the eyeball 14 is β * Oc * SINθ _X ≈ {(Xd + Xe) / 2} -Xc.
It can be obtained from the relational expression of. Further, FIGS. 4 and 5 show an example of calculating the angle of rotation θ _X when the observer's eyeball rotates in a plane perpendicular to the Y axis, but the observer's eyeball is perpendicular to the X axis. The method for calculating the angle of rotation θy when rotating in a plane is the same.

When the rotation angles θx and θy of the optical axis of the observer's eyeball 14 are calculated in the previous step, in step S006, the position of the observer's line of sight (gaze) on the display element 10 using θx and θy. The position of the point. Hereinafter referred to as the gazing point) is obtained. Assuming that the gazing point position is the coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display element 10.
Hx = m × (Ax × θx + Bx)
Hy = m × (Ay × θy + By)
Can be calculated. At this time, the coefficient m is a constant determined by the configuration of the finder optical system of the camera, and is a conversion coefficient for converting the rotation angles θx and θy into the position coordinates corresponding to the center c of the pupil 141 on the display element 10, and is determined in advance. It is assumed that the coefficient is stored in the memory unit 4. Further, Ax, Bx, Ay, and By are line-of-sight correction coefficients that correct individual differences in the line of sight of the observer, and are acquired by performing calibration work described later, and are stored in the memory unit 4 before the line-of-sight detection routine starts. It shall be remembered.

After calculating the coordinates (Hx, Hy) of the center c of the pupil 141 on the display element 10 as described above, the coordinates are stored in the memory unit 4 in step S007, and the line-of-sight detection routine is completed.

The above shows a method for acquiring gaze point coordinates on a display element using the corneal reflection images of the light sources 13a and 13b, but the method is not limited to this, and any method for acquiring the eyeball rotation angle from the captured eyeball image. For example, the present invention is applicable. The line-of-sight detection routine corresponds to the line-of-sight position detecting means according to claim 1.

<Calibration work>
As described above, the gaze point position is estimated by acquiring the rotation angles θx and θy of the eyeball from the eyeball image in the line-of-sight detection routine and performing a coordinate conversion of the pupil center position to the corresponding position on the display element 10. There is.

However, as shown in FIG. 3 (b), the values of the line-of-sight correction coefficients Ax, Ay, Bx, and By must be adjusted to appropriate values by the user due to factors such as individual differences in the shape of the human eyeball. , The position B that the user is actually gazing at and the calculated position of the estimated gazing point C are displaced from each other. In the above example, although the person at position B wants to be watched, the camera mistakenly estimates that the background is being watched, and the focus cannot be detected and adjusted appropriately.

Therefore, it is necessary to perform calibration work, acquire an appropriate correction coefficient value for the user, and store it in the camera before taking an image with the camera.

Conventionally, the calibration work is performed by highlighting a plurality of indexes having different positions as shown in FIG. 3C in the field of view of the finder before imaging and having the observer see the indexes. It is known as a known technique to perform a gazing point detection flow at the time of gazing at each optotype and to obtain an appropriate value of the coefficient from a plurality of estimated gazing point coordinates calculated and the position of each optotype coordinate. ..

However, if the surrounding environment such as brightness or the posture of the photographer changes, the position of the gazing point detected by the camera changes, so it is necessary to stop imaging and perform calibration work frequently every time the situation changes. There was a problem that it could not be maintained and became complicated. Therefore, there is a need for a system that automatically calibrates and ensures accuracy without interfering with the user's imaging operation.

<Automatic calibration system>
In order to solve the above problems, in the present invention, by incorporating the calibration work into the shooting operation and automatically performing the calibration, it is not necessary to stop the imaging, and the user's convenience is improved. The principle is explained below.

In the calibration work, the most important point is to identify which part on the display element 10 the user is actually looking at. Conventionally, as described above, an index to be viewed is shown on the display element 10 as shown in FIG. 3 (c), and the user is made to perform an operation of gazing at the index to specify the gaze point. There is a problem that the user imaging operation cannot be performed. Therefore, if there is a timing that can identify the point that the user is actually gazing at during the user's imaging operation, it is used to detect the difference in coordinates between the estimated gazing point from the eyeball image and the actual gazing point. , It is desirable to calibrate from the difference data.

In the present invention, when a moving subject is imaged, the user gazes at the movement of the subject displayed on the display element 10 in the work of identifying the point that the user is actually gazing at during the image pickup operation of the user. It is done by judging whether or not it is chasing with. If it is determined that the subject is being followed, even if there is a discrepancy between the estimated gazing point from the eyeball image and the position of the moving subject, the actual gazing point is assumed to be the subject.

An important point of the present invention is how to determine whether or not to follow a subject moving by the line of sight, and as a means thereof, a motion vector of the subject obtained from a time-series through image by the image sensor 2 ( (Hereinafter referred to as optical flow) and the motion vector of the estimated gazing point obtained from the time-series data of the estimated gazing point obtained from the eyeball image obtained by the image sensor 17 for the eyeball are compared to determine the degree of agreement. ..

FIG. 7 shows a situation in which a user visually recognizes a runner, which is a moving subject, through a display element 10 in the finder, and the above operation will be described using this.

In FIG. 7A, the user is actually gazing at the runner ( _point J1), which is _a moving subject, at time t1, but the line-of-sight correction coefficient is not at an appropriate value and is estimated by the camera. An example is shown in which the viewpoint is erroneously recognized as S ₁ (coordinates (Hx ₁ , Hy ₁ )). In this example, the gazing point moves in parallel by the vector Z and is displaced. At this time, the user wants to focus on the runner ( _point J1), but the background at the _point S1 is erroneously focused. At this time, the camera cannot determine where the gaze point is actually seen by the user only from the data obtained at time t1. However, by accumulating data in chronological order, the determination becomes possible. First, it is assumed that the camera stores the through image and the estimated gaze point position coordinates (Hx1, Hy1) at the time t1 in the memory unit.

Next, FIG. 7B shows the situation of the display element 10 in the finder at the time t2 when a predetermined time Δt has elapsed from the time t1. The position of the runner actually gazing at the user in the through image moves, moves from point J1 to point J2, and the gazing point position estimated by the camera at this point is point S ₂ (coordinates (Hx ₂ , Hy). ₂ )) is determined to be. There is still a gap between the points J ₂ and the point S ₂ , and when the focus is on S ₂ , which is the estimated gaze point, the background is in focus. _At this time, by using the stored through image at time t1 and the estimated gaze _point position and comparing with the same information at _time t2, the image when the time t1 is changed to _time t2. It is possible to obtain the illustrated motion vector ΔO indicating the amount and direction of movement of the subject in the subject and the illustrated estimated gazing point movement vector ΔH indicating the amount and direction of movement of the estimated gazing point.

When the user follows the movement of the subject in the image with his / her eyes, even if there is a positional deviation between the actual gazing point and the estimated gazing point, the subject movement is the time-series movement amount and direction of both. The vector ΔO and the estimated gaze point movement vector ΔH are almost equal. Therefore, if there is an image region having a motion vector that substantially matches the movement vector ΔH in the vicinity of the estimated gazing point in the through image, it can be determined that the region is the region that the user is actually gazing at. ..

In FIG. 7B, the region of the subject of the person shown in the image displayed on the display element 10 has a motion vector ΔO that substantially coincides with the estimated gaze point movement vector ΔH, and the region is actually gazed at. It can be judged that the point _J2 is. If the point J2, which is the actual gazing point, can be specified, the line _- of-sight correction coefficient can be recalculated and corrected in the same manner as in the conventional calibration described above.

With the above configuration, the camera can automatically determine the difference between the estimated gazing point and the actual gazing point position and perform calibration without causing the user to perform a specific operation other than the normal imaging operation. ..

<Acquisition of optical flow>
In order to perform the above, it is necessary to acquire the motion vector of the object to be photographed, and in this embodiment, it is realized by the following configuration.

In recent years, the functionality of image sensors has been improved, and the number of pixels and the frame rate have been improved. A method of obtaining vector data (optical flow) indicating the movement of an object between a plurality of images using this signal is known. In this embodiment, the optical flow is acquired from the through image. The image processing unit included in the CPU 3 can generate an optical flow based on a comparison between a plurality of images obtained from the image pickup element 2, and the image pickup element 2 and the image processing unit can use the optical flow calculation means according to claim 1. Configure.

FIG. 8 is a diagram showing an example of how to obtain an optical flow, which is a blur detection signal based on a plurality of image comparisons. Corresponds to the specific operation of the CPU 8 in FIG. Although the so-called block matching method will be described with reference to FIG. 8, other methods may be used.

8 (a) is an image acquired at a certain time t _n , FIG. 8 (b) is an image acquired at a time t _{n + 1} after (a), and FIG. 5 (c) is the time t _n and t. It is a figure which superimposed and displayed the image acquired by _{n + 1} , and schematically showed the detected vector.

In FIG. 8, 61 is an image acquired at time t _n , 62 is a subject, 63 is a region of interest in image 61, 64 is an image acquired at time t _{n + 1} , and 65 is 63, which is a position in the screen. Are acquired with an arrow schematically indicating that 66 is searching for the same region, 67 is an region in the image 64 corresponding to the region 63 of the subject of the image 61, and 68 is acquired at time nt _n and t _{n + 1} . The image in which the images are superimposed is shown, and 69 shows the movement vector detected in the region 63 in the image 61, respectively.

As shown in FIG. 8, two images 61 and 64 acquired at different times are prepared. Focusing on the region 63 in which the subject 62 exists in one of the images 61. The size of this area can be set arbitrarily, but may be, for example, 8x8 pixels.

The location of the area 63 in the image 64 may be searched based on the absolute difference value (= SAD) or the like. Specifically, the SAD may be calculated by shifting the region as shown by the arrow 66 in a predetermined range around the region 65 whose position in the image corresponds to the image 64. When SAD is used as a reference, the minimum position is the corresponding region 67. As a result, it can be seen that the region 63 moves like the vector 69 between the two images 68.

The above operation is performed for a plurality of areas set in the screen (multiple movement vectors are detected in the screen. After that, vector selection is performed focusing on the main subject, using a known technique of RANSAC (= Random Sample Consensus). One evaluation value of the movement vector between two images may be determined by a method such as obtaining an estimated value by a method such as (the explanation is omitted because it is not a main part).

<Operation of camera with line-of-sight detection function and automatic calibration function>
The operation of the camera having the line-of-sight detection function and the automatic calibration function in this embodiment, the principle of which has been described so far, will be described with reference to the flowchart of FIG.

When the camera power is turned on and the flow is started, in step S101, the image sensor 2 starts acquiring a through image, transmits an image signal to the CPU 3, and the CPU 3 displays the acquired through image on the display element 10. The user visually recognizes the subject by seeing the through image displayed on the display element 10 in the finder. The display element 10 corresponds to the display means according to claim 1.

In step S102, it is determined whether or not the power is turned off, and if it is turned off, the flow ends, and if it is not turned off, the process proceeds to step S103.

In step S103, acquisition of the eyeball image of the user who started to visually recognize the subject in step S101 is started, and by carrying out the above-mentioned line-of-sight detection routine, the estimated gaze point position coordinates (Hx, Hy) on the display element 10 are obtained. To get.

Further, following the acquisition of the estimated gazing point position coordinates in the previous step, in step S104, the through image obtained from the image pickup device 2 is stored in the memory unit 4.

In step S105, it is determined whether or not there are two or more histories of the estimated gazing point position coordinates stored in the memory unit 4. If there are less than two, the process proceeds to step S106. If two or more of them exist, the gazing point position movement vector and the optical flow can be acquired, so that the process proceeds to step S201.

In step S106, among the indexes showing the 180 focus detection points on the display element 10 in the finder as shown in FIG. 3A, the estimated viewpoint position of the user stored in the memory unit 4 The index at the position closest to the last memorized coordinate in the coordinate history is determined to be the focus detection point, distance measurement is performed using the corresponding distance measurement pixel on the image sensor 2, and the focus is shown in the figure. The target is highlighted as in point A.

Next, in step S107, a standby for a predetermined time is performed.

In step S108, it is determined whether or not the release button described above is pressed and SW1 is ON. If the user agrees to focus at the position of the optotype highlighted in step S106, the user presses the release button at the time of waiting for a predetermined time in the previous step to turn on SW1. If SW1 is ON, the process returns to step S109, and if SW1 is not ON, the process returns to step S103 to reacquire the gazing point position.

In step S109, the lens group is moved to the in-focus position corresponding to the distance measurement result acquired in step S104, and the process proceeds to step S110.

In step S110, the release button is further pushed in, and it is determined whether or not SW2 is turned on. If the user agrees to shoot at the current in-focus position, SW2 will be turned on. If SW2 is ON, the process returns to step S111, and if SW2 is not ON, the process returns to step S106, and the distance measurement / display is repeated at the same position.

In step S111, an image pickup operation is performed, an image signal is acquired by the image pickup element 2, transmitted to the CPU 3, and stored in the memory unit 4.

In step S112, the image acquired in the previous step is displayed on the display element 10 for a predetermined time, and the process proceeds to step S113.

In step S113, the estimated gazing point history data and the through image history stored in the memory unit 4 are erased. After that, the process returns to step S101, and the through image is reacquired and displayed.

In step S201, the latest data and the data stored immediately before the latest data are compared with the history of the estimated gazing point position coordinates stored up to the previous step and the history of the through image, and the above-mentioned estimated gazing point is determined. The estimated gaze point movement vector, which is a movement vector in time, and the optical flow, which is a movement vector of the subject in a predetermined time on the screen, are acquired. The operation of this step corresponds to the line-of-sight movement vector measuring means and the optical flow calculating means in claim 1.

In step S202, the degree of allowable difference used as a reference in the process of determining the agreement between the estimated gaze point movement vector and the optical flow vector in the screen, which is performed later, is calculated. Here, the degree of permissible difference varies depending on the size of the estimated gaze point movement vector calculated in the previous step, and the larger the estimated gaze point movement vector, the larger the permissible difference degree (even if the difference is large, it is permissible). do. This is because as the moving speed of the subject in the finder field of view increases, the difference between the movement of the line of sight and the movement of the subject becomes more likely to occur when the movement is followed by the line of sight, and the accuracy of capturing the subject is prevented from decreasing. .. The speed of the subject is judged from the size of the movement vector (estimated gaze point movement vector) of the line of sight that follows it, and is reflected in the calculation of the degree of allowable difference. This corresponds to the means for calculating the degree of allowable difference in claim 2.

In step S203, it is determined whether or not there is a subject area in the screen having an optical flow that matches the estimated gaze point movement vector acquired in the previous step by a predetermined degree or more. The determination of the match between the two vectors is performed using the allowable difference degree calculated in the previous step, and if the difference between the two vectors falls within the range of the allowable difference degree, it is judged to be a match. If there is a matching area, the coordinates are acquired and the process proceeds to step S204. The operation in this step corresponds to the movement vector matching degree determination means in claim 1. If there is no matching area, the process returns to step S106, and distance measurement is performed at the latest estimated gaze point position.

In step S204, the positions of the coordinates of the subject area having an optical flow that matches the estimated gaze point movement vector acquired in the previous step by a predetermined degree or more and the coordinates of the latest estimated gaze point are compared, and both are within a predetermined value. If the position is close to, it is determined that the area is the point that the user is actually paying attention to, and the process proceeds to step S205. This is because the estimated gazing point position deviates from the actual gazing point due to an error such as an individual difference, but the amount of the deviation is not large and it can be judged that both exist in the vicinity. If the vectors are separated by a predetermined distance or more, it is considered that the coincidence between the vectors is a coincidence, and the process returns to step S106 to perform distance measurement at the latest estimated gazing point position.

In step S205, assuming that the region having the optical flow is the point that the user is actually gazing at, distance measurement is performed at that position, and the index of the focus detection point at the position corresponding to the region is highlighted. The process of specifying the point actually being watched in steps S203 to S205 corresponds to the subject area specifying means in claim 1. Through the above process, even if there is an error in the line-of-sight detection result, it becomes possible to focus on the subject that the user is actually paying attention to, and the subject capture accuracy is improved.

Next, in step S206, a standby for a predetermined time is performed, and in step S207, the release button is pressed to determine whether or not SW1 is ON. If the user agrees to focus at the position of the optotype highlighted in step 205, the user presses the release button at the time of waiting for a predetermined time in the previous step to turn on SW1. If SW1 is ON, the process proceeds to step S208, and if it is not ON, the process returns to step S102 to reacquire the gazing point position.

In step S208, the lens group is moved to the in-focus position corresponding to the distance measurement result acquired in step S205, and the process proceeds to step S209.

In step S209, the release button is further pushed in, and it is determined whether or not SW2 is turned on. If the user agrees to shoot at the current in-focus position, SW2 will be turned on. If SW2 is ON, the process proceeds to step S210. If SW2 is not ON, the process returns to step S205, and the distance measurement / display is repeated at the same position.

In step S210, an image pickup operation is performed, an image signal is acquired by the image pickup element 2, transmitted to the CPU 3, and stored in the memory unit 4.

In step S211 it is determined that the imaging operation was performed because the user recognized the highlighted area having the optical flow as the correct gaze position, and the coordinates of the area were used to correct the above-mentioned line-of-sight correction coefficient. It is stored in the memory unit 4. The operation of this step corresponds to the line-of-sight position detection result correction means in claim 1. After that, the process proceeds to step S112 to display the acquired image.

With the above configuration, in a subject recognition device having a line-of-sight detection device, it is possible to accurately recognize the subject to be gazed by the photographer and to automate the calibration operation of the gaze position to improve the convenience of the photographer. did it.

1 Shooting lens 2 Image sensor 3 CPU
4 Memory unit 10 Display element 11 Display element drive circuit 12 Eyepiece 13a to f Illumination light source 14 Eyeball 15 Light divider 16 Light receiving lens 17 Eyepiece image pickup element 111 Aperture 112 Aperture drive unit 113 Lens drive motor 114 Lens drive member 115 Photocoupler 116 pulse plate 117 mount contact 141 pupil 142 cornea 143 glow

Claims (2)

Imaging means and
Display means for displaying captured images and
A line-of-sight position detecting means for detecting the line-of-sight position of the photographer,
An optical flow calculation means that calculates an optical flow based on an image acquired by the image pickup means, and an optical flow calculation means.
A line-of-sight movement vector measuring means for measuring a movement amount and a movement direction in which the photographer's line-of-sight position measured by the line-of-sight position detecting means moves at a predetermined time.
An optical flow calculated by the optical flow calculation means and a line-of-sight movement vector acquired by the line-of-sight movement vector acquisition means are compared, and the degree of coincidence between the two is determined from the difference.
When the determination means determines that the two match, an image region having the optical flow is extracted, and the subject region specifying means for identifying the region as the subject region is used.
By using the position coordinates of the specified subject area, the line-of-sight position detection result correction means for correcting the line-of-sight position detection result, and the line-of-sight position detection result correction means.
A subject recognition device characterized by having. The permissible difference degree calculating means for calculating the permissible difference degree, which is the permissible range of the difference between the two vectors, in which the line-of-sight movement vector and the optical flow are considered to match, is provided. The permissible difference degree is determined by the magnitude of the line-of-sight movement vector measured by the line-of-sight movement vector measuring means, and the larger the line-of-sight movement vector is, the larger the permissible difference degree is, and the movement vector matching degree determination means. The subject recognition device according to claim 1, wherein when both vectors are within the range of the allowable difference degree, it is considered that both vectors match.

Images