JP2023083695A - Electronic apparatus - Google Patents

Abstract

To provide a technique to prevent an object not intended by a user from being selected as a target to allow selection of a target according to the intention of the user.SOLUTION: An electronic apparatus of the present invention has: first detection means that detects a line-of-sight position that is a position a user sees; second detection means that detects an object and acquires object information that is information related to the object; receiving means that can receive a specific operation from the user; selection means that, in response to the specific operation, selects the object the user sees as a target based on the line-of-sight position detected by the first detection means; and recording means that, in response to the specific operation, records object information on the target selected by the selection means in a storage unit. When the specific operation is not performed, the selection means selects the target based on the object information stored by the storage unit.SELECTED DRAWING: Figure 8

Description

The present invention relates to an electronic device such as an imaging device, and more particularly to technology for selecting an object from one or more objects.

Imaging devices (cameras) have been proposed that can detect a subject and identify the type of the subject (eg, person, animal, vehicle, etc.). Also, as such a camera, a camera has been proposed that selects a main subject, which is the target of focus adjustment or exposure adjustment, from among the detected subjects.

When multiple subjects are captured in a captured image (captured image), the subject near the center of the captured image or a subject with a large captured size (size on the captured image) is generally the main subject. is selected as

However, the user (photographer) does not necessarily want to select a subject near the center of the captured image or a subject with a large captured size as the main subject. For example, there are cases where the user wants to place a main subject such as a person at the edge of the captured image and take a picture. In such a case, if a large subject such as a train appears in the center of the captured image, the large subject will be selected as the main subject against the user's intention. Although it is possible for the user to manually select the main subject, it takes time and effort.

A camera has also been proposed that can estimate a user's viewpoint (the position the user is looking at) in a captured image, and can select an object near the viewpoint as the main object in response to the user's viewpoint confirmation operation. Even with such a camera, in order to select a subject arranged at the edge of the captured image or a small subject as the main subject, it takes time and effort to perform a point-of-regard fixing operation every time the shooting scene changes.

Japanese Patent Application Laid-Open No. 2002-200002 discloses a technique for storing information about subjects photographed in the past by a user and selecting a main subject based on the information.

JP 2021-32927 A

However, with the technique disclosed in Patent Document 1, when a plurality of subjects appear in a captured image, not only information about the main subject but also information about subjects other than the main subject are stored. Even if only the information of the main subject is stored, if the main subject is erroneously determined, the information of the subject unintended by the user will be stored. As a result, there is a risk that a subject not intended by the user will be selected as the main subject based on information about previously photographed subjects. Such a problem arises not only in the case of selecting a main subject from one or more subjects, but also in various cases of selecting a target object from one or more objects.

SUMMARY OF THE INVENTION An object of the present invention is to provide a technology that prevents an object not intended by the user from being selected as an object and allows the user to select an object that matches his/her intention.

The electronic device of the present invention comprises first detection means for detecting a line-of-sight position, which is the position at which the user is looking, and second detection means for detecting an object and acquiring object information, which is information related to the object. a receiving means capable of receiving a specific operation from the user; and an object viewed by the user based on the line-of-sight position detected by the first detecting means in response to the specific operation. selecting means for selecting an object as an object; and recording means for recording object information of the object selected by the selecting means in a storage unit in response to the specific operation, wherein the selecting means is not performed, the object is selected based on the object information stored in the storage unit.

Advantageous Effects of Invention According to the present invention, it is possible to suppress selection of an object unintended by the user as an object and select an object that meets the user's intention.

It is a figure which shows an example of the external appearance of a camera. It is a figure which shows an example of an internal structure of a camera. 3 is a block diagram showing an example of an electrical configuration within the camera; FIG. It is a figure which shows an example of the visual field in a finder. It is a figure which shows an example of the optical system for performing line-of-sight detection. FIG. 4 is a diagram showing an example of an eye image and its luminance information; 5 is a flowchart showing an example of line-of-sight detection operation; 6 is a flowchart showing an example of main subject selection operation; 7 is a flowchart showing an example of priority determination operation based on type information; 7 is a flowchart showing an example of priority determination operation based on identification information; 4 is a diagram showing an example of subject information stored in a memory unit; FIG. FIG. 5 is a diagram for explaining a specific example of camera operation; FIG. 10 is an external view of another electronic device;

Embodiments of the present invention will be described below with reference to the accompanying drawings.

<Configuration example>
1A and 1B show an example of the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to this embodiment. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a back perspective view. As shown in FIG. 1A, the camera 1 has a photographing lens unit 1A and a camera housing 1B. The camera housing 1B is provided with a release button 5, which is an operation member (operation receiving section) capable of receiving a photographing operation from a user (photographer). As shown in FIG. 1B, an eyepiece lens 12 ( eyepiece optical system) is arranged. Note that the eyepiece optical system may include a plurality of lenses. Operation members 41 to 43 capable of receiving various operations from the user are also arranged on the rear surface of the camera housing 1B. For example, the operation member 41 is a touch panel capable of receiving touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-way key that can be pushed in each of four directions. The operation member 41 (touch panel) includes a display panel such as a liquid crystal panel, and has a function of displaying an image on the display panel.

FIG. 2 is a cross-sectional view of the camera 1 cut along the YZ plane formed by the Y-axis and Z-axis shown in FIG.

Two lenses 101 and 102, a diaphragm 111, a diaphragm driving section 112, a lens driving motor 113, a lens driving member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118 are provided in the photographing lens unit 1A. etc. are included. The lens driving member 114 is composed of a driving gear or the like, and the photocoupler 115 detects the rotation of the pulse plate 116 interlocked with the lens driving member 114 and transmits it to the focus adjustment circuit 118 . The focus adjustment circuit 118 drives the lens drive motor 113 based on the information from the photocoupler 115 and the information (lens drive amount information) from the camera housing 1B, and moves the lens 101 to adjust the focus position. change. A mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B. Although two lenses 101 and 102 are shown for simplicity, more than two lenses are actually included in the photographing lens unit 1A.

The camera housing 1B includes an imaging element 2, a CPU 3, a memory section 4, a display device 10, a display device driving circuit 11, and the like. The imaging element 2 is arranged on the intended imaging plane of the photographing lens unit 1A. A CPU 3 is a central processing unit of a microcomputer and controls the camera 1 as a whole. The memory unit 4 stores images captured by the imaging element 2 and the like. The display device 10 is composed of liquid crystal or the like, and displays a captured image (object image) or the like on the display surface of the display device 10 . A display device drive circuit 11 drives the display device 10 . A user can view an image (such as an image captured by the imaging device 2 ) displayed on the display surface of the display device 10 through the eyepiece 12 .

The camera housing 1B also includes light sources 13a and 13b, a light splitter 15, a light receiving lens 16, an eye imaging element 17, and the like. The light sources 13a and 13b are light sources for illuminating the eyeball 14 of the user. The light sources 13a and 13b have conventionally been used in single-lens reflex cameras in order to detect the line-of-sight direction (line-of-sight direction; the direction in which the user is looking) from the relationship between the reflected image (corneal reflection image; Purkinje image) due to the corneal reflection of light and the pupil. etc. Specifically, the light sources 13 a and 13 b are infrared light emitting diodes or the like that emit infrared light that is insensitive to the user, and are arranged around the eyepiece lens 12 . An optical image of the illuminated eyeball 14 (eyeball image; an image formed by reflected light emitted from the light sources 13 a and 13 b and reflected by the eyeball 14 ) passes through the eyepiece 12 and is reflected by the light splitter 15 . Then, the eyeball image is formed by the light receiving lens 16 on the eye imaging device 17 which is a two-dimensional array of photoelectric elements such as CCD and CMOS. The light-receiving lens 16 positions the pupil of the eyeball 14 and the eye imaging device 17 in a conjugate imaging relationship. The line-of-sight direction of the eyeball 14 is detected from the positional relationship between the pupil and the corneal reflection image in the eyeball image formed on the eye imaging device 17 by a predetermined algorithm, which will be described later.

FIG. 3 is a block diagram showing an example of an electrical configuration within the camera 1. As shown in FIG. A line-of-sight detection circuit 201, an image processing circuit 202, an automatic focus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, and the like are connected to the CPU 3. FIG. The CPU 3 also transmits a signal to the focus adjustment circuit 118 arranged in the photographing lens unit 1A and the diaphragm control circuit 206 included in the diaphragm driving section 112 in the photographing lens unit 1A through the mount contact 117. do. A memory unit 4 attached to the CPU 3 has a function of storing image signals from the image sensor 2 and the image sensor 17 for eyes, and a function of storing sight line correction parameters for correcting individual differences in sight lines, which will be described later.

The line-of-sight detection circuit 201 A/D-converts the output of the eye image sensor 17 in the state where the eyeball image is formed on the eye image sensor 17 (the eye image obtained by capturing the eye), and transmits the result to the CPU 3 . do. The CPU 3 extracts feature points necessary for line-of-sight detection from the eye image according to a predetermined algorithm, which will be described later. position; where the user is looking) is calculated.

The image processing circuit 202 performs various processing based on the image captured by the image sensor 2 . For example, the image processing circuit 202 performs amplification, logarithmic compression, A/D conversion, etc. of a signal obtained from the image sensor 2 that also serves as a photometric sensor, specifically, a luminance signal corresponding to the brightness of the object scene. Then, the resulting object field luminance information is sent to the CPU 3. The field brightness information is information indicating the brightness of each pixel of the captured image. Also, the image processing circuit 202 detects a specific subject (specific object) from the image captured by the image sensor 2 and sends the resulting subject information and subject area information to the CPU 3 . Subject information (object information) is information related to the detected subject, and subject area information (object area information) is information indicating the area of the detected subject. Subject information and subject area information can also be obtained by analyzing subject field luminance information.

The autofocus detection circuit 203 A/D-converts signal voltages from a plurality of detection elements (plurality of pixels) included in the image sensor 2 and used for phase difference detection, and sends them to the CPU 3 . The CPU 3 calculates the distance to the object corresponding to each focus detection point from the signals of the plurality of detection elements. This is a well-known technique known as imaging plane phase difference AF. In this embodiment, as an example, the field of view in the finder in FIG. , each has a focus detection point.

A switch SW1 and a switch SW2 are connected to the signal input circuit 204 . The switch SW1 is turned on by the first stroke (half-press) of the release button 5 to start photometry and focus detection (distance measurement) of the camera 1 and the like. The switch SW2 is turned on by the second stroke (full depression) of the release button 5 to start the photographing operation. ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU3. Photometry and focus detection are performed within a set photometry area or focus detection area, or within a selected (determined) main subject area. For example, in photometry, the image processing circuit 202 determines the brightness of the target area from field brightness information, and in focus detection, the automatic focus detection circuit 203 determines the distance to the subject existing in the target area. be.

A light source drive circuit 205 drives the light sources 13a and 13b.

FIG. 4 is a diagram showing an example of a viewfinder viewfinder, showing a state in which the display device 10 operates (a state in which an image is displayed). As shown in FIG. 4, the finder field includes a focus detection area 400, 180 distance measuring point indices 401, a field mask 402, and the like. Each of the 180 ranging point indices 401 is superimposed on the through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging plane. . Further, among the 180 ranging point indicators 401, the ranging point indicator 401 corresponding to the current viewpoint A (estimated position) is highlighted with a frame or the like and displayed.

<Example of gaze detection operation>
A line-of-sight detection method will be described with reference to FIGS. FIG. 5 is a diagram for explaining the principle of the line-of-sight detection method, and is a diagram showing an example of an optical system for performing line-of-sight detection. As shown in FIG. 5, the light sources 13a and 13b are arranged substantially symmetrically with respect to the optical axis of the light receiving lens 16, and illuminate the eyeball 14 of the user. Part of the light emitted from the light sources 13 a and 13 b and reflected by the eyeball 14 is collected by the light receiving lens 16 onto the eye imaging device 17 . FIG. 6A is a diagram showing an example of an eye image captured by the eye imaging device 17 (an eyeball image projected on the eye imaging device 17), and FIG. is a diagram showing an example of the output intensity of . FIG. 7 is a flow chart showing an example of the line-of-sight detection operation.

The line-of-sight detection operation is started, for example, when the camera 1 is activated, and is repeated. When the line-of-sight detection operation starts, the light sources 13a and 13b emit infrared light toward the user's eyeball 14 in step S701 of FIG. The user's eyeball image illuminated by the infrared light is imaged on the eye imaging device 17 through the light receiving lens 16 and photoelectrically converted by the eye imaging device 17 . This provides an electrical signal of the eye image that can be processed.

In step S<b>702 , the line-of-sight detection circuit 201 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 17 to the CPU 3 .

In step S703, the CPU 3 detects coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S702.

Infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the eyeball 14 of the user. At this time, the corneal reflection images Pd and Pe formed by a part of the infrared light reflected on the surface of the cornea 142 are condensed by the light receiving lens 16 and formed on the eye imaging element 17 to form an eye image. are corneal reflection images Pd' and Pe' at . Similarly, the light beams from the ends a and b of the pupil 141 are also imaged on the eye imaging device 17 to form pupil end images a' and b' in the eye image.

FIG. 6(B) shows luminance information (luminance distribution) of the area α in the eye image of FIG. 6(A). FIG. 6B shows the luminance distribution in the X-axis direction, with the horizontal direction of the eye image being the X-axis direction and the vertical direction being the Y-axis direction. In this embodiment, the X-axis (horizontal) coordinates of the corneal reflection images Pd' and Pe' are Xd and Xe, and the X-axis coordinates of the pupil edge images a' and b' are Xa and Xb. As shown in FIG. 6B, extremely high levels of brightness are obtained at the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from the X coordinate Xa to the X coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by forming an image of the light flux from the pupil 141 on the eye imaging device 17), the X coordinates Xd, Xe Extremely low levels of luminance are obtained except for In the area of the iris 143 outside the pupil 141 (the area of the iris image outside the pupil image obtained by forming an image of the light flux from the iris 143), a brightness intermediate between the above two types of brightness is obtained. Specifically, in the region where the X coordinate (coordinate in the X-axis direction) is smaller than the X coordinate Xa and in the region where the X coordinate is larger than the X coordinate Xb, a brightness intermediate between the above two types of brightness is obtained.

The X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained from the luminance distribution shown in FIG. 6B. Specifically, the coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates with extremely low brightness can be obtained as the coordinates of the pupil edge images a' and b'. . Further, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16 is small, the pupil center image c′ ( The X coordinate Xc of the pupil image center) can be expressed as Xc≈(Xa+Xb)/2. That is, the X coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

In step S704, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of the eyeball 14 with respect to the light receiving lens 16, and can be calculated using the function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

In step S<b>705 , the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16 as information on the line-of-sight direction of the eyeball 14 . The X coordinate of the midpoint between the corneal reflection image Pd and the corneal reflection image Pe and the X coordinate of the center of curvature O of the cornea 142 substantially match. Therefore, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, then the rotation angle θx of the eyeball 14 in the ZX plane (the plane perpendicular to the Y axis) is given below. It can be calculated by the formula 1 below. The rotation angle θy of the eyeball 14 within the ZY plane (the plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating the rotation angle θx.

β×Oc×SINθx≈{(Xd+Xe)/2}−Xc (Formula 1)

In step S706, the CPU 3 estimates the user's viewpoint on the display surface of the display device 10 using the rotation angles θx and θy calculated in step S705. Assuming that the coordinates (Hx, Hy) of the viewpoint correspond to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following equations 2 and 3.

Hx=m×(Ax×θx+Bx) (Formula 2)
Hy=m×(Ay×θy+By) (Formula 3)

The parameter m in Equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16, etc.) of the camera 1, and the rotation angles θx and θy are set to the coordinates corresponding to the pupil center c on the display surface of the display device 10. Transform coefficients to transform. It is assumed that parameter m is determined in advance and stored in memory unit 4 . The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in line-of-sight, and are obtained by performing calibration work. It is assumed that the parameters Ax, Bx, Ay, and By are stored in the memory unit 4 before the line-of-sight detection operation starts.

In step S707, the CPU 3 associates the viewpoint coordinates (Hx, Hy) with the time when the eye image was obtained, stores them in the memory unit 4, and finishes the line-of-sight detection operation.

Although an example of using a corneal reflection image has been described, the method of line-of-sight detection is not limited to this, and any method of detecting a line of sight based on an eye image may be used. Also, an example of obtaining the coordinates (Hx, Hy) of the viewpoint as the result (final result) of the line-of-sight detection has been described, but what kind of information (line-of-sight information) such as the rotation angles θx, θy may be Information may be obtained as a result of gaze detection.

<Example of main subject selection operation>
The main subject selection operation will be described with reference to FIG. FIG. 8 is a flow chart showing an example of main subject selection operation. The main subject selection operation is started, for example, when the camera 1 is activated, and is repeated. The main subject selection operation includes line-of-sight detection operation. In the camera 1, when a main subject (object) is selected by the main subject selection operation, focus adjustment (including focus detection) and exposure adjustment (including photometry) are performed on the main subject.

When the main subject selection operation starts, in step S801, the CPU 3 detects the user's viewpoint on the display surface of the display device 10 by the line-of-sight detection operation described above. When an image captured by the imaging device 2 is displayed on the display surface of the display device 10, the user's viewpoint on the display surface of the display device 10 is regarded as the user's viewpoint on the image captured by the imaging device 2. can also

In step S<b>802 , the CPU 3 controls the image processing circuit 202 to acquire subject information and subject area information, which are results of detecting a specific subject from the image captured by the image sensor 2 . A specific subject is, for example, a specific type of subject such as a person, an animal, or a vehicle. Individual subjects of a specific type may be treated as specific subjects without distinction, or individual subjects of a specific type may be distinguished and treated as specific subjects. The subject information is information related to the detected subject, and includes, for example, type information indicating the type of the detected subject and identification information (for example, an identifier) for identifying the detected subject. The detection method of the subject is not particularly limited. For example, a trained model obtained by learning using an image of a specific subject is used to detect the specific subject based on the color, shape, etc. of the specific subject. good too.

In step S803, the CPU 3 determines whether or not the user has performed a viewpoint confirmation operation (specific operation). When the CPU 3 determines that the viewpoint confirmation operation has been performed, the process proceeds to step S806, and when it determines that the viewpoint confirmation operation has not been performed, the process proceeds to step S804. The viewpoint fixing operation is an operation for fixing a viewpoint, and is an operation for conveying the viewpoint intended by the user to the camera 1 . The viewpoint fixing operation is performed using the operation members 41 to 43, for example. The viewpoint fixing operation may be the first stroke (half-press) of the release button 5 . In that case, after the process of step S806, photometry and focus detection (distance measurement) of the camera 1 are started.

In step S804, CPU 3 executes a priority determination operation. The priority determination operation is an operation for determining the priority of the subject detected in step S802. The priority determination operation will be described later with reference to FIG.

In step S805, the CPU 3 selects the subject with the highest priority determined in step S804 among the subjects detected in step S802 as the main subject.

In step S806, CPU 3 selects the subject viewed by the user as the main subject based on the viewpoint detected in step S801. Specifically, the CPU 3 selects a subject existing in the vicinity of the viewpoint (within a predetermined range from the viewpoint) among the subjects detected in step S802 as the main subject. Note that if there are multiple subjects near the viewpoint, it is preferable to select the subject closest to the viewpoint as the main subject. If the subject detected in step S802 does not exist near the viewpoint, the CPU 3 controls the image processing circuit 202 to detect an object existing in the viewpoint (subject not detected in step S802) as the main subject. do. For example, the CPU 3 detects an area made up of pixels at the viewpoint and pixels having a color similar to the color of the pixels at the viewpoint as the area of the main subject. The predetermined range that defines the vicinity of the viewpoint may be a fixed range or a range that can be set (changed) by the user using the operation members 41 to 43 or the like. The predetermined range may be automatically switched according to changes in the angle of view, that is, the focal length of the photographing lens unit 1A.

In step S807, the CPU 3 records the subject information of the main subject selected in step S806 in the memory section 4 (storage section) in association with the current time. Here, the subject information may be type information indicating the type of the main subject. The type in the type information is, for example, "person" or "animal". "Animals" may be subdivided into "dogs," "cats," "birds," and the like. The subject information may be identification information for identifying the main subject, such as "Person A", "Person B", and "Dog C". Further, when the main subject is selected when the subject detected in step S802 does not exist near the viewpoint, information such as "other" may be recorded as subject information, or nothing may be stored. may

<Example of priority determination operation>
The priority determination operation (step S804 in FIG. 8) when subject information is type information will be described with reference to FIG. FIG. 9 is a flow chart showing an example of the priority determination operation when subject information is type information.

When the priority determination operation starts, in step S901, the CPU 3 initializes the priorities of all subjects detected in step S802 of FIG. 8 to default values.

In step S<b>902 , the CPU 3 deletes subject information stored in the memory unit 4 for which a predetermined time or longer has passed since it was recorded. By doing so, the memory unit 4 stores only subject information recorded before the predetermined time from the present. Here, it is preferable that the predetermined period of time is not too long, for example, it is preferable that it is about several minutes to one day. Since the user does not always shoot the same type of subject, if too old subject information remains, when the user shoots another type of subject, the main subject will not match the user's intention. There is a risk that it will be lost. If the predetermined time is appropriately set, the main subject can match the user's intention even when the user shoots another type of subject. The predetermined time may be a fixed time or a time that can be set (changed) by the user using the operation members 41 to 43 or the like.

Note that the CPU 3 may delete subject information so that the memory section 4 stores a predetermined number or less of subject information. As with the predetermined time period described above, the predetermined number is preferably not too large. Also, the predetermined number may be a fixed number, or may be a number that can be set (changed) by the user using the operation members 41 to 43 or the like. Also, the CPU 3 may delete all subject information stored in the memory section 4 in response to the power-off operation. The CPU 3 may selectively or collectively delete the subject information stored in the memory section 4 in response to the user's operation using the operation members 41 to 43 or the like.

In step S903, the CPU 3 makes settings for performing the processing of steps S904 to S909 for each subject type indicated by the subject information stored in the memory unit 4. FIG. In other words, the CPU 3 makes settings for repeatedly executing the processing of steps S904 to S909 by the number of object types indicated by the object information stored in the memory unit 4. FIG.

In step S904, the CPU 3 determines whether or not the number of pieces of subject information of the subject type i (processing target) stored in the memory unit 4 is equal to or greater than a predetermined number. When determining that the number of pieces of subject information of the subject type i stored in the memory unit 4 is equal to or greater than the predetermined number, the CPU 3 advances the processing to step S905. When determining that the number of pieces of subject information of the subject type i stored in the memory unit 4 is less than the predetermined number, the CPU 3 advances the process to step S910.

In step S905, the CPU 3 increases the priority of subject type i. At this time, the CPU 3 may increase the priority by a predetermined amount of change that does not depend on the number of subject information of the subject type i stored in the memory section 4 . The CPU 3 may increase the priority with a larger amount of change as the number of pieces of subject information of the subject type i stored in the memory unit 4 increases.

In step S906, the CPU 3 determines whether or not the subject type for which the memory unit 4 stores the most subject information is the subject type i. When determining that the subject type for which the memory unit 4 stores the most subject information is the subject type i, the CPU 3 advances the process to step S907. When determining that the subject type for which the memory unit 4 stores the most subject information is not the subject type i, the CPU 3 advances the process to step S908.

In step S907, the CPU 3 increases the priority of subject type i.

In step S908, the CPU 3 determines whether or not the subject information of the subject type i has been recorded in the memory unit 4 before a predetermined time from now. If the CPU 3 determines that the subject information of the subject type i has been recorded in the memory unit 4 before the predetermined time from now, the CPU 3 advances the process to step S909. If the CPU 3 determines that the object information of the object type i has not been recorded in the memory unit 4 before the predetermined time from now, the CPU 3 advances the process to step S910.

In step S909, the CPU 3 increases the priority of subject type i. At this time, C
The PU 3 may increase the priority by a predetermined amount of change that does not depend on the time when the subject information of the subject type i was last recorded in the memory section 4 . The CPU 3 may increase the priority with a larger amount of change as the time at which the subject information of the subject type i was last recorded in the memory unit 4 is closer to the present.

Note that the CPU 3 increases the priority of the subject type i when there is subject information within a predetermined order counted from the side closest to the current time recorded in the memory section 4 as the subject information of the subject type i. may In other words, the CPU 3 determines the priority of the subject type i when the subject information of the subject type i exists in the predetermined number of subject information counted from the side closest to the current time recorded in the memory section 4. You can make it higher.

In step S910, the CPU 3 determines whether or not the processes of steps S904 to S909 have been performed for all subject types indicated by subject information stored in the memory section 4. FIG. When determining that the processes of steps S904 to S909 have been performed for all subject types, the CPU 3 advances the process to step S911. When determining that there is a subject type for which steps S904 to S909 have not been processed, the CPU 3 updates the subject type i and advances the process to step S904.

When proceeding to step S911, the CPU 3 performs the process of applying the priority of the subject type determined in steps S904 to S909 to all subjects detected in step S802 of FIG. do it against For example, the CPU 3 applies the priority of the subject type "animal" determined in steps S904 to S909 to the animal detected in step S802.

In step S911, among the subjects detected in step S802, the CPU 3 initializes the priority of subjects existing outside a predetermined range from the viewpoint detected in step S801 to a default value. Thereby, the user's intention can be more reflected in the priority of the subject. Specifically, the change in priority based on the subject information stored in the memory unit 4 (application of the priority determined in steps S904 to S909) exists within a predetermined range from the viewpoint detected in step S801. This is effective only for subjects that As a result, a subject existing near the current viewpoint of the same type as the main subject selected in response to the past viewpoint confirmation operation (a subject existing near the viewpoint at the time of the past viewpoint confirmation operation) is prioritized. higher than the default value. Here, the predetermined range may be a fixed range or a range that can be set (changed) by the user using the operation members 41 to 43 or the like. The predetermined range may be automatically switched according to changes in the angle of view, that is, the focal length of the photographing lens unit 1A.

In step S912, the CPU 3 gives higher priority to subjects existing within a predetermined range (near the viewpoint) from the viewpoint detected in step S801 among the subjects detected in step S802. At this time, the CPU 3 may increase the priority of the subject by a predetermined amount of change that does not depend on the distance between the subject and the viewpoint. The CPU 3 may increase the priority of the subject with a larger amount of change as the subject is closer to the viewpoint. The predetermined range may be a fixed range or a range that can be set (changed) by the user using the operation members 41 to 43 or the like. The predetermined range may be automatically switched according to changes in the angle of view, that is, the focal length of the photographing lens unit 1A.

In step S<b>913 , the CPU 3 gives higher priority to subjects existing within a predetermined range from the center of the image captured by the image sensor 2 among the subjects detected in step S<b>802 . At this time, the CPU 3 may increase the priority of the subject by a predetermined amount of change that does not depend on the distance between the subject and the center of the image. The CPU 3 may increase the priority of the subject by increasing the amount of change as the subject approaches the center of the image. The predetermined range may be a fixed range or a range that can be set (changed) by the user using the operation members 41 to 43 or the like. The predetermined range may be automatically switched according to changes in the angle of view, that is, the focal length of the photographing lens unit 1A.

In step S914, among the subjects detected in step S802, the CPU 3 gives higher priority to subjects whose imaging size (size on the image captured by the imaging device 2) is equal to or greater than a predetermined size. At this time, the CPU 3 may increase the priority of the subject by a predetermined amount of change that does not depend on the imaging size. The CPU 3 may increase the priority of the subject by increasing the amount of change as the imaging size increases. The predetermined size may be a fixed size, or may be a size that can be set (changed) by the user using the operation members 41 to 43 or the like. The predetermined size may be automatically switched according to changes in the angle of view, that is, the focal length of the photographing lens unit 1A.

In steps S904 to S909, the priority is raised based on the past subject information stored in the memory unit 4. In steps S911 to S914, the priority is raised based on the current viewpoint, subject position, subject size, and the like. Increased.

The priority determination operation (step S804 in FIG. 8) when subject information is identification information will be described with reference to FIG. FIG. 10 is a flowchart showing an example of the priority determination operation when subject information is identification information.

Steps S1001 and S1002 are the same as steps S901 and S902 in FIG.

In step S1003, the CPU 3 makes settings for performing the processing of steps S1004 to S1009 for each object identified by the object information stored in the memory unit 4. FIG. In other words, the CPU 3 makes settings for repeatedly executing the processing of steps S1004 to S1009 for the number of subjects identified by the subject information stored in the memory section 4. FIG.

In step S1004, the CPU 3 determines whether or not the number of pieces of subject information of the subject i (processing target) stored in the memory unit 4 is equal to or greater than a predetermined number. When determining that the number of pieces of object information about the object i stored in the memory unit 4 is equal to or greater than the predetermined number, the CPU 3 advances the processing to step S1005. When determining that the number of pieces of object information about object i stored in memory unit 4 is less than the predetermined number, CPU 3 advances the process to step S1010.

In step S1005, the CPU 3 increases the priority of subject i. At this time, the CPU 3 may increase the priority by a predetermined amount of change that does not depend on the number of subject information of the subject i stored in the memory section 4 . The CPU 3 may increase the priority with a larger amount of change as the number of pieces of subject information of the subject i stored in the memory section 4 increases.

In step S1006, the CPU 3 determines whether or not the subject for which the memory unit 4 stores the most subject information is the subject i. If the memory unit 4 determines that the subject for which the most subject information is stored is the subject i, the CPU 3 advances the process to step S1007. When the memory unit 4 determines that the subject for which the most subject information is stored is not the subject i, the CPU 3 advances the processing to step S1008.

In step S1007, the CPU 3 increases the priority of subject i.

In step S1008, the CPU 3 determines whether or not the subject information of the subject i has been recorded in the memory section 4 before a predetermined time from now. If the CPU 3 determines that the subject information of the subject i has been recorded in the memory unit 4 before the predetermined time from now, the CPU 3 advances the process to step S1009. If the CPU 3 determines that the subject information of the subject i has not been recorded in the memory unit 4 before the predetermined time from now, the CPU 3 advances the process to step S1010.

In step S1009, the CPU 3 increases the priority of subject i. At this time, the CPU 3 may increase the priority by a predetermined amount of change that does not depend on the time when the subject information of the subject i was last recorded in the memory unit 4 . The closer the time at which the subject information of the subject i was last recorded in the memory section 4 is to the present, the CPU 3 may increase the amount of change and increase the priority.

Note that if there is subject information within a predetermined order counted from the side closest to the current time recorded in the memory section 4 as the subject information of subject i, the CPU 3 may increase the priority of subject i. good. In other words, the CPU 3 increases the priority of the subject i when the subject information of the subject i exists in the predetermined number of subject information counted from the side closest to the current time recorded in the memory unit 4 . may

In step S1010, the CPU 3 determines whether or not the processes of steps S1004 to S1009 have been performed for all subjects indicated by the subject information stored in the memory section 4. FIG. When determining that the processes of steps S1004 to S1009 have been performed for all subjects, the CPU 3 advances the process to step S1011. If it is determined that there is a subject for which steps S1004 to S1009 have not been processed, CPU 3 updates subject i and advances the process to step S1004.

Note that since the priority of the subject type is determined in steps S904 to S909 of FIG. 9, processing to apply the priority of the subject type to the subject of the subject type is necessary when proceeding to step S911. . In steps S1004 to S1009, since the priority of individual subjects such as person A, person B, and dog C is determined, such processing is unnecessary.

Steps S1011 to S1014 are the same as steps S1011 to S1014 in FIG.

The number of possible priorities (the number of priority levels) is not particularly limited. A score may be calculated as the priority. The priority may have an upper limit, and the priority may not be higher than the upper limit. Furthermore, the amount of change in priority may be varied for each determination condition.

Also, the flowcharts of FIGS. 9 and 10 may be modified as appropriate. For example, although the processing proceeds to step S910 when the determination condition in step S904 of FIG. 9 is not satisfied, the processing may proceed to step S906. The same applies when the determination condition in step S1004 of FIG. 10 is not satisfied. At least one of the determination conditions shown in FIGS. 9 and 10 may be omitted, or another determination condition based on past subject information stored in the memory unit 4 may be added.

<Example of subject information stored in the memory unit>
11A to 11D are diagrams showing examples of subject information stored in the memory unit 4. FIG. 11A and 11B show examples in which the subject information is type information, and FIGS. 11C and 11D show examples in which the subject information is identification information. For simplicity of explanation, it is assumed that the maximum number (upper limit number) of subject information stored in the memory unit 4 is seven. It is assumed that the predetermined number in step S904 of FIG. 9 and step S1004 of FIG. 10 is 3, and the subject types or subjects for which the memory unit 4 stores three or more pieces of subject information are given higher priority. In FIGS. 11(A) to 11(D), subject types or subjects for which the memory unit 4 stores three or more pieces of subject information are indicated in gray. Also, the times a1 to a7, b1 to b7, c1 to c7, and d1 to d7 are assumed to be closer to the present as the numbers are smaller. Times a1, a2, b1, b2, c1, c2, d1, d2 are times from the present to a predetermined time before steps S908 and S1008, and times a3 to a7, b3 to b7, c3 to c7, d3 ∼d7 is the time more than a predetermined time before the present time. It is assumed that the priority is increased by one. Only the subject information stored in the memory unit 4 is considered, and the current viewpoint, subject position, subject size, etc. are not considered. That is, only steps S903 to S910 and steps S1003 to S1010 are considered, and steps S911 to S914 and steps S1011 to S1014 are not considered.

In FIG. 11A, the subject type for which the memory unit 4 stores three or more pieces of subject information is only a person, so 1 is added to the priority of the person (step S905). In addition, since the memory unit 4 stores the most object information of a person, 1 is added to the priority of the person (step S907). Further, since the time a1 at which the subject information of the person was last recorded in the memory unit 4 is the time before the predetermined time from the present time, 1 is added to the priority of the person (step S909). As a result, the priority of the person is 3, and the priority of the others is 0.

In FIG. 11B, since the object types for which the memory unit 4 stores three or more pieces of object information are a person and an animal, 1 is added to each of the priority of the person and the priority of the animal ( step S905). In addition, since the subject type for which the memory unit 4 stores the most subject information is the person, 1 is added to the priority of the person (step S907). Furthermore, since the time b1 at which the subject information of the person was last recorded in the memory unit 4 is the time before the predetermined time from the present time, 1 is added to the priority of the person (step S909). On the other hand, the time b3 when the subject information of the animal was last recorded in the memory unit 4 is the time more than the predetermined time before the present time, so the priority of the animal cannot be increased in step S909. As a result, the priority of the person is 3, the priority of the animal is 1, and the priority of the others is 0.

In FIG. 11C, the person A is the only subject for which three or more pieces of subject information are stored in the memory unit 4, so 1 is added to the priority of the person A (step S1005). Also, since the memory unit 4 stores the most object information of the person A, 1 is added to the priority of the person A (step S1007). Furthermore, since the time c1 at which the subject information of person A was last recorded in the memory unit 4 is the time from the present to the predetermined time ago, 1 is added to the priority of person A (step S1009). As a result, the priority of the person A becomes 3, and the priority of the others becomes 0.

In FIG. 11D, there is no subject for which the memory section 4 stores three or more pieces of subject information. As a result, the priority of any subject becomes zero. That is, according to the subject information stored in the memory unit 4, there is no difference in priority between subjects.

<Specific example of operation>
A specific example of the operation of the camera 1 will be described with reference to FIG.

In the state (a) of FIG. 12, the subject 12A (person) and subject 12B (dog) are shown in the image captured by the imaging device 2, and the subject 12A and subject 12B are detected. Here, it is assumed that the user wants to select subject 12B as the main subject. However, in general center priority or size priority, the subject 12A, which is close to the center of the image and has a large size on the image, is selected as the main subject, and a main subject frame (selection frame) is displayed for the subject 12A.

State (b) is a state after the user performs a viewpoint confirmation operation while looking at subject 12B, subject 12B located near the viewpoint is selected as the main subject, and a main subject frame is displayed for subject 12B. there is

State (c) is a state after the user has changed the orientation of the camera 1, for example. The image does not include the subjects 12A and 12B, but only the subject 12C (flower). Here, it is assumed that the camera 1 cannot detect plants (neither subject 12A nor subject 12B is detected, of course). Therefore, the main subject is not selected (the state in which the subject 12B was selected as the main subject is canceled), and the main subject frame is not displayed.

State (d) is a state after the user changes the composition of the image to the same composition as the composition of state (a), for example, by changing the orientation of the camera 1 again. As in state (a), the subject 12A is selected as the main subject, and the main subject frame is displayed for the subject 12A.

In a conventional camera, no matter how many times the operations of states (b) to (d) are repeated, the subject 12A is selected as the main subject in the composition of states (a) and (d) before the viewpoint confirmation operation is performed. end up

On the other hand, in the camera 1 according to the present embodiment, subject information related to the subject 12B is recorded in the memory unit 4 each time the user performs the viewpoint fixing operation in state (b), and the priority of the subject 12B is increased. Then, after the state (b) to (d) has been performed for a predetermined number of times or more, the state (c) composition is changed to the same composition as the state (a) composition, and then the state (d) is entered. Instead, the subject 12B is selected as the main subject before the viewpoint confirmation operation is performed, as in state (e).

<Summary>
As described above, according to the present embodiment, the subject that the user is looking at is selected as the main subject and the subject information of the main subject is recorded in the memory unit in response to the viewpoint confirmation operation. Then, when the viewpoint confirmation operation is not performed, the main subject is selected based on the subject information stored in the memory unit. For example, when the viewpoint confirmation operation is not performed, the subject whose subject information is stored in the memory unit is selected as the main subject. This makes it possible to suppress selection of a subject unintended by the user as the main subject, and to select a main subject that matches the user's intention.

The above-described embodiment is merely an example, and configurations obtained by appropriately modifying or changing the configuration of the above-described embodiment within the scope of the present invention are also included in the present invention. A configuration obtained by appropriately combining the configurations of the above embodiments is also included in the present invention.

For example, an example of viewing a subject with an EVF (electronic viewfinder) has been described, but the present invention can also be applied when viewing a subject with an OVF (optical viewfinder). The present invention can be applied to devices other than imaging devices (cameras), and can also be applied to viewing three-dimensional computer graphics. INDUSTRIAL APPLICABILITY The present invention is applicable to various electronic devices capable of detecting viewpoints.

<Examples of application to other electronic devices>
FIG. 13A is an external view of a notebook personal computer 1310 (notebook PC). In FIG. 13A , an imaging unit 1315 that captures an image of a user viewing a display unit 1311 of the notebook PC 1310 is connected to the notebook PC 1310 , and the notebook PC 1310 acquires imaging results from the imaging unit 1315 . Then, the notebook PC 1310 detects the user's viewpoint based on the imaging result. The present invention can also be applied to the notebook PC 1310 .

FIG. 13B is an external view of the smartphone 1320. FIG. In FIG. 13B, the smartphone 1320 detects the viewpoint of the user viewing the display unit 1322 of the smartphone 1320 based on the imaging result of the in-camera 1321 (front camera). The present invention can also be applied to smart phones 1320 . Similarly, the present invention can be applied to various tablet terminals.

FIG. 13C is an external view of the game machine 1330. FIG. In FIG. 13C, a head-mounted display 1335 (HMD) that displays a VR (Virtual Reality) image of a game on a display unit 1336 is connected to the game machine 1330 . The HMD 1335 has a camera 1337 that captures the eyes of the user wearing the HMD 1335 , and the game machine 1330 acquires the captured result from the HMD 1335 . Then, the game machine 1330 detects the user's viewpoint based on the imaging result. The present invention is also applicable to the game machine 1330. FIG. Components of the game machine 1330 may be built into the HMD 1335 . Therefore, the present invention is also applicable to the HMD1335. In the same way that the present invention can be applied when viewing a VR image displayed on an HMD, an AR (Augmented Reality) image displayed on the lens portion of a glasses-type wearable terminal or the like, or a real object beyond the lens portion can be viewed. The present invention is also applicable to viewing. The present invention is applicable to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology in the same way that the present invention is applicable to VR technology and AR technology.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1: Camera 3: CPU 5: Release button 201: Line-of-sight detection circuit 202: Image processing circuit

Claims (20)

a first detection means for detecting a line-of-sight position, which is the position at which the user is looking;
a second detection means for detecting an object and acquiring object information, which is information related to the object;
a receiving means capable of receiving a specific operation from the user;
selection means for selecting an object viewed by the user as a target object based on the line-of-sight position detected by the first detection means in response to the specific operation;
recording means for recording the object information of the object selected by the selection means in a storage unit in response to the specific operation;
The electronic device, wherein the selection means selects the object based on the object information stored in the storage unit when the specific operation is not performed. 2. The electronic device according to claim 1, wherein the object information is information indicating a type of object related to the object information. 2. The electronic device according to claim 1, wherein the object information is information for identifying an object related to the object information. The selection means, when the specific operation is not performed,
changing the priority of the object detected by the second detection means from a default value based on the object information stored in the storage unit;
4. The electronic device according to any one of claims 1 to 3, wherein an object having the highest priority after change is selected as the target object. 5. The electronic device according to claim 4, wherein the selection unit gives higher priority to objects whose number of pieces of object information stored in the storage unit is equal to or greater than a first number. 6. The electronic device according to claim 4, wherein the selection means gives a higher priority to an object having the largest number of pieces of object information stored in the storage unit. 7. The electronic device according to any one of claims 4 to 6, wherein said selection means gives higher priority to an object whose object information has been recorded in said storage unit from the present to a first time ago. device. 7. The selection means increases the priority of objects related to object information within a predetermined order counted from the side whose time recorded in the storage unit is closest to the present. 1. The electronic device according to item 1. 9. The electronic device according to any one of claims 4 to 8, wherein the storage unit stores object information recorded from the present to a second time ago. 10. The electronic device according to any one of claims 4 to 9, wherein the storage unit stores a second number or less of object information. further comprising imaging means;
The electronic device according to any one of claims 4 to 10, wherein said second detection means detects an object from the image captured by said imaging means. 12. The electronic device according to claim 11, wherein said selection means gives high priority to objects existing within a first range from the center of said image. 13. The electronic device according to claim 11, wherein the selection means gives a higher priority to an object whose size on the image is equal to or larger than a predetermined size. 14. The method according to any one of claims 4 to 13, wherein said selection means gives high priority to objects existing within a second range from said line-of-sight position detected by said first detection means. electronics. The selection means changes the priority based on the object information stored in the storage unit for an object existing within a second range from the line-of-sight position detected by the first detection means. 15. The electronic device according to any one of claims 4 to 14, characterized in that: 16. The method according to any one of claims 1 to 15, wherein said selection means selects an object whose object information is stored in said storage unit as said target object when said specific operation is not performed. 1. The electronic device according to item 1. a first detection means for detecting a line-of-sight position, which is the position at which the user is looking;
a second detection means for detecting an object;
a receiving means capable of receiving a specific operation from the user;
selection means for selecting an object viewed by the user as a target object based on the line-of-sight position detected by the first detection means in response to the specific operation;
has
The selection means is
selecting the first object as the target object before the specific operation is performed in a first state in which the first object and the second object are detected by the second detection means;
changing the object to the second object in response to the specific operation;
canceling the state in which the second object is selected as the object in response to a change to a second state in which neither the first object nor the second object is detected by the second detection means; After performing the operation more than a predetermined number of times,
An electronic device, wherein in the first state, the second object is selected as the target object before the specific operation is performed. a first detection step of detecting the line-of-sight position, which is the position where the user is looking;
a second detection step of detecting an object and acquiring object information that is information related to the object;
a receiving step of receiving a specific operation from the user;
a selection step of selecting an object viewed by the user as a target object based on the line-of-sight position detected in the first detection step in response to the specific operation;
a recording step of recording the object information of the target selected in the selection step in a storage unit in response to the specific operation;
A control method for an electronic device, wherein, in the selecting step, the target object is selected based on the object information stored in the storage unit when the specific operation is not performed. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 17. A computer-readable storage medium storing a program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 17.

Images