JP2020003740A - Image capturing device and control method therefor - Google Patents

Abstract

To accurately focus on a main object intended by a user.SOLUTION: A CPU 111 is configured to control driving of a focusing lens 126 to focus on an object detected based on a focus detection result, and then determine the object to be a first object to be tracked by the focusing lens on the basis of significance of the object whose focus detection result differs from a position corresponding to the object by less than a threshold.SELECTED DRAWING: Figure 5

Description

  The present invention relates to determining a subject to be focused.

  2. Description of the Related Art Conventionally, an imaging apparatus that performs autofocus (also referred to as AF) based on a focus state of a focus detection area has been known.

  Patent Literature 1 discloses a technique in which an imaging apparatus automatically selects one focus detection area used for focus adjustment from a plurality of focus detection areas.

JP-A-2002-072069

  According to Patent Literature 1, there is no need for the photographer to manually select the focus detection area, and thus there is an advantage that the operation procedure at the time of photographing can be simplified. However, in Patent Literature 1, there is the following problem in a case where AF is continuously performed for photographing a moving object. Even when the subject being tracked as the main subject to be kept in focus is different from the main subject intended by the user, the focus is continued on the erroneously detected main subject. Also, even when the main subject is correct, if the main subject changes dynamically and the tracked subject is no longer the main subject intended by the user, the camera continues to focus on the main subject intended by the user. There were times when I could not do that.

  The present invention has been made in view of the above problems, and has as its object to provide an imaging apparatus and a control method thereof that can more accurately focus a main subject intended by a user.

  In order to achieve the above object, the present invention provides a focus detection unit that performs focus detection based on a signal output from an image sensor, a control unit that controls driving of a focus lens, and the control unit controls the focus detection. A first determining unit that controls driving of the focus lens so as to focus on a subject detected based on the result, and then determines a main subject to be followed by the focus lens; And a first determination means for determining the first subject based on the saliency of the subject whose difference between the focus detection result and the focus detection result is less than the threshold value.

  According to the present invention, it is possible to more accurately focus on a main subject intended by a user.

FIG. 3 is a functional block diagram of a portion related to an imaging device of a camera. FIG. 2 is a cross-sectional view illustrating the first embodiment. This is the pixel structure of the image sensor The pupil division function of the image sensor 5 is a flowchart of a servo shooting mode according to the first embodiment. 6 is a flowchart illustrating a flow of a color saliency calculation process according to the first embodiment. FIG. 3 is a block diagram illustrating an evaluation pixel and an estimation area according to the first embodiment. FIG. 4 is a block diagram illustrating an evaluation area according to the first embodiment. 6 is a flowchart of a first subject determination process according to the first embodiment. 6 is a flowchart of a second subject determination process according to the first embodiment. 5 is a flowchart of a transfer determination process according to the first embodiment. 13 is a flowchart of a focus detection process according to the second embodiment. 13 is a flowchart of a second subject determination process according to the second embodiment. 9 is a flowchart of a transfer determination process according to the second embodiment. 15 is a flowchart of a transfer determination process according to the third embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[Example 1]
(Imaging system)
FIG. 2 is a sectional view showing Embodiment 1 of the camera including the imaging device according to the present invention.

  FIG. 1 is a functional block diagram of a portion related to an imaging device of the camera according to the first embodiment.

  Generally, a camera has two modes: a mode in which a lens is driven with respect to the image plane of a subject at a certain time (one-shot shooting mode), and a mode in which the lens is driven while predicting the image plane of a future subject (servo shooting mode). There are types. In the first embodiment, the movement when the camera is set to the servo shooting mode will be described.

  The camera of this embodiment is a digital still camera of an interchangeable lens type that uses a combination of a camera body 100 having an image sensor 103 and an imaging lens 120 having a photographic optical system.

  The camera body 100 includes an imaging element 103, a display 105, a CPU 110, an imaging control circuit 102, a memory circuit 101, an interface circuit 106, an image processing unit 107, and an electric contact 104.

  The image sensor 103 is formed of a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, or the like, and is arranged on a predetermined image forming surface of the photographing lens 120 of the camera body 100. Details of the image sensor 103 will be described later.

  The display 105 is formed by an LCD (Liquid Crystal Display) panel or the like, and displays a captured image and displays information such as captured information. In the live view mode, the display 105 displays a real-time moving image of an area to be shot. Here, the live view mode is a mode in which a subject image from the photographing lens 120 is captured by the image sensor 103 and a low-pixel moving image for preview is displayed on the display 105 in real time. In the live view mode, it is possible to execute phase difference AF on the imaging surface, which will be described later. Further, during continuous shooting, a low-pixel still image during each continuous shooting is displayed on the display device 105.

  The CPU 110 controls the camera as a whole and performs various calculations and determinations. The CPU 110 includes a subject detection unit 111, a phase difference detection unit 112, a first subject determination processing unit 113, a second subject determination processing unit 114, a transition determination processing unit 115, and a focus adjustment unit 116. Further, detection of the focus state of the photographing lens 120 (also referred to as focus detection) and an instruction to drive the focus lens 126 are performed based on the detection result. When focus detection is performed on a plurality of regions, the selection process is also performed.

  The subject detection unit 111 performs a process of detecting a subject based on an image signal obtained from the image sensor 103. The process of detecting a subject includes a face detection process of detecting a face. In addition, as one of the methods for detecting a subject, a saliency level described later is calculated. Further, a tracking position is calculated by detecting a subject based on characteristics of an image signal obtained from the image sensor 103. Instead of the subject detection unit 111, the detection may be performed by known template matching processing or histogram matching processing.

  The phase difference detecting means 112 detects a phase difference between a pair of image signals obtained from the image sensor 103.

  The first subject determination processing unit 113 determines to which subject position the focus position is to be moved. In other words, a subject to be a main subject (also referred to as a first subject) is determined.

  The second subject determination processing unit 114 determines a second subject candidate that can be the first subject.

  The transfer determination processing unit 115 determines whether to focus on the second object more than the first object determined at that time, that is, determine whether the second object is the main object.

  The focus adjustment unit 116 instructs the lens CPU 122 to move the focus position based on the defocus amount converted from the phase difference signal detected by the phase difference detection unit 112 at the first subject position.

  The image sensor control circuit 102 drives and controls the image sensor 103 according to an instruction from the CPU 110. The memory circuit 101 stores an image signal captured by the image sensor 103. Further, the memory circuit 101 stores the light reception distribution of the image sensor 103.

  The interface circuit 106 outputs an image signal processed by the image processing unit 107 to the outside of the camera.

  The image processing unit 107 performs image processing on an image signal captured by the image sensor 103.

  The electrical contact 104 contacts the electrical contact 124 of the taking lens 120 and is used for communication of power and various signals.

  The taking lens 120 is an interchangeable lens that is detachable from the camera body 100.

  The photographing lens 120 includes a photographing optical system, a lens CPU 122, a focus lens driving mechanism 121, a diaphragm driving mechanism 123, a diaphragm 125, and an electric contact 124. The photographing optical system includes a focus lens 126.

  The photographing optical system forms an optical image of a subject to be photographed on the image sensor 103. The imaging optical system includes a plurality of lens groups, and is driven by the focus lens driving mechanism 121 so as to move the focal position near the imaging surface of the imaging element 103 along the Z direction. In this embodiment, the operation of adjusting the focus position to the subject is also referred to as focusing.

  The lens CPU 122 receives the focus adjustment information sent from the CPU 110 of the camera body 100 via the electric contact 124, and drives the focus lens drive mechanism 121 based on the focus adjustment information.

  The aperture driving mechanism 123 has a mechanism for driving the aperture 125 and its actuator, and drives the aperture 125 according to an instruction from the lens CPU 110.

(Structure of imaging device)
Subsequently, a pixel structure of the image sensor 103 in the present embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating a pixel structure of the image sensor 103. FIG. 3A is an explanatory diagram of a pixel array of the image sensor 103, and FIG. 3B is a cross-sectional view of the pixel 210G taken along a ZX plane.

  FIG. 3A shows a pixel array of the imaging element 103 (two-dimensional CMOS sensor) in a pixel range of 4 rows × 4 columns. The Bayer array is adopted as the array of the pixels 210. As two pixels in the diagonal direction, a pixel 210G having a spectral sensitivity of G (green) is arranged. Further, as the other two pixels, a pixel 210R having an R (red) spectral sensitivity and a pixel 210B having a B (blue) spectral sensitivity are arranged. Each of the pixels 210R, 210G, and 210B has two sub-pixels 201a and 201b for pupil division. The sub-pixel 201a is a first pixel that receives a light beam that has passed through the first strong pupil region of the imaging optical system. The sub-pixel 201b is a second pixel that receives a light beam that has passed through the second pupil region of the imaging optical system. Each pixel functions as an imaging pixel and a focus detection pixel.

  Regarding the coordinate axes indicated by X, Y, and Z in FIG. 3A, the XY plane is located in the plane of FIG. 3A, and the Z axis is an axis perpendicular to the plane of the paper. The sub-pixels 201a and 201b are arranged along a direction parallel to the X axis.

  Regarding the coordinate axes indicated by X, Y, and Z in FIG. 3B, the XZ plane is located in the plane of the paper of FIG. 3B, and the Y axis is an axis perpendicular to the plane of the paper. The detector has a photodiode composed of a p-type layer 200 and an n-type layer. The micro lens 202 is arranged at a position separated from the light receiving surface by a predetermined distance in the Z-axis direction. The micro lens 202 is formed on the color filter 203.

  In this embodiment, all the pixels 210R, 210G, and 210B of the image sensor 103 are provided with pupil-dividing sub-pixels 201a and 201b. The sub-pixels 201a and 201 are used as focus detection pixels. However, the present embodiment is not limited to this, and a configuration in which pupil-divideable focus detection pixels are provided in only some of all pixels may be employed.

(Concept of pupil division function by image sensor)
Subsequently, the pupil division function of the image sensor 103 will be described with reference to FIG. FIG. 4 is an explanatory diagram of a pupil division function of the image sensor 103, and shows a state of pupil division in one pixel portion.

  Regarding the coordinate axes (X, Y, Z) of the pixel portion shown in the lower part of FIG. 4, the XZ plane is located in the plane of FIG. 4, and the Y axis is an axis perpendicular to the plane of the paper. The pixel includes a p-type layer 300 and n-type layers 301a and 301b. The p-type layer 300 and the n-type layer 301a constitute the sub-pixel 201a in FIG. 3, and the p-type layer 300 and the n-type layer 301b constitute the sub-pixel 201b. The micro lens 304 is arranged on the Z axis.

  Also, an exit pupil 302 and a frame 303 (for example, an aperture frame or a lens frame) are shown in the upper part of FIG. Regarding the coordinate axes (X, Y, Z) shown in the upper part of FIG. 4, the XY plane is located in the plane of the paper of FIG. 4, and the Z axis is an axis perpendicular to the plane of the paper.

  Since one pixel is provided with n-type layers 301a and 301b embedded in the p-type layer 300, two sub-pixels are formed. The two sub-pixels are regularly arranged along the x direction. Further, since the two sub-pixels are eccentric in the + X direction and the −X direction, pupil division can be performed using one microlens 304. 4, a pupil 302a of the image signal A and a pupil 302b of the image signal B are shown as the exit pupils 302. The image signal A is a first image signal acquired by a sub-pixel corresponding to the n-type layer 301a eccentric in the −X direction. The image signal B is a second image signal acquired by the sub-pixel corresponding to the n-type layer 301b decentered in the + X direction.

  As described above, in the present embodiment, the image signal A (first signal) is obtained by arranging a plurality of sub-pixels 301a regularly in the X direction as shown in FIG. This is the obtained signal. That is, the image signal A is one of the image signals obtained from a pair of light beams that pass through different exit pupil regions of the imaging optical system and have different base lengths according to the aperture value. The image signal B (second signal) is a signal obtained by arranging a plurality of sub-pixels 301b regularly in the X direction as shown in FIG. 3A and obtaining the plurality of sub-pixels 301b. . That is, the image signal B is the other of the image signals obtained from a pair of light beams that pass through different exit pupil regions of the imaging optical system and have different base lengths according to the aperture value.

  The CPU 110 calculates a relative image shift amount between the image signal A and the image signal B by the phase difference detection unit 111, and calculates a defocus amount by the defocus amount conversion unit 112. In this embodiment, the configuration corresponding to the subject having the luminance distribution in the X direction has been described. However, in the present embodiment, it is also possible to adopt a configuration corresponding to a subject having a luminance distribution in the Y direction by developing a similar configuration in the Y direction.

(Overall flow of automatic selection servo AF)
Next, the overall flow of the automatic selection servo AF will be described with reference to the flowchart of FIG. In the automatic selection servo AF according to the present embodiment, the imaging apparatus automatically selects one focus detection area to be used for focus adjustment from a plurality of focus detection areas, mainly for photographing a moving object. Is a mode in which the process of performing focus adjustment using is repeated.

  The outline of the automatic selection servo AF of this embodiment is as follows. First, a certain subject is determined as a main subject by a first subject determination process (described later). In a second subject determination process (described later) performed separately, a subject different from the main subject is determined as a subject (also referred to as a second subject) that is a candidate for the main subject. Then, in the transfer determination process (described later), whether the focus position should be adjusted to the second object more than the first object determined at that time, that is, whether the second object is the main object Judge. A process is performed in which the subject in accordance with the result of this determination is kept in focus with the main subject.

  First, in S101, the CPU 110 acquires an image signal from the image sensor 103.

  Next, in step S102, based on the image signal acquired in step S101, the CPU 110 performs focus detection for each frame of the entire screen.

  In S103, the CPU 110 selects the closest position from the focus detection of each frame performed in S102, and the CPU 110 sets the AF frame at the closest subject position. However, if the AF frame has already been set at the closest subject position, the AF frame may be fixed to the same AF frame as the previous time. You may reset it.

  In S104, it is determined whether the focus detection result of the AF frame position calculated by the CPU 110 is within the focusing range. If it is within the focusing range, the process proceeds to S108, and if it is outside the focusing range, the process proceeds to S105.

  In S105, the focus detection result of the AF frame position calculated by the CPU 110 is converted into a lens drive amount by the focus adjustment unit 116, and the focus of the lens 120 is adjusted.

  In S108, the CPU 110 performs the first subject determination process, and determines a region to start the tracking process. The details of this process will be described later (first subject determination process 1).

  In S109, the CPU 110 sets an AF frame at the first subject position determined in S108.

  In S110, the CPU 110 acquires an image signal from the image sensor 103.

  In S111, the CPU 110 performs a tracking process based on the image signal acquired in S110. The subject to be tracked is a first subject and a second subject candidate.

  In S112, the CPU 110 performs a focus detection process based on the image signal acquired from the image sensor 103 in S110.

  In S113, the CPU 110 performs a second subject determination process from the second subject candidates detected in S111. Details of this processing will be described later (second subject determination processing).

  In S114, the CPU 110 determines whether or not the second subject has been transferred as the first subject. The details of this process will be described later (transit determination process).

  In S115, as a result of the transfer determination performed in S114, the CPU 110 proceeds to S116 if it is determined to transfer, and proceeds to S118 if it is determined not to transfer.

  In S116, CPU 110 determines the second subject as a new first subject.

  In S117, the CPU 110 performs a second subject determination process.

  In S118, the focus detection result of the AF frame position calculated by the CPU 110 is converted into a lens drive amount by the focus adjustment unit 116, and the focus adjustment of the lens 120 is controlled.

  When the second object is determined as a new first object, the AF frame position is reset, and the focus detection result is calculated again. The amount may be converted into an amount and the focus of the lens 120 may be adjusted.

  In S119, the CPU 110 determines whether or not the release switch (SW2) is pressed. If the release switch (SW2) is pressed, the process proceeds to S120, and if not, the process proceeds to S121.

  In S120, the memory circuit 101 records the image signal acquired by the image sensor 103.

  In S121, the CPU 110 determines whether or not the AF switch (SW1) is being pressed. If the AF switch (SW1) is being pressed, the process proceeds to S109, and if not, the entire flow of the automatic selection servo AF is ended.

  As described above, in the automatic selection servo AF of this embodiment, first, a certain subject is determined as the first subject. In the second subject determination process, a subject different from the first subject is determined as a main subject candidate. When it is determined that the second subject should be focused more than the first subject, the second subject is set as the first subject which is the main subject. Accordingly, even when the determined main subject is different from the subject intended by the user, or when the subject to be determined as the main subject changes according to the movement of the subject, the appropriate subject is determined as the main subject. be able to.

(First subject determination processing 1)
Here, the details of the first subject determination processing 1 (S114) in the flowchart of FIG. 5 will be described with reference to the flowchart of FIG. As described above, the first subject determination process is a process of determining the first subject that is the main subject. In step S901, the CPU 110 detects a subject from an image signal acquired from the image sensor 103.

  In S902, the CPU 110 determines whether a face has been detected as a subject in S901. If a face has been detected, the process proceeds to S903, and if no face has been detected, the process proceeds to S904.

  If the CPU 110 determines in step S <b> 903 that there is a face in step S <b> 901, the CPU 110 compares the difference between the defocus amount of the in-focus position and the defocus amount of the position corresponding to the face with a first threshold. The difference between the defocus amount at the in-focus position and the defocus amount at the position corresponding to the face is determined by If it is less than the first threshold, the process proceeds to S909, and if it is more than the first threshold, the process proceeds to S904.

  Note that the wider the threshold value, the easier it is to focus on the face subject, but there is also a possibility of focusing on a face subject that the user does not intend, such as a background audience. Therefore, the threshold at this time may be set to the same threshold on the infinity side and the closest side from the in-focus position, or may be set narrower on the infinity side and wider on the closest side.

  In step S904, the CPU 110 calculates the saliency from the image signal acquired from the image sensor 103, and determines whether there is a subject having a salient feature (also referred to as a salient subject) based on the saliency. Do. The saliency of this embodiment is the degree of saliency of the feature, and it is assumed that the more distinct the background is, the higher the saliency is. When the saliency is equal to or greater than a second predetermined threshold, it is determined that there is a salient subject.

  In this embodiment, the saliency is determined based on hue, saturation, and luminance. For example, in a pink flower scene surrounded by green leaves, the position of the flower is the main subject, and the flower can be inferred to be the main subject by determining the saliency. Also, a ship at sea, a bird flying in the sky, and the like can be inferred to be the main subject by determining the saliency for the same reason. As described above, the saliency can be one of the influential indices in determining the main subject. Therefore, in this embodiment, the salient subject is set as a candidate for determining the first subject. The saliency calculation processing method in S904 will be described later.

In S905, the CPU 110 determines whether or not there is a prominent subject in S904. If a prominent subject is found, the process proceeds to S906; otherwise, the process proceeds to S907.
In step S906, the CPU 110 compares the difference between the defocus amount at the in-focus position and the defocus amount of the prominent subject with a third threshold value, and determines whether the prominent subject is near the in-focus position. I do. This is because a subject such as a signboard that the user does not want to be the main subject but has a high degree of saliency may be in the background or at the edge of a plane, and therefore it is desired to exclude a prominent subject at a position largely deviating from the current focus position. When the difference between the defocus amount of the focused position and the defocus amount of the prominent subject is less than the third threshold, and the position of the prominent subject on the plane is near the position of the in-focus position on the plane, The process proceeds to S908. Here, when the distance between the center coordinate position of the prominent subject on the XY axis plane and the coordinate position of the in-focus position when the defocus direction (depth direction) is the Z-axis direction is smaller than the fourth threshold value. Is determined to be near. If a plurality of salient subjects exist in the threshold range, the subject with the highest saliency is selected, and the process proceeds to S908. The difference between the defocus amount of the focused position and the defocus amount of the salient subject is equal to or greater than a third threshold value, or not close to the plane position of the salient subject and the focus position on the plane, or both are satisfied. In this case, the process proceeds to S907.

  In step S907, the CPU 110 determines the first subject to be a position where the focus detection result obtained in step S103 indicates the closest distance.

  In S908, the CPU 110 determines the first subject as the prominent subject determined in S904.

  In S909, the CPU 110 determines the first subject as a face.

(Color saliency calculation processing)
The color saliency calculation processing based on the image signal will be described with reference to FIGS.

  First, in S601 of FIG. 6, the CPU 110 determines an evaluation area in which an image signal is evaluated. The evaluation area is an area corresponding to a subject whose saliency is to be evaluated. In FIG. 7, reference numeral 701 denotes an input image, 702 denotes an evaluation pixel, and 703 denotes an evaluation area candidate. As shown in FIGS. 7A to 7C, each evaluation area candidate is calculated in a plurality of sizes. The size of the evaluation area candidate is calculated by multiplying a predetermined value by n, and is set to, for example, 16, 32, or 64 pixels. Evaluation pixels are sequentially selected from all the pixels of the input image by raster processing.

  Next, in S602 of FIG. 6, the CPU 110 determines an outer evaluation area with respect to the evaluation area. The outer evaluation area is an area corresponding to the background. In FIG. 8, reference numeral 801 denotes an input image, 802 denotes an evaluation pixel, 803 denotes an evaluation area, and 804 denotes an outer evaluation area. As shown in FIG. 8, the outer evaluation area has the same center position as the evaluation area, the outer diameter size is a value obtained by adding a predetermined value to the size of the evaluation area, and is a donut-shaped area excluding the evaluation area. The predetermined value to be added is determined by the ratio of the input image, and is, for example, 10% of the horizontal size of the input image.

  Next, in S603 of FIG. 6, the CPU 110 calculates the difference between the image signals corresponding to the evaluation area and the outer evaluation area as a color saliency evaluation value. The difference D is calculated by the following equation.

d _H is the difference calculated from the hue information of the evaluation area and the outer evaluation area, p _Hi is the number of pixels whose hue information of the evaluation area is i, and q _Hi is i of the outer evaluation area and i Represents the number of pixels. d _S , p _Si , and q _Si are the same values for the saturation information, and d _V , p _Vi , and q _Vi are the same values for the luminance information. M is a maximum value that can be taken by hue information, saturation information, and luminance information.

  Next, in S604 of FIG. 6, the CPU 110 determines whether or not there is an unprocessed evaluation area. If there is an unprocessed evaluation area, the process returns to S601 of FIG. 6 to repeat a series of processes, and if not, the process is terminated.

  Through the above processing, the color saliency can be calculated based on the assumption that the difference between the image signal of the evaluation area and the image signal of the outer evaluation area in the subject area increases.

(Second subject determination process)
Here, the details of the second subject determination processing (S113 and S117) in the flowchart of FIG. 5 will be described with reference to FIG. As described above, the second subject determination process is a process of determining the second subject as a main subject candidate.

  First, in step S <b> 1001, the CPU 110 performs subject detection using the subject detection unit 111 from an image signal acquired from the image sensor 103.

  Next, in S1002, as a result of the detection in S1001, the CPU 110 determines whether or not there is a subject that is different from the first subject and is a face.

  If it is determined in S1002 that there is a subject that is different from the first subject and is a face, the CPU 110 determines the face as the second subject in S1003. Then, the second subject determination process ends, with the second subject being detected. In this way, a subject whose face is likely to be the main subject is determined as a main subject candidate. Accordingly, when the first subject is not the main subject intended by the user, or when a subject different from the first subject is more suitable for the main subject, another subject more suitable for the main subject is regarded as one subject. Can be determined.

  If it is not determined in S1002 that there is a subject that is different from the first subject and is a face, the CPU 110 does not determine the second subject and does not detect the second subject in S1004. As the state, the second subject determination processing ends. If no face is detected as a subject different from the first subject, it is assumed that no subject other than the first subject is suitable for the main subject. For this reason, by not determining a subject that is a candidate for the main subject, it is possible to continue focusing on a subject that is more suitable for the main subject.

(Transfer judgment processing)
Here, the details of the transfer determination processing S114 in the flowchart of FIG. 5 will be described with reference to FIG. As described above, the transfer determination process is a process of determining whether the second subject, not the first subject at that time, is a subject to be focused.

  In S1101, the CPU 110 determines whether the subject determined in the first subject determination process in S108 is a face.

  If it is determined in S1101 that the first subject is a face, the CPU 110 sets the transfer determination result to FALSE in S1102, and ends the transfer determination processing.

  When the first subject is a face, the reason why the first subject is not changed is that when the subject that is currently following is a face, it is highly likely that the subject is the main subject, and it is better not to change the subject. That's why.

  If it is determined in S1101 that the first subject is not a face, the CPU 110 proceeds to S1103 and determines whether or not the second subject has been detected in the second subject determination processing in S113.

  If it is determined in step S1103 that the second subject has not been detected, the CPU 110 determines that the transfer target does not exist, and sets the transfer determination result to FALSE in step S1102, and ends the transfer determination process.

  If it is determined in S1103 that the second subject has been detected, the CPU 110 determines in S1104 whether the second subject is a subject closer to the first subject.

  If it is determined in S1104 that the second subject is a subject closer to the first subject, the CPU 110 sets the transfer determination result to TRUE in S1106, and ends the transfer determination processing.

  In this case, the reason why the second subject is changed to the first subject is that the second subject is a subject closer to the first subject and the first subject is in the background. This is because there is a high possibility that the closest subject is the main subject.

  If it is determined in S1104 that the second object is not the object on the near side of the first object, the CPU 110 determines in S1105 that the second object is an infinite object within a predetermined defocus from the first object. It is determined whether there is.

  In step S1105, when it is determined that the difference between the defocus amounts of the second subject and the first subject is less than the fifth threshold value and the second subject is a subject on the infinite side of the first subject, The CPU 110 sets the transfer determination result to TRUE in S1106. Then, the transfer determination processing ends.

  In step S <b> 1105, it is determined that the difference between the defocus amounts of the second subject and the first subject is equal to or greater than a fifth threshold value, and the second subject satisfies either the closest side of the first subject or both of them. In this case, the transfer determination result is set to FALSE in S1102. Then, the transfer determination processing ends.

  Here, the reason for changing the second subject to the first subject is as follows. When the difference between the defocus amounts of the second subject and the first subject is less than a fifth threshold value and the second subject is a subject on the infinite side of the first subject, for example, if the first subject is on the ground It is conceivable that the person is closer than the person. This is because in such a case, the face as the second subject is likely to be the main subject.

  On the other hand, when the difference between the defocus amounts of the second subject and the first subject is equal to or greater than the fifth threshold and the second subject is a subject on the infinite side of the first subject, the subject is in the background or the like. This is because there is a high possibility that the face is not the main subject.

  In the processing in S1104 and S1105, the CPU 110 determines from the focus detection results calculated in the focus detection processing in S112 by using the respective focus detection results corresponding to the first subject position and the second subject position.

  As described above, in the present embodiment, erroneous detection of the main subject can be reduced by various measures, so that it is possible to keep focusing on the correct main subject. Furthermore, by dynamically reselecting the main subject as a result of the focus detection, it is possible to keep focusing on the correct main subject.

[Example 2]
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. The present embodiment is different from the first embodiment in that, for example, it is determined whether or not the subject is a moving object in a focus detection process, and the result of the determination is considered in a changeover determination. Further, for example, the second embodiment differs from the first embodiment in that the second subject determination process takes into account saliency. The description of the same configuration as that of the first embodiment will be omitted, and the following description will focus on parts that differ from the first embodiment. In the present embodiment, the CPU 110 is configured to perform a moving object prediction process and a moving object determination process (also referred to as moving object detection), which will be described later, in addition to the process described in the first embodiment.

(Focus detection processing)
Here, details of the focus detection processing S112 of the second embodiment in the flowchart of FIG. 5 will be described with reference to FIG.

  In step S1201, the CPU 110 performs a focus detection process based on the image signal acquired from the image sensor 103 in step S110.

  In step S1202, the CPU 110 selects a focus detection result corresponding to the first subject position based on the focus detection result obtained in step S1201.

  In S1203, the CPU 110 performs a first moving object determination process based on the focus detection result corresponding to the first subject position selected in S1202.

  In the first moving object determination process, the focus detection result corresponding to the first subject position is stored as a history, and it is determined whether the object is a moving object based on the change. The result of the determination is also referred to as a first moving object detection result.

  Note that the moving object determination process may employ any method as long as it is a method for determining whether or not the subject is a moving object. For example, when the focus detection result changes in the same direction for a certain period or more and the amount of change exceeds a predetermined value, it can be determined that the object is a moving object.

  If it is determined in step S1204 that the first object is a moving object, in step S1205, the CPU 110 performs the first moving object prediction process based on the focus detection result corresponding to the first object position selected in step S1202. Do.

  In the first moving object prediction process, it is possible to predict the in-focus position of the first subject at an arbitrary time from the history of the focus detection results corresponding to the first subject position. The in-focus position of the first subject at the accumulation time is predicted.

  In addition, as a method of the moving object prediction processing, any method may be adopted as long as the method can predict the in-focus position of the subject at an arbitrary time. For example, the prediction can be performed using a statistical operation disclosed in Japanese Patent Application Laid-Open No. 2001-21794.

  In S1206, it is determined whether or not the second subject has been determined as a result of the second subject determination processing S113.

  If the second subject has been determined in S1206, the CPU 110 selects a focus detection result corresponding to the second subject position based on the focus detection result acquired in S1201 in S1207.

  In S1208, the CPU 110 performs a second moving object determination process based on the focus detection result corresponding to the second subject position selected in S1207.

  In the second moving object determination process, the focus detection result corresponding to the second object position is stored as a history, and it is determined whether the object is a moving object based on the change. The result of the determination is also referred to as a second moving object detection result.

  If it is determined in S1209 that the second object is a moving object, in S1210, the CPU 110 executes the second moving object prediction process based on the focus detection result corresponding to the second object position selected in S1207. Do.

  In the second moving object prediction processing, it is possible to predict the in-focus position of the second object at an arbitrary time from the history of the focus detection result corresponding to the second object position. The in-focus position of the subject is predicted.

(Second subject determination process)
Here, details of the second subject determination processing (S113 and S117) of the second embodiment in the flowchart of FIG. 5 will be described with reference to FIG. As described above, in the second subject determination processing according to the second embodiment, the saliency is also considered.

  First, in step S1301, the CPU 110 performs subject detection using the subject detection unit 111 from the image signal acquired from the image sensor 103.

  Next, in S1302, the CPU 110 determines whether or not a face has been detected as a result of the detection in S1301.

  If it is determined in S1302 that there is a face, in S1303, the CPU 110 compares the defocus amount of the first subject with the defocus amount of the face position. If the difference is less than the fifth threshold, the face is determined as the second subject in S1304, and the second subject is determined to be in the detected state, and the second subject determination processing ends.

  If the difference between the defocus amount of the first subject and the defocus amount of the face position is equal to or more than the fifth threshold, the detected face is not determined as the second subject, and the process proceeds to S1305.

  The wider this threshold value is, the easier it is to focus on the face subject, but there is also a possibility that the focus will be on a face subject that the user does not intend, such as a background audience. Therefore, the threshold at this time may be set to the same threshold on the infinity side and the closest side from the in-focus position, or may be set narrower on the infinity side and wider on the closest side.

  If it is not determined in S1302 that there is a face, the CPU 110 performs saliency determination from the image signal acquired from the image sensor 103 in S1305.

  In S1306, the CPU 110 determines whether or not there is a prominent subject in S1305. If a prominent subject is found, the process proceeds to S1307, and if not, the process proceeds to S1309.

  In S1307, the CPU 110 compares the defocus amount of the first subject with the defocus amount of the prominent subject. If the difference is less than the sixth threshold value, the prominent subject is determined to be the second subject in S1308, and the second subject determination process ends with the second subject being detected.

  The focus detection results of the first subject and the prominent subject are compared, and if the defocus difference is equal to or larger than the sixth threshold, the prominent subject is not determined as the second subject.

  The wider this threshold value is, the easier it is to focus on a prominent subject, but there is also a possibility of focusing on an unintended subject such as a signboard with a prominent background.

  Therefore, the threshold value at this time may be set to the same threshold value on the infinity side and the closest side from the first subject, or may be set narrower on the infinity side and wider on the closest side.

  In S1309, the CPU 110 determines whether the first subject exists in the central area on the screen.

  If it is determined in S1309 that the first subject exists in the center area of the screen, in S1310, the CPU 110 determines the position at which the focus detection result indicates the closest distance as the second subject. Then, the second subject determination process ends with the second subject detected.

  This is because if the first subject is located in the central area, there is a possibility that the first subject is a main subject if there is a close subject in other areas.

  If it is determined in S1309 that the first subject does not exist in the center area of the screen, in S1311, the CPU 110 determines the center position of the screen as the second subject and sets the second subject as a detected state. The second subject determination processing ends.

  This is because when the first subject does not exist in the central area, the main subject may exist in the central area.

  Here, in order to change the ease of determining the center of the screen as the second subject according to the range of the central region, the range of the central region may be narrowed or widened.

(Transfer judgment processing)
Here, the details of the transfer determination processing S114 of the second embodiment in the flowchart of FIG. 5 will be described with reference to FIG. In the second embodiment, as described above, the transfer determination is performed based on the result of the moving object determination in the focus detection processing.

  In S1401, the CPU 110 determines whether the first subject is a moving object as a result of the first moving object determination process in S1203.

  If it is determined in S1401 that the first subject is a moving object, the CPU 110 sets the transfer determination result to FALSE in S1402, and ends the transfer determination processing.

  When the first subject is a moving body, the reason why the first subject is not changed is that when the currently following subject is a moving body, it is highly likely that the first subject is the main subject, and it is better not to change the subject. That's why.

  If it is determined in S1401 that the first subject is not a moving object, the CPU 110 proceeds to S1403, and determines whether the second subject is a moving object as a result of the second moving object determination process in S1208.

  If it is determined in S1403 that the second subject is a moving object, the CPU 110 sets the transfer determination result to TRUE in S1404, and ends the transfer determination processing.

  If it is determined in S1403 that the second subject is not a moving object, the CPU 110 sets the transfer determination result to FALSE in S1402, and ends the transfer determination processing.

  In this case, the reason why the second subject is changed to the first subject is that the subject that is currently following is not a moving subject and the second subject is a moving subject, and the second subject is the main subject. This is because the possibility is high.

  In S118 in the second embodiment, the focus adjustment unit 116 converts the focus position of the AF frame position of the subject selected as the first subject calculated by the CPU 110 into a lens drive amount, and drives the focus lens 126. Focus adjustment.

  As described above, according to the present invention, it is possible to continuously focus on the correct main subject by determining the moving subject from the focus detection result and dynamically reselecting the main subject for the subject determined to be the moving subject. it can.

[Example 3]
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. The description of the same configuration as that of the second embodiment will be omitted, and the following description will focus on parts that are different from the second embodiment.

(Transfer judgment processing)
Here, details of the transfer determination processing S114 of the third embodiment in the flowchart of FIG. 5 will be described with reference to FIG.

  In S1501, the CPU 110 determines whether or not the first subject is a moving object as a result of the first moving object determination process in S1203.

  If it is determined in S1501 that the first subject is a moving object, in S1502, the CPU 110 determines whether the subject determined in the first subject determination processing in S108 is a face.

  If it is determined in S1502 that the first subject is a face, in S1503, the CPU 110 determines whether the second subject is a moving object as a result of the second moving object determination process in S1208.

  If it is determined in S1503 that the second subject is a moving object, in S1504, the CPU 110 determines whether the subject determined in the second subject determination processing in S113 is a face.

  If it is determined in S1503 that the second subject is not a moving object, the CPU 110 sets the transfer determination result to FALSE in S1507, and ends the transfer determination processing.

  If it is determined in step S1504 that the second subject is a face, in step S1505, the CPU 110 compares the defocus amount of the second subject with the defocus amount of the second subject, thereby obtaining the second subject. It is determined whether the subject is closer than the first subject. If it is determined in S1504 that the second subject is not a face, the CPU 110 sets the transfer determination result to FALSE in S1507, and ends the transfer determination processing.

  In addition, when determining whether the second subject is closer to the first subject, it can be performed using the focus detection result of the position of each target subject, but it is determined in each moving object prediction process. The determination may be made based on the in-focus position. Also in the determination in the subsequent processing, the determination may be made using the in-focus position obtained in the moving object prediction processing, instead of the defocus amount which is the focus detection result.

  If it is determined in S1505 that the second subject is closer to the first subject than the first subject, the CPU 110 sets the transfer determination result to TRUE in S1506, and ends the transfer determination processing.

  If it is determined in S1505 that the second subject is not closer than the first subject, the CPU 110 sets the transfer determination result to FALSE in S1506, and ends the transfer determination processing.

  In this case, the reason for changing the second subject to the first subject is that when the second subject is closer to the first subject, the subject in front is more likely to be the main subject. That's why.

  If it is determined in S1502 that the first subject is not a face, the CPU 110 determines in S1508 whether the second subject is a moving object as a result of the second moving object determination process in S1208.

  If it is determined in S1508 that the second subject is a moving object, in S1509, the CPU 110 determines whether the subject determined in the second subject determination processing in S113 is a face.

  If it is determined in S1508 that the second subject is not a moving object, the CPU 110 sets the transfer determination result to FALSE in S1513, and ends the transfer determination processing.

  If it is determined in S1509 that the second subject is a face, the CPU 110 determines in S1510 that the difference between the defocus amount of the second subject and the defocus amount of the first subject is less than the seventh threshold. It is determined whether there is.

  If it is determined in S1509 that the second subject is not a face, the CPU 110 sets the transfer determination result to FALSE in S1513, and ends the transfer determination processing.

  If it is determined in S1510 that the difference between the defocus amount of the second subject and the defocus amount of the first subject is less than the seventh threshold, the CPU 110 determines in S1511 that the second subject is the first subject. It is determined whether or not the object exists in a region near a predetermined position of the subject.

  If it is determined in step S1505 that the difference between the defocus amount of the second subject and the defocus amount of the first subject is equal to or greater than the seventh threshold, the CPU 110 sets the transfer determination result to FALSE in step S1513 and sets the transfer to FALSE. The determination processing ends.

  If it is determined in step S1511 that the position of the second object on the plane is in the vicinity of the position of the first object on the plane, the CPU 110 sets the transfer determination result to TRUE in step S1512, and determines the transfer. The process ends. Here, the distance between the position of the center coordinates of the first subject and the position of the center coordinates of the first subject on the XY-axis plane when the defocus direction (the depth direction) is the Z-axis direction is an eighth threshold value. If the distance is less than the distance, it is determined that the position of the second subject on the plane is in the vicinity of the position of the first subject on the plane.

  In this case, the reason why the second subject is changed to the first subject is that there is a high possibility that the second subject and the first subject are the same subject, and in that case, it is better to focus on the face. That's why.

  If it is determined in S1501 that the first subject is not a moving object, in S1514, the CPU 110 determines whether the subject determined in the first subject determination processing in S108 is a face.

  If it is determined in S1514 that the first subject is a face, in S1515, the CPU 110 determines whether or not the second subject is a moving object as a result of the second moving object determination processing in S1208.

  If it is determined in S1515 that the second subject is a moving object, in S1516, the CPU 110 determines whether the subject determined in the second subject determination processing in S113 is a face.

  If it is determined in S1515 that the second subject is not a moving object, the CPU 110 sets the transfer determination result to FALSE in S1519, and ends the transfer determination processing.

  If it is determined in S1516 that the second subject is a face, the CPU 110 sets the transfer determination result to TRUE in S1517, and ends the transfer determination processing.

  In this case, the reason why the second subject is changed to the first subject is that when both the second subject and the first subject are faces, the face that is a moving object may be the main subject. Because it is expensive.

  If it is determined in S1516 that the second subject is not a face, the CPU 110 determines whether the face priority setting is ON in S1518.

  Note that the face priority setting is a camera setting value stored in the memory circuit 101 according to a user's designation. When the face priority setting is ON, when a face is detected as a subject, the CPU 110 controls to focus on the face with priority.

  When the face priority setting is ON, it is also possible to display a frame indicating the position of the face on the display 105 when the subject detection unit 111 detects a face.

  If it is determined in S1518 that the face priority setting is ON, the CPU 110 sets the transfer determination result to FALSE in S1519, and ends the transfer determination processing.

  If it is determined in S1518 that the face priority setting is not ON, the CPU 110 sets the transfer determination result to TRUE in S1517, and ends the transfer determination processing.

  In this case, the reason why the second subject is changed to the first subject is that, if the face priority setting is not ON, a moving object that is not a face is a main subject than a subject that is a face but is not a moving object. This is because the nature is high.

  If it is determined in S1514 that the first subject is not a face, in S1520, the CPU 110 determines whether the second subject is a moving object as a result of the second moving object determination process in S1208.

  If it is determined in S1520 that the second subject is a moving object, the CPU 110 sets the transfer determination result to TRUE in S1521, and ends the transfer determination processing.

  In this case, the reason why the second subject is changed to the first subject is that the moving subject is more likely to be the main subject than the non-moving subject.

  If it is determined in S1520 that the second subject is not a moving object, in S1522, the CPU 110 determines whether the subject determined in the second subject determination processing in S113 is a face.

  If it is determined in S1522 that the second subject is a face, the CPU 110 determines whether the face priority setting is ON in S1523.

  If it is determined in S1523 that the face priority setting is ON, the CPU 110 sets the transfer determination result to TRUE in S1521, and ends the transfer determination processing.

  In this case, the reason why the second subject is changed to the first subject is that when the face priority setting is ON, the face is more likely to be the main subject than the non-face subject.

  If it is determined in S1523 that the face priority setting is not ON, the CPU 110 sets the transfer determination result to FALSE in S1524, and ends the transfer determination processing.

  If it is determined in S1522 that the second subject is not a face, the CPU 110 sets the transfer determination result to FALSE in S1524, and ends the transfer determination processing.

  As described above, according to the present invention, it is possible to continuously focus on the correct main subject by determining the moving subject from the focus detection result and dynamically reselecting the main subject for the subject determined to be the moving subject. it can.

  Further, by determining whether or not to reselect the main subject according to whether or not the subject is a face, it is possible to prevent the main subject from being reselected by mistake.

[Other Examples]
Note that, in the above-described embodiment, an example in which the determination is mainly performed using the defocus amount has been described. For example, information corresponding to the image shift amount and the distance may be used instead.

  In the above-described embodiment, an example in which a face is detected has been described. However, the present invention is not limited to a face as long as it is a predetermined subject detected by the tracking processing. For example, a template for tracking may be set in advance corresponding to an object to be photographed, and an object may be detected instead of the face in the present embodiment.

  The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

REFERENCE SIGNS LIST 100 Camera body 103 Image sensor 110 CPU
111 subject detection unit 112 phase difference detection means 113 first subject determination processing unit 114 second subject determination processing unit 115 transfer determination processing unit 116 focus adjustment unit 121 focus lens driving mechanism

In order to achieve the above object, the present invention provides a focus detection unit that performs focus detection based on a signal output from an image sensor, a control unit that controls driving of a focus lens, and the control unit controls the focus detection. after controlling the driving of the focus lens so focuses the detected object based on the results, a first determining means for determining a first object to be follow the focus lens, corresponding to the subject when the difference of the focus detection result of the position that is equal to or greater than a threshold, and determining the first object based on the saliency, the difference of the focus detection result of the position corresponding to the object is less than the threshold value case, the irrespective of the prominence, characterized by being configured to have a first determination means for determining the object as the first object.

Claims (8)

Focus detection means for performing focus detection based on a signal output from the image sensor,
Control means for controlling driving of the focus lens;
First determining means for controlling the driving of the focus lens so that the control means focuses on a subject detected based on the result of the focus detection, and then determining a main subject to be followed by the focus lens. An imaging apparatus comprising: a first determination unit configured to determine a first subject based on a degree of saliency of a subject whose difference in focus detection result from a position corresponding to the subject is less than a threshold.   2. The apparatus according to claim 1, wherein the first determination unit determines, as the first object, a subject corresponding to an area in which the saliency is equal to or greater than a third threshold value among a plurality of areas. 3. Imaging device.   The first determining unit determines the first subject based on a focus detection result when there is no region in which the saliency is equal to or greater than a third threshold value among a plurality of regions. The imaging device according to claim 1 or 2, wherein   The third threshold value when the saliency is a second value larger than the first value is larger than the third threshold value when the saliency is a first value. Item 4. The imaging device according to Item 3.   When a face is detected as a subject, the first determining means determines a first value based on the saliency when a defocus amount in an area corresponding to the face is equal to or greater than a first threshold. The imaging device according to any one of claims 1 to 4, wherein a subject is determined.   When a face is detected as a subject, and the defocus amount in an area corresponding to the face is less than a first threshold, the first determining unit determines the subject corresponding to the face as a first subject. The imaging apparatus according to claim 5, wherein the subject is determined as the subject.   The imaging apparatus according to any one of claims 1 to 6, wherein the saliency is an evaluation value calculated based on at least one of hue, saturation, and luminance of the image signal. apparatus. A focus detection step of performing focus detection based on a signal output from the image sensor,
A control step of controlling driving of the focus lens;
A first determining step of controlling the driving of the focus lens so as to focus on a subject detected based on the result of the focus detection in the control step, and then determining a main subject to be followed by the focus lens. A first determining step of determining a first subject based on the degree of saliency of the subject whose difference in focus detection result from the position corresponding to the subject is less than a threshold value. Method.

Images