JP2018017876A - Imaging apparatus and control method of the same, and image processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently detect a subject included in a scene, and improve accuracy of a focus adjustment relative to a main subject.SOLUTION: An imaging apparatus includes: focus detection means that detects a defocus amount for each of a plurality of predetermined focus detection areas from an image signal output from an imaging device; generation means that generates distance distribution information on the basis of the defocus amount; focus adjustment means that conducts focus adjustments on the basis of the distance distribution information and the defocus amount; and control means that, when generating the distance distribution information by the generation means, performs control so as to take a picture by setting an aperture to be included in an imaging optical system to a first aperture value to be a predetermined first depth of field, and when conducting the focus adjustment by the focus adjustment means, performs control so as to take the picture by setting an aperture to a second aperture value to be a second depth of filed shallower than the first depth of field.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus, a control method therefor, and an image processing apparatus and method, and more particularly to a focus adjustment technique based on imaging plane phase difference AF.

  Shooting while looking at a live view (LV) screen, which has been increasing in demand in recent years, requires a quality of focusing operation in addition to the responsiveness to focus quickly as AF control. Recently, an AF method capable of focusing at high speed and high quality during LV shooting has been proposed. One of the methods is AF of the imaging plane phase difference detection method in which AF by the phase difference detection method is performed using an image sensor for taking an image.

  As one imaging plane phase difference detection method, in Patent Document 1, a photodiode is divided in one pixel condensed by one microlens, and each photodiode passes through a different pupil region of the imaging lens. Is configured to receive light. Then, by comparing the outputs of the two photodiodes, the imaging plane phase difference detection AF can be performed. By using the imaging surface phase difference detection method, AF control can be performed by the phase difference detection method even during LV shooting, and focusing can be performed at high speed with high quality.

  The focus result obtained in this way is not used only for determining the focus position at the time of shooting, but a plurality of focus detection areas are set, and a plurality of subjects are determined from distance distribution information (distance map) in the screen. There are cases where the position is acquired and used for processing at the time of shooting and determination at the time of image processing.

  Patent Document 2 discloses that focus detection is performed by dividing an imaging range into a plurality of parts, and shooting modes are switched according to the type of subject acquired based on distance information. In focus detection at this time, a phase difference detection method, a contrast method, or a hybrid method combining the two is used.

JP 2001-083407 A JP 2009-272799 A

  However, depending on the position of the focus lens, there are scenes in which distance map information cannot be created by a single exposure. In such a case, it is necessary to search the subject by moving the focus lens many times and performing exposure to perform focus detection. Further, focus detection using the contrast method takes time because the focus lens is driven frequently, and the appearance of the LV screen is not good.

  The present invention has been made in view of the above problems, and an object of the present invention is to efficiently detect a subject included in a scene and improve the accuracy of focus adjustment to a main subject.

  In order to achieve the above object, the imaging apparatus of the present invention includes a plurality of photoelectric conversion elements for each of a plurality of microlenses, captures a subject image incident via an imaging optical system, and outputs an image signal. A focus detection unit that generates a pair of images having parallax from the image signal output from the image sensor and detects a defocus amount for each of a plurality of predetermined focus detection regions; Creation means for creating distance distribution information based on the defocus amount, focus adjustment means for performing focus adjustment based on the distance distribution information and the defocus amount, and creation of the distance distribution information by the creation means In this case, shooting is performed with the aperture included in the imaging optical system set to a first aperture value that is set to a predetermined first depth of field, and the focus adjustment unit performs focusing. Control means for controlling to perform shooting by setting the aperture to a second aperture value that is a second depth of field shallower than the first depth of field when performing the adjustment; Have.

  According to the present invention, it is possible to efficiently detect a subject included in a scene and improve the accuracy of focus adjustment to the main subject.

1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. FIG. 3 is a schematic diagram illustrating a relationship between a pupil of an imaging optical system and light received by an imaging element. 5 is a flowchart showing focus detection processing in the first embodiment. FIG. 3 is a diagram illustrating an example of a focus detection area in the first embodiment. FIG. 5 is a diagram showing an image signal obtained from a focus detection area in the first embodiment. The figure showing the change of the correlation amount and correlation change amount in 1st Embodiment. The figure showing the correlation variation | change_quantity in 1st Embodiment, and the change of correlation amount. 6 is a flowchart illustrating imaging processing according to the first embodiment. 5 is a flowchart illustrating still image shooting processing according to the first embodiment. 6 is a flowchart showing main subject detection processing in the first embodiment. 6 is a flowchart illustrating subject detection processing in the first embodiment. 5 is a flowchart showing AF processing in the first embodiment. FIG. 5 is a diagram illustrating an example of a focus detection area and a weight value in the first embodiment. FIG. 5 is a diagram illustrating an example of a scene to be photographed and a focus detection area in the first embodiment. FIG. 3 is a diagram schematically illustrating a positional relationship between the imaging device and a subject in the first embodiment. The figure explaining depth of field and image collapse. FIG. 3 is a diagram illustrating an example of subject detection in the first embodiment. The figure which shows another example of the scene image | photographed in 1st Embodiment. The figure which shows the 1st setting and 2nd setting in 2nd Embodiment. The figure which shows the focus detection process sequence in 2nd Embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
With reference to FIG. 1, the structure of the imaging device in 1st Embodiment is demonstrated. FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 101. The imaging apparatus 101 is a digital still camera or video camera that can record moving image or still image data obtained by photographing a subject on various media such as a magnetic tape, a solid-state memory, an optical disk, and a magnetic disk. It is not limited to.

  The photographing lens 111 (lens unit) includes an imaging optical system such as a fixed lens 112, a diaphragm 113, and a focus lens 114. The diaphragm control unit 115 drives the diaphragm 113 to adjust the aperture diameter of the diaphragm 113 to adjust the light amount during photographing. The focus control unit 116 determines a drive amount for driving the focus lens 114 based on the shift amount of the photographing lens 111 in the focus direction. The focus adjustment state is controlled by driving the focus lens 114. Automatic focus adjustment (AF) control is realized by movement control of the focus lens 114 by the focus control unit 116. The focus lens 114 is a focus adjustment lens, and is simply illustrated as a single lens in FIG. 1, but is normally configured by a plurality of lenses. The aperture controller 115 and the focus controller 116 are controlled by the lens controller 117.

  The light beam incident through the photographing lens 111 forms an image on the light receiving surface of the image sensor 121 and is converted into an electric signal by the image sensor 121. The image sensor 121 is configured by a CCD, a CMOS sensor, or the like, and photoelectrically converts a subject image (optical image) into signal charges by a plurality of photoelectric conversion elements. The signal charge accumulated in each photoelectric conversion element is sequentially read from the image pickup element 121 as a voltage signal (an image pickup signal and a focus detection signal) corresponding to the signal charge by a driving pulse output from the timing generator 122.

  Here, the image sensor 121 will be described with reference to FIG. The image sensor 121 is composed of m × n pixels that are two-dimensionally arranged, and 201 represents a partial cross section of the image sensor 121. Each pixel 202 is provided with a microlens 203 and two photoelectric conversion elements 204 and 205, and can generate an image signal used for automatic focus adjustment by an imaging surface phase difference detection method. A light beam that has passed through different regions 207 and 208 of the pupil 206 of the imaging optical system of the photographing lens 111 is provided in each pixel 202 via a microlens 203 disposed in each pixel 202 with the optical axis 209 as the center. Light is received by the two photoelectric conversion elements 204 and 205. Accordingly, two types of signals, that is, an imaging signal and a focus detection signal can be acquired from one pixel 202.

  By adding the outputs of the two photoelectric conversion elements 204 and 205, an imaging signal can be acquired and adjusted as an image signal (image data) by the imaging signal processing unit 124. Further, by handling the outputs of the two photoelectric conversion elements 204 and 205, two images having different parallaxes (parallax images) can be acquired, and the focus detection signal processing unit 125 performs focus detection calculation. In the description of the present embodiment, an image obtained by adding the outputs of the two photoelectric conversion elements 204 and 205 is an A + B image, and an image obtained by handling the outputs of the two photoelectric conversion elements is an A image and a B image, respectively. Call it. Note that the method of generating the focus detection signal is not limited to the method of the present embodiment, and other methods may be used. Details of the focus detection process will be described later.

  Returning to FIG. 1, the CDS / AGC / AD converter 123 performs correlated double sampling to remove reset noise, gain adjustment, signal signal extraction for the image signal and focus detection signal read from the image sensor 121. Digitize. Then, the CDS / AGC / AD converter 123 outputs the imaging signal to the imaging signal processing unit 124 and the imaging surface phase difference type focus detection signal to the focus detection signal processing unit 125, respectively. The focus detection signal processing unit 125 performs a correlation operation on the two image signals for focus detection output from the CDS / AGC / AD converter 123 to obtain an image shift amount and reliability information (two image coincidence, two images). Steepness etc.) is calculated. In addition, a focus detection area for performing focus detection in the imaging screen is set and arranged.

  The imaging signal processing unit 124 stores the imaging signal output from the CDS / AGC / AD converter 123 in the SDRAM 136 via the bus 131. The image signal stored in the SDRAM 136 is read by the display control unit 132 via the bus 131 and displayed on the display unit 133. In the operation mode in which the imaging signal is recorded, the image signal stored in the SDRAM 136 is recorded on the recording medium 135 by the recording medium control unit 134. The ROM 137 stores a control program executed by the camera control unit 140 and various data necessary for the control, and the flash ROM 138 stores various setting information related to the operation of the imaging apparatus 101 such as user setting information. Yes.

  The face detection unit 139 performs face detection from image data stored in the SDRAM 136 using a known method. Known techniques for face detection include methods that use knowledge about the face (skin color information, parts such as eyes, nose, mouth, etc.), a method of configuring a classifier for face detection by a learning algorithm represented by a neural network, etc. There is.

  The camera control unit 140 converts the defocus amount output from the focus detection signal processing unit 125 into a lens drive amount, and then transmits the lens drive amount to the lens control unit 117 and transmits it to the focus control unit 116 to realize automatic focus adjustment. doing. Further, based on the instruction from the operator or the magnitude of the pixel signal of the image data temporarily accumulated in the SDRAM 136, the accumulation time of the image sensor 121, the gain setting value of the CDS / AGC / AD converter 123, the timing The set value of the generator 122 is determined.

  Next, focus detection processing will be described using the flowchart of FIG. The focus detection area in the first embodiment is set by equally dividing the entire screen two-dimensionally. By arranging a plurality of focus detection areas in this way, distance distribution information (distance map) on the imaging surface can be created. In S311, one focus detection area is selected from the two-dimensionally arranged focus detection areas. In step S312, a pair of focus detection signals (A image and B image) are acquired from the image sensor 121 for the selected focus detection region. Next, in S313, filter processing for extracting a signal component in a predetermined frequency band from the pair of focus detection signals acquired in S312 is performed. Subsequently, in S314, a correlation calculation for calculating a correlation amount between the acquired pair of focus detection signals is performed. The correlation calculation is performed for each scanning line in the focus detection area. Next, in S315, a correlation change amount is calculated from the correlation amount. Then, based on the correlation change amount calculated in S315, an image shift amount between the A image and the B image is calculated in S316, and then reliability information (two image coincidence, two image steepness, etc.) is calculated in S317. calculate.

  The processing from S311 to S317 is performed on all the focus detection areas arranged, and if it is determined in S318 that the processing has been completed for all focus detection areas, the process proceeds to S319. Finally, the defocus amount is calculated by multiplying the image shift amount of the two images calculated in S319 by a predetermined conversion coefficient. However, since the accuracy of focus detection is assumed to be low when the reliability is equal to or lower than the threshold, the defocus amount calculated in the focus detection area whose reliability is equal to or lower than the threshold is not used. At this time, the camera control unit 140 acquires parameter information used for automatic focus detection possessed by the imaging apparatus 101 or the photographing lens 111. The imaging parameters are information including aperture information of the aperture 113 in the imaging lens 111 and sensor gain applied to the imaging device 121 in the imaging apparatus 101. The imaging parameters are not dependent on the configuration of the present embodiment. What is necessary is just to acquire necessary information according to the configuration of the apparatus 101 as appropriate. Based on the camera parameters, the camera control unit 140 provides information so that processing related to signal generation in the focus detection signal processing unit 125 and a region for focus detection can be set.

  Next, details of the focus detection process described above will be further described with reference to FIGS. FIG. 4 shows an example of the focus detection area 402 on the pixel array 401 of the image sensor 121 in the focus detection process. The shift areas 403 on both sides of the focus detection area 402 are areas necessary for correlation calculation. For this reason, a region 404 obtained by combining the focus detection region 402 and the shift region 403 is a pixel region necessary for the correlation calculation. In the figure, p, q, s, and t represent coordinates in the horizontal direction (x-axis direction), p and q represent the x-coordinates of the start point and end point of the area 404, and s and t represent the focus detection areas The x coordinate of the start point and end point of 402 is shown.

  FIG. 5 shows an example of a pair of image signals for focus detection subjected to filtering. A solid line 501 indicates the A image signal of the A image, and a broken line 502 indicates the B image signal of the B image. 5A shows the image signals 501 and 502 before the shift, and FIGS. 5B and 5C show the A image signal 501 and the B image signal 502 in the plus direction from the state of FIG. The state shifted in the minus direction is shown. When calculating the correlation amount between the pair of image signals 501 and 502, both the image signals 501 and 502 are shifted one bit at a time in the direction of the arrow.

  Next, a method for calculating the correlation amount COR will be described. First, as shown in FIGS. 5B and 5C, the A image signal 501 and the B image signal 502 are each shifted by 1 bit, and the A image signal 501 and the B image signal 502 at each shift position are shifted. Calculate the sum of absolute differences. When the number of shifts is i, the minimum shift amount is p−s, the maximum shift amount is q−t, x is the start coordinate of the focus detection region 402, and y is the end coordinate of the focus detection region 402, the correlation amount COR is It can be calculated by the following equation (1).

  FIG. 6A is a diagram illustrating an example of a change in the correlation amount, where the horizontal axis of the graph indicates the shift amount and the vertical axis indicates the correlation amount. In the correlation amount waveform 601, reference numerals 602 and 603 indicate the vicinity of the extreme value. Here, the degree of coincidence between the A image signal 501 and the B image signal 502 becomes the highest in the shift amount corresponding to a smaller correlation amount among the extreme values 602 and 603 in the correlation amount 601 that changes with the shift amount.

  Next, a method for calculating the correlation change amount ΔCOR will be described. The difference in the correlation amount at every other shift in the waveform of the correlation amount 601 shown in FIG. 6A is calculated as the correlation change amount. At this time, assuming that the shift amount is i, the minimum shift amount is ps, and the maximum shift amount is qt, the correlation change amount ΔCOR can be calculated by the following equation (2).

  FIG. 6B shows an example of the relationship between the shift amount and the correlation change amount ΔCOR. The horizontal axis of the graph indicates the shift amount, and the vertical axis indicates the correlation change amount. The correlation change amount 604 that changes with the shift amount changes from positive to negative around 605 and 606. A state in which the correlation change amount 604 is 0 at 605 and 606 is called zero crossing, and the degree of coincidence between the A image signal 501 and the B image signal 502 is the highest. Therefore, the shift amount giving the zero cross becomes the image shift amount.

  FIG. 7A is an enlarged view of the portion 605 in FIG. 6B, and 607 is a part of the correlation change amount 604. A method of calculating the image shift amount PRD will be described with reference to FIG. The shift amount (k−1 + α) giving the zero cross is divided into an integer part β and a decimal part α. The decimal part α can be calculated by the following equation (3) from the similar relationship between the triangle ABC and the triangle ADE in FIG.

一方、整数部分βは、図７（ａ）から以下の式（４）によって算出することができる。
β＝ｋ−１ …（４）
以上のようにして得られたαとβの和から、像ずれ量ＰＲＤを算出することができる。

On the other hand, the integer part β can be calculated by the following equation (4) from FIG.
β = k−1 (4)
The image shift amount PRD can be calculated from the sum of α and β obtained as described above.

  Also, as shown in FIG. 6B, when there are a plurality of zero crossings of the correlation change amount 604, the one with the larger steepness of the change of the correlation change amount 604 in the vicinity thereof is set as the first zero cross. This steepness is an index indicating the ease of focus detection, and the larger the value, the easier the focus detection is. The steepness maxder can be calculated by the following equation (5).

このように、第１の実施形態では、相関変化量のゼロクロスが複数存在する場合は、その急峻性によって第１のゼロクロスを決定し、この第１のゼロクロスを与えるシフト量を像ずれ量とする。

As described above, in the first embodiment, when there are a plurality of correlation change amount zero crosses, the first zero cross is determined based on the steepness, and the shift amount giving the first zero cross is set as the image shift amount. .

  Next, a method for calculating the reliability of the image shift amount will be described. The reliability of the image shift amount can be defined by the matching degree fnclvl (hereinafter referred to as “two-image matching degree”) of the A image signal and the B image signal and the steepness of the correlation change amount described above. The degree of coincidence of two images is an index representing the accuracy of the image shift amount. In the correlation calculation method in the present embodiment, the smaller the value, the better the accuracy.

  FIG. 7B is an enlarged view of the portion indicated by 602 in FIG. 6A, and 608 is a part of the correlation amount 601. The two-image coincidence degree fnclvl can be calculated by the following equation (6).

  Next, the operation of the imaging apparatus 101 will be described using the flowcharts of FIGS. FIG. 8 is a flowchart illustrating a procedure of shooting processing of the imaging apparatus 101. When the photographing process is disclosed, an initialization process is performed in S801, and the process proceeds to S802. In S802, it is determined whether the shooting mode is the movie shooting mode or the still image shooting mode. If the shooting mode is the movie shooting mode, the process proceeds to S803 to perform the movie shooting process. If the shooting mode is the still image shooting mode, the process proceeds to S804. The photographing process is performed, and the process proceeds to S805. Note that the still image shooting process performed in S804 will be described in detail later.

  In step S805, it is determined whether the shooting process has been stopped. If the shooting process has been stopped, the shooting process is terminated. The shooting process is stopped when the camera is turned off or when an operation other than shooting is performed, such as camera user setting process or playback process for checking captured images / movies. . If it is determined in step S805 that the shooting process has not been stopped, the process advances to step S806 to determine whether the shooting mode has been changed. If it has been changed, the process returns to S801, and after performing initialization processing, processing for the changed shooting mode is performed. If it has not been changed, the process returns to S802, and the processing in the current shooting mode is continued.

  Next, still image shooting processing accompanied by main subject detection processing at the time of focus detection will be described with reference to the flowchart of FIG. In S901, if the main subject has not yet been detected in the current shooting scene, the process proceeds to S902. In step S902, a first setting that can detect the positional relationship between a plurality of subjects in a short time is performed. In this first setting, the defocus amount can be detected in a wide range on the optical axis of the photographing lens 111 in one vertical period, but the accuracy of the position to be detected is low. After performing the first setting, in step S904, main subject detection processing is performed to detect at least one subject, and a main subject is selected therefrom. When the main subject detection process in S904 is completed, the process proceeds to S906.

  On the other hand, if the main subject has already been detected in the processing up to the previous time, the process proceeds to S903 and the second setting is performed. This second setting is characterized in that the range in which the defocus amount can be detected on the optical axis of the photographing lens 111 is narrow, but the focus detection accuracy is high. After performing the second setting, the process proceeds to S905, where a highly accurate focus detection process is performed, and the focal length with respect to the main subject is detected. Note that the focus detection processing in S905 is performed as described above with reference to FIGS. When the focus detection process is completed, the process proceeds to S906.

  As described above, in the first embodiment, depending on whether or not the main subject has already been detected, two types of settings with different characteristics are switched between the main subject detection and the focus detection to meet each requirement. Process using the specified settings. In the first embodiment, the first setting and the second setting are switched by setting the aperture stop that controls the exposure amount of the imaging optical system to different predetermined aperture values. The specific setting contents of the first setting and the second setting will be described later.

  In step S906, the AF instruction switch is determined. If the AF instruction switch is on, the process proceeds to S907, where AF processing is performed, and the focus lens is driven according to the defocus amount and reliability acquired by focus detection. On the other hand, if the AF instruction switch is off in S906, the process proceeds to S908 to check the shooting instruction switch. If it is determined in S908 that the shooting instruction switch is off, the process proceeds to S911, the focus stop state is canceled, and the still image shooting process flow is ended. Here, the in-focus stop state refers to a state in which the defocus amount with respect to the main subject is within a predetermined value and the camera is ready to perform a shooting process while focusing on the main subject.

  If it is determined in step S908 that the shooting instruction switch is on and it is determined in step S909 that the in-focus state is not stopped, the process proceeds to step S907 and AF processing is performed. If it is determined in step S908 that the shooting instruction switch is on and it is determined in step S909 that the in-focus state is stopped, the main subject is in focus, and the process advances to step S910 to perform shooting processing. After completing the shooting process in S910, the focus stop state is canceled in S911, and the still image shooting process is ended.

  Next, the main subject detection process performed in S904 will be described with reference to the flowchart of FIG. The detection of the main subject includes a flow of detecting at least one subject according to the defocus amounts calculated in the plurality of focus detection areas, and selecting one main subject from the at least one subject. It has become.

  First, in S1000, focus detection processing is performed on all the set focus detection areas. The focus detection process in S1000 is the same as the focus detection process in S905 as described above with reference to FIGS. 3 to 7. In S1000, the reliability and defocus amount in each focus detection area are obtained. Subsequently, in S1001, it is determined whether the reliability value obtained in S1000 is greater than or equal to a threshold value in one or more focus detection areas. Here, as described above, the reliability is a value defined by the matching degree (hereinafter referred to as two-image matching degree) fnclvl of the pair of image signals A and B and the steepness maxder of the correlation change amount. Further, the threshold value used for the reliability determination is defined as a value indicating that the detected defocus amount can be ensured to be within a predetermined range near the in-focus state. If it is determined in S1001 that the reliability is greater than or equal to the threshold value in one or more focus detection areas, there is an area where the defocus amount up to the subject can be detected, and the process advances to S1002. On the other hand, if the reliability is not determined to be greater than or equal to the threshold value in all focus detection areas, the subject cannot be detected, and thus the main subject detection processing flow ends.

  In S1002, subject detection is performed. Details of the subject detection process in S1002 will be described later. In S1002, a “total weight value” indicating the likelihood of the subject is output for each detected subject. In S1003, it is determined whether or not at least one subject has been detected in S1002. If YES in step S1002, the process advances to step S1004 to record the subject distance in the optical axis direction of the camera lens in the SDRAM 136. The distance to the subject is calculated based on the defocus amount calculated in the focus detection process in S1000. In S1005, one main subject is selected from the subjects detected in S1002. As a method of selecting the main subject in the first embodiment, the subject having the largest “weight total value” indicating the probability of the subject is selected as the main subject.

  Next, details of the subject detection processing performed in S1002 will be described using the flowchart of FIG. In the subject detection process according to the present embodiment, a region that compares the defocus amounts of the focus detection regions and outputs an equivalent defocus amount is defined as a group (region group).

  In step S1101, a focus detection area for confirming whether there is a clump is selected. Next, in S1102, the defocus amount of the focus detection area set in S1101 is compared with the defocus amount of the adjacent focus detection area, and if the defocus amount difference is within a predetermined threshold, 1 is set. Consider it as a lump. Note that the comparison of the defocus amount for the mass detection is not limited to the comparison of adjacent regions, but may be compared with other regions. In addition, a mass can be regarded as a mass if the reliability is equal to or higher than a threshold value and one or more focus detection regions. The processing of S1101 and S1102 is repeated until the focus detection areas to be confirmed are changed in order, so that the processing is completed for all the focus detection areas. The process proceeds to S1104.

  In S1104, the number N of clusters detected in S1101 and S1102 is counted. If it is determined in S1105 that the number N of lumps is not greater than 0, that is, if no lumps are detected, the subject detection process is terminated. On the other hand, in S1105, if the number N of chunks is larger than 0 in S1104, that is, if one or more chunks are detected, the process proceeds to S1106. In S1106, i indicating the number of the detected mass is initialized to 0. Next, in S1107, a total weight value for the i-th lump (hereinafter referred to as “chunk (i)”) is calculated.

  Here, the total weight value will be described in detail with reference to FIG. Reference numeral 1301 in FIG. 13 denotes an imageable area, and reference numeral 1302 denotes the arranged focus detection area. The total weight value is a weight value set in advance according to the image height stored in the ROM 137 for each focus detection area 1302 arranged as shown in FIG. 13 (a numerical value described in each focus detection area). ) To calculate. The total weight value is calculated by referring to the weight value and calculating the sum of the weight values of the areas constituting the cluster (i) detected in the processing of S1102. Note that the weight values in FIG. 13 are examples, and the number of focus detection areas and the weight values may be set to other settings. By using this total weight value, the probability of the subject can be quantified according to the position on the imaging surface and the size of the subject. For example, the weight sum value increases as the subject is closer to the center and as the size of the mass with respect to the imageable area is larger. Note that the weight setting method is not limited to the method of determining according to the image height or size, and the weight is multiplied by the detected defocus amount or the reliability value of the focus detection area. The total value may be obtained.

  If it is determined in S1108 that the total weight value of the chunk (i) is equal to or greater than the threshold value, the process proceeds to S1109, and the chunk (i) is regarded as a subject. On the other hand, if it is determined that the total weight value of the chunk (i) is smaller than the threshold value, the process proceeds to S1110, and the chunk (i) is not regarded as a subject. As the threshold value of the total weight value, a threshold value corresponding to the exposure condition and the weight setting of each focus detection area is stored in the ROM 137 in advance. If it is determined whether the chunk (i) is a subject, i is incremented by 1 in S1111 and the process proceeds to the next chunk (i). When the processing from S1107 to S1111 is repeated N times, which is the number of clusters, and the subject determination is completed N times, the processing exits from S1112 and ends the subject detection processing.

  Next, the AF process performed in S907 of FIG. 9 will be described in detail using the flowchart of FIG. In step S1201, it is determined whether the defocus amount with respect to the current main subject is within a predetermined value and the focus is stopped. When the defocus amount with respect to the main subject is outside the predetermined range and the focus is not stopped, the process proceeds from S1201 to S1202. On the other hand, when it is determined that the in-focus state is within a predetermined value of the defocus amount with respect to the main subject, the process proceeds to S1203 and the AF process is terminated while the in-focus state is maintained.

  In S1202, it is determined whether or not the reliability is greater than or equal to a threshold in one or more focus detection areas. If it is determined that the reliability is greater than or equal to the threshold in one or more focus detection areas, the process advances to step S1204. On the other hand, if it is determined in S1202 that the reliability is not greater than or equal to the threshold value in one or more focus detection areas, the process proceeds to S1205, the focus lens 114 is moved to infinity, and the AF process ends. By moving the focus lens 114 to infinity, it is possible to search for a wider range of subjects and increase the possibility of detecting a defocus amount whose reliability is equal to or greater than a threshold value. In S1205, if the detected defocus direction can be ensured even if the reliability is smaller than the threshold, the focus lens 114 may be driven based on that direction.

  On the other hand, in S1204, it is determined whether one or more subjects could be detected in the subject detection process of FIG. If one or more subjects can be detected, the main subject is selected in S1005 of FIG. 10, and thus the defocus amount of the main subject is acquired in S1206. At this time, as the defocus amount of the main subject, for example, an average value of the defocus amounts of the focus detection areas included in the main subject is used. The defocus amount of the main subject is not limited to the average value, and the defocus amount of the focus detection region including the center of gravity position of the main subject region may be used. If it is determined in S1204 that one or more subjects could not be detected, the process proceeds to S1207, and the focus is determined according to the defocus amount of the focus detection area having the largest weight total value among the focus detection areas whose reliability is equal to or greater than the threshold. The lens is driven and the AF process is terminated.

  Next, in S1208, it is determined whether the defocus amount of the main subject acquired in S1206 is within a threshold value. This threshold value is a threshold value for determining whether or not the main subject is in focus and is ready for photographing, and one depth is set as a threshold value. The threshold value may be changed according to the scene. If it is determined in step S1208 that the defocus amount of the main subject is within the threshold value, the process proceeds to step S1209 to shift to a focus stop state, and the AF process ends. On the other hand, if the defocus amount of the main subject is outside the threshold range, the process proceeds to S1210, and the drive speed of the focus lens 114 is set. The driving speed of the focus lens 114 is switched in accordance with the total weight of the main subject and the second setting, details of which will be described later. Then, in step S1211, the focus lens 114 is driven according to the defocus amount of the main subject, and the AF process ends.

  Next, operations from detection of the main subject to AF processing will be described assuming an actual scene with reference to FIGS. Here, a main subject detection method in the case where a plurality of focus detection areas 1302 are arranged in a scene in which two subjects, that is, a subject A1401 and a subject B1402, as shown in FIG. 14 will be described.

  Since the main subject is not detected at first, the first setting is used. In the first setting, the aperture 113 is set to an aperture value excluding the wide aperture. Details of the first setting will be described with reference to FIG. FIG. 15 schematically illustrates the positional relationship in the optical axis direction among the imaging apparatus 101, the subject A1401, the subject B1402, and the in-focus position 1501 of the object scene. 1502 is a range in which the defocus amount can be detected with the first setting, 1503 is a range in which the defocus amount can be detected with the second setting, and 1504 and 1505 indicate the defocus amounts of subject A and subject B, respectively.

  The detection range 1502 of the first setting can be widened by setting the diaphragm 113 on the side opposite to the open side (small diaphragm side). The reason will be described with reference to FIG. FIG. 16A shows an A image 1601 and a B image 1602 of a subject with small image blur (high contrast value), and FIG. 16B shows an A image 1601 and a B image 1602 of a subject with large image blur. Yes. By setting the aperture 113 to the small aperture side, the depth of field is deepened and the amount of image blur for a wide range of subjects can be reduced. Therefore, a waveform as shown in FIG. Can be acquired. Therefore, the degree of image coincidence and steepness are increased during correlation calculation, and a defocus amount with high reliability can be detected. Compared to FIG. 16A, the shapes of the A image and the B image in FIG. 16B are asymmetric and have no edges, so that the shift amount 1604 between the two images can be detected in the focus detection process. I find it difficult.

  By setting the aperture value on the small aperture side in the aperture 113 in this way, a wide range of subjects can be detected. However, there is a demerit that a defocus amount error due to an image shift amount detection error is large. This demerit is caused by a large conversion coefficient that is multiplied by the image shift amount when calculating the defocus amount. This means that the sensitivity of the image shift amount with respect to the defocus amount is large. Therefore, a small detection error of the image shift amount 1603 is output as a large defocus amount detection error. Although there is such a demerit, the requirement for the first setting is not the detection accuracy but the ability to detect the defocus amount of more subjects, so the aperture stop is set to the small stop side. The aperture value used for the first setting may be written in the ROM 137 in advance or may be set by the user.

  FIG. 17 shows the result of calculating the defocus amount using the first setting for each focus detection region 1302 in the scene of FIG. In FIG. 17, for the convenience of explanation, the defocus amount of each focus detection area 1302 is displayed in each focus detection area 1302 with characters and symbols. Here, the focus detection area 1302 in which the same character (A, B) is written is an area where the difference is within a threshold as a result of comparing the defocus amounts in S1102 of FIG. A focus detection area 1302 in which “x” is written is an area in which the reliability is lower than the threshold and the defocus amount cannot be detected. The clusters detected from the result of the defocus amount are the cluster A1701 and the cluster B1702.

  Then, for each detected cluster, the focus detection area 1302 included in each cluster is associated with the weight in FIG. 13, and the total weight value of the cluster A 1701 and the cluster B 1702 is calculated. As a result, the total weight value of the mass A1701 is 34 (= 4 + 8 + 4 + 2 + 4 + 4 + 2 + 2 + 2 + 2), and the total weight value of the mass B1702 is 12 (= 1 + 2 + 2 + 1 + 2 + 4). When the threshold value of the total weight value for considering the subject is set to 8, for example, both the chunk A1701 and the chunk B1702 can be regarded as subjects, and the subject A1401 and the subject B1402 can be detected. Since the total weight value of the subject A1401 is higher, the subject A1401 is set as the main subject.

  When the face 1801 detected by the face detection unit 139 exists as shown in FIG. 18 apart from the main subject 1401 detected by the total weight value, the face 1801 is not the main subject 1401 detected by the total weight value. It is also good. Alternatively, the plurality of faces may be regarded as the main subject, the aperture stop may be set so that the plurality of faces as the main subject enter the depth of field, and the focus driving may be performed.

  Since the main subject 1401 has been detected by the processing described above, the focus detection processing is performed with the aperture 113 set to the second setting. The aperture value of the second setting is set to an aperture value that is closer to the open side than the first setting. The reason is that, in the second setting, the purpose is to focus on the main subject, and the accuracy of the defocus amount is required more than the range in which the defocus amount can be detected. When the aperture value on the open side is set, the depth of field becomes shallow, so that the amount of image blur of a subject at a position outside the depth of field increases rapidly. As a result, the range of the subject from which the waveform as shown in FIG. 16A can be acquired is narrowed, so that the detection error of the image shift amount can be reduced and the detection accuracy of the defocus amount can be increased. it can.

  The aperture value of the second setting is set to an aperture value at which the position + α of the main subject 1401 falls within the depth of field, as indicated by a range 1503 in FIG. Setting the aperture value makes it impossible to detect the position of the subject 1402, but since the subject position is recorded in the SDRAM 136 in S1004 of FIG. 10, it is not necessary to detect another subject in the second setting.

Next, the main subject 1401 is focused along the AF process of FIG. At this time, if AF processing is performed on the main subject 1401, the defocus amount gradually decreases. Therefore, the aperture is reset to the open side as needed according to the size of the defocus amount to improve detection accuracy. May be. At this time, the focus lens drive setting in S1210 of FIG. 12 is set according to the evaluation value that can be calculated by the following equation (7).
Evaluation value = Total weight of main subject / Aperture value of second setting (7)
When this evaluation value is high, the reliability of the subject detection and the detected defocus amount is high, so the drive speed of the focus lens 114 is increased and the focus lens 114 is moved to the in-focus position in a short time. On the contrary, when the evaluation value is low, the focus lens 114 is driven with a value smaller than the detected defocus amount, so that the focus lens is gradually driven close to the focus and the focus accuracy is improved.

  As described above, in addition to the main subject, the positions and defocus amounts of a plurality of subjects in the screen are also acquired and stored, so even after focusing on the main subject, Shooting can be performed using the information. For example, the aperture stop may be controlled so that an arbitrary subject enters the depth of field, or the white balance may be changed according to the subject selected by the user.

  As described above, according to the first embodiment, first, the main setting is obtained by acquiring the positions of a plurality of subjects in a wide range in the first setting in a short time, and the main setting is selected by switching to the second setting. Can focus well.

<Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described. In the second embodiment and the first embodiment, the target for changing the setting is different between the first setting and the second setting set in S902 and S903 in FIG. Since other processing contents and the configuration of the imaging apparatus are the same as those described in the first embodiment, the description thereof is omitted.

  In the second embodiment, the parameter for calculating the correlation amount between the two image signals performed in S314 of FIG. 3 is switched between the first setting and the second setting. As described in FIG. 5 of the first embodiment, in the correlation calculation, the A image and the B image are shifted and the difference is calculated as the correlation amount. At this time, the shift width (shift width [bit]) and the shift distance (shift range [bit]) are switched between the first setting and the second setting.

  The first setting and the second setting in the second embodiment will be described in detail with reference to FIG. First, in the first setting (S902), as described in the first embodiment, it is necessary to detect many subjects in a wide range in a short time. Since the width of the shift range in the correlation calculation is proportional to the range in which the defocus amount can be detected, a wider range of defocus amounts can be detected by widening the shift range. However, since it is one of the requirements to detect in a short time, the entire calculation time is shortened by increasing the shift width.

  Next, in the second setting (S903), as described in the first embodiment, focus detection accuracy for the main subject is obtained. Therefore, the accuracy of detecting the shift amount of the two images is increased by reducing the shift width for the focus detection area including the main subject detected in the first setting. However, if an attempt is made to shift the same shift range as in the first setting, the number of shifts increases and a long time is required for calculation, so the shift range is narrowed. The reason why the accuracy is increased is that the resolution of the horizontal axis (shift amount) of the correlation change amount in FIG. By performing the second setting in this way, focus detection accuracy for the main subject is increased.

  In the second embodiment, the focus detection accuracy of a subject other than the main subject can be improved by newly setting the focus detection region other than the main subject. In this setting, as shown in FIG. 19A, the shift width is set to be as small as the main subject area, and the shift range is set to shift a wide range as in the first setting. By performing this setting, the defocus amount can be detected with high accuracy over a wide range. The disadvantage of this setting is that it takes a lot of time for the correlation calculation.

  FIG. 20 shows the processing time of each focus detection area. 2001 is an arrow indicating the passage of time, and the processing time has passed from left to right. 2002 represents the exposure time of the image sensor 121, 2003 represents the acquisition time of image data used for correlation calculation (S312), and 2004 and 2005 represent the focal points in the focus detection area of the main subject area and the areas other than the main subject, respectively. This is the time for performing the detection process. As described above, the other areas with respect to the main subject area require much time for correlation calculation. However, since the focus detection speed is not required in the area other than the main subject, the processing sequence as shown in FIG. 20 is established.

  Note that the second setting may be as shown in FIG. 19B if the accuracy of the defocus amount in the subject area other than the main subject may be equal to the accuracy in the first embodiment.

  As described above, according to the second embodiment, in addition to the same effects as those of the first embodiment, it is possible to detect the defocus amount with high accuracy for a subject other than the main subject.

<Third Embodiment>
Hereinafter, a third embodiment of the present invention will be described. In the third embodiment and the above-described first and second embodiments, the targets to be changed in the first setting and the second setting set in S902 and S903 in FIG. 9 are different. Since other processing contents and the configuration of the imaging apparatus are the same as those described in the first embodiment, the description thereof is omitted.

  In the third embodiment and the first embodiment, the objects whose settings are changed are different between the first setting and the second setting. In the third embodiment, the filter characteristic of the filter process (S313) for the image data in the focus detection signal processing unit 125 described in FIG. 3 is switched between the first setting and the second setting. By using this method, it is possible to acquire the positions of many subjects in a short time and to focus on various subjects with high accuracy.

  In the third embodiment, the filter characteristics used in the filter processing performed in S313 of FIG. 3 are switched between the first setting and the second setting. The coefficient of the filter to be used is stored in the ROM 137 in advance. In the first setting (S902), a filter that passes an image signal having a low-frequency image frequency is set. As the subject moves away from the focal point and moves away from the depth of field, the image signal becomes blurred. If the image signal is blurred, the degree of image coincidence between the two images decreases as described with reference to FIG. 16, so that a highly reliable defocus amount cannot be detected. By using a filter that allows low-frequency image signals to pass through such a blurred image signal, it is possible to detect the position of a subject that is not highly accurate but greatly deviates from the depth of field.

  Next, after a subject is detected with the first setting, a filter having a feature of allowing an image signal having a high-frequency image frequency to pass is used with the second setting. By using a high-pass filter, it is possible to detect the focus with high accuracy for an object with high contrast. However, when a high-pass filter is used, there is a demerit that it is easy to detect a noise component or the like for a blurred image and to easily detect it. For this reason, a scene in which the defocus amount up to the main subject detected in the first setting is larger than a predetermined value and the defocus amount of the main subject cannot be acquired with a high-pass filter is assumed. For such scenes, a filter that passes the mid-range image frequency is stored in the ROM 137 in advance, and the de-focus amount is detected using the mid-range filter until the defocus amount can be detected by the high-pass filter. The focus amount may be detected. Even if the defocus amount is smaller than a predetermined value, a low-pass filter or a mid-pass filter may be used for a subject with a low contrast value. The determination of the contrast value uses evaluation values calculated respectively for the A image and the B image using the following equations (8) and (9) at the time of image data acquisition (S312).

  In the second setting, the setting may be changed in the main subject area and the area other than the main subject. For example, since it is necessary to detect the defocus amount with high accuracy in the main subject region, a high-pass filter is used, and in a region other than the main subject, a low-pass filter is used to always detect the position of the subject in a wide range. use.

  As described above, according to the third embodiment, first, the main setting is obtained by acquiring the positions of a plurality of subjects in a wide range in the first setting in a short time, and the main setting is selected by switching to the second setting. Can focus well.

<Other embodiments>
Note that the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device.

  Further, the present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  101: imaging device, 102: photographing lens, 113: aperture, 114: focus lens, 115: aperture control unit, 116: focus control unit, 117: lens control unit, 121: imaging device, 124: imaging signal processing unit, 125 : Focus detection signal processing unit, 140: camera control unit, 137: ROM, 139: face detection unit

Claims (29)

An image sensor that includes a plurality of photoelectric conversion elements for each of a plurality of microlenses, captures an image of a subject that has entered through an imaging optical system, and outputs an image signal;
Focus detection means for generating a pair of images having parallax from the image signal output from the image sensor and detecting a defocus amount for each of a plurality of predetermined focus detection areas;
Creating means for creating distance distribution information based on the defocus amount;
Focus adjusting means for performing focus adjustment based on the distance distribution information and the defocus amount;
When creating the distance distribution information by the creating means, shooting is performed by setting the aperture included in the imaging optical system to a first aperture value that is a predetermined first depth of field, When performing the focus adjustment by the focus adjusting means, the aperture is set to a second aperture value that is a second depth of field shallower than the first depth of field, and shooting is performed. An imaging apparatus comprising: control means for controlling.   The imaging apparatus according to claim 1, further comprising an image processing unit that detects a subject based on the distance distribution information and detects a main subject from the detected subject.   The imaging apparatus according to claim 2, wherein the image processing unit detects a subject by grouping focus detection areas in which a difference in the defocus amount is within a predetermined threshold as a group of areas. . For each of the plurality of focus detection areas, further comprising storage means for storing weight values determined under predetermined conditions,
The image processing means determines, for each of the region groups, the region group as a subject when a total value of the weight values of the focus detection regions constituting each region group is larger than a predetermined threshold value. The imaging apparatus according to claim 3.   The imaging apparatus according to claim 4, wherein the image processing unit determines a subject having the largest total value among the total values of the weight values of the subjects as a main subject.   The imaging apparatus according to claim 2, wherein the focus adjustment unit performs focus adjustment based on a defocus amount of a focus detection area corresponding to the main subject. The focus detection unit further detects the reliability of the defocus amount,
The said image processing means detects a to-be-photographed object using the focus detection area | region which has a defocus amount whose reliability is more than the predetermined threshold value, The object of any one of Claim 2 thru | or 6 characterized by the above-mentioned. Imaging device.   The focus adjustment unit drives the focus lens included in the imaging optical system to a position at infinity when the reliability is lower than a predetermined threshold in all of the plurality of focus detection regions. The imaging apparatus according to claim 7, wherein the imaging apparatus is characterized.   The focus adjustment means is based on the defocus amount detected in the focus detection area of the area group having the highest weight total value when there is no area group in which the total weight value is larger than the predetermined threshold. The imaging apparatus according to claim 4, wherein a focus lens included in the imaging optical system is driven.   The focus adjusting unit calculates a driving speed of a focus lens included in the imaging optical system according to a total value of the weight values and the second aperture value in the area of the main subject. The imaging device according to claim 4 or 5. Face detection means for detecting a face area from an image represented by an image signal output from the image sensor;
The image processing means determines the face area as a main subject when the area of the main subject and the face area detected by the face detection means do not match. The imaging apparatus according to item 1. A focus detection unit that generates a pair of images having parallax from an image signal output from the image sensor and detects a defocus amount for each of a plurality of predetermined focus detection regions;
Creating means for creating distance distribution information based on the defocus amount;
Focus adjusting means for obtaining a driving amount of a focus lens based on the distance distribution information and the defocus amount;
The focus detection unit uses the first setting when the distance distribution information is generated by the generation unit, and the second setting when the driving amount of the focus lens is determined by the focus adjustment unit. An image processing apparatus for detecting the defocus amount using An image processing means for detecting a subject based on the distance distribution information and detecting a main subject from the detected subject;
The focus detection unit detects a defocus amount by performing a correlation calculation while shifting the pair of images with each other,
In the first setting, one shift width is set as the first shift width, and the shift range is set as the first shift range.
In the second setting, for the focus detection area corresponding to the main subject, the shift width is set to a second shift width that is narrower than the first shift width, and the shift range is set to be smaller than the first shift range. The image processing apparatus according to claim 12, wherein the second shift range is also narrow. Filter means for filtering the image signal;
Image processing means for detecting a subject based on the distance distribution information and detecting a main subject from the detected subject;
The focus detection unit detects a defocus amount using the image signal filtered by the filter unit,
In the first setting, the first filter is used in the filter means, and in the second setting, a second filter is used for a focus detection region corresponding to the main subject, and the first filter is used. The image processing apparatus according to claim 12, wherein a frequency band extracted by is lower than a frequency band extracted by the second filter.   In the second setting, a third filter that extracts a signal in a frequency band lower than the frequency band extracted by the second filter is used for a focus detection region that does not correspond to the main subject. The image processing apparatus according to claim 14.   16. The image processing unit according to claim 13, wherein the image processing unit detects a subject by grouping focus detection areas in which the difference in defocus amount is within a predetermined threshold as a group of areas. The image processing apparatus according to item 1. For each of the plurality of focus detection areas, further comprising storage means for storing weight values determined under predetermined conditions,
The image processing means determines, for each of the region groups, the region group as a subject when a total value of the weight values of the focus detection regions constituting each region group is larger than a predetermined threshold value. The image processing apparatus according to claim 16.   The image processing apparatus according to claim 17, wherein the image processing unit determines a subject having the largest total value among the total values of the weight values of the subjects as a main subject.   The image processing apparatus according to claim 13, wherein the focus adjustment unit performs focus adjustment based on a defocus amount of a focus detection area corresponding to the main subject. The focus detection unit further detects the reliability of the defocus amount,
20. The image processing means detects a subject using a focus detection region having a defocus amount whose reliability is equal to or greater than a predetermined threshold value. Image processing apparatus.   The focus adjusting unit obtains a driving amount for driving the focus lens to an infinite position when the reliability is lower than a predetermined threshold in all of the plurality of focus detection regions. The image processing apparatus according to claim 20.   The focus adjustment means is based on the defocus amount detected in the focus detection area of the area group having the highest weight total value when there is no area group in which the total weight value is larger than the predetermined threshold. The image processing apparatus according to claim 17, wherein the driving amount of the focus lens is obtained. Face detection means for detecting a face area from an image represented by an image signal output from the image sensor;
The image processing means defines the face area as a main subject when the area of the main subject and the face area detected by the face detection means do not coincide with each other. The image processing apparatus according to item 1. An image sensor that includes a plurality of photoelectric conversion elements for each of a plurality of microlenses, captures an image of a subject that has entered through an imaging optical system, and outputs an image signal;
An image pickup apparatus comprising: the image processing apparatus according to any one of claims 12 to 23. A plurality of photoelectric conversion elements are provided for each of the plurality of microlenses, and a diaphragm included in the imaging optical system is determined in advance from an imaging element that captures a subject image incident through the imaging optical system and outputs an image signal. For each of a plurality of predetermined focus detection areas, a pair of images having parallax is generated from an image signal obtained by performing shooting by setting the aperture value to be the first depth of field. A first focus detection step for detecting a defocus amount;
A creation step of creating distance distribution information based on the defocus amount detected in the first focus detection step;
A pair of images having parallax from an image signal obtained by shooting from the image pickup device by setting a stop included in the image pickup optical system to a predetermined stop value that is a predetermined second depth of field. And a second focus detection step of detecting a defocus amount for each of a plurality of predetermined focus detection regions,
A focus adjusting unit, which performs a focus adjustment based on the distance distribution information and the defocus amount detected in the second focus detection step;
The method for controlling an imaging apparatus, wherein the second depth of field is shallower than the first depth of field. The focus detection unit generates a pair of images having parallax from the image signal output from the image sensor, and detects the defocus amount for each of a plurality of predetermined focus detection areas using the first setting. A first focus detection step,
A creation step of creating distance distribution information based on the defocus amount detected in the first focus detection step;
A second focus detection step of generating a pair of images having parallax from the image signal output from the image sensor and detecting a defocus amount using a second setting for each of the plurality of focus detection regions. When,
A focus adjusting step for obtaining a driving amount of a focus lens based on the distance distribution information and the defocus amount detected in the second focus detecting step;
An image processing method comprising:   A program for causing a computer to execute each step of a control method for an imaging apparatus according to claim 25.   A program for causing a computer to execute each step of the image processing method according to claim 26.   A computer-readable storage medium storing the program according to claim 27 or 28.

Images