JP6080579B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus that performs imaging control based on an evaluation value obtained from an image, and a control method therefor.

  To suppress overexposure and underexposure, the dynamic range that is wider than the shooting dynamic range can be obtained by combining the portions where the gradation information is recorded from multiple images shot from the same scene with different exposures. A method for obtaining an image having the same is known. In this specification, such dynamic range expansion processing is referred to as HDR (High Dynamic Range) processing.

  Conventionally, HDR processing has been used only for still image shooting, but it can also be applied to moving image shooting by generating each frame by applying HDR processing. Hereinafter, the HDR processing for the moving image frame is referred to as moving image HDR processing. In the moving image HDR process, a plurality of images with different exposures are required to generate each frame, and therefore images with different exposures are sequentially captured.

  When such shooting is performed sequentially, focus control is performed by evaluating the in-focus state from the contrast information of the image. In contrast detection type focus control (hereinafter referred to as contrast AF control), AF control that follows the subject is performed. May not be accurate.

  In Patent Document 1, an exposure value obtained by superimposing a gain corresponding to an exposure difference on an evaluation value (hereinafter simply referred to as an AF evaluation value) used for contrast AF obtained from a plurality of exposure images. A technique for removing the effect of the difference on the AF evaluation value has been proposed.

JP 2002-323649 A

  However, in Patent Document 1, for example, when the subject moves, the influence of the exposure difference on the AF evaluation value may not be removed, and control according to the shooting environment is not considered. In addition, this problem is not limited to the AF evaluation value, and can be a problem in general evaluation values based on information obtained from the captured image.

  The present invention has been made in view of the above-described problems of the prior art, and an imaging apparatus with improved follow-up capability of imaging control using evaluation values obtained from images captured with different exposure amounts and a control method thereof The purpose is to provide.

The above-described object is to provide an imaging unit that sequentially captures images with different exposure amounts for each frame, an image processing unit that generates a moving image using a composite image obtained by combining a plurality of images obtained by the imaging unit , and a frame image ; Detection means for detecting the amount of movement of a subject between images , generation means for generating an evaluation value used for focus control of the imaging means , and control means for controlling focus of the imaging means based on the evaluation value generated by the generation means And when the difference in exposure amount between the plurality of images is greater than the first threshold , the control means performs focus control using an evaluation value generated from an image having the largest exposure amount among the plurality of images. If the difference in exposure amount between the plurality of images is smaller than the first threshold value, one of the images is selected according to the amount of movement of the subject, and the evaluation value generated from the selected image is used for the foreground. It is achieved by an imaging device and performs scan control.

  With such a configuration, according to the present invention, it is possible to provide an imaging apparatus and a control method thereof that improve the follow-up performance of focus control using AF evaluation values obtained from images captured with different exposure amounts.

1 is a block diagram illustrating an example of a functional configuration of an imaging apparatus according to an embodiment The figure which shows the example of the image which concerns on HDR processing, and its luminance histogram typically Timing chart showing timing examples of shooting, composition, AF evaluation value acquisition, and focus control during moving image HDR processing in the embodiment Flowchart for explaining control of exposure step amount of low-exposure image in the embodiment The figure which shows the example of relationship between focus lens position and contrast AF evaluation value Flowchart for explaining focus control in the first embodiment Flowchart for explaining AF evaluation value selection processing in S608 of FIG. The figure which shows the example of the imaging | photography information which can be utilized for selection and the synthesis | combination of AF evaluation value in embodiment Flowchart for explaining focus control in the second embodiment Flowchart for explaining AF evaluation value composition processing in S901 of FIG. (A) is an example of the relationship between the amount of motion and the synthesis coefficient in the second embodiment, (b) is a diagram showing an example of the relationship between the exposure step amount and the synthesis coefficient in the second embodiment. Diagram showing examples of multiple AF evaluation frames Flowchart for explaining focus control in the third embodiment FIG. 2 is a block diagram illustrating an example of a functional configuration of an AF evaluation value selection unit according to the first embodiment. The block diagram which shows the function structural example of AF evaluation value synthetic | combination part in 2nd Embodiment

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.
First, the technique of Patent Document 1 will be described with reference to FIG. 5 for the reason that the influence of the exposure difference on the AF evaluation value may not be removed when the subject moves, for example.
FIG. 5 shows a change in the AF evaluation value according to the focus lens position, with the horizontal axis as the focus lens position and the vertical axis as the AF evaluation value. As shown in the figure, when the focus lens position becomes the in-focus position, the AF evaluation value becomes the maximum value.

  FIG. 5A shows an AF evaluation value 501 calculated from an image obtained by photographing the same still subject with appropriate exposure, and an AF evaluation value 502 calculated from an image obtained by underexposure (low exposure image). FIG. 5B shows an AF evaluation value 501 calculated from an image obtained by photographing the same moving subject with appropriate exposure, and an AF evaluation value 502 calculated from a low-exposure image.

  When changing the exposure time to make a difference in the exposure amount, if the speed of the moving subject is constant, the amount of subject blurring in the captured image is smaller in the image captured due to underexposure. There is a similar relationship regarding the amount of camera shake. Since the contrast is higher when there is less blur, the AF evaluation value 502 calculated from an image shot with underexposure is higher than the AF evaluation value 501 calculated from an image shot with appropriate exposure, and the difference is the difference between moving subjects. Larger when shooting.

As described above, when an image obtained by sequentially capturing moving subjects while changing the exposure amount is input, the amount by which the AF evaluation value fluctuates for each frame becomes large, and it may be difficult to evaluate the in-focus state. . In this case, it is impossible to focus following the moving subject.
In addition, this problem is not limited to AF evaluation values. In general evaluation values based on information obtained from captured images, such as evaluation values for subject detection, evaluation values for exposure control, and evaluation values for white balance processing. It can be a challenge.

  Therefore, in the present embodiment, among the evaluation values obtained from each of a plurality of images taken under different exposure conditions, it is determined that the reliability is higher from the movement of the imaging device or the subject, the state of overexposure / blackout, etc. At least one evaluation value to be selected is selected and used.

  FIG. 1 is a block diagram illustrating a functional configuration example of an imaging apparatus according to the first embodiment of the present invention. Each functional block may be realized by individual hardware, or may be realized by executing software by a programmable processor. In addition, the functions described as separate functional blocks in the figure may be realized by a single piece of hardware, or a single functional block may be realized by a plurality of pieces of hardware.

  A subject image formed on the image sensor 102 by the optical system 101 including the focus lens is converted into an image signal by the image sensor 102. The image processing unit 103 applies white balance processing, demosaic processing, color (luminance / color difference signal) conversion processing, γ correction processing, and the like to this image signal. In addition, the image processing unit 103 calculates an evaluation value by a known technique such as matching using the input image signal, and performs subject detection processing. For example, face detection is performed. The obtained detection result of the subject detection is used for control such as weighting the subject detection region when calculating each evaluation value in the present embodiment, or the subject detection region is displayed to the user when displaying an image. It is used to let you know. Similar to the AF evaluation value, such an evaluation value for subject detection may cause a problem as to which exposure image is appropriate for obtaining the evaluation value depending on the shooting environment.

  In the white balance processing, a white area that is supposed to be white in the image of the image signal is extracted, and an evaluation value of the white area is calculated. Alternatively, the evaluation value is calculated by integrating the entire image signal, or the evaluation value using the image signal in the face area detected by the above-described method as the skin color area is calculated. A gain for white balance is calculated from at least one of these evaluation values, and the image signal is corrected with the gain. Similar to the AF evaluation value, the evaluation value for such white balance processing may cause a problem as to which exposure image is appropriate for obtaining the evaluation value depending on the shooting environment.

  The image signal output from the image processing unit 103 is input to the frame memory 105 or the image composition unit 106 in accordance with an instruction from the system control unit 110. This input switching control will be described in detail in the description of the image composition process.

  The image composition unit 106 synthesizes the image output from the image processing unit 103 and the image stored in the frame memory 105 so as to suppress blackout and overexposure, and expands the dynamic range. HDR image).

  The image signal output from the image sensor 102 is also input to the AF evaluation value calculation unit 107, the motion vector detection unit 108, and the histogram calculation unit 109. The AF evaluation value calculation unit 107 calculates an AF evaluation value of an image signal corresponding to an area (AF evaluation frame) designated by the system control unit 110 and inputs the AF evaluation value to the system control unit 110.

  The motion vector detection unit 108 calculates the motion vector information of the AF evaluation frame from a plurality of image signals photographed at different timings input from the image sensor 102 and inputs the motion vector information to the system control unit 110.

  Further, the histogram calculation unit 109 calculates histogram information (hereinafter referred to as luminance histogram information) about the luminance signal of the pixels in the AF evaluation frame as the luminance information of the image signal from the image sensor 102, and the system control unit 110. To enter.

  The system control unit 110 includes, for example, a programmable processor such as a CPU or MPU, a ROM that stores a program, and a RAM that is used to execute the program. The system control unit 110 controls the operation of the entire imaging apparatus by executing the program. The system control unit 110 performs control of the focal length (field angle), aperture value, focus position (focusing distance), etc. of the optical system 101, control of the driving method of the image sensor 102, control of the switch 104, and the like. These controls are performed based on the AF evaluation value from the AF evaluation value calculation unit 107, the luminance histogram information from the histogram calculation unit 109, the motion vector information from the motion vector detection unit 108, and the like.

  In addition, the system control unit 110 sends an AF evaluation value, luminance histogram, and motion vector detection information image area (AF evaluation frame) to the AF evaluation value calculation unit 107, histogram calculation unit 109, and motion vector detection unit 108. To set. Although there is no restriction | limiting in the setting method, For example, it can set using the coordinate which specifies AF evaluation frame.

  Note that FIG. 1 and the above description relate to the minimum configuration necessary for understanding the present invention out of the configurations of the imaging apparatus according to the present embodiment. There exists a configuration of such an imaging device.

Next, HDR processing in the imaging apparatus according to the present embodiment will be described.
In the HDR processing, the image composition unit 106 that composes images and generates an HDR image will be described first. The image synthesizing unit 106 generates one frame of HDR image with an expanded dynamic range by synthesizing a plurality of images included in a group of images continuously shot with different exposures based on one image. To do.

  Various proposals have already been made for synthesizing HDR images, and any known method can be used. For example, as described in Japanese Patent Application Laid-Open No. 2011-221924, a method of performing image composition processing after correcting a shift caused by camera shake or subject movement for an image other than the reference image is used. Can do.

An example of HDR processing and exposure control for capturing an image used for HDR processing will be described.
Various settings are possible for the number of images used for HDR processing (compositing processing) and the exposure conditions when each image is captured. As an example in the present embodiment, as shown in FIG. 2, a composite image (HDR) of one frame from two frames of a reference exposure image 201 photographed with a reference exposure and a low exposure image 202 photographed with an exposure amount smaller than the reference exposure. Image) 203 is obtained. In the present embodiment, the reference exposure is referred to as a proper exposure image in order to set the reference exposure as a proper exposure. An exposure control amount (exposure time, ISO sensitivity, aperture) for proper exposure is determined based on a luminance evaluation value obtained from an image and a program diagram. FIG. 2 also shows luminance histograms 204 to 206 obtained by the histogram calculation unit 109 for the proper exposure image 201, the low exposure image 202, and the composite image 203. The luminances Low and Llow in the luminance histogram information 204 and 205 are thresholds for determining an area where whiteout and blackout occur. Pixels 207 and 209 having a luminance of Llow or lower are regarded as blackout pixels, and pixels 208 and 210 having a luminance of Lhigh or higher are regarded as whiteout pixels. The thresholds Lhigh and Llow are determined in advance in consideration of the dynamic range of the image sensor and the like.

  In the present embodiment, the low-exposure image 202 is shot with an exposure setting that is N steps darker than the exposure setting of the proper exposure image 201. Hereinafter, N is referred to as an exposure step amount. In the present embodiment, the exposure step amount N at the time of the next photographing is dynamically changed according to the degree of overexposure or underexposure in the photographed low-exposure image. There are exposure time, ISO sensitivity, and the like as a method of providing an exposure step of the exposure step amount N, and one or both of them may be used to achieve the step amount N.

The control of the exposure step amount N in the present embodiment will be described using the flowchart of FIG.
The system control unit 110 sets the exposure step amount N to the initial value N0 at the start of shooting (S401). The system control unit 110 captures a proper exposure image and a low exposure image (S402), and acquires luminance histogram information of a predetermined area of the low exposure image from the histogram calculation unit 109 (S403).

  Then, the system control unit 110 determines whether or not whiteout occurs in the HDR image using the low-exposure image based on the luminance histogram information (S404). Specifically, if the number of pixels having a luminance equal to or greater than the whiteout threshold Lhigh (the integrated value of the frequency of the area 210 in FIG. 2) is equal to or greater than a predetermined number, whiteout occurs in the HDR image using the low-exposure image. Judge that. In this case, the system control unit 110 determines that the current exposure step amount N is insufficient, and sets the current exposure step amount N + the predetermined step amount N ′ (for example, 1/1) as the exposure step amount N for the next low-exposure image. (3 steps) is set (S404). That is, in the next shooting of a low exposure image, the exposure step amount is further increased to reduce the exposure amount.

  On the other hand, when the number of pixels having a luminance equal to or greater than the whiteout threshold Lhigh is less than the predetermined number, the system control unit 110 determines whether or not the image is dark (S406). Specifically, if the number of pixels having a luminance equal to or lower than the threshold Llow (the integrated value of the frequency of the region 209 in FIG. 2) is equal to or greater than a predetermined number, it is determined that blackout has occurred. In this case, since the whiteout does not occur and the blackout occurs, the system control unit 110 reduces the exposure amount of the low exposure image more than necessary (the exposure step amount N is too large). to decide.

  Then, the system control unit 110 sets the current exposure step amount N−predetermined step amount N ′ (for example, 1/3 step) as the exposure step amount N of the low exposure image to be photographed next (S407). That is, in the next shooting of a low-exposure image, the exposure step amount is reduced and the exposure amount is increased.

In S406, when the number of pixels having the luminance equal to or lower than the threshold Llow is less than the predetermined number, the system control unit 110 does not change the current exposure step amount N and maintains the exposure setting. By performing shooting while continuing the above operations, a low-exposure image can be shot with an exposure step amount N optimum for HDR processing, and a good HDR image can be generated.
Also, when the subject area is detected by the image processing unit 103, the subject area is similarly evaluated by the method described above in S403 to S406, and the determination result of the subject area is weighted together with the overall result. Then, the exposure step amount N is controlled.

When photographing the low-exposure image 202, the system control unit 110 exposes the exposure time (charge accumulation time) of the image sensor 102 so that an image with a small exposure amount by an amount corresponding to the exposure step amount N with respect to the proper exposure can be obtained. ) To control. If the exposure time of the proper exposure image is 1, it may be controlled to shoot with an exposure time of 1/2 ^N. For example, if the exposure step amount N = 2, 1/4 of the exposure time of the proper exposure image. And it is sufficient.

Next, the timing of shooting, composition, AF evaluation value acquisition, and focus lens control during moving image HDR processing will be described using the timing chart of FIG.
A capture pulse 301 is a trigger for the image sensor 102 to start image capture (charge reading). The shutter pulse 302 determines the exposure (charge accumulation) start timing of the image sensor 102. In FIG. 3, the first frame is exposed at a time Tlow from the shutter pulse S1 to the capture pulse C1, and a low-exposure image 304 is taken. The second frame is exposed for a time Tbase from S2 to C2, and a proper exposure image 305 is taken. Hereinafter, similarly, a low exposure image is taken in the third frame, a proper exposure image is taken in the fourth frame, and a low exposure image is taken in the fifth frame. Similarly, when three or more frames of images are combined to generate a one-frame HDR image, the frames used for combining are sequentially and sequentially photographed.

  In the present embodiment, since the proper exposure image is taken last in the group of images used for synthesis, the image synthesis process is performed after the proper exposure image is taken. Therefore, the system control unit 110 controls the switching of the switch 104 so that the low-exposure image 304 that has been shot first is stored in the frame memory 105. Then, the image synthesis unit 106 synthesizes the low-exposure image 304 (one frame before) stored in the frame memory 105 and the captured appropriate exposure image 305 (of the current frame) to generate one HDR video frame. An HDR image 306 is generated. Therefore, the frame rate of the HDR moving image according to the present embodiment is ½ of the imaging frame rate.

  The AF evaluation value is acquired at the timing of the timing pulse 307. The AF evaluation value of the low-exposure image 304 acquired by the capture pulse V1 is acquired at the timing of the pulse A1, that is, one frame later. The AF evaluation value is acquired for each frame of the captured image, not the composite image.

  The focus lens is driven at the timing of the timing pulse 308. The focus lens is driven at the same frequency as the HDR moving image frame rate by focus control described later. The system control unit 110 performs focus control based on the AF evaluation value acquired at the timing of the pulses A1 and A2 at the timing of the pulse L1.

Next, focus control in the present embodiment will be described.
The AF evaluation value calculated by the AF evaluation value calculation unit 107 is a value reflecting the level of contrast of the image, and can be calculated by a known and arbitrary method. For example, as described in Japanese Patent Application Laid-Open No. 2011-13324, a predetermined band pass filter is applied to an image to extract a contrast component, and a peak value for each line of the contrast signal is integrated. Can be sought.

  The system control unit 110 evaluates the focus state (in-focus state) of the optical system 101 based on the AF evaluation value calculated by the AF evaluation value calculation unit 107, and calculates the position of the focus lens that maximizes the AF evaluation value. . Then, the system control unit 110 instructs the optical system 101 to move (shift) the focus lens to the calculated position, thereby realizing focus control.

With reference to FIG. 5 again, the characteristics of the proper exposure AF evaluation value and the low exposure AF evaluation value will be described. It should be noted that the AF evaluation values shown in FIG. 5 are corrected so that the brightness levels of the proper exposure image and the low exposure image are aligned. For example, the correction process is to calculate the ratio Td between the exposure time Tbase of the proper exposure image and the exposure time Tlow of the low exposure image as shown in the following equation 1, and superimpose the Td as a gain on the AF evaluation value of the low exposure image. Good.
Td = Tbase / Tlow (Formula 1)

As shown in FIG. 5A, for a still subject, the AF evaluation value 502 of the low exposure image has a larger amount of noise (poor S / N) than the AF evaluation value 501 of the proper exposure image. There is.
As shown in FIG. 5B, for a moving subject, the AF evaluation value 502 of the low-exposure image has a smaller contrast decrease due to the movement of the subject than the AF evaluation value 501 of the proper exposure image, and the AF evaluation value It is characterized by a large maximum value.
All of these features are due to the short exposure time.

Next, focus control in the present embodiment will be described with reference to the flowchart of FIG.
The system control unit 110 of this embodiment has an AF evaluation value selection unit 1401 as shown in FIG. The AF evaluation value selection unit 1401 determines the AF evaluation value of the proper exposure image and the AF evaluation value of the low exposure image from the motion vector information Mv from the motion vector detection unit 108 and the exposure step amount N calculated by the system control unit 110. Among them, one of which is determined to have high reliability is selected. The selected AF evaluation value is used in the system control unit 110.

  As shown in the timing chart of FIG. 3, the proper exposure image 305 and the low exposure image 304 are alternately captured, and the AF evaluation value calculation unit 107 calculates an AF evaluation value for each captured image (S601). The AF evaluation value selection unit 1401 in the system control unit 110 switches processing according to the AF evaluation value acquired at the timing of the timing pulse 307 (S602).

  When the low-exposure AF evaluation value is acquired, the correction unit 1402 corrects the low-exposure AF evaluation value so that the luminance levels of the appropriate exposure image and the low-exposure image are aligned (S603). This correction may be the multiplication of Td described above. Then, the correction unit 1402 holds the corrected AF evaluation value in the holding unit 1403 for the AF evaluation value selection process performed for the next frame.

On the other hand, when the appropriate exposure AF evaluation value is acquired, the selection unit 1405 acquires the motion vector information Mv related to the AF evaluation frame from the motion vector detection unit 108 (S607). The motion vector detection unit 108 extracts an image within the AF evaluation frame in one image as a template from the appropriate exposure image and the low exposure image one frame before, and searches for a highly correlated area in the other image. The position difference is calculated as a motion vector. The motion vector detection unit 108 obtains and outputs motion vector information Mv from the horizontal motion vector Mvx and the vertical motion vector Mvy according to the following Expression 2.
Mv = √ (Mvx ² + Mvy ² ) (Formula 2)
Thus, in the present embodiment, the motion vector information Mv is the motion amount itself, but in other embodiments, the motion vector information Mv may be other forms of information representing the motion amount.

In step S <b> 608, the selection unit 1405 acquires the exposure step amount N described above.
Then, the selection unit 1405 performs a process of selecting a low exposure AF evaluation value and a proper exposure AF evaluation value (S608).

The AF evaluation value selection process in S608 will be described using the flowchart shown in FIG.
First, the selection unit 1405 compares the exposure step amount N with the threshold ThN (701). When the exposure level difference N is larger than the threshold ThN, that is, when the exposure level difference between the appropriate exposure image and the low exposure image is large, as described with reference to FIG. 5, the S / N decreases in the AF evaluation value of the low exposure image. For this reason, the reliability of the low-exposure AF evaluation value is lowered, and the possibility that the focusing accuracy is lowered is larger than that in the case where the proper exposure AF evaluation value is used.

  Accordingly, when the exposure step amount N is larger than the threshold value ThN, the selection unit 1405 determines to select an appropriate exposure AF evaluation value that is determined to have a reliability higher than the low exposure AF evaluation value, and the appropriate value obtained in the current frame. SW 1404 is controlled to select an exposure AF evaluation value. Note that the threshold ThN may be set to a predetermined exposure step amount, or may be controlled according to, for example, exposure information (such as a gain setting for a proper exposure image).

  On the other hand, when the exposure step amount N is equal to or smaller than the threshold ThN, the selection unit 1405 compares the motion vector information Mv with the threshold ThMv in S703. When the motion vector information Mv is larger than the threshold ThMv, that is, when the motion of the subject or area for which the AF evaluation value is calculated is large, as described with reference to FIG. 5, the appropriate exposure image has a larger contrast than the low exposure image. Decreases. For this reason, the reliability of the appropriate exposure AF evaluation value is lower than that when the low exposure AF evaluation value is used, and there is a high possibility that the focusing accuracy is lowered when the appropriate exposure AF evaluation value is used.

Therefore, when the motion vector information Mv corresponding to the motion amount is larger than the threshold value ThMv, the selection unit 1405 determines to select a low-exposure AF evaluation value that is determined to be more reliable. Then, the selection unit 1405 controls the SW 1404 so as to select the low-exposure AF evaluation value for the previous frame stored in the holding unit 1403 (S704). On the other hand, when the motion vector information Mv is equal to or less than the threshold ThMv, the selection unit 1405 determines to select an appropriate exposure AF evaluation value that is generally determined to be higher in reliability than the low exposure AF evaluation value. Then, the selection unit 1405 controls the SW 1404 so as to select a proper exposure AF evaluation value obtained in the current frame.
Note that the threshold ThMv may be set to a predetermined amount of motion, or may be controlled according to, for example, exposure information (such as setting the gain of a proper exposure image). Further, when the motion vector information Mv is equal to or less than the threshold value ThMv, both AF evaluation values may be determined to be used because both have higher reliability than the reference.

  In the present embodiment, the reliability of the AF evaluation value to be selected is determined based on the magnitude of the motion vector information Mv and the magnitude of the exposure step amount N, and at least one of which the reliability is determined to be higher than the reference. The configuration for selecting is described. However, the determination of reliability can also be based on other information. For example, it is possible to select an AF evaluation value using shooting information as shown in FIG.

  For example, when the imaging device has an image blur correction function, the device motion amount (camera shake amount) Hv obtained by a known method from means for detecting the motion of the device such as a gyro sensor is used similarly to the motion vector information Mv. AF evaluation value may be selected. This is because that the amount of camera shake is large, it is thought that the contrast of the properly exposed image due to movement is greatly reduced and the reliability is lowered. Therefore, the selection unit 1405 determines the selection of the low exposure AF evaluation value when the camera shake amount Hv is larger than the predetermined threshold ThHv, and the selection of the appropriate exposure AF evaluation value when the camera shake amount Hv is less than the threshold ThHv. To control.

Further, for example, the luminance histogram 204 as shown in FIG. 2 is calculated for the pixels within the AF evaluation frame of the appropriate exposure image, and the number of pixels having a luminance value equal to or higher than a predetermined luminance value Lhigh (up to Lmax) is calculated. The ratio Vh of the total number of pixels (hereinafter referred to as the amount of whiteout) is calculated. The whiteout amount Vh can be calculated by the following equation. In FIG. 8, “high exposure image” is described in the sense of an image having the largest exposure amount among a group of images having different exposure amounts. In this embodiment, since the proper exposure image is the image with the largest exposure amount, the amount of whiteout is calculated for the proper exposure image.

  If the amount of whiteout Vh is large, it is considered that there are many whiteout pixels in the AF evaluation frame of the appropriate exposure image and the reliability of the AF evaluation value is low. Therefore, the selection unit 1405 determines to select a low-exposure AF evaluation value that is determined to be more reliable when the whiteout amount Vh is larger than the threshold value ThVh. On the other hand, the selection unit 1405 determines to select an appropriate exposure AF evaluation value that is generally considered to be more reliable than the low exposure AF evaluation value if it is equal to or less than the threshold value ThVh. The selection unit 1405 controls the SW 1404 according to the determination result.

Further, for example, a luminance histogram 205 as shown in FIG. 2 is calculated for the pixels in the AF evaluation frame of the low-exposure image, and a pixel having an integration value equal to or less than a predetermined luminance value Llow (up to 0), The selection may be made according to the ratio Vl of the number of pixels (hereinafter referred to as the blackout amount). The blackout amount Vl can be calculated by the following equation.

  The large blackout amount Vl is considered that there are many blackout pixels in the AF evaluation frame of the low exposure image and the reliability of the low exposure AF evaluation value is low. Therefore, the selection unit 1405 determines to select the appropriate exposure AF evaluation value when the blackout amount Vl is larger than the threshold value ThVl. The selection unit 1405 also determines to select the low-exposure AF evaluation value if the blackout amount V1 is equal to or less than the threshold ThVh, and controls the SW 1404. Note that the “low exposure image” in FIG. 8 means an image with the smallest exposure amount among a group of images having different exposure amounts.

  Then, the system control unit 110 evaluates the focus state (in-focus state) of the optical system 101 based on the AF evaluation value selected in this way, and calculates the focus lens position where the AF evaluation value is maximized. Then, the system control unit 110 instructs the optical system 101 to move (shift) the focus lens to the calculated position. Then, as indicated by the timing pulse 308 in FIG. 3, the focus lens is driven in the next frame in which the proper exposure AF evaluation value is acquired.

  As described above, in the present embodiment, among the AF evaluation values obtained from each of a plurality of images shot under different exposure conditions, the reliability is higher due to the movement of the imaging device and subject, the situation of overexposure / blackout, etc. One that is determined to be high is dynamically selected and used for focus control of the optical system. For this reason, it is possible to realize contrast AF with good followability to the subject even when shooting for HDR video.

  Note that the focus control in the present embodiment is effective even when HDR images are generated after the proper exposure image and the low exposure image are all recorded without performing image synthesis at the time of shooting. Further, the focus control in the present embodiment can be applied to shooting other than a combination of a proper exposure image and a low exposure image. For example, the present invention can be applied even when a high-exposure image and a low-exposure image having a larger exposure amount than the proper exposure are taken. In this case, the high-exposure image may be applied to the above-described proper exposure image for processing.

  In the present embodiment, an AF evaluation value is mainly used as an example of an evaluation value for obtaining an evaluation value of at least one image from a plurality of exposure images, and a focus lens position or the like is used as a shooting parameter to be controlled based on the evaluation value. Although the control value has been described, the present invention is not limited to this. The evaluation values to which the present embodiment can be applied can be applied to various evaluation values obtained from captured images, such as the above-described subject detection evaluation values, white balance evaluation values, and exposure control evaluation values. The subject detection result is used for AF, exposure control (control of imaging parameters), white balance processing (control of image processing parameters), and the like. The white balance evaluation value is naturally used for white balance processing. Evaluation values for exposure control are used to control exposure control values as imaging parameters.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. The present embodiment is the same as the first embodiment except that the system control unit 110 includes an AF evaluation value synthesis unit instead of the AF evaluation value selection unit, and focus control. Therefore, the following is a feature of the present embodiment. Only the relevant parts will be described.

  The system control unit 110 of this embodiment has an AF evaluation value synthesis unit 1501 as shown in FIG. 15 instead of the AF evaluation value selection unit. 15, common reference numerals are used for the same components as those in FIG. 14, and duplicate descriptions are omitted. The AF evaluation value synthesis unit 1501 calculates a synthesis coefficient based on the motion vector information Mv of the image in the AF evaluation frame calculated by the motion vector detection unit 108 and the exposure step amount N calculated by the system control unit 110. To do. Then, the AF evaluation value combining unit 1501 outputs an AF evaluation value obtained by combining the appropriate exposure AF evaluation value and the low exposure AF evaluation value based on the combination coefficient to the system control unit 110. The system control unit 110 uses the synthesized AF evaluation value for evaluation of the focus state of the optical system 101 and focus control.

  FIG. 9 is a flowchart for explaining the focus control operation in the present embodiment, and the same reference numerals as in FIG. 6 are used for the processes common to the first embodiment. The focus control operation of the present embodiment is different in that it has S901 for performing AF evaluation value synthesis processing instead of the AF evaluation value selection processing in S608. Specifically, the higher the reliability is, the greater the weight is assigned, and a plurality of AF evaluation values are weighted and combined to generate one (synthesized) AF evaluation value. Therefore, only the AF evaluation value composition processing in S901 will be described using the flowchart of FIG.

First, the first synthesis coefficient calculation unit 1505 calculates a synthesis coefficient Cm based on the motion vector information Mv (S1001).
FIG. 11A shows an example of the relationship between the present embodiment, with the horizontal axis representing motion vector information Mv (motion amount) and the vertical axis representing the synthesis coefficient Cm. The synthesis coefficient in the present embodiment is a value indicating the weight of the low exposure AF evaluation value at the time of synthesis in%. Therefore, the synthesis coefficient 100 means that the low-exposure AF evaluation value is used as it is as the AF evaluation value after synthesis. In the present embodiment, the synthesis coefficient Cm is obtained by the following equation using the threshold ThMv. Note that the threshold ThMv can be determined in the same manner as in the first embodiment.
Cm = (Mv−ThMv) × Km (Formula 5)
However, if Cm <0, Cm = 0,
If Cm> 100, Cm = 100.

  Here, Km is the slope of the changing portion of the synthesis coefficient Cm in FIG. In particular, when the motion vector information Mv changes for each frame at a value in the vicinity of the threshold ThMv, a value that smoothly changes the composition coefficient Cm in the vicinity of the threshold so that the AF evaluation value after the composition does not vary greatly. Is set to Km.

Further, the second synthesis coefficient calculation unit 1506 calculates a synthesis coefficient Ce based on the exposure step amount N (S1002).
FIG. 11B shows an example of the relationship between the two in the present embodiment, with the horizontal axis representing the exposure step amount N and the vertical axis representing the synthesis coefficient Ce. The formula for calculating the synthesis coefficient is obtained by the following formula using the threshold ThN. Note that the threshold ThN can be determined in the same manner as in the first embodiment.
Ce = 100− (N−ThN) × Ke (Formula 6)
However, if Ce <0, Ce = 0,
If Ce> 100, Ce = 100.

Here, Ke is the slope of the changing portion of the synthesis coefficient Ce in FIG. In particular, when the exposure step amount N changes for each frame at a value in the vicinity of the threshold ThN, a value for smoothly changing the composition coefficient Ce in the vicinity of the threshold so that the AF evaluation value after the composition does not increase. Is set to Ke.
Note that the synthesis coefficients Cm and Ce may be calculated in parallel.

Next, the coefficient synthesizing unit 1507 synthesizes the synthesis coefficients Ce and Cm to calculate the synthesis coefficient C (S1003).
The synthesis coefficient C can be calculated by simple multiplication as shown by the following equation, for example.
C = (Ce × Cm) / 100 (Formula 7)
Note that the synthesis coefficient C can also be calculated by other methods such as weighting and synthesis of two synthesis coefficients using a value obtained by evaluating exposure information (shutter speed, gain setting, etc.) as a weight. Since the faster the shutter speed, the less blurring occurs, so the weight of the synthesis coefficient Cm can be increased as the shutter speed increases. Further, since the dark portion noise increases as the gain increases, the weight of the synthesis coefficient Ce can be reduced as the gain increases.

Next, the evaluation value synthesis unit 1504 synthesizes the AF evaluation value calculated from the proper exposure image and the AF evaluation value calculated from the low exposure image using the synthesis coefficient C. When the proper exposure AF evaluation value is Afb, the low exposure AF evaluation value is Afl, and the composite AF evaluation value is Afc, a composite AF evaluation value can be obtained according to the following formula.

Here, the case has been described where the synthesis coefficient of the AF evaluation value is calculated from the synthesis coefficient based on the motion vector information and the synthesis coefficient based on the exposure step amount. However, it is also possible to use the shooting information as shown in FIG. 8 instead of the motion vector information and the exposure step amount and obtain the synthesis coefficient by the same method.
For example, the synthesis coefficient can be calculated based on the relationship between the following photographing information and the synthesis coefficient.
The synthesis coefficient is increased as the camera shake amount Hv is larger than the predetermined threshold value ThHv.
As the amount of whiteout Vh becomes larger than a predetermined threshold value ThVh, the synthesis coefficient is increased.
As the blackout amount Vl becomes larger than a predetermined threshold value ThVl, the synthesis coefficient is reduced.

  The system control unit 110 evaluates the focus state (in-focus state) of the optical system 101 using the composite AF evaluation value, and calculates the focus position of the optical system 101 that maximizes the composite AF evaluation value. Then, the focus control is realized by instructing the optical system 101 to move (shift) the focus lens to the calculated position.

  Note that, as in the first embodiment, the focus control in the present embodiment is also performed when HDR images are generated after all the proper exposure images and low exposure images are recorded without performing image synthesis at the time of shooting. It is valid. Further, the focus control in the present embodiment can be applied to shooting other than a combination of a proper exposure image and a low exposure image. For example, the present invention can be applied even when a high-exposure image and a low-exposure image having a larger exposure amount than the proper exposure are taken. In this case, the high-exposure image may be applied to the above-described proper exposure image for processing.

  In this embodiment, since an AF evaluation value synthesized with a weight according to the degree of reliability is used, the reliability is higher than that in the first embodiment in which one AF evaluation value is selected according to the magnitude relationship of a predetermined threshold. It is possible to control the focus using the AF evaluation value that is reflected finely. Therefore, further improvement in the accuracy of focus control is expected. From the viewpoint of using the weight of the evaluation value with high reliability higher than the weight of the evaluation value with low reliability, the first embodiment that selects and uses the evaluation value with high reliability is the present embodiment. It can be said that it is one form.

  In the present embodiment, an AF evaluation value is mainly used as an example of an evaluation value for obtaining an evaluation value of at least one image from a plurality of exposure images, and a focus lens position or the like is used as a shooting parameter to be controlled based on the evaluation value. Although the control value has been described, the present invention is not limited to this. The evaluation values to which the present embodiment can be applied can be applied to various evaluation values obtained from captured images, such as the above-described subject detection evaluation values, white balance evaluation values, and exposure control evaluation values. The subject detection result is used for AF, exposure control (control of imaging parameters), white balance processing (control of image processing parameters), and the like. The white balance evaluation value is naturally used for white balance processing. Evaluation values for exposure control are used to control exposure control values as imaging parameters.

(Third embodiment)
Next, a third embodiment of the present invention will be described. Since this embodiment is the same as the first embodiment except for the difference in focus control processing due to the existence of a plurality of AF evaluation frames, only the characteristic features of this embodiment will be described below.

  The AF evaluation value calculation unit 107 (FIG. 1) in this embodiment has a plurality of functional blocks for calculating an AF evaluation value, and each functional block calculates an AF evaluation value for a different AF evaluation frame designated in advance. To do. Therefore, for example, the AF evaluation value calculation unit 107 calculates AF evaluation values Af1 to Af5 for a plurality of AF evaluation frames 1201 to 1205 shown in FIG.

  The positions of the AF evaluation frames 1201 to 1205 may be fixed or changeable by the user. Further, the user may be able to select from many fixed positions, or an arbitrary position may be designated. In any case, the AF evaluation value calculation unit 107 assigns an internal functional block so as to calculate an AF evaluation value for each of the set AF evaluation frames.

Further, in order to calculate the motion vector information Mv1 to Mv5 for each of the plurality of AF evaluation frames 1201 to 1205, the motion vector detection unit 108 also includes a plurality of functional blocks for calculating the motion vector information.
It is not essential that the AF evaluation value calculation unit 107 and the motion vector detection unit 108 have a plurality of functional blocks corresponding to a plurality of AF evaluation frames. If there is no problem in processing time, the AF evaluation value and the motion vector information may be calculated using functional blocks smaller than the number of AF evaluation frames.

In the present embodiment, the AF evaluation value selection process described in the first embodiment is performed for each of the plurality of AF evaluation frames 1201 to 1205.
That is, for each AF evaluation frame, the system control unit 110 selects an optimal AF evaluation value for focus control from the appropriate exposure AF evaluation value and the low exposure AF evaluation value based on the shooting information. The shooting information in the present embodiment is motion vector information Mv corresponding to the exposure step amount N and the AF evaluation frame.

FIG. 13 is a flowchart for explaining the focus control operation in the present embodiment, and the same reference numerals as in FIG. 6 are used for the processes common to the first embodiment.
As shown in the timing chart of FIG. 3, the AF evaluation value calculation unit 107 alternately calculates the appropriate exposure AF evaluation value and the low exposure AF evaluation value in the AF evaluation frames 1201 to 1205 (S1301). At the timing of the timing pulse 307, the AF evaluation value selection unit 1401 in the system control unit 110 acquires the AF evaluation value of each AF evaluation frame from the AF evaluation value calculation unit 107. Then, the AF evaluation value selection unit 1401 advances the process to S603 when the low exposure AF evaluation value is acquired, and proceeds to S1303 when the appropriate exposure AF evaluation value is acquired.

  When the low-exposure AF evaluation value group is acquired, correction processing in the correction unit 1402 (S603) and holding in the holding unit 1403 (S604) are performed for each AF evaluation frame.

On the other hand, when a proper exposure AF evaluation value group is acquired, the AF evaluation value selection process described in the first embodiment is performed for each AF evaluation frame.
The acquisition of the motion vector information Mv (S606), the acquisition of the exposure step amount N (S607), and the selection of the AF evaluation value used for focus control (S608), except that S606 and S608 are processed for each AF evaluation frame. This is the same processing as in the first embodiment.

  In this embodiment, since the AF evaluation value selection process is the same as that in the first embodiment, instead of selection using the motion vector information Mv and the exposure step amount N, as shown in FIG. An AF evaluation value may be selected using shooting information. In this case, the process is the same as that described in the first embodiment except that the process is performed for each AF evaluation frame.

  When the AF evaluation value selection unit 1401 applies the processing of S606 to S608 for all AF evaluation frames (S1304, YES), the system control unit 110 sets AF evaluation frames used for focus control as AF evaluation frames 1201-1205. (S1305).

  A method for selecting an AF evaluation frame used for focus control from a plurality of AF evaluation frames is not particularly limited, and any known and arbitrary method can be used. For example, it is described in JP-A-2006-330211. The method can be used.

  The system control unit 110 evaluates the focus state (focused state) of the optical system 101 based on the (selected) AF evaluation value obtained for the selected AF evaluation frame, and the focus lens that maximizes the AF evaluation value The position of is calculated. Then, the system control unit 110 instructs the optical system 101 to move (shift) the focus lens to the calculated position, thereby realizing focus control.

(Modification)
In the present embodiment, a mode has been described in which the AF evaluation value selected by the method described in the first embodiment is used for each of a plurality of AF evaluation frames. However, as described in the second embodiment, an AF evaluation value obtained by combining the appropriate exposure AF evaluation value and the low exposure AF evaluation value may be obtained for each AF evaluation value.

  In the present embodiment, an AF evaluation value is mainly used as an example of an evaluation value for obtaining an evaluation value of at least one image from a plurality of exposure images, and a focus lens position or the like is used as a shooting parameter to be controlled based on the evaluation value. Although the control value has been described, the present invention is not limited to this. The evaluation values to which the present embodiment can be applied can be applied to various evaluation values obtained from captured images, such as the above-described subject detection evaluation values, white balance evaluation values, and exposure control evaluation values. The subject detection result is used for AF, exposure control (control of imaging parameters), white balance processing (control of image processing parameters), and the like. The white balance evaluation value is naturally used for white balance processing. Evaluation values for exposure control are used to control exposure control values as imaging parameters.

(Other embodiments)
In the above-described embodiment, in order to facilitate explanation and understanding, the case where two types of photographing of proper exposure and low exposure (underexposure) are periodically performed has been described. However, the present invention is applicable even when three or more types of photographing are periodically performed. In this case, basically, an AF evaluation value obtained from an image having an appropriate exposure or an exposure amount closest thereto may be selected. Also, if the amount of movement is large or the amount of whiteout is large, the AF evaluation value of the low-exposure image depending on the degree is used. If the amount of blackout is large, the AF of the high-exposure image depending on the degree is used. An evaluation value may be used. When combining is performed as in the second embodiment, an AF evaluation value of an image having a larger difference from the appropriate exposure may be set to a smaller maximum value of the combination coefficient. Alternatively, even when three or more types of images are taken, AF evaluation values obtained from the two types of images described in the above embodiment may be used.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (6)

Imaging means for sequentially imaging with different exposure amounts for each frame ;
Image processing means for generating a moving image having a composite image obtained by combining a plurality of images obtained by the imaging means as a frame image ;
Detecting means for detecting the amount of movement of the subject between the images;
Generating means for generating an evaluation value used for focus control of the imaging means ;
Control means for controlling the focus of the imaging means based on the evaluation value generated by the generation means,
Wherein,
When the difference in exposure amount between the plurality of images is larger than a first threshold value, focus control is performed using an evaluation value generated from an image having the largest exposure amount among the plurality of images,
When the difference in exposure amount between the plurality of images is smaller than a first threshold value, one of the images is selected according to the amount of movement of the subject, and focus control is performed using an evaluation value generated from the selected image. An imaging apparatus characterized by performing The control means has the largest exposure amount among the plurality of images when a difference in exposure amount between the plurality of images is smaller than a first threshold value, or when a movement amount of the subject is larger than a second threshold value. The imaging apparatus according to claim 1, wherein focus control is performed using an evaluation value generated from a small image. The imaging apparatus according to claim 1, wherein the composite image is an image in which a dynamic range is expanded with respect to the plurality of images. An imaging step of sequentially imaging with different exposure amounts for each frame by the imaging means;
  An image processing unit that generates a moving image using a combined image obtained by combining the plurality of images obtained in the imaging step as a frame image; and
  A detecting step in which the detecting means detects the amount of movement of the subject between the images;
  A generating step of generating an evaluation value used for focus control of the imaging unit;
  And a control unit that performs focus control of the imaging unit based on the evaluation value generated in the generation step,
  In the control step, the control means includes:
    When the difference in exposure amount between the plurality of images is larger than a first threshold value, focus control is performed using an evaluation value generated from an image having the largest exposure amount among the plurality of images,
    When the difference in exposure amount between the plurality of images is smaller than a first threshold value, one of the images is selected according to the amount of movement of the subject, and focus control is performed using an evaluation value generated from the selected image. A method for controlling an imaging apparatus, comprising: In the control step, the control means includes the plurality of images when the difference in exposure amount between the plurality of images is smaller than a first threshold, or when the amount of movement of the subject is larger than a second threshold. 5. The method according to claim 4, wherein focus control is performed using an evaluation value generated from an image with the smallest exposure amount . 6. The method according to claim 4, wherein the composite image is an image in which a dynamic range is extended with respect to the plurality of images.