JP2017034536A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an excellent image to which an optimal WB correction is applied to each image region of a stroboscope light emission image even when there is a motion in a subject.SOLUTION: A WB control part (303) performs a WB correction of a strobe light emission image by using a first WB correction value corresponding to a strobe light and a second WB correction value corresponding to an environment light. A CPU (314) acquires a subject distance, extracts a mobile region, and determines a synthesis ratio of the strobe light emission image and a strobe non-light emission image in each subject region on the basis of a light amount ratio of the strobe light and environment light. The CPU(314) estimates a strobe irradiation amount on the basis of the subject distance and an arrangement light characteristic of the strobe light, calculates ratios of the strobe irradiation amount and the environment light, set them as the synthesis ratio of the mobile region, and performs synthesis in the other subject region to each subject region by using the determined synthesis ratio.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program that perform white balance correction for each subject area of a photographed image.

  In general, an imaging apparatus (hereinafter referred to as a camera) such as a digital camera or a digital video camera has a white balance control function for adjusting the color tone of a captured image. The white balance control is roughly classified into manual white balance control and auto white balance control. Manual white balance control is image processing in which a white subject is imaged in advance to obtain a white balance coefficient, and the white balance coefficient is applied to the entire captured image for one screen to perform white balance correction. On the other hand, the auto white balance control automatically detects a white part from the captured image and calculates a white balance coefficient from the color component of the part and the average value of each color component of the entire captured image. In auto white balance control, white balance correction is performed by applying the white balance coefficient to the captured image.

  Here, for example, in the case of taking an image with strobe light emission, the subject in the shooting area captured by the camera is not only strobe light but also illumination light from a lighting device such as a room light. Often hits. As described above, when there is light from various light sources in the shooting area, the white balance control described above applies the white balance coefficient to the entire shot image and performs white balance control. It is difficult to obtain a tinted image. For example, when the white balance is adjusted to a high color temperature light source such as strobe light, the white balance is not adjusted to a low color temperature light source such as an incandescent lamp. On the other hand, when the white balance is adjusted to the low color temperature light source of the incandescent lamp, the white balance does not match the high color temperature light source of the strobe light. For example, when white balance control is performed by adjusting white balance between both light sources, the white balance does not match any of both light sources. In this way, when white balance is adjusted between the two light sources, the image of the area irradiated with the strobe light is bluish, and the image of the area irradiated with the light of the low color temperature light source is reddish. It becomes a color like this.

  On the other hand, Patent Document 1 proposes an electronic still camera for the purpose of obtaining an image having a tint suitable for a plurality of light sources as described above. The electronic still camera described in Patent Document 1 compares a strobe light emission image and a strobe non-light emission image for each arbitrary subject area to obtain a data ratio, and determines the contribution of strobe light based on the ratio value. Then, the electronic still camera of Patent Document 1 performs white balance control by selecting a white balance coefficient for each region with respect to captured image data at the time of strobe light emission according to the degree of contribution. Hereinafter, the white balance control performed by selecting the white balance coefficient for each region of the image in this manner is referred to as partial white balance (partial WB) control. In addition, Patent Document 1 describes that a strobe irradiation amount is estimated by taking a difference between a strobe non-emission image and a strobe emission image, and white balance control is performed according to the strobe irradiation amount.

  In Patent Document 2, as an example of partial WB control, a subject blur region and a subject non-blurring region are detected from a captured image, and white balance control is performed on each of the subject blur region and the non-blurring region. An image processing apparatus has been proposed. The white balance coefficient for the non-blurring region is calculated based on the extracted light component and the subject blur region, respectively, by extracting the strobe light emission component and the external light component from the strobe light emission image and the non-light emission image. On the other hand, the white balance coefficient for the subject blur area is calculated by interpolation based on the white balance coefficient of the non-blurr area. Then, the image processing apparatus of Patent Document 2 performs white balance control that applies the calculated white balance coefficient to the image areas of the subject blur area and the non-blurr area on the captured image at the time of flash emission. . Japanese Patent Application Laid-Open No. H10-228561 describes that a high frequency component of an image is detected and a subject blur region is detected based on the high frequency component.

Japanese Patent No. 3540485 Japanese Patent No. 4389671

  As described above, in the technique described in Patent Document 1, white balance is applied to each image area of a captured image at the time of flash emission based on a ratio of data obtained by comparing a flash emission image and a non-flash emission image for each arbitrary subject area. Control is performed. However, in the technique described in Patent Document 1, for example, when a subject or the like is moving, it becomes impossible to accurately determine the ratio of data for each image area, and wrong white balance control is performed. There is a risk of generating an image like a pattern. As described above, in the technique described in Patent Document 1, when a subject or the like is moving, a good image in which optimum white balance control is performed on each image area in a captured image at the time of flash emission is obtained. Difficult to get.

  In the technique described in Patent Document 2, each image region such as a subject blur region is detected based on a high-frequency component of the image, and the white balance coefficient for each image region of the captured image at the time of strobe light emission is strobe light emission. It is calculated based on the light components of the image and the non-light emitting image. However, in the technique of Patent Document 2, for example, when an image in a region itself has a high-frequency component such as a fine pattern, and an image region of a high-frequency component due to subject blurring or camera shake caused by movement of a subject or the like Cannot be distinguished. For this reason, for example, if an image area with a fine pattern is erroneously detected as an image area with movement such as a subject blur area, white balance control for each image area is also erroneous. As described above, the technique disclosed in Patent Document 2 obtains a good image in which optimum white balance control is performed on each image area in a captured image at the time of flash emission when a subject or the like moves. It may not be possible.

  The present invention has been made in view of such problems. The present invention is an image processing that makes it possible to obtain a good image in which optimum white balance correction has been performed on each image area of a captured image at the time of flash emission even when the subject or the like is moving. An object is to provide an apparatus, an image processing method, and a program.

  The image processing apparatus according to the present invention uses the first white balance correction value corresponding to strobe light to correct the white balance of the first image taken at the time of strobe emission under ambient light, and supports environment light. Using the second white balance correction value, correction means for correcting the white balance of the second image taken under ambient light without strobe light emission, and when the first image is taken An acquisition means for acquiring a subject distance representing a distance to each subject; and a subject area of a moving subject that has moved from the first image during the shooting of the first image and the second image. Based on the extraction means for extracting, the first image and the second image, a light quantity ratio of strobe light and ambient light is obtained for each subject area of the first image, and the light quantity ratio for each subject area Based on before The first image is captured based on a determining means for determining a composition ratio for composing the first image and the second image for each subject area, and the subject distance and the light distribution characteristic of the strobe light. Calculating means for estimating the amount of strobe light irradiation for each subject area and calculating the ratio between the amount of strobe light irradiation for each subject area and the ambient light, and for the subject area of the moving subject The calculated ratio is used as the combination ratio, and the subject area of the subject that is not the moving subject is used as the first ratio after the white balance correction is performed using the combination ratio determined by the determination unit. The image processing apparatus includes combining means for combining the image and the second image for each subject area.

  According to the present invention, it is possible to obtain a good image in which optimum white balance correction is performed on each image area of a captured image at the time of flash emission even when the subject or the like is moving. .

It is a figure which shows schematic structure of the imaging device of embodiment. 1 is an overall process flow diagram of an imaging apparatus according to a first embodiment. It is a figure which shows the state which arranged the imaging | photography control in the imaging device in time series. It is a flowchart of a WB correction value calculation process. It is a figure used for description of block division. It is a characteristic view which shows an example of the relationship of the color evaluation value for performing white detection. It is a figure used for description of the correspondence between color temperature and CxCy value. It is a flowchart at the time of performing a subject's motion determination. It is a figure used for description of the motion determination based on a difference color evaluation value. It is a flowchart of an image composition process. It is a processing flowchart which calculates | requires the ratio of composition from a distance map and imaging | photography conditions. It is a processing flowchart of the principal part of 2nd Embodiment. It is a processing flow figure of the principal part of 3rd Embodiment.

  Hereinafter, a preferred embodiment of the present invention will be described. The image processing apparatus according to the present embodiment is applied to, for example, an imaging apparatus such as a digital still camera or a digital video camera, an image processing apparatus that processes an image signal captured by the imaging apparatus, or an information processing apparatus that itself has an imaging function. Is possible. Examples of such information processing apparatuses include various information processing apparatuses such as a tablet terminal, a smartphone, an electronic game machine, a navigation apparatus, and a personal computer. In addition, each process to be described later according to the present embodiment can be realized by either a hardware configuration or a software configuration by a computer program, and the software configuration is realized by a CPU or the like executing the computer program according to the present embodiment. Is possible. These are similarly applied not only to the following first embodiment but also to other embodiments described later.

<First Embodiment>
FIG. 1 shows a schematic configuration example of an imaging apparatus as an application example of the image processing apparatus of the present embodiment. In FIG. 1, a lens barrel 315 includes a lens optical system including a focus lens and a zoom lens, a diaphragm, a mechanical shutter, and the like. The lens barrel control circuit 313 drives and controls the aperture, the mechanical shutter, the focus lens, and the like of the lens barrel portion 315 based on values such as an exposure value calculated by a CPU 314 described later and a driving amount of the focus lens. The lens optical system of the lens barrel 315 forms an optical image of a subject or the like on the imaging surface of the image sensor 301.

  The image pickup element 301 is a solid-state image pickup element such as a CCD image sensor or a CMOS sensor, and an RGB primary color filter having a Bayer arrangement, for example, is arranged on the image pickup surface, and a color image can be picked up. For example, the image sensor 301 captures an image and outputs an image signal under the control of the CPU 314. In the present embodiment, the image sensor 301 includes a preprocessing circuit such as an A / D converter, and a digital image signal via the preprocessing circuit is sent to the memory 302. The memory 302 stores an image signal captured by the image sensor 301 under the control of the CPU 314, and outputs the stored image signal. The image signal output from the memory 302 is sent to the peripheral light amount correction unit 320 and the CPU 314.

  In the present embodiment, the imaging element 301 has a plurality of focus detection pixels arranged on substantially the entire surface within the effective imaging area of the imaging surface. The focus detection pixel is a pixel for generating two images for correlation calculation. Signals of two images for correlation calculation generated by the focus detection pixels are sent to an arithmetic circuit (not shown) provided as a peripheral circuit of the image sensor 301. The arithmetic circuit calculates the phase difference between the two images by correlation calculation. The phase difference signal calculated by the arithmetic circuit from the output signals of all the focus detection pixels of the image sensor 301 is used to calculate the driving amount of the focus lens at the time of focusing control on the subject, or a distance map as described later. It is used for the generation calculation. Note that focusing control based on the phase difference signal calculated from the output signal of the focus detection pixel is an existing technique, and thus detailed description thereof is omitted.

  The peripheral light amount correction unit 320 increases the gain corresponding to each image height on the imaging surface of the image sensor 301 so as to compensate for the decrease in the peripheral light amount that occurs according to the zoom position or aperture value of the zoom lens of the lens barrel unit 315. Determine the amount. Then, the peripheral light amount correction unit 320 corrects the decrease in the peripheral light amount by performing gain adjustment on the image signal based on the gain increase amount. Note that the CPU 314 may calculate the gain increase amount. The image signal after the peripheral light amount correction by the peripheral light amount correction unit 320 is sent to the white balance control unit 303. In the following description, white balance is expressed as “WB”.

  The CPU 314 calculates a shutter speed and an aperture value for photographing with an exposure value so that the entire captured image has a predetermined luminance, and a focus lens for focus control for focusing on a subject in the focus area. Calculate the drive amount. The exposure value (shutter speed and aperture value) calculated by the CPU 314 and the driving amount of the focus lens are sent to the lens barrel control circuit 313. As a result, the lens barrel control circuit 313 drives the aperture, the mechanical shutter, the focus lens, and the like of the lens barrel 315 based on the values supplied from the CPU 314 as described above.

  The CPU 314 also performs strobe light emission control that emits strobe light, strobe non-emission control that does not emit strobe light, and strobe light emission control when strobe light is emitted. When performing the flash emission control, the CPU 314 calculates the flash emission amount such that the subject area of the captured image has a predetermined luminance, along with the control of the shutter speed and the aperture value for the lens barrel 315 described above. Then, the CPU 314 controls the strobe light emission unit 316 so that strobe light emission of the calculated strobe light emission amount is performed. As a result, the strobe light emitting unit 316 emits strobe light having the strobe light emission amount calculated by the CPU 314. The strobe light emitting unit 316 may be a built-in strobe device or an external strobe device.

  In the present embodiment, as will be described later, an image captured by emitting strobe light under ambient light is appropriately referred to as a “strobe emission image”. In the present embodiment, this strobe light emission image is an image that is finally acquired as a captured image (an image that the photographer wants to acquire as a captured image). Hereinafter, this finally acquired captured image is referred to as “main exposure. Indicated as “image”. This main exposure image (flash emission image) is the first image in the present embodiment. In the present embodiment, as will be described later, the second image captured only with ambient light (external light) that does not emit strobe light is appropriately referred to as a “strobe non-light-emitting image”. The strobe non-emission image is the second image in the present embodiment.

  The WB control unit 303 is first and second correction units that perform first and second white balance correction processes in the present embodiment. The WB control unit 303 uses a first white balance correction value (first WB correction value) calculated by the CPU 314 as will be described later, and uses a first WB to be described later for the strobe light emission image that is the main exposure image. Perform correction processing. In addition, the WB control unit 303 uses a second white balance correction value (second WB correction value) calculated by the CPU 314 as described later, and a second described later with respect to a strobe light emission image that is the main exposure image. WB correction processing is performed. Details of the first and second WB correction value calculation processes and the first and second WB correction processes will be described later. A color conversion MTX (matrix) circuit 304 is used for the strobe emission image after the first WB correction processing is performed by the WB control unit 303 and the strobe emission image signal after the second WB correction processing are performed. And Y (luminance signal) generation circuit 311.

  The color conversion MTX circuit 304 multiplies the strobe light emission image signal after the first WB correction processing by a color gain so that the image is reproduced with an optimum color later, and further, the color difference between RY and BY. It converts into a signal and produces | generates a 1st color difference signal. Similarly, the color conversion MTX circuit 304 multiplies the strobe light emission image signal after the second WB correction processing by a color gain so as to be reproduced with an optimum color later, and further performs RY, BY color difference. It converts into a signal and produces | generates a 2nd color difference signal. The first and second color difference signals after color conversion by the color conversion MTX circuit 304 are sent to an LPF (low-pass filter) circuit 305.

  The LPF circuit 305 limits the band of the first color difference signal, and outputs the first band limited color difference signal after the band limitation to a CSUP (Chroma Suppress) circuit 306. Similarly, the LPF circuit 305 limits the band of the second color difference signal and outputs the second band limited color difference signal after the band limitation to the CSUP circuit 306.

  The CSUP circuit 306 reduces the false color component of the saturated portion of the first band limited color difference signal, and outputs the first false color reduced color difference signal after the false color component reduction to the RGB conversion circuit 307. Similarly, the CSUP circuit 306 reduces the false color component of the saturated portion of the second band limited color difference signal, and outputs the second false color reduced color difference signal after the false color component reduction to the RGB conversion circuit 307.

  The Y (luminance signal) generation circuit 311 generates the luminance signal Y from the strobe light emission image after the first WB correction processing under the control of the CPU 314, and outputs it to the edge enhancement circuit 312 as the first luminance signal. Similarly, the Y generation circuit 311 generates a luminance signal Y from the strobe light emission image after the second WB correction process under the control of the CPU 314, and outputs it to the edge enhancement circuit 312 as the second luminance signal.

  The edge enhancement circuit 312 performs edge enhancement processing on the first luminance signal under the control of the CPU 314, and outputs the first edge enhancement luminance signal after the edge enhancement processing to the RGB conversion circuit 307. Similarly, the edge enhancement circuit 312 performs edge enhancement processing on the second luminance signal under the control of the CPU 314, and outputs the second edge enhanced luminance signal after the edge enhancement processing to the RGB conversion circuit 307. .

  The RGB conversion circuit 307 generates a first RGB image signal using the first false color reduction color difference signal from the CSUP circuit 306 and the first edge enhancement luminance signal from the edge enhancement circuit 312. Similarly, the RGB conversion circuit 307 generates a second RGB image signal using the second false color reduction color difference signal from the CSUP circuit 306 and the second edge enhancement luminance signal from the edge enhancement circuit 312. . The first and second RGB image signals are sent to a γ (gamma) correction circuit 308.

  The γ correction circuit 308 performs gradation correction (γ correction) based on the γ curve for the first RGB image signal, and outputs the first corrected RGB image signal to the color luminance conversion circuit 309. Similarly, the γ correction circuit 308 performs γ correction on the second RGB image signal and outputs the second corrected RGB image signal to the color luminance conversion circuit 309.

  The color luminance conversion circuit 309 converts the first corrected RGB image signal into a first YUV image signal. Similarly, the color luminance conversion circuit 309 converts the second corrected RGB image signal into a second YUV image signal. These first and second YUV image signals are sent to the image composition processing unit 325.

  The image composition processing unit 325 generates a composite image signal by combining the first YUV image signal and the second YUV image signal based on the image composition ratio generated by the CPU 314 as described later. Details of the image composition processing in the image composition processing unit 325 will be described later. The composite YUV image signal by the image composition processing unit 325 is sent to the JPEG compression circuit 310.

  The JPEG compression circuit 310 compresses the combined YUV image signal from the image combining processing unit 325 into a format such as JPEG. The compressed image signal is recorded on an external recording medium (not shown) or an internal recording medium.

  Although not shown in FIG. 1, the imaging apparatus according to the present embodiment also includes operation devices such as various switches, dials, and shutter buttons. The shutter button is configured by a switch that can be pressed halfway and fully. An operation input signal by the photographer via the operation device is sent to the CPU 314. The photographer can input operation instructions such as setting of shooting parameters and shooting operations by operating the operation device. Although not shown in FIG. 1, the imaging apparatus according to the present embodiment also includes a display device such as an EVF (electronic viewfinder) or a rear display panel that can display a live view image or the like.

  Hereinafter, the overall processing flow in the imaging apparatus of the present embodiment will be described separately for each block representing each processing, with reference to FIG. Further, FIG. 3 shows the shooting control operation control by the CPU 314 of the imaging apparatus of the present embodiment side by side in time series.

  As illustrated in FIG. 3, the CPU 314 causes the live view control 601 to capture a live view image periodically (for each frame) before a user operation SW1 corresponding to a so-called half-press of the shutter button, for example. When the user operation SW1 is performed, the CPU 314 performs AF lock control 603 and AE lock control 604. The AF lock control 603 is a control for locking the focal length in the auto focus control for driving the focus lens of the lens barrel 315. The AE lock control 604 is a control for locking the exposure value (shutter speed, aperture value) in the automatic exposure control.

  When the user operation SW2 corresponding to the full press of the shutter button is performed, the CPU 314 performs the test light emission control 607 and the main exposure control 608. The test light emission control 607 is a control for causing the strobe light emission unit 316 to perform test light emission before the main exposure is performed. Based on the exposure amount obtained by the test light emission, a proper exposure is obtained by the strobe light emission at the time of the subsequent main exposure. Is performed in order to obtain the amount of strobe light emission that can be obtained. In the main exposure control 608, the strobe light emitting unit 316 is caused to emit light and an image is taken. Further, before the test light emission control 607, the external light control 606 performs photographing by exposure of only ambient light, that is, external light. Hereinafter, the ambient light is appropriately referred to as “external light”. It should be noted that the image capturing by the exposure of only the external light may be performed immediately after the main exposure control 608 or may be performed during the live view control 601.

  In the present embodiment, as in the main exposure control 608 in FIG. 3, the main exposure image that is captured by causing the strobe to emit light under ambient light is a strobe light emission image. Further, in the present embodiment, as in the external light control 606 in FIG. 3 immediately before (or immediately after) the photographing by the main exposure control 608, an image photographed only with ambient light that is not subjected to strobe light emission is a strobe non-light emitting image. It is. In addition, an image captured during the live view control 601 may be used as the non-flash image.

  2 is the data of the main exposure image acquired by the main exposure control 608 in FIG. 3 after the user operation SW2 in FIG. 3 (full pressing operation of the shutter button) is performed by the photographer, for example. is there. The strobe light emission image 100 is stored in the memory 302 of FIG.

  The strobe non-emission image 102 is, for example, a captured image acquired by the external light control 606 in FIG. 3 after the user operation SW2 is performed by the photographer, or the live view control 601 in FIG. This is the latest EVF image acquired in (1). In the description of FIG. 2, a case where the strobe non-light emitting image 102 is an EVF image acquired by the live view control 601 of FIG. 3 will be described as an example. The non-flash image 102 is stored in the memory 302 of FIG.

  The strobe light distribution lens peripheral light amount 137 is strobe light distribution information and lens peripheral light amount data handled by the CPU 314 in FIG. Details of the strobe light distribution information and the lens peripheral light amount data will be described later. Note that information on the strobe light distribution lens peripheral light amount 137 is stored in an information storage memory or the like not shown in FIG.

  The light emission amount information 110 is information on a guide number that is a strobe light emission amount in the strobe light emitting unit 316. Details of the light emission amount information 110 will be described later. The light emission amount information 110 is stored in an information storage memory or the like not shown in FIG.

  The first WB correction value calculation process 111 is a process for calculating a first WB correction value, which will be described later, based on the light emission amount information 110, for example, and is performed by the CPU 314 in FIG. Further, the first WB correction value calculation process 111 may be a process for calculating the first WB correction value based on the strobe light emission image 100. FIG. 2 shows an example in which the first WB correction value is calculated based on the light emission amount information 110. The first WB process 1091 in FIG. 2 is a process for performing the first WB correction using the first WB correction value on the strobe light emission image 100 which is the main exposure image, and the WB control unit 303 in FIG. Is done. The first WB correction value calculation process 111 may be performed by the WB control unit 303 in FIG. Details of the first WB correction value calculation process 111 and the first WB process 1091 will be described later.

  The second WB correction value calculation process 107 is a process for calculating a second WB correction value, which will be described later, from the non-flash image 102, and is performed by the CPU 314 in FIG. The second WB process 1092 in FIG. 2 is a process for performing the second WB correction using the second WB correction value on the strobe light emission image 100 which is the main exposure image, and the WB control unit 303 in FIG. Is done. The second WB correction value calculation process 107 may be performed by the WB control unit 303 in FIG. Details of the second WB correction value calculation process 107 and the second WB process 1092 will be described later.

  Details of the first WB correction value calculation process 111 and the second WB correction value calculation process 107 will be described below. Here, although an example in which the CPU 314 in FIG. 1 performs the first and second WB correction value calculation processes 111 and 107 will be described, the first and second WB correction value calculation processes 111 and 107 WB control are described. This may be performed by the unit 303.

  The first WB correction value calculation processing 111 can calculate the first WB correction value based on the light emission amount information 110 or the strobe light emission image 100 as described above. An example in which the first WB correction value is calculated based on the image 100 will be described. In this case, in the first WB correction value calculation process 111, the CPU 314 performs a first operation from the strobe light emission image 100 (main exposure image) that is captured during strobe emission under ambient light as in the main exposure control 608 in FIG. A WB correction value is calculated.

  FIG. 4 is a detailed flowchart of processing in which the CPU 314 calculates the first WB correction value from the strobe light emission image 100 photographed in the main exposure control 608 of FIG. In the flowchart of FIG. 4, the CPU 314 arbitrarily determines the strobe light emission image 100 acquired by the main exposure control 608 of FIG. 3 for each of a plurality of RGB pixels, as shown in FIG. Divide into m blocks 201. In the following description, the block 201 is simply expressed as a block. After step S101, the CPU 314 advances the process to step S102.

  In step S102, for each block divided into m as described above, the CPU 314 performs an addition average for each color of RGB for all pixels in the block to obtain color average values R [i], G [i], B [i] is calculated. Then, the CPU 314 calculates the color evaluation values Cx [i] and Cy [i] for each block according to the equation (1). Note that “i” in R [i], G [i], B [i] and Cx [i], Cy [i] is, for example, the i-th block among the 1st to m-th blocks. Represents.

Cx [i] = (R [i] −B [i]) / Y [i] × 1024
Cy [i] = (R [i] + B [i] −2G [i]) / Y [i] × 1024 Expression (1)
However, Y [i] = R [i] + 2G [i] + B [i], 1024 is the number of gradations.

  After step S102, the CPU 314 advances the process to step S103. In step S103, the CPU 314 performs white detection using a graph having coordinate axes as shown in FIG. The color evaluation value Cx of the x coordinate in FIG. 6A is a color evaluation value when a negative direction is taken of white of a high color temperature subject, and a color evaluation value is obtained when the positive direction is taken of white of a low color temperature subject. Is shown. Moreover, the color evaluation value Cy of the y coordinate means the degree of the green component of the light source, and indicates that the green (green) component increases as it goes in the negative direction. Here, since the strobe light is a known light source, the CPU 314 uses the white detection range 401 limited as shown in FIG. 6A for the strobe emission image by the main exposure control 608 in FIG. Perform detection.

  The graph of FIG. 7A and the table of FIG. 7B are blackbody radiation axes, and show the relationship between the color temperature and the color evaluation values Cx and Cy. These graphs in FIG. 7A and the table in FIG. 7B are obtained by photographing an image with an imaging device in advance with a known color temperature light source on a black body radiation axis, and calculating color evaluation values Cx, Cy from the photographed image. It is calculated by calculating. Of course, not only the color temperature direction but also the correspondence relationship with the color evaluation values Cx and Cy may be obtained for a light source including many color components in the green direction and magenta direction such as a white fluorescent lamp. Good. Further, if the color evaluation values Cx and Cy corresponding to the light emission color temperature of the strobe are obtained, the first WB correction value corresponding to the strobe light source can be calculated if the color temperature of the strobe light is known. It becomes.

  Returning to the flowchart of FIG. 4, in step S103, the CPU 314 determines that the color evaluation values Cx [i] and Cy [i] of the i-th block calculated in step S102 are the strobes shown in FIG. It is determined whether or not it is included in the white detection range 401 for light. If the CPU 314 determines in step S103 that the color evaluation values Cx [i] and Cy [i] of the i-th block are included in the white detection range 401, the CPU 314 determines that the image of that block is white. Then, the process proceeds to step S104.

  In step S104, the CPU 314 determines the color average values R [i], G [i], and B [i] of the i-th block determined to be included in the white detection range 401 in step S103 by the following equation (2). Accumulate as follows. After step S104, the CPU 314 advances the process to step S105. On the other hand, if the CPU 314 determines in step S103 that the color evaluation values Cx [i] and Cy [i] of the i-th block are not included in the white detection range 401, the addition in step S104 is not performed. Then, the process proceeds to step S105. The processing of step S103 and step S104 can be expressed by equation (2). Expression (2) is an expression for obtaining integrated values SumR, SumG, and SumB of the color average values R [i], G [i], and B [i] of the i-th block.

  Here, when the color evaluation values Cx [i] and Cy [i] are included in the white detection range 401 in FIG. 6A, the CPU 314 sets Sw [i] in Expression (2) to “1”. On the other hand, if it is not included in the white detection range 401, Sw [i] in Expression (2) is set to “0”. Accordingly, the CPU 314 performs addition (integration) of the color average values R [i], G [i], and B [i] or adds (accumulation) for each block according to the determination result of step S103. The process of whether or not to perform is substantially switched.

  Next, when the processing proceeds to step S105, the CPU 314 determines whether or not the processing from step S102 to step S104 is completed for all blocks. If there is an unprocessed block, the CPU 314 returns the process to step S102 and performs each process after step S102. On the other hand, if the CPU 314 determines in step S105 that the processing has been completed for all blocks, the process proceeds to step S106.

  In step S106, the CPU 314 calculates the first WB correction values WBCo1_R1, WBCo1_G1, and WBCo1_B1 according to the equation (3) using the integral values SumR, SumG, and SumB calculated in the above equation (2). In Expression (3), the integral values SumR, SumG, and SumB calculated from Expression (2) from the strobe light emission image are expressed as sumR1, sumG1, and sumB1.

WBCo1_R1 = sumY1 × 1024 / sumR1
WBCo1_G1 = sumY1 × 1024 / sumG1 Formula (3)
WBCo1_B1 = sumY1 × 1024 / sumB1
However, sumY1 = (sumR1 + 2 × sumG1 + sumB1) / 4

  In the description so far, the example in which the first WB correction value is calculated based on the strobe light emission image 100 has been described. However, the CPU 314 performs the first WB correction value calculation processing 111 based on the light emission amount information 110. It is also possible to calculate the first WB correction value. That is, since the strobe light is a known light source, the CPU 314 may obtain the first WB correction value by using the strobe light amount information of the strobe light emitting unit 316 described above as the light amount information 110 in FIG. it can. The calculation process of the first WB correction value based on the light emission amount information 110 will be described later.

  Next, as the second WB correction value calculation process 107 in FIG. 2, an example in which the CPU 314 in FIG. 1 calculates the second WB correction value based on the non-flash image 102 will be described. In the second WB correction value calculation processing 107, the CPU 314 performs a second WB correction from the strobe non-emission image 102 (EVF image) taken under the ambient light in which the flash emission is not performed in the live view control 601 in FIG. Calculate the value. The second WB correction value calculation process 107 may be performed by the WB control unit 303 in FIG. Further, the second WB correction value may be calculated from the strobe non-light emission image photographed by the external light control 606 in FIG.

  When calculating the second WB correction value from the strobe non-emission image 102 which is an EVF image, the CPU 314 performs the second WB correction value WBCol_R2, WBCol_G2 in the same manner as in the first WB correction value calculation process 111 described above. , WBCol_B2 is calculated.

  In the process of calculating the second WB correction value, the CPU 314 divides the strobe non-emission image into m blocks as described above in step S101 of FIG.

  In the case of the second WB correction value calculation process, in step S104 of FIG. 4, for example, the CPU 314 converts the color evaluation values Cx [i] and Cy [i] of the i-th block to FIG. It is determined whether or not it is included in the external light white detection range 402 shown. The reason why the wide white detection range 402 as shown in FIG. 6B is used is that strobe light is a known light source, whereas external light (environment light) is not a known light source. This is because the detection range cannot be limited like the white detection range 401 in FIG. In the white detection range 402 for external light shown in FIG. 6B, a white object is captured in advance under a different light source, and a color evaluation value obtained from the captured image data is plotted along the black body radiation axis. It is generated by. The white detection range 402 depends on whether the outside light (environment light) is, for example, sunlight, cloudy light, or light of any kind of light source such as indoor fluorescent light or light from a light bulb. It can be set individually. Which one of the white detection ranges 402 corresponding to each type of external light is used is determined depending on, for example, selection of a plurality of light source photographing modes prepared according to the type of each external light. The color evaluation values Cx and Cy in FIG. 6B are expressed in the same manner as in FIG. 6A described above, and as the color evaluation value Cy becomes negative, the green component increases and approaches, for example, a fluorescent light source. It is shown that.

  In the second WB correction value calculation process, the CPU 314 calculates the second WB correction values WBCo1_R2, WBCo1_G2, and WBCo1_B2 in step S106 of FIG. In addition, when the integral value calculated by the above-described equation (2) from the non-flash image is expressed as sumR2, sumG2, and sumB2, sumR2, sumG2, and sumB2 are used instead of sumR1, sumG1, and sumB1 in equation (3). It is done. In Expression (3), the second WB correction values WBCo1_R2, WBCo1_G2, and WBCo1_B2 are calculated instead of WBCo1_R1, WBCo1_G1, and WBCo1_B1.

The description returns to the processing flow of FIG.
A first development process 1131 in FIG. 2 is a process corresponding to the color conversion MTX circuit 304 to the color luminance conversion circuit 309 in FIG. 1, and after the first WB process 1091 using the first WB correction value. Development processing of the strobe light emission image 100 (main exposure image). An image after the strobe light emission image 100 (main exposure image) is converted into a YUV image by the first development processing 1131 is the above-described first YUV image. The first YUV image is represented by a luminance color evaluation value Y1 [i] and color difference color evaluation values u1 [i], v1 [i]. The first YUV image (Y1 [i], u1 [i], v1 [i]) is sent to an image composition process 114 described later.

  Similarly, the second development process 1132 is a process corresponding to the color conversion MTX circuit 304 to the color luminance conversion circuit 309 in FIG. 1, and is performed after the second WB process 1092 using the second WB correction value. Development processing of the strobe light emission image 100 (main exposure image). The image after the strobe light emission image 100 (main exposure image) is converted into the YUV image by the second development processing 1132 is the above-described second YUV image. The second YUV image is represented by a luminance color evaluation value Y2 [i] and a color difference color evaluation value u2 [i], v2 [i]. The second YUV image (Y2 [i], u2 [i], v2 [i]) is sent to an image composition process 114 described later.

  The resizing process 101 and the resizing process 106 are processes performed by the CPU 314 in FIG. 1, for example. The resizing process 101 and the resizing process 106 are performed when the non-flash image 102 is an image having a resolution lower than that of the main exposure image 100, such as an image for EVF acquired by the live view control 601 in FIG. Process. Specifically, in the resizing process 101, a process of resizing the main exposure image 100 (flash emission image) to a size corresponding to an image for EVF is performed. In the resizing process 101, after resizing the main exposure image (flash emission image 100), the resized image is divided into blocks as described above. The resizing process 106 is a process for resizing image composition ratio information described later from a size corresponding to the EVF image to a size corresponding to the main exposure image 100. Note that if the strobe non-emission image is an image having the same resolution as the main exposure image acquired by the main exposure control 608, such as an image acquired by the external light control 606 in FIG. The resizing process 106 can be omitted. Whether or not to execute the resizing process 101 and the resizing process 106 may be switched depending on whether an EVF image is used as the strobe non-light-emitting image or a photographed image by the external light control 606 is used.

  The exposure amount calculation process 103 is, for example, a process performed by the CPU 314 in FIG. Based on the ISO sensitivity, aperture value, and shutter speed when the flash emission image 100 and the non-flash image 102 are respectively captured in the exposure amount calculation process 103, the CPU 314 performs exposure when the images are captured. Calculate the amount. Here, the exposure may differ between the flash emission image 100 and the non-flash image 102 due to differences in ISO sensitivity, aperture value, shutter speed, and the like. Further, as will be described later, in the present embodiment, the strobe light emission amount is obtained based on the comparison of the light amount of the strobe light and the ambient light (external light) in the area-specific strobe light amount calculation processing 104 performed later. For this reason, the CPU 314 determines whether the strobe light emission image 100 and the strobe non-light emission image 102 are captured based on the ISO sensitivity, the aperture value, and the shutter speed when the strobe light emission image 100 and the strobe non-light emission image 102 are captured. The set exposure amount is obtained. Further, the CPU 314 obtains a difference between the exposure amount of the strobe light emission image 100 and the exposure amount of the strobe non-light emission image 102 in the exposure amount calculation processing 103.

  In the exposure amount calculation process 103, the CPU 314 divides the strobe non-emission image 102 into blocks as described above. In the case of the present embodiment, the area of the block of the flash emission image 100 subjected to the resizing process 101 as described above and the area of the block of the flash non-light emission image 102 of the image size for EVF are made the same area with respect to the angle of view. ing. Then, the CPU 314 obtains integrated values sumR2, sumG2, and sumB2 for each block in the exposure amount calculation process 103 in the same manner as described above for the strobe non-light-emitting image 102. Further, in the exposure amount calculation process 103, the CPU 314 performs a conversion process according to the above-described exposure amount difference with respect to the integrated value for each block of the strobe non-light-emitting image 102.

  The area-specific strobe light emission calculation process 104 (hereinafter referred to as the light emission quantity calculation process 104) is a process performed by the CPU 314 in FIG. In the light emission amount calculation processing 104, the CPU 314 performs strobe light emission for each image region based on the exposure amount difference obtained in the exposure amount calculation processing 103, the strobe light emission image 100 subjected to the resizing processing 101, and the strobe non-light emission image 102. Find the amount.

  Here, in this embodiment, WB correction is performed for each image area. The image area in this embodiment is an area of various subject images such as a person, a building, the sky, and a vehicle, and is described as a subject area in this embodiment. For this reason, the CPU 314 obtains a strobe light emission amount for each subject area based on the strobe light emission image 100 subjected to the resizing process 101 and the strobe non-light emission image 102 in the light emission amount calculation process 104. The strobe emission amount for each subject area is the amount of irradiation with the strobe light for each subject area, and will be referred to as the strobe irradiation amount for each subject area in the following description.

  The outside light / strobe light ratio generation process 133 for each area (hereinafter referred to as a light ratio generation process 133) is a process performed by the CPU 314 in FIG. In the light ratio generation processing 133, the CPU 314 performs strobe light and ambient light for each subject area based on the strobe irradiation amount by the light emission amount calculation processing 104 and the strobe light emission image 100 and the strobe non-light emission image 102 after the resizing processing 101. The light quantity ratio of (external light) is obtained.

  The image composition ratio determination process 105 (hereinafter referred to as composition ratio determination process 105) is a process performed by the CPU 314 in FIG. In the composition ratio determination process 105, the CPU 314 determines a composition ratio for each subject area when the strobe light emission image 100 and the strobe non-light emission image 102 are combined based on the light amount ratio obtained in the light ratio generation process 133. .

  The first color information acquisition process 1201 is a process performed by the CPU 314 in FIG. In the first color information acquisition process 1201, the CPU 314 acquires the strobe light emission image 100, calculates the color average values R [i], G [i], B [i] for each block, and further for each block. Are integrated values sumR1, sumG1, and sumB1. In the following description, the color average values R [i], G [i], and B [i] for each block calculated from the strobe light emission image 100 are R1 [i], G1 [i], and B1 [i. ]. Details of the first color information acquisition processing 1201 will be described later. The average color values R1 [i], G1 [i], B1 [i] and integrated values sumR1, sumG1, sumB1 for each block of the strobe light emission image 100 by the first color information acquisition processing 1201 are the strobe light emission image color. It is sent to the correction process 121.

  The second color information acquisition process 1202 is a process performed by the CPU 314 in FIG. In the second color information acquisition processing 1202, the CPU 314 acquires the strobe non-light-emitting image 102, calculates the color average values R [i], G [i], and B [i] for each block, and further blocks The integral values sumR2, sumG2, and sumB2 are calculated for each. In the following description, the color average values R [i], G [i], and B [i] for each block calculated from the non-flash image 102 are represented by R2 [i], G2 [i], and B2 [ i]. Details of the processing in the second color information acquisition processing 1202 will be described later. The color average values R2 [i], G2 [i], B2 [i] and the integrated values sumR2, sumG2, sumB2 for each block of the strobe non-light-emitting image 102 obtained by the second color information acquisition processing 1202 are strobe light-emitting image colors. It is sent to the taste correction process 121.

  The strobe emission image color correction process 121 (hereinafter referred to as the color correction process 121) is a process performed by the CPU 314 in FIG. The CPU 314 performs strobe correction based on the information obtained from the first and second color information acquisition processes 1201 and 1202, the light emission amount information 110, and the light amount ratio obtained by the light ratio generation process 133 as the color correction process 121. A color evaluation value for each block of the luminescent image 100 is calculated. Details of the hue evaluation value calculation process for each block of the strobe light emission image 100 by the hue correction process 121 will be described later. The color evaluation value of the strobe light emission image 100 calculated by the color correction processing 121 is sent to the moving body region extraction processing 123.

  The moving body region extraction process 123 is a process performed by the CPU 314 in FIG. The CPU 314 detects a subject area (hereinafter referred to as a moving body area) of a moving subject that has moved when the strobe light emission image 100 is captured as the moving body area extraction process 123. Specifically, the CPU 314 obtains a difference value between the tint evaluation value of the strobe light emission image 100 and the tint evaluation value of the strobe non-light emission image 102 in the moving body region extraction processing 123. Then, the CPU 314 extracts a moving body region from the strobe light emission image 100 based on the difference value between the tint evaluation values. Details of the mobile object region extraction process in the mobile object region extraction process 123 will be described later. The information on the mobile body area extracted by the mobile body area extraction process 123 is sent to the mobile body area synthesis ratio correction process 140.

  The strobe light ratio estimation process 130 (hereinafter referred to as the ratio estimation process 130) is a process performed by the CPU 314 in FIG. In the ratio estimation process 130, the CPU 314 performs the strobe light and the ambient light (for the subject area) on the basis of the distance map, strobe light distribution information, the lens peripheral light amount, and the light emission amount information 110 described later. The ratio (light quantity ratio) to external light) is calculated. Details of the ratio calculation process for each subject area in the ratio estimation process 130 will be described later. The ratio information for each subject area obtained by the ratio estimation process 130 is sent to the moving object area synthesis ratio correction process 140.

  The moving body region composition ratio correction process 140 (hereinafter referred to as composition ratio correction process 140) is a process performed by the CPU 314 in FIG. In the composition ratio correction process 140, the CPU 314 sets the composition ratio calculated in the composition ratio determination process 105 for the mobile body area extracted in the mobile object area extraction process 123 as the ratio by the ratio estimation process 130. Details of the ratio correction process for the moving object region in the combination ratio correction process 140 will be described later. Information on the composition ratio after the correction processing by the composition ratio correction process 140 is sent to the resizing process 106.

  The resizing process 106 is a process performed by the resizing processing unit 324 in FIG. Since the composition ratio after the correction processing by the composition ratio correction processing 140 is information corresponding to the size of the EVF image, the resizing processing unit 324 in FIG. 1 resizes the information to correspond to the main exposure image 100. Information on the composition ratio after resizing by the resizing processing 106 is sent to the image composition processing 114.

  The image composition processing 114 is processing performed by the image composition processing unit 325 in FIG. In the image composition processing 114, the image composition processing unit 325 converts the first YUV image after the first development processing 1131 and the second YUV image after the second development processing 1132 into an image after the resizing processing 106. A composite YUV image is generated by combining based on the composite ratio. In the case of the present embodiment, the composition ratio for the moving body region is corrected to the ratio obtained by the ratio estimation process 130 described above. Therefore, the first and second YUV images are synthesized with the ratio of the ratio estimation process 130 in the moving body area, and the first and first YUV images are synthesized with the synthesis ratio of the image synthesis ratio determination process 105 in the other subject areas. This is a composite image of the two YUV images. The composite YUV image signal after the image composition processing by the image composition processing 114 is sent from the image composition processing unit 325 in FIG. 1 to the JPEG compression circuit 310 in FIG.

  In FIG. 8, when the CPU 314 in FIG. 1 calculates a tint evaluation value from the strobe light emission image 100 and the strobe non-light emission image 102 and extracts a moving body region from the strobe light emission image 100 based on the color evaluation value. The flowchart of a motion area | region determination process is shown. The flowchart of FIG. 8 shows processing from the first and second color information acquisition processing 1201 and 1202 to the moving body region extraction processing 123 of FIG.

  In step S501 of FIG. 8, the CPU 314 acquires the strobe non-light emitting image 102 in the second color information acquisition process 1202 of FIG. After step S501, the CPU 314 advances the process to step S502.

  The process of step S502 is included in the second color information acquisition process 1202 and the color correction process 121 of FIG. In the second tint information acquisition process 1202 in step S502, the CPU 314 determines the color average values R2 [i], G2 [i], B2 for each block as described above from the strobe non-light emitting image 102 acquired in step S501. [i] is calculated. Further, in the second color information acquisition process 1202, the CPU 314 uses the color average values R2 [i], G2 [i], and B2 [i] for each block to perform the same calculation as the above-described equation (2). Thus, the integral values sumR2, sumG2, and sumB2 for each block are calculated. Further, in the tint correction processing 121 in step S502, the CPU 314 calculates the luminance value b [i] for each block in the strobe non-light-emitting image 102 by the equation (4).

  b [i] = 0.3 * R2 [i] + 0.6 * G2 [i] + 0.1 * B2 [i] (4)

  After step S502, the CPU 314 advances the process to step S503. The process of step S503 is included in the first color information acquisition process 1201 in FIG. In the first color information acquisition process 1201 in step S503, the CPU 314 acquires the strobe light emission image 100. After step S503, the CPU 314 advances the process to step S504.

  The process of step S504 is included in the first color information acquisition process 1201 in FIG. In the color information acquisition process 1201 in step S <b> 504, the CPU 314 acquires the color temperature information of the strobe from the light emission amount information 110. The color temperature of the strobe is determined from the relationship between the color temperature and the light emission time as the light emission characteristic of the xenon tube when the strobe light emitting unit 316 has, for example, a xenon tube. It can be calculated. After step S504, the CPU 314 advances the process to step S505.

  The process of step S505 is included in the second color information acquisition process 1202 and the color correction process 121 in FIG. In the first color information acquisition process 1201 in step S505, the CPU 314 calculates the color average values R1 [i], G1 [i], and B1 [i] for each block from the strobe light emission image 100 acquired in step S503. . Further, the CPU 314 uses the color average values R1 [i], G1 [i], and B1 [i] for each block of the strobe light emission image 100, and calculates the integral value sumR1 for each block by the same calculation as the above-described equation (2). , SumG1, sumB1 are calculated. In the tint correction process 121 in step S505, the CPU 314 calculates the luminance value a [i] for each block in the strobe light emission image 100 using Expression (5).

  a [i] = 0.3 * R1 [i] + 0.6 * G1 [i] + 0.1 * B1 [i] (5)

  After step S505, the CPU 314 advances the process to step S506. The process of step S506 is included in the color correction process 121 of FIG. In the tint correction processing 121 in step S506, the CPU 314 estimates the amount of strobe light irradiation for each block. Specifically, the CPU 314 calculates the luminance value b [i] for each block of the strobe non-light-emitting image 102 acquired in step S502 and the luminance for each block of the strobe light-emitting image 100 calculated in step S505 according to equation (6). The difference c [i] from the value a [i] is obtained. The difference c [i] between the luminance value b [i] and the luminance value a [i] is the estimated strobe light irradiation amount.

  c [i] = a [i] −b [i] (6)

  When the exposure amount and sensitivity setting such as the shutter speed, aperture value, and ISO sensitivity setting are different between the strobe light emission image 100 and the strobe non-light emission image 102, the luminance component b [i] of the strobe non-light emission image 102 is different. The exposure difference is corrected. As a result, the exposure amount of the strobe component can be calculated. After step S506, the CPU 314 advances the process to step S507.

  The process of step S507 is included in the color correction process 121 of FIG. In the tint correction processing 121 in step S507, the CPU 314 evaluates the color of the strobe image 100 based on the integrated value for each block in steps S502 and S505, the strobe irradiation amount in step S506, and the strobe color temperature information in step S504. Calculate the value.

  In the color tone correction process 121 of FIG. 2 in step S507, the CPU 314 obtains the color evaluation value of the strobe non-light emitting image 102. Specifically, the CPU 314 uses the integrated values sumR2, sumG2, and sumB2 for each block in step S502, and performs the same calculation as in the above equation (3), using the WB coefficients WB_R_evf [i], WB_G_evf [i], WB_B_evf [i ] Is calculated. WB_R_evf [i] is an R (red) WB coefficient, WB_G_evf [i] is a G (green) WB coefficient, and WB_B_evf [i] is a B (blue) WB coefficient. Then, in the tint correction process 121, the CPU 314 calculates the tint evaluation values WCx_evf [i] and WCy_evf [i] for each block in the strobe non-light-emitting image 102 by the following formulas (7) and (8). .

R2 ′ [i] = WB_R_evf [i] × R2 [i]
G2 ′ [i] = WB_G_evf [i] × G2 [i] (7)
B2 ′ [i] = WB_B_evf [i] × B2 [i]

WCx_evf [i] = (R2 ′ [i] −B2 ′ [i]) / Y2 ′ [i] × 1024
WCy_evf [i] = (R2 ′ [i] + B2 ′ [i] −2G2 ′ [i]) / Y2 ′ [i] × 1024
However, Y2 ′ [i] = (R2 ′ [i] + 2G2 ′ [i] + B2 ′ [i]) / 4 (8)

  In the color correction processing 121 in step S507, the CPU 314 obtains the color evaluation value of the strobe light emission image 100. The CPU 314 calculates the WB coefficients WB_R_flash [i], WB_G_flash [i], and WB_B_flash [i] using the integration values sumR1, sumG1, and sumB1 for each block in step S505, by the same calculation as the above-described equation (3). . WB_R_flash [i] is an R (red) WB coefficient, WB_G_flash [i] is a G (green) WB coefficient, and WB_B_flash [i] is a B (blue) WB coefficient. However, in the case of the strobe light emission image 100, R1 [i], G1 [i], and B1 [i] are used in place of R2 [i], G2 [i], and B2 [i] in the above equation (4). . In Expression (4), WB_R_flash [i], WB_G_flash [i], and WB_B_flash [i] are used instead of WB_R_evf [i], WB_G_evf [i], and WB_B_evf [i]. In the formula (4), R1 ′ [i], G1 ′ [i], and B1 ′ [i] are used instead of R2 ′ [i], G2 ′ [i], and B2 ′ [i].

  When obtaining the color evaluation value of the strobe light emission image 100, the CPU 314 corrects the WB coefficients WB_R_flash [i], WB_G_flash [i], and WB_B_flash [i] in the tint correction processing 121 in step S507. Specifically, the CPU 314 uses the WB coefficients WB_R_evf [i], WB_G_evf [i], WB_B_evf [i] calculated from the non-flash image 102, the stroboscopic irradiation amount for each subject area, and the strobo color temperature information. The corresponding correction process is performed. Here, the correction value for correcting the WB coefficients WB_R_flash [i], WB_G_flash [i], and WB_B_flash [i] is obtained as a variable value according to the ratio between the ambient light (external light) and the strobe light. Specifically, the CPU 314 sets the luminance component synthesis ratio α [i] obtained by Expression (9) as the correction value for the WB coefficient.

  α [i] = c [i] / (a [i] + c [i]) (9)

  In addition, the CPU 314 uses the WB coefficients and the luminance component synthesis ratio α [i] in steps S502 and S505, and uses the WB coefficients WB_R_cap [i], WB_G_cap [i], and WB_B_cap [i] according to the ratio of the ambient light to the strobe light. ] Is calculated. Formulas for calculating the WB coefficients WB_R_cap [i], WB_G_cap [i], and WB_B_cap [i] are expressed by Expression (10). The WB coefficients WB_R_cap [i], WB_G_cap [i], and WB_B_cap [i] are corrected WB coefficients in the strobe light emission image 100.

WB_R_cap [i] = WB_R_evf [i] × (1−α [i]) + WB_R_flash [i] × α [i]
WB_G_cap [i] = WB_G_evf [i] × (1−α [i]) + WB_G_flash [i] × α [i]
WB_B_cap [i] = WB_B_evf [i] × (1−α [i]) + WB_B_flash [i] × α [i]
... Formula (10)

  Further, the CPU 314 multiplies the WB coefficients WB_R_cap [i], WB_G_cap [i], and WB_B_cap [i] by the color average values R1 [i], G1 [i], and B1 [i] of the strobe light emission image 100 to obtain R1 ". [i], G1 "[i], B1" [i] are obtained. The calculation formula of these R1 "[i], G1" [i], B1 "[i] is expressed by the following equation (11).

R1 "[i] = WB_R_cap [i] × R1 [i]
G1 "[i] = WB_G_cap [i] × G1 [i] (11)
B1 "[i] = WB_B_cap [i] × B1 [i]

  Then, the CPU 314 obtains the tint evaluation values WCx_cap [i] and WCy_cap [i] in the strobe light emission image 100 by the calculation of Expression (12).

WCx_cap [i] = (R1 "[i] -B1" [i]) / Y1 "[i] * 1024
WCy_cap [i] = (R1 "[i] + B1" [i] -2G1 "[i]) / Y1" [i] × 1024
However, Y1 "[i] = (R1" [i] + 2G1 "[i] + B1" [i]) / 4 ... Formula (12)

  After step S507, the CPU 314 advances the process to step S508. The process of step S508 is included in the moving body region extraction process 123 of FIG. Specifically, the CPU 314 calculates the absolute difference between the tint evaluation values WCx_cap [i] and WCy_cap [i] of the strobe light emission image 100 and the tint evaluation values WCx_evf [i] and WCy_evf [i] of the strobe non-light emission image 102. ΔWCx and ΔWCy are obtained. Further, the CPU 314 obtains the total difference absolute value for each block as the difference color evaluation value ΔWCxWCy. The calculation of the difference absolute values ΔWCx, ΔWCy and the difference color evaluation value ΔWCxCy for each block in step S507 can be expressed by Expression (13).

ΔWCx = | WCx_cap [i] −WCx_evf [i] |
ΔWCy = | WCy_cap [i] −WCy_evf [i] | Expression (13)
ΔWCxCy = ΔWCx + ΔWCy

  After step S508, the CPU 314 advances the process to step S509. The process of step S509 is included in the moving body region extraction process 123 of FIG. In the moving body region extraction process 123 of step S508, the CPU 314 performs a motion determination from the difference color evaluation value ΔWCxCy calculated for each block in step S508 as a motion determination process in the moving body region extraction process 123 of FIG. Do.

  FIG. 9 shows an example in which motion determination is performed from the difference color evaluation value ΔWCxCy for each block. Each square in FIG. 9 corresponds to a block, and the number in each square represents a specific numerical value of the difference color evaluation value ΔWCxCy of each block. As illustrated in FIG. 9, the CPU 314 performs a motion determination using a threshold value on the difference color evaluation value ΔWCxCy. In FIG. 9, when a value of “50” is set as the threshold value, each block with a thick frame in which the value of the difference color evaluation value ΔWCxCy is “50” or more is represented by each of the subject areas of the moving subject. An example of determining a block is given.

  As described above, the strobe irradiation amount is calculated from the difference luminance value between the strobe light emission image 100 and the strobe non-light emission image 102. For example, in the image area where the subject is not moving, the strobe irradiation amount is appropriately calculated. . Therefore, if the subject does not move and the calculation of the strobe irradiation amount is successful, an appropriate value is also calculated for the WB coefficient, and a correct value can also be calculated for the tint evaluation value. On the other hand, when the subject moves, there is a possibility of calculating an incorrect strobe exposure amount, and the WB coefficient cannot be calculated correctly. As a result, the color evaluation value is also calculated incorrectly. It will be. Therefore, in this embodiment, it is possible to determine whether or not the subject has moved by using the absolute difference value of the color evaluation value.

  In the present embodiment, the threshold value for the difference color evaluation value ΔWCxCy is set to a constant value. However, even if the threshold value is made variable according to the color evaluation values WCx_evf [i] and WCy_evf [i] of the strobe non-emission image 102. Good. That is, the difference color evaluation value ΔWCxCy varies depending on the color depending on the color of the subject and the color temperature that is the shooting environment when the strobe is not lit. The accuracy of region determination can be improved.

  Further, as described above, the difference color evaluation value ΔWCxCy used for determining the movement area of the subject area is multiplied by the WB coefficient corresponding to the color temperature of the strobe light, but the correction of the color component has not been performed. Absent. In this case, for example, a color component can be accurately calculated for an image region with a gray color, but there may be some error for a chromatic color. For example, in the case of a low color temperature light source, a reddish subject or a bluish subject under a high color temperature light source tends to develop a strong color. In order to reduce an error in the difference in color developability for each color temperature, the color correction according to the color temperature of the ambient light and the WB coefficient in the non-flash image 102, and the correction for each color of the subject are performed. It is possible to improve. For example, the correction gains of the color evaluation values WCx and WCy are calculated in advance for each of the color temperature and the applied WB coefficient for the color evaluation values of both the non-flash image 102 and the flash image 100. It is feasible. Expression (14) shows an arithmetic expression for obtaining the color evaluation values WCx ′ and WCy ′ obtained by correcting the color evaluation values WCx and WCy according to the WB coefficient and the color temperature. However, a, b, c, and d in Equation (14) are correction coefficients calculated for each color temperature.

  The correction coefficients a, b, c, and d are obtained by, for example, photographing a color chart for each color temperature and calculating the color evaluation values WCx and WCy for each color. For example, the color evaluation values WCx, WCx, Using WCy as a reference, the color evaluation value is determined in advance as a coefficient that is approximately the same value at each color temperature. Of course, the calculated coefficient can be obtained for intermediate color temperatures other than the acquired color temperature by performing linear interpolation or the like for each color temperature. Such a color conversion process has the same effect as a color reproduction matrix that absorbs a color difference for each color temperature, and a moving subject is obtained by absorbing the color difference for each color temperature by these processes. Can be accurately detected.

  FIG. 10 shows a detailed flowchart of the image composition processing 114 of FIG. 2 performed by the image composition processing unit 325 of FIG. In step S201 in FIG. 10, the image composition processing unit 325 has developed the strobe light emission image 100 after the WB correction processing is performed using the first WB correction value in the first development processing 1131 in FIG. A first YUV image is acquired. The first YUV image acquired here is represented by the color evaluation values Y1 [i], u1 [i], and v1 [i] described above. After step S201, the image composition processing unit 325 advances the process to step S202.

  In step S202, the image composition processing unit 325 performs the second development processing 1132 in FIG. 2 to develop the second strobe image 100 after the WB correction processing is performed using the second WB correction value. A YUV image is acquired. The second YUV image acquired here is represented by the color evaluation values Y2 [i], u2 [i], and v2 [i] described above. After step S202, the image composition processing unit 325 advances the process to step S203.

  In step S203, the image composition processing unit 325 divides the first YUV image and the second YUV image into blocks as described above. After step S203, the image composition processing unit 325 advances the processing to step S204. In step S204, the image composition processing unit 325 acquires the luminance component composition ratio α [i] calculated by the above-described equation (9) as the image composition ratio α [i]. After step S204, the image composition processing unit 325 advances the processing to step S205.

  In step S205, the image composition processing unit 325 obtains the first YUV image obtained in step S201 and the first YUV image obtained in step S202 by the calculation of Expression (15) using the image composition ratio α [i] obtained in step S204. The second YUV image is combined for each block. A composite YUV image obtained by combining the first YUV image and the second YUV image is represented by color evaluation values Y3 [i], u3 [i], and v3 [i].

Y3 [i] = Y1 [i] * α [i] + Y2 [i] * (1−α [i])
u3 [i] = u1 [i] * α [i] + u2 [i] * (1−α [i]) (15)
v3 [i] = v1 [i] * α [i] + v2 [i] * (1−α [i])

  Further, in order to alleviate the color shift at the boundary portion of the block, in step S205, the image composition processing unit 325 performs pixel interpolation processing, thereby performing image composition for each pixel from the image composition ratio α [i] for each block. The ratio rate α ′ [j] may be calculated. Note that [j] of α ′ [j] represents the j-th pixel. For example, bilinear interpolation processing can be used as the pixel interpolation processing, and the image composition processing unit 325 calculates the image composition ratio α ′ [j] for each pixel from the image composition ratio α [i] for each block by the bilinear interpolation processing. To do. In this example, in step S205, the image composition processing unit 325 uses the image composition ratio α ′ [j] for each pixel to calculate the first YUV image and the second YUV image by the calculation of Expression (16). Are combined to generate a composite YUV image. Note that the first YUV image when image synthesis is performed for each pixel is represented by color evaluation values Y1 [j], u1 [j], and v1 [j] for each pixel, and the second YUV image is It is represented by color evaluation values Y2 [j], u2 [j], and v2 [j] for each pixel. The composite YUV image is represented by color evaluation values Y3 [j], u3 [j], and v3 [j] for each pixel.

Y3 [j] = Y1 [j] * α ′ [j] + Y2 [j] * (1−α ′ [j])
u3 [j] = u1 [j] * α ′ [j] + u2 [j] * (1−α ′ [j]) (16)
v3 [j] = v1 [j] * α ′ [j] + v2 [j] * (1−α ′ [j])

  11 shows the ratio between the strobe amount of the strobe light emitting unit 316 and the ambient light (external light) based on the distance map, the strobe light distribution information, the lens peripheral light amount, and the light emission amount information 110 in the ratio estimation process 130 of FIG. The processing flow which calculates is shown. In the present embodiment, the ratio estimation process 130 by the CPU 314 corresponds to a ratio calculation unit. FIG. 11 is a diagram showing in detail each process included in the light ratio estimation process 130 of FIG. Description of the strobe light emission image 100 and the moving body region composition ratio correction processing 140 in FIG. 11 is omitted.

  The strobe light distribution lens peripheral light amount 137 of FIGS. 11 and 2 includes strobe light distribution information and lens peripheral light amount data. The strobe light distribution information is information representing strobe light distribution characteristics calculated by measuring in advance the light distribution during strobe light emission. Specifically, the amount of light around the lens of a photographed image obtained by performing strobe light emission from an appropriate distance with respect to a gray card surface having a reflectance of 18% in the dark in advance and taking the zoom lens of the lens barrel 315 as the wide-angle end. This is light distribution information calculated by correcting the drop amount.

  When calculating the strobe light distribution information, the amount of deviation between the strobe and the optical axis of the lens can be absorbed by varying the shooting distance and acquiring the light distribution information for each shooting distance. is there. Another method is to calculate the light intensity distribution to the periphery from the strobe mounting position and the strobe reflector or Fresnel lens shape, and then calculate the strobe light distribution from the focal length of the lens. Is possible. Then, by obtaining light distribution information according to the focal length of the lens, it is possible to calculate the strobe light distribution when the focal length changes. For example, if the focal length of the zoom lens is double the focal length from the wide-angle end, a horizontal and vertical range of 50% from the center in the strobe light distribution at the wide-angle end measured in advance may be used as the light distribution region. Good. As for the lens peripheral light amount drop amount, the peripheral light amount drop amount may be measured for each zoom position or aperture value, or based on the optical design value based on the lens design information and the peripheral incident light amount characteristic of the image sensor 301. It may be calculated.

  The distance map generation process 131 in FIG. 11 is a process performed by the CPU 314 in FIG. 1, for example. In the distance map generation process 131, the CPU 314 multiplies the phase difference generated by the correlation calculation of the output signals of the focus detection pixels of the image sensor 301 by a conversion coefficient that converts the phase difference into a defocus amount. Calculate the defocus amount. Then, the CPU 314 creates a focus shift amount map on the imaging surface of the image sensor 301 from each focus shift amount corresponding to each of all focus detection pixels of the image sensor 301.

  Here, the focus shift amount map is distribution data in which the position of each focus detection pixel is associated with each focus shift amount on the imaging surface of the image sensor 301. In the distance map generation process 131, the CPU 314 considers lens information in the lens optical system of the lens barrel unit 315, and sets each focus shift amount of the focus shift amount map as distance information to the subject (hereinafter referred to as subject distance). .). This makes it possible to obtain distribution data in which the position of each focus detection pixel on the shooting screen of the image sensor 301 is associated with the subject distance. In the distance map generation process 131, the CPU 314 extracts a subject area based on the subject distance distribution data. For example, regions having substantially the same subject distance in the subject distance distribution are connected and grouped to extract the contour of the subject included in the shooting screen. As a result, a subject distance map in which each subject area is associated with the subject distance is obtained. This subject distance map is a distance map generated by the distance map generation processing 131.

  Note that the function of the focus detection pixel when generating the distance map may be included in all the pixels in the effective imaging region of the imaging element 301, or may be included in every several pixels. . When the pixels for every several pixels are the focus detection pixels, the focus detection pixels are arranged so as to be uniformly distributed on the imaging surface of the image sensor 301. Information on the distance map generated by the distance map generation process 131 is sent to the strobe component estimation process 132.

  The strobe component estimation process 132 in FIG. 11 is a process performed by the CPU 314 in FIG. 1, for example. In the strobe component estimation process 132, the CPU 314 calculates a strobe component based on the strobe light distribution lens peripheral light amount 137, the light emission amount information 110, and the distance map.

  The light emission amount information 110 is information on a guide number that is a strobe light emission amount in the strobe light emitting unit 316. Specifically, the correspondence between the strobe flash time, the voltage value of the strobe capacitor terminal or the current value actually consumed at the time of light emission, and the guide number is measured in advance, and the amount of light emitted as information indicating the correspondence Information 110 is prepared. The CPU 314 can obtain a guide number for strobe light emission based on the correspondence relationship in the light emission amount information 110. As another method, a light amount sensor may be provided in the strobe light emitting unit 316, and the CPU 314 may determine a guide number according to a light amount value detected by the light amount sensor during strobe light emission.

  Here, in general, the guide number is defined as follows. For example, in the case of ISO 100 sensitivity, the guide number is defined as in Expression (17).

Guide number = Subject distance × Aperture value (17)
The aperture value is an F value.

  Further, the square value of the subject distance and the strobe light amount are inversely proportional, and the ISO sensitivity and the strobe light amount are proportional. For example, when the subject distance is doubled, the strobe light amount on the subject is ¼, and when the ISO sensitivity is doubled, the strobe light amount on the imaging surface is doubled. In the present embodiment, for example, at the standard ISO sensitivity and the standard subject distance, the amount of light on the imaging surface when the strobe light is applied to the gray card surface with a reflectance of 18% in the dark is calculated in advance and held in a ROM (not shown) or the like. ing. Accordingly, the CPU 314 can calculate the strobe light amount component for each subject area based on information such as the subject distance, guide number, aperture value, ISO sensitivity, and the like. The guide number is defined as a value representing an appropriate amount of light at the center of the screen. For this reason, the CPU 314 can correct the light amount of the peripheral portion of the strobe irradiation amount in accordance with the peripheral light amount drop amount on the imaging surface for the peripheral portion of the screen.

  In the strobe component estimation process 132, the CPU 314 indicates the amount of light incident on the imaging surface in accordance with the strobe light distribution amount of the strobe light distribution lens peripheral light amount 137, the lens peripheral light amount, the strobe light emission amount (guide number), and the subject distance. Estimate the strobe component. Expression (18) shows an arithmetic expression for the estimated amount of the strobe component at the coordinates (x, y) of the imaging surface of the image sensor 301.

f (x, y) = GN × St (x, y) × LensSHD (x, y) / (Dist (x, y) × FNo) × (R (x, y) /0.18) × AETarget
... Formula (18)
f (x, y): the amount of light incident on the image sensor 301 when stroboscopic irradiation is performed
GN: Guide number
St (x, y): Light distribution information, and surrounding light distribution when the light distribution at the center of the image is “1”
LensSHD (x, y): Peripheral light quantity ratio of the lens when the light distribution at the center of the image is “1”
Dist (x, y): Subject distance for each subject area
R (x, y): Subject reflectance
AETarget: Output level of the image sensor 301 obtained at the time of appropriate exposure shooting

  Here, for example, when a gray card with a reflectance of 18% is photographed, Expression (18) can be simplified as Expression (19) at the center of the photographed image. Further, when the strobe light is photographed with a guide number (GN) that provides proper exposure, the output value of the image sensor 301 is obtained from the equation (17) as the output level value obtained when photographing is performed with proper exposure. It turns out that it is obtained.

  f (x, y) = G.N. × / (Dist (x, y) × FNo) × AETarget Expression (19)

  In addition, among the external light components (environmental light components) included in the strobe non-light emitting image 102 and the strobe light emitting image 100, the light amount component by the light source and the subject reflectance cannot be separated, so the subject reflectance cannot be measured. For this reason, the reflectance of the subject may be simply calculated as a reflectance of 18% in a simple manner, or the assumption of the reflectance may be changed according to the color and luminance value of the strobe light emission image 100. For example, in the case of a yellow subject, it is possible to improve the accuracy by estimating the reflectance according to the color and the luminance value of the strobe light emission image 100, such as setting the reflectance to 70%. In addition, by taking two shots with different strobe light guide numbers and obtaining the difference between the two shot images by the two shots, the external light component can be removed, and the difference between these images is the strobe. It becomes the amount of light fluctuation. In addition, as described above, since it is possible to estimate the amount of strobe light applied to the subject from the difference between the subject distance and the guide number, the ratio of the strobe amount to the actual image signal value is calculated to calculate the ratio of the subject. It is possible to measure the reflectance.

Returning to FIG.
The external light / strobe light ratio generation process 138 for each region (hereinafter referred to as the light ratio generation process 138) uses the value of the strobe component calculated by the strobe component estimation process 132 from the value of the image signal of the strobe emission image 100. By subtracting, the amount of external light components is calculated. Then, the light ratio generation process 138 obtains the ratio of the strobe light to the outside light (light quantity ratio) for each subject area by the same process as the light ratio generation process 133 described above. Information on the ratio calculated by the light ratio generation process 138 is sent to the combination ratio correction process 140.

  As described above, the composition ratio correction process 140 uses the composition ratio calculated in the composition ratio determination process 105 for the moving object area extracted in the moving object area extraction process 123, as the subject by the strobe outside light ratio estimation process 130. Modify according to the ratio of each area.

  As described above, in the present embodiment, as described above, the first YUV image and the second YUV image are combined by the image combining process 114 in accordance with the image combining ratio α [i]. Here, generally, the additiveness of light is established for the color when light from a plurality of light sources is mixed. In other words, if the light intensity ratio between the strobe light quantity and the external light component is known, an image after WB correction processing corresponding to the light of each light source is synthesized according to the light quantity ratio between the strobe light quantity and the external light quantity. An image subjected to appropriate WB control is obtained. In the present embodiment, with respect to the subject area where the subject or the like has not moved in the strobe light emission image 100 that is the main exposure image, an image that is WB corrected with the first WB correction value and the second WB correction. An image subjected to WB correction with values is synthesized. As described above, in the subject area where the subject or the like is not moving, the light amount ratio can be accurately calculated because the external light amount and the strobe light amount can be accurately calculated.

  On the other hand, when the subject or the like is a moving subject, the light amount ratio cannot be accurately calculated by comparing the strobe non-emission image 102 and the strobe emission image 100 for the moving body region. For this reason, in this embodiment, the ratio of strobe light to outside light is estimated based on the distance map calculated from the strobe light emission image 100 that is the main exposure image and the shooting information at that time. In the present embodiment, with respect to the moving object region, the ratio is used as the image composition ratio to perform image composition.

  According to the present embodiment, the moving body region is determined using the strobe light emission image 100 and the strobe non-light emission image 102, the distance information of the subject is acquired, and the composition ratio based on the distance information is obtained for the moving body movement region. Is determined. For this reason, according to the present embodiment, even if the subject moves in a scene that emits strobe light, optimal white balance control can be performed for each subject region, and both the main subject and the background have an appropriate color tone. Can be acquired.

  In the present embodiment, an example is given in which a distance map is obtained by detecting a phase difference in each focus detection pixel. However, the distance measurement by the phase difference detection method can be calculated accurately because the depth of field is sufficiently shallow with respect to the distance if it is a short-distance region. Because the depth is deep, the distance cannot be calculated accurately. However, in the short-distance region, the difference in the amount of strobe light irradiation is large depending on the subject distance, so that the color unevenness due to the difference in the amount of strobe light irradiation increases. For this reason, it is necessary to accurately control the WB correction so as not to cause color unevenness. However, since the fluctuation of the strobe irradiation amount with respect to a long distance is reduced, it is not necessary to accurately calculate the distance. .

<Second Embodiment>
Hereinafter, an imaging apparatus according to the second embodiment will be described. The second embodiment is an example in the case where a moving body region is extracted using information of a distance map. FIG. 12 is a diagram showing a part different from the processing flow of FIG. 2 in the first embodiment described above in the imaging apparatus of the second embodiment. In the case of the second embodiment, instead of the first and second color information acquisition processes 1201 and 1202 and the strobe light emission image color correction process 121 in the process flow of FIG. Processing 1311, 1312, 1341, 1342, 135, 136 is used. In the second embodiment, the processes other than extracting the moving body region based on the information of the distance map are the same as those in the process flow of FIG. 2 described above, and thus the description thereof is omitted. .

  As shown in FIG. 12, in the case of the second embodiment, the strobe light emission image 100 is sent to the first distance map generation process 1311. The first distance map generation process 1311 is a process performed by, for example, the CPU 314 in FIG. As the first distance map generation process 1311, the CPU 314 generates a distance map from the strobe light emission image 100 in the same manner as described in the distance map generation process 131 of FIG. 11 described above. Hereinafter, the distance map generated by the first distance map generation process 1311 is referred to as a “first distance map”. The information of the first distance map is sent to the first distance map reliability generation process 1341 and the distance map difference extraction process 136.

  The strobe non-light-emitting image 102 is sent to the second distance map generation process 1312. The second distance map generation process 1322 is a process performed by the CPU 314 in FIG. As the second distance map generation process 1311, the CPU 314 generates a distance map from the strobe non-light emitting image 102 as described above. Hereinafter, the distance map generated by the second distance map generation process 1312 is referred to as a “second distance map”. The information of the second distance map is sent to the second distance map reliability generation process 1342 and the distance map high reliability area extraction process 135.

  The first distance map reliability generation process 1341 (hereinafter referred to as the first reliability generation process 1341) is, for example, a process performed by the CPU 314 in FIG. As described above, the distance map is generated based on the phase difference obtained by the correlation calculation of the output signals of all the focus detection pixels. As the first reliability generation process 1341, the CPU 314 determines the reliability of the phase difference that is the result of the correlation calculation for the first distance map. The reliability in this case corresponds to the degree of coincidence of the signals of the two images by the focus detection pixels, and it can be said that the reliability of the focus detection result is high when the degree of coincidence of the signals of the two images is high. In the first reliability generation process 1341, the CPU 314 determines the reliability of the phase difference detection result of the correlation calculation based on whether or not the degree of coincidence of the signals of the two images by the focus detection pixels exceeds a certain threshold value. I do. As described above, the CPU 314 determines the reliability of each phase difference when the first distance map is generated in the first reliability generation process 1341. Information on the determination result of the reliability of the first distance map by the first reliability generation process 1341 is sent to the distance map high reliability area extraction process 135.

  The second distance map reliability generation process 1342 (hereinafter referred to as the second reliability generation process 1342) is, for example, a process performed by the CPU 314 in FIG. As the second reliability generation process 1342, the CPU 314 determines the reliability of each phase difference for the second distance map in the same manner as the first reliability generation process 1341. Information on the determination result of the reliability of the second distance map by the second reliability generation process 1342 is sent to the distance map high reliability area extraction process 135.

  The distance map high reliability area extraction process 135 (hereinafter referred to as reliability area extraction process 135) is, for example, a process performed by the CPU 314 in FIG. As the reliability region extraction processing 135, the CPU 314 compares the reliability of each phase difference in the first distance map with the reliability of each phase difference in the second distance map, and selects a reliability with a small difference between the two. Then, in the reliability region extraction processing 135, the CPU 314 determines, in the first and second distance maps, the reliability of the regions where the reliability of the selected phase differences is higher than a predetermined value. Are extracted as high regions. Hereinafter, an area extracted as having high reliability in both the first and second distance maps is referred to as “high reliability area”. The information on the high reliability area extracted by the reliability area extraction process 135 is sent to the mobile object area extraction process 123.

  The distance map difference extraction process 136 (hereinafter referred to as difference extraction process 136) is a process performed by the CPU 314 of FIG. As the difference extraction process 136, the CPU 314 obtains a difference between the distance information of the first distance map and the second distance map, and extracts an area where distance information having a large difference is gathered. Hereinafter, an area where distance information having a large difference is collected is referred to as a “large difference distance area”. The information on the large difference distance area extracted by the difference extraction process 136 is sent to the moving object area extraction process 123.

  In the case of the second embodiment, the CPU 314 extracts a region that is a large difference distance region and a high reliability region as a moving body region extraction process 123 as a moving body region of the strobe light emission image 100. The information on the moving body area extracted by the second moving body area extraction process 123 is sent to the composition ratio correction process 140. Since the combination ratio correction process 140 is the same as that in the first embodiment, the description thereof is omitted.

  As described above, according to the second embodiment, the moving object region is extracted based on the distance map, and the distance information can be extracted for each pixel by using the distance map. It has an advantage that a moving body region can be extracted with high resolution. Further, according to the second embodiment, it is not necessary to evaluate the color for each image area as in the case of the first embodiment, so that the amount of calculation can be reduced.

  However, in the case of the second embodiment, for example, in the case of a dark subject or a subject with low contrast, the distance based on the above-described phase difference may not be measured, and it is difficult to extract the moving body region using the distance map. There is a risk of becoming. In such a case, switching is performed so that the moving body region is extracted by the color evaluation of the first embodiment, and the distance is measured when the area of the region where the distance can be measured is equal to or larger than a certain area. Different usage such as performing mobile object region extraction using a map may be performed. Further, when the mobile body region extraction based on the distance map and the mobile body region extraction based on the color evaluation are used separately, which one to use may be determined by a method such as linear interpolation.

<Third Embodiment>
Hereinafter, an imaging apparatus according to the third embodiment will be described. The third embodiment is an example in which the image composition ratio of the moving body region is determined based on the correlation calculation result between the strobe light / external light quantity ratio and the subject distance. FIG. 13 is a diagram showing an extracted part of the imaging apparatus according to the third embodiment, which is different from the processing flow of FIG. 2 in the first embodiment described above. In the case of the third embodiment, in addition to the process flow of FIG. 2 described above, processes 139, 134, 150, 160, 161 described later are provided. In the third embodiment, the other processes other than the added processes are the same as those in the process flow of FIG. 2 described above, and a description thereof will be omitted. However, in the case of the third embodiment, the strobe outside light ratio estimation processing 130 of FIG. 2 is not performed, and the moving object region image is based on the correlation calculation result between the strobe light and outside light amount ratio and the subject distance. A composite ratio is required.

  As shown in FIG. 13, in the case of the third embodiment, the strobe light emission image 100 is sent to a distance map generation process 139 and a subject area determination process 160. The distance map generation process 139 is the same process as the distance map generation process 131 of FIG. 11 and the first distance map generation process 1311 of FIG. The information of the distance map generated by the distance map generation process 139 is sent to the distance map reliability generation process 134, the subject area distance information acquisition process 161, and the distance / light quantity ratio correlation calculation process 150.

  The distance map reliability generation process 134 is the same process as the first reliability generation process 1341 described with reference to FIG. The distance map reliability information generated by the distance map reliability generation process 134 is sent to the composition ratio correction process 140.

  The subject area determination process 160 is a process performed by the CPU 314 of FIG. The CPU 314 determines the subject area included in the flash emission image 100 that is the main exposure image as the subject area determination processing 160. The CPU 314 performs image processing based on face image area detection processing, processing for detecting an image area having a similar color based on saturation, hue, luminance value, and the like as subject area determination processing 160, autofocus information, and distance information based on a distance map. The subject region is determined by processing for detecting the region. Note that these face image detection processing, image region detection processing using similar colors, and image region detection processing based on distance information are generally known techniques, and thus detailed description thereof is omitted here. The subject area determination process 160 may be a process in which these processes are combined. Information on the subject area determined by the subject area determination processing 160 is sent to the subject region distance information acquisition processing 161.

  The subject area distance information acquisition process 161 (hereinafter referred to as a distance information acquisition process 161) is, for example, a process performed by the CPU 314 in FIG. As the distance information acquisition process 161, the CPU 314 calculates a distribution of distance information for each subject area, and calculates a representative value of the distance for each subject area. As the representative value of the distance for each subject area, for example, a value obtained by using a statistical method such as an average value or a median value is used. The representative value of the distance information for each subject area obtained by the distance information acquisition process 161 is sent to the composition ratio correction process 140.

  The distance / light quantity ratio correlation calculation processing 150 (hereinafter referred to as correlation calculation processing 150) is, for example, processing performed by the CPU 314 in FIG. As the correlation calculation processing 150, the CPU 314 correlates between the subject distance and the light amount ratio for each subject region based on the distance map from the distance map generation processing 139 and the light amount ratio from the light ratio generation processing 133 described above. Perform the operation. As a correlation calculation method at this time, a general method such as a least square method is used. The correlation coefficient obtained by the correlation calculation by the correlation calculation process 150 is sent to the synthesis ratio correction process 140.

  In the combination ratio correction process 140 in the case of the third embodiment, the correlation coefficient from the correlation calculation process 150, the reliability from the distance map reliability generation process 134, and the distance information representative from the distance information acquisition process 161 are represented. Based on the value, the composition ratio is corrected for each moving body region. Specifically, the CPU 314 performs the composition ratio correction processing 140 according to the third embodiment for a moving body region whose distance map reliability is higher than a threshold based on the correlation coefficient by the correlation calculation processing 150. To decide. On the other hand, for a mobile object region whose reliability of the distance map is lower than the threshold, the composition ratio correction process 140 obtains a correlation coefficient by a correlation operation between the representative value of the distance of the mobile object region and the light amount ratio, and the correlation coefficient Based on the above, the composition ratio is calculated. Then, the information of the composition ratio after the correction processing by the composition ratio correction processing 140 is sent to the resize processing 106 in FIG.

  In the third embodiment, when calculating the correlation coefficient of the strobe light irradiation amount, the strobe light irradiation distribution is used, and the strobe light ratio is corrected to be dependent on the strobe light amount. Thus, it is possible to improve the accuracy of the correlation calculation between the subject distance and the light amount ratio. In other words, since the light amount ratio between the strobe light amount and the external light amount depends on the strobe light amount, the strobe light ratio is corrected according to the strobe light distribution when calculating the correlation coefficient. More specifically, the strobe light distribution information is calculated from the strobe light distribution lens peripheral light amount 137 of FIG. 1 and the focal length at the time of photographing with respect to the strobe light amount when calculating the correlation coefficient. Multiply the strobe light amount by the ratio to the irradiation amount at the center of the image. In this way, the correlation between the strobe light amount and the distance can be obtained with respect to data having no strobe irradiation unevenness, and the correlation coefficient between the distance, the outside light, and the strobe light amount can be obtained with higher accuracy. On the other hand, when calculating the strobe light amount according to the subject distance from the correlation coefficient calculated as described above, the strobe light amount is multiplied by the irradiation light amount ratio according to the strobe light amount distribution in the imaging surface. Thus, the light quantity ratio between the strobe light and the external light quantity can be calculated with higher accuracy.

  As described above, in the third embodiment, the correlation between the distance map and the light amount ratio is calculated from the strobe light emission image 100, and correction is performed using the distance map of the strobe light emission image 100. As described above, since the correlation coefficient between the distance map of the same strobe light emission image 100 and the strobe light emission amount is calculated, an error due to the movement of the subject does not occur. Therefore, the WB correction is accurately performed for the moving body region. Can be done. In the third embodiment, the relationship between the distance and the subject area can be obtained from the strobe light emission image 100 by determining the subject area of the strobe light emission image 100 and generating the distance map. For example, in the subject area in the strobe light emission image 100, the subject distance is often uniform. Therefore, in the present embodiment, even when the distance map is not accurately obtained, the calculation accuracy of the light amount ratio is improved. Can be made.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Each of the above-described embodiments is merely a specific example for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  100 Strobe light emission image (main exposure image), 101 Resize processing, 102 Strobe non-light emission image (EVF image), 103 Exposure amount calculation processing, 104 Strobe light emission amount calculation processing for each area, 105 Image composition ratio determination processing, 106 Resizing processing, 107 2nd WB correction value calculation processing, 1091 1st WB processing, 1092 2nd WB processing, 110 light emission amount information, 111 1st WB correction value calculation processing, 1131 1st development processing, 1132 2nd Development processing, 114 image composition processing, 1201 first color information acquisition processing, 1202 second color information acquisition processing, 121 strobe emission image color correction processing, 123 moving body region extraction processing, 130 strobe external light ratio estimation Processing, 131 distance map generation processing, 132 strobe component estimation processing, 138 outside light / stoke for each region Robo light ratio generation process, 1311 first distance map generation process, 1312 second distance map generation process, 1341 first distance map reliability generation process, 1342 second distance map reliability generation process, 136 distance map difference Extraction processing, 135 Distance map high reliability region extraction processing, 137 Strobe light distribution lens peripheral light amount information, 140 Moving body region composition ratio correction processing, 150 Distance light amount ratio correlation calculation processing, 301 Image sensor, 302 Memory, 303 WB control unit 304 color conversion MTR circuit, 305 LPF circuit, 306 CSUP circuit, 307 RGB conversion circuit, 308 gamma correction circuit, 309 color luminance conversion circuit, 310 JPEG compression circuit, 311 Y generation circuit, 312 edge enhancement circuit, 313 lens barrel control Circuit, 314 CPU, 315 lens barrel, 316 Flash output unit, 320 peripheral light amount correction unit, 325 image synthesis processing unit

Claims (9)

The first white balance correction value corresponding to the strobe light is used to correct the white balance of the first image taken when the strobe light is emitted under the ambient light, and the second white balance correction value corresponding to the ambient light. Correcting means for correcting the white balance of the second image taken under ambient light without strobe light emission,
An acquisition means for acquiring a subject distance representing a distance to each subject when the first image is captured;
Extraction means for extracting from the first image a subject area of a moving subject that has moved during the shooting of the first image and the shooting of the second image;
Based on the first image and the second image, a light quantity ratio between strobe light and ambient light is obtained for each subject area of the first image, and based on the light quantity ratio for each subject area, the first light quantity ratio is obtained. Determining means for determining a synthesis ratio for synthesizing the first image and the second image for each subject area;
Based on the subject distance and the light distribution characteristic of the strobe light, the amount of strobe light irradiation for each subject region when the first image is taken is estimated, and the strobe light irradiation for each subject region is estimated. A calculating means for calculating a ratio between the amount and the ambient light;
The white balance correction is performed using the calculated ratio as the composition ratio for the subject area of the moving subject, and using the composition ratio determined by the determining means for the subject area of the subject that is not the moving subject. An image processing apparatus, comprising: a combining unit that combines the first image and the second image after being subjected to each subject area. The first white balance correction value corresponding to the strobe light is used to correct the white balance of the first image taken when the strobe light is emitted under the ambient light, and the second white balance correction value corresponding to the ambient light. Correcting means for correcting the white balance of the second image taken under ambient light without strobe light emission,
An acquisition means for acquiring a subject distance representing a distance to each subject when the first image is captured;
Extraction means for extracting from the first image a subject area of a moving subject that has moved during the shooting of the first image and the shooting of the second image;
Based on the first image and the second image, a light quantity ratio between strobe light and ambient light is obtained for each subject area of the first image, and based on the light quantity ratio for each subject area, the first light quantity ratio is obtained. Determining means for determining a synthesis ratio for synthesizing the first image and the second image for each subject area;
Correlation calculating means for calculating a correlation coefficient by performing a correlation calculation between the subject distance and the light amount ratio;
For the subject area of the moving subject, the composite ratio corrected based on the calculated correlation coefficient is used, and for the subject area of the subject that is not the moving subject, the composite ratio determined by the determining means is used. An image processing apparatus comprising: a combining unit that combines the first image and the second image after the white balance correction for each subject area. The acquisition means obtains the reliability of the subject distance for the subject area of the moving subject,
The synthesis means includes
For the subject area of the moving subject whose reliability value of the subject distance is equal to or greater than a predetermined threshold, the composite ratio corrected based on the correlation coefficient by the correlation calculating means is used.
For a subject area of a moving subject whose reliability value of the subject distance is lower than a predetermined threshold, a representative value of the subject distance is obtained, and the phase is calculated by a correlation calculation between the representative value of the subject distance and the light amount ratio. Using the composite ratio corrected based on the correlation coefficient by the correlation calculation between the representative value of the subject distance and the light amount ratio, by obtaining the relationship number,
The image processing apparatus according to claim 2, wherein the first image and the second image after the white balance correction are combined for each subject area.   The extraction means obtains first color information for each subject area of the first image and second color information for each subject area of the second image, and obtains the first color information and the first color information. The image processing apparatus according to claim 1, wherein a subject area having a difference between the two color information is extracted as a subject area of the moving subject.   The extraction means uses a subject area having a difference between a subject distance when the first image is taken and a subject distance when the second image is taken as a subject area of the moving subject. The image processing apparatus according to claim 1, wherein the image processing apparatus is extracted. The determining means determines a composition ratio for each pixel by an interpolation process for the composition ratio,
The image processing apparatus according to claim 1, wherein the synthesizing unit synthesizes the first image and the second image based on a synthesis ratio for each pixel. The correction means corrects the white balance of the first image taken at the time of flash emission under the ambient light using the first white balance correction value corresponding to the flash light, and the second corresponding to the environmental light. Using the white balance correction value to correct the white balance of the second image taken under ambient light without flash emission;
An obtaining unit obtaining a subject distance representing a distance to each subject when the first image is captured;
An extracting unit extracting, from the first image, a subject area of a moving subject that has moved during the shooting of the first image and the shooting of the second image;
Based on the first image and the second image, the determining means obtains a light quantity ratio of strobe light and ambient light for each subject area of the first image, and based on the light quantity ratio for each subject area. Determining a combining ratio for combining the first image and the second image for each subject area;
The calculation means estimates the amount of strobe light irradiation for each subject area when the first image is taken based on the subject distance and the light distribution characteristic of the strobe light, and Calculating a ratio between the amount of strobe light irradiation and ambient light;
Combining means uses the calculated ratio as the combining ratio for the subject area of the moving subject, and uses the combining ratio determined by the determining means for the subject area of the subject that is not the moving subject, An image processing method comprising: synthesizing the first image after the white balance correction and the second image for each subject area. The correction means corrects the white balance of the first image taken at the time of flash emission under the ambient light using the first white balance correction value corresponding to the flash light, and the second corresponding to the environmental light. Using the white balance correction value to correct the white balance of the second image taken under ambient light without flash emission;
An obtaining unit obtaining a subject distance representing a distance to each subject when the first image is captured;
An extracting unit extracting, from the first image, a subject area of a moving subject that has moved during the shooting of the first image and the shooting of the second image;
Based on the first image and the second image, the determining means obtains a light quantity ratio of strobe light and ambient light for each subject area of the first image, and based on the light quantity ratio for each subject area. Determining a combining ratio for combining the first image and the second image for each subject area;
A step of calculating a correlation coefficient by performing a correlation calculation between the subject distance and the light amount ratio;
The composition means uses the composition ratio corrected based on the calculated correlation coefficient for the subject area of the moving subject, and the composition determined by the determination means for the subject area of the subject that is not the moving subject. An image processing method comprising: combining the first image after the white balance correction and the second image for each subject area using a ratio.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-6.

Images