JP4905513B2 - Image processing method, image processing apparatus, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a high-quality image through virtual adjustment processing of flash light intensity in image correction, based on images photographed with or without emitting flash light. <P>SOLUTION: A flash component image is calculated, on the basis of a first image photographed without emitting a flash light and a second image photographed with emitting the flash light, and an intensity adjustment flash component image generated by carrying out an intensity adjustment of the flash component image is applied to generate a final adjustment image. The present configuration makes it possible to generate a high-quality image, with which saturation pixels, namely, white-out is reduced in an image of non-linear transformation, such as gamma correction. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to an image processing method, an image processing apparatus, and a computer program. More specifically, a pixel value correction process using an image shot with flash light and an image shot without flash is executed, the optimal flash intensity is estimated, and correction based on the estimation result is performed. The present invention relates to an image processing method, an image processing apparatus, and a computer program that can generate a high-quality image.

  A flash (electric flash, strobe) is used as an auxiliary light source during camera photography. In recent years, DSC (Digital Still Camera) is rapidly spreading, but flash photography is often performed also in DSC. By using a flash, fill-in light (a technique that weakens the shadow when the shadow of a person's face is too hard), backlight correction (a technique that prevents the face from being blackened when shooting a person with the sun behind), A variety of photos such as catchlight (a technique that shoots the eyes beautifully by putting a "spot of light" that shines in the eyes of the eyes), or daytime synchro (daylight sync, a technique used as an auxiliary beam in the daytime and evening) Shooting can be performed. On the other hand, when flash photography is performed, color balance may be lost or whiteout may occur. An object of the present invention is to provide an appropriate means capable of correcting such a bad phenomenon caused by flash photography.

  Generally, in a digital camera, white balance (WB) adjustment is performed so that a white subject is photographed white. For example, when shooting in a light component environment where the color temperature of the light irradiated to the subject, such as natural light, illumination light, or flash (strobe) is high and blue (B) is strong, the sensitivity to blue light is suppressed. When shooting in a light component environment where the sensitivity to light of red (R) is relatively high and the color temperature of light irradiated to the subject is low and the light of red (R) is strong, White balance adjustment such as suppressing sensitivity to light and relatively increasing sensitivity to blue (B) light is performed.

  In the white balance adjustment, an adjustment process in which an adjustment parameter corresponding to a light source used at the time of shooting is set is usually executed. For example, when flash photography is performed, white balance adjustment is performed according to parameters according to the light component of the flash light to be applied.

  However, if flash photography is performed in the presence of external light other than flash light, the subject is irradiated with two types of light, flash light and external light, and the reflected light reaches the image sensor of the camera, and photography is performed. It will be. When such shooting is performed, if white balance adjustment is performed according to the flash light, the subject area that is exposed to the flash light is adjusted to a natural color, but the flash light does not reach and only the external light is reflected. If the white balance adjustment according to the parameter setting according to the light component of the flash light is performed on the area captured as light, for example, the background image area, the appropriate white balance adjustment is not performed and an unnatural color is displayed. It will be output as the area you have.

  On the other hand, if the white balance adjustment according to the background, that is, the image was shot only with outside light, the white balance adjustment of the entire shot image will be performed, and the part that is exposed to a lot of flash light will be adjusted to an unnatural color. become.

  Several configurations for dealing with such problems have been proposed. For example, Patent Document 1 acquires an image shot without flash and a shot image with flash emitted, divides the two shot images into blocks, and compares brightness values for each block. FIG. 3 shows a configuration in which different white balance adjustment is performed for each block on an image captured by emitting a flash based on a comparison result of luminance values.

  White balance adjustment is performed by selecting either white balance adjustment according to the flash light, white balance adjustment between the flash light and external light, or white balance adjustment according to the external light for each block. . However, in such a configuration, it is necessary to perform processing in units of blocks, and there are problems such as block distortion, and problems that cannot be performed correctly when the subject moves.

  Patent Document 2 discloses the following processing configuration. That is, first, the aperture is opened and the exposure time is shortened, and the flash is emitted to take a picture. Thereafter, the picture is taken without the flash being emitted under the originally intended exposure conditions. Here, the former is a first image and the latter is a second image. Further, in the first image, a pixel having a predetermined level or higher is registered as a main subject area, and the other is registered as a background area. After that, the first image is adjusted to the white balance according to the flash, the second image is adjusted to the white balance according to the outside light, and the final recorded image is obtained by combining the main subject area of the first image and the background area of the second image. The configuration is as follows.

  However, with this configuration, white balance adjustment cannot be performed correctly for a subject that has been exposed to both external light and flash light.

  Patent Document 3 discloses a configuration in which image blur detection means is added to the configuration of Patent Document 2 described above. When it is determined that there is blurring, the first image is used as a recorded image as it is, and the combination processing of the first image and the second image is not executed. Therefore, when blur is detected, unnaturalness caused by the difference in color temperature between the flash light and the external light cannot be resolved.

  Further, Patent Document 4 divides the luminance value of the corresponding pixel of the image shot with the flash emitted and the image shot without the flash emission to determine the contribution rate of the flash light, and based on this contribution rate, A configuration is disclosed in which white balance adjustment is performed on an image shot with light emission.

  However, in this configuration, the white balance parameter for flash light and external light is simply interpolated based on the contribution ratio of the flash light and the image captured by mixing the reflected light from the flash light and external light, and finally A simple image. However, when considering the physical light reflection model, the component due to flash light and the component due to external light should be processed independently. An optimum result image cannot be generated only by processing.

JP-A-8-51632 JP 2000-308068 A JP 2000-307940 A JP-A-8-340542

  The present invention has been made in view of the above-described problems of the prior art, and enables an optimal white balance adjustment of an image taken in an environment where external light and flash light are mixed, and the intensity of the flash light. If the light intensity is too strong or too weak, the optimal flash light intensity is estimated, and correction based on the adjusted image corresponding to the estimated optimal flash light intensity is performed to generate a high-quality image. It is an object of the present invention to provide an image processing method, an image processing apparatus, and a computer program that enable this.

The first aspect of the present invention is:
A flash component image calculating step for calculating a flash component image based on a first image captured without flashing and a second image captured with flash firing;
An intensity adjustment value calculating step for calculating an intensity adjustment value of the flash component image;
Applying the intensity adjustment value to calculate an intensity adjustment flash component image to calculate an intensity adjustment flash component image; and
A final adjusted image generating step for generating a final adjusted image based on the first image and the intensity adjusted flash component image;
A motion detection step of performing motion detection based on the first image and a third image taken without flashing;
A correction propriety determination step for determining whether correction is possible or not according to the size of the region of the motion part detected in the motion detection step;
If it is determined that correction is possible in the correction possibility determination step, the pixel value correction of the motion part detected in the motion detection step is executed, and if it is determined that correction is impossible in the correction possibility determination step, the flash is emitted. A white balance adjustment step of performing pixel value correction of the second image captured
An image processing method characterized by comprising:

  Furthermore, in an embodiment of the image processing method of the present invention, the flash component image calculating step includes a step of calculating difference image data between the first image and the second image, and for the difference image data Executing a white balance adjustment process based on a parameter corresponding to the flash light component, and the intensity adjustment value calculating step is a step of calculating an intensity adjustment value of the flash component image subjected to the white balance adjustment process. It is characterized by being.

Furthermore, in one embodiment of the image processing method of the present invention, the final adjusted image generation step includes:
(A) For the first image, a white balance adjusted first image that has been subjected to white balance adjustment processing based on a parameter corresponding to an external light component;
(B) an intensity-adjusted flash component image obtained by performing intensity adjustment on a flash component image obtained by performing white balance adjustment processing based on a parameter corresponding to the flash light component;
It is a step of executing a final adjusted image generation process based on the two images (a) and (b).

  Furthermore, in an embodiment of the image processing method of the present invention, the intensity adjustment value calculation step is based on the first image that has been subjected to white balance adjustment processing based on the external light component correspondence parameter and the flash light component correspondence parameter. The intensity adjustment value is calculated as an adjustment value for suppressing the number of pixels having a saturated pixel value in a non-linear conversion processing image generated based on a composite image with a flash component image subjected to white balance adjustment processing. .

  Furthermore, in an embodiment of the image processing method of the present invention, the nonlinear conversion processing is gamma correction, and the intensity adjustment value calculation step includes an adjustment value that suppresses the number of pixels having a saturated pixel value in the gamma correction image. The intensity adjustment value is calculated as follows.

  Furthermore, in an embodiment of the image processing method of the present invention, the image processing method further controls the flash emission to execute shooting of a plurality of images including the first image and the second image. And a photographing step.

Furthermore, the second aspect of the present invention provides
An image processing device,
A flash component image calculating unit that calculates a flash component image based on a first image captured without the flash being emitted and a second image captured with the flash being emitted;
An intensity adjustment value calculation unit for calculating an intensity adjustment value of the flash component image;
Applying the intensity adjustment value to calculate an intensity adjustment flash component image, an intensity adjustment flash component image calculation unit;
A motion detection unit that performs motion detection based on the first image and a third image captured without flashing;
A correction propriety determination unit that determines whether correction is possible according to the size of the region of the motion part detected by the motion detection unit;
When the correction availability determination unit determines that correction is possible, the pixel value correction of the motion part detected by the motion detection unit is performed, and when the correction availability determination unit determines that correction is not possible, the flash is emitted. A white balance adjustment unit that performs pixel value correction on the second image that has been captured,
An image processing apparatus characterized by comprising:

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the flash component image calculation unit includes a difference image calculation unit that calculates difference image data between the first image and the second image, and the difference image data. A white balance adjustment unit that executes a white balance adjustment process based on a parameter corresponding to the flash light component, and the intensity adjustment value calculation unit adjusts the intensity of the flash component image that has been subjected to the white balance adjustment process. It is the structure which calculates a value, It is characterized by the above-mentioned.

Furthermore, in one embodiment of the image processing apparatus of the present invention, the final adjusted image generation unit includes:
(A) For the first image, a white balance adjusted first image that has been subjected to white balance adjustment processing based on a parameter corresponding to an external light component;
(B) an intensity-adjusted flash component image obtained by performing intensity adjustment on a flash component image obtained by performing white balance adjustment processing based on a parameter corresponding to the flash light component;
The present invention is characterized in that the final adjusted image generation processing is executed based on the two images (a) and (b).

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the intensity adjustment value calculation unit is based on the first image on which the white balance adjustment processing based on the external light component correspondence parameter is executed and the flash light component correspondence parameter. The intensity adjustment value is calculated as an adjustment value for suppressing the number of pixels having a saturated pixel value in a nonlinear conversion processed image generated based on a composite image with a flash component image subjected to white balance adjustment processing. Features.

  Furthermore, in one embodiment of the image processing apparatus of the present invention, the nonlinear conversion processing is gamma correction, and the intensity adjustment value calculation unit adjusts the number of pixels having a saturated pixel value in the gamma corrected image. The intensity adjustment value is calculated as follows.

  Furthermore, in an embodiment of the image processing apparatus of the present invention, the image processing apparatus further controls the flash emission to capture a plurality of images including the first image and the second image. It has the imaging part which performs.

Furthermore, the third aspect of the present invention provides
A flash component image calculating step for calculating a flash component image based on a first image captured without flashing and a second image captured with flash firing;
An intensity adjustment value calculating step for calculating an intensity adjustment value of the flash component image;
Applying the intensity adjustment value to calculate an intensity adjustment flash component image to calculate an intensity adjustment flash component image; and
A final adjusted image generating step for generating a final adjusted image based on the first image and the intensity adjusted flash component image;
A motion detection step of performing motion detection based on the first image and a third image taken without flashing;
A correction propriety determination step for determining whether correction is possible or not according to the size of the region of the motion part detected in the motion detection step;
If it is determined that correction is possible in the correction possibility determination step, the pixel value correction of the motion part detected in the motion detection step is executed, and if it is determined that correction is impossible in the correction possibility determination step, the flash is emitted. A white balance adjustment step of performing pixel value correction of the second image captured
There is a computer program characterized by comprising:

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, the flash component image is calculated based on the first image captured without the flash and the second image captured with the flash, and the intensity of the flash component image is calculated. Saturated pixels in the image after nonlinear conversion processing such as gamma correction executed when generating the output image because the final adjusted image is generated by applying the intensity adjusted flash component image generated by executing adjustment It is possible to generate a high-quality image in which pixels having values, that is, overexposure are reduced.

  Further, according to the configuration of the present invention, the first image that has been subjected to the white balance adjustment by executing the white balance adjustment process based on the parameter corresponding to the external light component is performed on the first image that is taken without emitting the flash. Further, the intensity of the flash component image that has been subjected to the white balance adjustment process based on the parameter corresponding to the flash light component is adjusted for the flash component image, and the final adjusted image is generated based on these images. Since it is configured, optimal white balance adjustment using parameters corresponding to individual images is executed.

It is a figure which shows the structure of the image processing apparatus of this invention. It is a flowchart explaining the procedure of the image processing method of this invention. It is a flowchart explaining the procedure of the white balance adjustment process based on several image data in the image processing method of this invention. It is a flowchart explaining the production | generation processing procedure of the intensity | strength adjustment flash component image in the image processing method of this invention. It is a figure explaining the detail of the production | generation process of the intensity | strength adjustment flash component image in the image processing of this invention. It is a figure explaining the detail of the production | generation process of the intensity | strength adjustment flash component image in the image processing of this invention. It is a figure explaining the detail of the production | generation process of the intensity | strength adjustment flash component image in the image processing of this invention. It is a figure explaining the mechanism which performs pixel value adjustment processing based on a plurality of image data in image processing of the present invention.

  Hereinafter, the details of the present invention will be described with reference to the drawings. FIG. 1 shows a configuration example of an image processing apparatus according to the present invention, and shows an example of an image processing apparatus provided with an imaging unit.

  An image processing apparatus shown in FIG. 1 includes a lens 101, a diaphragm 102, a solid-state image sensor 103, a correlated double sampling circuit 104, an A / D converter 105, a DSP block 106, a timing generator 107, a D / A converter 108, and a video encoder 109. , A video monitor 110, a codec (CODEC) 111, a memory 112, a CPU 113, an input device 114, a flash controller 115, and a flash light emitting device 116.

  The input device 114 includes operation buttons such as a recording button on the camera body. The DSP block 106 is a block having a signal processing processor and an image RAM. The signal processing processor can perform pre-programmed image processing on image data stored in the image RAM. ing. Hereinafter, the DSP block is simply referred to as a DSP.

The overall operation of this embodiment will be described below.
Incident light that has passed through the optical system and reached the solid-state image sensor 103 first reaches each light receiving element on the imaging surface, is converted into an electrical signal by photoelectric conversion at the light receiving element, and is subjected to noise by the correlated double sampling circuit 104. After being removed and converted into a digital signal by the A / D converter 105, it is temporarily stored in an image memory in the digital signal processing unit (DSP) 106. If necessary, the flash light emitting device 116 can be caused to emit light via the flash control device 115 during photographing.

  In the state during imaging, the timing generator 107 controls the signal processing system so as to maintain image capture at a constant frame rate. A stream of pixels is also sent to the digital signal processing unit (DSP) 106 at a constant rate, and after appropriate image processing is performed there, the image data is sent to the D / A converter 108 and / or the codec (CODEC) 111 or both. Sent. The D / A converter 108 converts the image data sent from the digital signal processing unit (DSP) 106 into an analog signal, which is converted into a video signal by the video encoder 109 so that the video signal can be monitored by the video monitor 110. In this embodiment, the video monitor 110 serves as a camera finder. The codec (CODEC) 111 encodes image data sent from the digital signal processing unit (DSP) 106, and the encoded image data is recorded in the memory 112. Here, the memory 112 may be a recording device using a semiconductor, a magnetic recording medium, a magneto-optical recording medium, an optical recording medium, or the like.

  The above is the description of the entire system of the digital video camera of this embodiment. The image correction processing of the captured image is mainly executed in the digital signal processing unit (DSP) 106. Details of the image processing will be described below.

  The image processing is performed by sequentially executing processing described in a predetermined program code on the image signal stream input to the digital signal processing unit (DSP) 106. In the following description, the execution sequence of each process in the program will be described with reference to flowcharts. The present invention is not limited to the program processing mode described below, but can be realized by a hardware configuration having a function of executing a processing sequence described below.

  FIG. 2 is a flowchart for explaining the procedure of pixel value correction processing including white balance (WB) adjustment processing executed on the stream of the input image signal in the digital signal processing unit (DSP) 106.

In step S101, the diaphragm is set in advance, using a shutter speed, and taken without flash, and stores the no flash captured image as image data I ₁ on the memory in step S102. In step S103, the diaphragm is set in advance as in step S101, by using the shutter speed, photographed and flash emission, stores this flash has photographed image in step S104 as image data I ₂ in the memory.

In step S105, the diaphragm is set in advance as in step S101, by using the shutter speed, without flash, and photographing again, the memory of this no flash captured image as image data I ₃ at step S106 Store.

  Note that the photographing in steps S101, S103, and S105 is executed as continuous photographing, for example, continuous photographing at intervals of about 1/100 seconds. Using a plurality of images obtained from each photographing step, pixel value correction such as white balance (WB) adjustment processing is performed to generate a final adjusted image as a final pixel value corrected image.

In addition, the image data I ₁ , I ₂ , and I ₃ stored in the memory in steps S102, S104, and S106 are images subjected to camera shake correction. That is, if camera shake occurs during the shooting of the _three images I ₁ , I ₂ , and I ₃ , they are corrected in advance and stored in the memory. That is, when the captured image is a camera shake occurrence image, camera shake correction is performed between steps S101 and S102, steps S103 and S104, and steps S105 and S106, and the corrected image is stored in the memory. To store. Therefore, the image data I ₁ , I ₂ , and I ₃ stored in the memory are images as if they were taken continuously with the camera fixed to a tripod.

  Note that a conventionally known process can be applied to the camera shake correction process. For example, a method of detecting a shift using an acceleration sensor and shifting a lens, or a method of reading an image with a resolution larger than a target resolution using an image sensor and reading an appropriate part so that the shift does not occur, Conventionally used methods such as a method of correcting camera shake only by image processing without using a sensor are applied.

Next, in step S107, it is detected whether or not there is an image blur due to the movement of the subject itself during the photographing of the three images in steps S101, S103, and S105. The process for detecting whether there is an image blur due to the movement of the subject itself is performed by comparing two images from three images. For example, the moving part can be detected using the images I ₁ and I ₃ . As an example, there is a method of taking a difference with respect to each pixel of the image I ₁ and the image I ₃ and registering it as a portion where there is a motion when the difference is equal to or greater than a certain threshold value. If it is determined that there is no image blur due to the movement of the subject itself (step S108: No), the process proceeds to step S112. If a motion is detected (step S108: Yes), the process proceeds to step S109.

  In step S109, it is determined whether correction processing for performing appropriate white balance (WB) adjustment is possible for the moving portion detected in step S107. For example, a method of applying this determination processing based on the ratio of the number of pixels registered as a moving part in step S107 to the number of pixels of the entire image is applied. For example, when the ratio [ratio A] of the number of pixels registered as a moving part to the number of pixels of the entire image is equal to or larger than a predetermined threshold [Threshold], it is determined that correction is impossible and the ratio is less than the threshold. If there is, it is determined that correction is possible.

  If it is determined in step S109 that correction is not possible, the process proceeds to step S113. If it is determined that correction is possible, the process proceeds to step S110.

In step S113, with respect to captured image data I ₂ of there flash performs white balance (WB) adjustment, and generates an output image R, the process ends. Parameter values used for white balance adjustment include parameters set according to external light components, parameters set according to flash light components, or parameters set based on intermediate components between external light and flash light The white balance (WB) adjustment in which these parameters are set is executed. This white balance adjustment method is a conventional method and will not be described in detail. The parameter to be applied is a parameter indicated by a 3 × 3 matrix, and is a matrix to be applied to the conversion of the color component constituting the color of each pixel. As the 3 × 3 matrix, a matrix in which the components other than the diagonal components are set to 0 is applied.

  In step S110, pixel value correction including white balance adjustment processing based on a plurality of image data is executed. In step S110, only the correction of the pixel values of the pixels excluding the motion part detected in the motion part detection process of step S107 is executed, and the pixel value correction process of the motion part is executed in step S111. Alternatively, pixel value correction including white balance adjustment processing may be executed for all pixels in step S110, and correction may be executed again for only the moving partial pixels in step S111.

  In addition, the pixel value correction process of the motion part performed in step S111 is performed by, for example, a process applying a radiation definition function (Radial Basis Function) or a process applying a smoothing filter. Details of these processes are described in Japanese Patent Application No. 2003-31630, which is a patent application of the same applicant as the present invention.

  Next, white balance (WB) adjustment processing based on a plurality of image data in step S110 and step S112 will be described. The processes in step S110 and step S112 are the same process. That is, in step S112, when no motion is detected in the motion part detection process in step S107, pixel value correction is executed for all the images. In step S110, the motion detected in the motion part detection process in step S107. The pixel value correction of the pixels excluding the portion or the pixel value correction for all the pixels is executed by the same processing sequence. Details of this processing will be described with reference to FIG.

Step S201 Oite, taking the difference of each color component of the pixel flashlight has photographed image component data _{I 2} and the flash light without captured image _{I 1} (e.g., each of the RGB channels), the difference image _{F =} I 2 -I ₁ The flash component image is generated and stored in the memory. If the subject does not move between step S101 in which shooting is performed without flash light and step S103 in which shooting is performed with flash light, the difference image F = I ₂ −I ₁ is in a state where there is no external light. Thus, only the flash light is irradiated, and only the flash light is reflected on the subject, and is incident on the solid-state image sensor of the camera, that is, an image equivalent to a captured image, that is, a flash component image.

  Next, in step S202, white balance (WB) adjustment according to the color temperature of the flash light is performed on the difference image F. That is, the white balance (WB) adjustment is executed based on the parameter set for the difference image data F, which is a flash component image, according to the flash light component. Further, when the flash light is too bright or too dark, the level is adjusted so that the brightness of the image is optimal, and a corrected difference image F ′ as a corrected flash component image is generated.

In step S203, the flash light without captured image data I _1, executes the white balance (WB) adjustment matched to ambient light. That is, the white balance (WB) adjustment is performed on the photographed image data I ₁ without flash light based on the parameter corresponding to the external light component, and the corrected image I ₁ ′ is generated.

This is performed by white balance (WB) adjustment that has been widely known. For example, the technique described in JP-A-2001-78202 can be applied. In Japanese Patent Application Laid-Open No. 2001-78202, the object color component data and the spectral distribution of external light are illuminated from the difference image F between the image I ₂ captured with flash emission and the image I ₁ captured without flash and the known flash spectral characteristics. Calculated as component data. With this illumination component data, by executing the white balance (WB) adjustment of the flashlight without captured image I _1, to generate a corrected image I '.

  Next, in step S204, when the flash light is too bright or too dark, the intensity (gain) of the corrected difference image F ′ that is the flash component image corrected to optimize the brightness of the final result image is set. The intensity adjustment flash component image F ″ as the gain adjustment difference image is generated by adjusting.

The outline of the generation process of the intensity adjustment flash component image F ″ is as follows. That is, the corrected difference image F ′, the corrected image I ₁ ′, and statistics such as a histogram based on the pixel values of these images, and the synthesis Using the statistics about the image [F ′ + I ₁ ′] that has undergone nonlinear transformation such as gamma correction, the optimum gain of the corrected difference image F ′ is obtained, and the intensity adjustment as a gain adjustment difference image with adjusted brightness The flash component image F ″ is generated.

  Hereinafter, details of the process of generating the intensity adjustment flash component image F ″ (gain adjustment difference image) executed in step S204 will be described with reference to the flow shown in FIG.

First, in step S301, based on the corrected difference image F ′, which is a flash component image that has been subjected to white balance adjustment, a portion in the image that is considered to be captured of an object that is exposed to a certain amount of flash light. A flash mask M _f to be represented is obtained. An example of a specific method for obtaining the flash mask _Mf will be described below.

  First, with respect to the corrected difference image F ′, pixels whose luminance is equal to or higher than a predetermined threshold (Th1) are detected. The average value of the brightness of the extracted pixels having the brightness value equal to or greater than the threshold (Th1) is calculated. When the luminance of each pixel constituting the corrected difference image F ′ is equal to or greater than the calculated average value, the flash light is relatively large, and is a portion having a large influence on the final image obtained by combining the external light and the flash light. I can say that.

  Note that the pixel corresponding to the moving part of the subject itself detected in step S107 described with reference to FIG. 2 is excluded from the flash mask setting target pixels because the movement may appear as a flash component. . The pixel value correction process of the moving part is executed in step S111 in the processing flow in the figure.

That is,
(A) For the corrected difference image F ′, a pixel whose luminance is equal to or higher than a predetermined threshold value (b) A pixel that does not correspond to the moving part of the subject itself (a), a pixel region that satisfies the two conditions of (b) A mask to be configured is generated as a flash mask _Mf .

In step S302 and subsequent steps, an analysis process is performed on the image of the flash mask M _f portion composed of pixels satisfying both the above conditions (a) and (b), and the optimum gain g of the corrected difference image F ′ is calculated. As a gain adjustment difference image,
F "= gF '
Ask for.

The processing after step S302 will be described with reference to FIGS. 5, 6, and 7. FIG. In step S302, the average brightness value of the corrected difference image F ′ and the flash mask M _f portion of the corrected image I ₁ ′ is calculated.

FIG. 5A shows a histogram as a frequency distribution diagram of the luminance (pixel value) of the flash mask M _f portion of the corrected image I ₁ ′. The horizontal axis indicates the pixel value, and the vertical axis indicates the frequency. The pixel value indicates a luminance value of, for example, 0 (min) to 255 (max). Note that, for the sake of brevity, a description will be given by applying one histogram. However, in the case of an RGB color image, a histogram of each channel of each pixel, for example, each channel of RGB is generated, and each channel is generated. The processing described below is executed.

The luminance (pixel value) of the pixels constituting the flash mask M _f portion of the corrected image I ₁ ′ is distributed between a minimum value (min), for example, a pixel value = 0 to a maximum value (max), for example, a pixel value = 255. and has a pixel value represents an average value of the luminance value (avg ₁ hereinafter) as the average value (avg ₁₎ of the flash mask _{M f} part of the corrected image _{I 1} '.

FIG. 5 (2) shows a histogram as a frequency distribution diagram of the luminance (pixel value) of the flash mask _Mf portion of the corrected difference image F ′. The luminance (pixel value) of the pixels constituting the flash mask M _f portion of the correction difference image F ′ is the average value (referred to as avg _F ) of the luminance value of the flash mask M _f portion of the correction difference image F ′. avg _F ).

In step S302 of the flow shown in FIG. 4, the histograms shown in FIGS. 5 (1) and (2) are generated, and the average value in each histogram, that is,
Average value [avg ₁ ] of luminance (pixel value) of the flash mask M _f portion of the corrected image I ₁ ′
Average value [avg _F ] of the luminance (pixel value) of the flash mask M _f portion of the corrected difference image F ′
Is calculated.

In step S303, a composite image R _tmp = I ₁ ′ + F ′ of the corrected image I ₁ ′ and the corrected differential image F ′ is generated. This is generated by adding the pixel value of the corresponding pixel of the corrected difference image F ′ to the pixel value of each pixel of the corrected image I ₁ ′.

The histogram shown in FIG. 5 (3) is a frequency distribution diagram of the luminance (pixel value) of the flash mask M _f portion of the corrected image I ₁ ′ and the composite image R _tmp = I ₁ ′ + F ′ of the corrected difference image F ′. The histogram of is shown. The brightness (pixel value) of the pixels constituting the flash mask M _f portion of the composite image R _tmp = I ₁ '+ F' is set as a histogram that is shifted to the maximum pixel value (max) side as a whole, and the composite image R _tmp = The average value of the luminance values of the flash mask M _f portion of I ₁ '+ F' is the pixel value shown as the average value (avg _{1 + F} ).

  The luminance of each pixel constituting the image output from the solid-state image sensor has a linear relationship with the amount of light incident on the solid-state image sensor. On the other hand, an image output as a final image by a general digital still camera or the like is obtained by performing nonlinear conversion such as gamma correction on an image output from a solid-state imaging device. Gamma correction is a correction process for correctly displaying the brightness and color saturation of an image, and when the pixel value obtained by imaging with a digital still camera is output as it is, when it is observed by the human eye, May become unnatural. In order to solve this, the input pixel value and the output pixel value are subjected to nonlinear conversion processing using a predetermined γ curve, which is called gamma correction.

Here, this nonlinear transformation is denoted as T. In step S304, an image obtained by performing nonlinear transformation T on the composite image R _tmp = I ₁ ′ + F ′ of the corrected image I ₁ ′ and the corrected differential image F ′
Nonlinear transformation image R _tmp '= T (F ′ + I ₁ ′)
Is generated.

Here, the non-linear conversion image R _tmp ′ = T (F ′ + I ₁ ′) is non-linear with respect to each channel of each pixel of the composite image R _tmp = (F ′ + I ₁ ′), for example, the pixel value of each RGB channel. A nonlinear transformed image obtained by applying transformation T is shown.

FIG. 6 shows a nonlinear transformation T applied to the pixel value of each channel of each pixel of the corrected image I ₁ ′ and the combined image (F ′ + I ₁ ′) of the corrected difference image F ′ shown in FIG. 8 shows a histogram as a frequency distribution diagram of the luminance (pixel value) of the flash mask M _f part for the nonlinear transformed image T (F ′ + I ₁ ′) obtained in this way. The luminance (pixel value) of the pixels constituting the flash mask M _f portion of the linearly transformed image T (F ′ + I ₁ ′) is set as a histogram that is shifted to the maximum pixel value (max) side as a whole.

In step S305, the standard deviation is calculated from the histogram shown in FIG. As shown in FIG. 6, the average pixel value of the pixels constituting the flash mask M _f portion of the linearly transformed image T (F ′ + I ₁ ′) is [avg _rslt ]. The standard deviation is obtained as [std _rslt ].

As shown in FIG. 6, the histogram formed by the pixel values of the pixels constituting the flash mask M _f portion of the linearly transformed image T (F ′ + I ₁ ′) has a maximum value (max), for example, near pixel value = 255. It can be seen that a large number of pixels are distributed in the area and whiteout occurs. The overexposed pixel is a pixel that should exist in the overexposed portion 201 shown in FIG. That is, the pixel values of many pixels are set to the maximum value (maximum luminance value), that is, the saturated pixel value by gamma correction. This state is called so-called overexposure that cannot express the original pixel level difference.

  An ideal image is an image that is sufficiently bright and does not cause overexposure. An image having many saturated pixel values as shown in FIG. 6 due to gamma correction does not have an ideal image feature.

Therefore, in the present invention, the corrected image I ₁ ′ that has been subjected to the white balance adjustment process based on the external light component corresponding parameter, and the corrected difference image that is the flash component image that has been subjected to the white balance adjustment process based on the flash light component corresponding parameter An intensity adjustment value, that is, a gain (g) is calculated as an adjustment value for suppressing the number of pixels having a saturated pixel value in the nonlinear transformation processing image generated based on the composite image with F ′, and this gain is applied, The correction difference image F ′ is corrected to generate an intensity adjustment flash component image.

  Thus, in the present invention, the final result image (final adjustment image) is adjusted after virtually adjusting the intensity of the flash light by adjusting the gain (referred to as g) of the correction difference image F ′. Generate.

In step S306, in order to generate a bright and non-overexposed image as a final adjusted image,
The average value (avg ₁ ) of the luminance values of the flash mask M _f portion of the corrected image I ₁ ′ shown in FIG.
The average value (avg _F ) of the luminance values of the flash mask M _f portion of the correction difference image F ′ shown in FIG.
Standard deviation [std _rslt ] in the pixel value histogram of the pixels constituting the flash mask M _f portion of the linearly transformed image T (F ′ + I ₁ ′) shown in FIG.
Is used to calculate the gain g of the optimum correction difference image F ′, and in step S307, the gain g is multiplied by the value of each channel of each pixel of the correction difference image F ′ to obtain the intensity adjustment flash component image F ″. That is,
F "= gF '
As a result, an intensity adjustment flash component image F ″ is obtained.

  In general, when the measurement pixel values are distributed according to a normal distribution, about 68% of the measurement pixels in the range of “average value” ± “standard deviation” and “average value” ± “twice the standard deviation”. There are about 95% of the measured pixels in the range. A similar tendency can be considered when the pixel value distribution does not strictly follow the normal distribution.

Therefore, when many overexposures occur as in the pixel value histogram of the pixels constituting the flash mask M _f portion of the linearly transformed image T (F ′ + I ₁ ′) shown in FIG. 6, the flash mask M _f portion The luminance [average value] + [standard deviation], that is, the point [A] shown in FIG. 6 is often larger than the maximum pixel value (max).

Therefore, in the process of calculating the gain g for the corrected difference image F ′ in step S306, the “average value” obtained from the pixel value histogram of the pixels constituting the flash mask M _f portion of the linearly transformed image T (F ′ + I ₁ ′). "±" k times the standard deviation "(where k is a predetermined fixed scalar value, such as 1, 1.5, 2, etc.)
The gain g is set so that it does not take a range larger than the maximum pixel value (max) and is set to be sufficiently bright.

  For example, a histogram as shown in FIG. 7 can be considered as an ideal image histogram with little or no overexposure.

A luminance histogram of the flash mask _Mf portion of the gain adjustment nonlinear conversion image T (gF ′ + I ₁ ′) is set as a histogram as shown in FIG. 7, and a final output image is generated based on the gain adjustment nonlinear conversion image. Thus, it is possible to obtain an ideal image that is sufficiently bright and does not cause overexposure.

The average value of the histogram shown in FIG. 7 is [avg ′ _rslt ] and the standard deviation is [std ′ _rslt ] as shown in FIG. 7.

Here, the point [A] indicated by [average value] + [standard deviation] is smaller than the maximum value (max) shown in FIG. 7, and the average value [avg ′ _rslt ] is a sufficiently bright pixel value. ing. For example, it is a value having a sufficient pixel level (luminance value) that is larger than the intermediate pixel value [(min + max) / 2].

Here, it is assumed that the histogram as shown in FIG. 7 is a luminance histogram of the flash mask _Mf portion of the gain adjustment nonlinear conversion image T (gF ′ + I ₁ ′).

In this embodiment, in a flash mask M _f portion of the gain adjustment nonlinear transformation image _{T (gF '+ I 1'} ) is that overexposure is to reduce and sufficient bright image, gain adjustment nonlinear transformation image T (gF '+ I _1' The average value [avg ′ _rslt ] of the luminance histogram (FIG. 7) of the flash mask M _f portion of) is equivalent to the value obtained by dividing k times the standard deviation [std ′ _rslt ] from the maximum luminance value (max). Adjust the gain g.

The flash mask M _f portion of the gain-adjusted nonlinear conversion image T (gF ′ + I ₁ ′) generated by applying such a gain g is a sufficiently bright image with little overexposure.

When the gain g for generating such an optimal result image is calculated, if the gain g is not extremely large or small, the nonlinear transformed image T (F ′ + I ₁ ′) and the gain adjusted nonlinear transformed image Since it is considered that the standard deviation of the luminance of the flash mask M _f portion of T (gF ′ + I ₁ ′) is not significantly different, the histogram shown in FIG. 7, that is, the gain-adjusted nonlinear transformation image T (gF ′ + I ₁ ′) standard deviation in the luminance histogram of the flash mask _{M f} moiety [std _'rslt] is a histogram shown in FIG. 6, that is, the gain adjustment before the nonlinear transformation image T (F' brightness of the flash mask _{M f} portion of the + _{I 1} ') Assume that it is the same size as the standard deviation [std _rslt ] in the histogram. That is,
std ' _rslt = std _rslt
It is.

The average value [avg ₁ ] of the histogram of the corrected image I ₁ ′ shown in FIG. 5A and the average value [g × avg] of the luminance of the flash mask M _f portion of the corrected differential image F ′ subjected to the gain g. _F ]] and a scalar value calculated by performing nonlinear transformation T on the added value [avg ₁ + g × avg _F ],
T (avg ₁ + g × avg _F )
Is equivalent to the average value [avg ′ _rslt ] (see FIG. 7) of the luminance of the flash mask M _f portion of the final gain-adjusted nonlinear transformed image T (gF ′ + I ₁ ′). That is,
avg ′ _rslt = T (avg ₁ + g × avg _F )
Assume that

That is, the average value [avg ′ _rslt ] of the final gain-adjusted nonlinear transformation image T (gF ′ + I ₁ ′) to be obtained is equal to T (avg ₁ + g × avg _F ). When this value becomes the same value as (max−k × std _rslt ), an image corresponding to the histogram shown in FIG. 7 is set, and as a result, a histogram corresponding to a sufficiently bright image with little overexposure is obtained. .

That is,
T (avg ₁ + g × avg _F ) = max−k × std _rslt (Formula 1)

A gain [g] that satisfies the above (Equation 1) may be calculated. As a formula for calculating the gain [g], the following (formula 2) obtained by modifying the above (formula 1) is applied.
g = {T ⁻¹ (max−k × std _rslt ) −avg ₁ } ÷ avg _F (Expression 2)
Here, T ⁻¹ represents the inverse transformation of the nonlinear transformation T.

In step S306 of the flow shown in FIG. 4, a gain g as an intensity adjustment value of the flash component image (corrected difference image F ′) is calculated based on the above (Equation 2). As understood from the above (Formula 2),
Gain [g] is
a) Average luminance value (avg ₁ ) of the flash mask M _f portion of the corrected image I ₁ ′ shown in FIG.
b) Average value (avg _F ) of the luminance values of the flash mask M _f portion of the correction difference image F ′ shown in FIG.
c) Standard deviation [std _rslt ] in the pixel value histogram of the pixels constituting the flash mask M _f portion of the nonlinear transformed image T (F ′ + I ₁ ′) shown in FIG.
d) Fixed scalar value [k]
It is also calculated based on each value.
The fixed scalar value [k] is a fixed value such as k = 1 or k = 1.5, and is a preset value.

In step S307, the intensity-adjusted flash component image F ″ is obtained by multiplying the value of each channel of each pixel of the corrected difference image F ′ by g.
F "= gF '
This intensity adjustment flash component image F ″ is set as a flash component image to be applied to the generation of the final adjustment image.

  The above process is the process of step S204 shown in FIG. Although the method for automatically calculating the gain g has been described here, the user can freely set the gain g and adjust it to the desired flash intensity.

After step S204 is completed, in step S205, the intensity adjustment flash component image F ″ set as the flash component image and the image I ₁ ′ are combined to generate a white balance adjustment image R as the final adjustment image. ,
R = I ₁ '+ F "
As a result, a white balance adjustment image R is generated.

  The above process is the detail of the process in step S112 and step S110 shown in FIG.

If no moving part is detected in step S107, the white balance adjustment image R obtained by synthesizing the intensity adjustment flash component image F ″ acquired in step S112 and the image I ₁ ′ in step S112, that is,
R = I ₁ '+ F "
Is the final adjusted image. After this, the final adjusted image R is subjected to nonlinear conversion processing such as gamma correction and output.

In addition, when a moving part is detected in step S107, the white balance adjustment image R acquired by the above-described process in step S110, that is,
R = I ₁ '+ F "
In step S111, the pixel value correction of the moving part is executed. As described above, the correction of the pixel value of the moving part in step S111 is performed by applying a radiation defining function (Radial Basis Function) described in Japanese Patent Application No. 2003-31630, which is a patent application of the same applicant as the present invention. The process is executed by applying a smoothing filter.

  If a moving part is detected in step S107, the image in which the pixel value of the moving part is corrected is determined as a final adjusted image in step S111.

In the pixel value correction in step 110, correction is performed on all the image constituent pixels without distinguishing whether the pixel is a moving part pixel or a non-moving part pixel, and the white balance in step 110 is performed. Adjustment image R, that is,
R = I ₁ '+ F "
In step 111, the image R may be corrected by executing exception correction processing for only the pixel value of the motion portion detected in step 107 after the generation of the image. Specifically, a method of synthesizing a final image by referring to the pixel value of the portion of the image R where there is no motion, using the pixel value of the image I _{2 with} flash emission of the portion where the motion is detected as an input, etc. Is applicable.

  FIG. 8 is a block diagram showing a functional configuration of a digital signal processing unit (DSP) (corresponding to the DSP 106 in FIG. 1) that executes processing according to the present embodiment.

  Processing in the digital signal processing unit (DSP) shown in FIG. 8 will be described in comparison with the flowchart shown in FIG.

In steps S101 to S106 in FIG. 2, the photographed non-flashing image I ₁ , the flashing image I ₂ , and the non-flashing image I ₃ are stored in the frame memories 301, 302, and 303, respectively. As a frame memory for storing images, a memory configured in a digital signal processing unit (DSP) may be applied, or a bus-connected memory (memory 112 in FIG. 1) may be applied.

The motion detection process in step S107 is executed in the motion detection unit 309. The motion detection process is executed as a detection process using difference data based on the non-flashing image I ₁ and the non-flashing image I ₃ .

  The white balance adjustment process based on the multiple image data executed in step S110 and step S112 is the process described above with reference to FIGS.

First, a flashed no image _{I 1,} on the basis of the flashed image existed _{I 2,} obtains the difference image data _{F =} I 2 -I ₁ in the difference image calculating unit 304 that is configured in the flash-component image output unit 330 (FIG. 3, S201). Next, the difference image data F = I ₂ −I ₁ , that is, the difference image F corresponding to the image photographed under the irradiation condition of only the flash light, is set by the white balance adjustment unit 307 based on the flash light component. The correction difference image F ′ as the flash component image corrected by executing the white balance adjustment process according to the parameters is obtained (FIG. 3, S202).

Further, the white balance adjustment unit 305 executes the white balance adjustment process according to the parameter set based on the estimated value of the external light component estimated by the external light component estimation unit 306 for the non-flash image I ₁ (see FIG. 3, S203).

  Further, the intensity adjustment value (gain) calculation unit 321 calculates a gain [g] as an intensity adjustment value for making the correction difference image F ′ an image with no or less overexposure, and the intensity adjustment flash component image. The calculation unit 322 calculates the intensity adjustment flash component image F ″ = gF ′. This process is the process of step S203 in FIG. 3, that is, the process according to the flow shown in FIG.

Further, in the final adjustment image calculation unit (pixel value addition unit) 308, the corrected image I ₁ ′ output from the white balance adjustment unit 305 and the intensity adjustment flash component image F output from the intensity adjustment flash component image calculation unit 322. The pixel values of “are added (FIG. 3, S205).

  When the captured image does not include a moving part, the image data having the pixel value added by the pixel value adding unit 308 is output as a white balance adjustment image without executing processing in the moving part correction pixel value calculating unit 310 The data is output via the unit 312. The output destination is a D / A converter 108 (see FIG. 1) that performs digital-analog conversion, a codec 111 that performs encoding processing, and the like.

On the other hand, when the motion detection unit 309 detects the motion region of the subject itself as a result of motion detection based on the difference data based on the non-flashing image I ₁ and the non-flashing image I ₃ , the motion partial correction pixel location calculation unit In 310, the pixel value of the portion with motion is corrected (transformed) by processing using a radial basis function, processing using a smoothing filter, and the like, and the portion with motion is corrected by the corrected pixel value. An image having the replaced pixel value data is output via the output switching unit 312.

The white balance adjustment unit 311 executes step S113 in the processing flow of FIG. That is, although the motion area by the motion detector 309 is detected, such as a high percentage of the total image of the motion area, if it is determined that uncorrectable, enter the flashed there image I _2, a predetermined parameter The white balance adjustment according to the above is executed, and this is output via the output switching unit 312.

  In the configuration shown in FIG. 8, each processing unit is shown separately in order to explain the function. However, as an actual process, a program in the DSP is executed according to a program that executes a process according to each process flow described above. It can be executed by the processor.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims section described at the beginning should be considered. In the above-described embodiment, the term “flash” is used as an illumination device that emits light when the subject is dark. However, the term “flash” may also be used instead of the flash. The present invention is applied to an illumination device that emits light when the subject is dark.

The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.
For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.
Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the present invention, the flash component image is calculated based on the first image captured without the flash and the second image captured with the flash. Since the final adjustment image is generated by applying the intensity adjustment flash component image generated by executing the intensity adjustment of the flash component image, nonlinear conversion such as gamma correction executed when generating the output image It is possible to generate a pixel having a saturated pixel value in the processed image, that is, a high-quality image with reduced overexposure, and can be applied to, for example, correction processing of an image shot with a digital camera.

  Further, according to the configuration of the present invention, the first image that has been subjected to the white balance adjustment by executing the white balance adjustment process based on the parameter corresponding to the external light component is performed on the first image that is taken without emitting the flash. Further, the intensity of the flash component image that has been subjected to the white balance adjustment process based on the parameter corresponding to the flash light component is adjusted for the flash component image, and the final adjusted image is generated based on these images. Since it is configured, optimal white balance adjustment using parameters corresponding to individual images is executed, and can be applied, for example, to correction processing of images taken with a digital camera.

DESCRIPTION OF SYMBOLS 101 Lens 102 Diaphragm 103 Solid-state image sensor 104 Correlated double sampling circuit 105 A / D converter 106 DSP block 107 Timing generator 108 D / A converter 109 Video encoder 110 Video monitor 111 Codec (CODEC)
112 memory 113 CPU
114 Input device 115 Flash control device 116 Flash light emitting device 301, 302, 303 Frame memory 304 Difference image calculation unit 305 White balance adjustment unit 306 Ambient light component estimation unit 307 White balance adjustment unit 308 Pixel value addition unit 309 Motion detection unit 310 Motion Partial correction pixel value calculation unit 311 White balance adjustment unit 312 Output switching unit 321 Intensity adjustment value (gain) calculation unit 322 Intensity adjustment flash component image calculation unit 330 Flash component image calculation unit

Claims (6)

An image processing method executed in an image processing apparatus,
A first image I ₁ that is a continuously shot image and a captured image without flash emission , a second image I ₂ that is a captured image with flash emission, and a third image I ₃ that is a captured image without flash emission are input.
And the first image I _1, performs motion detection processing based on said third image I _3,
In the motion detection process, when it is determined that there is a motion part and it is determined that correction is not possible in the correction possibility determination process according to the size of the motion part area, a white balance adjustment process is performed on the second image I ₂ . To generate a white balance adjustment image R,
In the motion detection process, when it is determined that there is no moving part, or
When it is determined that there is a moving part and it is determined that correction is possible in the correction possibility determination process according to the size of the moving part area,
Calculating a difference image F between the first image I ₁ and the second image I ₂ ;
For the difference image F, a difference correction image F ′ obtained by performing white balance adjustment processing based on the flash light component correspondence parameter is generated,
Generating a first corrected image I ₁ ′ obtained by executing white balance adjustment processing based on the external light component corresponding parameter for the first image ;
As an adjustment value for suppressing the number of saturated pixels in the nonlinear transformation processed image generated based on the composite image R _tmp of the difference correction image F ′ and the first correction image I ₁ ′, the difference correction image F ′ Calculate the strength adjustment value g,
An intensity adjustment difference correction image F ″ is generated by intensity adjustment applying the intensity adjustment value g to the difference correction image F ′,
An image processing method, comprising: executing a process of generating a white balance adjustment image R by a synthesis process of the first correction image I ₁ ′ and the intensity adjustment difference correction image F ″ . The nonlinear conversion process is a gamma correction,
The image processing method according to claim 1, wherein the intensity adjustment value g is an intensity adjustment value that suppresses the number of pixels having a saturated pixel value in a gamma corrected image . An image processing device,
A first image I ₁ that is a continuously shot image and a captured image without flash emission , a second image I ₂ that is a captured image with flash emission, and a third image I ₃ that is a captured image without flash emission are input.
And the first image I _1, performs motion detection processing based on said third image I _3,
In the motion detection process, when it is determined that there is a motion part and it is determined that correction is not possible in the correction possibility determination process according to the size of the motion part area, a white balance adjustment process is performed on the second image I ₂ . To generate a white balance adjustment image R,
In the motion detection process, when it is determined that there is no moving part, or
When it is determined that there is a moving part and it is determined that correction is possible in the correction possibility determination process according to the size of the moving part area,
Calculating a difference image F between the first image I ₁ and the second image I ₂ ;
For the difference image F, a difference correction image F ′ obtained by performing white balance adjustment processing based on the flash light component correspondence parameter is generated,
Generating a first corrected image I ₁ ′ obtained by executing white balance adjustment processing based on the external light component corresponding parameter for the first image ;
As an adjustment value for suppressing the number of saturated pixels in the nonlinear transformation processed image generated based on the composite image R _tmp of the difference correction image F ′ and the first correction image I ₁ ′, the difference correction image F ′ Calculate the strength adjustment value g,
An intensity adjustment difference correction image F ″ is generated by intensity adjustment applying the intensity adjustment value g to the difference correction image F ′,
An image processing apparatus that executes a process of generating a white balance adjustment image R by a synthesis process of the first correction image I ₁ ′ and the intensity adjustment difference correction image F ″ . The nonlinear conversion process is a gamma correction,
The image processing apparatus according to claim 3 , wherein the intensity adjustment value g is an intensity adjustment value that suppresses the number of pixels having a saturated pixel value in a gamma corrected image . The image processing apparatus further includes:
4. The image processing apparatus according to claim 3 , further comprising: an imaging unit that controls flash emission to capture a plurality of images including the first image I ₁ and the second image I _{2. 5} . A computer program for executing image processing in an image processing apparatus;
A process of inputting a first image I ₁ that is a continuously shot image that is a captured image without flash emission , a second image I ₂ that is a captured image with flash emission, and a third image I ₃ that is a captured image without flash emission; ,
A motion detection process based on the first image I ₁ and the third image I ₃ ;
In the motion detection process, when it is determined that there is a motion part and it is determined that correction is not possible in the correction possibility determination process according to the size of the motion part area, a white balance adjustment process is performed on the second image I ₂ . Processing to generate the white balance adjustment image R,
In the motion detection process, when it is determined that there is no moving part, or
When it is determined that there is a moving part and it is determined that correction is possible in the correction possibility determination process according to the size of the moving part area,
A process of calculating a difference image F between the first image I ₁ and the second image I ₂ ;
A process for generating a difference-corrected image F ′ obtained by executing a white balance adjustment process based on the flash light component correspondence parameter for the difference image F;
Processing for generating a first corrected image I ₁ ′ in which white balance adjustment processing based on an external light component corresponding parameter is performed on the first image ;
As an adjustment value for suppressing the number of saturated pixels in the nonlinear transformation processed image generated based on the composite image R _tmp of the difference correction image F ′ and the first correction image I ₁ ′, the difference correction image F ′ Processing for calculating the intensity adjustment value g;
A process of generating an intensity adjustment difference correction image F ″ by intensity adjustment applying the intensity adjustment value g to the difference correction image F ′;
A computer program for executing a process of generating a white balance adjustment image R by a synthesis process of the first correction image I ₁ ′ and the intensity adjustment difference correction image F ″ .

Images