JP2013070241A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct deterioration in the gradation characteristics or color reproduction characteristics occurring due to short exposure time when compounding a plurality of high sensitivity images.SOLUTION: The image processing apparatus comprises image acquisition units 103, 104 acquiring a plurality of high sensitivity images captured based on a first exposure time, and at least one low sensitivity image captured based on a second exposure time longer than the first exposure time, a compounding unit 103 generating a composite image by compounding a plurality of high sensitivity images, a first statistic calculation unit 104 calculating a statistic based on the pixel value of at least partial area of a composite image, a second statistic calculation unit 104 calculating a statistic based on the pixel value of at least partial area of a low sensitivity image corresponding at least to a partial area of a composite image, and a correction unit 104 correcting the pixel value of a composite image so as to bring the statistic calculated by the first statistic calculation unit 104 close to the statistic calculated by the second statistic calculation unit 104.

Description

  The present invention relates to an image processing technique, and more particularly to an image processing technique for generating a composite image by combining a plurality of images.

  In an imaging apparatus such as a digital camera, it is effective to secure a sufficient exposure time in order to obtain an image with little noise when capturing a still image. However, if the exposure time is increased, the image is blurred due to camera movement or subject movement due to camera shake. As a method for dealing with such image blur, an electronic blur correction method has been proposed. In Patent Document 1 and Patent Document 2, an image having no blur is obtained by continuously performing photographing with a short exposure time with little blur and performing composition processing after performing alignment processing of a plurality of obtained images. A technique for obtaining the above is disclosed.

  However, if the exposure time is shortened, sufficient charge is not accumulated in the image sensor, so shooting is performed in an area with poor linearity characteristics (proportional relationship between exposure time and pixel value), and gradation characteristics and color reproduction characteristics deteriorate. Problem arises. Patent Document 3 discloses a technique for correcting the linearity characteristic of the output of the image sensor based on the linearity characteristics measured in advance for the RAW image data output from the image sensor.

JP 2008-5084 A JP 2006-157568 A JP 2003-23642 A

  However, it is cumbersome to measure linearity characteristics with each camera in advance, and sufficient effects cannot be obtained only by correction based on the previous measurement results.

  The present invention has been made in view of the above problems, and corrects the deterioration of gradation characteristics and color reproduction characteristics caused by a short exposure time when combining a plurality of images, thereby correcting gradation characteristics and color reproduction. An object is to provide a technique capable of obtaining a composite image with good characteristics.

  An image processing apparatus according to an aspect of the present invention acquires a plurality of high-sensitivity images shot based on a first exposure time, and based on a second exposure time longer than the first exposure time. An image acquisition unit that acquires at least one low-sensitivity image captured in the above; a synthesis unit that generates a composite image by combining the plurality of high-sensitivity images; and at least a region of the composite image. A first statistic calculator that calculates a statistic based on a pixel value of the image, and a statistic based on a pixel value of at least a partial area of the low-sensitivity image corresponding to at least a partial area of the composite image And correcting the pixel value of the composite image so that the statistic calculated by the second statistic calculator and the statistic calculated by the second statistic calculator are closer to the statistic calculated by the second statistic calculator. Correction unit , Comprising a.

  An image processing method according to another aspect of the present invention obtains a plurality of high-sensitivity images taken based on a first exposure time, and has a second exposure time longer than the first exposure time. Acquiring at least one low-sensitivity image captured based on the image, generating a composite image by combining the plurality of high-sensitivity images, and pixels of at least a part of the region of the composite image Calculating a statistic based on a value; calculating a statistic based on a pixel value of at least a part of the low sensitivity image corresponding to at least a part of the composite image; and the composite image The pixel value of the composite image is corrected so that the statistic based on the pixel value of at least a part of the region is closer to the statistic based on the pixel value of the at least a part of the low-sensitivity image. It has a step that, a.

  An image processing program according to still another aspect of the present invention acquires a plurality of high-sensitivity images taken based on a first exposure time, and a second exposure time longer than the first exposure time. Acquiring at least one low-sensitivity image captured based on the image, generating a composite image by combining the plurality of high-sensitivity images, and at least part of a region of the composite image Calculating a statistic based on a pixel value; calculating a statistic based on a pixel value of at least a part of the low-sensitivity image corresponding to at least a part of the composite image; The statistic based on the pixel value of at least a partial area of the image is made closer to the statistic based on the pixel value of at least a partial area of the low-sensitivity image. And correcting the pixel value, which is an image processing program for causing a computer to execute the.

  According to the present invention, the gradation characteristics and color reproduction characteristics of the composite image are corrected according to the gradation characteristics and color reproduction characteristics of the low-sensitivity image captured in the region where the linearity characteristics of the image sensor are good. A composite image having good characteristics and color reproduction characteristics can be obtained.

FIG. 1 is a diagram illustrating a schematic configuration of an imaging apparatus equipped with an image processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing a detailed configuration of the gradation / color correction unit. FIG. 3 is a block diagram showing a detailed configuration of the first cumulative histogram calculation unit and the second cumulative histogram calculation unit. FIG. 4 is a diagram illustrating an example of a histogram calculation result. FIG. 5 is a diagram illustrating an example of a cumulative histogram. FIG. 6 is a diagram for explaining a method of calculating gradation correction characteristics by the gradation correction characteristic calculation unit. In FIG. 7, the image processing unit converts an image in the RAW data format into image data composed of R, G, and B information per pixel, or a luminance signal (Y) and two color difference signals (Cb, Cr) per pixel. FIG. 2 is a diagram illustrating a schematic configuration of an imaging apparatus including an image processing apparatus configured such that a plurality of image composition processing units and gradation / color correction units perform respective processes after being converted into image data including the above information. FIG. 8 is a block diagram illustrating a detailed configuration of the gradation / color correction unit of the image processing apparatus according to the second embodiment. FIG. 9 is a diagram for explaining the effect of correction by the image processing apparatus according to the second embodiment. FIG. 10 is a diagram for explaining the effect of correction by the image processing apparatus according to the third embodiment.

<First Embodiment>
FIG. 1 is a diagram illustrating a schematic configuration of an imaging apparatus equipped with an image processing apparatus according to the first embodiment. The image pickup apparatus includes an optical system 100, an image pickup element 101, a frame memory 102, a multiple-sheet composition processing unit 103, a gradation / color correction unit 104, and an image processing unit 105. Among these, the image processing apparatus according to the first embodiment includes a multiple-sheet composition processing unit 103, a gradation / color correction unit 104, and an image processing unit 105.

  Subject light input to the image sensor 101 through the optical system 100 including a lens or the like is photoelectrically converted and temporarily stored in the frame memory 102.

  In this embodiment, first, a plurality of high-sensitivity images are taken with an exposure time that does not cause camera shake, and stored in the frame memory 102. Next, one low-sensitivity image is captured with an exposure time longer than the exposure time when the high-sensitivity image was captured, and stored in the frame memory 102. For example, when the limit of the exposure time at which camera shake does not occur is 1/50 seconds and the ISO sensitivity that can obtain an image with appropriate brightness with an exposure time of 1/50 seconds is ISO 3200, a plurality of images (for example, 4 to 8) high-sensitivity images. Next, an image with the same brightness is taken with low sensitivity. For example, when the ISO sensitivity is ISO200, the exposure time is 16/50 seconds.

  The multiple-sheet composition processing unit 103 reads out a plurality of high-sensitivity images from the frame memory 102 and combines the read-out multiple high-sensitivity images to generate a composite image. For the synthesis process, for example, a known method as disclosed in Patent Document 1 or Patent Document 2 described above can be used. By this combining process, noise in the high-sensitivity image can be reduced.

  The composite image generated by the multiple image combining processing unit 103 is generated by combining high-sensitivity images captured in an area where the linearity characteristics of the image sensor are poor, although noise is reduced. Characteristics and color reproduction characteristics are not good. On the other hand, the low-sensitivity image stored in the frame memory 102 may be affected by camera shake, but is an image shot in an area where the linearity characteristic of the image sensor is good. The characteristics are good.

  The gradation / color correction unit 104 corrects the gradation characteristics and color reproduction characteristics of the composite image using the low-sensitivity image stored in the frame memory 102 and outputs the corrected image to the image processing unit 105.

  The image processing unit 105 outputs a RAW data format composite image (Bayer data format image) output from the gradation / color correction unit 104 as image data including R, G, and B information per pixel, or 1 The image data is converted into image data including information of a luminance signal (Y) per pixel and two color difference signals (Cb, Cr), and further, edge enhancement processing, noise reduction processing, and the like are performed.

  FIG. 2 is a block diagram showing a detailed configuration of the gradation / color correction unit 104. The tone / color correction unit 104 includes a first cumulative histogram calculation unit 200, a second cumulative histogram calculation unit 201, a tone correction characteristic calculation unit 202, and a tone correction unit 203.

  The first cumulative histogram calculation unit 200 calculates a cumulative histogram of the pixel values of all pixels for the composite image generated by the multiple-sheet composite processing unit 103. The second cumulative histogram calculation unit 201 calculates a cumulative histogram of the pixel values of all the pixels for the low sensitivity image stored in the frame memory 102.

  FIG. 3 is a block diagram showing a detailed configuration of the first cumulative histogram calculation unit 200 and the second cumulative histogram calculation unit 201. The first cumulative histogram calculation unit 200 and the second cumulative histogram calculation unit 201 are respectively a color separation unit 300, an R histogram calculation unit 301, a GR histogram calculation unit 302, a GB histogram calculation unit 303, and a B histogram calculation unit 304. An R cumulative histogram calculation unit 305, a GR cumulative histogram calculation unit 306, a GB cumulative histogram calculation unit 307, and a B cumulative histogram calculation unit 308.

  An image in the Bayer data format is input to the color separation unit 300. Specifically, the composite image is input to the color separation unit 300 of the first cumulative histogram calculation unit 200, and the low sensitivity image is input to the color separation unit 300 of the second cumulative histogram calculation unit 201. Each pixel of these Bayer data format images has color information of any one of R (red), GR (green), GB (green), and B (blue).

  The color separation unit 300 extracts each color component from an input image in the Bayer data format, and extracts an image of a pixel having an R component (hereinafter referred to as an R image) and an image of a pixel having a GR component (hereinafter referred to as a GR image). , A pixel image having a GB component (hereinafter referred to as a GB image) and a pixel image having a B component (hereinafter referred to as a B image) are generated. The R image is output to the R histogram calculation unit 301, the GR image is output to the GR histogram calculation unit 302, the GB image is output to the GB histogram calculation unit 303, and the B image is output to the B histogram calculation unit 304.

  The R histogram calculation unit 301, the GR histogram calculation unit 302, the GB histogram calculation unit 303, and the B histogram calculation unit 304 each calculate a pixel value histogram for all the pixels of the input image.

  FIG. 4 is a diagram illustrating an example of a histogram calculation result. In FIG. 4, the horizontal axis represents the pixel value, and the vertical axis represents the number of pixels. The pixel value has a minimum value of 0 and a maximum value of 255. Each histogram calculation unit 301 to 304 calculates the number of pixels indicating a certain pixel value for all pixels of the input image, and creates a histogram of pixel values.

  The histogram calculated by the R histogram calculation unit 301 is output to the R cumulative histogram calculation unit 305. Similarly, the histogram calculated by the GR histogram calculation unit 302 is calculated by the GR cumulative histogram calculation unit 306, and the histogram calculated by the GB histogram calculation unit 303 is calculated by the GB cumulative histogram calculation unit 307 and by the B histogram calculation unit 304. The histograms thus output are output to the B cumulative histogram calculation unit 308, respectively.

  The R cumulative histogram calculator 305, the GR cumulative histogram calculator 306, the GB cumulative histogram calculator 307, and the B cumulative histogram calculator 308 create a cumulative histogram based on the input pixel value histogram.

  FIG. 5 is a diagram illustrating an example of a cumulative histogram. In FIG. 5, the horizontal axis represents the pixel value, and the vertical axis represents the frequency. The pixel value has a minimum value of 0 and a maximum value of 255, and the frequency has a minimum value of 0% and a maximum value of 100%.

  The cumulative histogram is obtained by integrating the calculation results of the pixel value histogram in order from the pixel value 0 to the pixel value 255, and plotting the ratio of the calculated value to the total number of pixels for each pixel value. For example, in FIG. 5, the frequency at the pixel value X is 50%, which means that the number of pixels from the pixel value 0 to the pixel value X is 50% of the total number of pixels. The cumulative histogram created by each of the cumulative histogram calculators 305 to 308 is output to the gradation correction characteristic calculator 202.

  The gradation correction characteristic calculation unit 202 outputs the cumulative histogram calculation result of the composite image output from the first cumulative histogram calculation unit 200 and the cumulative histogram calculation result of the low sensitivity image output from the second cumulative histogram calculation unit 201. And a gradation correction characteristic that makes the gradation and color reproduction characteristics of the composite image close to the gradation and color reproduction characteristics of the low-sensitivity image.

  FIG. 6 is a diagram for explaining a method of calculating the gradation correction characteristic by the gradation correction characteristic calculation unit 202. In FIG. 6, the horizontal axis represents the pixel value, and the vertical axis represents the frequency. Further, the solid line in FIG. 6 represents the cumulative histogram of the composite image, and the dotted line represents the cumulative histogram of the low-sensitivity image.

  Here, gradation correction characteristics are calculated by performing gradation correction processing on the composite image so that the cumulative histogram of the corrected composite image matches the cumulative histogram of the low-sensitivity image. For example, when the frequency of the cumulative histogram of the pixel value X in the composite image is 50% and the pixel value corresponding to the cumulative histogram frequency of 50% in the low-sensitivity image is the pixel value Y, the pixel value X is changed to the pixel value Y. A tone correction characteristic to be corrected is obtained. Such association is performed for all pixel values, and a pixel value conversion table is created. This pixel value conversion table is the gradation correction characteristic calculation result.

  The calculation of the gradation correction characteristics as described above is performed for each of R, GR, GB, and B colors. By performing gradation correction for each color, the color reproduction characteristics are corrected as a result.

  Note that the tone characteristics and colors of the composite image are not required to match the cumulative histogram of the low-sensitivity image completely with the tone correction characteristics that substantially match the tone correction characteristics. The effect of correcting the reproduction characteristics can be obtained. Alternatively, a correction value may be calculated so that the cumulative histogram of the composite image becomes closer to the cumulative histogram of the low-sensitivity image. Specifically, tone correction characteristics such that the cumulative histogram of the composite image is changed so as to reduce the difference in pixel values corresponding to each frequency value between the cumulative histogram of the composite image and the cumulative histogram of the low-sensitivity image. Even so, the effect of correcting the tone characteristics and color reproduction characteristics of the composite image can be obtained. Alternatively, tone correction such that the cumulative histogram of the composite image is changed so that the difference between the pixel values corresponding to each frequency value between the cumulative histogram of the composite image and the cumulative histogram of the low-sensitivity image is equal to or less than a predetermined threshold. Even with the characteristics, the effect of correcting the gradation characteristics and color reproduction characteristics of the composite image can be obtained.

  The gradation correction unit 203 performs gradation correction for each color of R, GR, GB, and B of the composite image based on the gradation correction characteristic calculated by the gradation correction characteristic calculation unit 202, that is, the pixel value conversion table. Processing is performed, and the composite image after gradation correction is output to the image processing unit 105.

  As described above, the image processing apparatus according to the first embodiment acquires a plurality of high-sensitivity images photographed based on the first exposure time, and based on the second exposure time longer than the first exposure time. At least one low-sensitivity image captured in this way is acquired and a plurality of high-sensitivity images are combined to generate a composite image. Then, a statistic based on a pixel value of at least a part of the composite image and a statistic based on a pixel value of at least a part of the low-sensitivity image corresponding to at least a part of the composite image are calculated. Then, the pixel value of the composite image is corrected so that the statistic of the composite image becomes closer to the statistic of the low-sensitivity image. The gradation characteristics and color reproduction characteristics of the composite image are corrected according to the gradation characteristics and color reproduction characteristics of the low-sensitivity image captured in the area where the linearity characteristic of the image sensor is good, so that the floor caused by the short exposure time. It is possible to correct a deterioration in tone characteristics and color reproduction characteristics and obtain a composite image with favorable gradation characteristics and color reproduction characteristics.

  The conventional apparatus that measures the linearity characteristic in advance has a limit to the response when the linearity characteristic changes due to a temperature change or the like. However, according to the image processing apparatus in the first embodiment, the statistical amount based on the pixel values of at least a partial area of the composite image is set to at least one of the low-sensitivity images corresponding to at least a partial area of the composite image. Because the pixel value of the composite image is corrected so that it is closer to the statistic based on the pixel value of the area of the area, even if the linearity characteristics change due to temperature changes, etc., the gradation characteristics and color reproduction characteristics must be corrected reliably Can do.

  In particular, since the statistic is a cumulative histogram representing the cumulative value of the frequency of each pixel value, tone characteristics and color reproduction characteristics can be corrected effectively.

  In the above description, even when a positional deviation occurs between the synthesized image and the low-sensitivity image, the processing is performed without considering the positional deviation. It is also possible to correct the tone characteristics and the color reproduction characteristics of the composite image after performing the alignment process by detecting the positional deviation. By performing the alignment process between the synthesized image and the low-sensitivity image, the accuracy of the gradation correction process can be further improved.

  Further, in order to reduce the calculation amount of the first cumulative histogram calculation unit 200 and the second cumulative histogram calculation unit 201, the area for calculating the histogram may be a part of the image, or each of the composite image and the low-sensitivity image. Alternatively, a reduced image with a reduced number of pixels may be generated to calculate a histogram. When the area for calculating the histogram is only a part of the image, it is necessary to make a part of the composite image and a part of the low-sensitivity image correspond to each other.

  The multi-sheet composition processing unit 103 and the gradation / color correction unit 104 performed processing on an image in the Bayer data format. On the other hand, for image data consisting of R, G, B information per pixel, or image data consisting of information of luminance signal (Y) and two color difference signals (Cb, Cr) per pixel, Processing can also be performed.

  In FIG. 7, the image processing unit 105 converts an image in the RAW data format (an image in the Bayer data format) into image data consisting of R, G, and B information per pixel, or luminance signal (Y) per pixel and 2 Outline of an imaging apparatus equipped with an image processing apparatus having a configuration in which a plurality of image composition processing units 103 and gradation / color correction units 104 perform respective processes after conversion into image data including information of two color difference signals (Cb, Cr). It is a figure which shows a structure. About the same structure as the structure shown in FIG. 1, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  The image processing unit 105 converts a RAW data format image (Bayer data format image) output from the image sensor 101 into image data composed of R, G, and B information per pixel, or a luminance signal per pixel ( Y) and a synchronization process for converting the image data into information of two color difference signals (Cb, Cr), and an edge enhancement process and a noise reduction process are performed. The image that has been subjected to the processing described above by the image processing unit 105 is stored in the frame memory 102.

  The multiple-sheet compositing processing unit 103 generates a composite image using a plurality of high-sensitivity image data after the synchronization processing instead of an image in the Bayer data format.

  The gradation / color correction unit 104 does not require the color separation unit 300 in the internal configuration, and calculates a cumulative histogram for each of R, G, and B, or each of Y, Cb, and Cr. Processing, processing for calculating gradation correction characteristics, and gradation correction processing are performed.

<Second Embodiment>
In the image processing apparatus according to the first embodiment, the gradation / color correction unit 104 calculates a cumulative histogram for the entire composite image and low-sensitivity image, and brings the cumulative histogram of the composite image closer to the cumulative histogram of the low-sensitivity image. Therefore, the gradation characteristics and color reproduction characteristics of the composite image are improved by performing the gradation correction processing for the composite image. In the image processing apparatus according to the second embodiment, a method in which processing performed by the gradation / color correction unit is simplified will be described. Note that the gradation / color correction unit in the second embodiment and the gradation / color correction unit 104 in the first embodiment differ in the content of processing performed internally, and therefore the gradation / color correction unit in the second embodiment. The code of the color correction unit is 104A.

  FIG. 8 is a block diagram illustrating a detailed configuration of the gradation / color correction unit 104A of the image processing apparatus according to the second embodiment. The gradation / color correction unit 104A includes a first average value calculation unit 400, a second average value calculation unit 401, a correction value calculation unit 402, and a pixel value correction unit 403.

  The first average value calculation unit 400 calculates the average value of the pixel values of all the pixels of the composite image generated by the multiple-sheet composition processing unit 103. Further, the second average value calculation unit 401 calculates the average value of the pixel values of all the pixels of the low sensitivity image read from the frame memory 102. The composite image and the low-sensitivity image are images in the Bayer data format, and each pixel corresponds to one of R, GR, GB, and B. Therefore, the average value of the pixel values is also calculated for each color component. .

  The correction value calculation unit 402 matches the average value of the pixel values of the composite image calculated by the first average value calculation unit 400 with the average value of the pixel values of the low sensitivity image calculated by the second average value calculation unit 401. Calculate the correction value to be corrected. That is, the correction value is calculated by subtracting the average pixel value of the composite image from the average pixel value of the low-sensitivity image. The correction value is calculated for each of the R, GR, GB, and B color components.

  Note that the tone characteristics of the composite image are not the correction value that completely matches the average value of the pixel values of the composite image with the average value of the pixel values of the low-sensitivity image, And the effect of correcting the color reproduction characteristics can be obtained. Or you may make it calculate the correction value which makes the average value of the pixel value of a synthesized image approach the average value of the pixel value of a low sensitivity image. Specifically, even if the correction value reduces the difference between the average pixel value of the composite image and the average pixel value of the low-sensitivity image, the tone characteristics and color reproduction characteristics of the composite image are corrected. The effect to do is obtained. Alternatively, even if the correction value is such that the difference between the average value of the pixel values of the composite image and the average value of the pixel values of the low-sensitivity image is equal to or less than a predetermined threshold, the tone characteristics and color reproduction characteristics of the composite image are reduced. The effect of correcting is obtained.

  The pixel value correcting unit 403 adds the correction values of the respective colors calculated by the correction value calculating unit 402 to the pixel values of the respective colors of the combined image, thereby generating a corrected combined image.

  The correction value calculation unit 402 can also calculate a correction value by subtracting the average value of the pixel values of the low-sensitivity image from the average value of the pixel values of all the pixels of the composite image. In this case, the pixel value correction unit 403 generates a corrected composite image by subtracting the correction value of each color calculated by the correction value calculation unit 402 from the pixel value of each color of the composite image.

  FIG. 9 is a diagram for explaining the effect of correction by the image processing apparatus according to the second embodiment. In FIG. 9, the horizontal axis represents the brightness of the subject, and the vertical axis represents the pixel value. In FIG. 9, the solid line represents an example of the relationship between the brightness of the subject and the pixel value in the composite image, and the dotted line represents an example of the relationship between the brightness of the subject and the pixel value in the low-sensitivity image.

  For example, when a subject having the same brightness is photographed and a certain offset occurs in the pixel value in high-sensitivity photographing and low-sensitivity photographing, as shown in FIG. 9, the pixel value of the composite image and the low-sensitivity image A certain offset occurs between the pixel values. Therefore, by calculating the difference between the average value of the pixel values of the composite image and the average value of the pixel values of the low-sensitivity image, the offset value described above is estimated, and the pixel value of the composite image is calculated based on the estimated offset value. to correct. As described above, when the offset of a certain pixel value that does not depend on the brightness of the subject is the dominant factor of the error between the synthesized image and the low-sensitivity image, the simple correction process as described above is performed. As a result, the tone characteristics and color reproduction characteristics of the composite image can be improved.

  Note that examples of the occurrence of a constant offset regardless of the brightness of the subject include cases caused by noise differences between high-sensitivity shooting and low-sensitivity shooting, and errors in black level estimation processing (optical black correction processing). There may be cases.

  The first average value calculation unit 400 and the second average value calculation unit 401 have been described as calculating the average value of the pixel values of all the pixels. However, a pixel whose pixel value is the minimum value 0 or the maximum value 255 is selected. The average value may be excluded from the calculation. By excluding the pixel whose pixel value is the minimum value or the maximum value from the calculation of the average value, the offset value between the pixel value of the synthesized image and the pixel value of the low-sensitivity image can be obtained with higher accuracy. However, in general, since the ratio of the pixels having the minimum or maximum pixel value among all the pixels is low, it is necessary to exclude the pixels having the minimum or maximum pixel value from the calculation of the average value. However, the calculation accuracy of the offset value is not so low.

  As in the image processing apparatus according to the first embodiment, after detecting the positional deviation between the composite image and the low-sensitivity image and performing the alignment process, the gradation characteristics and color reproduction characteristics of the composite image are changed. You may make it correct | amend. Further, in order to reduce the calculation amount of the first average value calculation unit 400 and the second average value calculation unit 401, the area for calculating the average value may be only a part of the image, or each of the composite image and the low-sensitivity image. A reduced image with a reduced number of pixels may be generated and the average value calculated. Further, the multi-sheet compositing processing unit 103 and the gradation / color correction unit 104A performed processing on an image in the Bayer data format. However, image data including R, G, and B information per pixel, or 1 Processing may be performed on image data including information of a luminance signal (Y) per pixel and two color difference signals (Cb, Cr).

  As described above, according to the image processing apparatus of the second embodiment, a plurality of high-sensitivity images photographed based on the first exposure time are acquired, and the second exposure time longer than the first exposure time. At least one low-sensitivity image photographed based on the above is acquired, and a composite image is generated by combining a plurality of high-sensitivity images. Then, an average value of pixel values (statistics based on the pixel value) of at least a part of the composite image is set to a pixel value of at least a part of the low-sensitivity image corresponding to the part of the composite image. The pixel value of the synthesized image is corrected so as to be closer to the average value (statistics based on the pixel value). The gradation characteristics and color reproduction characteristics of the composite image are corrected according to the gradation characteristics and color reproduction characteristics of the low-sensitivity image captured in the area where the linearity characteristic of the image sensor is good, so that the floor caused by the short exposure time. It is possible to correct a deterioration in tone characteristics and color reproduction characteristics and obtain a composite image with favorable gradation characteristics and color reproduction characteristics. In particular, since the average value of the pixel values is used as the statistic, it is possible to obtain a composite image with favorable gradation characteristics and color reproduction characteristics with a small amount of calculation compared to the case where the cumulative histogram is used as the statistic.

  In addition, the correction value is added to each pixel of the composite image (or subtraction depending on the correction value) to correct the pixel value of the composite image, so an error due to a fixed offset occurs in high-sensitivity shooting and low-sensitivity shooting. In this case, it is possible to effectively obtain a composite image having good gradation characteristics and color reproduction characteristics.

<Third Embodiment>
In the second embodiment, the method of simply correcting the composite image has been described assuming a case where a certain offset occurs in the pixel value in high-sensitivity shooting and low-sensitivity shooting. In the third embodiment, a method of simply correcting a composite image will be described assuming a case where an error due to a certain gain occurs in high-sensitivity shooting and low-sensitivity shooting.

  The configuration of the image processing apparatus in the third embodiment is the same as the configuration of the image processing apparatus in the second embodiment. However, in the image processing apparatus according to the third embodiment, the operations of the correction value calculation unit and the pixel value correction unit of the gradation / color correction unit 104A are different from those of the second embodiment. In the following description, the reference numerals of the correction value calculation unit and the pixel value correction unit in the third embodiment are 402A and 403A, respectively.

  The correction value calculation unit 402A calculates a correction value that matches the average value of the pixel values of the composite image with the average value of the pixel values of the low-sensitivity image. The correction value is calculated for each of the R, GR, GB, and B color components. As in the second embodiment, a correction value may be calculated so that the average value of the pixel values of the composite image substantially matches the average value of the pixel values of the low-sensitivity image. Or you may make it calculate the correction value which makes the average value of the pixel value of a synthesized image approach the average value of the pixel value of a low sensitivity image. Specifically, even if the correction value is such that the ratio between the average value of the pixel values of the composite image and the average value of the pixel values of the low-sensitivity image is closer to 1, the tone characteristics and color reproduction characteristics of the composite image The effect of correcting is obtained. Alternatively, the ratio 1 between the average value of the pixel values of the composite image and the average value of the pixel values of the low-sensitivity image (the average value of the pixel values of the composite image and the average value of the pixel values of the low-sensitivity image are completely the same) Even if the correction value is such that the absolute value of the difference from the case is equal to or less than a predetermined threshold value, the effect of correcting the tone characteristics and color reproduction characteristics of the composite image can be obtained.

  The pixel value correction unit 403A multiplies the correction value of each color calculated by the correction value calculation unit 402A by the pixel value of each color of the composite image to generate a corrected composite image.

  FIG. 10 is a diagram for explaining the effect of correction by the image processing apparatus according to the third embodiment. In FIG. 10, the horizontal axis represents the brightness of the subject, and the vertical axis represents the pixel value. In FIG. 10, a solid line represents an example of the relationship between the brightness of the subject and the pixel value in the composite image, and a dotted line represents an example of the relationship between the brightness of the subject and the pixel value in the low-sensitivity image.

  For example, when an object having the same brightness is photographed and an error due to a certain gain occurs in the pixel value in high-sensitivity photographing and low-sensitivity photographing, as shown in FIG. An error due to a certain gain occurs between the pixel value of the image.

  The correction value calculation unit 402A obtains the estimated value of the gain by dividing the average value of the pixel values of the low-sensitivity image by the average value of the pixel values of the composite image for each of the R, GR, GB, and B color components. A correction value is calculated. The pixel value correction unit 403A corrects the pixel value of the composite image by multiplying the pixel value of each color of the composite image by the correction value of each color calculated by the correction value calculation unit 402A. In this way, when the error due to a constant gain that does not depend on the brightness of the subject is the dominant factor, the tone characteristics and color reproduction characteristics of the composite image can be reduced by performing the simple correction processing as described above. Can be improved.

  Here, the correction value calculation unit 402A calculates the correction value for each of the R, GR, GB, and B color components by dividing the average value of the pixel values of the composite image by the average value of the pixel values of the low-sensitivity image. The pixel value correction unit 403A can obtain the same result by dividing the pixel value of each color of the composite image by the correction value of each color calculated by the correction value calculation unit 402A.

  Note that examples of errors that occur due to a constant gain regardless of the brightness of the subject are due to errors in shutter speed between high-sensitivity shooting and low-sensitivity shooting, or due to errors in gain processing according to sensitivity. There are some cases.

  In addition, as described in the second embodiment, the first average value calculation unit 400 and the second average value calculation unit 401 change the average value of pixels whose pixel value is 0 of the minimum value or 255 of the maximum value. You may make it exclude from calculation.

  As in the image processing apparatus according to the first embodiment, after detecting the positional deviation between the composite image and the low-sensitivity image and performing the alignment process, the gradation characteristics and color reproduction characteristics of the composite image are changed. You may make it correct | amend. Further, in order to reduce the calculation amount of the first average value calculation unit 400 and the second average value calculation unit 401, the area for calculating the average value may be only a part of the image, or each of the composite image and the low-sensitivity image. A reduced image with a reduced number of pixels may be generated and the average value calculated. Further, the multi-sheet compositing processing unit 103 and the gradation / color correction unit 104A performed processing on an image in the Bayer data format. However, image data including R, G, and B information per pixel, or 1 Processing may be performed on image data including information of a luminance signal (Y) per pixel and two color difference signals (Cb, Cr).

  As described above, according to the image processing apparatus in the third embodiment, the plurality of high-sensitivity images captured based on the first exposure time are acquired, and the second exposure time longer than the first exposure time. At least one low-sensitivity image photographed based on the above is acquired, and a composite image is generated by combining a plurality of high-sensitivity images. Then, an average value (a statistic based on the pixel values) of at least a part of the composite image and a pixel value of at least a part of the low-sensitivity image corresponding to the at least part of the composite image The pixel value of the composite image is corrected so that the average value (statistics based on the pixel value) of the image is substantially the same. The gradation characteristics and color reproduction characteristics of the composite image are corrected according to the gradation characteristics and color reproduction characteristics of the low-sensitivity image captured in the area where the linearity characteristic of the image sensor is good, so that the floor caused by the short exposure time. It is possible to correct a deterioration in tone characteristics and color reproduction characteristics and obtain a composite image with favorable gradation characteristics and color reproduction characteristics. In particular, since the average value of the pixel values is used as the statistic, it is possible to obtain a composite image with favorable gradation characteristics and color reproduction characteristics with a small amount of calculation compared to the case where the cumulative histogram is used as the statistic.

  In addition, since the pixel value of the composite image is corrected by multiplying each pixel of the composite image by the correction value, it is effective when an error due to a certain gain occurs in high-sensitivity shooting and low-sensitivity shooting. A composite image having good gradation characteristics and color reproduction characteristics can be obtained.

  In the above description of the first to third embodiments, hardware processing is assumed as processing performed by the image processing apparatus, but the present invention is not limited to such a configuration. For example, a configuration in which software processing is performed by a computer is also possible. In this case, the computer includes a main storage device such as a CPU and a RAM, and a computer-readable storage medium storing a program for realizing all or part of the above processing. Here, this program is called an image processing program. The CPU reads out the image processing program stored in the storage medium and executes information processing / calculation processing, thereby realizing the same processing as that of the above-described image processing apparatus.

  Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, and the like. Alternatively, the image processing program may be distributed to a computer via a communication line, and the computer that has received the distribution may execute the image processing program.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment.

101 ... Image sensor 103 ... Multiple sheet composition processing unit (image acquisition unit, composition unit)
104, 104A: gradation / color correction unit (image acquisition unit)
105 Image processing unit 200 First cumulative histogram calculation unit (first statistic calculation unit)
201 ... 2nd cumulative histogram calculation part (2nd statistic calculation part)
202 ... gradation correction characteristic calculation unit (correction value calculation unit)
203 ... gradation correction unit (correction unit)
400 ... 1st average value calculation part (1st statistic calculation part)
401 ... 2nd average value calculation part (2nd statistic calculation part)
402: Correction value calculation unit (correction value calculation unit)
403 ... Pixel value correction unit (correction unit)

Claims (8)

A plurality of high-sensitivity images photographed based on the first exposure time are acquired, and at least one low-sensitivity image photographed based on the second exposure time longer than the first exposure time is obtained. An image acquisition unit to acquire;
Combining a plurality of high-sensitivity images to generate a combined image; and
A first statistic calculator that calculates a statistic based on pixel values of at least a partial region of the composite image;
A second statistic calculator that calculates a statistic based on a pixel value of at least a partial area of the low-sensitivity image corresponding to at least a partial area of the composite image;
A correction unit that corrects the pixel value of the composite image so that the statistic calculated by the first statistic calculation unit is closer to the statistic calculated by the second statistic calculation unit;
An image processing apparatus comprising: The first statistic calculator calculates a cumulative histogram representing a cumulative value of the frequency of each pixel value of the composite image as the statistic,
The second statistic calculation unit calculates a cumulative histogram representing a cumulative value of the frequency of each pixel value of the low sensitivity image as the statistic.
The image processing apparatus according to claim 1. The correction unit is configured to generate a composite image based on a gradation conversion characteristic determined based on a cumulative histogram calculated by the first statistic calculator and a cumulative histogram calculated by the second statistic calculator. Correcting the pixel value of the composite image by performing gradation conversion processing on the pixel;
The image processing apparatus according to claim 2. The first statistic calculator calculates an average value of pixel values of the composite image as the statistic,
The second statistic calculator calculates an average value of pixel values of the low-sensitivity image as the statistic.
The image processing apparatus according to claim 1. A correction value calculation unit that calculates a correction value for bringing the average value of the pixel values calculated by the first statistic calculation unit closer to the average value of the pixel values calculated by the second statistic calculation unit; ,
The correction unit corrects the pixel value of the composite image by adding or subtracting the correction value to the pixel value of the pixel of the composite image.
The image processing apparatus according to claim 4. A correction value calculation unit that calculates a correction value for bringing the average value of the pixel values calculated by the first statistic calculation unit closer to the average value of the pixel values calculated by the second statistic calculation unit; ,
The correction unit corrects the pixel value of the composite image by multiplying or dividing the pixel value of the pixel of the composite image by the correction value;
The image processing apparatus according to claim 4. A plurality of high-sensitivity images photographed based on the first exposure time are acquired, and at least one low-sensitivity image photographed based on the second exposure time longer than the first exposure time is obtained. A step to obtain,
Generating a composite image by combining the plurality of high-sensitivity images; and
Calculating a statistic based on pixel values of at least a portion of the composite image;
Calculating a statistic based on pixel values of at least a portion of the low sensitivity image corresponding to at least a portion of the composite image;
Correcting a pixel value of the composite image so that a statistic based on a pixel value of at least a partial region of the composite image is closer to a statistic based on a pixel value of at least a partial region of the low-sensitivity image. When,
An image processing method comprising: A plurality of high-sensitivity images photographed based on the first exposure time are acquired, and at least one low-sensitivity image photographed based on the second exposure time longer than the first exposure time is obtained. A step to obtain,
Generating a composite image by combining the plurality of high-sensitivity images; and
Calculating a statistic based on pixel values of at least a portion of the composite image;
Calculating a statistic based on pixel values of at least a portion of the low sensitivity image corresponding to at least a portion of the composite image;
Correcting a pixel value of the composite image so that a statistic based on a pixel value of at least a partial region of the composite image is closer to a statistic based on a pixel value of at least a partial region of the low-sensitivity image. When,
An image processing program for causing a computer to execute.

Images