JP2008219230A - Imaging apparatus, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To use entropy of each image generated from one image and having a different exposure as an index of visibility, without obtaining a plurality of images having different exposures by photographing. <P>SOLUTION: An imaging apparatus includes a generation means for generating, from one source image, the plurality of exposed images having different exposures by adding different gains to the source image; a differentiation means for differentiating the degree number of each column in a histogram corresponding to each of the plurality of exposed images according to the above gains; and a calculation means for calculating the entropy corresponding to each exposed image based on each histogram having the different degree number according to the gain. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus capable of optimizing the exposure of each part of a captured image, and particularly to an imaging apparatus and an image processing method that use entropy in the exposure optimization process.

  Conventionally, as an index for measuring the visibility of an image, the spread of a histogram is sometimes obtained using entropy (amount of information). If this entropy is Ec (entropy of a color image), Ec is given by the following equation (1).

但し、記号「ｉ」は画像の画素値（０≦ｉ＜階調数Ｎ）を、記号Ｐｉは画素値ｉの出現確率を示している。また、記号「ｒ」、「ｇ」及び「ｂ」はカラー画像における赤（Ｒ）、緑（Ｇ）及び青（Ｂ）を示している。

Here, the symbol “i” indicates the pixel value (0 ≦ i <the number of gradations N) of the image, and the symbol Pi indicates the appearance probability of the pixel value i. Symbols “r”, “g”, and “b” indicate red (R), green (G), and blue (B) in the color image.

  An image with high visibility includes a wide range of pixel values, that is, the range of values taken by the pixel value i is wide, so that the entropy is high. On the other hand, the brightly flying image and the darkly crushed image have relatively low entropy because the histogram is biased.

<Dynamic range compression technology using entropy>
By the way, when shooting with an imaging device such as a digital camera, when the brightness of the shooting scene changes greatly, regardless of the shutter speed or the like, the image is unevenly exposed, that is, the image is overexposed or underexposed. May occur. In this regard, a method of synthesizing a plurality of images (multiple exposure images) with different exposures has been proposed as dynamic range (DR) compression of images.

  A method using entropy is known as a reference for the synthesis of this multiple exposure image. For example, in Non-Patent Document 1, a plurality of images with different exposures (exposure amounts) are acquired by photographing a plurality of times with different exposures, for example, different shutter speeds, and the exposure of each part of the image is corrected from the plurality of images. In addition, a technique for generating a single image (Gothasby method) is disclosed. In this technique, each multiple exposure image is divided into a plurality of block images, and the block images having the highest entropy (information amount; average information amount) are compared by comparing the entropy between the block images at the same position in each multiple exposure image. select. The block images (each having the maximum entropy) at the respective positions thus selected are combined to create one image. However, since the brightness (brightness) of the block images is different simply by combining them, that is, a so-called mosaic image in which the joints of the block images are unnatural, so-called using LPF (Low Pass Filter) or the like. By performing the blurring process, the pixel values of each block image are blended (blending process; also referred to as entropy / blending process) to form a natural image.

  Further, for example, in Patent Document 1, an overexposed image and an underexposed image (multi-exposure image) are acquired by photographing, and each multiple is based on the entropy of each block image obtained by dividing these images in the same manner as described above. A technique for determining a composite ratio of exposure images is disclosed.

Further, for example, in Patent Document 2, pseudo multiple exposure images are generated from one image, and the above-mentioned Non-Patent Document 1 is based on block values such as average luminance values in each block image obtained by dividing these images. A technique for blending multiple exposure images in the same manner as in the above method is disclosed.
A. Gostasby, “Fusion of Multi-Exposure Images”, Image and Vision Computing, vol. 23, 2005, pp. 611-618 JP 2004-229259 A Japanese Patent Application No. 2006-263026

  However, in the techniques of Non-Patent Document 1 and Patent Document 1, it is necessary to capture a plurality of images (multiple exposure images) with different exposures, and this imaging takes time. In addition, subject blurring may occur between the plurality of images, resulting in a reduction in image quality of an image created from them. Further, in the technique disclosed in Patent Document 2, although it is not necessary to capture a plurality of images with different exposures, there is no problem that the imaging takes time, but a pseudo-multiple exposure image (pseudo multiple exposure image) is generated. ) Cannot be obtained correctly as the entropy corresponding to each multiple exposure image, and the entropy cannot be used as a visibility index.

  Specifically, when generating a pseudo multiple exposure image, a gain (digital gain) is basically applied to an underexposed image (underimage), that is, as shown in FIG. 12, for example, an original image I0 (underimage). The pseudo multiple exposure images I1 and I2 are generated by applying gains of various values, for example, a gain of 1.5 times or 2 times.

  FIG. 13 is an example of a histogram of an original image I0 (left side figure) and a pseudo multiple exposure image I2 (right side figure) obtained by multiplying the original image I0 by a gain of two, which are referred to as histograms H1 and H2, respectively. In this case, when entropy Ec is obtained by the above equation (1), the result (entropy value) is the same in any case. This is because, in the pseudo multiple exposure image I2, the histogram H2 has a so-called tooth-like shape (comb shape) in which each column is open, and the pixel value (pixel value at each column position; horizontal axis coordinate value). Is twice that of the original image I0, but the frequency itself does not change, so the occurrence probability remains the same, and the total occurrence probability (= entropy) does not change.

  The present invention has been made in view of the above circumstances, and there is no need to shoot each of a plurality of images (multiple exposure images) with different exposures, and subject blurring between the plurality of images is prevented. Provided are an imaging apparatus and an image processing method capable of obtaining entropy of images with different exposures generated from a single image as entropy corresponding to each image and using the entropy as a visibility index. For the purpose.

  The imaging apparatus according to the present invention includes a generating unit that generates a plurality of exposure images with different exposures by applying different gains to the original image from one original image, and each of the plurality of exposure images according to the gain. Difference means for differentiating the frequency of each column of the histogram corresponding to, and a calculation means for calculating entropy corresponding to each exposure image based on each histogram in which the frequency of each column differs according to the gain. It is characterized by.

  According to the above configuration, a plurality of exposure images with different exposures are generated by applying different gains to the original image from one original image by the generation unit, and a plurality of exposure images are generated by the difference unit according to the gain. The frequency of each column of the histogram corresponding to each is made different, and the entropy corresponding to each exposure image is calculated based on each histogram in which the frequency of each column differs according to the gain. That is, since each exposure image is created from one image, there is no need to shoot a plurality of images (multi-exposure images) with different exposures, and subject blurring between the plurality of images can be prevented. . In addition, since the frequency of each column in the histogram of each exposure image differs depending on the gain, the entropy obtained from these histograms also differs in each exposure image (the difference in entropy of each exposure image occurs). Therefore, the entropy of each exposure image generated from one image can be obtained as the entropy corresponding to each exposure image, and the entropy can be used as an indicator of visibility. By using it, it becomes possible to obtain one image with high image quality with appropriate exposure and contrast from a plurality of exposed images.

  Further, in the above configuration, the different means includes an intermittent histogram in which a gap is formed between each column by multiplying each pixel value of the original image corresponding to each exposure image by the gain. It is preferable to interpolate so that there is no gap between the columns (claim 2).

  According to this, an intermittent histogram in which a gap is left between columns by multiplying each pixel value of the original image corresponding to each exposure image by a gain by the difference means, a gap between the columns is obtained. Since interpolation is performed so as to disappear, it is possible to easily obtain a histogram of each exposure image having a frequency different depending on the gain by using a simple method called interpolation.

  Further, in the above-described configuration, from among the entropy calculated by the calculating unit corresponding to the dividing unit that divides each exposed image into a plurality of partial images and each partial image at the same position of each exposed image. It is preferable that the apparatus further comprises selection means for selecting the maximum entropy, and combining means for combining the partial images having the maximum entropy.

  According to this, each exposure image is divided into a plurality of partial images by the dividing means, and each of the entropies calculated by the calculating means corresponding to each partial image at the same position of each exposure image by the selecting means. The maximum entropy is selected. Then, since the partial images having the maximum entropy are combined by the combining means, the entropy of each part of the image (each partial image) is maximized from the plurality of exposed images using the entropy information 1 Two composite images, that is, high-quality images with appropriate exposure and contrast can be obtained.

  Further, in the above configuration, the histogram is a histogram in which a horizontal axis coordinate is a pixel value corresponding to a gradation value and a vertical axis coordinate is a frequency, and the difference means is a histogram corresponding to each of the plurality of exposure images. Preferably, the horizontal axis coordinates are reduced in accordance with the gain when the histogram is generated.

  According to this, the histogram is a histogram in which the horizontal coordinate is a pixel value corresponding to the gradation value, and the vertical coordinate is the frequency. Then, the difference unit is a generation unit that generates a histogram corresponding to each of the plurality of exposure images, and the horizontal axis coordinate is reduced according to the gain when the histogram is generated by the difference unit. In other words, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of simply creating a histogram having the horizontal axis coordinates reduced according to the gain.

  Further, in the above configuration, when the number of gradations of the original image is N and the gain is G, the creation unit creates the histogram having the horizontal axis coordinate having a maximum gradation value of N / G. (Claim 5).

  According to this, a histogram having a horizontal axis coordinate having a maximum gradation value of N / G is created by the creating means, and therefore a histogram having a horizontal axis coordinate reduced according to the gain is easily created. be able to.

  In the above configuration, the creation unit divides each pixel value of the original image by the G value, and counts the frequency of the pixel value in the horizontal coordinate corresponding to the integer value obtained by the division. (Claim 6).

  According to this, each pixel value of the original image is divided by the value of G by the creating means, and the frequency of the pixel value in the horizontal axis coordinate corresponding to the integer value obtained by the division is counted. Using a simple frequency counting method, a histogram having horizontal axis coordinates with a maximum gradation value of N / G can be easily created from the original image.

  In the above configuration, it is preferable that the difference unit is an image smoothing unit that spatially smoothes each exposed image before calculating the entropy.

  According to this, since the different means is an image smoothing means for spatially smoothing each exposure image before entropy calculation, that is, each exposure image is spatially smoothed by the image smoothing means. Therefore, the histogram created from each of the smoothed exposure images becomes a histogram having a different frequency depending on the gain. As described above, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of spatially smoothing each exposure image.

  In the above configuration, it is preferable that the image smoothing unit performs low-pass filter processing on each of the exposed images.

  According to this, since the low-pass filter process is performed on each exposure image by the image smoothing means, each exposure image can be easily spatially smoothed by a simple method called low-pass filter processing.

  Further, in the above configuration, the image forming apparatus further includes a creating unit that creates a histogram for each of the plurality of exposure images, and the difference unit smoothes the histogram of each exposure image created by the creating unit before calculating the entropy. Preferably, it is a histogram smoothing means.

  According to this, the difference means is a histogram smoothing means for smoothing the histogram of each exposure image created by the creation means before calculating the entropy, that is, the histogram of each exposure image is obtained by the histogram smoothing means. Since smoothing is performed, the smoothed histogram becomes a histogram having a different frequency depending on the gain. In this way, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of smoothing the histogram of each exposure image.

  In the above configuration, it is preferable that the histogram smoothing unit smoothes the histogram by performing a moving average process on the histogram created by the creating unit.

  According to this, since the histogram smoothing means smoothes the histogram by performing the moving average process on the histogram created by the creating means, the histogram averaging can be performed by a simple method of performing the moving average process on the histogram. Can be easily smoothed.

  Further, in the above configuration, the synthesizing unit includes a pixel value of the partial image having the maximum entropy and a weight according to a distance from the position of the pixel value of the partial image to a predetermined reference position of the partial image. It is preferable to combine the partial images by weighted averaging.

  According to this, the pixel value of the partial image having the maximum entropy and the weight according to the distance from the position of the pixel value of the partial image to the predetermined reference position of the partial image are weighted and averaged by the combining means. Since the partial images are combined, the pixel values of the partial images are blended so that there is no seam between the partial images (not a mosaic image), and the overall exposure and contrast are optimized. A single high-quality composite image can be obtained.

  Further, in the above configuration, the camera further includes a photographing unit that obtains the original image by photographing, the photographing unit is a linear log sensor having a photoelectric conversion characteristic including a linear characteristic and a logarithmic characteristic, and the generating unit includes the linear log Preferably, each exposure image is generated by applying different gains to the original image unified with the linear characteristics after unifying photoelectric conversion characteristics in the original image obtained by the sensor into the linear characteristics (claim). Item 12).

  According to this, the photographing means for obtaining the original image is a linear log sensor having a photoelectric conversion characteristic composed of a linear characteristic and a logarithmic characteristic, and the photoelectric conversion characteristic in the original image obtained by the linear log sensor by the generating means. Since each exposure image is generated by applying different gains to the original image unified to the linear characteristic after the linear characteristic is unified, even when the photographing means is such a linear log sensor, Each exposure image can be easily generated from the original image obtained by the linear log sensor.

  According to the image processing method of the present invention, the first step of generating a plurality of exposure images having different exposures by applying different gains to the original image from one original image, and depending on the gain The entropy corresponding to each exposure image based on the second step of varying the frequency of each column of the histogram corresponding to each of the plurality of exposure images, and each histogram having a different frequency of each column according to the gain. And a third step of calculating each of them (claim 13).

  According to this, in the first step, a plurality of exposure images with different exposures are generated from one original image by applying different gains to the original image. In the second step, a plurality of exposure images are generated according to the gain. In the third step, the entropy corresponding to each exposure image is calculated on the basis of the histograms having different frequencies for each column in accordance with the gain. Is done. That is, since each exposure image is created from one image, there is no need to shoot a plurality of images (multi-exposure images) with different exposures, and subject blurring between the plurality of images can be prevented. . In addition, since the frequency of each column in the histogram of each exposure image differs depending on the gain, the entropy obtained from these histograms also differs in each exposure image (the difference in entropy of each exposure image occurs). Therefore, the entropy of each exposure image generated from one image can be obtained as the entropy corresponding to each exposure image, and the entropy can be used as an indicator of visibility. By using it, it becomes possible to obtain one image with high image quality with appropriate exposure and contrast from a plurality of exposed images.

  Further, in the above configuration, in the creation of a histogram corresponding to each of the plurality of exposure images, the second step corresponds to the abscissa of the histogram with the pixel value corresponding to the gradation value as a coordinate according to the gain. It is preferable that the reduction step be performed (claim 14).

  According to this, when the histogram corresponding to each of the plurality of exposure images is created in the second step, the horizontal axis coordinate of the histogram having the pixel value corresponding to the gradation value as a coordinate is reduced according to the gain. Since it is a process, it is possible to vary the frequency of the histogram corresponding to each exposure image according to the gain by a simple method of simply creating a histogram having horizontal axis coordinates reduced according to the gain. Become.

  In the above configuration, it is preferable that the second step is a step of spatially smoothing each of the exposed images before performing the third step.

  According to this, since the second step is a step of spatially smoothing each exposure image before performing the third step, a histogram created from each smoothed exposure image is The histogram has a different frequency depending on the gain. As described above, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of spatially smoothing each exposure image.

  Further, in the above configuration, the second step may be a step of smoothing a histogram of each exposure image created corresponding to each of the plurality of exposure images before performing the third step. Preferred (claim 16).

  According to this, since the second step is a step of smoothing the histogram of each exposure image created corresponding to each of the plurality of exposure images before performing the third step, this step The smoothed histogram becomes a histogram having different frequencies depending on the gain. In this way, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of smoothing the histogram of each exposure image.

  Since each exposure image is created from one image, it is not necessary to shoot a plurality of images (multi-exposure images) having different exposures, and subject blurring between the plurality of images is prevented. Further, since the frequency of the histogram of each exposure image differs depending on the gain, the entropy obtained from this histogram also differs for each exposure image. Therefore, since the entropy of each exposure image generated from one image can be obtained as the entropy corresponding to each exposure image, and the entropy can be used as a visibility index, for example, this entropy information is used. Thus, it is possible to obtain one image with high image quality in which exposure and contrast are optimized from a plurality of exposed images.

(Embodiment 1)
FIG. 1 is a schematic block diagram illustrating mainly an imaging process of a digital camera 1 which is an example of an imaging apparatus according to the first embodiment. As shown in FIG. 1, the digital camera 1 includes a lens unit 2, an image sensor 3, an amplifier 4, an A / D conversion unit 5, an image processing unit 6, an image memory 7, a control unit 8, a monitor unit 9, and an operation unit 10. I have. The lens unit 2 functions as a lens window for capturing subject light (light image), and an optical lens system (optical axis L of the subject light) for guiding the subject light to the imaging sensor 3 disposed inside the camera body. For example, a zoom lens or a focus lens). The lens unit 2 includes an aperture and a shutter (both not shown) for adjusting the amount of light transmitted through the lens, and the drive of the aperture and shutter is controlled by the control unit 8.

  The image sensor 3 performs photoelectric conversion into image signals of R, G, and B components according to the amount of light of the subject light image formed by the lens unit 2 and outputs the image signal to the subsequent amplifier 4. In the present embodiment, the imaging sensor 3 is a linear sensor included in a general digital camera, that is, an imaging sensor having a photoelectric conversion characteristic composed of one type of linear characteristic with respect to a linear log sensor described later. The digital camera 1 may be configured to be able to shoot a moving image. In this case, it is sufficient to use one frame image (still image) in the moving image. An image obtained by this general linear sensor is referred to as a “linear image” with respect to a “linear / logarithmic image” in the case of a linear log sensor described later.

  The amplifier 4 amplifies the image (video) signal output from the imaging sensor 3 and includes, for example, an AGC (auto gain control) circuit, and adjusts the gain (amplification factor) of the output signal. The amplifier 4 may include a CDS (correlated double sampling) circuit that reduces sampling noise of the image signal as an analog value in addition to the AGC circuit. The gain value for the AGC circuit is set by the control unit 8. The A / D converter 5 converts the analog value image signal amplified by the amplifier 4 into a digital value image signal, and each pixel signal obtained by receiving light at each pixel of the image sensor 3 is, for example, Convert to 12-bit pixel data.

  The image processing unit 6 performs various image processing (digital signal processing) on the image signal obtained by the A / D conversion processing by the A / D conversion unit 5. In this embodiment, there is a main feature point in the dynamic range compression (DR compression) process or the exposure optimization process in the image processing unit 6. Various image processing including processing relating to this feature point in the image processing unit 6 will be described in detail later. The image memory 7 includes a memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores image data obtained by photographing by the image sensor 3, image data after image processing by the image processing unit 6, and the like. To do.

  The control unit 8 includes a ROM that stores each control program, a RAM that temporarily stores data, and a control program that is read from the ROM and executed (central processing unit: CPU). It governs the operation control. The control unit 8 calculates various control parameters and the like required by each unit of the device based on various signals from each unit of the device such as the imaging sensor 3, the image processing unit 6, or the operation unit 10, and transmits the calculated control parameters. Control the behavior. Specifically, the control unit 8 is a base for performing automatic exposure control (AE control), automatic focus control (AF control), white balance (WB) processing, and the like from the image signal captured by the image sensor 3. Evaluation values (control parameters), that is, AE evaluation values, AF evaluation values, WB evaluation values, and the like are calculated. The control unit 8 uses these evaluation values to control imaging operation and zoom with respect to the imaging sensor 3 and the lens unit 2 (aperture and shutter) via a timing generation unit (timing generator) and a driving unit (both not shown). Perform focus drive control and perform image processing control on the image processing unit 6. The control unit 8 also performs display control on the monitor unit 9 and input / output control of image signals to the image memory 7.

  The monitor unit 9 displays a monitor such as an image captured by the image sensor 3 (an image stored in the image memory 7). Specifically, the monitor unit 9 is, for example, a liquid crystal display (LCD) composed of a color liquid crystal display element disposed on the back of the camera, or an electronic viewfinder (EVF) that forms an eyepiece. Finder).

  The operation unit 10 is used to give an operation instruction (instruction input) to the digital camera 1 by a user. For example, various operation switches such as a power switch, a release switch, a mode setting switch for setting various shooting modes, and a new selection switch. (Button) group. For example, when a release switch is pressed (turned on), a subject is imaged by an imaging operation, that is, the imaging sensor 3, and predetermined image processing is performed on the image data obtained by the imaging, and then the image memory 7. A series of operations such as recording in the above are executed.

  Here, details of the configuration and operation of the image processing unit 6 will be described. FIG. 2 is a functional block diagram showing a circuit configuration example of the image processing unit 6. As shown in the figure, the image processing unit 6 includes a basic processing unit 61, a WB correction unit 62, a DR compression unit 63, a color interpolation unit 64, a color correction unit 65, a γ correction unit 66, a color space conversion unit 67, and noise processing. A portion 68 is provided.

  The basic processing unit 61 performs basic image processing with respect to subsequent image processing, in this case, black reference correction processing and FPN correction processing. The black reference correction process is a process of correcting the black level (image signal level in darkness) of the digital image signal input from the A / D conversion unit 5 to a reference value. That is, when the image signal level input from the A / D converter 5 is SD1, and the image signal level in the dark is SD2, the calculation of SD1-SD2 is performed.

  The FPN correction process is a process for removing fixed pattern noise (FPN) of the signal obtained by the black reference correction process. The fixed pattern noise is noise caused by variations in output values of image signals generated by the respective pixels, which are caused by variations in threshold values of the respective pixel circuits of the image sensor 3. The WB correction unit 62 performs WB correction on the input image, that is, color balance correction accompanying white change caused by the color temperature change of the subject light source. Based on the WB evaluation value given from the control unit 8, the WB correction unit 62 performs correction for converting the level of each pixel data of each color component R, G, B so that the color balance of the image signal becomes a predetermined color balance. Do.

  The DR compression unit 63 performs processing (DR compression processing) for compressing the dynamic range (DR) of the input image so that the input image falls within the dynamic range of the display system (monitor or the like), for example. In the present embodiment, the exposure of each part of the image is optimized in the DR compression process. The process for optimizing the exposure of each part of the image is called the exposure optimization process (the DR compression process is substantially performed by performing the exposure optimization process of this embodiment). This DR compression processing (exposure optimization processing) in the DR compression unit 63 will be described in detail later. The DR compression unit 63 may be located at any position as long as it is provided upstream of a later-described γ correction unit 66. In this embodiment, since the memory capacity is small, the DR compression unit 63 is provided before color interpolation (Bayer interpolation), that is, before the color interpolation unit 64.

  The color interpolation unit (Bayer interpolation unit) 64 performs color interpolation processing for interpolating data at pixel positions where the frame image is insufficient for each color component R, G, B of the input image signal. That is, the color filter structure of the image sensor 3 employs a so-called Bayer system in which, for example, G is checkered and R and B are line-sequentially arranged (hereinafter referred to as G-checkered RB line-sequentially arranged). Since the color information is insufficient, the color interpolation unit 64 interpolates pixel data at nonexistent pixel positions using a plurality of actual pixel data.

  The color correction unit 65 performs color correction processing for correcting the hue (color balance; saturation) of the image signals of the color components R, G, and B input from the color interpolation unit 64. The color correction unit 65 has three types of conversion coefficients for converting the level ratio of each image signal of the color component RGB, and corrects the hue of the image data by converting the level ratio with the conversion coefficient according to the shooting scene. .

  The γ correction unit 66 performs gamma correction processing that performs nonlinear conversion on the input image using a predetermined gamma characteristic. Specifically, the γ correction unit 66 sets the level of the image signal for each color component to the monitor unit 9 or an externally output monitor television so that the input image signals of the color components RGB have appropriate output levels. Nonlinear correction is performed using a predetermined gamma correction table (gamma correction LUT) according to the display characteristics (gradation characteristics; nonlinear display characteristics; γ curve) of the display medium (display medium). The gamma correction table corresponding to the display characteristics of the display medium is stored in advance in the γ correction unit 66, for example.

  The color space conversion unit 67 converts the luminance (Y) and the blue color difference (Cb) and the red color difference (Cr) from the RGB display system represented by the RGB of the input image data, that is, the red, green, and blue gradations. ) To perform a color space conversion process for color space conversion to a YCbCr display system (YCC display system). The noise processing unit 68 performs noise removal processing for removing noise components of the input image signal. The noise processing unit 68 performs appropriate noise removal processing, for example, by performing coring processing using a coring coefficient.

  Here, the details of the configuration and operation of the DR compression unit 63 will be described. FIG. 3 is a functional block diagram showing a circuit configuration example of the DR compression unit 63. The DR compression unit 63 includes an image generation unit 20, an entropy calculation unit 30, and a blending unit 40. The image generation unit 20 generates a plurality of images (referred to as multiple exposure images) having different exposures from one input image I (original image I), that is, various predetermined values are generated for the original image I. A plurality of pseudo multiple exposure images are generated by applying a gain of magnification (digital gain; G). Here, for example, images obtained by multiplying the original image I by, for example, 2, 4, and 8 times gains (gain values 2.0, 4.0, and 8.0) are generated as pseudo multiple exposure images M1, M2, and M3. . However, the pseudo multiple exposure images to be generated are not limited to the three images M1, M2, and M3, that is, any number (type) of gain and its value applied to the original image I can be adopted. In this regard, the image generation unit 20 may obtain the gain from the output information (for example, the original image I) of the imaging sensor 3, or the representative value of each block image at the time of AE by the method of Patent Document 2 above. For example, you may obtain | require from pixel value average (block average; Ajk).

  The entropy calculation unit 30 performs a predetermined calculation related to entropy to obtain later-described maximum entropy information used when the generated pseudo multiple exposure images are combined (blended). The entropy calculation unit 30 includes a block division unit 31, a histogram creation unit 32, an entropy calculation unit 33, and a selection unit 34. The block dividing unit 31 divides the original image I into a predetermined number of divisions, that is, j × k blocks (block images). Note that each j × k block image of the original image I corresponds to each block image obtained by dividing each pseudo multiple exposure image into j × k.

  The histogram creating unit 32 creates a histogram for each block image divided by the block dividing unit 31. The histogram creation unit 32 uses the gain information output from the image generation unit 20 (gain values 2.0, 4.0, 8.0; these gain values are represented by “G”), and the horizontal axis gradations. A histogram in which the number (maximum gradation value on the horizontal axis) is reduced from the number of gradations N to N / G is created (normally the horizontal axis of the histogram is N, but in this embodiment it is N / G). . The symbol “N” indicates the number of gradations of the image, and the symbol “/” indicates division. Here, an 8-bit image is handled, and in this case, N = 256 (gradation). However, when there are a plurality of types of gains used when generating each multiple exposure image, the maximum gain is used. For example, when a plurality of gains such as “2.1”, “2.4”, and “2.5” are applied when generating a single multiple exposure image from the original image, The corresponding gain value G is “2.5” which is the maximum value among them. In this sense, hereinafter, the gain value corresponding to each multiple exposure image is set to Gmax instead of G. That is, the histogram creation unit 32 creates a histogram with the horizontal axis of the histogram having the number of gradations N / Gmax.

  Therefore, histograms are created with N / 2.0, N / 4.0, and N / 8.0 on the horizontal axis corresponding to multiple exposure images with gain values of 2.0, 4.0, and 8.0, respectively. . In actual calculation, the histogram creating unit 32 has N / Gmax arrays corresponding to the respective multiple exposure images (pixel values 1, 2, 3,... At the coordinate positions corresponding to the respective columns on the horizontal axis of the histogram). , N / Gmax, where the columns at each coordinate position, that is, the frequency values are all initially set to zero, may be prepared (this array information may be stored in advance in the histogram creation unit 32). ) The i / Gmax-th array that is an integer value is counted up (frequency count; 1 is added) for each pixel value (pixel value i) in each of the j × k block images in the original image I. .

  Specifically, for example, in the case of a multiple exposure image having a Gmax of 2.0, if a certain pixel value i in a certain block image of the original image I is 60, i / Gmax = 60/2 = 30, The count value of the frequency of the 30th array position corresponding to this value 30, that is, the horizontal coordinate, is increased by “1” (the frequency at the pixel value 30 of the histogram with the horizontal axis of N / Gmax = 256/2 is counted up. ) If the pixel value i is 61, i / Gmax = 61/2 = 30.5 (value that cannot be divided; non-integer), but in this case, 30.5 is its fraction (value below the decimal point). The frequency at the 30th array position, which is the same as when the pixel value i is 60, is counted up, with 0.5 being rounded down to 30. Thus, the frequency at each array position corresponding to the i / Gmaxth (integer value) for each pixel value i is counted up. When the pixel value i is 62, i / Gmax = 31, and the frequency at the 31st array position is counted up. Thereby, for example, as shown on the right side of FIG. 4 (when N / Gmax = 256/2), a histogram H4 having a maximum gradation value of 128 (= 256/2) is created.

  Although the case where Gmax is 2.0 has been described here, similarly, for example, when the value of Gmax is other than “4.0”, for example, the value of the pixel value i is “60”, “61”, “62”. ”,“ 63 ”,“ 64 ”, i / Gmax = (60/4 = 15: integer), (61/4 = 15.25; non-integer), (62/4 = 15.5; (Non-integer), (63/4 = 15.75; non-integer), (64/4 = 16; integer), and the pixel values i = 60, 61, 62, 63 are all histograms (N / Gmax). The frequency of the same 15th array position in (with a horizontal axis of 256/4) is counted up, that is, counted four times. On the other hand, when the pixel value i = 64, the frequency at the 16th array position, which is one higher than the 15th, is counted up. It should be noted that the pixel value (i / Gmax) counted at the nth array position on the horizontal axis coordinate with the gradation number N / Gmax may be considered as [X] by a so-called Gaussian symbol (X = i / Gmax). This [X] is the maximum integer not exceeding X ([X] = n, n ≦ X <n + 1). For example, when n = 30, 30 ≦ X <31, and in the case of 30.5, [30.5] = 30th, and in the case of 15.25, 15.5, 15.75 [15.25], [15.5], and [15.75] = 15th respectively.

  In this manner, the histogram creation unit 32 has the horizontal axes of 256/2, 256/4, and 256/8 based on the gain information from the original image I (the pixel value i in each block image of the original image I). J × k histograms are created. Note that the gain information used in the histogram creation unit 32 is not output from the image generation unit 20 to the entropy calculation unit 30, but is the same gain information (when the multiple exposure image is generated in the image generation unit 20) ( (Gain value) may be stored in the histogram creating unit 32.

  Thus, obtaining a histogram with the horizontal axis number of gradations set to N / Gmax (described in the case of 256/2) is the normal horizontal axis gradation number N (256) as shown in FIG. The number of gradations of the histogram H1 of the original image I was considered (assumed) as 128 (= 256/2), that is, two adjacent histograms H1 (Gmax = 2) as indicated by reference numerals 901, 902. The histogram H3 shown in FIG. 4 is changed to the histogram H4 when the corresponding column is considered as one column (for example, one column obtained by averaging the frequencies of the two columns as indicated by dotted lines). It corresponds to that. It can be said that the histograms H1 and H2 shown in FIG. 13 are histograms in which the frequencies are added in the horizontal axis direction. Further, the histogram H4 is interpolated where there is no column in the histogram H2 (denoted as a missing tooth type histogram), that is, interpolated so that the gap between the pillars is filled and there is no gap. It can be said that this is a histogram in which the state is eliminated.

  However, for convenience of explanation, the histogram H3 is a histogram that assumes two columns as one from the histogram H1 of the original image I as described above, and the overall frequency itself of the histogram of the original image I is the histogram H1. (The frequency of a certain column in the histogram H3, for example, the column of the pixel value i = 1 is converted with the same 256 gradations as in the histogram H1, the pixel value i = 1, 2 in the histogram H1). Can be thought of as the average value of the frequency of each column x 2). Further, when the gain value is other than 2.0, for example, when N / Gmax = 256/4, similarly, the histogram H3 (the number of gradations is 128 above) that is considered as one of the four columns of the histogram H1. Changes to a histogram H4 with 64 horizontal gradations.

  This means that the frequency of each column of the histogram corresponding to each multiple exposure image is different depending on the gain. For example, when considering a histogram obtained by reducing the horizontal coordinate in the case of gain values of 2.0, 4.0, and 8.0, the frequency of each column of each histogram, for example, the frequency of a column with a pixel value i = 1 is The values are different from each other. From a different perspective, the frequency of the pixel value i = 1 of the histogram obtained by reducing the horizontal coordinate is compared with the frequency of the column of the pixel value i = 1 at the same column position in the histogram H1 of the original image I. It is a different (high) value depending on the gain.

  In the above description, the width of each column of the histogram is changed for each multiple exposure image generated by changing the gain from one image (original image I) (maximum tone values 256, 128 on the horizontal axis, 64..., That is, the width in the horizontal axis direction of each column of each histogram becomes larger (thicker) as the above becomes 256/2, 256/4, 256/8. However, in all the multiple exposure images, the width of each column of the histogram does not change (becomes a common column width), or the maximum gradation value on the horizontal axis is the same coordinate position and a constant value, for example The above-described “reduction of the horizontal coordinate” may be performed so as not to change at 256 (or 128). In this case, the total width of all the columns from the pixel value 1 to the coordinate position of the maximum pixel value in the histogram in the case of 256/8 (maximum gain) (width in the horizontal axis direction of the entire histogram) is the smallest ( The gradation value at the maximum pixel value is 32).

  Thus, in the past, when the pixel value was gained, the overall frequency (total frequency) did not change, but the overall frequency changed according to the gain, that is, according to the gain as described above. The frequency of each column of the histogram is different. Since the frequency changes in this way also changes the entropy, it is possible to evaluate the visibility of the image by the entropy according to the gain value.

  The entropy calculation unit 33 calculates the entropy of each block image based on the histogram for each block image created by the histogram creation unit 32. The entropy (E) is calculated by the following equation (2).

但し、記号「Ｐ’ｉ」は、各ブロック画像における画素値ｉ×Ｇｍａｘ〜（ｉ＋１）×Ｇｍａｘの範囲の出現確率を示す。この出現確率は、上記ヒストグラムにおける各配列（ｉ）の柱の度数値÷ブロック画像の画素数（ｉの度数／ブロック画素数）である。

However, the symbol “P′i” indicates the appearance probability of the pixel value i × Gmax to (i + 1) × Gmax in each block image. This appearance probability is the frequency value of the column of each array (i) in the histogram divided by the number of pixels of the block image (frequency of i / number of block pixels).

With respect to the appearance probability P′i, for example, the pixel value i of the histogram H4 shown in FIG. 4 is “30” (30 to 31), and the frequency of the shaded column is 50, and the pixel of the block image for which the histogram is obtained If the number is 100 × 100 (10000), the appearance probability P ′ ₃₀ = 50/10000 for this pixel value i = 30. The appearance probability P′i is a pixel value 60 to 62 (i × Gmax to (i + 1)) obtained by multiplying the gain value 2 as a pixel value in the block image (original image I) having 10,000 pixels. × Gmax), that is, the appearance probability of the pixel value 60 and the pixel value 61 (not including the pixel value 62) counted as the frequency of the column corresponding to the gradation range when considered in the histogram of the original image I It is.

  The selection unit 34 sets each entropy of j × k block images corresponding to each gain (each N / Gmax (= 2, 4, 8)) at the same position, that is, the same block arrangement position (j, k). The block images having the highest entropy, that is, the maximum entropy image number Ijk (referred to as maximum entropy information) are selected (determined). Specifically, for example, the maximum of the three entropy values of the three block images corresponding to the gains 2, 4, and 8 at the same (3, 5) (j = 3, k = 5) block arrangement position A block image having the entropy value (information indicating which of the three block images has the maximum entropy value; image number Ijk) is selected. Such a selection operation is performed over all blocks (j × k).

The blending unit 40 synthesizes (blends) the pseudo multiple exposure images M1, M2, and M3 output from the image generation unit 20 based on the maximum entropy information output from the entropy calculation unit 30. That is, only the block images having the maximum entropy at the block arrangement positions (j, k) of the pseudo multiple exposure images M1, M2, and M3 are synthesized. That is, the block arrangement position (2, 1) is the block image of the image M1 (the image number I ₂₁ is “M1” or a predetermined number indicating M1), and the block arrangement position (2, 2) is the image M3 (image number I ₂₂ is a block number having a predetermined number indicating “M3” or M3), and so on, only the block images having the maximum entropy are gathered together so as to be combined into a single image and output as a composite image O. . The blending unit 40 divides each pseudo multiple exposure image M1, M2, and M3 into j × k pieces corresponding to the j × k block images of the original image I before performing the composition processing. Keep it.

  Actually, the blend unit 40 performs the above synthesis using the following equations (3) to (5). That is, the output value (O (x, y)) of a certain pixel of interest (x, y) of the image is the pixel value Ijk (x, y) of the coordinates (x, y) in the block image of the maximum entropy image number Ijk. And a weighted average of the weight Wjk (x, y) obtained from the distance from the target pixel (x, y) to the center (center pixel) of the block image (j, k).

但し、記号「ｎ_ｒ」、「ｎ_ｃ」はｊ×ｋ配列のブロック数（ｊが１〜ｎ_ｒ、ｋが１〜ｎ_ｃ）を示す。また、Ｗｊｋ（ｘ、ｙ）は以下の（４）式で与えられる。

However, the symbols “n _r ” and “n _c ” indicate the number of blocks in the j × k array (j is 1 to n _r , k is 1 to n _c ). Wjk (x, y) is given by the following equation (4).

但し、Ｇｊｋ（ｘ、ｙ）はガウシアンを示しており、以下の（５）式で与えられる。

However, Gjk (x, y) indicates Gaussian and is given by the following equation (5).

但し、記号「ｘ_ｊｋ」、「ｙ_ｊｋ」は、各ブロック画像の中心画素の座標（ｘ_ｊｋ，ｙ_ｊｋ）を示す。また、記号「σ」は標準偏差であって所定のブレンド半径を示している。

However, the symbols “x _jk ” and “y _jk ” indicate the coordinates (x _jk , y _jk ) of the center pixel of each block image. The symbol “σ” is a standard deviation and indicates a predetermined blend radius.

  What is performed in the above arithmetic processing can be rephrased as follows. In other words, each block image has a pixel value Ijk (x, y) of the block image having the maximum entropy, so that the gain (exposure; brightness) is adjusted and the amount of information is large (high image quality with high contrast). A) suitable image. However, if each block image is set to the pixel value Ijk (x, y) as it is, a boundary is generated in each block image and the whole becomes a so-called mosaic image. Therefore, there is no seam (border) between the block images by multiplying this Ijk (x, y) by a weight (weight Wjk (x, y)) considering the distance to the block center. As in the case of (not a mosaic image), it is possible to obtain a single high-quality image (synthesized image O) in which the pixel values of each block image are blended to optimize the overall exposure and contrast.

  The weight in consideration of the distance to the block center is a weight according to the distance from the target pixel to the center pixel of the block image when a target pixel (x, y) is considered in a block image. The ratio of using the value of the pixel value Ijk (x, y) of the block image for the pixel of interest that is closer to the center pixel is greater in weight, that is, the pixel that is closer to the center of the block. On the other hand, the ratio of using the pixel value Ijk (x, y) of the block image decreases as the pixel is farther away, that is, the pixel is closer to the block boundary.

  Note that the pixel for obtaining the distance from a certain pixel of interest (x, y) is not necessarily the pixel at the center of the block (center coordinate), and the position (reference position, reference) in each block when blending Any position (reference position, reference coordinates) or pixel (reference pixel) may be used. In the above description, each block is divided into rectangular areas. However, the shape of the area may not be rectangular, and may be a shape (curved area) divided by a curve, for example. In short, a single image may be divided into partial regions having an arbitrary shape (free shape). However, the reference position is, for example, a spatial or luminance reference position. Further, “centroid” or “centroid” may be obtained as a position corresponding to the center of the partial area. The same applies to each embodiment described later.

  FIG. 5 is a flowchart illustrating an example of an operation related to the DR compression processing (exposure optimization processing) in the DR compression unit 63 of the digital camera 1 according to the first embodiment. First, a captured image (linear image) is acquired by imaging with the imaging sensor 3 (linear sensor) (step S1). Next, the image generation unit 20 applies various gains (for example, gain values 2.0, 4.0, and 8.0) to the original image I to generate a plurality of pseudo multiple exposure images corresponding to the respective gains. (Step S2) The histogram creation unit 32 creates a histogram with the horizontal axis set to N / Gmax for each block image of the original image I divided by the block division unit 31 (step S3). Then, the entropy calculation unit 33 calculates the entropy of each block image (step S4), and the selection unit 34 compares the block images at the same block arrangement position (j, k) with each other, so that the block image having the highest entropy is obtained. (Maximum entropy image number Ijk) is selected (step S5). Then, the blend unit 40 synthesizes (blends) the selected block images with the maximum entropy to obtain one image (synthesized image O). That is, each block unit is obtained from a plurality of pseudo multiple exposure images. One image with optimized exposure is synthesized (step S6).

(Embodiment 2)
In the first embodiment, the number of gradations on the horizontal axis is set to N / Gmax when creating the histogram. Instead, in the second embodiment, the generated pseudo multiple exposure image is spatially converted. Smoothing is performed (histogram creation is performed with the normal number of gradations N). Accordingly, the digital camera 1 in the second embodiment includes a configuration of a DR compression unit 63 a instead of the DR compression unit 63.

  FIG. 6 is a functional block diagram illustrating a circuit configuration example of the DR compression unit 63a in the digital camera 1 according to the second embodiment. The DR compression unit 63a differs from the DR compression unit 63 in an image smoothing processing unit 50, an entropy calculation unit 30a, and a blending unit 40a. The image smoothing processing unit 50 performs spatial smoothing processing (referred to as image smoothing processing) on the pseudo multiple exposure images M1, M2, and M3 generated by the above-described method by the image generating unit 20. . Specifically, predetermined LPF processing is performed on the pseudo multiple exposure images M1, M2, and M3. Note that pseudo multiple exposure images M1 ', M2', and M3 'are obtained by this smoothing process.

  The entropy calculation unit 30a calculates the entropy of each block image in the pseudo multiple exposure images M1 ', M2', and M3 'obtained by the image smoothing processing unit 50. That is, the block dividing unit 31a divides each image M1 ', M2', and M3 'into a predetermined number of divisions, for example, j * k block images as described above. The histogram creation unit 32a creates a histogram of each j × k block image in each image M1 ′, M2 ′, and M3 ′. The histogram created in this way is, for example, a histogram H6 described later. At this time, the horizontal axis of the histogram is assumed to be the number of gradations N of the image as usual (256 in the case of an 8-bit image as described above).

  Based on this histogram, the entropy calculation unit 33a calculates the entropy of each block image using the following equation (6).

但し、記号「Ｐｉ」は、各ブロック画像における画素値ｉの出現確率（上記ヒストグラム作成部３２ａにより作成されたヒストグラムの各配列（ｉ）の柱の度数値÷ブロック画像の画素数）を示す。

However, the symbol “Pi” indicates the appearance probability of the pixel value i in each block image (the frequency value of the column in each array (i) of the histogram created by the histogram creation unit 32a / the number of pixels in the block image).

  The selection unit 34a calculates the entropy of the j × k block images in the images M1 ′, M2 ′, and M3 ′ calculated by the entropy calculation unit 33a between the block images at the same block arrangement position (j, k). And the image number Ijk (maximum entropy information) of the block image having the highest entropy is selected. The maximum entropy information obtained here is input to the blend unit 40a.

  The blending unit 40a performs synthesis (blending processing) of the pseudo multiple exposure images M1 ', M2', and M3 'based on the input maximum entropy information. That is, like the blend unit 40, only the block images with the maximum entropy in each of the images M1 ′, M2 ′, and M3 ′ are collected and combined into one image using the above equations (3) to (5). Output as a composite image O.

  In this embodiment, since the pseudo multiple exposure image is first subjected to the smoothing process and the histogram is created from the pseudo multiple exposure image after the smoothing process, for example, the pseudo multiple exposure image M ′ When considering (when the gain value is 2), as shown in FIG. 7, the histogram H5 of the original image I (corresponding to the histogram H1 of FIG. 13) changes like a histogram H6. In this case, the horizontal axis coordinate value (pixel value) of the histogram H6 is twice that of the histogram H5. However, unlike the histogram H2 shown in FIG. 13, the “tooth missing” of the histogram is eliminated, that is, the interpolation of the histogram. Has been done. The histogram H6 has a smaller height (frequency at each pixel value i) (the height is about half) than the histogram H2, but the total value ( The overall frequency) is larger than the overall frequency of the histogram H2. As indicated by “−Pi · log (Pi)” in the above equation (6), the smaller the appearance probability Pi (the height of the column), the smaller the information in the whole (−Pi · log (Pi)). This is based on the theory that the amount (entropy) increases.

  Also in this case, the frequency of each column of the histogram corresponding to each multiple exposure image is different depending on the gain. That is, in the same manner as described above, for example, when considering the histogram of each multiple exposure image with gain values of 2.0, 4.0, and 8.0, the frequency of each column of each histogram, for example, the pixel value i = 1. The frequency of the column is different from each other, and the frequency of the pixel value i = 1 in this histogram is compared with the frequency of the column of the pixel value i = 1 at the same column position in the histogram H1 of the original image I. Each has a different value depending on the gain.

  As described above, also in the second embodiment, the total frequency (entropy) does not change even when the pixel value is gained by the conventional gain, but the total frequency (entropy) changes according to the gain. Since the frequency of each column of the histogram differs according to the gain, it is possible to evaluate the visibility of the image by entropy according to the gain value.

  FIG. 8 is a flowchart illustrating an example of an operation related to the DR compression process (exposure optimization process) in the DR compression unit 63a of the digital camera 1 according to the second embodiment. First, a captured image (linear image) is acquired by imaging with the imaging sensor 3 (linear sensor) (step S11). Next, the image generation unit 20 applies various gains (for example, gain values 2.0, 4.0, and 8.0) to the original image I to generate a plurality of pseudo multiple exposure images corresponding to the respective gains. (Step S12) The image smoothing processing unit 50 performs image smoothing processing such as LPF processing on each generated pseudo multiple exposure image (Step S13). Next, for each block image obtained by dividing each of the pseudo multiple exposure images (images M1 ′, M2 ′, and M3 ′) after the smoothing process by the block dividing unit 31a by the histogram creating unit 32a, the horizontal axis indicates the gradation. A number N of histograms are created (step S14). Then, the entropy of each block image is calculated by the entropy calculation unit 33a (step S15), and the block images at the same block arrangement position (j, k) in the images M1 ′, M2 ′, and M3 ′ are calculated by the selection unit 34a. By comparison, the block image having the highest entropy (maximum entropy image number Ijk) is selected (step S16). Then, the selected block image having the maximum entropy is synthesized (blended) by the blend unit 40a to obtain one image (synthesized image O). That is, each block unit is obtained from a plurality of pseudo multiple exposure images. A single image with optimized exposure is synthesized (step S17).

(Embodiment 3)
In the first embodiment, the number of gradations on the horizontal axis is set to N / Gmax when creating a histogram. Instead, in the third embodiment, the creation of a histogram is performed with a normal number of gradations N. And smoothing the created histogram (interpolating the tooth loss type histogram). Accordingly, the digital camera 1 according to the third embodiment includes the configuration of the DR compression unit 63 c instead of the DR compression unit 63.

  FIG. 9 is a functional block diagram illustrating a circuit configuration example of the DR compression unit 63c in the digital camera 1 according to the third embodiment. The DR compression unit 63c is different from the DR compression unit 63 in the entropy calculation unit 30c and the blending unit 40c (the image smoothing processing unit 50 is deleted and the histogram smoothing processing unit 35 is added compared to the DR compression unit 63b). ing).

  The entropy calculation unit 30c calculates the entropy of each block image in the pseudo multiple exposure images M1, M2, and M3 obtained by the image generation unit 20. The block dividing unit 31c divides each of the images M1, M2, and M3 into a predetermined number of divisions, for example, j × k block images as described above. The histogram creation unit 32c creates a histogram of each j × k block image in each of the images M1, M2, and M3. The histogram created in this way is, for example, the above-described histogram H6. At this time, the horizontal axis of the histogram is assumed to be the number of gradations N of the image as usual (256 in the case of an 8-bit image as described above).

  The histogram smoothing processing unit 35 performs smoothing processing (referred to as histogram smoothing processing) on the histogram of each block image created by the histogram creating unit 32c. Specifically, the histogram is directly smoothed by performing a moving average process (determining a moving average value) on N (256) arrays (columns) after counting up each array position in the creation of the histogram. To do. By the smoothing by the moving average, as shown in a histogram H6 described later, the gap between the columns is interpolated by a column having a frequency obtained by averaging the frequencies of the columns before and after this. Note that this method is not limited to the method based on the moving average, and any method may be used as long as the histogram is smoothed. For example, the gap between the columns is a column having the same frequency as that of the column before or after the column. An interpolated method may be used.

  The entropy calculation unit 33c calculates the entropy of each block image from the smoothed histogram of each block image using the equation (6) in the same manner as described above. The selection unit 34c compares the entropies of j × k block images in the images M1, M2, and M3 calculated by the entropy calculation unit 33c between the block images at the same block arrangement position (j, k). Thus, the image number Ijk (maximum entropy information) of the block image having the highest entropy is selected. The maximum entropy information obtained here is input to the blend unit 40c.

  The blend unit 40c performs synthesis (blend processing) of the pseudo multiple exposure images M1, M2, and M3 based on the input maximum entropy information. That is, similarly to the blending unit 40, using the above equations (3) to (5), only the block images with the maximum entropy in each of the images M1, M2, and M3 are collected and combined into one image to be a combined image O. Output as.

  In this embodiment, since the obtained histogram itself is smoothed in this way, the histogram H5 of the original image I is also the histogram H6 in this case as in the second embodiment (assuming the gain value is 2). (See FIG. 7), and “tooth missing” such as the histogram H2 is eliminated (interpolated). Therefore, in the case of the third embodiment as well, the total frequency does not change even if the pixel value is gained conventionally, but the total frequency (entropy) changes according to the gain. Since the frequency of each column of the histogram differs depending on the image, it becomes possible to evaluate the visibility of the image by entropy according to the gain value.

  FIG. 10 is a flowchart illustrating an example of an operation related to DR compression processing (exposure optimization processing) in the DR compression unit 63c of the digital camera 1 according to the third embodiment. First, a captured image (linear image) is acquired by imaging with the imaging sensor 3 (linear sensor) (step S21). Next, the image generation unit 20 applies various gains (for example, gain values 2.0, 4.0, and 8.0) to the original image I to generate a plurality of pseudo multiple exposure images corresponding to the respective gains. (Step S22). Each generated pseudo multiple exposure image is input to the entropy computing unit 30c and divided into a plurality of block images by the block dividing unit 31c, and the histogram creating unit 32c has a histogram with the number of gradations N on the horizontal axis for each block image. Is created (step S23). Then, the histogram smoothing processing unit 35 performs a moving average process on the histogram to smooth the histogram of each block image (step S24). Next, the entropy calculation unit 33c calculates the entropy of each block image (step S25), and the selection unit 34c compares the block images at the same block arrangement position (j, k) in the images M1, M2, and M3. Then, the block image with the highest entropy (maximum entropy image number Ijk) is selected (step S26). Then, the selected block image having the maximum entropy is synthesized (blended) by the blend unit 40c to obtain one image (synthesized image O). That is, each block unit is obtained from a plurality of pseudo multiple exposure images. One image with optimized exposure is synthesized (step S27).

  As described above, according to the imaging apparatus (digital camera 1) according to each of the embodiments described above, the generation unit (image generation unit 20) causes a gain (for example, a gain value) that is different from one original image I to the original image I. 2.0, 4.0, 6.0) to generate a plurality of exposure images (multi-exposure images; pseudo-multi-exposure images) having different exposures, and a difference means (histogram creation unit 32, image smoothing processing unit) 50 or the histogram smoothing processing unit 35), the frequency of each column of the histogram (for example, histograms H4 and H6) corresponding to each of the plurality of exposure images is made different according to the gain (gain value), and the calculation means (entropy) The calculation units 33, 33a, 33c) each exposure based on the histograms (histograms H4, H6) in which the frequency of each column differs according to the gain. Entropy corresponding to the image are calculated. That is, since each exposure image is created from one image (original image I), it is not necessary to shoot each of a plurality of exposure images having different exposures, and the occurrence of subject blur between these plurality of images is prevented. The In addition, since the frequency of each column of the histogram of each exposure image differs depending on the gain, the entropy obtained from these histograms (each histogram corresponding to each exposure image) also differs for each exposure image (each Differences in the entropy of the exposed image). Therefore, the entropy of each exposure image generated from one image can be obtained as the entropy corresponding to each exposure image, and the entropy can be used as an indicator of visibility. By using it, it becomes possible to obtain one image with high image quality with appropriate exposure and contrast from a plurality of exposed images.

  Further, an intermittent histogram (for example, histogram H2) in which a gap is left between the columns by multiplying each pixel value of the original image corresponding to each exposed image by a gain by the difference means, Since the interpolation is performed so that there is no gap, it is possible to easily obtain histograms (for example, histograms H4 and H6) of the exposure images having different frequencies depending on the gain by using a simple method called interpolation.

  Further, each exposure image is divided into a plurality of partial images (block images) by the dividing means (block dividing sections 31, 31a, 31c), and the same image of each exposure image is obtained by the selecting means (selecting sections 34, 34a, 34c). The maximum entropy (maximum entropy image number Ijk) is selected from each entropy calculated by the calculation means corresponding to each partial image at the position. Then, since the partial images having the maximum entropy are synthesized by the synthesizing means (blending units 40, 40a, and 40c), each part of the image (each partial image) is obtained from a plurality of exposure images using the information of entropy. It is possible to obtain a single composite image in which the entropy of all of the images is maximized, that is, a high-quality image in which exposure and contrast are optimized.

  Also, the histogram is a histogram in which the horizontal axis coordinates are pixel values corresponding to the gradation values and the vertical axis coordinates are frequencies. The difference means is a creation means (histogram creation section 32) that creates a histogram (for example, histogram H4) corresponding to each of the plurality of exposure images. The difference means creates a horizontal axis when creating the histogram (histogram H4). The coordinates are reduced according to the gain. In other words, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of simply creating a histogram having the horizontal axis coordinates reduced according to the gain.

  Further, since the histogram having horizontal axis coordinates with the maximum gradation value of N / G (N / Gmax) is created by the creation means (histogram creation unit 32), the horizontal axis coordinates reduced according to the gain. It is possible to easily create a histogram (histogram H4) having

  Further, each pixel value i of the original image is divided by the G value by the creating means (histogram creating unit 32), and the frequency of the pixel value on the horizontal axis corresponding to the integer value obtained by the division is counted (counted up). Therefore, it is possible to easily create a histogram (histogram H4) having a horizontal axis coordinate having a maximum gradation value of N / G from the original image I by using a simple frequency counting method. it can.

  Further, since the difference means is an image smoothing means (image smoothing processing unit 50) that spatially smoothes each exposure image before entropy calculation, that is, each exposure image is spatially converted by the image smoothing means. Therefore, the histogram (for example, the histogram H6) created from each of the smoothed exposure images becomes a histogram having a different frequency depending on the gain. As described above, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of spatially smoothing each exposure image.

  Further, since the LPF processing is performed on each exposure image by the image smoothing means, each exposure image can be spatially smoothed easily by a simple method called LPF processing.

  Further, the difference means is a histogram smoothing means (histogram smoothing processing unit 35) for smoothing a histogram (for example, histogram H2) of each exposure image created by the creating means before calculating the entropy. Since the histogram (histogram H2) of each exposure image is smoothed by the smoothing means, this smoothed histogram (for example, histogram H6) becomes a histogram having different frequencies depending on the gain. In this way, the frequency of the histogram corresponding to each exposure image can be varied according to the gain by a simple method of smoothing the histogram of each exposure image.

  In addition, since the histogram smoothing unit smoothes the histogram by performing the moving average process on the histogram created by the creating unit, the histogram can be easily obtained by a simple method of performing the moving average process on the histogram. Can be smoothed.

  Further, the pixel value (pixel value Ijk (x, y)) of the partial image having the maximum entropy and the pixel value (target pixel (x, y)) of the partial image (block image (j, k)) are combined by the combining means. ) And a weight (weight Wjk (x, y)) corresponding to a distance from the position of the partial image (block image (j, k)) to a predetermined reference position (center pixel). Since the partial images are combined, the pixel values of the partial images are blended (entropy / blend processing) so that there is no seam between the partial images (not a mosaic image). It is possible to obtain a single high-quality composite image with an appropriate contrast.

  Further, according to the image processing method of the present invention, in the first step, a plurality of exposure images (multiple exposure images; pseudo-exposures) having different exposures are obtained by applying different gains to the original image I from one original image I. Multiple exposure images) are generated, and in the second step, the frequencies of the respective columns of the histograms (for example, histograms H4 and H6) corresponding to each of the plurality of exposure images are made different according to the gain (gain value). In step 3, the entropy corresponding to each exposure image is calculated based on the histograms (histograms H4 and H6) in which the frequencies of the columns differ according to the gain. That is, since each exposure image is created from one image (original image I), there is no need to shoot each of a plurality of images (multi-exposure images) with different exposures, and subject blurring between the plurality of images is not necessary. Occurrence is prevented. In addition, since the frequency of each column of the histogram of each exposure image differs depending on the gain, the entropy obtained from these histograms (each histogram corresponding to each exposure image) also differs for each exposure image (each Differences in the entropy of the exposed image). Therefore, the entropy of each exposure image generated from one image can be obtained as the entropy corresponding to each exposure image, and the entropy can be used as an indicator of visibility. By using it, it becomes possible to obtain one image with high image quality with appropriate exposure and contrast from a plurality of exposed images.

  Further, when the second step is to create a histogram corresponding to each of the plurality of exposure images, the horizontal axis coordinate of the histogram (for example, histogram H4) having the pixel value corresponding to the gradation value as a coordinate corresponds to the gain. Since the process is reduced, the frequency of the histogram (histogram H4) corresponding to each exposure image is simply set according to the gain by a simple method of creating a histogram having the horizontal axis coordinate reduced according to the gain. Can be different.

  In addition, since the second step is a step of spatially smoothing each exposure image before performing the third step, a histogram (for example, histogram H6) created from each of the smoothed exposure images. ) Becomes a histogram having different frequencies depending on the gain. As described above, the frequency of the histogram (histogram H6) corresponding to each exposure image can be varied according to the gain by a simple method of spatially smoothing each exposure image.

  Furthermore, since the second step is a step of smoothing a histogram (for example, histogram H2) of each exposure image created corresponding to each of the plurality of exposure images before performing the third step. The histogram (for example, histogram H6) smoothed by this process becomes a histogram having different frequencies depending on the gain. As described above, the frequency of the histogram (histogram H6) corresponding to each exposure image can be varied according to the gain by a simple method of smoothing the histogram of each exposure image.

In addition, this invention can take the following aspects.
(A) In each of the embodiments described above, a general linear sensor is used as the image sensor 3. However, when the sensor incident luminance is low (in the dark) as an image sensor capable of photographing in a wide dynamic range (wide DR). A linear characteristic region in which the output pixel signal is linearly converted and output, and a logarithmic characteristic region in which the output pixel signal is logarithmically converted and output when the sensor incident luminance is high (during light). A sensor (referred to as a linear log sensor) having conversion characteristics (referred to as linear / logarithmic characteristics; see the left side of FIG. 11) may be used. An image having linear / logarithmic characteristics obtained by this linear log sensor is expressed as a linear / logarithmic image.

  When the linear log sensor is used, a method for generating a multiple exposure image is different. That is, in this case, first, the logarithmic characteristic in the linear / logarithmic characteristic is converted into a linear characteristic and temporarily unified into an image having a linear characteristic. A plurality of multiple exposure images (see the right side of FIG. 11) are generated by applying a gain (for example, 1.0, 0.5, 0.25 times) with a predetermined magnification to the unified linear characteristic image. However, if the maximum pixel value max (for example, max = 256 for an 8-bit image) is exceeded, clipping is performed. These calculations are performed by the image generation unit 20. In this way, a plurality of multiple exposure images (pseudo multiple exposure images) can be easily generated from an image (original image I; linear / logarithmic image) obtained by the linear log sensor. The subsequent exposure optimization process for the generated multiple exposure image is the same as that in each of the above embodiments.

  (B) In each of the above embodiments, the exposure optimization processing for the original image I is executed in the digital camera 1 (DR compression units 63, 63a, 63c). It is good also as a structure performed in the predetermined | prescribed process part outside one. Specifically, for example, a user interface configured to be able to transmit information using a storage medium such as a direct connection or wireless connection (network connection) with a digital camera 1 using a USB or the like or a memory card is provided. It may be executed by a predetermined host provided, for example, a PC (Personal Computer) or a PDA (Personal Digital Assistant).

1 is a schematic block configuration diagram mainly relating to imaging processing of a digital camera which is an example of an imaging apparatus according to a first embodiment. FIG. It is a functional block diagram which shows the example of 1 circuit structure of the image process part in the said digital camera. It is a functional block diagram which shows the example of 1 circuit structure of DR compression part in the said image processing part. FIG. 6 is a graph for conceptually explaining what kind of histogram the original image changes to as a result of histogram creation in the first embodiment. It is a flowchart which shows an example of the operation | movement regarding DR compression process (exposure optimization process) in DR compression part of the digital camera which concerns on 1st Embodiment. It is a functional block diagram which shows the example of 1 circuit structure of DR compression part in the digital camera which concerns on 2nd Embodiment. FIG. 11 is a graph for conceptually explaining what histogram of an original image changes by image smoothing processing in the second embodiment. It is a flowchart which shows an example of the operation | movement regarding DR compression process (exposure optimization process) in DR compression part of the digital camera which concerns on 2nd Embodiment. It is a functional block diagram which shows the example of 1 circuit structure of DR compression part in the digital camera which concerns on 3rd Embodiment. It is a flowchart which shows an example of the operation | movement regarding DR compression process (exposure optimization process) in DR compression part of the digital camera which concerns on 3rd Embodiment. FIG. 10 is a graph for explaining a method of generating a multiple exposure image when the imaging sensor is a linear log sensor, which is a modification of each of the above embodiments. It is a graph for demonstrating the production | generation (gain) of a pseudo multiple exposure image. It is a graph for demonstrating the production | generation (histogram) of a pseudo multiple exposure image.

Explanation of symbols

1 Digital camera (imaging device)
3 Imaging sensor (photographing means)
6 Image processing units 63, 63a, 63c DR compression unit 20 Image generation unit (generation means)
30, 30a, 30c Entropy calculation unit 31, 31a, 31c Block division unit (division means)
32 Histogram creation part (difference means)
32a, 32c Histogram creation unit 33, 33a, 33c Entropy calculation unit (calculation means)
34, 34a, 34c Selection unit (selection means)
35 Histogram smoothing processing unit (difference means, histogram smoothing means)
40, 40a, 40c Blend part (synthesis means)
50 Image smoothing processing unit (difference means, image smoothing means)
H1, H3, H5 Histogram H2 Histogram (intermittent histogram, histogram according to claims 10 and 16)
H4 histogram (histogram according to claim 1, 4, 5, 13, 14)
H6 histogram (histogram according to claims 1 and 13)

Claims (16)

Generating means for generating a plurality of exposure images with different exposures by applying different gains to the original image from one original image;
Different means for varying the frequency of each column of the histogram corresponding to each of the plurality of exposure images according to the gain;
An imaging apparatus comprising: a calculating unit that calculates entropy corresponding to each exposure image based on each histogram in which the frequency of each column differs according to the gain. The difference means is
Interpolating an intermittent histogram in which there is a gap between the columns by multiplying each pixel value of the original image corresponding to each exposure image by the gain so that there is no gap between the columns. The imaging apparatus according to claim 1. Dividing means for dividing each of the exposed images into a plurality of partial images;
A selection unit that selects the maximum entropy from among each entropy calculated by the calculation unit, corresponding to each partial image at the same position of each exposure image,
The imaging apparatus according to claim 1, further comprising a combining unit that combines the partial images having the maximum entropy. The histogram is a histogram in which a horizontal axis coordinate is a pixel value corresponding to a gradation value and a vertical axis coordinate is a frequency,
The difference means is
Creating means for creating a histogram corresponding to each of the plurality of exposure images,
The imaging apparatus according to claim 1, wherein the horizontal axis coordinates are reduced according to the gain when the histogram is created. When the number of gradations of the original image is N and the gain is G,
The creating means includes
The imaging apparatus according to claim 4, wherein the histogram having the horizontal axis coordinate having a maximum gradation value of N / G is created. However, the symbol “/” indicates division. The creating means includes
6. The imaging according to claim 5, wherein each pixel value of the original image is divided by the G value, and the frequency of the pixel value in the horizontal coordinate corresponding to the integer value obtained by the division is counted. apparatus. The difference means is
The imaging apparatus according to claim 1, wherein the imaging apparatus is an image smoothing unit that spatially smoothes each exposure image before calculating the entropy. The image smoothing means includes
The imaging apparatus according to claim 7, wherein low-pass filter processing is performed on each exposure image. A creation means for creating a histogram of each of the plurality of exposure images;
The difference means is
The imaging apparatus according to claim 1, wherein the imaging apparatus is a histogram smoothing unit that smoothes a histogram of each exposure image created by the creation unit before calculating the entropy. The histogram smoothing means includes:
The imaging apparatus according to claim 9, wherein the histogram is smoothed by performing a moving average process on the histogram created by the creation unit. The synthesis means includes
By calculating the weighted average of the pixel value of the partial image having the maximum entropy and the weight according to the distance from the position of the pixel value of the partial image to the predetermined reference position of the partial image, the partial images The imaging apparatus according to claim 3, wherein: It further comprises photographing means for obtaining the original image by photographing,
The photographing means is a linear log sensor having a photoelectric conversion characteristic composed of a linear characteristic and a logarithmic characteristic,
The generating means unifies the photoelectric conversion characteristics in the original image obtained by the linear log sensor to the linear characteristics, and then applies different gains to the original images unified to the linear characteristics, so that each exposure image The imaging apparatus according to claim 1, wherein: A first step of generating, from one original image, a plurality of exposure images having different exposures by applying different gains to the original image;
A second step of varying the frequency of each column of the histogram corresponding to each of the plurality of exposure images according to the gain;
And a third step of calculating entropy corresponding to each exposure image based on each histogram in which the frequency of each column differs according to the gain. The second step includes
The step of reducing a horizontal axis coordinate of the histogram having a pixel value corresponding to a gradation value as a coordinate when generating a histogram corresponding to each of the plurality of exposure images according to the gain. 14. The image processing method according to 13. The second step includes
The image processing method according to claim 13, wherein the exposure process is a process of spatially smoothing each exposure image before performing the third process. The second step includes
The image processing method according to claim 13, wherein before performing the third step, the step of smoothing a histogram of each exposure image created corresponding to each of the plurality of exposure images.

Images