JP2008085634A - Imaging apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain a high quality image, wherein the entire image is properly exposed without any bias in each part of the image while shortening an operation processing time and reducing costs without photographing a plurality of images of different exposures and without calculating image entropy. <P>SOLUTION: This imaging apparatus comprises: an image dividing means for dividing one basic image into a plurality of partial areas; a feature amount calculating means for calculating a feature amount of the partial area image of each partial area divided by the image dividing means; a partial area correction amount calculating means for calculating a partial area correction amount being a correction amount about lightness in each partial area image on the basis of the feature amount calculated by the feature amount calculating means; a pixel correction amount calculating means for calculating a pixel correction amount being a correction amount for each pixel value of the basic image on the basis of the partial area correction amount calculated by the partial area correction amount calculating means; and an image correcting means for correcting each pixel value of the basic image on the basis of the pixel correction amount calculated by the pixel correction amount calculating means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an image processing method capable of optimizing the exposure of each part of a captured image.

  Conventionally, when shooting with an imaging device such as a digital camera, the brightness of the shooting scene changes greatly, regardless of the shutter speed, etc., the exposure image is biased, that is, overexposed or underexposed in the captured image. May occur.

In this regard, for example, in Non-Patent Document 1, a plurality of images with different exposures (exposure amounts) are acquired by photographing a plurality of times with different exposures (for example, shutter speed), and each part of the image is obtained from the plurality of images. There is known a method for generating a single image in which the exposure of the image is corrected (for example, Gostasby's method; see Non-Patent Document 1). In this method, as shown in FIG. 11, a plurality of shot images having different exposures, for example, four shot images of images a to d are divided into a predetermined number, for example, 16 blocks, and an image of each block (referred to as a block image) Entropy is obtained, and the block at the same position in each photographed image, for example, the four block images in blocks 801 to 804 are compared with each other, and the block image having the highest entropy (information amount; average information amount) is selected. To do. The selected item is an image of a block 805 in an image e described later. Similarly, for the other block regions, the block image having the highest entropy is selected from the four block images. The block images (each having the maximum entropy) selected at each position are combined to create one image e. However, as it is, the image e has a different brightness (brightness) in each block area, in other words, a so-called mosaic image in which the connection between the block images is unnatural. A so-called blurring process (hereinafter referred to as blending or blending process) is performed so as to eliminate the unnatural joint using LPF (Low Pass Filter) or the like, thereby obtaining a natural image e ′.
A. Gostasby, “Fusion of Multi-Exposure Images”, Image and Vision Computing, vol. 23, 2005, pp. 611-618

However, the above conventional methods have the following problems (1) to (3).
(1) It takes time to capture a plurality of images with different exposures. In addition, subject blurring may occur between the plurality of images, resulting in a deterioration in image quality of images created from these.
(2) Since it is necessary to store a plurality of images, a memory having a large capacity is required, resulting in an increase in cost.
(3) Since histogram calculation and logarithmic calculation are required to obtain the entropy of the image, the calculation processing becomes complicated, the program and circuit scale increase, and the cost increases.

  A technique is known in which a plurality of simulated images with different exposures are generated from a single image, and entropy calculation and blending are performed in the same manner as the Gostasby method. Although the problem 1) does not occur, the problems (2) and (3) still remain.

  The present invention has been made in view of the above circumstances, and without taking a plurality of images with different exposures or calculating the entropy of the image, the calculation processing time is shortened, and the memory, program, and circuit scale are reduced. An image pickup apparatus and an image processing method capable of obtaining a high-quality image in which exposure is not biased in each part of the image and the whole is in an appropriate exposure state (brightness) while achieving cost reduction by suppressing an increase in size The purpose is to provide.

  An imaging apparatus according to the present invention includes an image dividing unit that divides one base image into a plurality of partial regions, and a feature amount calculation that calculates a feature amount of the partial region image for each partial region divided by the image dividing unit. A partial region correction amount calculating unit that calculates a partial region correction amount that is a correction amount related to brightness for each partial region image based on the feature amount calculated by the feature amount calculating unit; and the partial region correction amount A pixel correction amount calculation unit that calculates a pixel correction amount that is a correction amount for each pixel value of the base image based on the partial region correction amount calculated by the calculation unit; and a pixel calculated by the pixel correction amount calculation unit And image correction means for correcting each pixel value of the base image based on a correction amount.

  According to the above configuration, one base image is divided into a plurality of partial regions by the image dividing unit, and the feature amount of the partial region image is calculated for each partial region divided by the image dividing unit by the feature amount calculating unit. The In addition, the partial region correction amount calculation unit calculates a partial region correction amount that is a correction amount related to brightness for each partial region image based on the feature amount calculated by the feature amount calculation unit. Then, a pixel correction amount that is a correction amount for each pixel value of the base image is calculated by the pixel correction amount calculation unit based on the partial region correction amount calculated by the partial region correction amount calculation unit. Based on the pixel correction amount calculated by the pixel correction amount calculation means, each pixel value of the base image is corrected.

  Further, in the above configuration, the pixel correction amount calculation unit is configured to set a distance from a predetermined pixel position of the base image to a predetermined reference position in the partial region image with respect to the partial region correction amount of each partial region. It is preferable to calculate the pixel correction amount by integrating a value obtained by weighting using a corresponding weight for at least a partial region in the vicinity of the predetermined pixel in the base image. (Claim 2)

  According to this, the weight corresponding to the distance from the predetermined pixel position of the base image to the predetermined reference position in the partial area image is used for the partial area correction amount of each partial area by the pixel correction amount calculation means. The pixel correction amount is calculated by integrating the weighted values for at least the partial areas in the vicinity of the predetermined pixel in the base image.

  Further, in the above configuration, the feature amount calculating unit calculates any one of an average value, a minimum value, and a maximum value of pixel values in the partial area image and a median value of the pixel value distribution of the partial area image. It is preferable to calculate as an amount. (Claim 3)

  According to this, any one of the average value, the minimum value, the maximum value, and the median value of the pixel value distribution of the partial region image is calculated as the feature amount by the feature amount calculation means. The

  Further, in the above configuration, the partial region correction amount calculation means has a gain for converting the feature amount of each partial region image into a predetermined target value so as to be corrected according to the brightness of each partial region image. It is preferable to calculate the partial area correction amount. (Claim 4)

  According to this, the gain for converting the feature amount of each partial region image into a predetermined target value to be corrected according to the brightness of each partial region image by the partial region correction amount calculation means is the partial region correction amount. Is calculated as

  Further, in the above configuration, the partial region correction amount calculating means includes an offset for converting the feature amount of each partial region image into a predetermined target value to be corrected according to the brightness of each partial region image. It is preferable to calculate the partial area correction amount. (Claim 5)

  According to this, the offset for converting the feature amount of each partial region image into the predetermined target value to be corrected according to the brightness of each partial region image by the partial region correction amount calculation means is the partial region correction amount. Is calculated as

  In the above configuration, it is preferable that the conversion characteristic for converting the feature amount into a predetermined target value is conversion in which a dark portion of the base image is lifted to be bright and a change in a bright portion is suppressed. . (Claim 6)

  According to this, the conversion characteristic for converting the feature amount into the predetermined target value is raised so that the dark part of the base image becomes bright (the brightness or brightness of the dark part is increased), and the change of the bright part is suppressed. Conversion (in which the change in brightness and brightness of the bright part is suppressed).

  Further, in the above configuration, a photoelectric characteristic comprising a linear characteristic that is output by linearly converting an electrical signal with respect to the incident light amount, and a logarithmic characteristic that is output by logarithmically converting the electrical signal with respect to the incident light amount. An imaging unit having conversion characteristics and configured to be able to obtain a linear / logarithmic image by photographing, and for the linear / logarithmic image obtained by the imaging unit, a logarithmic image in the linear / logarithmic image is a linear image. Conversion processing means for performing processing for converting the entire image into linear characteristics by converting the image into linear characteristics, and the image dividing means includes the base image in which the linear / logarithmic image is unified with linear characteristics by the conversion processing means. The unified linear image is preferably divided into the plurality of partial regions. (Claim 7)

  According to this, the imaging means has a linear characteristic in which the electric signal is linearly converted with respect to the incident light quantity and is output, and a logarithmic characteristic in which the electric signal is logarithmically converted with respect to the incident light quantity and output. It is possible to obtain a linear / logarithmic image by photographing, and a logarithm in the linear / logarithmic image with respect to the linear / logarithmic image obtained by the imaging means by the conversion processing means. Processing for converting the image into a linear image and unifying the entire image into linear characteristics is performed. Then, the unified linear image as the base image obtained by unifying the linear / logarithmic image with the linear characteristic by the conversion processing means is divided into a plurality of partial regions by the image dividing means.

  Further, in the above configuration, the partial region correction amount calculation means calculates the partial region correction amount so that a value obtained by multiplying or adding the partial region correction amount to each pixel value of the base image does not exceed a predetermined pixel value. It is preferable to calculate. (Claim 8)

  According to this, the partial region correction amount calculation unit calculates the partial region correction amount so that a value obtained by multiplying or adding the partial region correction amount to each pixel value of the base image does not exceed the predetermined pixel value.

  According to the image processing method of the present invention, the first step of dividing one base image into a plurality of partial areas by the image dividing means, and the partial areas for each partial area divided by the image dividing means Based on the second step of calculating the feature quantity of the image by the feature quantity calculation means and the feature quantity calculated by the feature quantity calculation means, a partial area correction amount that is a correction amount relating to the brightness for each partial area image is determined. Based on the third step calculated by the partial region correction amount calculating means and the partial region correction amount calculated by the partial region correction amount calculating means, a pixel correction amount that is a correction amount for each pixel value of the base image is determined. A fourth step of calculating by the pixel correction amount calculating means, and a fifth step of correcting each pixel value of the base image by the image correcting means based on the pixel correction amount calculated by the pixel correction amount calculating means. Characterized in that it has a. (Claim 9)

  According to this, in the first step, one base image is divided into a plurality of partial regions by the image dividing unit, and in the second step, the partial region image is divided for each partial region divided by the image dividing unit. The feature amount is calculated by the feature amount calculation means. In the third step, the partial region correction amount calculating unit calculates a partial region correction amount that is a correction amount related to brightness for each partial region image based on the feature amount calculated by the feature amount calculating unit. In the fourth step, based on the partial region correction amount calculated by the partial region correction amount calculation unit, a pixel correction amount that is a correction amount for each pixel value of the base image is calculated by the pixel correction amount calculation unit. In the fifth step, each pixel value of the base image is corrected by the image correction unit based on the pixel correction amount calculated by the pixel correction amount calculation unit.

  In the above configuration, in the fourth step, the pixel correction amount calculation unit performs a predetermined region in the partial region image from a predetermined pixel position of the base image with respect to the partial region correction amount of each partial region. A step of calculating the pixel correction amount by accumulating a value obtained by weighting using a weight according to a distance to the reference position of at least a partial region in the base image near the predetermined pixel. preferable. (Claim 10)

  According to this, in the fourth step, the distance from the predetermined pixel position of the base image to the predetermined reference position in the partial area image with respect to the partial area correction amount of each partial area by the pixel correction amount calculation means The pixel correction amount is calculated by integrating the weighted values using the weights corresponding to the values in at least the partial area near the predetermined pixel in the base image.

  According to the imaging apparatus of the first aspect, one base image is divided into a plurality of partial areas, and a feature amount is calculated for each partial area. Further, a partial area correction amount related to brightness is calculated for each partial area image based on the feature amount. Then, a pixel correction amount for each pixel value of the base image is calculated based on the partial region correction amount, and each pixel value of the base image is corrected based on the pixel correction amount, that is, the entire base image is corrected. There is no need to shoot multiple images with different exposures or calculate the entropy of the image, shortening the processing time, and memory (memory usage) and entropy for storing multiple images A high-quality image with no exposure bias (brightness) in each part of the image and proper exposure (brightness) as a whole, while reducing costs by reducing the size of the computer program and circuit scale Can be obtained.

  According to the imaging device according to claim 2, the partial region correction amount is weighted by a weight according to a distance from a predetermined pixel position of the base image to a predetermined reference position in the partial region image, and the weighted portion Pixels used for so-called blending processing that eliminates differences in exposure (brightness) between the partial area images because the area correction amount is obtained by integrating at least a partial area near the predetermined pixel in the base image. The correction amount can be obtained with an easy method and with high accuracy.

  According to the imaging apparatus according to claim 3, any one of an average value, a minimum value, and a maximum value of pixel values (that is, luminance values) in the partial region image and a median value of the pixel value distribution of the partial region image is characterized. Since the configuration is calculated as a quantity, the feature quantity of each partial area image can be easily obtained.

  According to the imaging device of the fourth aspect, since the gain is calculated as the partial region correction amount of each partial region image, the partial region correction amount can be easily obtained.

  According to the imaging device of the fifth aspect, since the offset is calculated as the partial region correction amount of each partial region image, the partial region correction amount can be easily obtained.

  According to the imaging device of the sixth aspect, the conversion characteristic for converting the feature value into the predetermined target value is a conversion in which the dark part of the base image is lifted so that the bright part is brightened and the change of the bright part is suppressed. Therefore, using this conversion characteristic, that is, using the partial region correction amount obtained from this conversion characteristic, the dark part of the base image is lifted to become bright and the change of the bright part is suppressed. It is possible to easily optimize the exposure of each part of the base image. Since the exposure can be optimized based on the conversion characteristics as described above, the degree of freedom of the exposure optimization method, such as how to optimize the exposure, can be set by arbitrarily setting the conversion characteristics. Get higher.

  According to the imaging device of the seventh aspect, the linear / logarithmic image obtained by the imaging means is converted into a unified linear image by the conversion processing means, and this is used as a base image (divided by the image dividing means). Therefore, even when the imaging unit is such that a linear / logarithmic image can be obtained by imaging (that is, an imaging unit having a photoelectric conversion characteristic composed of a linear characteristic and a logarithmic characteristic here; for example, a linear log sensor), The obtained linear / logarithmic image is converted into a unified linear image, and exposure optimization can be performed as in the case of a linear image obtained by a general imaging means.

  According to the imaging device of the eighth aspect, the partial region correction amount is set to a value such that a value obtained by multiplying or adding the partial region correction amount to each pixel value of the base image does not exceed the predetermined pixel value. The output pixel value (luminance value) overflows (saturates) due to the exposure optimization processing in the bright portion of the base image, that is, the region where the lightness is particularly high by setting the predetermined pixel value as the maximum value of the pixel value, for example. This can be prevented or alleviated.

  According to the image processing method of the ninth aspect, one base image is divided into a plurality of partial areas, and a feature amount is calculated for each partial area. Further, a partial area correction amount related to brightness is calculated for each partial area image based on the feature amount. Then, a pixel correction amount for each pixel value of the base image is calculated based on the partial region correction amount, and each pixel value of the base image is corrected based on the pixel correction amount, that is, the entire base image is corrected. There is no need to shoot multiple images with different exposures or calculate the entropy of the image, shortening the processing time, and memory (memory usage) and entropy for storing multiple images A high-quality image with no exposure bias (brightness) in each part of the image and proper exposure (brightness) as a whole, while reducing costs by reducing the size of the computer program and circuit scale Can be obtained.

  According to the image processing method of the tenth aspect, the partial region correction amount is weighted by the weight according to the distance from the predetermined pixel position of the base image to the predetermined reference position in the partial region image. Since the partial area correction amount is integrated for at least the partial area in the vicinity of the predetermined pixel in the base image, the pixel correction amount is used, so that it is used for so-called blending processing that eliminates the difference in exposure (brightness) between the partial area images. The pixel correction amount can be obtained with an easy method and with high accuracy.

(Embodiment 1)
FIG. 1 is a schematic block diagram illustrating mainly an imaging process of a digital camera which is an example of an imaging apparatus according to the first embodiment. As shown in FIG. 1, the digital camera 1 includes a lens unit 2, an image sensor 3, an amplifier 4, an A / D conversion unit 5, an image processing unit 6, an image memory 7, a control unit 8, a monitor unit 9, and an operation unit 10. I have.

  The lens unit 2 functions as a lens window for capturing subject light (light image), and an optical lens system (optical axis L of the subject light) for guiding the subject light to the imaging sensor 3 disposed inside the camera body. For example, a zoom lens, a focus lens, and other fixed lens blocks) that are arranged in series. The lens unit 2 includes an aperture and a shutter (both not shown) for adjusting the amount of light transmitted through the lens, and the drive of the aperture and shutter is controlled by the control unit 8.

  The image sensor 3 performs photoelectric conversion into image signals of R, G, and B components according to the amount of light of the subject light image formed by the lens unit 2 and outputs the image signal to the subsequent amplifier 4. In the present embodiment, the imaging sensor 3 is a linear sensor included in a general digital camera, that is, an imaging sensor having a photoelectric conversion characteristic composed of one type of linear characteristic with respect to a linear log sensor described later. The digital camera 1 may be configured to be capable of moving image shooting. In this case, one frame image (still image) in the moving image shot by the imaging sensor 3 may be used. Note that an image obtained by imaging with this general linear sensor is referred to as a “linear image” in contrast to a “linear / logarithmic image” in the case of a linear log sensor described later.

  The amplifier 4 amplifies the image (video) signal output from the imaging sensor 3 and includes, for example, an AGC (auto gain control) circuit, and adjusts the gain (amplification factor) of the output signal. The amplifier 4 may include a CDS (correlated double sampling) circuit that reduces sampling noise of the image signal as an analog value in addition to the AGC circuit. The gain value for the AGC circuit is set by the control unit 8.

  The A / D converter 5 converts the analog image signal (analog signal) amplified by the amplifier 4 into a digital image signal (digital signal), and receives light at each pixel of the image sensor 3. Each pixel signal obtained in this way is converted into, for example, 12-bit pixel data.

  The image processing unit 6 performs various image processing (digital signal processing) on the image signal obtained by the A / D conversion processing by the A / D conversion unit 5. In this embodiment, there is a main feature point in the dynamic range compression (DR compression) process or the exposure optimization process in the image processing unit 6. Various image processing including processing relating to this feature point in the image processing unit 6 will be described in detail later.

  The image memory 7 includes a memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores image data that has undergone image processing in the image processing unit 6. The image memory 7 has a capacity capable of storing image data for a predetermined frame by, for example, photographing.

  The control unit 8 includes a ROM that stores each control program, a RAM that temporarily stores data, and a control program that is read from the ROM and executed (central processing unit: CPU). It governs the operation control. The control unit 8 calculates various control parameters and the like required by each unit of the device based on various signals from each unit of the device such as the imaging sensor 3, the image processing unit 6, or the operation unit 10, and transmits the calculated control parameters. Control the behavior. Specifically, the control unit 8 is a base for performing automatic exposure control (AE control), automatic focus control (AF control), white balance (WB) processing, and the like from the image signal captured by the image sensor 3. Evaluation values (control parameters), that is, AE evaluation values, AF evaluation values, WB evaluation values, and the like are calculated. The control unit 8 uses these evaluation values to control imaging operation and zoom with respect to the imaging sensor 3 and the lens unit 2 (aperture and shutter) via a timing generation unit (timing generator) and a driving unit (both not shown). Perform focus drive control and perform image processing control on the image processing unit 6. The control unit 8 also performs display control on the monitor unit 9 and input / output control of image signals to the image memory 7.

  The monitor unit 9 displays a monitor such as an image captured by the image sensor 3 (an image stored in the image memory 7). Specifically, the monitor unit 9 is, for example, a liquid crystal display (LCD) composed of a color liquid crystal display element disposed on the back of the camera, or an electronic viewfinder (EVF) that forms an eyepiece. Finder).

  The operation unit 10 is used to give an operation instruction (instruction input) to the digital camera 1 by a user. For example, various operation switches such as a power switch, a release switch, a mode setting switch for setting various shooting modes, and a new selection switch. (Button) group. For example, when a release switch is pressed (turned on), a subject is imaged by an imaging operation, that is, the imaging sensor 3, and predetermined image processing is performed on the image data obtained by the imaging, and then the image memory 7. A series of operations such as recording in the above are executed.

  Here, details of the configuration and operation of the image processing unit 6 will be described. FIG. 2 is a functional block diagram illustrating a circuit configuration example of the image processing unit 6. As shown in the figure, the image processing unit 6 includes a basic processing unit 61, a WB correction unit 62, a DR compression unit 63, a color interpolation unit 64, a color correction unit 65, a γ correction unit 66, a color space conversion unit 67, and noise processing. A portion 68 is provided.

  The basic processing unit 61 performs basic image processing with respect to subsequent image processing, in this case, black reference correction processing and FPN correction processing. The black reference correction process is a process of correcting the black level (image signal level in darkness) of the digital image signal input from the A / D conversion unit 5 to a reference value. That is, when the image signal level input from the A / D converter 5 is SD1, and the image signal level in the dark is SD2, the calculation of SD1-SD2 is performed.

  The FPN correction process is a process for removing fixed pattern noise (FPN) of the signal obtained by the black reference correction process. The fixed pattern noise is noise caused by variations in output values of image signals generated by the respective pixels, which are caused by variations in threshold values of the respective pixel circuits of the image sensor 3.

  The WB correction unit 62 performs WB correction on the input image, that is, color balance correction accompanying white change caused by the color temperature change of the subject light source. Based on the WB evaluation value given from the control unit 8, the WB correction unit 62 performs correction for converting the level of each pixel data of each color component R, G, B so that the color balance of the image signal becomes a predetermined color balance. Do.

  The DR compression unit 63 performs processing (DR compression processing) for compressing the dynamic range (DR) of the input image so that the input image falls within the dynamic range of the display system (monitor or the like), for example. In the present embodiment, the exposure of each part of the image is optimized in the DR compression process. The process for optimizing the exposure of each part of the image is called the exposure optimization process (the DR compression process is substantially performed by performing the exposure optimization process of this embodiment). This DR compression processing (exposure optimization processing) in the DR compression unit 63 will be described in detail later. The DR compression unit 63 may be located at any position as long as it is provided upstream of a later-described γ correction unit 66. In this embodiment, since the memory capacity is small, the DR compression unit 63 is provided before color interpolation (Bayer interpolation), that is, before the color interpolation unit 64.

  The color interpolation unit (Bayer interpolation unit) 64 performs color interpolation processing for interpolating data at pixel positions where the frame image is insufficient for each color component R, G, B of the input image signal. That is, the color filter structure of the image sensor 3 employs a so-called Bayer system in which, for example, G is checkered and R and B are line-sequentially arranged (hereinafter referred to as G-checkered RB line-sequentially arranged). Since the color information is insufficient, the color interpolation unit 64 interpolates pixel data at nonexistent pixel positions using a plurality of actual pixel data.

  The color correction unit 65 performs color correction processing for correcting the hue (color balance; saturation) of the image signals of the color components R, G, and B input from the color interpolation unit 64. The color correction unit 65 has three types of conversion coefficients for converting the level ratio of each image signal of the color component RGB, and corrects the hue of the image data by converting the level ratio with the conversion coefficient according to the shooting scene. .

  The γ correction unit 66 performs gamma correction processing that performs nonlinear conversion on the input image using a predetermined gamma characteristic. Specifically, the γ correction unit 66 sets the level of the image signal for each color component to the monitor unit 9 or an externally output monitor television so that the input image signals of the color components RGB have appropriate output levels. Nonlinear correction is performed using a predetermined gamma correction table (gamma correction LUT) according to the display characteristics (gradation characteristics; nonlinear display characteristics; γ curve) of the display medium (display medium). The gamma correction table corresponding to the display characteristics of the display medium is stored in advance in the γ correction unit 66, for example.

  The color space conversion unit 67 converts the luminance (Y) and the blue color difference (Cb) and the red color difference (Cr) from the RGB display system represented by the RGB of the input image data, that is, the red, green, and blue gradations. ) To perform a color space conversion process for color space conversion to a YCbCr display system (YCC display system).

  The noise processing unit 68 performs noise removal processing for removing noise components of the input image signal. The noise processing unit 68 performs appropriate noise removal processing, for example, by performing coring processing using a coring coefficient.

  Here, the details of the configuration and operation of the DR compression unit 63 will be described. FIG. 3 is a functional block diagram illustrating a circuit configuration example of the DR compression unit 63. As shown in the figure, the DR compression unit 63 includes a block value calculation unit 631, a block gain calculation unit 632, a blend gain calculation unit 633, and a multiplication unit 634. The block average calculation unit 631 divides (divides) the image (input image, original image) I input to the DR compression unit 63 into a plurality of m × n blocks (block images), and each of the divided blocks is divided. A feature amount representing a feature, that is, a value representing each block (referred to as a block value) is calculated. The block value calculation unit 631 calculates Ajk (1 ≦ j ≦ m, 1 ≦ k ≦ n), which is an average (block average) of pixel values (luminance values) in each block, as the block value. However, the block value is not limited to the block average Ajk (average value), and is the minimum value or maximum value of the pixel values in each block, or the median value (so-called median) of the pixel value distribution (luminance distribution) of each block. Also good. In short, the value of Ajk may be a value representing each block (the same applies to the second embodiment).

The block gain calculation unit 632 calculates a correction amount related to lightness (exposure, luminance) in each block, specifically, a gain that converts the block average Ajk calculated by the block value calculation unit 631 into a predetermined value. It is. This conversion is a conversion in which the dark part of the image is brightly raised (increased brightness or brightness), and the bright part has little change in brightness or brightness (a change in brightness or brightness is suppressed). This is indicated by the conversion characteristic 41 (gain characteristic) in FIG. This conversion characteristic 41 is represented by a conversion expression (conversion function) shown in the following expression (1).
y = exp (log (x) * c) * n (1) (Equation (1) is y = x ^ c * n (the symbol “^” is a factorial))
However, the symbols “c” and “n” indicate coefficients. The symbol “*” indicates multiplication (the same applies hereinafter).

  Since the conversion characteristic 41 passes through two points of a point S (Amin, t0) and a point T (Amax, t1) as shown in FIG. 4, the simultaneous values obtained by substituting the coordinate values of these points S and T, respectively. From the equation, two unknowns c and n in the equation (1) can be calculated. Points S and T are control points for controlling the conversion characteristic 41, that is, gain. The x coordinate values at these control points, that is, Amin and Amax, are the minimum value and the maximum value of the block average Ajk, respectively. However, the x-coordinate value at each control point may not be the minimum and maximum values of the block average. For example, the median value in the block where the block average Ajk is minimum and maximum (the pixel value at the central position of each block). ).

  Further, the y coordinate value t0 at the control point S is a value after the above Amin is converted by the conversion characteristic 41, and an appropriate brightness after the gamma correction by the γ correction unit 66 at the subsequent stage (for example, a gradation in an 8-bit image) Value (about 128) (a target level in the conversion). The y coordinate value t1 at the control point T is a value (target level) after the above Amax is converted by the conversion characteristic 41, and is the maximum pixel value (for example, a gradation value 255 in an 8-bit image). If the value of t1 is lowered, the output pixel value (luminance value) may overflow (saturate) after blending (due to the exposure optimization process) in a bright portion of the image, that is, a region where the brightness is particularly high. Since it is prevented or alleviated, it is desirable to lower the value of t1 in an image with many bright parts, that is, to set a value smaller than a predetermined value (threshold) that prevents or alleviates the overflow. The block gain calculation unit 632 calculates the conversion characteristic 41 when calculating the block gain (the same applies to the conversion characteristic 51 described later). Also, the conversion characteristic is not limited to this conversion characteristic 41. For example, an arbitrary one can be adopted by controlling the position of the control point (a point other than the control point may be controlled). is there.

The block gain calculation unit 632 calculates the gain (block gain) Bjk (1 ≦ j ≦ m, 1 ≦ k ≦ n) based on the information of the conversion characteristic 41 set in this way and the block average Ajk. . That is, the block gain Bjk is the ratio (input / output ratio) between the input and the output in the conversion characteristic 41, and is calculated by the following equation (2).
Bjk = exp (log (Ajk) * c) * n / Ajk (2)
However, the symbol “/” indicates division (the same applies hereinafter).

<Modification>
In the above description, the conversion characteristic 41 (characteristic that draws a logarithm that changes logarithmically) shown in FIG. 4 is used as the conversion that brightens the dark part of the image and the bright part has little change. Instead, for example, a conversion characteristic 51 (a characteristic composed of two linear characteristics having different slopes of the low luminance part and the high luminance part) as shown in FIG. 5 may be used. In this case, the conversion characteristic 51 is given by the following equations (3 · 1) and (3 · 2).
y = a0 * x (0 ≦ x ≦ Amin) (3.1)
y = a1 * x + b1 (x> Amin) (3.2)
However, similarly to the conversion characteristic 41, the conversion characteristic 51 passes through the control points P and Q, and the values of Amin, Amax, t0, and t1 are also set similarly to the conversion characteristic 41.

In this case, the block gain Bjk is calculated by the following equations (4 · 1) and (4 · 2).
Bjk = (a0 * Ajk) / Ajk = a0 (0 ≦ x ≦ Amin) (4.1)
Bjk = (a1 * Ajk + b1) / Ajk = a1 + b1 / Ajk (x> Amin) (4.2)
Note that, as described above, not only the conversion characteristic 51 but also any conversion characteristic can be adopted.

  The blend gain calculation unit 633 uses the block gain Bjk calculated by the block gain calculation unit 632 to obtain a blend gain for each pixel of the input image I (each pixel value of the input image I (x, y) described later). A certain blend gain P (x, y) is calculated. The blend gain P (x, y) is calculated by the following equation (5).

但し、Ｗｊｋ（ｘ，ｙ）は、各ブロックにおける各画素に対するウエイト（重み、加重平均）を示しており、以下の（６）式で与えられる。

However, Wjk (x, y) indicates the weight (weight, weighted average) for each pixel in each block, and is given by the following equation (6).

但し、Ｇｊｋ（ｘ，ｙ）はガウシアンを示しており、以下の（７）式で与えられる。

However, Gjk (x, y) indicates Gaussian and is given by the following equation (7).

但し、ｘ_ｊｋ、ｙ_ｊｋは、各ブロック画像の中心座標或いは各ブロックの中心画素の座標（ｘ_ｊｋ，ｙ_ｊｋ）を示す。また、記号「σ」は標準偏差であって所定のブレンド半径を示している。

Here, x _jk and y _jk indicate the center coordinates of each block image or the coordinates (x _jk , y _jk ) of the center pixel of each block. The symbol “σ” is a standard deviation and indicates a predetermined blend radius.

The multiplier 634 multiplies each pixel value I (x, y) of the input image I by the blend gain P (x, y) calculated by the blend gain calculator 633 (multiplies the blend gain for each pixel of the input image). Is. As a result of this multiplication processing, an output image O is output from the multiplication unit 634. The multiplication process in the multiplication unit 634 is expressed by the following equation (8).
O (x, y) = P (x, y) * I (x, y) (8)
However, the pixel value O (x, y) indicates each pixel value of the output image O.

  What is performed in the above arithmetic processing can be rephrased as follows. That is, each block image is multiplied by the block gain Bjk calculated based on the conversion characteristic 41 (51), so that a suitable pixel value (a dark portion is brightly raised and a change in the bright portion is suppressed as described above). Brightness value). However, if the Bjk (fixed value) is directly applied to each pixel I (x, y) of the image I, a boundary is generated in each block image, and the whole becomes a so-called mosaic image. Therefore, instead of simply multiplying Bjk, a value (P (x, y)) obtained by adding a weight (weight) considering the distance to the block center to this Bjk is obtained, and this is obtained for each pixel. By multiplying by I (x, y), there is no boundary between each block image (not a mosaic image), that is, an image in which pixel values of each block image are blended and the overall exposure is optimized (output) Image O) can be obtained.

  The weight in consideration of the distance to the block center is a weight according to the distance from the target pixel to the central pixel of the block when a target pixel is considered in a block (block image). A pixel of interest closer to the center pixel has a greater weight, that is, a pixel closer to the center of the block has a higher proportion (weight) of using the value of Bjk representing this block, The ratio of using the Bjk value of this block decreases as the pixel is farther away, that is, closer to the block boundary.

In the actual calculation, the coordinates (x _jk , y _jk ) of the center pixel of each block (j, k) from a certain pixel I (x, y) (s assumed as the target pixel I (x, y)) in the image I are used. ) Is obtained by Gaussian G _jk (x, y) shown in the above equation (7), and this value is calculated from the _target pixel I (x, y) of all the blocks as shown in the above equation (6). What is divided by the integrated value (total value) of the distance to the center pixel is the weight (W _jk (x, y)) for Bjk of each block (j, k) in a certain pixel I (x, y) of the image I. y)). Further, W _jk (x, y) * Bjk obtained by multiplying this Bjk by the weight W _jk (x, y) is set for all blocks (j = 1 to m, k = 1 to n) (all blocks P (x, y) is obtained by integrating (summing up). In this way, for a certain pixel I (x, y) of the image I, obtaining P (x, y) while considering all the blocks of the image I is so-called blending of the mosaic image. become.

P (x, y) is obtained by integrating (summing) W _jk (x, y) * Bjk for “all” blocks (j = 1 to m, k = 1 to n) as described above. In other words, j may not be the entire range from 1 to m and k may be from 1 to n. For example, a block in the vicinity of the target pixel (a predetermined block in the vicinity) or a block having the target pixel It may be accumulated for partial blocks of image I such as neighboring blocks (values of j and k may be arbitrary). In short, any configuration may be used as long as it is integrated with respect to a predetermined number of blocks (block range) at a predetermined position so that blending can be suitably performed.

  Further, the pixel for obtaining the distance from a certain pixel of interest does not necessarily have to be the pixel at the center of the block (center coordinate), and may be any position (reference position, reference point) that is a reference in each block when blending. Any position (reference position, reference coordinate) or pixel (reference pixel) may be used. In the above description, each block is divided into rectangular areas. However, the shape of the area may not be rectangular, and may be a shape (curved area) divided by a curve, for example. In short, a single image may be divided into partial regions having an arbitrary shape (free shape). However, the reference position is, for example, a spatial or luminance reference position. Further, “centroid” or “centroid” may be obtained as a position corresponding to the center of the partial area. The same applies to the second embodiment described later.

  FIG. 6 is a flowchart illustrating an example of an operation related to the DR compression processing (exposure optimization processing) in the DR compression unit 63 of the digital camera 1 according to the first embodiment. First, a captured image (linear image) is acquired by imaging with the imaging sensor 3 (linear sensor) (step S1). Next, the block value calculation unit 631 divides the linear image (input image I) into blocks, and calculates a block value in each of the divided blocks, that is, a block average Ajk (step S2). Based on the calculated block average Ajk and the conversion characteristic 41 (51) (see FIGS. 4 and 5), the block gain calculation unit 632 calculates the block gain Bjk in each block (step S3). Based on the calculated block gain Bjk, the blend gain calculator 633 calculates the blend gain P (x, y) for each pixel of the input image I (step S4). The multiplication unit 634 multiplies each pixel value I (x, y) of the input image I by the calculated blend gain P (x, y). As a result of this multiplication, the output image O is output from the multiplier 634 (step S5).

(Embodiment 2)
In the first embodiment, a general linear sensor is used as the image sensor 3. However, in the second embodiment, as the image sensor 3 capable of shooting in a wide dynamic range (wide DR), for example, FIG. When the sensor incident luminance is low (when dark), the output pixel signal (output electrical signal generated by photoelectric conversion) is linearly converted and output, and the sensor incident luminance is high A photoelectric conversion characteristic composed of a logarithmic characteristic region in which the output pixel signal is logarithmically converted (during light), that is, a photoelectric conversion characteristic (linear / logarithmic characteristic) in which the low luminance side is linear and the high luminance side is logarithmic. It has a sensor (called linear log sensor).

  Specifically, for example, the imaging sensor 3 adds a logarithmic conversion circuit including a P-type (or N-type) MOSFET to a solid-state imaging device in which photoelectric conversion elements such as photodiodes are arranged in a matrix. A so-called CMOS image sensor is used in which the output characteristics of the solid-state imaging device are logarithmically converted with respect to the amount of incident light by utilizing the sub-threshold characteristics of the MOSFET. However, it is not limited to a CMOS image sensor, and may be a VMIS image sensor, a CCD image sensor, or the like.

  By the way, exposure control (AE control) is performed in the digital camera 1, and the definition regarding this concept will be described. Unlike so-called silver-salt cameras, in imaging devices such as digital cameras and digital movies, the control element for AE control is related to the photoelectric conversion characteristics of the image sensor (by changing the photoelectric conversion characteristics artificially). There is a method of controlling and a method of adjusting the total amount of light reaching the imaging surface of the imaging sensor and the integration time of the photoelectric conversion current after photoelectric conversion. The former is called “dynamic range control”, and the latter is called “exposure amount control”. “Dynamic range control” is executed, for example, by controlling photoelectric conversion characteristics of the image sensor, that is, by controlling a switching point (hereinafter referred to as an inflection point) between a linear characteristic region and a logarithmic characteristic region. The “exposure amount control” is executed, for example, by adjusting the aperture amount of the aperture, adjusting the shutter speed of the mechanical shutter, or controlling the integration time of charges by controlling the reset operation for the image sensor.

  In the present embodiment, a predetermined exposure evaluation value related to exposure control is detected based on the luminance information of the subject obtained from the photographed image by the image sensor 3, and the controller 8 uses the exposure evaluation value to obtain a required exposure. In addition to controlling the amount of exposure based on the photoelectric conversion characteristics to ensure the amount, and using this exposure evaluation value to control the dynamic range based on the photoelectric conversion characteristics to ensure the required dynamic range, Control exposure. Thus, exposure control of the digital camera 1 can be performed by performing exposure amount control and dynamic range control in association with the photoelectric conversion characteristics (linear / logarithmic characteristics) of the image sensor 3 (linear log sensor). Depending on the subject brightness, the subject can be imaged in an optimal exposure state and with a predetermined dynamic range secured.

  The dynamic range control is performed by controlling the position of the inflection point in the linear / logarithmic characteristics. This inflection point is controlled by a predetermined control signal for each pixel circuit of the image sensor 3 by the control unit 8. In the above exposure amount control, the output of the image sensor for a certain subject luminance for exposure amount setting becomes a predetermined target output level in the linear / logarithmic linear characteristic region (such photoelectric conversion characteristics). By performing the exposure amount control using the exposure amount setting value, an appropriate exposure amount (brightness) that can obtain a high-contrast image signal even in a low-luminance subject (dark part) is ensured. .

  Corresponding to the use of the image sensor 3, in the present embodiment, the DR compression unit 63 shown in FIG. 2 has the block configuration shown in FIG. If the DR compression unit shown in FIG. 8 is a DR compression unit 63a, the DR compression unit 63a is similar to the DR compression unit 63 in that it includes a block value calculation unit 631a, a block gain calculation unit 632a, a blend gain calculation unit 633a, and a multiplication unit. 634a and a log linear conversion unit 635 in addition to these.

  The log linear conversion unit 635 converts the logarithmic image of the linear / logarithmic image into a linear characteristic with respect to the input image I, that is, an image (referred to as a linear / logarithmic image) obtained by imaging by the imaging sensor 3 (linear log sensor). This is converted to an image (this conversion is called “log linear conversion”). This is so that image data consisting of two types of characteristics, so-called linear and logarithmic, can be handled in a unified manner with one type of image data of linear characteristics. An image unified with linear characteristics (linear image) is referred to as “unified linear image”. FIG. 8 shows this unified linear image It.

The block value calculation unit 631a divides the unified linear image It obtained by the conversion processing by the log linear conversion unit 635 into a plurality of m × n blocks, like the block value calculation unit 631, and calculates the block average Ajk ( 1 ≦ j ≦ m, 1 ≦ k ≦ n) is calculated. Similar to the block gain calculation unit 632, the block gain calculation unit 632a calculates a gain that converts the block average Ajk calculated by the block value calculation unit 631a into a predetermined value. However, in the present embodiment, the conversion formula 71 in this conversion uses the conversion formulas represented by the following formulas (9.1) and (9.2).
y = x (0 ≦ x ≦ t0) (9.1)
y = a * x + b (x> t0) (9.2)

  FIG. 9 shows the conversion characteristic 71. The conversion characteristic 71 is y = x in the range of 0 ≦ x ≦ t0 as shown in the above equation (9.1). As described above, the linear log sensor is used in the present embodiment as described above, and the dark portion of the image, that is, the linear characteristic region on the low luminance side is already at the appropriate brightness (brightness, brightness level) at the stage of AE control. This is because the dark portion is not changed, that is, gain = 1 (Ajk / Ajk = 1). Note that the conversion characteristic 71 passes through the control points M and N as shown in FIG. The control points M and N correspond to the control points P and Q. However, for the control point M, the x coordinate value is set to t0 which is the same as the y coordinate value so as not to change the dark part (gain = 1). These t0, t1, and Amax are set in the same manner as described above.

In this case, the block gain Bjk is represented by the following expressions (10 · 1) and (10 · 2).
Bjk = 1 (0 ≦ x ≦ t0) (10 · 1)
Bjk = a + b / Ajk (x> t0) (10 · 2)
As described above, any conversion characteristic can be adopted in addition to the conversion characteristic 71.

  The blend gain calculation unit 633a calculates the blend gain P (x, y) in the same manner as the blend gain calculation unit 633. The blend gain P (x, y) is calculated by the following equation (11). Wjk (x, y) in the equation (11) is the same as the equation (6).

但し、Ｃｊｋ（ｘ，ｙ）（単にＣｊｋとも表記する）は、制限付きのブロックゲインであり、以下の（１２）式（条件式）により定義されるものである。

However, Cjk (x, y) (also simply referred to as Cjk) is a limited block gain, and is defined by the following expression (12) (conditional expression).

if (Bjk * It (x, y)> Vmax)
Cjk (x, y) = Vmax / It (x, y)
else
Cjk (x, y) = Bjk (12)
However, Vmax is a maximum pixel value (for example, a gradation value of 255 in an 8-bit image).

  This equation (12) is such that the value obtained by adding the gain Bjk of a certain block to a certain pixel value It (x, y) of the unified linear image It does not exceed the maximum pixel value Vmax that is a threshold value. That is, even if the block gain is multiplied by the pixel value It (x, y), the multiplication value is limited to be equal to or less than the maximum pixel value Vmax (so that it is clipped at Vmax). In the present embodiment, linear / logarithmic images (wide DR images) are handled by a linear log sensor. Therefore, without such a condition (restriction), the gain value in a region with a particularly high brightness (high luminance region) does not become small. For this reason, there is a possibility that the area around this area is saturated, and the area is brightly lit (whiteout). However, the threshold value is not limited to the maximum pixel value Vmax, and an arbitrary value may be set. That is, a value obtained by adding the gain Bjk to the pixel value It (x, y) may be prevented from exceeding a predetermined pixel value.

Similarly to the multiplication unit 634, the multiplication unit 634a multiplies each pixel value It (x, y) of the unified linear image It by the blend gain P (x, y) calculated by the blend gain calculation unit 633a, and outputs the output image O ( x, y). The multiplication process in the multiplication unit 634a is expressed by the following equation (8) ′.
O (x, y) = P (x, y) * It (x, y) (8) ′

  FIG. 10 is a flowchart illustrating an example of an operation related to the DR compression process (exposure optimization process) in the DR compression unit 63a of the digital camera 1 according to the second embodiment. First, a captured image (linear / logarithmic image) is acquired by imaging with the imaging sensor 3 (linear log sensor) (step S11). Next, the log linear conversion unit 635 performs log linear conversion on the linear / logarithmic image to obtain a unified linear image It (step S12). Then, the unified linear image It is divided into blocks by the block value calculation unit 631a, and the block value, that is, the block average Ajk in each of the divided blocks is calculated (step S13). Based on the calculated block average Ajk and the conversion characteristic 71 (see FIG. 9), the block gain calculation unit 632a calculates the block gain Bjk in each block (step S14). Based on the calculated block gain Bjk, the blend gain calculation unit 633a calculates a blend gain P (x, y) for each pixel of the unified linear image It (step S15). The multiplication unit 634a multiplies each pixel value It (x, y) of the unified linear image It by the calculated blend gain P (x, y). As a result of this multiplication, the output image O is output from the multiplier 634a (step S16).

  As described above, according to the imaging apparatus (digital camera 1) according to each of the embodiments described above, one base image I (It) is converted into a plurality of partial regions (by the block value calculation unit 631 (631a) (image dividing unit)). The feature value (Ajk) of the partial region image is calculated for each partial region divided by the image dividing unit by the block value calculation unit 631 (631a) (feature amount calculation unit). Further, the partial gain correction, which is a correction amount related to brightness, for each partial area image based on the feature quantity calculated by the feature quantity calculation means by the block gain calculation section 632 (632a) (partial area correction quantity calculation means). A quantity (block gain Bjk, Cjk or Fjk described later) is calculated. Then, each pixel value (I (x, y), It (x, y)) of the base image is calculated based on the partial area correction amount calculated by the partial area correction amount calculating means by the pixel correction amount calculating means. A pixel correction amount (blend gain P (x, y)) that is a correction amount for the pixel is calculated, and based on the pixel correction amount calculated by the pixel correction amount calculation unit by the multiplication unit 634 (634a) (image correction unit). Thus, each pixel value of the base image is corrected.

  In this way, one base image is divided into a plurality of partial areas, and a feature amount is calculated for each partial area. Further, a partial area correction amount related to brightness is calculated for each partial area image based on the feature amount. Then, a pixel correction amount for each pixel value of the base image is calculated based on the partial region correction amount, and each pixel value of the base image is corrected based on the pixel correction amount, that is, the entire base image is corrected. There is no need to shoot multiple images with different exposures or calculate the entropy of the image, shortening the processing time, and memory (memory usage) and entropy for storing multiple images A high-quality image with no exposure bias (brightness) in each part of the image and proper exposure (brightness) as a whole, while reducing costs by reducing the size of the computer program and circuit scale Can be obtained.

  Further, the pixel correction amount calculating means weights the partial region correction amount of each partial region according to the distance from a predetermined pixel position (target pixel) of the base image to a predetermined reference position in the partial region image. The pixel correction amount is calculated by integrating the weighted values using (Wjk (x, y)) for all the partial areas of the base image. In this way, the partial region correction amount is weighted by the weight according to the distance from the predetermined pixel position of the base image to the predetermined reference position in the partial region image, and this weighted partial region correction amount is used in the base image. A pixel correction amount (P (x, y)) is accumulated at least for a partial area in the vicinity of the predetermined pixel (or in the vicinity of a partial area including the predetermined pixel) (may be integrated for all partial areas). Therefore, the pixel correction amount used for the so-called blending process for eliminating the difference in exposure (brightness) between the partial area images can be obtained with an easy method and with high accuracy.

  Also, any one of an average value, a minimum value, a maximum value of pixel values (that is, luminance values) in the partial region image (block image), and a median value of the pixel value distribution of the partial region image by the feature amount calculation unit Is calculated as a feature value, the feature value of each partial region image can be easily obtained.

  Further, the partial region correction amount calculation means calculates a gain for converting the feature amount of each partial region image into a predetermined target value as a partial region correction amount so as to be corrected according to the brightness of each partial region image. . That is, since the gain (block gain) is calculated as the partial region correction amount of each partial region image, the partial region correction amount can be easily obtained.

  In addition, the conversion characteristic (conversion characteristic 41, 51, or 71) for converting the feature amount into a predetermined target value is a conversion in which the dark part of the base image is raised so that the change in the bright part is suppressed. Therefore, using this conversion characteristic, that is, using the partial region correction amount obtained from this conversion characteristic, the dark part of the base image is lifted to become bright and the change of the bright part is suppressed. It is possible to easily optimize the exposure of each part of the base image. Since the exposure can be optimized based on the conversion characteristics as described above, the degree of freedom of the exposure optimization method, such as how to optimize the exposure, can be set by arbitrarily setting the conversion characteristics. Get higher.

  In addition, the imaging sensor 3 (imaging means) linearly outputs an electrical signal that is linearly converted with respect to the incident light amount, and a logarithm that is output after the electrical signal is converted logarithmically with respect to the incident light amount. And a linear / logarithmic image can be obtained by photographing, and the linearity obtained by the imaging means by the log-linear converter 635 (conversion processing means). The logarithmic image in the linear / logarithmic image is converted into a linear image to unify the entire image into linear characteristics. Then, the image dividing means divides the unified linear image as a base image obtained by unifying the linear / logarithmic image into linear characteristics by the conversion processing means into a plurality of partial regions. Thus, the linear / logarithmic image obtained by the imaging means is converted into a unified linear image by the conversion processing means, and this is used as a base image (divided by the image dividing means). Even when a linear / logarithmic image is obtained by photographing (that is, an imaging means having a photoelectric conversion characteristic composed of a linear characteristic and a logarithmic characteristic here; for example, a linear log sensor), the obtained linear / logarithmic image is obtained. Can be converted into a unified linear image, and exposure optimization can be performed as in the case of a linear image obtained by a general imaging means (linear sensor).

  Further, the partial region correction amount is calculated by the partial region correction amount calculation means so that a value obtained by multiplying or adding the partial region correction amount to each pixel value of the base image does not exceed a predetermined pixel value. The partial region correction amount is set to a value such that a value obtained by multiplying or adding the partial region correction amount to each pixel value of the base image does not exceed the predetermined pixel value. By setting the maximum value or the like, it is possible to prevent or alleviate the overflow (saturation) of the output pixel value (brightness value) due to the exposure optimization process in the bright portion of the base image, that is, the region where the lightness is particularly high. it can.

  Further, according to the image processing method according to the present embodiment, one basic image is divided into a plurality of partial areas by the image dividing unit in the first step, and divided by the image dividing unit in the second step. The feature amount of the partial region image is calculated for each partial region by the feature amount calculation means. In the third step, the partial region correction amount calculating unit calculates a partial region correction amount that is a correction amount related to brightness for each partial region image based on the feature amount calculated by the feature amount calculating unit. In the fourth step, based on the partial region correction amount calculated by the partial region correction amount calculation unit, a pixel correction amount that is a correction amount for each pixel value of the base image is calculated by the pixel correction amount calculation unit. In the fifth step, each pixel value of the base image is corrected by the image correction unit based on the pixel correction amount calculated by the pixel correction amount calculation unit.

  Thus, according to the image processing method, one base image is divided into a plurality of partial areas, and a feature amount is calculated for each partial area. Further, a partial area correction amount related to brightness is calculated for each partial area image based on the feature amount. Then, a pixel correction amount for each pixel value of the base image is calculated based on the partial region correction amount, and each pixel value of the base image is corrected based on the pixel correction amount, that is, the entire base image is corrected. There is no need to shoot multiple images with different exposures or calculate the entropy of the image, shortening the processing time, and memory (memory usage) and entropy for storing multiple images A high-quality image with no exposure bias (brightness) in each part of the image and proper exposure (brightness) as a whole, while reducing costs by reducing the size of the computer program and circuit scale Can be obtained.

  Further, in the fourth step, the pixel correction amount calculation means determines the partial region correction amount of each partial region according to the distance from a predetermined pixel position of the base image to a predetermined reference position in the partial region image. The pixel correction amount is calculated by integrating the values weighted using the weights for all the partial areas of the base image. That is, the partial region correction amount is weighted by a weight according to the distance from a predetermined pixel position of the base image to a predetermined reference position in the partial region image, and the weighted partial region correction amount is at least the relevant value in the base image. Since the pixel correction amount is obtained by integrating the partial regions in the vicinity of the predetermined pixel, the pixel correction amount used for so-called blending processing that eliminates the difference in exposure (brightness) between the partial region images can be obtained in an easy method and It can be obtained with high accuracy.

In addition, this invention can take the following aspects.
(A) The imaging sensor 3 in the second embodiment may not be a linear log sensor. For example, the imaging sensor 3 may have different shutter speeds or aperture values as long as a wide DR image can be obtained (wide DR shooting is possible). It may be a type that synthesizes and outputs multiple images with different exposures obtained by shooting, or knee processing (processing that compresses by applying a predetermined gain only to a bright image portion (high luminance area)) May be the output type. However, the image sensor 3 is not limited to the one that obtains the wide DR image, and any type of image sensor can be used including one that obtains a normal linear image as in the first embodiment.

(B) In the first embodiment, the block gain calculation unit 632 calculates the gain (Bjk) for converting the block average Ajk into a predetermined value. Instead of the “gain”, the “offset (block Offset) ”may be calculated. This offset, like the gain, can be said to be a value for converting the feature amount (block average Ajk) of each block image into a predetermined target value. In this case, the above equation (2) is expressed by the following equation (2) ′.
Fjk = exp (log (Ajk) * c) * n−Ajk (2) ′
However, Fjk indicates a block offset (the same applies hereinafter).

In addition, the above expressions (4-1) and (4-2) are represented by the following expressions (4-1) 'and (4-2)'.
Fjk = a0 * Ajk-Ajk = (a0-1) * Ajk (4.1) '
Fjk = (a1 * Ajk + b1) −Ajk = (a1-1) * Ajk + b1 (4.2) ′
In this case, the blend gain calculation unit 633 calculates a blend offset (this is referred to as a blend offset Q (x, y)) from the block offset Fjk. That is, the value obtained by replacing Bjk on the right side of the above equation (5) with the block offset Fjk is calculated as Q (x, y). Then, the multiplication unit 634 multiplies each pixel value I (x, y) of the input image I by this blend offset Q (x, y), and outputs an output image O.

(C) Similarly to the modification (B), when calculating an offset instead of a gain, the above expressions (10 · 1) and (10 · 2) are expressed by the following (10 · 1) ′, (10 · 2) It will be expressed by the formula.
Fjk = 1 (0 ≦ x ≦ t0) (10 · 1) ′
Fjk = (a * Ajk + b) −Ajk = (a−1) * Ajk + b (10 · 2) ′

Further, the above expression (12) is represented by the following expression (12) ′.
if (Fjk + It (x, y)> Vmax)
Cjk (x, y) = Vmax−It (x, y)
else
Cjk (x, y) = Fjk (12) ′
In this case as well, as in the modification (B), the blend gain calculation unit 633a calculates the blend offset Q (x, y) from the block offset Fjk, and the multiplication unit 634a calculates each pixel value of the unified linear image It. It (x, y) is multiplied by this blend offset Q (x, y), and an output image O is output.

  Thus, since the offset of each block image is calculated by the blend gain calculation unit 633 (633a), a correction amount relating to exposure (brightness, luminance) in each block image can be easily obtained.

  (D) In the first and second embodiments, the blend process for the image (I or It), that is, the process of calculating the blend gain or the blend offset, multiplying the image (I or It), and outputting as the image O is performed. Although it is configured to be executed in the digital camera 1 (DR compression units 63 and 63a), the configuration is not limited thereto, and may be configured to be executed in a predetermined processing unit outside the digital camera 1. Specifically, for example, a user interface configured to be able to transmit information using a storage medium such as a direct connection or wireless connection (network connection) with a digital camera 1 using a USB or the like or a memory card is provided. This blending process may be executed in a predetermined host such as a PC (Personal Computer) or a PDA (Personal Digital Assistant).

1 is a schematic block configuration diagram mainly relating to imaging processing of a digital camera which is an example of an imaging apparatus according to a first embodiment. FIG. It is a functional block diagram which shows the example of 1 circuit structure of the image process part in the said digital camera. It is a functional block diagram which shows the example of 1 circuit structure of DR compression part in the said image processing part. It is a graph which shows an example of the conversion characteristic used at the time of the block gain calculation in the block gain calculation part in the said DR compression part. It is a graph which shows one modification of the conversion characteristic used at the time of the block gain calculation in the block gain calculation part in the said DR compression part. It is a flowchart which shows an example of the operation | movement regarding DR compression process (exposure optimization process) in DR compression part of the digital camera which concerns on 1st Embodiment. It is a graph which shows the photoelectric conversion characteristic (linear / logarithmic characteristic) of the imaging sensor (linear log sensor) in the digital camera which concerns on 2nd Embodiment. It is a functional block diagram which shows the example of 1 circuit structure of DR compression part in the digital camera which concerns on 2nd Embodiment. It is a graph which shows an example of the conversion characteristic used at the time of block gain calculation in the block gain calculation part in DR compression part shown in FIG. It is a flowchart which shows an example of the operation | movement regarding DR compression process (exposure optimization process) in DR compression part of the digital camera which concerns on 2nd Embodiment. It is a schematic diagram explaining the method to produce | generate the one image by which the exposure of each part of the image was correct | amended from the conventional several image.

Explanation of symbols

1 Digital camera (imaging device)
3 Imaging sensor (imaging means)
6 Image processing unit 63, 63a DR compression unit 631, 631a Block value calculation unit (image division unit, feature amount calculation unit)
632, 632a Block gain calculation unit (partial region correction amount calculation means)
633, 633a Blend gain calculation unit (pixel correction amount calculation means)
634, 634a Multiplier (image correction means)
635 log linear converter (conversion processing means)

Claims (10)

Image dividing means for dividing one base image into a plurality of partial areas;
Feature quantity calculating means for calculating the feature quantity of the partial area image for each partial area divided by the image dividing means;
A partial region correction amount calculating unit that calculates a partial region correction amount that is a correction amount relating to brightness for each partial region image based on the feature amount calculated by the feature amount calculating unit;
A pixel correction amount calculating unit that calculates a pixel correction amount that is a correction amount for each pixel value of the base image based on the partial region correction amount calculated by the partial region correction amount calculating unit;
An image pickup apparatus comprising: an image correction unit that corrects each pixel value of the base image based on the pixel correction amount calculated by the pixel correction amount calculation unit.   The pixel correction amount calculation means uses a weight corresponding to a distance from a predetermined pixel position of the base image to a predetermined reference position in the partial region image for the partial region correction amount of each partial region. The imaging apparatus according to claim 1, wherein the pixel correction amount is calculated by integrating a weighted value for at least a partial region in the vicinity of the predetermined pixel in the base image.   The feature amount calculating means calculates, as the feature amount, one of an average value, a minimum value, and a maximum value of pixel values in the partial region image and a median value of pixel value distribution of the partial region image. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized.   The partial region correction amount calculating means calculates, as the partial region correction amount, a gain for converting the feature amount of each partial region image into a predetermined target value to be corrected according to the brightness of each partial region image. The imaging apparatus according to claim 1, wherein   The partial region correction amount calculating means calculates, as the partial region correction amount, an offset for converting the feature amount of each partial region image into a predetermined target value to be corrected according to the brightness of each partial region image. The imaging apparatus according to claim 1, wherein   5. The conversion characteristic for converting the feature quantity into a predetermined target value is a conversion in which a dark part of the base image is lifted to be bright and a change in a bright part is suppressed. 5. The imaging device according to 5. It has a photoelectric conversion characteristic consisting of a linear characteristic in which an electric signal is linearly converted with respect to the incident light amount and output, and a logarithmic characteristic in which the electric signal is converted logarithmically with respect to the incident light amount, and output. Imaging means configured to be able to obtain a linear / logarithmic image by imaging;
Conversion processing means for performing processing for converting the logarithmic image in the linear / logarithmic image into a linear image and unifying the entire image into linear characteristics with respect to the linear / logarithmic image obtained by the imaging means,
The image dividing means divides the unified linear image as the base image, in which the linear / logarithmic image is unified into linear characteristics by the conversion processing means, into the plurality of partial regions. 6. The imaging device according to any one of 6.   The partial region correction amount calculation means calculates the partial region correction amount so that a value obtained by multiplying or adding the partial region correction amount to each pixel value of the base image does not exceed a predetermined pixel value. The imaging device according to claim 7. A first step of dividing one base image into a plurality of partial regions by an image dividing unit;
A second step of calculating a feature amount of the partial region image by the feature amount calculation unit for each partial region divided by the image dividing unit;
A third step of calculating, by the partial region correction amount calculating means, a partial region correction amount that is a correction amount relating to brightness for each of the partial region images based on the feature amount calculated by the feature amount calculating unit;
A fourth step of calculating a pixel correction amount, which is a correction amount for each pixel value of the base image, by the pixel correction amount calculating unit based on the partial region correction amount calculated by the partial region correction amount calculating unit;
An image processing method comprising: a fifth step of correcting each pixel value of the base image by the image correction unit based on the pixel correction amount calculated by the pixel correction amount calculation unit.   In the fourth step, a distance from a predetermined pixel position of the base image to a predetermined reference position in the partial region image with respect to the partial region correction amount of each partial region by the pixel correction amount calculation unit 10. The step of calculating the pixel correction amount by accumulating a value obtained by weighting using a weight according to at least a partial region in the vicinity of the predetermined pixel in the base image. An image processing method described in 1.

Images