JP2011100204A - Image processor, image processing method, image processing program, imaging apparatus, and electronic device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor suitable for converting luminance information on an input image to form an output image different in gradation characteristics, and an image processing method, an image processing program, an imaging apparatus and an electronic device. <P>SOLUTION: An imaging system includes an HDR imaging apparatus and an image processor 20. The image processor 20 includes an illumination component converting section 20a generating luminance image data Y and illumination component data L based on color HDR image data; a limit control section 20b determining a limit value lim of a gain coefficient k corresponding to the brightness of an image based on the average luminance AVE of the luminance image data Y and a threshold of luminance; a coefficient computing section 20c computing compressed illumination component data L' from the illumination component data L, computing a gain coefficient k based on L and L' and setting the gain coefficient k to lim when the computed k exceeds lim; and a range compressing section 20d compressing a dynamic range of the color HDR image data based on the gain coefficient k. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, an imaging apparatus, an electronic apparatus, and the like suitable for converting the luminance of an input image to generate an output image having different gradation characteristics.

Conventionally, many methods based on the retinex theory considering human visual characteristics have been published as methods for compressing the dynamic range of an input image by converting luminance information of the input image. For example, Non-Patent Document 1 concludes from experiments that “the light image entering the eye is represented by the product of the illumination component and the reflection component, but it is the reflection component that shows a strong correlation with the eye”. Based on this theory, when compressing the dynamic range, a method for compressing the dynamic range of the image while preserving image quality degradation by preserving the reflection component of the image and compressing only the illumination component has been proposed.
That is, when the luminance component of the original image is I, the illumination component is L, and the reflection component is R, these relationships can be expressed by the following expression (1).

I = L × R (1)

For example, in Patent Document 1, L ′ obtained by compressing the illumination component L is obtained from the following equation (2), and the obtained component L ′ is multiplied by the reflected component R as shown in the following equation (3), so that The luminance component I ′ of the image is obtained.

L ′ = exp (log (L) × c) × n (2)
I ′ = L ′ × R (3)

In the above equation (2), c and n are coefficients.
As a method of estimating the illumination component L, “L = LPF (I)” obtained by multiplying the luminance component I of the original image by a low-pass filter (for example, a Gaussian filter) is used as the illumination component L, and this L is used as a reflection component R (R = I / L) is obtained. In addition, when these formulas are modified, they can be expressed as the following formula (4).

I ′ = I × (L ′ / L) (4)

That is, the luminance component I ′ of the image with the compressed dynamic range can be expressed as the luminance component I of the original image multiplied by the change amount “L ′ / L” of the illumination component as a coefficient.

In Patent Document 2, the luminance component I of the input image is converted into the luminance component O of the output image in the step “I → Id → f (Id) → Ad → Adb → O”.
Here, Id is obtained by nonlinear space processing of I, f (Id) is obtained by subjecting Id to dynamic range compression processing, and Ad is obtained by dividing f (Id) by Id (Ad = f (Id ) / Id). Further, Adb is obtained by performing blur processing on Ad, and the luminance component O of the output image is expressed by the following expression (5) in the case of a color image (RGB).

Ro = AdbRi
Go = AdbGi
Bo = AdbBi (5)

Further, Id can be expressed in the form of log as shown in the following formula (6).

Id = log (LPF (exp (μ · I))) / μ (6)

As seen in the above prior art, log characteristics are often used in dynamic range compression. This is because the human visual characteristic is close to the log characteristic (when the physical quantity of brightness becomes 10 times, it feels twice as a sensation quantity), so by compressing the highlight part according to the sensation quantity, it looks Natural dynamic range compression becomes possible.

Further, as described above, the compression of the high dynamic range image can be finally transformed into a form in which the luminance component I of the original image is multiplied by a coefficient as in the above equations (4) and (5). . At this time, the coefficient is k, and k includes log characteristics. For example, when the luminance component I (pixel value I) of the dynamic range A is compressed into an image of the dynamic range B (pixel value O = I × k), the coefficient k is “k = L ′ / L” when the processing is simplified. Can be expressed. However, L ′ is obtained from “L ′ = B / A × log (L)”, and L is an illumination component estimated from the pixel value I. Thereby, L ′ and L have a relationship as shown in FIG.
On the other hand, as shown in FIG. 7B, the coefficient k has a steep characteristic that takes a constant value on the bright side and a large value on the dark side.

JP 2006-352825 A JP 2008-511048 A E.H.Land, J.J.McCann, "Lightness and retinex theory", journal of the Optical Society of America, 61 (1), 1 (1971)

However, in the dynamic range compression method in the above-described prior art, for example, as shown in FIG. 7B, the coefficient (gain k) increases as the luminance (A) of the original image is lower (darker). For this reason, there is a possibility that the difference in the low gradation portion becomes small and the contrast on the dark side is lowered.
Therefore, the present invention has been made paying attention to such unsolved problems of the conventional technology. According to some aspects of the present invention, an image processing apparatus, an image processing method, an image processing program, an imaging apparatus, and an electronic apparatus that are suitable for generating luminance information of an input image and generating output images having different gradation characteristics Equipment can be provided.

[Mode 1] In order to achieve the above object, an image processing apparatus according to mode 1 includes:
Image conversion means for converting the luminance of the input image to generate an output image having different gradation characteristics,
The image conversion means has a conversion characteristic of increasing the gain for converting the luminance as the luminance of the input image is lower, and setting an upper limit for the gain, and making the upper limit variable.
With such a configuration, when converting the luminance of the input image, the image conversion means increases the gain for converting the luminance to the set upper limit value as the luminance of the input image is lower. be able to.
Accordingly, by appropriately setting the upper limit value of the gain, it is possible to reduce the image quality deterioration due to the decrease in the contrast of the low gradation portion in the output image after converting the luminance of the input image.

Further, the upper limit of the gain can be changed according to the characteristics of the input image and the desired characteristics of the output image.
As a result, it is possible to obtain an output image having desired image quality characteristics while reducing image quality deterioration due to a decrease in contrast.
Here, the luminance conversion may be performed in units of one pixel or may be performed in units of a plurality of pixels at a specific position. For example, in the case of a plurality of pixels, the average luminance is obtained, an upper limit of the gain corresponding to the average luminance is set, and the luminance is converted according to the upper limit.

[Mode 2] Furthermore, the image processing apparatus according to mode 2 is the same as the image processing apparatus according to mode 1.
The image conversion means converts the luminance of the input image to generate an output image in which the dynamic range of the input image is compressed.
With such a configuration, for example, when the input image is a high dynamic range image, the image conversion means matches the luminance of the input image with the gradation range that can be output by an output device such as a display device. And an output image in which the dynamic range of the input image is compressed to a lower range can be generated.

[Mode 3] Furthermore, the image processing apparatus of mode 3 is the image processing apparatus of mode 1 or 2,
The image conversion means increases the upper limit when the luminance of the input image is lower than a set luminance threshold, and increases the upper limit when the luminance is higher than the luminance threshold. Reduce.
With such a configuration, the upper limit of the gain can be increased when the luminance of the input image is relatively low, and the upper limit of the gain can be decreased when the luminance of the input image is relatively high.

  As a result, for dark images, the upper limit of gain can be adjusted so that the luminance after conversion does not become too low. Therefore, it is possible to generate an image that secures necessary brightness while limiting the low brightness portion of the input image so as not to be too bright. For bright images, the upper limit of gain can be adjusted so that the luminance after conversion does not become too high as compared to dark images. Therefore, it is possible to generate a sharp image with no black floating while limiting the low luminance portion of the input image so that it is not too bright.

[Mode 4] Furthermore, the image processing apparatus according to mode 4 is the image processing apparatus according to any one of modes 1 to 3,
The image conversion means changes the upper limit according to the average luminance of the input image.
With such a configuration, the upper limit can be changed according to the average brightness obtained by averaging the brightness of each pixel constituting the input image, and thus, for example, obtained by imaging a subject in a relatively dark place An input image and an input image obtained by imaging a subject in a relatively bright place can be easily distinguished. For example, the upper limit of the gain is reduced for an input image obtained by imaging in a relatively dark place, and the upper limit of the gain for an input image obtained by imaging in a relatively bright place. Can be raised.
As a result, it is possible to easily set an upper limit corresponding to the brightness of the input image.

[Embodiment 5] Furthermore, the image processing apparatus of Embodiment 5 is the image processing apparatus of any one of Embodiments 1 to 4,
The image conversion means changes the upper limit so that the output image has an image characteristic at the time of conversion by any one of a plurality of types of imaging elements having different conversion methods for converting an incident light amount into an electrical signal. Let
With such a configuration, the image conversion unit gains the output image so that the output image has different image characteristics depending on the conversion method of the various imaging elements when the input image is a captured image captured by the imaging device. The upper limit of can be changed.
As a result, the effect that the input image can be converted into an output image having the image quality characteristics of a desired image sensor can be obtained.

  Here, as the image pickup device, for example, there is a linear conversion type image pickup device (hereinafter referred to as a linear sensor) in which an electrical signal is linearly converted according to the amount of incident light and output. In addition, there is a logarithmic conversion type imaging device (hereinafter referred to as a log sensor) in which an electrical signal is logarithmically converted with respect to an incident light amount for output characteristics of the sensor. In addition, there is a linear logarithm conversion type imaging device (hereinafter referred to as a linear log sensor) that can automatically switch between a linear operation state that is an operation state of a linear sensor and a logarithmic operation state that is an operation state of a log sensor. .

[Mode 6] On the other hand, in order to achieve the above object, an image processing method according to mode 6 includes:
Including an image conversion step of converting the luminance of the input image to generate an output image having different gradation characteristics,
In the image conversion step, as the luminance of the input image is lower, the gain for converting the luminance is increased, and an upper limit is set for the gain, and conversion processing is performed to make the upper limit variable.
Thereby, the same operation and effect as those of the image processing apparatus of aspect 1 can be obtained.

[Mode 7] In order to achieve the above object, an image processing program according to mode 7
Including an image processing program for causing a computer to execute a process having an image conversion step of converting the luminance of an input image to generate an output image having different gradation characteristics;
In the image conversion step, as the luminance of the input image is lower, the gain for converting the luminance is increased, and an upper limit is set for the gain, and conversion processing is performed to make the upper limit variable.
With such a configuration, when the program is executed by a computer, the same operations and effects as those of the image processing apparatus described in the form 1 can be obtained.

[Mode 8] In order to achieve the above object, an imaging apparatus according to mode 8
Imaging means for imaging a subject;
An image processing apparatus according to any one of forms 1 to 5,
The image processing apparatus performs conversion processing using a captured image obtained by imaging with the imaging unit as the input image.
With such a configuration, operations and effects equivalent to those of the image processing apparatus according to Embodiments 1 to 5 can be obtained.
[Embodiment 9] In order to achieve the above object, an electronic apparatus of embodiment 9
The imaging apparatus according to the eighth aspect is provided.
With such a configuration, operations and effects equivalent to those of the imaging device according to mode 8 can be obtained.

1 is a block diagram illustrating a configuration of an imaging system 1 according to the present invention. 1 is a block diagram showing a configuration of an HDR imaging device 10. FIG. 2 is a block diagram illustrating a configuration of an image processing device 20. FIG. 3 is a flowchart showing range conversion processing in the image processing apparatus 20. (A) is a figure which shows an example of the relationship between the average luminance AVE of the color HDR image data I, and the limit value lim of the gain coefficient k, and (b) is the gain coefficient k and the illumination component data L. It is a figure which shows an example of a relationship. (A) and (b) are examples of the relationship between the luminance of the color HDR image data I and the luminance of the color LDR image data O when the limit value lim is set to “5” and “0.25”. FIG. (A) is a figure which shows an example of the relationship between the illumination component in image data, and the illumination component after compression, (b) is a figure which shows an example of the relationship between the illumination component and the gain coefficient k for brightness | luminance conversion. It is.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. 1 to 6 are diagrams illustrating embodiments of an imaging apparatus, an imaging method, and an electronic apparatus according to the present invention.
First, the configuration of an imaging system according to the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing a configuration of an imaging system 1 according to the present invention. 2A is a block diagram showing the internal configuration of the HDR imaging device 10, FIG. 2B is a schematic diagram showing the internal configuration of the image sensor 11, and FIG. It is a block diagram which shows the internal structure. FIG. 3 is a block diagram showing the internal configuration of the image processing apparatus 20.

As illustrated in FIG. 1, the imaging system 1 includes an HDR imaging device 10 and an image processing device 20.
As illustrated in FIG. 2A, the HDR imaging device 10 includes an imaging element 11, memories 12 to 14, and an image synthesis unit 15.
The imaging element 11 has a function of imaging a subject with three or more types of exposure amounts. As shown in FIG. 2B, the lens 11a, the microlens 11b, the color filter array 11c, and the HDR sensor 11d It is comprised including.

Hereinafter, for convenience of explanation, the imaging device 11 performs imaging with exposure times T1, T2, and T3 (T1 <T2 <T3) corresponding to three types of exposure amounts L1, L2, and L3 (L1 <L2 <L3). The configuration will be described as follows.
The lens 11a collects reflected light from the subject and guides it to the microlens. There are types such as a single focus lens, a zoom lens, and an auto iris lens depending on the imaging conditions.

The microlens 11b collects light transmitted through the lens on each sensor cell (pixel) of the sensor cell array included in the HDR sensor.
The color filter array 11c separates light having a wavelength corresponding to one predetermined type of color element from light transmitted through the microlens 11b and makes the separated light incident on each corresponding pixel (hereinafter referred to as CF). Part) and at least the number of pixels.

The HDR sensor 11d controls the exposure times T1 to T3 by an electronic shutter system and outputs three types of digital image data having different exposure amounts.
Specifically, the HDR sensor 11d includes a reference timing generator, a scanning line scanner, a sensor cell array, and a horizontal transfer unit (not shown).
The reference timing generator generates a reference timing signal based on the vertical synchronization signal and the horizontal synchronization signal, and outputs the reference timing signal to the scanning line scanner.

The scanning line scanner generates a reset line selection signal that enables a line to be reset based on various signals from the reference timing generator. Then, the generated reset line selection signal is output to the sensor cell array.
Further, the scanning line scanner generates a readout line selection signal that enables the line on which charge accumulation for the set exposure time has been performed as a readout line of the pixel signal after resetting. The generated read line selection signal is output to the sensor cell array.

  The sensor cell array includes a light receiving region having a configuration in which a plurality of sensor cells (pixels) including light receiving elements (photodiodes, etc.) are arranged in a two-dimensional matrix, which is configured using CMOS technology. On the other hand, the address line, the reset line, and the readout line are connected in common. Then, various drive signals (selection signals) are transmitted to the sensor cells constituting each line through the three control lines, and when the address line and the readout line become valid, the accumulated charge (pixel signal) is transmitted through the signal line. Transfer (output) to the horizontal transfer unit.

The horizontal transfer unit performs A / D conversion on pixel signal (analog signal) data read from each pixel of the sensor cell array, and sequentially outputs the data serially to the memories 12 to 14 in line units.
The memories 12 to 14 temporarily store the image data corresponding to the three types of exposure amounts L1 to L3 obtained by imaging the subject with a plurality of different exposure times T1 to T3 in the image sensor 11, and take synchronization. It serves as a buffer to be output to the image composition unit 15.

As shown in FIG. 2B, the image composition unit 15 includes an HDR signal processing unit 15a and a frame memory 15b.
Although not shown, the HDR signal processing unit 15a includes a preprocessing unit, a synthesis processing unit, a color processing unit, and a memory interface (hereinafter referred to as a memory IF).
The preprocessing unit performs fixed pattern noise removal processing, clamping processing, and the like on the pixel signals (digital pixel data S1 to S3) from the memories 12 to 14.

The composition processing unit performs HDR composition processing based on the image data from the memories 12 to 14 and the exposure ratio information. Then, the image data after the HDR synthesizing process is stored as HDR image data in the frame memory 15b via the memory IF.
The color processing unit performs color interpolation processing using the HDR pixel data (pixel data of the pixel to be processed) obtained by combining and the HDR pixel data around the pixel to be processed. That is, processing (color signal processing) for generating color signals (data) defined in the RGB color space is performed for each point of the image. Specifically, the color processing unit converts the HDR pixel data into color HDR pixel data corresponding to each color element of RGB for each pixel by color interpolation processing. Note that color HDR image data is composed of color HDR pixel data for one frame.

The memory IF has a function of arbitrating data writing to and data reading from the frame memory 15b. Specifically, these mediations are performed so that reading and writing are normally performed in response to data read requests and write requests from the preprocessing unit, the synthesis processing unit, and the color processing unit. Then, data read processing from the frame memory 15b and data write processing to the frame memory 15b are executed in response to requests from the components.
The frame memory 15b is a memory that stores each color HDR pixel data constituting the color HDR image data after the synthesis in the HDR signal processing unit 15a in a frame unit (one image unit) in an address area corresponding to each pixel position.

Next, the internal configuration of the image processing apparatus 20 will be described with reference to FIG.
As shown in FIG. 3, the image processing apparatus 20 includes an illumination component conversion unit 20a, a limit control unit 20b, a coefficient calculation unit 20c, a range compression unit 20d, an inverse gamma processing unit 20e, and a storage unit 20f. Consists of including.

  The illumination component conversion unit 20a includes color HDR pixel data Ir (x, y) corresponding to red (R), green (G), and blue (B) constituting color HDR image data input from the HDR imaging device 10, It has a function of converting Ig (x, y) and Ib (x, y) into luminance pixel data Y (x, y). The luminance image data Y is composed of luminance pixel data Y (x, y) for one image. Further, (x, y) is a two-dimensional coordinate indicating the position of the pixel. For example, the upper left corner of the image is the coordinate with the origin (x, y) = (0, 0).

In the present embodiment, the illumination component conversion unit 20a performs the color HDR pixel data Ir (x, y), Ig (x, y), Ib (x, y) of the color HDR image data I according to the following equation (7). Is converted into luminance pixel data Y (x, y).

Y (x, y) = 0.3 × Ir (x, y) + 0.6 × Ig (x, y) + 0.1 × Ib (x, y) (7)

Further, the illumination component conversion unit 20a has a function of performing a blurring process on the luminance image data Y and converting the luminance image data Y into the illumination component data L.

In the present embodiment, the illumination component conversion unit 20a performs a blurring process using a known Gaussian filter according to the following equation (8), and converts each luminance pixel data Y (x, y) of the luminance image data Y into an illumination component. Conversion into data L (x, y).
Here, since the illumination component is composed of components having a relatively low frequency, the illumination component is extracted from the luminance component by applying a low-pass filter such as a Gaussian filter.

L (x, y) = Gauss (Y (x, y)) (8)

The illumination component conversion unit 20a outputs the luminance image data Y to the limit control unit 20b, and outputs the illumination component data L to the coefficient calculation unit 20c.

  The limit control unit 20b has a function of calculating the average luminance AVE based on the luminance image data Y from the illumination component conversion unit 20a. Further, it has a function of determining the limit (upper limit) of the gain coefficient k for luminance conversion when compressing the dynamic range of the color HDR image data I based on the calculated average luminance AVE. The limit control unit 20b outputs the determined limit information to the coefficient calculation unit 20c.

In the present embodiment, the limit control portion 20b, the average brightness AVE is the case of luminance higher than the threshold value Y _H of the high-luminance side of the luminance set in advance, it determines a limit value to be lower than normal. Further, when the average luminance AVE is lower than the preset lower luminance threshold Y _L , the limit value is determined to be higher than normal. When the average luminance AVE is a value between the high luminance side and low luminance side threshold values Y _H and Y _L , the normal limit value is determined.

Further, in the present embodiment, the limit control unit 20b determines the limit value so that the gradation characteristic of the output image becomes the gradation characteristic of the linear sensor when the average luminance AVE is relatively high when determining the limit value. To decide. In addition, when the AVE has a relatively low luminance, the limit value is determined so that the gradation characteristic of the output image becomes the gradation characteristic of the linear log sensor. When AVE is in the normal luminance range, the limit value is determined so that the gradation characteristic of the output image becomes the gradation characteristic of the log sensor.
The coefficient calculation unit 20c has a function of calculating a gain coefficient based on the illumination component data L from the illumination component conversion unit 20a.
In the present embodiment, the coefficient calculator 20c first compresses the illumination component from the dynamic range A to the dynamic range B (A> B) according to the following equation (9) using the illumination component data L. Compressed illumination component data L ′ is generated.

L ′ (x, y) = B / A × (A × log (L (x, y)) / log (A))
... (9)

Next, the coefficient calculation unit 20c uses the illumination component data L and the compressed illumination component data L ′ to perform color HDR pixel data Ir (x, y), Ig corresponding to each pixel according to the following equation (10). A gain coefficient k (x, y) for luminance conversion of (x, y) and Ib (x, y) is calculated.

k (x, y) = L ′ (x, y) / L (x, y) (10)

Further, the coefficient calculation unit 20c compares the calculated gain coefficient k (x, y) with the limit value lim acquired from the limit control unit 20b, and when k (x, y) is equal to or less than the limit value lim, The calculated k (x, y) is left as it is. On the other hand, if k (x, y) exceeds the limit value lim, k (x, y) = lim is set.

Then, the coefficient calculation unit 20c uses the value calculated by the above equation (10) or the limit value lim as the gain coefficient k (x, y), the color HDR pixel data Ir (x, y), Ig (x, y). ) And Ib (x, y) as a gain coefficient k (x, y) and output to the range compression unit 20d.
Although the above equation (9) is used for compression of the illumination component, the present invention is not limited to this compression method, and a compression method based on the inverse gamma characteristic shown in the following equation (11) may be used.

L ′ (x, y) = B / A × (A × (L (x, y) / A) ^0.4 ) (11)

  The range compression unit 20d determines the luminance values indicated by the color HDR pixel data Ir (x, y), Ig (x, y), and Ib (x, y) of the color HDR image data I acquired from the HDR imaging device 10, It has a function of multiplying the gain coefficient k (x, y) acquired from the coefficient calculation unit 20c. As a result, the dynamic range A of the color HDR image data is compressed to the dynamic range B. For example, when compressing a 10-bit dynamic range (0 to 1023) to an 8-bit dynamic range (0 to 255), “B / A” in the above equation (11) is “255/1023 = ≈0.25. "

Specifically, the range compression unit 20d performs color HDR pixel data Ir (x, y), Ig (x, y), Ib (x) of the color HDR image data I that is input image data according to the following equation (12). , Y) is range-compressed. This range-compressed color HDR pixel data (hereinafter referred to as color LDR pixel data) Or (x, y), Og (x, y), Ob (x, y) constitutes color LDR image data O. .

Or (x, y) = Ir (x, y) × k (x, y)
Og (x, y) = Ig (x, y) × k (x, y)
Ob (x, y) = Ib (x, y) × k (x, y) (12)

Further, the range compression unit 20d outputs the color LDR image data O obtained by the range compression to the inverse gamma processing unit 20e.
The inverse gamma processing unit 20e has a function of performing inverse gamma processing on the color LDR image data O acquired from the range compression unit 20d. Further, the inverse gamma processing unit 20e stores the color LDR image data O after the inverse gamma processing in the storage unit 20f.
Here, the reverse gamma processing is, for example, the reverse gamma “1 / γ = 1 / 2.2” so that the halftone is corrected when the image is viewed on a display device with a gamma “γ = 2.2”. The process is to apply the correction at “2≈0.45”.

The storage unit 20f includes a frame memory for storing the color LDR image data O, and stores the color LDR image data O after the inverse gamma processing in the frame memory in response to a write request from the inverse gamma processing unit 20e. It has a function to do.
Note that the image processing apparatus 20 may be configured such that each of the above-described components is entirely configured by hardware such as an electric circuit, or may include a processor such as a DSP as a component, and a part of processing may be realized by software. Good.

Next, the flow of range conversion processing in the image processing apparatus 20 will be described with reference to FIG. Here, FIG. 4 is a flowchart showing the range conversion processing in the image processing apparatus 20.
When the range conversion process is started in the image processing apparatus 20, first, the process proceeds to step S100 as shown in FIG.
In step S100, the image processing apparatus 20 determines whether or not the HDR imaging apparatus 10 has started imaging. If it is determined that the imaging has started (Yes), the process proceeds to step S102; otherwise (No). Repeats the determination process until shooting is started.

When the process proceeds to step S102, the image processing apparatus 20 determines whether or not the color HDR image data I has been acquired from the HDR imaging apparatus 10, and when it is determined that it has been acquired (Yes), the process proceeds to step S104. If not (No), the determination process is repeated until acquisition.
When the process proceeds to step S104, the illumination component conversion unit 20a converts the color HDR image data I (Ir, Ig, Ib) acquired in step S102 into the luminance image data Y using the above equation (7). . Then, the luminance image data Y obtained by the conversion is output to the limit control unit 20b, and the process proceeds to step S106.

In step S106, the illumination component conversion unit 20a performs a blurring process using the Gaussian filter shown in the above equation (8) on the luminance image data Y obtained by the conversion in step S104, and the illumination component data L is obtained. Generate and move to step S108.
In step S108, the limit control unit 20b calculates the average luminance AVE based on the luminance image data Y acquired from the illumination component conversion unit 20a, and proceeds to step S110.

In step S110 determines, at limit control unit 20b, compares the threshold value Y _H of the average brightness AVE and the high luminance side, AVE is whether greater than Y _H. If it is determined that “AVE> Y _H ” (Yes), the process proceeds to step S112, and if not (No), the process proceeds to step S128.
When the process proceeds to step S112, the limit control unit 20b determines that the limit value lim of the gain coefficient k is “lim = lim _S ”, and outputs the determined lim information to the coefficient calculation unit 20c. Migrate to

In step S114, the coefficient calculation unit 20c selects unprocessed data L (x, y) of the range compression process from the illumination component data L acquired from the illumination component conversion unit 20a. Then, by using the selected illumination component data L (x, y), compressed illumination component data L ′ (x, y) obtained by compressing the range of the illumination component is generated by the above formula (9) or (11). The process proceeds to step S116.
In step S116, the coefficient calculation unit 20c calculates the gain coefficient k (x, y) corresponding to each pixel of the color HDR image from the illumination component data L and the compressed illumination component data L ′ using the above equation (10). Then, the process proceeds to step S118.

  In step S118, the coefficient calculation unit 20c compares the gain coefficient k (x, y) calculated in step S116 with the limit value lim acquired from the limit control unit 20b. Then, it is determined whether or not k (x, y) is larger than lim. If it is determined that “k (x, y)> lim” (Yes), the process proceeds to step S120, and otherwise ( No) outputs the value of k (x, y) as it is to the range compressor 20d, and proceeds to step S122.

When the process proceeds to step S120, the coefficient calculation unit 20c changes “k (x, y) = lim” and outputs k (x, y) whose value is changed to lim to the range compression unit 20d. The process proceeds to step S122.
In step S122, the range compression unit 20d uses the k (x, y) acquired from the coefficient calculation unit 20c, and the corresponding color HDR pixel data Ir (x, y), Ig (x , Y) and Ib (x, y) are compressed. Then, the color LDR pixel data Or (x, y), Og (x, y), Ob (x, y) after range compression is output to the inverse gamma processing unit 20e, and the process proceeds to step S124.

In step S124, the image processing apparatus 20 determines whether or not processing has been completed for all pixels for one image. If it is determined that the processing has ended (Yes), the process proceeds to step S126; No) moves to step S114.
When the process proceeds to step S126, the image processing apparatus 20 determines whether or not the shooting process of the HDR imaging apparatus 10 is completed. When it is determined that the imaging process is completed (Yes), the series of processes is ended. If not (No), the process proceeds to step S102.

On the other hand, if the luminance average AVE is equal to or lower than the high luminance side threshold Y _H in step S110 and the process proceeds to step S128, the limit average control unit 20b sets the luminance average AVE to the low luminance side threshold Y _L (Y _H > Y _L ) to determine whether it is smaller. If it is determined by this determination that “AVE <Y _L ” (Yes), the process proceeds to step S130. If not (No), the process proceeds to step S132.
When the process proceeds to step S130, the limit control unit 20b determines that the limit value lim of the gain coefficient k is “lim = lim _L ”, and outputs the determined lim information to the coefficient calculation unit 20c. Migrate to

In step S128, when the luminance average AVE is “Y _H ≧ AVE ≧ Y _L ” and the process proceeds to step S132, the limit control unit 20b sets the limit value lim of the gain coefficient k to “lim = lim _M Decided. Then, the determined lim information is output to the coefficient calculation unit 20c, and the process proceeds to step S114.
Note that lim _S , lim _M , and lim _L have a relationship of “lim _S <lim _M <lim _L ”.

Next, the operation of the present embodiment will be described with reference to FIGS.
Here, FIG. 5A is a diagram illustrating an example of the relationship between the average luminance AVE of the color HDR image data I and the limit value lim of the gain coefficient k, and FIG. 5B is a diagram illustrating the gain coefficient k and the illumination. It is a figure which shows an example of the relationship with the component data. FIG. 6A is a diagram showing an example of the relationship between the luminance of the color HDR image data I and the luminance of the color LDR image data O when the limit value lim is set to “5”. FIG. 6B is a diagram illustrating an example of the relationship between the luminance of the color HDR image data I and the luminance of the color LDR image data O when the limit value lim is set to “0.25”. .
When the subject imaging process is started in the HDR imaging device 10 in response to a user operation, the light reflected from the subject is collected by the lens 11a and is incident on the microlens 11b. Incident light from the lens 11a is collimated by the microlens 11b and is incident on each pixel of the sensor cell array via the color filter array 11c.

On the other hand, in the HDR sensor 11d, the reset line R is set one line at a time from the start line, and the accumulated charge of each pixel is reset. Subsequently, for each scanning line, after resetting each pixel, a non-destructive readout line L1 is set at the elapse timing of the exposure time T1, and the pixel signal S1 is read from each pixel (sensor cell) of the scanned line.
Subsequently, for each scanning line, after resetting each pixel, the non-destructive readout line L2 is set at the elapse timing of the exposure time T2, and the pixel signal S2 is read from each pixel of the scanned line. Subsequently, the read & reset line L3 & R is set in order from the start line, the pixel signal S3 is read from each pixel of the scanned line, and then the accumulated charge of each pixel is reset.

Thereafter, while the photographing process is being performed, the setting of L1 to L3 & R, the reading of the pixel signals S1 to S3, and the reset process are repeatedly performed according to the above procedure.
The pixel signals S1 to S3 read out in this way are converted into digital pixel data S1 to S3 in the HDR sensor 11d, and sequentially to the memories 12 to 14 in units of lines and in the order of the pixel data S1 to S3. Is output.

On the other hand, the HDR signal processing unit 15a of the image synthesizing unit 15 executes the HDR synthesizing process when the pixel data S1, S2, and S3 for one line are stored in the memories 12, 13, and 14, respectively.
When the HDR synthesizing process is executed, the HDR signal processing unit 15a acquires pixel data S1 to S3 for one line from the memories 12 to 14. Then, the obtained S1 to S3 are combined to generate color HDR pixel data Ir (x, y), Ig (x, y), and Ib (x, y) at each pixel position. Then, the generated color HDR pixel data Ir (x, y), Ig (x, y), Ib (x, y) is stored in the memory area of the address corresponding to each pixel position of the frame memory 15b via the memory IF. To store.
In this way, color HDR pixel data is sequentially generated, and when the color HDR image data I for one image (one frame) is generated, the generated color HDR image data I is output to the image processing device 20.

  On the other hand, when the image processing device 20 determines that shooting has started in the HDR imaging device 10 (the branch of “Yes” in step S100), the image processing device 20 waits for the color HDR image data I to be transmitted from the HDR imaging device 10. To do. When the color HDR image data I for one frame is acquired from the HDR imaging device 10 (“Yes” in step S102), first, the illumination component conversion unit 20a converts the acquired color HDR image data I into luminance image data. Conversion to Y is performed (step S104).

Here, it is assumed that color HDR image data I in which each pixel data is expressed by 10 bits is acquired.
For example, the color HDR pixel data Ir (1, 1), Ig (1, 1), and Ib (1, 1) with the pixel position “(x, y) = (1, 1)” are “(Ir (1 , 1), Ig (1, 1), Ib (1, 1)) = (250, 752, 560) ”.
In this case, “Y (1, 1) = 0.3 × 250 + 0.6 × 752 + 0.1 × 560≈582” is calculated from the above equation (9).

In this way, each pixel data of the acquired color HDR image data I is converted into a luminance value, and luminance image data Y is generated by performing luminance conversion on all the pixel data.
Once the luminance image data Y is generated, the image processing device 20 then performs a blurring process using the Gaussian filter of the above equation (8) on the generated luminance image data Y in the illumination component conversion unit 20a. The illumination component data L is generated (step S106).

The Gaussian filter is obtained by generating a kernel (also called an operator) rate corresponding to n × m pixels (n and m are natural numbers of 3 or more) by a Gaussian distribution function. Using this kernel, blurring processing is performed by multiplying the luminance values of the pixel of interest and its surrounding pixels by a rate corresponding to each pixel position.
When the illumination component data L is generated by the blurring process, the illumination component conversion unit 20a outputs the generated luminance image data Y to the limit control unit 20b, and outputs the illumination component data L to the coefficient calculation unit 20c.

When the limit control unit 20b acquires the luminance image data Y from the illumination component conversion unit 20a, the limit control unit 20b calculates the average luminance AVE of the acquired luminance image data Y (step S108). In the present embodiment, the average luminance AVE is calculated by dividing the sum W of luminance values of all pixel data in the luminance image data Y by the number of pixels N (AVE = W / N).
After calculating the average luminance AVE, the limit control unit 20b then compares AVE with a preset threshold Y _H on the high luminance side, and determines whether or not AVE is larger than Y _H (Step S20). S110).

Here, since the color HDR image data I is 10-bit data, “768” is set as the threshold Y _H on the high luminance side, and “256” is set as the threshold Y _L on the low luminance side. .
In the present embodiment, the lim value of the gain coefficient k is further set so that the color LDR image data after range compression has a predetermined gradation characteristic. Specifically, with respect to the value range of the illumination component data L, it has log sensor characteristics in the low luminance range, linear log sensor characteristics in the middle luminance range, and linear sensor characteristics in the high luminance range. Limit values lim _S , lim _M , and lim _L are set as follows.

Here, since the color HDR image data I in the 10-bit gradation range is range-compressed into the color LDR image data O in the 8-bit gradation range, as shown in FIG. lim _S = 1/4 (0.25), lim _M = 5, lim _L = 22 ".
For example, when the luminance average AVE is “400”, it is smaller than “Y _H = 768” (the branch of “Yes” in step S110) and larger than “Y _L = 256” (in step S128). “No” branch).

Therefore, when the luminance average AVE is “400”, the limit control unit 20b determines that the limit value lim of the gain coefficient k is “lim = 5”, and outputs information on the determined lim to the coefficient calculation unit 20c. (Step S132).
When the coefficient calculation unit 20c acquires information on the limit value “lim = 5” from the limit control unit 20b, first, the coefficient calculation unit 20c compresses the range of the illumination component data L according to the above equation (10), and the compressed illumination component data L 'Is generated (step S114).

  For example, the color HDR pixel data Ir (x, y), Ig (x, y), and Ib (x, y) are “155, 158, 131”, and the luminance pixel data Y (x, y) is “154”. It is assumed that the illumination component data L (x, y) at this time is “150”. In this case, from the above equation (9), “L ′ (x, y) = 255/1023 × (1023 × log (150) / log (1023)) ≈184”.

When the compressed illumination component data L ′ (x, y) is calculated, the coefficient calculation unit 20c then calculates the gain coefficient k (x, y) for the pixel position (x, y) according to the above equation (10). Calculate (step S116).
Specifically, since L (x, y) is “150” and L ′ (x, y) is “184”, “k (x, y) = 184 / 150≈1 from the above equation (10). .2 ".
When the gain coefficient k is calculated, the coefficient calculation unit 20c then compares the calculated gain coefficient “k (x, y) = 1.2” with the limit value “lim = 5” (step S118). ). In this case, since “k (x, y) <lim” (the branch of “No” in step S118), information on the gain coefficient “k (x, y) = 1.2” is stored in the range compression unit 20d. Output to.

  Further, for example, the color HDR pixel data (Ir (x, y), Ig (x, y), Ib (x, y)) is “4, 25, 13”, and the luminance pixel data Y (x, y) is It is assumed that the illumination component data L (x, y) at this time is “15.” In this case, “L ′ (x, y) = 255/1023 from the above equation (9). × (1023 × log (15) / log (1023) ≈100 ”.

  Since the coefficient calculation unit 20c has L (x, y) of “15” and L ′ (x, y) of “100”, the gain coefficient k is calculated as “k (x, y) from the above equation (10). ) = 100 / 15≈6.6 ”. In this case, since “k (x, y)> lim” (the branch of “No” in step S118), the gain coefficient k (x, y) is set to the value of lim. That is, the information “k (x, y) = lim = 5” is output to the range compression unit 20d.

When the range compression unit 20d acquires the gain coefficient k (x, y) information from the coefficient calculation unit 20c, the color HDR pixel data Ir (x, y) at the pixel position (x, y) according to the above equation (12). , Ig (x, y) and Ib (x, y) are compressed (step S122).
When the gain coefficient k is “1.2”, “Or (x, y) = 1.2 × 155 = 186” and “Og (x, y) = 1.2 × 158 are obtained from the above equation (12). ≈ 190 ”and“ Ob (x, y) = 1.2 × 131 = 157 ”.

As a result, “(Or (x, y), Og (x, y), Ob (x, y)) = (186, 190, 157)” is obtained as the color LDR pixel data after range compression.
When the gain coefficient k is “5.0”, “Or (x, y) = 5.0 × 4 = 20” and “Og (x, y) = 5.0 are obtained from the above equation (12). × 25 = 125 ”and“ Ob (x, y) = 5.0 × 13 = 65 ”.

As a result, “(Or (x, y), Og (x, y), Ob (x, y)) = (20, 125, 65)” is obtained as the color LDR pixel data after range compression.
On the other hand, when the limit value “5” is not limited, the gain coefficient k is “6.6”. Therefore, from the above equation (12), “Or (x, y) = 6.6 × 4≈ 26 ”,“ Og (x, y) = 6.6 × 25 = 165 ”, and“ Ob (x, y) = 6.6 × 13≈86 ”.

Therefore, originally, “(Or (x, y), Og (x, y), Ob (x, y)) = (26, 165, 86)” is obtained, and the low-brightness portion that is too bright is determined as the gain coefficient. By providing a limit on k, it is possible to suppress the luminance to (20, 125, 65).
The color LDR pixel data (Or (x, y), Og (x, y), Ob (x, y)) calculated in this way is subjected to inverse gamma processing in the inverse gamma processing unit 20e. In the storage unit 20f, the data is stored in the frame memory.

Then, the same processing as described above is repeated, and when the range compression processing is completed for all the pixels of the color HDR image data I (“Yes” branch of step S124), it is then determined whether or not the photographing processing is completed. (Step S126). If it is determined that the shooting process has not been completed (“No” branch in step S126), the transmission of the color HDR image data I of the next frame is awaited (step S102), and it is determined that the shooting process has ended. (Step S126: “Yes” branch) terminates the series of processing.
When the limit value lim is “5”, as shown in FIG. 6A, the color LDR image data O (output value) is lower in luminance than the color HDR image data I (input value). It has the gradation characteristic of the linear log sensor, which changes linearly in the region and changes in the log sensor from the middle.

On the other hand, in the limit control unit 20b, for example, when the luminance average AVE is “800”, it becomes larger than “Y _H = 768” (the branch of “Yes” in Step S110), so the limit value of the gain coefficient k The lim is determined to be “lim = 0.25” which is the minimum value.
As shown in FIG. 5B, the fact that the limit value is smaller than the range of the value of the gain coefficient k means that all calculated k are fixed to lim (1/4), and a constant is multiplied. It becomes the same as that. Therefore, in this case, as shown in FIG. 6B, the color LDR image data O changes linearly with respect to the color HDR image data I, and has the gradation characteristics of the linear sensor.

In the limit control unit 20b, for example, when the luminance average AVE is “200”, it becomes smaller than “Y _H = 768” (the branch of “No” in Step S110), and “Y _L = 256”. Smaller than. Therefore, the limit value lim of the gain coefficient k is determined to be “lim = 22” which is the maximum value.
When the limit value is large, as shown in FIG. 5B, the brightness of the image can be ensured since the calculated k is used as long as the value of L is not so small. Furthermore, since the upper limit of k is limited so that it is not too bright in the low luminance part, it is possible to prevent the low luminance part from being excessively bright. In this case, the color LDR image data O has the gradation characteristics of the log sensor.

As described above, the imaging system 1 according to the present embodiment compresses the dynamic range by compressing the illumination light component in the luminance value indicated by each pixel data of the color HDR image data I in the image processing device 20 (range compression). )can do.
Further, when the range compression is performed, the gain coefficient k for compressing the illumination light component is calculated, the limit value lim of the gain coefficient k is determined, and the limit of the gain coefficient k for luminance conversion is limited based on the determined lim. be able to.
Further, when determining the limit value lim, the luminance image data Y is generated from the color HDR image data I, and the average luminance AVE of the luminance image data Y is calculated. Then, based on the average luminance AVE and preset luminance threshold values Y _H and Y _L (Y _H > Y _L ), the limit value lim of the luminance conversion gain coefficient k corresponding to the brightness of the image is determined. can do.

Furthermore, for a relatively high-brightness image, the limit value is smaller than usual (for example, a value smaller than the minimum value of the coefficient k), and the tone characteristics of the image after range compression are It is possible to set a limit value lim _S that is a gradation characteristic. For a relatively low-brightness image, the gradation characteristic of the linear log sensor is the gradation value of the image after the range compression with a smaller limit value (larger than lim _S ) than usual. The limit value lim _M can be set. For an image in the luminance range between Y _H and Y _L , it is possible to set a normal limit value lim (L) at which the tone characteristics of the image after range compression become the tone characteristics of the log sensor. it can.
As described above, with respect to a dark image, it is possible to generate an image that secures necessary brightness while limiting the low luminance portion of the input image so as not to be too bright. As for a bright image, it is possible to generate a sharp image with no black floating while limiting the low luminance portion of the input image so that it is not too bright.

Also, the image quality characteristic (gradation characteristic) of the output image can be dynamically changed to any one of the linear sensor, the linear log sensor, and the log sensor in accordance with the brightness of the input image.
In the said embodiment, the illumination component conversion part 20a, the limit control part 20b, the coefficient calculation part 20c, and the range compression part 20d respond | correspond to the image conversion means of any one of the forms 1-5.
Moreover, in the said embodiment, step S100-S132 respond | corresponds to the image conversion step as described in form 6 or 7.

In the above-described embodiment, the imaging system 1 corresponds to the imaging device according to the eighth aspect or the electronic device according to the ninth aspect.
In the imaging system 1 in the above embodiment, the HDR imaging device 10 and the image processing device 20 can be integrally formed to form a configuration of the imaging device. Alternatively, an electronic device such as a digital camera or a digital video camera can be configured in combination with another device (not shown) such as a display device or a memory device.

In the above embodiment, the HDR sensor 11d of the image sensor 11 has the sensor cell array configured using the CMOS technology. However, the configuration is not limited thereto. For example, other configurations such as a configuration having a sensor cell array composed of CCDs may be used.
In the above embodiment, the limit value lim is selected from the three types of lim _s , lim _M , and lim _H based on the comparison result between the average luminance AVE and the luminance threshold values Y _H and Y _L. However, the present invention is not limited to this configuration.
For example, the limit value, only the lim _H is the reference value for lim _H, configuration and adding or subtracting the correction amount corresponding to the magnitude of the average brightness AVE, selected from four or more limit values configuration Other configurations may be used.

In the above embodiment, the values of lim _s , lim _M , and lim _H are set to 0.25, 5, and 22. However, the values are not limited to these values, and other values may be used as long as the magnitude relationship is maintained. Good.
In the above embodiment, the values of lim _s , lim _M , and lim _H are set so that the gradation characteristics of the output image are the gradation characteristics of the linear sensor, linear log sensor, and log sensor. Not exclusively.
For example, other configurations such as setting a value that is not related to the gradation characteristics, or setting a value that becomes the gradation characteristics of another sensor may be used.

The above embodiments are preferable specific examples of the present invention, and various technically preferable limitations are given. However, the scope of the present invention is described in particular in the above description to limit the present invention. As long as there is no, it is not restricted to these forms. In the drawings used in the above description, for convenience of illustration, the vertical and horizontal scales of members or parts are schematic views different from actual ones.
Further, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging system, 10 ... HDR imaging device, 11 ... Imaging device, 11a ... Lens, 11b ... Micro lens, 11c ... Color filter array, 11d ... HDR sensor, 12-14 ... Memory, 15 ... Image composition part, 15a ... HDR signal processing unit, 15b ... frame memory, 20 ... image processing device, 20a ... illumination component conversion unit, 20b ... limit control unit, 20c ... coefficient calculation unit, 20d ... range compression unit, 20e ... inverse gamma processing unit, 20f ... Preservation part

Claims (9)

Image conversion means for converting the luminance of the input image to generate an output image having different gradation characteristics,
The image conversion means has a conversion characteristic that increases the gain for converting the luminance as the luminance of the input image is lower, and sets an upper limit for the gain, and makes the upper limit variable. .   The image processing apparatus according to claim 1, wherein the image conversion unit converts the luminance of the input image to generate an output image in which a dynamic range of the input image is compressed.   The image conversion means increases the upper limit when the luminance of the input image is lower than a set luminance threshold, and increases the upper limit when the luminance is higher than the luminance threshold. The image processing apparatus according to claim 1, wherein the image processing apparatus is lowered.   4. The image processing apparatus according to claim 1, wherein the image conversion unit changes the upper limit in accordance with an average luminance of the input image. 5.   The image conversion means changes the upper limit so that the output image has an image characteristic at the time of conversion by any one of a plurality of types of imaging elements having different conversion methods for converting an incident light amount into an electrical signal. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is configured to perform the above-described processing. Including an image conversion step of converting the luminance of the input image to generate an output image having different gradation characteristics,
In the image conversion step, the lower the luminance of the input image, the higher the gain for converting the luminance, the upper limit is set on the gain, and the conversion processing is performed to make the upper limit variable. Method. Including an image processing program for causing a computer to execute a process having an image conversion step of converting the luminance of an input image to generate an output image having different gradation characteristics;
In the image conversion step, the lower the luminance of the input image, the higher the gain for converting the luminance, the upper limit is set on the gain, and the conversion processing is performed to make the upper limit variable. program. Imaging means for imaging a subject;
An image processing apparatus according to any one of claims 1 to 5,
The image processing apparatus performs conversion processing using a captured image obtained by capturing with the imaging unit as the input image.   An electronic apparatus comprising the imaging device according to claim 8.

Images