JP2001086402A - Device and method for processing high dynamic range image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To drastically expand the dynamic range of an image processor itself by defining processing that transforms an input signal corresponding to an area where a characteristic between the incident luminance of a solid-state image pickup device and an output voltage lies in a non-linear relationship into a signal having a linear characteristic as one processing and executing processing that presupposes linearity between input signals on the basis of an output signal from a linear transforming means. SOLUTION: A CCD 100 provided with a color filter uses an output signal from a pixel provided with another adjacent color filter about a signal of a wavelength region where the signal can not be transmitted through the signal can not be. When the color space of an output of a color interpolating means 103 is an RGB space, signals outputted due to the color interpolation are defined as a red signal r0, a green signal g0 and a blue signal b0. A dynamic range compressing means 104 is used before color space conversion processing by a matrix means 105, and the bit widths of respective color signals outputted from the means 103 are reduced. Thus, a circuit scale in the means 105 is reduced.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a high dynamic image processing apparatus and method for processing a color video signal from an image sensor having a high dynamic range and displaying the image naturally on a display device having a relatively narrow dynamic range. About.

[0002]

2. Description of the Related Art In a conventional digital still camera or video camera, only a luminance region in which the characteristic between the incident luminance and the output voltage of a solid-state image sensor has a linear relationship has been processed. Therefore, also in the color separation processing for processing a signal from the imaging device, there is a precondition that a linear relationship is established between the incident luminance and the output voltage of the solid-state imaging device.

FIG. 10 is a block diagram showing an example of a color camera color signal processing method using a solid-state image pickup device in the prior art. The camera here includes a video camera for moving images and a digital still camera for still images. Hereinafter, a color camera color signal processing method will be specifically described by taking, as an example, a case where the solid-state imaging device 200 has the color filter shown in FIG.

[0004] An analog video output signal from the solid-state imaging device 200 is converted into a digital signal by an AD converter 1001. A signal in a wavelength range that does not pass through one color filter is calculated by the color interpolation unit 1002 using a signal output from an adjacent pixel having another color filter.

At this time, when the color space output from the color interpolation means 1002 is an RGB space, the signals output by the color interpolation here are a red signal r1, a green signal g1, and a blue signal b1. Each signal is then subjected to color correction by a matrix unit 1003 of three rows and three columns so as to be suitable for a desired color space. The output signal of the matrix means is a red signal r
Assuming that the green signal g2 and the blue signal b2 are 2, it can be obtained from the input signals red signal r1, green signal g1, and blue signal b1 using Equation 1. (Hereinafter, uppercase letters indicate a matrix or vector, lowercase letters indicate a scalar)

[0006]

(Equation 1)

However, the matrix M is a matrix of 3 rows and 3 columns.
Nine color separation coefficients or elements of arbitrary values are represented by m11, m12, m13, m21, m2 as shown in the following equation 2.
2, m23, m31, m32, and m33 are assumed to be included.

[0008]

(Equation 2)

As shown in FIG. 1, when white balance means 1004, 1005 and 1006 are provided, diagonal components m11 and m2 of the matrix are provided.
2 and m33 each become 1, and the matrix M has six color separation coefficients or elements. The output signals of the matrix means obtained by the above equation 2 are r2, g2, b
2 is white balance means 1004, 1005, 100
6 is converted as shown in Expression 3.

[0010]

(Equation 3)

However, white balance means 1004-1
The output signal of 006 is a red signal r3, a green signal g3, and a blue signal b.
3. Set the white balance coefficient to WB (wbr, wbg,
wbb). White balance means 1004-10
The output signal from the gamma correction means 1007, 100
8 and 1009, the output device 101
Output to 0.

[0012]

Therefore, the conventional image processing apparatus as described above has several problems to be solved as follows. For example, a camera placed in a natural environment such as a surveillance camera must be able to identify a target object regardless of the imaging state. In particular, in the case of backlight, the target image is often blackened and cannot be distinguished. The reason is that when the auto iris of the camera is activated, the shutter time and the aperture are adjusted according to the bright luminance of the background, and the luminance of the target object becomes relatively dark.

Conversely, when the auto iris is operated in accordance with the object, the background is saturated with white. The reason is that if the search area of the camera's auto iris is matched to a dark object, the shutter time and aperture are adjusted accordingly, the background brightness becomes relatively bright, exceeding the sensitivity area of the solid-state image sensor. It is. That is, the above-described phenomenon of blackening and saturation of white is caused by a narrow dynamic range of the solid-state imaging device, the image processing apparatus itself, or the image display unit with respect to the dynamic range of the image to be captured. It is. Therefore, a camera used for such a purpose is required to have a wider dynamic range of the solid-state imaging device, the image processing device itself, or the image display means.

Here, not only a luminance region in which the characteristic between the incident luminance and the output voltage of the solid-state image sensor is in a linear relationship, but also a luminance region in a non-linear relationship which is a saturation characteristic of the solid-state image sensor having a vertical overflow drain structure. By utilizing this, the dynamic range can be expanded. A similar method is described in Japanese Patent Application Laid-Open No. 9-65217, but this method does not use the nonlinear input / output sensitivity characteristics. In this conventional technique, the amount of charge flowing out of the photodiode to the vertical overflow drain is controlled by changing the voltage applied to the semiconductor substrate with time,
In this way, the amount of charge remaining in the photodiode is adjusted to expand the dynamic range.

Accordingly, an object of the present invention is to increase the dynamic range of the image processing apparatus itself by utilizing the non-linear characteristics of the current flowing over the barrier (overflow barrier) formed by the vertical overflow drain structure with respect to the above-mentioned conventional technology. It is an object of the present invention to provide a high dynamic image processing apparatus which can be greatly expanded while suppressing an increase in circuit scale and an increase in power consumption.

[0016]

In order to achieve the above object, according to the first aspect of the present invention, there is provided at least one solid-state image pickup device having a vertical overflow drain structure, wherein the solid-state image pickup device is provided at a subsequent stage. In a high dynamic range image processing apparatus that performs processing on the assumption of linearity between input signals and outputs the processed signal to a display unit, the characteristic between the incident luminance and the output voltage in the solid-state imaging device is non-linear. An input signal corresponding to a related region has linear conversion means for converting the input signal into a signal having linear characteristics as one process, and the linearity between the input signals is determined based on an output signal from the linear conversion means. Is performed on the assumption that

According to a second aspect of the present invention, in the first aspect of the present invention, the linear conversion means performs, as another processing, an area in which the characteristic between the incident luminance and the output voltage in the solid-state imaging device has a linear relationship. The input signal is converted into a signal having a linear characteristic.

According to a third aspect of the present invention, in the first or second aspect, the linear conversion means converts the input signal from the solid-state imaging device into a signal having linear characteristics. A lookup table system in which an input signal from the solid-state imaging device and an output signal after the conversion are stored in association with each other.

According to a fourth aspect of the present invention, in the invention according to any one of the first to third aspects, the processing assuming linearity between the input signals includes inputting an output signal from the linear conversion means. And a color separation process of outputting the processed signal to the display means. Between the color interpolation process and the color separation process, each color signal output from the color interpolation process is included. A dynamic range compression unit for compressing a dynamic range of each color signal so as to maintain linearity between the color signals.

According to a fifth aspect of the present invention, in the fourth aspect of the present invention, the dynamic range compression means calculates luminance from each color signal output in the color interpolation processing,
The color signals are divided by the same ratio according to the calculated luminance.

According to a sixth aspect of the present invention, in the fifth aspect of the present invention, in the dynamic range compressing means, the same ratio is obtained by limiting a frequency band of a luminance calculated from each of the color signals by a low-pass filter. And is calculated based on the output signal in accordance with a visual model equation.

According to a seventh aspect of the present invention, in the invention of the sixth aspect, the dynamic range compression means has a maximum value detecting section for calculating a maximum value of luminance calculated from each of the color signals, and When changing the expression according to the brightness of the image, the maximum value detected by the maximum value detection unit is used as the input of the visual model expression, and the maximum value of the output of the visual model expression by the input is the maximum value detection unit. The visual model formula is set so as to be the maximum value obtained in (1).

According to an eighth aspect of the present invention, in the invention of the seventh aspect, the dynamic range compressing means determines a maximum value obtained by the maximum value detecting section in order to set the visual model equation. A look-up table method in which coefficients of the visual model formula corresponding to the maximum value are stored in advance is adopted.

According to a ninth aspect of the present invention, in any one of the fourth to eighth aspects of the present invention, the high dynamic range image processing apparatus has a matrix means in the color separation processing.

The invention according to claim 10 is the invention according to claims 4 to 9
The high dynamic range image processing apparatus according to any one of claims 1 to 3, wherein the high dynamic range image processing apparatus includes a white balance unit at a stage subsequent to the matrix unit in the color separation processing.

According to the eleventh aspect of the present invention, there is provided at least one solid-state imaging device having a vertical overflow drain structure, wherein a process is performed at a subsequent stage of the solid-state imaging device assuming linearity between input signals, In the high dynamic range image processing method for outputting a signal after the processing to a display means, an input signal corresponding to a region where a characteristic between an incident luminance and an output voltage in the solid-state imaging device has a non-linear relationship is converted into a signal having a linear characteristic. And performing a process on the basis of an output signal from the linear conversion process on the assumption of linearity between the input signals.

According to a twelfth aspect of the present invention, in the invention of the eleventh aspect, in the linear conversion step, further, the input corresponding to a region where the characteristic between the incident luminance and the output voltage in the solid-state imaging device has a linear relationship. Converting a signal into a signal having linear characteristics is one process.

According to a thirteenth aspect of the present invention, in the eleventh or twelfth aspect of the invention, in the linear conversion step, when the input signal from the solid-state imaging device is converted into a signal having linear characteristics, the conversion operation is performed. As a system, a look-up table system in which an input signal from the solid-state imaging device and an output signal after the conversion are stored in association with each other is adopted.

According to a fourteenth aspect of the present invention, in the invention according to any one of the eleventh to thirteenth aspects, the processing based on the linearity between the input signals includes inputting an output signal from the linear conversion means. And a color separation process of outputting the processed signal to the display means. Between the color interpolation process and the color separation process, each color signal output from the color interpolation process is included. A dynamic range compression step of compressing a dynamic range of each color signal so as to maintain linearity between the color signals.

According to a fifteenth aspect of the present invention, in the invention of the fourteenth aspect, in the dynamic range compression step, a luminance is calculated from each color signal output in the color interpolation processing, and the luminance is calculated in accordance with the calculated luminance. And dividing the respective color signals by the same ratio.

According to a sixteenth aspect of the present invention, in the invention of the fifteenth aspect, in the dynamic range compression step, the same ratio is obtained by limiting a frequency band of a luminance calculated from each of the color signals by a low-pass filter. And is calculated based on the output signal in accordance with a visual model equation.

According to a seventeenth aspect of the present invention, in the invention of the sixteenth aspect, in the dynamic range compression step, a maximum value of luminance calculated from each of the color signals is obtained.
When changing the visual model formula according to the brightness of an image, the maximum value of the calculated luminance is used as the input of the visual model formula, and the maximum value of the output of the visual model formula by the input is the maximum value of the luminance. Thus, the visual model formula is set.

According to an eighteenth aspect of the present invention, in the invention of the seventeenth aspect, in the dynamic range compressing step, a maximum value of the luminance and a value corresponding to the maximum value are set in order to set the visual model equation. It is characterized by adopting a look-up table method in which coefficients of a visual model formula are stored in advance.

According to a nineteenth aspect of the present invention, in any one of the fourteenth to seventeenth aspects, the high dynamic range image processing method includes a matrix step in the color separation processing.

According to a twentieth aspect, in the invention according to any one of the fourteenth to nineteenth aspects, the high dynamic range image processing method includes a white balance step after the matrix step in the color separation processing. It is characterized by the following.

[0036]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Generally, in an image processing apparatus using a solid-state image sensor having nonlinear input / output characteristics, if processing in a linear region and processing in a nonlinear region are separated, signal processing means is required for each region. This increases the circuit scale and power consumption. Therefore, the nonlinear region is converted back to a signal in the linear region before the signal processing. By this linearization process, a process based on the linearity of signals between different pixels, such as a color interpolation process in a color separation process, can be performed only by a correction for expanding a bit width.

The color space conversion requires linearity only between color signals such as red, blue, and green, and does not require linearity between different pixels. Therefore, before performing color space conversion such as matrix processing or white balance, by compressing the dynamic range so that the linearity between color signals is maintained, the bit width expanded by the linearization processing can be reduced. . In particular, in color space conversion, since multiplication with a large circuit scale is frequently used, the circuit scale and power consumption can be significantly reduced by reducing the bit width.

That is, even if the dynamic range is widened, the portion where the bit width is widened is only required before and after the interpolation means, so that the circuit scale can be reduced. A matrix means,
Color separation processing circuits such as white balance means and gamma correction can be used as they are conventionally. In the dynamic range compression at this time, color reproducibility is maintained because each color signal is divided by the same ratio.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram for explaining the configuration of the image processing apparatus according to the present invention based on one solid-state image sensor. One embodiment of the present invention will be described with reference to FIG. The image processing apparatus referred to here includes a video camera for moving images and a digital still camera for still images. Hereinafter, the present embodiment will be described as an example in which the solid-state imaging device 100 has the color filter shown in FIG.

Light transmitted through an optical means such as a lens is photoelectrically converted in the solid-state image pickup device 100. The solid-state imaging device 100 has a non-linear input / output characteristic.
An analog video output signal from the solid-state imaging device 100 is A
Is converted into a digital signal raw by the A / D conversion means 101, and the digital output signal r of the A / D conversion means 101
aw is a signal r obtained by converting the nonlinear characteristic region of the digital output signal raw into linear characteristics by the linear conversion means 102.
awlinear.

In the solid-state imaging device 100 having a color filter as shown in FIG. 2, for a signal in a wavelength range that does not pass through one color filter, an output signal from an adjacent pixel having another color filter is used. Color interpolation means 103
Is calculated. At this time, when the color space of the output of the color interpolation means 103 is an RGB space, the signals output by the color interpolation here are a red signal r0, a green signal g0, and a blue signal b0.

Further, in this embodiment, before the color space conversion processing by the matrix means 105 described later, the dynamic range compression means 104 is used to execute the color interpolation means 1.
03, the bit width of each color signal output is reduced. As a result, the circuit scale required for the color space conversion processing in the matrix unit 105 can be reduced, and the power consumption can be reduced accordingly.

The signals r1, g1, and b1 of each color output from the dynamic range compressing means 104 are subjected to color correction by a matrix means 105 of three rows and three columns so as to be suitable for a desired color space, and r2, g2, b2 is output, and
Each of the color signals r2, g2, b2 is converted to a white balance
6 to 108, the white balance is adjusted.
3, g3, and b3 are output, and the color signals r3, g3, and b3 whose white balance has been adjusted are output from the gamma correction units 109 to
Gamma correction according to the display device is performed by 111 and the color signals r4, g4, b4 are output to the output device 112.

FIG. 3 is a diagram showing an example of linear and non-linear characteristics between the incident luminance and the output voltage in the solid-state imaging device. As shown in FIG. 3, for example, a solid-state imaging device having a vertical overflow drain structure has a non-linear characteristic in characteristics between incident luminance and output voltage. In the incident luminance region until the solid-state imaging device is saturated,
Has a linear characteristic as shown in FIG.

[0046]

(Equation 4)

Here, x is the incident luminance and y is the output voltage. When the incident luminance increases and becomes saturated, the solid-state imaging device has a logarithmic characteristic as shown in Expression 5 below.

[0048]

(Equation 5)

As a reference for the logarithmic characteristic of a solid-state imaging device having an overflow drain structure after saturation, Kawa
i, et. al: "Photo Response A
nal_y_sis in CCD Image Se
nsor with a VOD structure
", IEEE Transactions on El
electron Devices, Vol. 42, no.
4, Apr. 1995. There is.

FIG. 4 is a block diagram showing an example of the configuration of the linear conversion means 102. The conversion equation when the incident luminance is in the logarithmic area (non-linear area) is obtained from Equations 4 and 5 above, and is the conversion equation shown in Equation 6 below.

[0051]

(Equation 6)

On the other hand, when the incident luminance is in the linear region,
The linear conversion means 102 performs the conversion represented by the following Expression 7.

[0053]

(Equation 7)

However, when the conversion processing is performed by the linear conversion means 102 in accordance with Equation 7, the amount of calculation is large, and conditions are divided for the case where the incident luminance is in the linear region and the non-linear region (logarithmic region). There must be. Therefore, in the present invention, Equations 6 and 7 are preliminarily calculated including this condition division, and the results are stored in a look-up table (LUT). As a result, it is possible to perform moving image processing in which the calculation time required for this conversion processing is reduced, and it is possible to reduce the circuit scale and power consumption as a whole.

FIG. 4 shows a schematic configuration of the linear conversion means 102 using the above-mentioned lookup table (LUT). The update of the lookup table 400 is performed by the CPU 40
This is performed by the calculation of Equation 6 based on a, b, c, and d to which 1 is input. If the characteristic between the incident luminance and the output voltage in the solid-state imaging device 100 does not change with time,
The look-up table 400 need not be updated by the CPU 401. In this case, in particular, the CPU 40
1 need not be provided.

FIG. 5 shows the dynamic range compression means 10.
4 is a diagram showing a schematic configuration of FIG. Each color signal r0, g0, b0 input to the dynamic range compression means 104 is divided by the same value k shown in the following equation 8 by dividers 506 to 508, and output as color signals r1, g1, b1.

[0057]

(Equation 8)

It should be noted that since the value k to be divided by all the color signals is the same, color reproducibility is ensured. This value k is calculated as follows. First, the luminance is calculated by the luminance calculating means 500 from the input color signals r0, g0, b0, and the result is output as a luminance signal Y. Next, the luminance signal Y is applied to the low-pass filter 50.
2, the signal Y _low indicating the luminance whose frequency band is limited is output. Here, in order to compress the dynamic range, a visual model equation as shown in Equation 9 below is assumed. (See Fig. 7)

[0059]

(Equation 9)

For example, if the dynamic range is 1/256
Assuming that the compression, the bit width of the input signal to the dynamic range compression unit 104 18bit, if the bit width of the output signal of 10bit, the variable j is 2 ^{10/2 18/3}
= 2 ⁴ = 16, and the variable k is calculated as in the following equation 10 based on the above equations 8 and 9.

[0061]

(Equation 10)

A low-pass filter 50 is added to x in Expression 10 above.
The variable k can be calculated by substituting Y _low output from 2. However, in order to compress the dynamic range, in other words, in order not to widen the dynamic range, a certain lower limit k1 (k1>
It is necessary to provide 1). Such division according to Equation 10 is performed by division means 505.

Further, regarding the X ^2/3 part in the expression 10, Y _low output from the low-pass filter 502 and
By mounting the result of X ^2/3 corresponding to the Y _low as the look-up table 503, the processing speed can be improved. When the variable j is changed depending on the brightness of the image, the maximum value Y in the frame of the luminance Y obtained from the luminance Y signal by the maximum value detecting means 501
_{Use max} . That is, the look-up table 504 for calculating j is set so that the maximum value of Expression 9 becomes _Ymax .

FIG. 7 is a diagram showing an example of the configuration of the matrix means 105. Assuming that the output signals of the matrix means 105 are a red signal r1, a green signal g2, and a blue signal b1, the red signal r2, the green signal g2, and the blue signal b2 are obtained from the input signals red signal r1, green signal g1, and blue signal b1 according to the related art. Can be obtained in the same manner according to the equation (2).

Here, the matrix M is a matrix of 3 rows and 3 columns.
9 color separation coefficients or elements of any value, m11, m1
2, m13, m21, m22, m23, m31, m3
2, m33. Here, the white balance means 106 to 106
When 108 is provided, the diagonal components m11, m22, and m33 of the matrix M are each 1 and the matrix M has six color separation coefficients or elements. In FIG. 7, the matrix means 105 having six elements
Is shown. 700 to 705 are multiplication means.
6 to 708 are adding means.

FIG. 8 shows white balance means 106-1.
It is a figure which shows the schematic structure of 08. Output signals of the white balance means 106 to 108 are a red signal r3, a green signal g3,
Assuming a blue signal b3, these red signal r3 and green signal g
3. The blue signal b3 can be obtained from the input signals red signal r2, green signal g2, and blue signal b2 by Expression 2 in the same manner as in the related art. However, the coefficient of the white balance is WR (w
br, wbg, wbb). 800 to 802 are expressed by Equation 2.
Is a multiplication means that performs an operation according to

FIG. 9 is a block diagram showing an image processing apparatus constructed based on a plurality of (here, three) solid-state image sensors as another embodiment of the present invention. Note that the image processing apparatus here includes a video camera for moving images and a digital camera for still images as in the above-described embodiment. Hereinafter, an example will be described in which three solid-state imaging devices 200 to 202 are provided so that each of the red, green, and blue color signals can be output by one solid-state imaging device.

Light transmitted through a lens or the like is color-separated by a prism, and the color-separated light is output to the solid-state imaging devices 200 to 2.
02 is photoelectrically converted. This solid-state imaging device 200
To 202 have nonlinear input / output characteristics as in the above embodiment. Each analog output signal from the solid-state imaging devices 200 to 202 is
A / D conversion means 203 to 205 provided corresponding to
Are respectively converted into digital signals.

Each digital video output signal from the A / D conversion means 203 to 205 is
2. The non-linear characteristic area is converted to the linear characteristic area by the linear conversion means 206 to 208 provided corresponding to the A / D conversion means 203 to 205, and each digital video output signal converted to the linear characteristic area is The bit width is reduced by the dynamic range compression means 209.

The linear conversion means 206 to 208 and the dynamic compression means 209 are the same as the linear conversion means 102 and the dynamic compression means 104 described in the above embodiment. Dynamic compression means 209
The color signals r1, g1, and b1 outputted from are corrected by the matrix means 210 so as to be suitable for a desired color space. Further, each color signal r2, g2, b2 output from the matrix means 210 is
The white balance is adjusted by 11 to 213, and then the color signals r3, g3, b3 output from the white balance units 211 to 213 are converted to gamma correction units 214 to 213.
Gamma correction according to the display device is performed by 216 and output to the output device 217.

[0071]

As is apparent from the above description, as a first effect of the present invention, the dynamic range of the image processing apparatus can be greatly expanded.

The reason is that not only the luminance region where the characteristic between the input luminance and the output voltage of the solid-state imaging device has a linear relationship, but also
This is because a luminance region having a non-linear relationship is also used. For example, a solid-state imaging device having a vertical overflow drain structure has a non-linear region as shown in FIG. 4 in characteristics between input luminance and output voltage. In the solid-state imaging device having such characteristics, the dynamic range of the input luminance is wide because the slope of the logarithmic region is smaller than that of the linear region. For example, assume that a and b in Equations 4 and 5 are of the same order. At this time, in Expression 5, x = 10
If it is 00, since log1000 = 3, the slope of the logarithmic region with respect to the linear region is about 1/100 or less. Therefore, in the luminance region having a non-linear relationship, an incident luminance region that is 100 times or more the linear luminance region can be realized.

As a second effect, since the processing for the luminance regions having the linear and non-linear relationships described above is not divided, the circuit scale can be reduced as a whole.

The reason is that the incident luminance in the non-linear region is once converted into a linear relationship. Furthermore, since the luminance area in the linear area and the luminance area in the non-linear area are simultaneously converted using a look-up table, processing such as conditional branching for both is unnecessary.

As a third effect, even if the dynamic range of the image processing apparatus itself is expanded, the portion where the bit width is widened only inside the color interpolating means can be reduced, so that the circuit scale is reduced.

The reason is that the bit width is enlarged before the color interpolation means in the image processing apparatus, but the enlarged bit width is reduced after the color interpolation means.

As a fourth effect, in the image processing apparatus according to the present invention, a color separation processing circuit (color interpolation means) such as a matrix means, a white balance means, and gamma correction can be used by the same means as the conventional one.

The reason is that the bit width is narrowed by the dynamic range compression element n, so that the subsequent matrix means, white balance means and gamma correction are not affected.

As a fifth effect, the image processing apparatus of the present invention can always maintain color reproducibility regardless of the above processing.

The reason is that all the color signals are divided by the same ratio in the dynamic range compression means, so that the linearity between the color signals is maintained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of a high dynamic image processing device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a configuration example of a color filter array according to an embodiment of the present invention.

FIG. 3 is a diagram illustrating an example of linear and non-linear characteristics of the image sensor according to the embodiment of the present invention.

FIG. 4 is a diagram illustrating a configuration example of a linear conversion unit according to the embodiment of the present invention.

FIG. 5 is a diagram illustrating a configuration example of a dynamic range compression unit according to the embodiment of the present invention.

FIG. 6 is a diagram illustrating an example of a compression ratio calculation characteristic according to the embodiment of the present invention.

FIG. 7 is a diagram illustrating a configuration example of a matrix unit according to the embodiment of the present invention.

FIG. 8 is a diagram illustrating a configuration example of a white balance unit according to the embodiment of the present invention.

FIG. 9 is a block diagram showing a configuration example when the high dynamic range image processing apparatus of the present invention is applied to a three-plate color camera as another embodiment of the present invention.

FIG. 10 is a diagram showing a schematic configuration of a single-panel color camera array according to the related art.

[Explanation of symbols]

 100, 900 to 902 Image sensor 101, 303 to 305 AD converter 102, 906 to 908 Linear conversion means 103 Color interpolation means 104, 909 Dynamic range compression means 105, 910 Matrix means 106, 911 Red signal white balance means 107, 912 Green signal white balance means 108, 913 Blue signal white balance means 109 to 111, 914 to 916 Gamma correction means 112, 917 External device 400, 503, 504 Lookup table 401 CPU 500 Luminance calculation means 501 Maximum value detection means 502 Low pass filter 505 To 508 Division circuit 700 to 705, 800 to 802 Multiplication circuit 706 to 708 Addition circuit

Continued on the front page F term (reference) 5C022 AA01 AA13 AB19 AB31 AC42 AC69 5C024 AA01 BA01 CA15 CA21 DA01 FA01 GA11 GA44 HA08 HA11 HA23 5C065 AA01 AA03 BB01 BB12 BB14 BB48 CC01 DD02 DD11 DD17 DD18 DD19 GG13 GG29

Claims (20)

[Claims] 1. A solid-state imaging device having at least one solid-state imaging device having a vertical overflow drain structure, performing processing on the assumption of linearity between input signals at a subsequent stage of the solid-state imaging device, and In the high dynamic range image processing apparatus for outputting to a display means, it is preferable that an input signal corresponding to a region where a characteristic between an incident luminance and an output voltage in the solid-state imaging device has a nonlinear relationship is converted into a signal having a linear characteristic. A high dynamic range image processing apparatus, comprising: linear conversion means for processing; and performing processing on the basis of output signals from the linear conversion means on the assumption of linearity between the input signals. 2. The linear conversion means according to claim 1, wherein
2. The high dynamic range image processing apparatus according to claim 1, wherein an input signal corresponding to a region where the characteristic between the incident luminance and the output voltage in the solid-state imaging device has a linear relationship is converted into a signal having a linear characteristic. 3. The linear conversion means, when converting the input signal from the solid-state imaging device into a signal having linear characteristics, as the conversion operation method, the input signal from the solid-state imaging device and the converted signal. 3. A high dynamic range image processing apparatus according to claim 1, wherein a look-up table system in which output signals are stored in correspondence with each other is adopted. 4. The processing assuming the linearity between the input signals includes a color interpolation processing using an output signal from the linear conversion means as an input and a color separation processing for outputting a processed signal to the display means. And dynamic range compression for compressing the dynamic range of each color signal so as to maintain linearity between the color signals output from the color interpolation processing between the color interpolation processing and the color separation processing. 4. The high dynamic range image processing apparatus according to claim 1, further comprising a unit. 5. The dynamic range compression means calculates luminance from each color signal output in the color interpolation processing, and divides each of the color signals by the same ratio according to the calculated luminance. The high dynamic range image processing apparatus according to claim 4, wherein: 6. The dynamic range compression unit outputs the same ratio as a signal obtained by limiting the frequency band of a luminance calculated from each of the color signals by a low-pass filter, and based on the output signal, a visual model. 6. The high dynamic range image processing apparatus according to claim 5, wherein the calculation is performed according to an equation. 7. The dynamic range compression unit has a maximum value detection unit that calculates a maximum value of luminance calculated from each of the color signals. When the visual model formula is changed according to the brightness of an image, the maximum value detection unit determines the maximum value. The maximum value detected by the value detection unit is used as an input of the visual model expression, and the visual value is output so that the maximum value of the output of the visual model expression by the input becomes the maximum value obtained by the maximum value detection unit. 7. The high dynamic range image processing apparatus according to claim 6, wherein a model formula is set. 8. The dynamic range compression means sets a maximum value obtained by the maximum value detection unit and a coefficient of the visual model expression corresponding to the maximum value in advance to set the visual model expression. 8. The high dynamic range image processing apparatus according to claim 7, wherein a stored look-up table method is adopted. 9. The high dynamic range image processing apparatus according to claim 4, wherein said high dynamic range image processing apparatus includes a matrix unit in said color separation processing. 10. The high dynamic range image according to claim 4, wherein said high dynamic range image processing apparatus has a white balance means at a stage subsequent to said matrix means in said color separation processing. Processing equipment. 11. A device having at least one solid-state image sensor having a vertical overflow drain structure, performing a process on the assumption of linearity between input signals at a subsequent stage of the solid-state image sensor, and converting the signal after the process. In the high dynamic range image processing method for outputting to a display means, it is preferable that an input signal corresponding to a region where a characteristic between incident luminance and an output voltage in the solid-state imaging device has a non-linear relationship is converted into a signal having a linear characteristic. A high dynamic range image processing method, comprising: a linear conversion step as a process, wherein a process is performed on the basis of an output signal from the linear conversion step, assuming linearity between the input signals. 12. The linear conversion step further includes converting an input signal corresponding to a region where a characteristic between incident luminance and an output voltage in the solid-state imaging device has a linear relationship into a signal having a linear characteristic. The high dynamic range image processing method according to claim 11, wherein the processing is processing. 13. In the linear conversion step, when converting an input signal from the solid-state imaging device into a signal having linear characteristics, the conversion operation method includes an input signal from the solid-state imaging device and a signal after the conversion. 13. The high dynamic range image processing method according to claim 11, wherein a look-up table method in which an output signal is stored in association with the output signal is adopted. 14. The processing assuming linearity between the input signals includes a color interpolation processing in which an output signal from the linear conversion means is input and a color separation processing in which a processed signal is output to the display means. And dynamic range compression for compressing the dynamic range of each color signal so as to maintain linearity between the color signals output from the color interpolation processing between the color interpolation processing and the color separation processing. 14. The high dynamic range image processing method according to claim 11, further comprising a step. 15. In the dynamic range compression step, a luminance is calculated from each color signal output in the color interpolation processing, and each of the color signals is divided by the same ratio in accordance with the calculated luminance. The high dynamic range image processing method according to claim 14, wherein: 16. In the dynamic range compression step, the same ratio is output as a signal obtained by limiting a frequency band of a luminance calculated from each of the color signals by a low-pass filter, and a visual model is generated based on the output signal. 16. The high dynamic range image processing method according to claim 15, wherein the high dynamic range image processing method is calculated according to an equation. 17. In the dynamic range compression step, when a maximum value of luminance calculated from the respective color signals is obtained and the visual model formula is changed according to the brightness of an image,
The visual model expression is set such that the maximum value of the calculated luminance is input to the visual model expression, and the maximum value of the output of the visual model expression by the input is the maximum value of the luminance. The high dynamic range image processing method according to claim 16. 18. A look-up table in which, in the dynamic range compression step, a maximum value of the luminance and a coefficient of the visual model expression corresponding to the maximum value are stored in advance in order to set the visual model expression. The high dynamic range image processing method according to claim 17, wherein a method is adopted. 19. The high dynamic range image processing method according to claim 14, wherein said high dynamic range image processing method includes a matrix step in said color separation processing. 20. The high dynamic range image according to claim 14, wherein the high dynamic range image processing method includes a white balance step after the matrix step in the color separation processing. Processing method.