JP2009060675A - Imaging element - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve color reproducibility and noise characteristics by forming a color filter of an imaging element by a combination of five colors of primary color system RGB and complementary color system YC. <P>SOLUTION: An imaging element having a hybrid RGBYC color filter is composed by using primary color system RGB filters and complementary color system YC filters. G filters which directly relate to resolution and is close to a luminance signal that human eyes sense are arrayed in a checker shape so that the number of the G filters is four times larger than the number of filters of each of the other colors. An array shown in Fig. 10A is composed of low-sensitivity rows (G, R, G and B) and high-sensitivity rows (C, G, Y and G) that are alternately arrayed in each line. When signals are read by varying exposure time for each line, the signals that are read can easily have a wide dynamic range. An array shown in Fig. 10B is composed so that two Gs are included in each line and each row and two remaining colors are formed by a combination of a low-sensitivity color and a high-sensitivity color. Thus, the array shown in Fig. 10B has the small luminance difference in the horizontal direction and the vertical direction. Therefore, the reading method in the array shown in Fig. 10B is complicated compared with that in the array shown in Fig. 10A. However, since the spatial interpolation characteristic of the array shown in Fig. 10B is advantageous, a smooth gradation can be easily represented. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image sensor.

  In recent years, an increase in the number of pixels of an image sensor of an image input device has been remarkable, and the resolution of the image sensor has reached a level having smooth graininess that cannot be determined by human eyes. Under these circumstances, the demand for image quality is shifting from resolution to color reproducibility, noise reduction, and wide dynamic range. The present invention provides means and an apparatus for increasing the dynamic luminance range that can be scanned while maintaining image quality, particularly color reproducibility and noise characteristics at an acceptable level.

  First, an image processing apparatus that reproduces a more faithful color and can reduce noise is described in Patent Document 1 below.

JP 2003-284084 A

  Hereinafter, the apparatus described in Patent Document 1 will be described. A four-color filter 1 having the color arrangement shown in FIG. 1 is provided on the front surface of the image sensor. As indicated by the alternate long and short dash line, the color filter 1 includes an R filter that transmits only red (R) light, a B filter that transmits only blue (B) light, and green light in the first wavelength band. And a G1 filter that transmits only green light and a G2 filter that transmits only green light in the second wavelength band.

FIG. 2 shows an example of the configuration of a signal processing unit that performs signal processing on an image pickup signal obtained by an image pickup device having such a color filter 1, for example, a CCD. Reference numeral 10 denotes a front end to which four types of color signals (R signal, G1 signal, G2 signal, and B signal) from the image sensor are input. The front end 10 performs processing such as correlated double sampling processing for removing noise components, gain control processing, and digital conversion processing on the color signal from the image sensor. Image data from the front end 10 is LSI (Large Scale Integrated).
Circuit: a large-scale integrated circuit) is supplied to the signal processing unit 11.

  The signal processing unit 11 is connected to a microcomputer (not shown) via a microcomputer interface 12. The microcomputer controls the entire operation of the digital still camera, for example, according to a predetermined program. Further, each block constituting the signal processing unit 11 is controlled by the microcomputer via the microcomputer interface 12.

  The signal processing unit 11 performs interpolation processing, filtering processing, matrix calculation processing, luminance signal generation processing, color difference signal generation processing, and the like on the four types of color signals input from the front end 10. The image signal generated by the signal processing unit 11 is supplied to a display (not shown), and a captured image is displayed. Further, the image data from the signal processing unit 11 is compressed and stored in an internal storage medium, an external storage medium, or the like.

  Each block of the signal processing unit 11 will be described below. The offset correction processing unit 21 removes a noise component (offset component) included in the image signal supplied from the front end 10. The image signal from the offset correction processing unit 21 is output to the white balance correction processing unit 22 and white balance correction is performed. That is, the imbalance between the colors due to the difference in the color temperature environment of the subject and the difference in sensitivity due to the color filters (R, G1, G2, B) on the sensor is corrected.

  The output of the white balance correction processing unit 22 is supplied to the vertical direction synchronization processing unit 23. The vertical direction synchronization processing unit 23 synchronizes temporally different vertical image data using a delay element, for example, a small memory, for vertical interpolation processing and filtering processing.

  A plurality of image signals synchronized by the vertical direction synchronization processing unit 23 are supplied to the interpolation processing, filter processing, high-frequency correction processing, and noise processing unit 24. Interpolation processing for interpolating the color signal of 2 × 2 pixels, which is the minimum unit of the color filter (R, G1, G2, B), into the phase of the same space, filter processing for appropriately limiting the signal band, and high signal band A high frequency correction process for correcting the band component, a noise process for removing the noise component of the signal, and the like are performed.

  Image signals obtained by the processing unit 24, for example, RG1G2B four-color signals are supplied to the linear matrix processing unit 25. The linear matrix processing unit 25 performs 4-input / 3-output matrix calculation. By giving matrix coefficients of a 3 × 4 matrix, RGB color output can be obtained from the input image information of four colors of RG1G2B.

  The RGB output from the linear matrix processing unit 25 is supplied to the gamma correction processing units 26R, 26G, and 26B, respectively. The gamma correction processing units 26R, 26G, and 26B perform reverse correction of the nonlinear characteristics of the display device in advance, so that linear characteristics are finally realized.

  Output signals from the gamma correction processing units 26R, 26G, and 26B are supplied to a luminance (Y) signal generation processing unit 27 and a color difference (C) signal generation processing unit 28, respectively. The luminance signal generation processing unit 27 generates a luminance signal by synthesizing the gamma-corrected RGB signal with a predetermined synthesis ratio. The color difference signal generation processing unit 28 generates a color difference signal by synthesizing the gamma-corrected RGB signals with a predetermined synthesis ratio.

  The color difference signal generated by the color difference signal generation processing unit 28 is supplied to the band limiting and thinning processing unit 29, and a color difference signal in which the color difference signals Cb and Cr are time-division multiplexed is formed. As described above, the image processing apparatus using the four-color filter can improve the color reproducibility as compared with the apparatus using the three primary color filters.

  In general, the characteristics preferable as the spectral sensitivity of the image sensor are good color reproducibility and good noise characteristics. “Good color reproducibility” means that a color similar to that of the human eye can be sensed or a color difference with respect to the appearance of the human eye is small. The appearance of the human eye means the color that is visible to the human eye. “Noise characteristics are good” means that the amount of noise at a certain luminance level is small. Noise is broadly divided into luminance noise and color noise. Luminance noise depends on absolute sensitivity. Color noise is greatly related to the relationship between the spectral sensitivities of the color filters of the image sensor, that is, the shape of the spectral sensitivity curve. Dependent.

  A technique for generating a primary color RGB signal value by performing matrix conversion on an output signal of an image sensor that is linear in luminance is often performed in signal processing of a general image input apparatus as shown in FIG. This processing is called linear matrix processing. In most cases, an image input device (such as a scanner or a digital still camera) often observes and edits an input image on a personal computer (hereinafter abbreviated as PC) monitor. The target color space is set to the sRGB color space, which is a general PC monitor space.

The sRGB color space is defined by the International Electrotechnical Commission (IEC) as a standard color space for multimedia that the color image signal to be transmitted should comply with. By conforming to the standard color space, the color image sending side and receiving side can share the same color reproduction.

  Therefore, the target spectral sensitivity (represented in the figure as relative sensitivity) of the image sensor is an sRGB color matching function obtained by linearly converting a color matching function (see FIG. 3) that is the spectral sensitivity of the human eye using a 709 matrix. . The 709 matrix is described in the following Non-Patent Document 1.

"ITU-R BT.709-3," Basic Parameter Values for the HDTV Standard for the Studioand for International Program Exchange "(1998)”

In FIG. 3, a curve 31x indicates a function x (λ), a curve 31y indicates a function y (λ), and a curve 31z indicates a function z (λ). The color matching function graph shown in FIG.
Internationale de I'Eclairage (International Lighting Commission) 1931.

  FIG. 4 is a graph showing the sRGB color matching function. In FIG. 4, a curve 32r indicates a function r (λ), a curve 32g indicates a function g (λ), and a curve 32b indicates a function b (λ). Since the sRGB color matching function satisfies the router condition, it is possible to sense a color as seen with the eyes. The router condition is described in the following non-patent document 2.

Noboru Ota, "Color Engineering", ISBN: 4-501-61350-5, Tokyo Denki University Press (1993)

  However, the spectral sensitivity shown in FIG. 4 has negative spectral sensitivity, and in reality, it is impossible to create a three-color RGB filter having such spectral sensitivity. If an attempt is made to create a three-color RGB filter spectral sensitivity having a positive spectral sensitivity and satisfying the router condition, the spectral sensitivity as shown in FIG. 5 is obtained. In FIG. 5, a curve 33R represents a function sR (λ), a curve 33G represents a function sG (λ), and a curve 33B represents a function sB (λ).

  As can be seen from FIG. 5, the overlap of the spectral sensitivity curve 33R of the red component filter of this image sensor and the spectral sensitivity curve 33G of the green component filter is very large. This means that the two component signals are very similar. Therefore, when trying to calculate the three colors in the sRGB space, which is the target output signal, using the image sensor having the spectral sensitivity filter shown in FIG. 5, the matrix operation of the following equation (1) is required.

  As can be seen from the matrix coefficients in equation (1), in order to calculate the red component of the output signal, a very large matrix of 6.5614 and -5.5412 for the red and green components of the input signal. Multiplied by a factor. This means that the noise on the red and green signals of the image sensor is greatly increased.

  Therefore, in practice, a three-color RGB filter having spectral sensitivity as shown in FIG. 6 is used even if the router condition is not perfectly satisfied, that is, noise characteristics are good even if color reproducibility is somewhat sacrificed. . In FIG. 6, a curve 34R represents a function s1R (λ), a curve 34G represents a function s1G (λ), and a curve 34B represents a function s1B (λ). Since the spectrum of FIG. 6 does not satisfy the router condition, it cannot be linearly converted to an sRGB color matching function. Therefore, the approximate conversion matrix calculation to the sRGB color matching function is expressed by the following equation (2).

  In the matrix coefficient of Expression (2), it can be seen that the absolute values of all the coefficients are smaller than the matrix coefficient shown in Expression (1), and as a result of color separation, noise does not increase relatively.

  A general primary color RGB imaging device has a spectral sensitivity having a curve as shown in FIG. 6 for the reason described above, and is known to have excellent color reproducibility and noise characteristics. However, in actuality, the graph of the spectral sensitivity curve shown in FIG. In FIG. 7, a curve 35R indicates the spectral sensitivity of the R filter, a curve 35G indicates the spectral sensitivity of the G filter, and a curve 35B indicates the spectral sensitivity of the B filter.

  Due to various manufacturing limitations, it is difficult to increase the spectral sensitivity of each color filter. In order to maintain the sensitivity, methods such as increasing the cell size to some extent or multiplying by an electrical gain are used. However, increasing the cell size involves a sacrifice in resolution, and multiplying the gain sacrifices noise reduction.

  In summary, the primary color RGB imaging device is quite good in terms of color reproducibility, but has some room for improvement. In terms of noise characteristics, color separation noise is small, but noise is low due to low sensitivity. There is a tendency that luminance noise, which is a dominant component of, increases. That is, the color reproducibility is good, but there is a feature that a sense of noise is likely to occur.

  As the color filter, a complementary color filter is also known. For example, a complementary color checkered line sequential color filter is known in which four color filters of Y (yellow), C (cyan), M (magenta), and G (green) are arranged as shown in FIG. In FIG. 8, a 2 × 4 array surrounded by a broken line is the minimum unit of the array.

  A graph of the spectral sensitivity of the complementary color YCMG image sensor is shown in FIG. In FIG. 9, a curve 36Y indicates the spectral sensitivity of the Y filter, a curve 36C indicates the spectral sensitivity of the C filter, a curve 36M indicates the spectral sensitivity of the M filter, and a curve 36G indicates the spectral sensitivity of the G filter. Show. As shown in FIG. 9, since the sensitivity of each color filter is high, the complementary color YCMG image sensor is particularly strong in photographing in a dark place and has good luminance noise characteristics. However, since the spectral sensitivity overlap is very large, an attempt to improve color reproducibility requires a very large color separation coefficient, and there is a problem that color separation noise increases. Therefore, there is a problem that the color reproducibility cannot be pursued as compared with the primary color RGB imaging device.

  As described above, the complementary color YCMG image pickup device has a great room for driving in color reproducibility. Regarding noise characteristics, although color separation noise is large, luminance noise, which is a dominant component of noise, tends to be small because of high sensitivity. That is, the noise is good, but the color reproducibility is not so good.

  As described above, in the conventional image sensor, there is a problem that there is a lot of luminance noise due to low sensitivity when the primary color RGB image sensor is used, and on the other hand, when the complementary color YCMG image sensor is used, the color reproducibility is high. In addition, there is a problem that color separation noise characteristics are insufficient.

  Accordingly, an object of the present invention is to provide an image sensor that can solve the problems of these image sensors.

In order to solve the above-described problems, the present invention provides three primary color filters including an R (red) filter, a G (green) filter, and a B (blue) filter, a Y (yellow) filter, and a C ( A total of five color filters including two complementary color filters composed of cyan) filters,
This is an image sensor in which G (green) filters are arranged in a checkered pattern so as to obtain spatial information four times that of other colors.

  According to the present invention, color reproducibility and color separation noise can be reduced by combining the color filters of the image sensor with five colors of the primary color RGB complementary color system YC. Also, by using the output signals of the RGBYC5 color filter image sensor according to the brightness level, the image pickup with high sensitivity, that is, low noise, with better color reproducibility than using the primary color and complementary color image sensors alone. An element can be realized. As a result, it is possible to capture a scene with a wide dynamic range.

  In the present invention, an image pickup device including a hybrid RGBYC color filter is configured by using a primary color RGB filter and a complementary color YC filter. Various color arrangements are possible, but the color filters shown in FIGS. 10A and 10B are used as an example. In FIG. 10, RGB indicates a filter that transmits red, green, and blue light, respectively, and YC indicates a filter that transmits yellow and cyan light, respectively.

FIGS. 10A and 10B show the minimum unit (4 × 4) of one example of an array of color filters that can be used in the present invention and another example, respectively. The characteristics of the color filter arrangement shown in FIG. 10 are as follows. First, a G filter close to the luminance signal of the human eye that is directly related to the resolution is arranged in a checkered pattern four times as many as other color filters. It is that you are. In other words, the G filter occupies a ratio of 8 out of the 16 minimum units, and the remaining four color filters have a ratio of 2 each. The array shown in FIG. 10A has low sensitivity columns (G, R, G,
B) is an array in which rows (C, G, Y, G) with high sensitivity are alternately arranged. If reading is performed by changing the exposure time for each row, it is easy to read a wide dynamic range.

  The array shown in FIG. 10B is included in two Gs in each of the rows and columns, and the remaining two colors are a combination of a high sensitivity and a low sensitivity, so that the horizontal, This is an array with little luminance difference in the vertical direction. Therefore, compared with the arrangement shown in FIG. 10A, the readout method is complicated, but since the spatial interpolation characteristic is advantageous, it has a feature that smooth gradation expression can be easily realized.

  FIG. 11 shows a graph of spectral sensitivity of the hybrid RGBYC color filter. In FIG. 11, a curve 37Y indicates the spectral sensitivity of the Y filter, a curve 37C indicates the spectral sensitivity of the C filter, a curve 37R indicates the spectral sensitivity of the R filter, and a curve 37G indicates the spectral sensitivity of the G filter. The curve 37B shows the spectral sensitivity of the B filter. As shown in FIG. 11, the complementary colors Y, C have higher spectral sensitivity than the primary colors R, G, B.

  FIG. 12 shows the relationship between the amount of charge Q and the amount of light P stored in the image sensor for each color of the color filter. In FIG. 12, Qs represents the saturation charge amount, 38C represents the [PQ] characteristic relating to the C filter, 38Y represents the [PQ] characteristic relating to the Y filter, and 38R represents the [PQ characteristic relating to the R filter. -Q] characteristic, 38G shows the [PQ] characteristic for the G filter, and 38B shows the [PQ] characteristic for the B filter.

  As shown in FIG. 12, the cells in which the high sensitivity Y and C filters are arranged immediately accumulate necessary charges even with a small amount of light, while the cells in which the low sensitivity R and B filters are arranged. Since the rate at which charges are accumulated is slow even when a large amount of light is incident, they are not immediately saturated with charges. By utilizing this difference in characteristics, it is possible to capture an image with a very wide dynamic range.

  As an example, among the five color filters, since the cell in which the C filter is arranged is saturated first, the light amount Ph at which the charge accumulated in the cell in which the C filter is arranged is saturated is defined as a low / medium level boundary. A luminance region and a high luminance region are defined. In the case of a luminance image included in the low / medium luminance region, signal values of five colors R, G, B, Y, and C are used. In the case of a luminance image included in the high luminance region, Y is used. Since the accumulated charge of the cell in which the filters of C and C are arranged is saturated, the signal values of the three colors R, G and B are used. As described above, it is possible to generate an image with a very wide dynamic range by switching the color filter to be used according to the luminance of the image.

  As a method for determining the luminance of an image, for example, a method of performing determination based on signal values such as G and Y, which are signal values of an image sensor including a luminance component of human eyes, is possible.

In order to calculate the RGB signal value of the image from the signal value of the image sensor, some matrix calculation processing is required. For example, matrix calculation processing for converting 5 color RGBYC signal values (Rin, Gin, Bin, Yin, Cin) into generated RGB image signal values (Rout, Gout, Bout) uses 3 × 5 matrix coefficients. Is represented by the following equation (3).

  The matrix coefficient in Expression (3) is referred to as a linear matrix in the sense of a matrix calculation coefficient for a linear signal. See Non-Patent Document 3 below for the method of determining the linear matrix.

"Takami Mizunakura, Naoya Kato, Kenichi Nishio," Evaluation Method of CCD Color Filter Spectral Sensitivity Considering Noise ", Color Forum JAPAN2003 Proceedings, pp.29-32 (2003)"

Hereinafter, image data having different luminance will be described in detail.
In the case of low / medium luminance image data, the low-sensitivity R and B signal values are increased to some extent, the influence of noise is reduced,
When the signal value of C is not saturated, it is possible to use the signal values of all five colors of the hybrid image sensor. Compared to the three primary colors, the number of colors is five and the five colors have good color separation, so that it is possible to perform better image processing in both color reproducibility and noise reduction.

In the case of high luminance image data When the high sensitivity Y, C signal values are saturated, an image is generated with the remaining R, G, B signal values. This state is equivalent to the case where a primary color three-color image sensor is used, and the color reproducibility and noise reduction are the same as those of the primary color three-color image sensor.

  An image pickup apparatus using the above-described hybrid RGBYC image pickup device, for example, a digital still camera will be described below. However, the present invention is not limited to a still camera, and can be applied to a camera for moving image shooting.

  When the above-described hybrid RGBYC image sensor is used, it is possible to configure a camera with better color reproducibility and noise characteristics than a camera with a primary color system RGB image sensor or a complementary color YCMG image sensor.

  Using the color filter evaluation method described in Non-Patent Document 3 described above, draw a CN diagram for three image sensors: a hybrid RGBYC image sensor, a primary color system RGB image sensor, and a complementary color system YCMG image sensor. . A method for writing a CN diagram will be briefly described.

Precondition As a photographing condition, a standard light source D55 defined by CIE is used as a light source, and a Macbeth Color Checker is used as a target. As camera signal processing, processing as shown in FIG. 13 is assumed. White balance for Raw Data 41 obtained by integrating the product of the spectral sensitivity of the color filter of the image sensor, the spectral reflectance of the Macbeth Color Checker color patch, and the spectral radiance of the photographic light source as the overall characteristics of the digital still camera In the correction process 42, a gain is multiplied so that the level of each color data is equal to the achromatic color.

  Next, a color conversion process 43 is performed by a matrix process so as to bring a linear signal value close to the target color with respect to the luminance signal. This processing 43 corresponds to linear matrix processing. Hereinafter, this matrix is appropriately referred to as MAT. By the color conversion processing 43, the signal value becomes three colors of R (red), G (green), and B (blue). When the number of colors of the color filter is expressed as N, the white balance correction process 42 is an N × N diagonal matrix calculation process, and the color conversion process 43 is a 3 × N matrix calculation process.

  Finally, the camera GB 45 is obtained as the final output through the gamma correction processing 44. However, here, the inverse gamma of sRGB is used as the camera gamma. When configuring the hardware of the digital still camera, the same configuration as the signal processing configuration shown in FIG. 2 described above is used.

  The camera RGB thus obtained is assumed to be reproduced and observed on a standard sRGB monitor. That is, the camera RGB is assumed to be a color in the sRGB color space. However, since values outside the sRGB range are also stored in the calculation, it is considered that colors outside the sRGB color gamut are not clipped.

Definition of color reproduction evaluation index Calculates the color difference and Delta E (ΔE) value of the output signal of the photographed camera for the human eye. Since the camera RGB is assumed to be data in the sRGB color space, the corresponding L ^* a ^* b ^* value camera L ^{* a * b *} is calculated according to the flowchart shown in FIG. 14 using the ₇₀₉ matrix _M709 or the like. Is done.

The L ^* a ^* b ^* value is a color space defined from tristimulus values in standard light. CIE proposed U ^* V ^* W ^* as a uniform color space in 1964, and proposed L ^{* u *} v ^* space in 1976, which was modified from U ^* V ^* W ^* space. Further, in 1976, an L ^* a ^* b ^* space (CIELAB) was proposed as a color space corresponding to a perceptual color difference in any color region.

In FIG. 14, after the camera RGB 51 receives the sRGB gamma process 52, the conversion process 53 converts RGB into tristimulus values XYZ. In the conversion process 54, 709
Using the system matrix M ₇₀₉ 55, XYZ is converted to L ^* a ^* b ^* . Then, the final camera L ^{* a * b *} 56 is obtained.

The 709 matrix M ₇₀₉ is described in Non-Patent Document 1.

The L ^* a ^* b ^* value of the human eye appearance as the target value can be calculated from the spectral reflectance of Macbeth Color Checker and the spectral radiance of the standard light source D55.

  As an evaluation index of color reproducibility, the average ΔE value (ΔEa) of 24 colors of Macbeth Color Checker is adopted. This value becomes a function ΔEa (MAT) of the MAT coefficient during the signal processing of the camera, as shown by the following equation (4).

Definition of Noise Evaluation Index As a model of noise included in the output signal of the image sensor, a definition formula shown in the following formula (5) is used. This definition is described in Non-Patent Document 4 below.

"G. C. Holst," CCD ARRAYS CAMERAS and DISPLAYS 2nd Ed., "JCD Publishing (1998)"

In the formula (5), a _s and b _d, as described in Non-Patent Document 5 below, is a value determined by the CCD device characteristics (saturated electron amount, etc.).

Kenichi Nishio, "CCD Camera Color," Color Forum JAPAN '99, pp.143-147 (1999)

_{a s} · CV_ _CCD, the optical shot noise, i.e. represents a noise component that depends on the signal value, b _d is floor noise, namely, represents a noise component that does not depend on the signal value (see Non-Patent Document 4).

The noise Noise _ _raw signal processing of the camera, further is propagated to various color spaces by the L ^* a ^* b ^* conversion. The outline of the noise propagation model (see Non-Patent Document 6 below) is shown below.

P. D. Burns and R.S. Berns, "Error Propagation Analysis in Color Measurement and Imaging," Col. Res. Appl, Vol.22, pp.280-289 (1997)

If an input signal is linearly converted to Y = [y ₁ , y ₂ ,..., Y _m ] ^t by an (m × n) matrix A, the equation can be written as follows:

  Y = A ・ X ・ ・

  Now, it is assumed that the variance-covariance matrix of the input signal X is represented by the following equation (6).

  The diagonal component of this equation (6) is just the noise variance value of the input signal. If there is no correlation between the input signal values, the covariance component (that is, the off-diagonal component) in the matrix component is zero. At this time, the variance-covariance matrix of the output signal Y can be defined by the following equation (7).

This equation (7) is a theoretical expression for propagation of noise variance values between color spaces that can be converted by linear conversion.

In order to convert the final output signal camera RGB value to L ^* a ^* b ^* value, as shown in the section of color reproducibility definition, when converting from XYZ space to L ^* a ^* b ^* space Includes non-linear transformation. However, since the noise variance value is usually very small, the XYZ → L * a * b * conversion is approximately expressed by linear conversion using the Jacobian matrix J _{L * a * b} described in Non-Patent Document 6 described above. be able to. Therefore, an approximate matrix M _total for linearly converting the original signal into L ^* a ^* b ^* values is expressed by the following equation (8).

  Using this matrix of equation (8) and noise propagation theoretical equation (7), the noise variance value on the final output signal value can be calculated by the following equation (9).

The total noise value (TN value) defined by the following equation (10) is calculated using the noise amounts σ _L *, σ _a *, and σ _b * obtained from the equation (9). Further, the average value TNa of the TN values of each of the 24 colors of Macbeth Color Checker is used as a noise evaluation index.

The TN value is a value highly correlated with noise perceived by human eyes, taking into account both the brightness component and the color component of the noise. Using L ^* a ^* b ^* noise amount, it is defined by the following equation.

In Expression (11), W _L *, W _a *, and W _b * are weighting coefficients for the noise amounts σ _L *, σ _a *, and σ _b *, and are experimental and empirical values obtained by visual experiment. It was used.

Method for Determining Parameters during Camera Signal Processing Specifically, this is a method for determining MAT coefficients during signal processing. First, using the color reproduction evaluation index ΔEa and the noise evaluation index TNa, a CEM (Comprehensive Error Metric) value is defined by the following equation (12).

  By determining weighting coefficients wc and wn for each evaluation index and obtaining a MAT coefficient that minimizes the CEM value, it is possible to realize filter evaluation with adjusted color reproducibility and noise amount.

Method of creating CN diagram The MAT coefficient is calculated by changing wc and wn of the CEM value described above, and the color reproduction evaluation index ΔEa and the noise evaluation index TNa at that time are calculated. As shown in FIG. 15, a plot of [color reproduction versus noise] is created with ΔEa on the horizontal axis and TNa on the vertical axis (hereinafter, this plot is referred to as a CN diagram as appropriate). The CN diagram shows the ability of the color filter set to be evaluated.

  The lower left area in FIG. 15 is an area where both color reproduction and noise are good, and the upper right area is an area where both are bad. Therefore, the closer the plot is to the lower left, the better the filter set. By performing this plot for each filter set to be evaluated and comparing it, it is possible to intuitively determine which filter set is superior. In general, the trajectory of the CN diagram is a convex curve in the lower left corner, and the color reproducibility and noise characteristics that the noise characteristics are poor when the color reproducibility is good and the color reproducibility is poor when the noise characteristics are good are traded. Indicates an off relationship.

  A CN diagram of the hybrid image sensor, the primary color system three-color image sensor, and the complementary color system four-color image sensor according to the present invention is as shown in FIG. Reference numeral 61a is a plot of [color reproduction vs. noise] of the hybrid image sensor, reference numeral 61b is a plot of [color reproduction vs. noise] of the primary color system three color image sensor, and reference numeral 61c is a complementary color system four color image sensor. It is a plot of [color reproduction vs. noise] of an element.

  Since the lower left of the CN diagram shown in FIG. 16 is an area where both color reproduction and noise characteristics are good, an image sensor having a good trajectory can be said to have a trajectory closest to the lower left. As can be seen from FIG. 16, the plot 61a of the hybrid image sensor is located at the bottom left, so that it can be seen that both the color reproducibility and noise characteristics are superior to the primary color image sensor and the complementary color image sensor.

Actually, an RGBYC imaging device is created with the arrangement shown in FIG. 10, and an IC (Integrated Circuit) for performing signal processing shown in FIG. 13 is created. Gamma correction is RGB using sRGB gamma
Calculate the output signal. As the linear matrix coefficient, for example, a weight coefficient of wc: wn = 1: 3 indicated by ☆ 62 in the CN diagram (FIG. 16) is used. The coefficient of ☆ is shown in equation (13).

  It can be seen from equation (13) that the coefficient value is small and the color separation noise is not increased so much. This makes it possible to configure a camera using a five-color RGBYC image sensor.

  Next, an image input apparatus that can capture a wide dynamic range image using a hybrid image sensor will be described.

  An example of scanning an image with a wide dynamic range by using signal values of five colors depending on the luminance level of the pixel value will be described.

  There are various methods for calculating the luminance level of the pixel value. For example, a G signal close to the luminance component of the human eye is used as the luminance value. As the signal processing, the processing shown in FIG. 17 can be considered. LM (H) 71, LM (M) 72, and LM (L) 73 indicate linear matrix processing for high luminance, medium luminance, and low luminance, respectively. WB (H) 74, WB (M) 75, WB ( L) 76 indicates white balance processing, and 77, 78, and 79 indicate multipliers that multiply the output signals by gain values a, b, and c, respectively.

  FIG. 18 shows the transition of the weight gain coefficient with respect to the change in the luminance level (horizontal axis). In the high luminance region, the gain coefficient is (a = 1, b = c = 0), in the medium luminance region, the gain coefficient is (a = 0, b = 1, c = 0), and in the low luminance region. The gain coefficient is (a = b = 0, c = 1). In addition, in the vicinity of the boundary of each region, the rise and the fall of the gain coefficient are inclined to cross each other so that the change is not noticeable. Although omitted in FIG. 17, a gain coefficient control unit that controls the gain coefficient by, for example, the luminance level (for example, G signal) for each sample after A / D (analog-digital) conversion of the RGBYC signal is provided. ing.

  The linear matrix coefficient of each luminance level is determined as follows with reference to Non-Patent Document 3.

Low luminance region In the low luminance region, noise is less important than color reproducibility from the viewpoint of image quality. Therefore, as the linear matrix coefficient, a coefficient with an emphasis on noise characteristics is used using RGBYC5 color signal values. The coefficient of the point of wc: wn = 1: 10 (the position of the Δ mark 63 in FIG. 16) can be considered. The coefficient values are shown below.

  Since Expression (14) has a larger weight for noise than Expression (13), it can be seen that the absolute value of each coefficient is small and color separation noise is not increased.

Medium luminance region Linear matrix coefficients for medium luminance pixels are determined at the point where the balance between color reproducibility and noise characteristics is the best using RGBYC5 signal values. That is, in the CN diagram, a coefficient that is on the locus of the lower left position is determined. For example, a matrix coefficient in the vicinity of wc: wn = 1: 3 shown in Expression (13) is conceivable.

High luminance region Since the Y signal and C signal are considerably saturated in the linear matrix coefficient for high luminance pixels, the signal balance of RGB three colors is used to provide the best balance between color reproducibility and noise characteristics. Decide on points. That is, in the CN diagram, a coefficient that is on the locus of the lower left position is determined. For example, the coefficient of the point of wc: wn = 1: 3 (the position of the circle 64 in FIG. 16) can be considered. The coefficient is shown in the following formula (15).

The processing flow is as follows.
When a certain pixel signal is input, the input signal value is subjected to three types of linear matrix processing (LM (H) 71, LM (M) 72, LM (L) 73 in FIG. 17), and for each luminance level. A signal value is calculated.

  Each signal value is subjected to white balance processing (WB (H) 74, WB (M) 75, WB (L) 76 in FIG. 17), and as shown in FIG. 18, according to the luminance level of the pixel. , Multiplied by a predetermined gain coefficient (a, b, c) (multipliers 77, 78 and 79 in FIG. 17).

  Thereafter, the three types of output signals are added by the adders 80 and 81 in FIG. 17, and the output signal value of the pixel is determined.

  As a boundary between the low luminance level and the middle luminance level, for example, the boundary between ISO 400 and ISO 200 can be considered. The boundary between the medium luminance and the high luminance level is, for example, a case where one of the Y signal and the C signal is saturated.

  Although the embodiment of the present invention has been specifically described above, the present invention is not limited to the above-described embodiment, and various modifications based on the technical idea of the present invention are possible. For example, in the configuration of FIG. 17, different matrix calculation processing is performed for three levels of luminance levels, but different matrix calculation processing is performed for two levels (low / medium luminance level and high luminance level). You may do it. Alternatively, three colors of YCG may be used at the low luminance level, and five colors of RGBYC may be used at the medium luminance level. Further, the present invention can be applied to a device having three layers of CMOS structures and having a blue-sensitive layer, a green-sensitive layer, and a red-sensitive layer as well as a color film.

It is a basic diagram which shows the color arrangement | sequence of the color filter of the imaging device proposed previously. It is a block diagram which shows the structure of the signal processing part of the imaging device proposed previously. It is a graph which shows an example of a color matching function. It is a graph which shows an example of sRGB color matching function. It is a graph which shows the spectral sensitivity of the 3 color filter which satisfy | fills router conditions. It is a graph which shows the spectral sensitivity of a normal three primary color filter. It is a graph which shows the spectral sensitivity of a primary color system RGB image sensor. It is a basic diagram which shows the color arrangement | sequence of the color filter of a complementary color type | system | group YCMG image pick-up element. It is a graph which shows the spectral sensitivity of a complementary color type | system | group YCMG image pick-up element. It is a graph which shows the spectral sensitivity of a hybrid RGBYC image sensor. It is a basic diagram which shows the color arrangement | sequence of a hybrid RGBYC image pick-up element. It is a basic diagram for demonstrating the signal processing using the output signal of a hybrid RGBYC image pick-up element. It is a flowchart which shows the flow of the signal processing which can apply this invention. It is a flowchart which shows the flow of the color conversion process in this invention. It is a graph which shows the CN figure in this invention. It is a graph which compares and shows CN of a hybrid RGBYC image sensor, a primary color system image sensor, and a complementary color system image sensor. It is a block diagram which shows an example of the signal processing in one Embodiment of this invention. It is an approximate line figure showing an example of a weight gain coefficient transition under signal processing.

Explanation of symbols

37C, 37Y Spectral sensitivity of cyan and yellow filters 37R, 37G, and 37B Spectral sensitivity of red, green, and blue filters 42 White balance correction processing 43 Color conversion processing 44 Camera gamma correction processing 52 sRGB gamma processing 53 Convert RGB to XYZ Conversion Process 54 Converting XYZ to L ^* a ^* b ^* 61a [Color Reproduction vs. Noise] Plot of Hybrid Image Sensor 61b [Color Reproduction vs. Noise] Plot of Primary Color System 3 Color Image Sensor 61c Complementary Color System 4 [Color reproduction vs. noise] plot of color image sensor 71 High-brightness linear matrix processing 72 Medium-brightness linear matrix processing 73 Low-brightness linear matrix processing 77, 78, 79 Multiply gain factors a, b, c Multiplier

Claims (5)

Three primary color filters composed of R (red) filter, G (green) filter, and B (blue) filter, and two complementary color filters composed of Y (yellow) filter and C (cyan) filter And a total of five color filters,
An image sensor in which G (green) filters are arranged in a checkered pattern to obtain spatial information four times that of other colors. The minimum unit of the color filter array is 4 × 4, and each column and each row includes two G (green) filters,
The imaging device according to claim 1, wherein the remaining two color filters are a combination of a high sensitivity and a low sensitivity.   3. The image pickup device according to claim 2, wherein the combination of the remaining two color filters is a combination of a Y (yellow) filter and a B (blue) filter, or a combination of a C (cyan) filter and an R (red) filter. . The minimum unit of the color filter array is 4 × 4, and each column and each row includes two G (green) filters,
The imaging device according to claim 1, wherein columns and rows using the remaining two filters having high sensitivity and columns and rows using low sensitivity are alternately arranged. In the columns and rows using the remaining two color filters having high sensitivity, a C (cyan) filter and a Y (yellow) filter are used,
5. The image pickup device according to claim 4, wherein a B (blue) filter and an R (red) filter are used in columns and rows in which the remaining two color filters have low sensitivity.

Images