JP3039653B2 - Imaging device - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to an imaging device having a complementary color filter and the like.

2. Description of the Related Art Conventionally, in this type of apparatus, a color filter as shown in FIG.
Normally, luminance and two color difference signals RY and BY are finally obtained by performing signal processing as shown in the figure.

In such conventional color signal processing, first, arithmetic processing is performed from a color difference signal that is a result of subtracting outputs from pixels that are horizontally adjacent and have different color filters attached. Is common. For example, if the color filter array shown in FIG. 2A is interlaced scanned, the subtractor 304 obtains the subtraction result of C ₁ = (Mg−Gr) from the odd-numbered column of each field, and the even-numbered column Yields a subtraction result of C ₂ = (Ye−Cy). In contrast, the color signal processing unit 305 performs color processing calculations such as white balance and γ conversion by an appropriate method.

Next, these line-sequential color difference signals C ₁ / C ₂ are synchronized by a synchronization circuit 306 using a 1H (horizontal scanning time) delay line and the like. To rotate the chrominance axis appropriately to finally obtain two chrominance signals RY and BY.

However, such a color processing method has two fundamental problems as follows.

(A) It is difficult to maintain white balance.

In a three-tube camera or RGB primary color (pure color) type camera, white balance can be obtained by changing the ratio of R to B with respect to G according to the color temperature. Since it comes out in the form of color difference, for example,
The color difference signal for white is forcibly set to zero by adding or subtracting some of the luminance signal to or from the color difference signal in accordance with the color temperature, thereby achieving white balance. This method is not correct in principle, and it is extremely difficult to accurately perform white balance in a wide color temperature range.

(B) Since the γ conversion is performed with the color difference, the color reproducibility is not good.

In the RGB primary color type camera of a three-tube camera, the output R, G, B color-separated according to the NTSC method is multiplied by γ to G ^γ , G
^gamma, after obtaining a B ^gamma, obtain two color difference R ^gamma -Y, a B ^gamma -Y. However, Y (luminance signal) is Y = 0.30R ^γ + 0.59G ^γ +0.11
B ^γ .

However, in a complementary color camera, the color signal is first subtracted and then multiplied by γ, so that (Mg−Gr) ^γ
Γ is applied in the form of the difference as shown. Therefore,
No matter how the correction is made later, the color signal corresponding to the regular NTSC cannot be obtained, and the color reproducibility is not good.

In order to solve the above problem, for example, as shown in FIG. 4, using two color difference signals C ₁ and C ₂ and a luminance signal Y _L ′ passed through a low-pass filter, conversion to RGB is performed by an appropriate operation. In this state, a method of performing white balance and γ conversion and converting the data again into luminance and color difference can be considered.

According to this, white balance, γ in the state of R, G, B
Since the conversion can be performed, the problems as in the example of FIG. 3 described above can be solved to some extent.

However, in this way, once the color difference of the output difference in the horizontal direction is created and the color processing is performed based on this,
There is a problem that optimal color processing matching the spectral sensitivity of the filter cannot be performed and color reproducibility is not improved.

Therefore, the following method is considered to improve color reproducibility. In the NTSC system, ideal spectral characteristics for the three primary colors R, G, and B are defined, and these are defined as r (λ), g
(Λ) and b (λ). On the other hand, when the sensor as shown in FIG. 2A is used, the spectral characteristics of each output of Mg, Gr, Cy and Ye are respectively expressed as Mg (λ), Gr (λ), Cy (λ) and Ye ( λ).

There is a function F at this time, If possible, a similar function F is applied to the sensor outputs Mg, Gr, Cy, and Ye.
, The ideal R, G, B signals of NTSC should be obtained.

In reality, it is difficult to satisfy the expression (1) for all the wavelengths λ, so the following method is used. F,
Consider approximation by (3 × 4) linear matrix A = (a _ij ). Spectral characteristics resulting from the conversion by A are r ′ (λ), g ′ (λ), and b ′ (λ).

Here, the error function E is defined as follows, for example.

N is an integer, usually 300 nm <λ ₁ , λ ₂ ,..., Λ _N <800
nm.

The E (A) may be determined to A so as to minimize.

(5) is a so-called normal equation, which is a 12-element simultaneous linear equation. If this equation is taken, (a _ij ) is determined, and a good approximation of F can be obtained by using this. Of course, the error function is the choice of how to E, which not only, wavelength λ ₁ and r the appropriate weighting, g, it may be put in between the b. Using the A determined in this way, the sensor output (Mg, Cr, Cy, Ye) And perform the necessary color processing such as gamma conversion and white balance based on these R, G, B, and finally By performing the conversion according to the NTSC standard, a desired luminance and color difference signal can be obtained.

Here, R ^γ , G ^γ , and B ^γ are white-balanced R,
This is a signal obtained by performing γ conversion on G and B at approximately γ = 0.45 v.

On the other hand, as in the conventional example shown in FIG. 4, color difference signals C ₁ and C ₂ are generated from the difference between horizontally adjacent pixels.
Let us consider a case where color processing is performed based on these.

The two color difference signals C ₁ and C ₂ in the case of the color arrangement shown in FIG. ₂ are as follows: C ₁ = Mg−Gr C ₂ = Cy−Ye (8)

On the other hand, since the low-frequency component Y _L ′ of the luminance signal is formed by an appropriate weighted average of Mg, Gr, Cy, and Ce, Y _L ′ = k ₁ Mg + k ₂ Gr + k ₃ Cy + k ₄ Ye (9) I can write

In this case, the RGB conversion unit 406 uses a (3 × 3) matrix B
Is converted from C ₁ , C ₂ , Y _L ′ to R, G, B by Becomes Summarizing equations (8), (9) and (10), Compare (11) and (6) so that B M = A
There is no problem if B can be determined, but this is not possible. Because the operation of first taking the difference in the horizontal direction is performed, the dimension of the color information is 4 to 3 at this point.
, Ie from (Mg, Cy, Ye, Gr) to (Y _L ′, C ₁ , C ₂ ) 1
This is because they are falling. Since the expression (6) is optimized for color reproduction in the linear range, the color reproduction is always better than the expression (11) ′.

[Problems to be solved by the invention]

Further, the coefficient [a _ij ] of the linear matrix A must satisfy the following condition. For example, when an achromatic subject is output by the filters shown in FIG. 2A, output signals Mg (λ), Gr (λ), Cy corresponding to each filter are output.
(Λ) and Ye (λ) are converted into a matrix [a _ij ] (i = 3, j = 4)
When converted, the RGB signal after conversion is R (λ) = a ₁₁ Mg (λ) + a ₁₂ Gr (λ) + a ₁₃ Cy (λ) + a ₁₄ Ye (λ) G (λ) = a ₂₁ Mg (λ ) + A ₂₂ Gr (λ) + a ₂₃ Cy (λ) + a ₂₄ Ye (λ) B (λ) = a ₃₁ Mg (λ) + a ₃₂ Gr (λ) + a ₃₃ Cy (λ) + a ₃₄ Ye (λ) (12) '. At this time, the position of Mg, Gr of the filter coincides with the dark part of the subject, and the position of Cy, Ye coincides with the bright part of the subject. Therefore, no matter what interpolation filter is used, Mg (λ) = αGr ( λ) = V ₁ (λ) Cy (λ) = βYe (λ) = V ₂ (λ) (13) Α and β are parameters depending on the color temperature of the subject.

From equation (12) ′ (13), R (λ) = (a ₁₁ + a ₁₂ / α) V ₁ (λ) + (a ₁₃ + a ₁₄ / β) V ₂ (λ) G (λ) = (a _{21 + a 22 / α) V 1} (λ) + (a 23 + a 24 / β) V 2 (λ) B (λ) = (a 31 + a 32 / α) V 1 (λ) + (a 33 + a 34 / β ) V ₂ (λ) …… (14) Since this subject is achromatic, all V ₁ (λ), V ₂
If (λ) is not R (λ) = G (λ) = B (λ), a false color appears in the vertical direction. Moreover, α and β change depending on the color temperature of the subject, and R (λ), G (λ), B
Since (λ) changes, there is a problem that a false color appears every time the color temperature changes with only one matrix conversion as in the related art.

[Means for Solving the Problems] An object of the present invention is to provide a matrix conversion unit that forms another color signal from color signals of a plurality of colors, which is a particular problem. Is to solve the problem that a false color in the vertical direction is generated due to the change of the color.

Therefore, in the imaging apparatus of the present invention, a color conversion matrix that forms a plurality of color signals of colors different from the plurality of colors by performing a matrix conversion of the plurality of color signals obtained from the imaging device using a predetermined matrix coefficient. Means for detecting a change in the color temperature of the subject; and correcting the matrix coefficient in accordance with the change in the color temperature of the subject detected by the detection means. Correction means for correcting so as to reduce false colors in the vertical direction generated in the color conversion matrix means, and a two-dimensional digital interpolation filter is provided between the image sensor and the color conversion matrix means. The output of the image sensor is synchronized and then input to the color conversion matrix means.

[Action]

As described above, a color conversion matrix unit that forms a plurality of color signals of a color different from the plurality of colors by performing a matrix conversion of the plurality of color signals obtained from the imaging device using a predetermined matrix coefficient, Detecting means for detecting a change in the color temperature of the subject; and correcting the matrix coefficient in accordance with the change in the color temperature of the subject detected by the detecting means, thereby changing the color conversion due to the change in the color temperature of the subject. Correction means for correcting so as to reduce the false color in the vertical direction generated in the matrix means, and a two-dimensional digital interpolation filter is provided between the image sensor and the color conversion matrix means, and the output of the image sensor is provided. Since the signals are input to the color conversion matrix means after synchronization, the circuit configuration can be simplified, and It is possible to effectively prevent the generation of a false color in the vertical direction due to a change in color temperature, which is a problem peculiar to a color conversion matrix unit that converts a color signal of a color into a matrix.

[Example] Hereinafter, the present invention will be described based on the present invention.

(First Embodiment) FIG. 1 shows an embodiment in which the present invention performs interlaced scanning of a CCD equipped with a color filter as shown in FIG. 2 (a).

In this case, the four color signals of Mg, Gr, Cy, and Ye must be synchronized. This is because these four pieces of information are converted into R, G, B color signals by calculation.

This can be done without any problem if the structure allows simultaneous reading of four lines, such as a MOS type sensor. However, in a sensor such as a CCD that cannot do this, first, each color signal is two-dimensionally converted. It is necessary to interpolate and synchronize.

For example, in the case of a sensor output as shown in FIG.
Note that the sampling position is indicated by a circle in FIG. 2 (b). Other crosses are
Although there is other color information, but there is no Mg color information, interpolation is performed with appropriate weighting of data with A marks (A to H, etc.). This is the synchronization by the two-dimensional interpolation filter. This is done for each color.

With the above in mind, a description will be given below with reference to FIG.

The CCD sensor 101 is provided with four types of color filters as shown in FIG. The image signal read from the sensor 101 by interlaced scanning is adjusted by the AGC 119 and then A / D converted by the A / D converter 102 at a timing synchronized with the read clock. For color processing to be performed later, the A / D converter 102 has good linear characteristics, and is preferably performed with 8 bits or more from the viewpoint of quantization error.

The luminance signal has a high-frequency component detected by a 116 high-pass filter, and a low-frequency component of luminance obtained by a method described later.
Y _L is added to the adder 117, D / A converted by the D / A converter 118, and output.

On the other hand, the output of the A / D converter 102 has four interpolation filters 10
6, 107, 108 and 109 are input. These four interpolation filters are configured, for example, as shown in FIG. 5, and their outputs are synchronized color signals Mg, Cy, Ye, and Gr, respectively. Here, the operation of the interpolation filter shown in FIG. 5 will be described.

Assuming that the output from the sensor 101 is interlaced, the output of the A / D converter 102 is (Mg
The output is switched between the (Gr) line and the (Cy / Ye) line. Therefore, for example, if the interpolation filter is Mg (Mg / Gr)
When the switch 501 selects the output of the A / D during the scanning of the line and selects zero in the next 1H, the switch 50
In the output of 1, the data of the (Mg / Gr) line and zero for 1H are alternately output every 1H.

1H memory 502,503, coefficient multiplier 504,505,506 and adder 50
7 forms a vertical interpolation filter. For example, if the coefficients of 504 and 506 are set to 1/2 and the coefficient of 505 is set to 1, the output of 507 outputs the data of the (Mg / Gr) line and the average value of the preceding and following (Mg / Gr) lines every 1H. And interpolated vertically.

Next, the output of the adder 507 is input to the switch 508.
The input of the switch 508 is a readout clock φ for each pixel.
Since the Mg signal and the Gr signal appear alternately in synchronism with, the output of 507 is selected and output for the Mg signal and zero for the Gr signal. This is Daylay 509-514, Coefficient 515
521 and an adder 521 are input to a horizontal interpolation filter and interpolated in the horizontal direction. The coefficients of the coefficient units 515 to 521 are
For example, it is preferable to make the sum of all two as (1/8, 2/8, 3/8, 1/2, 3/8, 2/8, 1/8).

In the above, the interpolation filter 106 for Mg has been described.
If the selection of the switch 508 is reversed, the interpolation filter 1 for Gr can be used.
If 09 reverses the selection of switch 501 again, switch 5
Interpolation filters 107 and 108 for Cy and Ye can be configured according to the phase of 08, respectively.

In the above description, two 1H memories are used (1/2, 1,
Although the (1/2) interpolation filter is constructed, N (1 + 1) tap FIR digital filters of (N + 1) taps may be used. In this case, the vertical color band is preferable. Such a configuration is difficult with analog processing, and a configuration based on digital processing is desirable.

In the above description, the case where the four interpolation filters 106 to 109 are individually configured is shown. However, if they are configured together as shown in FIG. 6, the 1H memory, the delay, the adder, and the coefficient unit can be shared. As a result, the circuit scale can be significantly reduced.

That is, the outputs of the A / D converter 102 in FIG. 6 are the same as those shown in FIG. 5 in 1H memories 502 and 503 and coefficient units 504, 505 and 506.
To the vertical interpolation filter consisting of The output of the adder 601 shows the average value of the preceding and succeeding lines every 1H. Now, 50
If the output of 5 is a (Mg / Gr) line, the adder 60
The output of 1 indicates the average value of the (Cy / Ye) line before and after. In the next line, the output of 505 is a (Cy / Ye) line, so the interpolated signals of the (Mg / Gr) and (Cy / Ye) lines are displayed alternately in F1 and F2 every 1H. Will be Therefore, by selecting F1 and F2 for each 1H by the switch 602, the interpolation signals (Mg / Gr) and (Cy / Ye) can be taken out in synchronization in the vertical direction.

The signal of each (Mg / Gr) or (Cy / Ye) line is
It is input to a horizontal interpolation filter composed of delays 509 to 514 and coefficient units 515 to 521 similar to those shown in FIG. Adders 603,604,606,
Since 607 adds the output of every two taps, the interpolated output of Mg and Gr alternately appears in the output of 603 and 604, for example, every clock φ. Therefore, if the outputs of 603 and 604 are switched every φ by the switch of 605, two-dimensionally interpolated Mg and Gr signals can be obtained. The same applies to Cy and Ye.

In FIG. 1, if the synchronized signals of Mg, Cy, Ye, and Gr are obtained, all the subsequent arithmetic processing may be performed once every several times of the read clock for each pixel. This is because the color signal band is generally narrow. Therefore, a thinning-out process may be performed after the interpolation filter, and the subsequent arithmetic processing may be performed at a relatively low speed. In this case, the power consumption can be significantly reduced.

 Next, the RGB conversion units 110 to 110-6 will be described.

As described above, first, each of the spectral characteristics Mg, Gr, Cy, and Ye
(Λ), Gr (λ), Cy (λ), Ye (λ) from 380 nm to 780 nm
Up to Mg (λi), Gr (λi), Cy (λ
i), Ye (λi) (i = 1,..., 41) were obtained.

Next, NTSC RGB ideal spectral characteristics r (λi), g (λ
i) and b (λi) are read from, for example, “Handbook of Color Chemistry, The University of Tokyo Press (1981)”, and a normal equation with the same weight given by equation (5) is obtained.

At this time, in order to prevent a false color in the expression (14), R
In order to satisfy (λ) = G (λ) = B (λ), the following two conditions are simultaneously satisfied. _{That, a 11 + a 12 / α} = a 21 + a 22 / α = a 31 + a 32 / α a 13 + a 14 / β = a 23 + a 24 / β = a 33 + a 34 / β ...... (15) where Since α and β change depending on the color temperature of the subject, 2000K, 3000K, 4000K, 5000K, 6000
Α, β of achromatic subject at each color temperature of ゜ K, 7000 ゜ K
Is measured.

Now, the output of Gr (λ) multiplied by α and the output of Ye (λ) multiplied by β are Gr ′ (λ) and Ye ′ (λ), respectively, instead of equation (6). Consider converting by In that case (15) is rewritten with _{a 11 + a 12 = a 21} + a 22 = a 31 + a 32 a 13 + a 14 = a 23 + a 24 = a 33 + a 34 ...... (17). Therefore, equation (4) becomes I can write However, Gr '(λ) is α times Gr (λ), and Ye' (λ) is β times ye (λ).

E ′ ( A ) = E ( A ) + l ₁ (a ₁₁ + a ₁₂ −a ₂₁ −a ₂₂ ) + l ₂ (a ₁₁ + a ₁₂ −a ₃₁ −a ₃₂ ) + l ₃ (a ₁₃ + a ₁₄ −a ₂₃₎ −a ₂₄ ) + l ₄ (a ₁₃ + a ₁₄ −a ₃₃ −a ₃₄ )... (19) If E ( A ) takes the minimum value, E ′ ( A ) = E ( A ) and takes the minimum value.

Solving this as a normal equation, as in equation (5), gives
[A _ij ] is a function of l ₁ to l ₄ .

Since the coefficient to be found satisfies the condition of (17), l ₁ , l
_2, l _3, the evaluation function at the time of setting the l ₄ F a (l) is defined as follows.

Here, l = (l ₁ , l ₂ , l ₃ , l ₄ ).

Now, set l ₁ , l ₂ , l ₃ , l ₄ to some initial l ₁₀ , l ₂₀ , l ₃₀ , l _40, and set the parameters of the set values to △ l ₁ , △ l ₂ , △ l ₃ , △ l ₄
A for each of the various l ₁ , l ₂ , l ₃ , l _4
ij ] is obtained by solving equation (20), and F ( l ) is obtained from equation (21) using this [a _ij ].

When a set of l = (l ₁ , l ₂ , l ₃ , l ₄ ) that minimizes this F ( l ) is obtained, [a _ij ] corresponding thereto is a coefficient to be obtained. Therefore, [a _ij ] has a different optimum value for each color temperature because equation (16) differs depending on α and β. Therefore, a plurality of RGB conversion units 110-1 to 110-6 are provided to perform different matrix calculations for each color temperature.

Switching of the RGB conversion matrix depending on the color temperature may be performed by switching a manual switch based on judgment from external light, but in this embodiment, the color temperature detector 120 determines the red light component and the blue light component from the spectral characteristics of the external light. And the switch is automatically switched.

For color reproduction, a fixed multiplier may be used by approximating the matrix coefficient, but in this embodiment, 2000 K, 3000 K, 4000 K, 50 K
R, G, B conversion tables suitable for 00 ゜ K, 6000 ゜ K, and 7000 ゜ K are used. If the conversion matrix is obtained for each finer step of the color temperature, more perfect signal processing is possible.

The R, G, and B signals converted and formed in this manner are controlled again by the color temperature detector 120 in the white balance circuit 111, and the γ conversion circuit
After being converted γ in 112, to form a low-frequency luminance signal Y _L by the color difference matrix circuit 113, the color difference signals R-Y, the B-Y. Low frequency luminance signal Y _L is guided by an adder 117 to form a luminance signal as described above.

The color difference signals RY and BY are respectively supplied to D / A converters 114 and 115.
To become an analog signal.

Second Embodiment The RGB conversion and the white balance in FIG. 1 may be shared. That is, in FIG. 1, the white balance is obtained by converting the RGB signal from RGB to xR, G, yB in the white balance circuit 111, which is converted to the RGB conversion units 110-1 to 110-6. Are variable multipliers, and the coefficients are variably controlled in accordance with the output of the color temperature detector 120. Specifically, instead of the matrix A obtained in the first embodiment, The coefficients x and y are determined according to the color temperature of 2000 K to 7000 K divided into 32 steps. More accurate color reproduction can be achieved by increasing the number of color temperature steps.

[Effects of the Invention] As described above, according to the present invention, a plurality of color signals different from the plurality of colors are obtained by performing a matrix conversion on a plurality of color signals obtained from the image sensor using a predetermined matrix coefficient. Color conversion matrix means for forming a color signal; detection means for detecting a change in the color temperature of the object; and correcting the matrix coefficient in accordance with the change in the color temperature of the object detected by the detection means. Correction means for correcting so as to reduce false colors in the vertical direction generated in the color conversion matrix means due to the change in color temperature of the color conversion matrix means. By providing a digital interpolation filter and synchronizing the output of the imaging device and then inputting it to the color conversion matrix means, In addition to simplifying the configuration, it is possible to effectively prevent the generation of a false color in the vertical direction due to a change in color temperature, which is a problem peculiar to the color conversion matrix unit that converts the color signals of a plurality of colors into a matrix. .

[Brief description of the drawings]

FIG. 1 is a block diagram of a first embodiment of the present invention, FIGS. 2 (a) and 2 (b) are diagrams showing an arrangement of color filters, FIG. 3 is a block diagram of a conventional example, and FIG. FIG. 5 is a block diagram of a configuration example of an interpolation filter, and FIG. 6 is a block diagram of another embodiment of the interpolation filter. 101: Sensor 106 to 109: Interpolation filter 110-1 to 110-6: RGB converter at each color temperature of 2000 to 7000K 120: Color temperature detector

Continuation of the front page (72) Inventor Akihiko Shiraishi 770 Shimonoge, Takatsu-ku, Kawasaki-shi, Kanagawa Prefecture Inside the Tamagawa Works of Canon Inc. (56) References JP-A-63-105591 (JP, A) JP-A-2-107091 (JP) JP-A-63-158990 (JP, A) JP-A-61-150489 (JP, A) JP-A-49-131536 (JP, A)

Claims (2)

(57) [Claims] 1. A color conversion matrix means for forming a plurality of color signals of a color different from the plurality of colors by subjecting a plurality of color signals obtained from an image pickup device to matrix conversion using a predetermined matrix coefficient. Detecting means for detecting a change in the color temperature of the subject; and correcting the matrix coefficient in accordance with the change in the color temperature of the subject detected by the detecting means, thereby changing the color conversion due to the change in the color temperature of the subject. Correction means for correcting so as to reduce the false color in the vertical direction generated in the matrix means, and a two-dimensional digital interpolation filter is provided between the image sensor and the color conversion matrix means, and an output of the image sensor is provided. An image pickup apparatus wherein the image data is input to the color conversion matrix means after synchronizing the image data. 2. The method according to claim 1, wherein the matrix coefficient is a reference spectral characteristic r.
The method according to claim 1, wherein (λ), g (λ), and b (λ) are represented by a linear combination of spectral characteristics f ₁ (λ)... F _N (λ) of _N color filters. Imaging device.