JPH11243554A - Signal interpolation method for single-plate color camera - Google Patents

Abstract

PROBLEM TO BE SOLVED: To interpolate the relevant color signal of a pixel string which does not include the pixels of a certain color signal and to decrease the false color signals that occur at the boundary parts of an object image, by extracting an interpolation signal obtained through multiplication of the coefficient signals corresponding to the levels of 1st and 2nd low frequency signals by each other, using the extracted interpolation signal as a signal existing at the position of a 1st pixel. SOLUTION: First and 2nd low frequency signals, corresponding to a 1st pixel position, are extracted from the signal obtained by adding together the signals of each prescribed pixel. Then an interpolation signal obtained by multiplying the coefficient signals, corresponding to the levels of the 1st and 2nd low frequency signals by each other, is extracted and used as a signal existing at the 1st pixel position. That is, the low frequency components of R and G signals are extracted from the outputs of a solid-state image-pickup device 1 and a 1H delay circuit 15 and is added to a divider 8 to obtain a signal that corresponds to the ratio set between both signals with regard to this camera. Then the obtained signal is multiplied by the G signal obtained from an 1H delay circuit 14 by a multiplier 9 and selected by a gate circuit 25 to have the R signal of a pixel string including no R pixel interpolated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a signal interpolation method for a single-chip color camera for reducing a false color signal generated at a boundary portion of a subject image in a reproduced image of a single-chip color camera. The present invention relates to a signal interpolation method for a single-chip color camera using a so-called Bayer array color filter in which certain transmitted light minute color filters are arranged in a checkered pattern.

[0002]

2. Description of the Related Art A single-chip color camera that obtains a color video signal by using only one solid-state imaging device having pixels arranged two-dimensionally is used in a wide range of fields such as a home video camera and an electronic still camera. ing. In a single-chip color camera, several types of minute color filters are periodically associated with each pixel of a solid-state imaging device, and a color signal corresponding to light transmitted by the minute color filter is obtained from each pixel. Therefore, since each color signal corresponds to a subject image at a spatially different position, a false color signal is likely to be generated at the boundary between the subject images.

The cause of the generation of a false color signal generated at the boundary of a subject image will be described using a so-called Bayer array primary color filter in which certain transmitted light microcolor filters are arranged in a checkered pattern. The primary color filter of the Bayer array generally has a configuration shown in FIG. In FIG. 2, R is red light, G
Is a small color filter that transmits green light and B is blue light, each of which corresponds to one pixel of the solid-state imaging device. Further, the numbers described above the arrangement of the micro color filters are horizontal position numbers, and the numbers described on the left side are vertical position numbers. Hereinafter, in FIG. 2, a pixel whose horizontal position number is 2n and whose vertical position number is 2m is referred to as a “pixel at a horizontal position 2n and a vertical position 2m”.

In FIG. 2, consider a case where an achromatic subject image is formed where a portion with diagonal lines is dark and a portion without diagonal lines is bright. Assuming that the output of each pixel corresponding to a bright portion is 1.0 for each of the R, G, and B signals, and the output of each pixel corresponding to a dark portion is 0.2 for each of the R, G, and B signals, a vertical position of 2 m The output of each pixel in the pixel column is as shown in FIG. This is separated into an R signal and a G signal, and a signal that does not exist at the position of the pixel in each color signal is linearly interpolated from a signal obtained at a preceding and succeeding pixel of that color as is generally performed. 3
The R signal shown in FIG. 3B and the G signal shown in FIG. 3C are obtained. As can be seen from FIGS. 3B and 3C, at the pixel position at the horizontal position 2 (n + 1) +1, the R signal becomes larger than the signal to be obtained as indicated by the dotted line, and the horizontal position 2 (n + 1) At the position of the pixel (n + 1), the magnitude of the G signal is smaller than the signal that should be originally obtained. As a result, a false color occurs at each position.

The color filter of the Bayer arrangement shown in FIG. 2 has the same structure as that of the original color filter, for example, rotated 90 degrees around the pixel at the horizontal position 2n and the vertical position 2m. Therefore,
Although the above description has been made with respect to the pixel rows in the horizontal direction, the same applies to the pixel rows in the vertical direction.

The generation process of the false color signal in the single-chip color camera has been described above using the spatial signal waveform.
Next, a process of generating a false color signal in a single-chip color camera using signals in the frequency domain will be described.

The spatial function of each color signal obtained by the Bayer array primary color filter shown in FIG. 2 is a function representing the spatial position of a pixel from which the color signal is obtained,
This is obtained by multiplying a function representing the magnitude of a component corresponding to the color signal of the subject image. Therefore, it is known from the Fourier transform theorem that the frequency component of each color signal is a convolution of the function representing the position in the frequency domain with the frequency component of each color component of the subject. Here, the superposition integral is a relationship represented by the following equation (1).

[0008]

(Equation 1)

The color filter shown in FIG. 2 has two pixels in the horizontal direction.
Since the minimum unit of two pixels in the vertical direction is repeated, four types of functions representing the spatial positions of the pixels may be considered. Here, the pixel interval in the horizontal direction is dx, the pixel interval in the vertical direction is dy, and the length of one side of a square pixel is dw.
Further, the center position of the pixel of R whose horizontal position and vertical position are both 0 is set to the spatial coordinates (0, 0).

An R signal is obtained from a pixel at a horizontal position 2n and a vertical position 2m (n and m are arbitrary integers). If the two-dimensional frequency component is Sr (fx, fy), the following equation 2 is obtained. . Further, a G signal is obtained from the pixel at the horizontal position 2n + 1 and the vertical position 2m. If the two-dimensional frequency component is Sg1 (fx, fy), the following equation 3 is obtained. Similarly, a G signal is obtained from a pixel at a horizontal position 2n and a vertical position 2m + 1, and its two-dimensional frequency component is represented by S
If g2 (fx, fy) is given, the following equation 4 is obtained. In addition, horizontal position 2
From the pixel at n + 1 and the vertical position 2m + 1, a B signal is obtained. If the two-dimensional frequency component is Sb (fx, fy), the following equation 5 is obtained.

[0011]

(Equation 2)

[0012]

(Equation 3)

[0013]

(Equation 4)

[0014]

(Equation 5)

In Equations 2 to 5, fx and fy are frequencies in the horizontal and vertical directions, respectively. Frx and Fry are the R components of the subject image, Fgx and Fgy are the G components of the subject image, Fbx and Fby
Is a function representing the horizontal frequency component and the vertical frequency component of the B component of the subject image, respectively. Also,
“*” Represents the convolution integral shown in Equation 1. The delta function is a function representing a position where a harmonic component generated due to sampling is generated. Fax and Fay are frequency responses generated due to the aperture of the pixel, and are as shown in the following Expressions 6 and 7.

[0016]

(Equation 6)

[0017]

(Equation 7)

From the comparison between Equations 2 and 3, or between Equations 4 and 5, if the horizontal position differs by dx, the phase of the horizontal harmonic component generated due to sampling will be
It turns out that it rotates by -jkπ (k is an integer). Similarly, from the comparison of Equations 2 and 4 or the comparison of Equations 3 and 5, if the vertical position differs by dy, the phase of the vertical harmonic component generated due to sampling becomes −jlπ (l Is an integer).

Here, for example, the vertical position shown in FIG.
Notice only the 2m pixel column. Since it is not necessary to consider changes in the vertical direction on a pixel column having the same vertical position, only the horizontal frequency components need to be considered. At this time, Expressions 2 and 3 can be rewritten as Expressions 8 and 9 below.

[0020]

(Equation 8)

[0021]

(Equation 9)

In Equations 8 and 9, the subject image is achromatic and Fg
It is assumed that x (fx) = Frx (fx) and the frequency component in the horizontal direction is as shown in FIG. At this time, Equations 8 and 9
The horizontal frequency components Sr (fx) and Sg1 of each signal indicated by
(fx) is as shown in FIGS. 4B and 4C, respectively. The phase rotation of -jkπ of the harmonic component in Equation 9 is 1 / 2dx as is clear from the comparison between FIG. 4B and FIG. 4C.
, And the higher harmonic components generated around the center have the opposite polarity.

Here, when the frequency range from 0 to 1 / 4dx, which is indicated by oblique lines in Sr (fx) and Sg1 (fx), is extracted with a low-pass filter and used for a color signal, the frequency region centered on 1 / 2dx is generated. Some of the higher harmonic components mixed into the effective band. The frequency components mixed into the effective band are higher than 1 / 4dx of the subject image, but in the subject image having a wide range of frequency components such as a stepwise change in brightness, the components mixed into the effective band are ignored. It becomes impossible size. And Sr (fx) and Sg
At 1 (fx), the harmonic components are mixed in opposite phases, and the influence appears in the opposite direction, so that a false color signal corresponding to the size of the mixed components is generated.

The cause of the generation of the false color signal has been described in the frequency domain. The false color signal generated at the boundary portion of the achromatic subject image is particularly noticeable visually since the originally achromatic portion is colored and reproduced. On the other hand, a false color signal in which the hue at the boundary portion of a high-saturation subject image changes generally becomes inconspicuous because the frequency band of the color signal is limited to a narrow band. Therefore, a greater improvement effect can be expected by suppressing a false color signal generated at the boundary portion of the achromatic subject image than at the boundary portion of the high saturation subject image.

A first conventional method for reducing false color signals is a method disclosed by Japanese Patent Laid-Open No. 3-124190 by the inventor of the present invention. The method of JP-A-3-124190 is
For example, the ratio between the low-frequency component of Sr (fx) and the low-frequency component of Sg1 (fx) described above is multiplied by Sg1 (fx), and the obtained signal is used as an interpolation signal of Sr (fx). Things. Hereinafter, the method of Japanese Patent Application Laid-Open No. 3-124190 will be described with reference to the configuration diagram shown in FIG.

In the configuration shown in FIG. 5, for example, the operation of reducing the false color signal in the pixel column at the vertical position 2m in FIG. 2 is as follows. In FIG. 5, it is assumed that the solid-state imaging device 1 is combined with the Bayer array primary color filter shown in FIG. Further, it is assumed that the solid-state imaging device 1 is of a so-called all-pixel sequential reading type in which pixel columns in the vertical direction are sequentially read one by one.

The pixel signal obtained from the solid-state imaging device 1 is
R signal sr (x) in addition to sampler 2 and sampler 3
And a G signal sg1 (x). Further, the R signal and the G signal obtained from the sampler 2 and the sampler 3 are added to a low-pass filter 4 and a low-pass filter 5, respectively, which pass frequency components lower than fa. Here, the low-pass filter 4
Output signals srl (x), sg1l obtained from low-pass filter 5
The frequency components Srl (fx) and Sg1l (fx) of (x) are shown in FIG.
(D) and those shown by oblique lines in FIG. FIG.
(D), Srl (fx), Sg1l (f
x) are the frequency components Fr (fx) and Fg (fx) of the subject, respectively.
Of the components in the frequency range lower than fa, Frl (fx), Fgl (f
x) and harmonic components centered at 1 / 2dx and mixed into the band below fa, components in the frequency range higher than (1 / 2dx-fa)
Frh (fx) and Fgh (fx) are added. That is, Srl
(fx) and Sg1l (fx) have the relationship of Expression 10 and Expression 11.

[0028]

(Equation 10)

[0029]

[Equation 11]

FIG. 4 (d), (e) or equations (10) and (1)
If Frh (fx) and Fgh (fx) are smaller than 1, Srl (fx) and Sg1l
(fx) are the low-frequency components Frl (fx) and Fgl (f
x). When the change in the hue of the subject image in the local area is small, it can be expected that the frequency components of any two color signals around the area have a substantially constant ratio in the entire frequency range. When these conditions are satisfied, the ratio of the low frequency components becomes substantially equal to the ratio of the entire frequency components, and the following equation 12 holds.

[0031]

(Equation 12)

Further, according to Percival's theorem shown in the following equation 13, the power value obtained by integrating the function in the space domain is equal to the power value obtained by integrating the function in the frequency domain. Therefore, it is expected that the power value obtained by integrating the function in the space domain and the power value obtained by integrating the function in the frequency domain are close to each other even within a finite integration range. We expect that there is a correlation with the frequency component at.

[0033]

(Equation 13)

In the configuration shown in FIG. 5, the output signals of the low-pass filter 4 and the low-pass
And the gate circuit 7. Further, the output signals of the gate circuits 6 and 7 are applied to the divider 8. Here, the gate circuit 6 and the gate circuit 7 have a vertical position of 2 m, for example.
When the R signal is obtained from the solid-state imaging device 1 in the pixel row of, the output srl (x) of the low-pass filter 4 is added to the divisor side of the divider 8 and the output sg1l (x) of the low-pass filter 5 is added to the dividend side. Works. When a G signal is obtained from the solid-state imaging device 1, the operation is performed such that the output sg1l (x) of the low-pass filter 5 is added to the divisor side of the divider 8 and the output srl (x) of the low-pass filter 4 is added to the dividend side. I do. The result divider 8 outputs a signal corresponding to a value obtained by dividing Sg1l (fx) by Srl (fx) when an R signal is obtained from the solid-state imaging device 1, and outputs Srl (fx) when a G signal is obtained. Outputs a signal corresponding to the value divided by Sg1l (fx). Further, the output signal of the divider 8 is added to the multiplier 9 together with the pixel signal obtained from the solid-state imaging device 1 to obtain an interpolation signal sig (x).

As a result, for example, R
When a signal is obtained, the frequency component Sig (fx) of the interpolation signal sig (x) obtained from the multiplier 9 is represented by the following Expression 14.

[0036]

[Equation 14]

When Expressions 8 and 12 are used in Expression 14, Expression 14
Is rewritten as shown in the following Expression 15, so that Sig (fx) is as shown in FIG.

[0038]

(Equation 15)

Note that eg in Equation 15 is a coefficient representing the accuracy of the interpolation signal, and has a relationship of Equation 16 which is next to Equations 10, 11 and 12 below. From Equation 16, as the component of the band lower than fa of the subject image is larger than the component of the band higher than 1 / 2dx-fa, eg approaches 1 and the accuracy of the interpolation signal is improved.

[0040]

(Equation 16)

Comparison between Equation 15 and Equation 9 or FIG.
From the comparison between FIG. 6C and FIG. 6A, when eg is close to 1, the frequency component of the interpolation signal obtained from the multiplier 9 is an odd harmonic of the frequency component Sg1 (fx) of the G signal. It can be seen that the components have opposite phases. As a result, the frequency components of the G signal obtained by selecting and adding the G signal sg1 (x) obtained from the solid-state imaging device 1 and the interpolation signal sig (x) obtained from the multiplier 9 by the gate circuit 10 are odd-ordered. 6 (b) are almost canceled out.

Similarly, when the G signal is obtained from the solid-state imaging device 1, the multiplier 9 determines that the frequency component Sr (fx) of the R signal sr (x) has an odd-order harmonic component in phase opposite to that of the frequency component Sr (fx). A certain interpolation signal is obtained. As a result, the frequency components of the R signal obtained by selecting and adding the R signal obtained from the solid-state imaging device 1 and the interpolation signal obtained from the multiplier 9 by the gate circuit 11 and odd-order harmonic components are almost canceled. 6 (c).

The outputs of the gate circuits 10 and 11 obtained as described above are applied to the low-pass filter 12 and the low-pass filter 13, and a frequency band of 1 / 4dx or less is extracted to obtain a G signal and an R signal. As can be seen from FIGS. 6B and 6C, the G signal and the R signal have 1 /
Harmonic components generated around 2dx are sufficiently suppressed, and a false color signal resulting therefrom is improved.

In the above description, a pixel row in which the vertical position is 2 m and R and G are repeated is described as an example.
In the 2m + 1 pixel column, similar interpolation processing can be realized between the G signal and the B signal. However, there is no description of a method of interpolating a B signal whose vertical position does not exist in the pixel column of 2m or an R signal which does not exist in the pixel column of 2m + 1, and each color corresponding to every pixel column is not described. I can't get a signal.

On the other hand, a second conventional method of interpolating a B signal of a pixel row in which R and G are repeated or an R signal of a pixel row in which G and B are repeated in a Bayer array primary color type filter is notable. No. 7-236147. Japanese Patent Application Laid-Open No. 7-236147 describes an interpolation processing means for obtaining a plurality of color signals for a pixel by using signals of a plurality of pixels present around a certain pixel. At the same time, a method is described in which means for obtaining correlation values in the vertical direction and the horizontal direction of the pixel to be subjected to the interpolation processing are provided, and the operation of the interpolation processing is controlled based on the correlation. This method will be described with reference to FIG.

FIG. 7A shows, for example, nine pixels centered on the pixel at the horizontal position 2 (n + 1) and the vertical position 2m + 1 of the Bayer array primary color filter shown in FIG. . Here, the interpolation signals Rh, Gh, and Bh at the position of G22 when it is assumed that the change of the subject image in the horizontal direction is small are calculated by the following equations (17), (18), and (19).

[0047]

[Equation 17]

[0048]

(Equation 18)

[0049]

[Equation 19]

The interpolation signals Rv, G at the position G22 when it is assumed that the change of the
v and Bv are obtained by the following calculations of Expressions 20, 21, and 22.

[0051]

(Equation 20)

[0052]

(Equation 21)

[0053]

(Equation 22)

On the other hand, the vertical correlation coefficient Sv for determining the correlation in the vertical direction and the horizontal correlation coefficient Sh for determining the correlation in the horizontal direction are obtained by the following Expressions 23 and 24 or Expressions 25 and 26. Equations 23 and 24 are applied when the saturation of the subject image is high,
Equations 25 and 26 apply to achromatic colors. Further, the following Expressions 27 and 28 are obtained from the vertical correlation coefficient Sv and the horizontal correlation coefficient Sh.
Then, a horizontal coefficient Kh and a vertical coefficient Kv are obtained.

[0055]

(Equation 23)

[0056]

(Equation 24)

[0057]

(Equation 25)

[0058]

(Equation 26)

[0059]

[Equation 27]

[0060]

[Equation 28]

The coefficient Kh in the horizontal direction is an amplification factor of the interpolation signals Rh, Gh, and Bh obtained assuming that the change in the image of the subject in the horizontal direction is small. The coefficient Kv in the vertical direction is a coefficient of the change in the image of the subject in the vertical direction. This is the amplification factor of the interpolation signals Rv, Gv, Bv obtained as a small value. Rh and Rv, Gh amplified by each coefficient
Gv, Bh and Bv are added to obtain a final interpolation signal. Number 2
3, 24, 25, 26, 27, and 28, for example, in a subject image in which only a change in the vertical direction is large.
Since the value of Sv increases and Sh becomes 0, Kh = 1 and Kv = 0
Becomes At this time, the final interpolation signals are interpolation signals Rh, Gh,
It consists of Bh only. On the other hand, in a subject image in which only a change in the horizontal direction is large, the value of Sh becomes large, and Sv becomes zero.
Kh = 0 and Kv = 1. At this time, the final interpolation signal is
It is composed of only the interpolation signals Rv, Gv, and Bv obtained assuming that the change of the subject image in the vertical direction is small.

FIG. 7B shows the horizontal position in FIG.
9 pixels centered on the pixel at 2 (n + 1) +1 and the vertical position 2m + 1. Interpolation signal R assuming that the horizontal subject image change at the position of B22 is small.
h, Gh, and Bh are calculated by the following equations (29), (30), and (31), and the interpolation signals Rv, Gv, and Bv when the change in the subject image in the vertical direction is assumed to be small are: , 33, and 34.

[0063]

(Equation 29)

[0064]

[Equation 30]

[0065]

(Equation 31)

[0066]

(Equation 32)

[0067]

[Equation 33]

[0068]

(Equation 34)

In FIG. 7A, it is assumed that an achromatic subject image is formed in which a portion indicated by oblique lines is dark and a portion without oblique lines is bright. The magnitude of the signal in the dark part is R,
G and B are both 0.2, and the magnitude of the signal in the bright part is R,
Assuming that both G and B are 1.0, an interpolation signal is obtained according to Equations 17 to 28. At this time, Kh = 1, Kv = 0
Thus, Rh = 1.0, Gh = 1.0, and Bh = 1.0 are output as interpolation signals, which respectively match the color signals that should be obtained. However, it is assumed that the subject image shown in FIG. 7A is green, G in a portion without oblique lines is 1.0 and R and B are 0.2, and R, G and B in portions indicated by oblique lines are 0.2. When the interpolation signal is obtained according to Equations 17 to 28, Kh = 1 and Kv = 0, so that Rh = 1.0, Gh = 1.0, and Bh = 0.2 are output as interpolation signals. At this time, a false color signal is generated unlike the signal that should be obtained from the R signal.

As described above, in the second conventional method described in Japanese Patent Application Laid-Open No. Hei 7-236147, for example, an R signal corresponding to a pixel row having no R pixel is interpolated to generate a false color in an achromatic subject image. Although the color signal can be reduced, there is a problem that a false color signal cannot be prevented in a highly saturated subject image. Further, the method described in Japanese Patent Application Laid-Open No. Hei 7-236147 requires a complicated correlation detection method, such as changing the correlation determination method between an achromatic subject image and a high-saturation subject image, thereby increasing the circuit scale.

The above description is made with reference to FIGS. 7A and 7B.
As an example, the case where the interpolation processing is performed in the range of three pixels in the horizontal direction as shown in FIG. 1 is described. Japanese Patent Application Laid-Open No. Hei 7-236147 also describes the interpolation processing using the pixels in the range of five pixels in the horizontal direction. However, a false color signal cannot be reduced in a portion where a highly saturated subject image changes, so that the detailed description is omitted.

On the other hand, a third conventional method for interpolating a B signal of a pixel row in which R and G are repeated or an R signal of a pixel row in which G and B are repeated in a Bayer array primary color filter is disclosed in -501424.
The method of JP-A-61-501424 will be described below with reference to FIG.

First, the G signals at the positions of the R and B pixels in FIG. 8 are interpolated by the average value of the signals of the four G pixels at the top, bottom, left and right. That is, for example, the G signal at the position of B9 indicated by oblique lines in FIG. 8 is interpolated by G9 expressed by the following Expression 35.

[0074]

(Equation 35)

The method of interpolating the R and B signals after interpolating the G signal is as follows. When the left and right pixels at the position to be interpolated are the pixels of the color to be interpolated, for example, the R signal R2 at the position G2 indicated by the sand pattern in FIG.
It is obtained by the following equation (36).

[0076]

[Equation 36]

For example, when interpolating an R signal in a pixel row having no R pixel, if the upper and lower pixels are R pixels, for example, the R signal R5 at the G5 position shown by the double frame in FIG. Is obtained by the following equation (37).

[0078]

(37)

If the pixels in the four oblique directions are the pixels of the color to be interpolated, for example, B9 in FIG.
The R signal R9 at the position is obtained by the following equation (38). Similar processing may be performed when interpolating a B signal in a pixel column having no B pixel.

[0080]

(38)

Here, when the periphery of the pixel G5 shown in FIG. 8 (a) is extracted and shown in FIG. 8 (b), a shaded portion is dark, and a portion without the shaded portion is a bright green subject image. Suppose you image. G, R, and B in the shaded area are 0.2,
When G in the unshaded portion is 1.0 and R and B are 0.2, when R, G, and B at the position of G5 are obtained according to Formulas 35 to 37, G5 = 1.0, R5 = 0.35, and B5 = 0.25. Although the R signal is slightly larger than the signal to be originally obtained and the B signal is slightly smaller, it is greatly improved as compared with the above-mentioned second conventional method.

On the other hand, in FIG. 8B, G, R, and B in the hatched portions are 0.2, and R, G, and B in the non-hatched portions are 0.2.
It is assumed that an achromatic subject image of 1.0 is formed. Number 35
When R, G, and B at the position of G5 are obtained according to Expression 37, G5 = 1.0, R5 = 0.75, and B5 = 1.25. As described above, the R signal is slightly smaller than the signal to be originally obtained, and the B signal is slightly larger, so that the false color signal is larger than that of the second conventional method.

[0083]

As described above, in the above-mentioned first conventional method, the B signal of the pixel row in which R and G are repeated or G and B in the primary color filter of the Bayer array is used. There is a problem that it does not correspond to the process of interpolating the R signal of a repeated pixel row. Further, the above-mentioned second conventional method interpolates the B signal of a pixel row in which R and G are repeated and the R signal of a pixel row in which G and B are repeated in a Bayer array primary color filter to achromatic color. Although false color signals in a subject image can be sufficiently reduced, there is a problem that a large false color signal is generated in a highly saturated subject image. It also has the problem that the method of detecting the correlation is complicated. Furthermore, in the third conventional method described above, false color signals can be reduced both in the case of an achromatic subject image and in the case of a highly saturated subject image, but there is a problem that the amount of reduction is insufficient. Was.

An object of the present invention is to provide a color filter in which a color signal of a certain color signal does not exist even when a color filter of a Bayer array is used, and a false color when a subject image is achromatic. It is an object of the present invention to provide a signal interpolation method for a single-chip color camera, in which the generation of a color signal is sufficiently reduced and a false color signal is not greatly increased even when the saturation of a subject image is high.

[0085]

In order to achieve the above object, in a signal interpolation method for a single-chip color camera according to the present invention, a plurality of pixel groups having different spectral sensitivity characteristics are two-dimensionally arranged. Of the pixel groups corresponding to the first type
Are arranged in a relationship of interpolating each other between adjacent pixel columns, and at least pixels of a second pixel group corresponding to a second type different from the first type among the plurality of types of pixel groups are In the signal interpolation method for a single-chip color camera that generates at least the second type of signal corresponding to at least all pixel columns by using an imaging element arranged so as to exist every other pixel column, the second pixel group Takes out a first signal which is a signal of the first pixel group at a position of a first pixel present in a first pixel row arbitrarily selected from a pixel row in which the first pixel row does not exist, and is adjacent to the first pixel row. And a signal of a pixel belonging to the first pixel group on the second pixel column and the first signal on a third pixel column adjacent to the first pixel column and different from the second pixel column. From the signal obtained by adding the signals of the pixels belonging to the pixel group , And a signal of a pixel belonging to the second pixel group on the second pixel column and belonging to the second pixel group on the third pixel column. From the signal obtained by adding the pixel signals, the first
Extract a second low-frequency signal corresponding to the position of the pixel
A first interpolation signal obtained by multiplying the first signal by a first coefficient signal corresponding to the magnitude of the second low frequency signal with respect to the first low frequency signal is taken out, and the position of the first pixel is obtained. Is used as the second type of signal. At this time, when the first pixel belongs to the first pixel group, the signal of the first pixel is used as the first signal, and the first pixel does not belong to the first pixel group. Sometimes it is present on the first pixel column,
A second pixel belonging to the first pixel group adjacent on both sides to the pixel
It is desirable that the average signal of the signals of the pixel and the third pixel be the first signal.

Further, the image pickup device may be a combination of an optical low-pass filter for lowering the resolution in the pixel row direction.

Further, a plurality of types of pixel groups having different spectral sensitivity characteristics are two-dimensionally arranged, and a first pixel group corresponding to a first type among the plurality of types of pixel groups is mutually connected between adjacent pixel columns. The pixels of a second pixel group corresponding to a second type different from the first type among at least the plurality of types of pixel groups are arranged so as to be present in every other pixel column. A signal interpolation method for a single-chip color camera in which an image sensor is used to interpolate signals of the second type corresponding to positions of all pixels except the second pixel group by interpolation signals generated by at least two methods. And a signal of a pixel belonging to the first pixel group on a second pixel row adjacent to the first pixel row including a first pixel arbitrarily selected from the pixels not belonging to the second pixel group. Adjacent to the first pixel column, and Generating a first low-frequency signal corresponding to the position of the first pixel from a signal obtained by adding signals of pixels belonging to the first pixel group on a third pixel column different from the pixel column of From the signal obtained by adding the signal of the pixel not belonging to the first pixel group on the second pixel column and the signal of the pixel not belonging to the first pixel group on the third pixel column, the first
And generates a second low-frequency signal corresponding to the position of the pixel, and passes through the second pixel existing on the first pixel column and adjacent to the first pixel to the first pixel column. The signals of the pixels belonging to the first pixel group existing on the first line in the vertical direction and the second pixels are present on the first pixel column with respect to the first pixels. From the signal obtained by adding the signals of the pixels belonging to the first pixel group, which passes through the third pixel adjacent on the opposite side and is on the second line in the direction perpendicular to the first pixel column, And generating a third low-frequency signal corresponding to the pixel of the first pixel group and a signal of a pixel that does not belong to the first pixel group existing on the first line and the first low-frequency signal existing on the second line. A fourth low-frequency signal corresponding to the first pixel is generated from a signal obtained by adding signals of pixels that do not belong to the pixel group. A first coefficient signal corresponding to a ratio of an absolute value of a difference between the first low-frequency signal and the second low-frequency signal to an added value of the first low-frequency signal and the second low-frequency signal is calculated. A second low-frequency signal corresponding to a ratio of the absolute value of the difference between the third low-frequency signal and the fourth low-frequency signal to the sum of the third low-frequency signal and the fourth low-frequency signal. Generating a coefficient signal; generating a first amplification factor corresponding to a ratio of the first coefficient signal to an added value of the first coefficient signal and the second coefficient signal; Generating a second amplification factor corresponding to a ratio of the second coefficient signal to an added value of the second coefficient signal, and using the first amplification factor and the second amplification factor to perform the two methods; May be combined with the interpolation signal generated in step (1).

At this time, as a preferred configuration method, the interpolation signal generated by the above two methods is such that the signal corresponding to the ratio of the second low frequency signal to the first low frequency signal is the first pixel column. A first interpolation signal obtained by multiplying a signal corresponding to the first type at a position of the first pixel obtained from a signal of a pixel belonging to the first pixel group present above;
The signal corresponding to the ratio of the fourth low-frequency signal to the third low-frequency signal passes through the first pixel and passes through the first low-frequency signal.
Multiplied by the signal corresponding to the first type at the position of the first pixel obtained from the signal of the pixel belonging to the first pixel group existing on the third line in the direction perpendicular to the pixel column of A second interpolation signal, wherein the first interpolation signal is amplified at the second amplification degree, and the second interpolation signal is converted to the first interpolation signal.
Amplification with an amplification degree of

[0089]

With the above arrangement, a value close to the component ratio of the subject image can be obtained from the ratio of the output signals of the low-pass filter.
Even when a Bayer array color filter is used, it is possible to generate an interpolation signal that is sufficiently close to the color signal to be originally obtained in a pixel row in which a pixel of a certain color signal does not exist. As a result, the generation of a false color signal when the subject image is achromatic is sufficiently reduced, and at the same time, the false color signal does not greatly increase even when the subject image has high saturation. A signal interpolation method can be realized.

Also, when the two interpolation signals are weighted to generate the final interpolation signal, the amplification degree of the weight is obtained by using the output signal of the low-pass filter used for generating the interpolation signal. The configuration is not complicated.

[0091]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to the configuration diagram shown in FIG. In FIG. 1, a solid-state imaging device 1 is a combination of primary color filters of the Bayer array shown in FIG.

In the present invention, in the configuration of FIG. 1, the output of the solid-state imaging device 1 is set to 1H for a delay time equal to one horizontal scanning period.
It is added to the delay circuit 14. Further, the output signal of the 1H delay circuit 14 is applied to the 1H delay circuit 15. Thus, for example, when the signal of the pixel column at the vertical position 2 (m + 1) shown in FIG. 2 is obtained from the solid-state imaging device 1, the 1H delay circuit 14 outputs the signal of the pixel column at the vertical position 2m + 1, and 1H delay. The signal of the pixel column at the vertical position 2 m is simultaneously obtained from the circuit 15.

On the other hand, the signal of the pixel row at the vertical position 2 (m + 1) obtained from the solid-state imaging device 1 is applied to the samplers 2 and 3, and is separated into R and G signals, respectively. Similarly, a signal of a pixel row at a vertical position of 2 m obtained from the 1H delay circuit 15 is applied to a sampler 16 and a sampler 17, and is separated into an R signal and a G signal, respectively. Sampler 2,
Outputs of the samplers 3, 16 and 17 are respectively applied to a low-pass filter 4, a low-pass filter 5, a low-pass filter 18, and a low-pass filter 19 that pass a band lower than the frequency fa, and a low-frequency component is extracted. The output of the low-pass filter 4 and the output of the low-pass filter 18 obtained at this time are applied to an adder 20 to output the output of the low-pass filter 5 and the low-pass filter 19.
Is applied to the adder 21. As a result, adder 2
From 0, the low frequency component of the R signal of the pixel column at the vertical position 2m,
An addition signal of the low frequency component of the R signal of the pixel row at the vertical position 2 (m + 1) is obtained. Similarly, the adder 21 obtains an addition signal of the low-frequency component of the G signal of the pixel row at the vertical position 2m and the low-frequency component of the G signal of the pixel row at the vertical position 2 (m + 1).

Here, considering the signal of the pixel row at the vertical position 2m + 1 obtained from the 1H delay circuit 14 as a reference, the signal of the pixel row at the vertical position 2 (m + 1) after 1H is-in the vertical direction. In the case of using by shifting by dy, the signal of the pixel column at the vertical position 2m before 1H corresponds to using by shifting by + dy in the vertical direction. That is, if the low-frequency signal of the R signal of the pixel column at the vertical position 2m and 2 (m + 1) is srl (x, y), the adder 2
Sra (x, y) obtained from 0 is as shown in the following Expression 39.

[0095]

[Equation 39]

To determine the change in frequency response due to the shift of y, srl (x, y) = exp (jωy). As a result,
Equation 39 is rewritten as the following Equation 40, and the vertical position is 2m
The frequency response obtained by adding the pixel row of (1) and the pixel row of 2 (m + 1) and using it at the vertical position 2m + 1 is cosωdy.
That is, the frequency component Sra (fx, fy) of the signal sra (x, y) obtained from the adder 20 is obtained by limiting the horizontal band of Sr (fx, fy) with a low-pass filter, 1 /
In the case of 4dy, it is obtained by multiplying the frequency response cosωdy in the vertical direction where the response becomes 0. Adder 21
The same applies to the frequency component of the G signal obtained from.

[0097]

(Equation 40)

The 1H delay circuit 14, 1H delay circuit 1
5 is set to exactly one horizontal scanning period, the signal obtained from the solid-state imaging device 1 and the 1H delay circuit 1
The horizontal position of the pixel in FIG. 2 of the signal obtained from 5 is the same. Therefore, in the configuration of the second embodiment of the present invention shown in FIG. 9, the output of the solid-state imaging device 1 and the output of the 1H delay circuit 15 are added by the adder 20. The output of the adder 20 is, for example, the sum of pixels of the same color and the same color in the pixel row at the vertical position 2m and the pixel row at the vertical position 2 (m + 1). In addition to R, G signals can be collectively separated into R signals and G signals. At this time, the signals obtained by adding the outputs of the samplers 2 and 3 to the low-pass filters 4 and 5 are as shown in FIG.
It is evident that the signals obtained from the adders 20 and 21 are the same. Therefore, the description after the low-pass filter 4 and the low-pass filter 5 in the configuration of FIG. 9 is the same as that of the configuration of FIG. 1 described below.

By the way, the pixel row at the vertical position 2m and the vertical position
The frequency component Sr (fx, fy) of the R signal of the 2 (m + 1) pixel column is represented by Expression 2, and the frequency component Sg of the G signal
1 (fx, fy) was represented by Expression 3. Further, the frequency component Sg2 (fx, fy) of the G signal of the pixel row at the vertical position 2m + 1 is represented by Expression 4, and the frequency component Sb of the B signal
(fx, fy) was represented by Expression 5. Here, the subject image is achromatic and the two-dimensional frequency component of the subject image
When F (fx, fy) is a circle shown in FIG.
(fx, fy), Sg1 (fx, fy), Sg2 (fx, fy), Sb (fx, fy) have harmonic components as shown in FIGS. 10 (b) to 10 (e), respectively. It will be. In FIG. 10, the frequency component indicated by the hatched circle indicates that the phase is opposite to the frequency component of the subject image.

Comparison of Equations 2 and 3 or FIG. 10 (b)
From the comparison between FIG. 10C and FIG. 10C, in Sr (fx, fy) and Sg1 (fx, fy), the harmonic components generated at odd-order horizontal frequencies have opposite phases.
It can be seen that the phases of the harmonic components generated at the vertical frequency match. Here, if the horizontal frequency response of the low-pass filter 4, the low-pass filter 5, the low-pass filter 18, and the low-pass filter 19 is as shown in FIG. 11A, the signals obtained from the adders 20 and 21 are respectively shown in FIG. 11 (b) and 11 (c), the horizontal frequency band is limited to fa or less.

In the configuration shown in FIG. 1, the low-frequency component srl (x, y) of the R signal obtained from the adder 20 is added to the dividend side of the divider 8, and the low-frequency component of the G signal obtained from the adder 21 is added. The component sg1l (x, y) is added to the divisor side of the divider 8. FIG.
1 (b) and srl (x, y) and sg1l as shown in FIG.
Since the harmonic components of the vertical frequency of (x, y) have the same phase, the influence of the mixed components is in the same direction. Therefore, if the subject image is an achromatic color, the factor that the ratio of the output signals of the adder 20 and the adder 21 differs from the ratio of the R component and the G component of the subject image is an odd harmonic component in the horizontal frequency. The only effect is. This is the same as the case of considering only one pixel row in the first conventional method, and the adder becomes smaller as the component of the subject image in the frequency domain exceeding (1 / 2dx-fa) in the horizontal direction is smaller. 20 and the ratio of the low frequency component of the R signal obtained from the adder 21 and the low frequency component of the G signal are close to the ratio of the R component and the G component of the subject image.

On the other hand, the signal obtained from the 1H delay circuit 14 is applied to the phase adjuster 22. Further, the phase adjuster 22
Is output to the samplers 23 and 24, and when the signal of the pixel column at the vertical position 2m + 1 is obtained from the 1H delay circuit 14, for example, the signal is separated into a G signal and a B signal. The phase adjuster 22 compensates for a signal delay generally generated by a horizontal low-pass filter, and as a result, a signal obtained from the divider 8 and a signal obtained from the sampler 23 are shown in FIG.
Are adjusted so that the horizontal positions at are the same.

The output of the divider 8 obtained in this way and the G signal obtained from the sampler 23 are multiplied by the multiplier 9 to obtain the R interpolation signal sir corresponding to the pixel row at the vertical position 2m + 1.
Generate (x, y). The output signal sir (x, y) of the multiplier 9 is
The G signal of the pixel column at the vertical position 2m + 1 shown in FIG.
l (fx, fy) / Sg1l (fx, fy). Ie
The frequency component Sir (fx, fy) of sir (x, y) is expressed by the following equation 41 and is shown in FIG. 11D.

[0104]

[Equation 41]

Here, similarly to Equation 12 in the case of the first conventional technique, when the following Equation 42 can be expected, Equation 41 can be replaced with the following Equation 43. Comparison of Equations 2 and 43, or FIGS. 10B and 11
According to the comparison in (d), the R interpolation signal sir (x, y) obtained from the multiplier 9 is the R signal of the pixel row at the vertical position 2m or 2 (m + 1).
The odd-order harmonic components in the vertical direction are in antiphase with the signal. Therefore, the output of the multiplier 9 and the output of the sampler 24 are added to the gate circuit 25, and the interpolation signal sir (x, y) and the vertical position are calculated.
When the R signal sr (x, y) at 2m (m is an arbitrary integer) is selectively synthesized for each pixel column, odd-order harmonic components in the vertical direction cancel each other as shown in FIG. Being reduced. Similarly, if the B signal is selectively synthesized by the gate circuit 26, odd-order harmonic components in the vertical direction from Sb (fx, fy) are canceled.

[0106]

(Equation 42)

[0107]

[Equation 43]

Here, for example, if the subject image is achromatic,
It is assumed that R, G, and B in the hatched portion shown in FIG. 12 are 0.2, and R, G, and B in the portion without the hatched portion are 1.0. At this time, R of the pixel row at the vertical position 2 (m + 1) obtained from the low-pass filter 4
The low frequency component of the signal is 1.0, and the low frequency component of the R signal of the pixel row at a vertical position of 2 m obtained from the low-pass filter 18 is 0.2.
Similarly, the low frequency component of the G signal of the pixel row at the vertical position 2 (m + 1) obtained from the low-pass filter 5 is 1.0, and the G frequency of the pixel row at the vertical position 2m obtained from the low-pass filter 19 is G
It is clear that the low frequency component of the signal is 0.2. As a result, the magnitude of the signal obtained from the divider 8 via the adder 20 and the adder 21 becomes 1.0, and this is multiplied by the G signal obtained from the sampler 23 according to the equation (41).
The R signal at the horizontal position 2 (n + 1) and the vertical position 2m + 1 multiplied by (1) is 1.0, which is equal to the signal to be originally obtained.

On the other hand, the image of the subject is green, and R, G, and B in the hatched portion shown in FIG. 12 are 0.2, and G in the portion without the hatched portion is 1.
It is assumed that 0, R, and B are 0.2. At this time, the low-frequency component of the R signal of the pixel row at the vertical position 2 (m + 1) obtained from the low-pass filter 4 and the low-frequency component of the R signal of the pixel row at the vertical position 2m obtained from the low-pass filter 18 are both 0.2. The low-frequency component of the G signal of the pixel row at the vertical position 2 (m + 1) obtained from the low-pass filter 5 is 1.0, and the low-frequency component of the G signal of the pixel row at the vertical position 2m obtained from the low-pass filter 19 is 0.2. Becomes As a result, the adder 20, the adder 21
, The magnitude of the signal obtained from the divider 8 becomes 0.33. The magnitude of this signal multiplied by the G signal is 0.33 for the R signal at the horizontal position 2 (n + 1) and the vertical position 2m + 1. This is a magnitude closer to the signal to be originally obtained than in the above-described second conventional method and the third conventional method.

As described above, according to the embodiment of the present invention, even when a Bayer array color filter is used, a color signal can be interpolated by a pixel row from which a certain color signal cannot be obtained. In addition, when the subject image is an achromatic color, an interpolation signal that matches the signal to be originally obtained is obtained, so that false color signals can be sufficiently reduced. Further, even when the saturation of the subject image is high, an interpolation signal having a magnitude close to the signal that should be originally obtained is obtained, and the generation of a false color signal is suppressed.

In FIG. 1, among signals obtained from the low-pass filter 4 and the like, components of a region exceeding (1 / 2dx-fa) among the horizontal frequency components of the subject image are mixed. If an optical low-pass filter is inserted into the front surface of the light-receiving device 1 to reduce it, the ratio of the low-frequency components of the two color signals obtained from the low-pass filter can be made closer to the ratio of the color components of the subject image. As an example of the optical low-pass filter, as shown in FIG. 13, a quartz plate 54 that generates a double image shifted by a distance equal to the horizontal pixel interval dx of the solid-state imaging device 1 is applicable.

In the configuration shown in FIG. 1, the signal obtained from the phase adjuster 22 is added to the samplers 23 and 24,
Although the signal and the R signal or the B signal are separated, they may be added to the linear interpolation circuit 27 as in the third configuration example of the present invention shown in FIG. The linear interpolation circuit 27 not only separates the G signal and the B signal, but also realizes general linear interpolation at the same time. Here, the linear interpolation circuit 27 can be realized by the configuration shown in FIG. In FIG. 15, the input signal to the linear interpolation circuit 27 is applied to a pixel delay circuit 28, and the output of the pixel delay circuit 28 is further applied to a pixel delay circuit 29. The delay times of the pixel delay circuit 28 and the pixel delay circuit 29 are each equal to the delay time of one pixel. Further, the input signal to the linear interpolation circuit 27 and the output of the pixel delay circuit 29 are applied to the averaging circuit 30 to output a signal corresponding to the averaging value of the two inputs. The output of the pixel delay circuit 28 and the output of the averaging circuit 30 are simultaneously output to the gate circuit 3 respectively.
1 and the gate circuit 32. As a result, for example, the input to the linear interpolation circuit 27 is the horizontal position 2 (n +
1) When the signal is the G signal of the pixel at the vertical position 2m + 1, the B signal of the pixel at the horizontal position 2n + 1 is obtained from the pixel delay circuit 28;
The G signal of the pixel at the horizontal position 2n is obtained from the pixel delay circuit 29. At this time, the gate circuit 31 outputs the average value of the G signals of the pixels at the horizontal positions 2n and 2 (n + 1) obtained from the averaging circuit 30, and the gate circuit 32 outputs the horizontal position 2n obtained from the pixel delay circuit 28. It operates to output the B signal of the +1 pixel. The input to the linear interpolation circuit 27 is horizontal position 2 (n + 1)
+1 and the B signal of the pixel at the vertical position 2m + 1, if the gate circuit 31 outputs the signal from the pixel delay circuit 28 and the gate circuit 32 outputs the signal from the averaging circuit 30, G Separation of the signal and the B signal and general linear interpolation processing can be simultaneously achieved. The operation of the other parts of the configuration shown in FIG. 14 is the same as that of the configuration shown in FIG.

In the above description, an example has been described in which a color signal that does not exist in a certain horizontal pixel row is interpolated using the relationship between the color signal and the G signal in the upper and lower horizontal pixel rows. However, as described above, the color filter of the Bayer array is 9 colors.
Since the arrangement pattern does not change even when rotated by 0 degrees, interpolation can be performed between pixel rows in the vertical direction.
For this purpose, as in the configuration according to the fourth embodiment of the present invention shown in FIG. 16, a plurality of 1H delay circuits may be overlapped and the signals of the vertical pixel columns may be used simultaneously.

In the configuration of FIG. 16, the 1H delay circuit 14 in which the outputs of the solid-state imaging device 1 are connected in series, the 1H delay circuit 1
5, 1H delay circuit 33 and 1H delay circuit 34 are sequentially added. As a result, in FIG.
Signals of two pixel columns are obtained at the same time. The output of the solid-state imaging device 1 is sequentially applied to a pixel delay circuit 28 and a pixel delay circuit 29 which are connected in series with a delay time of one pixel. Similarly, the output of the 1H delay circuit 14 is sequentially applied to a pixel delay circuit 35 and a pixel delay circuit 36,
The output of the delay circuit 15 is sequentially applied to a pixel delay circuit 37 and a pixel delay circuit 38. The output of the 1H delay circuit 33 is sequentially applied to the pixel delay circuit 39 and the pixel delay circuit 40, and the output of the 1H delay circuit 34 is sequentially applied to the pixel delay circuit 41 and the pixel delay circuit 42.

The outputs of the solid-state image sensing device 1 and the pixel delay circuit 29 are applied to an averaging circuit 30, and a signal corresponding to the average value of both is output. Similarly, the outputs of the 1H delay circuit 14 and the pixel delay circuit 36 are the averaging circuit 43, and the outputs of the 1H delay circuit 15 and the pixel delay circuit 38 are the averaging circuits 44, 1
The outputs of the H delay circuit 33 and the pixel delay circuit 40 are added to an averaging circuit 45, the outputs of the 1H delay circuit 34 and the pixel delay circuit 42 are applied to an averaging circuit 46, and a signal corresponding to the average value of both input signals is output. Is done. As a result, when the signal of the pixel column at the horizontal position 2n + 1 in FIG. 2 is obtained from the pixel delay circuit 28, the pixel delay circuit 35, the pixel delay circuit 37, the pixel delay circuit 39, and the pixel delay circuit 41, the averaging circuit 30 , The averaging circuit 43, the averaging circuit 44, the averaging circuit 45, and the averaging circuit 46, the horizontal positions 2n and 2 (n +
A signal corresponding to the average value of the pixel row in 1) is obtained.

Further, the outputs of the averaging circuit 30, the averaging circuit 44 and the averaging circuit 46 are applied to a vertical low-pass filter 47, and the averaging circuit 43 and the averaging circuit 45 are added.
Is applied to a vertical low pass filter 48. Here, the vertical low-pass filter 47 is 1 times the output signal of the averaging circuit 30, 1/2 times the output signal of the averaging circuit 44,
Assume that 1/4 times the output signal of the averaging circuit 46 is added. The vertical low-pass filter 48 adds 1/2 times the output signal of the averaging circuit 43 and 1/2 times the output signal of the averaging circuit 45. At this time, the frequency response of the vertical low-pass filter 48 is as shown in Expression 40 as described above. On the other hand, the frequency response of the vertical low-pass filter 47 is (1 + cos2ωdy) / 2
Is required.

[0117]

[Equation 44]

As a result, the frequency response of the vertical low-pass filter 47 is as shown in FIG. 17A, and the frequency response of the vertical low-pass filter 48 is shown in FIG.
As shown in (b), the frequencies at which the response becomes 0 coincide with each other at 1 / 4dy. Thus, obtaining signals of five consecutive pixel columns at the same time is a minimum condition for forming a vertical low-pass filter having a frequency band close to that of two color signals obtained alternately. When only signals of three consecutive pixel columns are used, one of the two color signals is a signal from only one pixel, so that a vertical low-pass filter cannot be formed. At this time, the influence of the higher harmonic component mixed between the two color signals is greatly different.
It cannot be expected that the ratio of the two indicates the ratio of the color components of the subject image. As the number of taps of the vertical low-pass filter is increased by further increasing the number of 1H delay circuits, the response of the vertical low-pass filter of the two color signals becomes closer. Therefore, it is clear that this is preferable in obtaining the ratio of the color components of the subject image. is there.

Further, in the configuration shown in FIG. 16, the outputs of vertical low-pass filter 47 and vertical low-pass filter 48 are simultaneously applied to gate circuit 49 and gate circuit 50, respectively. The output of the gate circuit 49 is applied to the dividend side of the divider 8, and the output of the gate circuit 50 is applied to the divisor side of the divider 8. At this time, the gate circuit 49 and the gate circuit 50 output the R signal or the B signal to the dividend side of the divider 8.
The operation is performed so that the low frequency component of the signal is added and the low frequency component of the G signal is added to the divisor side of the divider 8. On the other hand, the outputs of the pixel delay circuit 35 and the pixel
To generate a linear interpolation signal in the vertical direction. Further, the outputs of the averaging circuit 51 and the pixel delay circuit 37 are simultaneously applied to a gate circuit 52 and a gate circuit 53, respectively. At this time, the gate circuits 52 and 53 operate so that the G signal is obtained from the gate circuit 52 and the R signal or the B signal is obtained from the gate circuit 53.

As a result, for example, when the G signal of the pixel at the horizontal position 2n + 1 and the vertical position 2 (m + 1) shown in FIG. 2 is obtained from the pixel delay circuit 37, the vertical low-pass filter 47 outputs the vertical position 2m. R signal at horizontal position 2n and 2 (n + 1), R signal at horizontal position 2n and 2 (n + 1) at vertical position 2 (m + 1), and horizontal position at vertical position 2 (m + 2) After adding the 2n and 2 (n + 1) R signals, a signal whose band is limited in the vertical direction is obtained. on the other hand,
Vertical position 2m + 1 from vertical low-pass filter 48
The G signal at the horizontal position 2n and 2 (n + 1) is added, and the G signal at the horizontal position 2n and 2 (n + 1) is added at the vertical position 2 (m + 1) +1. A signal is obtained.

At this time, the horizontal position obtained from the divider 8
A signal corresponding to the ratio of the low-frequency component of the R signal to the low-frequency component of the G signal in the pixel rows 2n and 2 (n + 1) is multiplied by a multiplier 9
Is added to At the same time, the multiplier 9 has a gate circuit 52
, The G signal of the pixel at the horizontal position 2n + 1 and the vertical position 2 (m + 1) is added. The output of the multiplier 9 is selected by the gate circuit 25, and becomes an R interpolation signal corresponding to the pixel at the horizontal position 2n + 1 and the vertical position 2 (m + 1). At this time, the horizontal position 2n + 1 obtained from the averaging circuit 51 and the vertical positions 2m + 1 and 2 (m + 1)
The signal corresponding to the average value of the B signal of +1 is the gate circuit 5
3. The B signal corresponding to the pixel at the horizontal position 2n + 1 and the vertical position 2 (m + 1) selected by the gate circuit 26.

The signal obtained from the pixel delay circuit 37 is shifted by one pixel, and the horizontal position 2 (n + 1) and the vertical position 2 (m
+1), the low-frequency component of the G signal of the pixel row at the horizontal position 2n + 1 and 2 (n + 1) +1 is obtained from the vertical low-pass filter 47. The low frequency component of the B signal is obtained. Also, the averaging circuit 5
From the horizontal position 2 (n + 1) and the vertical positions 2m + 1 and 2 (m + 1) +1
A signal corresponding to the average value of the signal is obtained. At this time, the gate circuit 49 and the gate circuit 50 add the low-frequency component of the G signal obtained from the vertical low-pass filter 47 to the divisor side of the divider 8 and the low-frequency component of the B signal obtained from the vertical low-pass filter 48 to the dividend side. , And the gate circuit 52 operates so that the G signal of the averaging circuit 51 is added to the multiplier 9. Further, the gate circuit 25 is connected to the horizontal position obtained from the pixel delay circuit 37 via the gate circuit 53.
The gate circuit 26 operates to output the R signal of 2 (n + 1) and the vertical position 2 (m + 1), and the gate circuit 26 operates to output the B interpolation signal obtained from the multiplier 9.

Further, even if the position of the signal obtained from the pixel delay circuit 37 in FIG.
The gate circuit 50 only needs to operate so that the low frequency component of the G signal is added to the divisor side of the divider 8 and the low frequency component of the R signal or B signal is added to the dividend side. Further, the gate circuit 52 and the gate circuit 53 add the G signal to the multiplier 9, and
The gate circuit 25 and the gate circuit 26 operate so that the R signal or the B signal is added to the gate circuit 25 and the gate circuit 2.
6 only has to operate so that an R signal and a B signal can be obtained from each.

As described above, if the configuration shown in FIG. 16 is used, the vertical interpolation process using the upper and lower horizontal pixel columns by the configuration shown in FIG. 1 can be replaced with the horizontal interpolation process using the left and right vertical pixel columns. The operation converted to the direction interpolation processing can be realized.

FIG. 18 shows the configuration of the fifth embodiment of the present invention. The configuration of FIG. 18 is different from the configuration of generating a horizontal interpolation signal using the left and right vertical pixel columns shown in FIG. 16 in that the vertical interpolation signal using the upper and lower horizontal pixel columns shown in FIG. Are combined. In FIG. 18, the operations of the parts denoted by the same reference numerals are the same as those in FIG. 16 or FIG. Here, a gate circuit 25-that outputs an R signal, a B signal, and a G signal after horizontal interpolation processing using left and right vertical pixel columns.
1. The outputs of the gate circuit 26-1 and the gate circuit 52 are applied to a gain control circuit 55, a gain control circuit 56, and a gain control circuit 57, respectively. Further, the gate circuit 25-2, the gate circuit 26-2, and the sampler 2 that output the R signal, the B signal, and the G signal after the vertical interpolation processing using the upper and lower horizontal pixel columns.
3 are output from a gain control circuit 58, a gain control circuit 59, and a gain control circuit 6, respectively.
Added to 0. Further, the gain control circuit 55
And the output of the gain control circuit 58 are added to the adder 61, the gain control circuit 56 and the gain control circuit 5.
The output of 9 is added by an adder 62, and the outputs of the gain control circuit 57 and the gain control circuit 60 are added by an adder 63.

Here, the vertical low frequency component srlv (x, y) or sblv (x, y) of the R signal or B signal obtained from the gate circuit 49 and the vertical direction of the G signal obtained from the gate circuit 50 are obtained. Are added to an adder 64 and a subtraction absolute value circuit 65, respectively, and the signal sav (x, y) corresponding to the sum of the two and the absolute value of the difference between the two are added. The output signal sdv (x, y) is output. Further, the output of the adder 64
sav (x, y) is applied to the divisor side of the divider 66, and the output sdv (x, y) of the subtraction absolute value circuit 65 is applied to the dividend side of the divider 66. As a result, the following equation 45 is obtained from the divider 66.
Sv (x, y) is obtained.

[0127]

[Equation 45]

On the other hand, R obtained from low-pass filter 4
Horizontal frequency component srlh (x, y) of signal or B signal
Alternatively, sblh (x, y) and the horizontal low-frequency component sglh (x, y) of the G signal obtained from the low-pass filter 5 are added to an adder 67 and a subtraction absolute value circuit 68, respectively, to add the two. A signal sah (x, y) corresponding to the value and a signal sdh (x, y) corresponding to the absolute value of the difference between the two are output. Output of adder 67
sah (x, y) is applied to the divisor side of the divider 69, and the output sdh (x, y) of the subtraction absolute value circuit 68 is applied to the dividend side of the divider 69. As a result, the following equation 46 is obtained from the divider 69.
Sh (x, y) is obtained.

[0129]

[Equation 46]

Output Sv (x, y) of divider 66 and divider 6
The output Sh (x, y) of No. 9 is applied to the adder 70 to output an added signal of both. Further, the output of the adder 70 is applied to the divisor side of the divider 71 and the divider 72. On the dividend side of the divider 71, the output signal Sv (x,
y) is added, and a divider 6 is provided on the dividend side of the divider 72.
Nine output signals Sh (x, y) are added. As a result, the divider 71 obtains the vertical interpolation coefficient Gv
(x, y) is obtained, and the horizontal interpolation coefficient Gh (x, y) expressed by the following Expression 48 is obtained from the divider 72.

[0131]

[Equation 47]

[0132]

[Equation 48]

The vertical interpolation coefficient Gv obtained from the divider 71
(x, y) is added to the multiplier 58, the multiplier 59, and the multiplier 60, and becomes the gain of the vertical interpolation signal using the upper and lower horizontal pixel columns. The horizontal interpolation coefficient Gh (x, y) obtained from the divider 72 is multiplied by the multiplier 55 and the multiplier 5
6. The gain is applied to the multiplier 57 to obtain the gain of the horizontal interpolation signal using the left and right vertical pixel columns.

When the subject image is achromatic and the change in the vertical direction is smaller than the change in the horizontal direction, srlv (x,
y) or sblv (x, y) and sglv (x, y) are closer values. At this time, it can be seen from the relations of Equations 45 to 48 that the vertical interpolation coefficient Gv (x, y) is small and the horizontal interpolation coefficient Gh (x, y) is large. As a result, the adders 61 to 63
, The horizontal interpolation signals using the left and right vertical pixel columns obtained from the gain control circuits 55 to 57 occupy a large proportion.

When the change in the horizontal direction is smaller than the change in the vertical direction, srlh (x, y) or sblh (x, y) and sg
lh (x, y) is closer to the horizontal interpolation coefficient Gh (x, y).
y) is small and the vertical interpolation coefficient Gv (x, y) is large. As a result, in the outputs of the adders 61 to 63, the vertical interpolation signals using the upper and lower horizontal pixel columns obtained from the gain control circuits 58 to 60 occupy a large proportion.

As described above, according to the structure shown in FIG.
An operation is performed to generate an interpolation signal using a low frequency component in a direction in which the difference between the signal (or the B signal) and the G signal is small. As a result, the change in the subject image is small, and the function of obtaining the interpolation signal generation coefficient from a direction close to an achromatic color is obtained. Therefore, the influence of the harmonic component on the coefficient is small, and the generation of the interpolation signal close to the signal to be obtained originally Can be expected. Furthermore, the weight of the interpolation signal in the vertical and horizontal directions is determined by using the output of the low-pass filter used when generating the interpolation signal, so that a large increase in the circuit scale can be prevented.

Although the embodiment of the present invention has been described by taking as an example the case where a color filter in which Gs shown in FIG. 2 are arranged in a checkered pattern is used, from the operation thereof, as shown in FIG.
It is clear that any filter can be applied to a color filter having a checkerboard pattern, such as a color filter having a checkerboard pattern.

Further, although the embodiment of the present invention has been described by taking the processing by hardware as an example, it is apparent that the present invention can also be realized by processing on a general-purpose computer by software.

[0139]

As described above, according to the signal interpolation method for a single-chip color camera of the present invention, even when a color filter of a Bayer array is used, a color signal cannot be obtained in a pixel row where a color signal cannot be obtained. Can be interpolated. In addition, when the subject image is an achromatic color, an interpolation signal that matches the signal to be originally obtained is obtained, so that false color signals can be sufficiently reduced. Further, even when the saturation of the subject image is high, an interpolation signal having a magnitude close to a signal that should be originally obtained can be obtained, and generation of a false color signal can be suppressed.

Further, according to the signal interpolation method for a single-chip color camera of the present invention, the weighting of the interpolation signal using the left and right vertical pixel columns and the weighting of the interpolation signal using the upper and lower horizontal pixel columns are performed. Since the determination is made using the output of the low-pass filter used for generation, it is possible to generate an interpolation signal close to a signal that should be originally obtained without greatly increasing the circuit scale.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a first embodiment of a signal interpolation method for a single-chip color camera according to the present invention.

FIG. 2 is a diagram illustrating a configuration of a Bayer array primary color filter used in the signal interpolation method of the single-chip color camera of the present invention.

FIG. 3 is a diagram for explaining a cause of generation of a false color signal in a conventional single-chip color camera.

FIG. 4 is a diagram illustrating an example of a frequency component of a signal of a pixel obtained from a solid-state imaging device in a conventional single-chip color camera.

FIG. 5 is a diagram showing a configuration of a first conventional single-chip color camera in a signal interpolation method.

FIG. 6 is a diagram illustrating an example of frequency components of a color signal obtained by a signal interpolation method of a first conventional single-panel color camera.

FIG. 7 is a diagram used to explain a calculation process in a second conventional single-chip color camera signal interpolation method.

FIG. 8 is a diagram used for describing arithmetic processing in a third conventional single-chip color camera signal interpolation method.

FIG. 9 is a diagram showing the configuration of a second embodiment of the signal interpolation method for a single-chip color camera according to the present invention.

FIG. 10 is a diagram illustrating an example of frequency components of a color signal obtained by a Bayer array primary color filter.

FIG. 11 is a diagram illustrating an example of a frequency component of a color signal obtained by a signal interpolation method for a single-chip color camera according to the present invention.

FIG. 12 is a diagram illustrating an operation of an interpolation process for reducing a false color signal by a signal interpolation method for a single-chip color camera according to the present invention.

FIG. 13 is a diagram illustrating the operation of an optical low-pass filter that can be used together in the signal interpolation method for a single-chip color camera of the present invention.

FIG. 14 is a diagram showing the configuration of a third embodiment of the signal interpolation method for a single-chip color camera according to the present invention.

FIG. 15 is a diagram showing a configuration of a linear interpolation circuit that can be used for the configuration of the third embodiment in the signal interpolation method for a single-chip color camera of the present invention.

FIG. 16 is a diagram showing the configuration of a fourth embodiment of the signal interpolation method for a single-chip color camera according to the present invention.

FIG. 17 is a diagram showing a frequency response of a vertical low-pass filter used in a fourth embodiment in a signal interpolation method for a single-chip color camera according to the present invention.

FIG. 18 is a diagram showing the configuration of a fifth embodiment of the signal interpolation method for a single-chip color camera according to the present invention.

FIG. 19 is a diagram showing an example of another filter to which the signal interpolation method for a single-chip color camera of the present invention can be applied.

[Explanation of symbols]

1 Solid-state image sensor 2, 3, 16, 17, 23, 24 Sampler 4, 5, 12, 13, 18, 19 Low-pass filter 6, 7, 10, 11, 25, 26, 31, 32, 49,
50, 52, 53 Gate circuit 8, 66, 69, 71, 72 Divider 9 Multiplier 14, 15, 33, 34 1H delay circuit 20, 21, 61, 62, 63, 64, 67, 70
Adder 22 phase adjuster 27 linear interpolation circuit 28, 29, 35, 36, 37, 38, 39, 40, 4
1, 42 pixel delay circuit 30, 43, 44, 45, 46, 51 averaging circuit 47, 48 vertical low-pass filter 55, 56, 57, 58, 59, 60 gain control circuit 65, 68 subtraction absolute value circuit 54 crystal plate

Claims (5)

[Claims] 1. A plurality of pixel groups having different spectral sensitivity characteristics are two-dimensionally arranged, and a first pixel group corresponding to a first type among the plurality of pixel groups is mutually connected between adjacent pixel columns. The pixels of a second pixel group corresponding to a second type different from the first type among at least the plurality of types of pixel groups are arranged so as to be present in every other pixel column. In a signal interpolation method for a single-chip color camera that generates the second type of signal corresponding to at least all pixel columns using an image pickup device, a second pixel group arbitrarily selected from a pixel column in which the second pixel group does not exist. A first pixel group signal which is a signal of the first pixel group at a position of a first pixel existing in one pixel column;
And a signal of a pixel belonging to the first pixel group on a second pixel column adjacent to the first pixel column and a signal of a pixel adjacent to the first pixel column and the second pixel column Extracting a first low-frequency signal corresponding to the position of the first pixel from a signal obtained by adding signals of pixels belonging to the first pixel group on a third pixel column different from the second pixel; A signal corresponding to the position of the first pixel is obtained from a signal obtained by adding a signal of a pixel belonging to the second pixel group on the column and a signal of a pixel belonging to the second pixel group on the third pixel column. 2 and takes out the second low frequency signal with respect to the first low frequency signal.
Extracting a first interpolation signal obtained by multiplying the first signal by a first coefficient signal corresponding to the magnitude of the low-frequency signal, and using the first interpolation signal as the second type signal at the position of the first pixel A signal interpolation method for a single-chip color camera. 2. When the first pixel belongs to the first pixel group, the signal of the first pixel is used as the first signal, and the first pixel belongs to the first pixel group. If not, it exists on the first pixel column and the first
A second pixel belonging to the first pixel group adjacent on both sides to the pixel
2. The signal interpolation method for a single-chip color camera according to claim 1, wherein an average value signal of the signals of the pixel and the third pixel is used as the first signal. 3. A signal interpolation method for a single-chip color camera according to claim 1, wherein said image pickup device is a combination of an optical low-pass filter for reducing resolution in a direction of said pixel row. 4. A plurality of pixel groups having different spectral sensitivity characteristics are two-dimensionally arranged, and a first pixel group corresponding to a first type among the plurality of pixel groups is mutually connected between adjacent pixel columns. The pixels of a second pixel group corresponding to a second type different from the first type among at least the plurality of types of pixel groups are arranged so as to be present in every other pixel column. A signal interpolation method for a single-chip color camera in which an image sensor is used to interpolate signals of the second type corresponding to positions of all pixels except the second pixel group by interpolation signals generated by at least two methods. And a signal of a pixel belonging to the first pixel group on a second pixel row adjacent to the first pixel row including a first pixel arbitrarily selected from the pixels not belonging to the second pixel group. Adjacent to the first pixel column,
A first low-frequency signal corresponding to the position of the first pixel is obtained from a signal obtained by adding signals of pixels belonging to the first pixel group on a third pixel column different from the second pixel column. Generating a signal of a pixel not belonging to the first pixel group on the second pixel row and a signal of a pixel not belonging to the first pixel group on the third pixel row, Generating a second low-frequency signal corresponding to the position of the first pixel;
Belongs to the first pixel group that passes through a second pixel adjacent to the first pixel and that is on a first line in a direction perpendicular to the first pixel column. A pixel signal and a third pixel present on the first pixel column and adjacent to the first pixel on the opposite side of the second pixel and perpendicular to the first pixel column Generating a third low-frequency signal corresponding to the first pixel from a signal obtained by adding the signals of the pixels belonging to the first pixel group existing on the second line in the direction of. The first present on
A fourth low-frequency signal corresponding to the first pixel from a signal obtained by adding a signal of a pixel not belonging to the pixel group and a signal of a pixel existing on the second line and not belonging to the first pixel group And the first low-frequency signal and the second signal with respect to the sum of the first low-frequency signal and the second low-frequency signal.
Generating a first coefficient signal corresponding to the ratio of the absolute value of the difference between the low-frequency signals of the third low-frequency signal and the third low-frequency signal with respect to the sum of the third low-frequency signal and the fourth low-frequency signal 4th above
Generating a second coefficient signal corresponding to the ratio of the absolute value of the difference between the low frequency signals, and corresponding to the ratio of the first coefficient signal to the sum of the first coefficient signal and the second coefficient signal. Generating a first amplification degree, and generating a second amplification degree corresponding to a ratio of the second coefficient signal to an added value of the first coefficient signal and the second coefficient signal; A signal interpolation method for a single-chip color camera, wherein the interpolation signals generated by the above two methods are synthesized by using the amplification degree of (1) and the second amplification degree. 5. The interpolation signal generated by the above two methods is:
A first signal obtained by obtaining a signal corresponding to a ratio of the second low-frequency signal to the first low-frequency signal from a pixel belonging to the first pixel group existing on the first pixel column; A first interpolation signal obtained by multiplying a signal corresponding to the first type at a pixel position and a signal corresponding to a ratio of the fourth low-frequency signal to the third low-frequency signal are converted to the first pixel. As described above, the first type at the position of the first pixel obtained from the signal of the pixel belonging to the first pixel group existing on the third line in the direction perpendicular to the first pixel column A second interpolation signal obtained by multiplying a corresponding signal by amplifying the first interpolation signal at the second amplification degree and amplifying the second interpolation signal at the first amplification degree. 5. The signal interpolation method for a single-chip color camera according to claim 4, wherein