JP4334484B2 - Pixel signal processing apparatus and method - Google Patents

Description

  The present invention relates to a pixel signal processing apparatus and method, and in particular, when each of pixels arranged on a two-dimensional plane does not have at least one color component value among a plurality of color component values, the pixel is present. The present invention relates to a pixel signal processing apparatus and method for obtaining a color image by generating non-color component values by interpolation.

  Such pixel signal processing is, for example, a plurality of color component values each generating a plurality of color component values, for example, any one of the three primary colors of red (R), green (G), and blue (B). A type of photoelectric conversion element is used as a part of a color imaging apparatus further including, for example, a Bayer-type imaging element on a two-dimensional plane, and is missing at each pixel position based on a pixel signal output from the imaging element It is used to interpolate the color component value (insufficient color component value).

  In a conventional imaging device having an imaging device in which red, green, and blue color filters are arranged in a Bayer shape, G, B or B, R or R, and G color component values are insufficient for each pixel. For example, as shown in Patent Document 1 below, the pixel signal of each pixel is replaced with an average value based on the local pixel signal distribution for each color in order to enhance the resolution, and this is assumed. An interpolation method based on the linear similarity ratio between known color geometric figures and missing color geometric figures is used.

JP 2001-197512 A (paragraphs 0048 to 0049, FIG. 7)

  This conventional method assumes that there is a positive correlation between color component values (for example, R, G, and B component values in a Bayer array) in a region near the interpolation target pixel. Therefore, interpolation is appropriately performed in areas where there is no positive correlation between color component values (for example, a boundary between one color and another color), for example, where there is no correlation, or where there is a negative correlation. There is a problem that interpolation error becomes large.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a pixel signal processing apparatus that can always perform interpolation using an optimal interpolation method regardless of how color component values change in a region near an interpolation processing target pixel.

This invention
Based on a set of pixel signals arranged on a two-dimensional plane and each having any one of the first to Nth spectral sensitivity characteristics, the hth (h is any one of 1 to N) In the pixel signal processing device that generates the pixel signal of the kth (k is any one of 1 to N except h) spectral signal at the interpolation target pixel position where the pixel signal of the spectral sensitivity characteristic exists. ,
H signal low frequency component generating means for generating a low frequency component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
H signal change component generation means for generating a change component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
K-signal low-frequency component generating means for generating low-frequency components of the pixel signal of the k-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
K signal change component generation means for generating a change component of the pixel signal of the kth spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A saturation coefficient calculation means for calculating a saturation coefficient based on the saturation of the pixel signal at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
The k-th signal and the h-th signal determined based on the difference between the value obtained by the k signal change component generation means and the value obtained by the h signal change component generation means and the saturation coefficient; , The difference between the value obtained by the k signal low frequency component generation means and the value obtained by the h signal low frequency component generation means, the h th pixel signal at the interpolation target pixel position, And an interpolation value calculating means for obtaining the k-th pixel signal at the interpolation target pixel position.

  According to the present invention, it is possible to accurately perform interpolation even if the correlation between the color component values is variously different, such as when the pixel to be interpolated is in the vicinity of the color boundary.

  Further, based on the difference between the value obtained by the k signal change component generation means and the value obtained by the h signal change component generation means and the saturation coefficient (for example, the difference is multiplied by the saturation coefficient). Therefore, the decorrelation value between the kth signal and the hth signal is calculated, and the interpolated value is obtained using the thus obtained decorrelation value. Interpolation can be performed with accuracy.

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is suitable for use as a part of a digital still camera, but the present invention is not limited to this. That is, the present invention generates missing (insufficient) signals by interpolation, and can be applied to the case of converting the resolution in a display device such as a projector in addition to an imaging device such as a digital still camera.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of an imaging apparatus provided with a pixel signal processing apparatus according to Embodiment 1 of the present invention.
The light incident from the lens 1 forms an image on the imaging surface of the two-dimensional image sensor 2 configured by, for example, a solid-state imaging device. The image sensor 2 includes a plurality of photoelectric conversion elements arranged two-dimensionally. The plurality of photoelectric conversion elements are arranged in a Bayer type, for example, as shown in FIG. ), Green (G), and blue (B) are covered with a color filter having spectral sensitivity characteristics corresponding to the three primary colors, and each photoelectric conversion element receives an analog signal of a color component corresponding to the color of the color filter. Is output.
Further, generally speaking, the present invention is based on a set of pixel signals of pixels arranged on a two-dimensional plane, each having any one of the first to Nth spectral sensitivity characteristics. Spectroscopy of k (k is any one of 1 to N excluding h) at the target pixel position where a pixel signal having spectral sensitivity characteristics of h (h is any one of 1 to N) exists. The present invention relates to a pixel signal processing apparatus and method for generating a pixel signal having sensitivity characteristics. In the following embodiment, N is 3, and pixels having first to Nth spectral sensitivity characteristics are R, G. , B pixels.

In FIG. 2, horizontal and vertical represent the horizontal direction (H) and vertical direction (V) of the imaging surface, respectively. The photoelectric conversion element constitutes a pixel, and the position occupied by each photoelectric conversion element on the imaging surface corresponds to the pixel position. Since each pixel is two-dimensionally arranged on the image pickup surface of the image pickup device, their position can be represented by a coordinate value on the HV coordinate plane (or HV plane). FIG. 2 shows only a part of the image sensor, that is, a range of 7 rows and 7 columns. The horizontal position of the center pixel is represented by i, the vertical position is represented by j, and the coordinate value is represented by (i, j). The horizontal position (row direction) of the surrounding pixels is represented by i-3, i-2. ,..., I + 3, the positions in the vertical direction (column direction) are represented by j-3, j-2,. Note that i and j are both integers. Pixels adjacent to each other in the horizontal direction differ by “1” in the horizontal direction, and pixels adjacent to each other in the vertical direction differ from each other by “1” in the vertical direction. .
In the following description, pixels corresponding to photoelectric conversion elements covered with R color filters are covered with R pixels, and pixels corresponding to photoelectric conversion elements covered with G color filters are covered with G pixels and B color filters. A pixel corresponding to the photoelectric conversion element is called a B pixel.

  The image sensor 2 photoelectrically converts incident light and outputs an analog signal having a level corresponding to the amount of incident light for each pixel. This analog signal is converted into a digital signal by the A / D converter 3 and outputted, and is written in the frame memory 4 as a color component value (pixel signal) possessed by each pixel. At this time, each signal is written in association with the position of each pixel on the imaging surface, and hence the position on the HV coordinate plane.

As described above, since each of the photoelectric conversion elements constituting each pixel is covered with the filter, it receives light of any one color of red, green, and blue. The color of light received by each photoelectric conversion element may be referred to as “light reception color”, and the color other than the light reception color for each pixel may be referred to as “insufficient color”.
From each of the photoelectric conversion elements constituting each pixel, only a signal representing one color component value corresponding to the received light color can be obtained. That is, for the R pixel, the R component value is known, while the G and B component values are unknown, and for the G pixel, the G component value is known, while the B and R component values are unknown, For the B pixel, the B component value is known, while the R and G component values are unknown. Since each pixel has all the R, G, and B color component values, a color image can be obtained. Therefore, an unknown color component value written in the frame memory 4 at each pixel position is an insufficient color component. Also called value. In the pixel signal processing of the present invention, a color component value (insufficient color component value) that is unknown in each pixel is obtained by interpolation.

  The pixel signals stored in the frame memory 4 are distributed and stored in the two-dimensional memories 6r, 6g, and 6b for each of the R, G, and B signals by the demultiplexer 5. That is, the R signal is stored in the two-dimensional memory 6r, the G signal is stored in the two-dimensional memory 6g, and the B signal is stored in the two-dimensional memory 6b.

  3, 4, and 5 respectively show the arrangement of R pixels, G pixels, and B pixels on the imaging surface of the image sensor 2 for each color. Also in each of the two-dimensional memories 6r, 6g, and 6b, the signal (color component value) of each pixel is written in association with the position on the imaging surface, that is, the position on the HV coordinate plane. Therefore, FIGS. 3, 4 and 5 also represent the positions on the HV coordinate plane of the pixel signals distributed from the demultiplexer 5 and stored.

  Note that the frame memory 4 is of a so-called interlaced readout method in which the image sensor 2 reads out one row at a time in two rows. Since all the pixel signals of one sheet (frame) are prepared, the reading is performed twice (two fields). This is necessary if you have to In the case of the so-called progressive readout type image sensor 2 that reads out the pixels in the pixel array shown in FIG. 2 one row at a time from the top, the pixel signal sent from the image sensor 2 is distributed by the demultiplexer 5 as it is. Therefore, the same operation can be realized without the frame memory 2.

  The low-pass filters 8r, 8g, and 8b are provided corresponding to the memories 6r, 6g, and 6b, respectively, and output low-frequency components of each color component with respect to the pixel signals read from the memories 6r, 6g, and 6b. To do. That is, each of the low-pass filters 8r, 8g, and 8b calculates, for each pixel, the low-frequency component of the pixel signal of each color at a plurality of pixel positions in an area near the pixel position (area including the pixel position). To do. The calculation method will be described in detail later. 6, 7 and 8 show output examples of the low-pass filters 8r, 8g and 8b.

  Similarly, the high-pass filters 7r, 7g, and 7b are also provided corresponding to the memories 6r, 6g, and 6b, respectively, and the high-frequency components of the respective color components with respect to the pixel signals read from the memories 6r, 6g, and 6b. Is output. That is, each of the high-pass filters 7r, 7g, and 7b calculates, for each pixel, the high-frequency component or “change component” of the pixel signal of each color at a plurality of pixel positions in a region near the pixel position. The calculation method will be described in detail later. 9, 10 and 11 show output examples of the high-pass filters 7r, 7g and 7b.

  As shown in FIGS. 6, 7, 8, 9, 10, and 11, the outputs of the low-pass filters 8r, 8g, and 8b (RLPF, GLPF, and BLPF), and the outputs of the high-pass filters 7r, 7g, and 7b. (RHPF, GHPF, BHPF) is obtained for all pixels.

  The saturation coefficient calculation means 9 calculates a saturation coefficient β from the pixel signals read from the two-dimensional memories 6r, 6g, 6b.

The decorrelation value calculation means 12 obtains a decorrelation value for each pixel based on the outputs of the high-pass filters 7r, 7g, 7b and the output of the saturation coefficient calculation means 9.
As shown in FIG. 12, for example, the non-correlation value calculation means 12 includes selection means 23k and 23h, a difference calculation means 25, a coefficient multiplication means 27, and a control means 30a.

The calculation means 10 is low for each pixel based on the pixel signals read from the two-dimensional memories 6r, 6g, 6b, the outputs of the low-pass filters 8r, 8g, 8b, and the output of the decorrelation value calculation means 12. A frequency component difference is obtained, and an interpolation value is obtained.
As shown in FIG. 13, for example, the calculation means 10 includes selection means 24k, 24h, 21, difference calculation means 26, coefficient multiplication means 28, addition means 29, and control means 30b.

The selection unit 21 selects one of the two-dimensional memories 6r, 6g, and 6b and supplies the pixel signal read from the selected two-dimensional memories 6r, 6g, and 6b to the addition unit 29.
The selection unit 23k receives the outputs RHPF, GHPF, and BHPF of the high pass filters 7r, 7g, and 7b, and selects and outputs one of them. The selection unit 23h receives the outputs RHPF, GHPF, and BHPF of the high-pass filters 7r, 7g, and 7b, and selects and outputs the other one of them.
The selection unit 24k receives the outputs RLPF, GLPF, and BLPF of the low-pass filters 8r, 8g, and 8b, and selects and outputs one of them. The selection unit 24h receives the outputs RLPF, GLPF, BLPF of the low-pass filters 8r, 8g, 8b, and selects and outputs the other one of them.

Selection by the selection means 23k and 23h is controlled by the control means 30a. Selection by the selection means 21, 24k, 24h is controlled by the control means 30b. The control means 30a and the control means 30b may be configured by the same unit. In other words, it may be constituted by a common control means (for example, constituted by a programmed computer).
When the interpolation target pixel has a color component value of the h-th color of R, G, and B, and the color component value of the k-th color of the interpolation target pixel is obtained by interpolation, the selection unit 21 selects the h-th color. The two-dimensional memory storing the color component values of the color is selected, and the color component value of the h-th color (for example, represented by h (i, j)) of the interpolation target pixel is read. The k-color high-pass filter output kHPF is selected, the selection means 23h selects the h-th color high-pass filter output hHPF, the selection means 24k selects the k-th color low-pass filter output kLPF, The selection unit 24h selects the output hLPF of the h-th color low-pass filter.

The difference calculation means 25 calculates a difference between the kth HPF signal kHPF and the hth HPF signal hHPF (the former minus the latter) (kHPF−hHPF). This difference (kHPF−hHPF) is also referred to as “high frequency component difference” or “change component difference”.
The difference calculating means 26 obtains the difference between the kth LPF signal kLPF and the hth LPF signal hLPF (the former minus the latter) (kLPF−hLPF). This difference (kLPF−hLPF) is also referred to as “low frequency component difference”. It may also be called “correlation value”.
The LPF 8r, 8g, 8b and the selecting unit 24h extract or generate a low frequency component of the pixel signal having the h-th spectral sensitivity characteristic at a plurality of pixel positions in the vicinity of the interpolation target pixel position. A low-frequency component of the pixel signal of the k-th spectral sensitivity characteristic at a plurality of pixel positions in the vicinity of the interpolation target pixel position by the LPF 8r, 8g, 8b and the selection unit 24k. K signal low-frequency component generating means for extracting or generating the signal is configured. Then, the h signal low frequency component generating means, the k signal low frequency component generating means, and the difference calculating means 26 are used for the kth spectral sensitivity characteristics at a plurality of pixel positions in the vicinity of the interpolation target pixel position. Difference calculating means for calculating the difference between the low frequency component of the pixel signal and the low frequency component of the pixel signal having the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region near the interpolation target pixel position is configured. Yes.

The coefficient multiplier 27 receives the output of the difference calculator 25 (kHPF−hHPF) and the output of the saturation coefficient calculator 9, and is output from the saturation coefficient calculator 9 to the output of the difference calculator 25 (kHPF−hHPF). The product βq (kHPF−hHPF) is output as a decorrelation value by multiplying the saturation coefficient β by a predetermined constant q. The output of the coefficient multiplication unit 27 is used as the output of the decorrelation value calculation unit 12.
The coefficient multiplying unit 28 multiplies the output (kLPF-hLPF) of the difference calculating unit 26 by a predetermined constant r and outputs a product r (kLPF-hLPF).
The adding unit 29 outputs the pixel value h output from the selecting unit 21, the uncorrelated value βq output from the uncorrelated value calculating unit 12 (kHPF−hHPF), and the value r (kLPF output from the coefficient multiplying unit 28. -HLPF) and the sum h + βq (kHPF-hHPF) + r (kLPF-hLPF)
Is output.
The output of the adding means 29 is used as the color component value (interpolation value) of the kth color of the interpolation target pixel.

  Among the above, the high-pass filters 7r, 7g, and 7b and the selection unit 23h change the change component (high-frequency component) of the h-th spectral sensitivity characteristic pixel signal at a plurality of pixel positions in the vicinity of the interpolation target pixel position. ) Is extracted or generated, and the high-pass filters 7r, 7g, and 7b and the selection unit 23k are used to generate the k-th change component at a plurality of pixel positions in the vicinity of the interpolation target pixel position. A k signal change component generating means for extracting or generating a change component (high frequency component) of the pixel signal of the spectral sensitivity characteristic is configured.

  Further, the difference calculation means 25 and coefficient multiplication means 27 shown in FIG. 12 and the selection means 21, difference calculation means 26, coefficient multiplication means 28 and addition means 29 shown in FIG. The decorrelation value (βq) between the k-th signal and the h-th signal determined based on the difference between the obtained value and the value obtained by the h signal change component generation means (kHPF−hHPF) and the saturation coefficient β. (KHPF−hHPF), the difference (kLPF−hLPF) between the value (kLPF) obtained by the k signal low frequency component generation means and the value (hLPF) obtained by the h signal low frequency component generation means, and the interpolation target pixel Interpolation value calculation means for obtaining the kth pixel signal at the interpolation target pixel position by adding the hth pixel signal (h) at the position is configured.

The interpolation value calculation means in the example shown in the figure is a selection means 21 for selecting a pixel signal of one color (hth color) at the position of the interpolation target pixel, and difference calculation for obtaining a difference (kLPF-hLPF) between low frequency components. And a pixel signal of one color (hth color) at the position of the interpolation target pixel selected by the selection unit 21 and a decorrelation value βq (kHPF−hHPF) obtained as an output of the difference calculation unit 25. And the addition unit 29 adds the difference between the low frequency components (kLPF−hLPF) obtained by the difference calculation unit 26 and the first predetermined coefficient (r) multiplied by the coefficient multiplication unit 28. A pixel signal of another color (kth color) at the target pixel position is obtained.
In the above example, the non-correlation value βq (kHPF−hHPF) is calculated by adding the saturation coefficient β calculated by the saturation coefficient calculation unit 9 to the difference between the high frequency components calculated by the difference calculation unit 25 (kHPF−hHPF) and the second coefficient It is obtained by multiplying by a predetermined coefficient (q).

  The interpolation value calculated by the interpolation value calculation means is stored in, for example, a two-dimensional memory (any one of 6r, 6g, 6b) for the pixel signal of the kth color, or is output from the output terminal 11.

  Hereinafter, the above interpolation method will be described in more detail.

As described above, FIG. 4 shows an arrangement of G signals on the HV coordinate plane.
The G signal shown in the figure is a signal obtained through the G color filter originally disposed on the image sensor 2, and is a signal obtained through the color filter, and blank portions are other R and B color filters. Are arranged, the G color signal is missing. It is necessary to interpolate the G signal at this missing location.
As a conventional method for interpolation, there is an average interpolation method (bilinear interpolation) using an average value of surrounding pixels, but high-precision interpolation cannot be expected at a portion where a signal change is large.

  Therefore, in the present embodiment, it is possible to accurately perform interpolation even in an area where there is no similarity in the color component value change in the area near the interpolation target pixel. Interpolation is always performed by the optimum interpolation method regardless of how the color component values change in the nearby region, and the interpolation calculation in the calculation means 10 is expressed by the following equation (1).

k (i, j)
= H (i, j) + βq {kHPF (i, j) -hHPF (i, j)}
+ R {kLPF (i, j) -hLPF (i, j)}
... (1)

In equation (1), k (i, j) is a color signal that is missing at coordinates (i, j) on the image sensor 2 and is a color signal to be interpolated. h (i, j) is a color signal (having a known value) that exists in advance at the position (i, j). kHPF and hHPF are HPF values calculated by a predetermined calculation from the k signal and the h signal at the position (i, j) and the surrounding pixel positions, respectively. kLPF and hLPF are LPF values calculated by another predetermined calculation from the k signal and h signal at the position (i, j) and the surrounding pixel positions, respectively.
Here, the HPF value is a true spatial high-frequency component estimated from a known pixel signal (image sensor output), and the LPF value is a true value estimated from a known pixel signal (image sensor output). It is a spatial low frequency component. The true value here refers to a value of a spatially continuous pixel signal that will be obtained when the pixel interval of the image sensor is infinitesimal and there is no photoelectric conversion error.
q and r are predetermined constants.

β is a coefficient determined by the saturation coefficient calculation means 9 based on the saturation (chroma) of the signal at the position (i, j) and the surrounding pixel positions. When the saturation is high, the value of β is When the value increases and the saturation is low, the value of β is set to be small. Therefore, when the saturation is low, the effect of suppressing the decorrelation value determined based on {kHPF (i, j) −hHPF (i, j)} works. Since the correlation between different colors tends to be strong in the achromatic color portion, the interpolation accuracy is improved by suppressing the non-correlation value in the achromatic color portion.
Thus, by changing the value of β, the same interpolation processing can be performed for both the achromatic color portion and the chromatic color portion, so that the circuit scale can be reduced as compared with the case where the interpolation is performed while switching a plurality of processing. Further, since the same processing is performed for all the pixels, an overall natural interpolation result can be obtained. On the other hand, if interpolation is performed while switching between a plurality of different processes, the interpolation results may not be continuously connected, which may cause false colors.

The value of β may be set as in the following formulas (2a) and (2b), for example.

c = | G′−R ′ | + | G′−B ′ | (2a)

When c ≦ Th1, β = f (c) = N × c / Th1
When c> Th1, β = f (c) = N
... (2b)

In Expression (2a), R ′, G ′, and B ′ represent R, G, and B pixel signals at the position (i, j), respectively. As this pixel signal, the pixel signal R, G, B of the pixel position output from the two-dimensional memory 6r, 6g, 6b is used as it is for the light-receiving color pixel signal at each pixel position. For the pixel signal of the insufficient color, a pixel signal having a value obtained by averaging, replacement, or the like based on the pixel signal in the area near the pixel position may be used, and the low-pass filters 8r, 8g, 8b You may use the pixel signal produced | generated by the same low pass filter. When the low-pass filter is used, the low-pass filters 8r, 8g, and 8b can also be used as the pixel signal generating units 9r, 9g, and 9b.
c represents saturation. When the saturation is low, the color components have the same value, so the difference between the color components is small. Conversely, when the saturation is high, the variation between the color components tends to increase. Therefore, for example, as in the above equation (2a), the difference between the R and B components with respect to the G component may be used as the saturation.

Note that the saturation c may be obtained by the following equation (2c) or (2d) instead of the equation (2a).
c = | Gm−Rm | + | Gm−Bm | (2c)
c = | Rm−Zm | + | Gm−Zm | + | Bm−Zm | (2d)
In the above formulas (2c) and (2d), Rm, Gm, and Bm are average values of R, G, and B pixel signals at the position (i, j), respectively, and Zm is (i, j) It is the average value of the pixel signals of all the pixels at the position.

  The saturation coefficient β is calculated from the saturation c according to the function f (c). Expression (2b) is shown as a graph in FIG. When the saturation c is a constant value, that is, the first threshold Th1 or less, the saturation coefficient β is changed according to the magnitude of the value. However, when the saturation c exceeds the first threshold Th1, the saturation coefficient β is fixed to N. To do. The first threshold Th1 and the upper limit N of the saturation coefficient may be appropriately set according to each system, apparatus, etc., but the value of N is, for example, 1.0, and the saturation coefficient is 0.0-1. You may change between 0.

  Further, in a system or apparatus that has a large effect obtained by setting the decorrelation value to “0” when the saturation c is lower than the second threshold Th2, as shown in FIG. 15, the second threshold Th2 is used. In the following, the value of the saturation coefficient β is set to “0”, and the value of the saturation coefficient β is equal to or greater than the first threshold Th3 (corresponding to the first threshold Th1 in FIG. 14 but indicated by a different sign). May be set to the upper limit value N, and the value of the saturation coefficient β may be increased linearly between the first threshold value Th3 and the second threshold value Th2. At this time, Th2 = Th3 may be satisfied.

In FIG. 14, the value of β is set to 0 only when the value of saturation c is 0, whereas in FIG. 15, β is set to 0 until saturation c reaches Th2. The range for completely suppressing the decorrelation value is determined by the value of Th2. For example, when capturing an image with low saturation as a whole or when emphasizing the resolution, the value of Th2 is increased, and a range in which the decorrelation value is suppressed is widened. By doing so, since β becomes 0 even when the saturation c is not 0, a strong correlation method is widely applied in the case of an achromatic color, and the image quality can be improved.
Further, even if the image is a monochrome image, the saturation c is not always 0 in a situation where there are pixels that are uneven or missing in the image. Even in such a case, when β is 0 when it is less than the threshold Th2, the image quality can be improved by treating the range less than the threshold Th2 as an achromatic color.

  Further, the saturation coefficient β may be set by a look-up table method. By using the look-up table method, it is possible to obtain a characteristic in which the saturation coefficient β changes in a complex manner with respect to the saturation c.

  Furthermore, the saturation c may be calculated using Cr, Cb after converting the R signal, G signal, and B signal into Y, Cr, Cb. At this time, Y, Cr, and Cb may be obtained approximately, for example, as in Expression (3).

Y = (Rave + GAve + Bave) / 3
Cr = | Y-Rave |
Cb = | Y-Bave |
c = Cr + Cb
... (3)

An example of the saturation coefficient calculating means 9 is shown in FIG. The saturation coefficient calculation means 9 shown in the figure is for obtaining the saturation coefficient β by the equations (2a) and (2b). The pixel signal generation means 9r, 9g, 9b, the saturation calculation means 9d, and the saturation coefficient Generating means 9e.
The pixel signal generation unit 9r generates a pixel signal R ′ at each pixel position based on the pixel signal read from the two-dimensional memory 6r. For example, when R is a light receiving color at each pixel position, the pixel signal R of the pixel position output from the two-dimensional memory 6r is output as it is as the pixel signal R ′, and R is an insufficient color at each pixel position. In this case, a pixel signal having a value obtained by averaging, replacement, or the like based on a pixel signal in a region near the pixel position may be used. When using the average, for example, a simple average of pixel signals of pixels having the same color pixel signal in a region where the distance between the pixels in the horizontal and vertical directions from each pixel position is “1” may be obtained. Here, “distance between pixels” means a difference in coordinate values between one pixel and another pixel in the horizontal direction or the vertical direction. In the case of replacement, the value of a pixel signal at a pixel position whose coordinate value differs by “1” in the horizontal direction or the vertical direction is used.
The pixel signal generation units 9g and 9b are configured in the same manner as the pixel signal generation unit 9r. The pixel signal generation unit 9g is based on the pixel signal read from the two-dimensional memory 6g, and the pixel signal G at each pixel position. ', And the pixel signal generator 9b generates a pixel signal B' at each pixel position based on the pixel signal read from the two-dimensional memory 6b.

The saturation calculation unit 9d calculates, for example, the above equation (2a) using the pixel signals R ′, G ′, and B ′ output from the pixel signal generation units 9r, 9g, and 9b, and the saturation c Ask for. The saturation calculation means 9d is, for example, a means for obtaining the absolute value of the difference between G ′ and R ′, a means for obtaining the absolute value of the difference between G ′ and B ′, and the sum of the absolute values obtained by these means. And means for obtaining.
The saturation coefficient generation unit 9e outputs the saturation coefficient β having the relationship of, for example, the equation (2b) using the saturation c calculated by the saturation calculation unit 9d. The saturation coefficient generation means 9e can be configured by a lookup table, for example.

The meaning of the calculation formula shown in Formula (1) will be described with reference to FIGS. In these drawings, the color signal level in each pixel and the position of each pixel on the image sensor 2 are shown. In these drawings, for the sake of simplicity, only one line of the image sensor 2 is described, and the calculation is limited to one-dimensional direction. Described above is the arrangement of each color filter, where h is an h pixel, k is a k pixel, and parentheses () of each pixel are coordinates indicating the pixel position. Curve a is the true value of the k signal, and curve b is the true value of the h signal. On the curves a and b, black circles (●) indicate pixel signal values of the k signal and h signal obtained from the image sensor 2. Curves c and d are the LPF values of the k and h signals, respectively, and curves e and f are the HPF values of the k and h signals, respectively. With reference to these drawings, a method for performing pixel interpolation of the k signal at the pixel position (i, j) will be specifically described.
FIG. 17 shows a case where there is a positive correlation between the k signal and the h signal, FIG. 18 and FIG. 19 show a case where there is no correlation between the k signal and the h signal, and FIG. The case where there is a negative correlation between the signal and the h signal is shown.

First, the case where there is a positive correlation between the k signal and the h signal will be described with reference to FIG. The difference between the curves c and d at the pixel position (i, j) is a value r (kLPF-hLPF) proportional to the difference between the low frequency components (kLPF-hLPF). Further, the difference between the curves e and f at the pixel position (i, j) is the difference between the high frequency components (kHPF−hHPF). The HPF values shown by the curves e and f are the same value when the changes of the k signal and the h signal are the same, so that they are overlapped and the high frequency component difference (kHPF−hHPF) is “0”.
In the bilinear method according to the prior art, the k signal at the pixel position (i−1, j) and (i + 1, j) is used, and the average value is set as the k signal at the pixel position (i, j). The signal level interpolated by the bilinear method is indicated by white triangles (Δ) in FIG. 17, but an interpolation error has occurred with the true value to be obtained.
In the interpolation method according to the present embodiment, the known value h (i, j) at the pixel position (i, j) is multiplied by the high-frequency component difference (kHPF−hHPF) by the coefficients β and q (both the coefficients β and q are combined). Multiplication) and the difference between the low frequency components (kLPF−hLPF) multiplied by the coefficient r are added, and when there is a correlation between the k signal and the h signal as shown in FIG. The difference is substantially “0”. After all, the known value h (i, j) at the pixel position (i, j) is added after the low frequency component difference (kLPF−hLPF) is multiplied by the coefficient r. That is the interpolation signal k (i, j). This interpolation signal k (i, j) is indicated by white circles (◯) in FIG. It can be seen that pixel interpolation is realized with high accuracy for the true value. Thus, according to the present embodiment, pixel interpolation can be performed with high accuracy even when the correlation between color signals is large.

  Next, a case where there is no correlation between color signals will be described. 18 and 19 show the color signal level in each pixel and the position of each pixel on the image sensor 2. In the signals shown in FIGS. 18 and 19, the k signal is constant, and there is no correlation between changes in the k signal and the h signal.

First, a description will be given with reference to FIG. In the example of FIG. 18, since there is no change in the k signal, the HPF value kHPF of the k signal indicated by the curve e is “0”. Interpolation using only the color correlation change by the method disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2001-197512) adds a value proportional to (kLPF-hLPF) to the signal of h (i, j). It corresponds to doing.
In this case, the pixel-interpolated signal level is a signal level indicated by a white square mark (□) in FIG. Since the signal level indicated by the white square mark is a value at a position away from the true value of the k signal, it can be seen that an interpolation error has occurred. This is because the k signal to be interpolated at the pixel position (i, j) has no correlation with the reference h signal.
On the other hand, in the pixel interpolation method according to the present embodiment, h signal h (i, j) at pixel position (i, j) is added to (kLPF-hLPF) multiplied by coefficient r, and (kHPF-hHPF) is further added. ) Multiplied by coefficients β and q. Since kHPF is “0”, (kHPF−hHPF) at the pixel position (i, j) is a negative value. Therefore, in the interpolation method according to the present embodiment, the interpolated signal level is the position of the white circle (o), and is interpolated with high accuracy with respect to the true value of the k signal. In the present embodiment, how much the signal change between the k signal and the h signal is not correlated is determined as a value of (kHPF−hHPF). For this reason, a value determined based on (kHPF−hHPF), specifically, a value obtained by multiplying (kHPF−hHPF) by coefficients β and q is called “non-correlated value”. On the other hand, the difference between the low frequency components (kLPF−hLPF) represents the degree of correlation, and the closer the degree of correlation, the closer to a certain value. Therefore, when (kLPF−hLPF) multiplied by coefficient r and (kHPF−hHPF) multiplied by coefficients β and q are added to h (i, j), there is no correlation between signals. However, pixel interpolation can be performed with high accuracy.

In contrast to FIG. 18, FIG. 19 shows another example in which the k signal changes and the h signal does not change and there is no correlation between the two color signals. Interpolation using only the color correlation change by the method disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2001-197512) adds the signal level of (kLPF-hLPF) to the signal of h (i, j). It corresponds to that.
In this case, the pixel-interpolated signal level is a signal level indicated by a white square mark (□) in FIG. Since the signal level indicated by the white square mark is a value at a position away from the true value of the k signal, it can be seen that an interpolation error has occurred.
On the other hand, in the pixel interpolation method according to the present embodiment, h signal h (i, j) at pixel position (i, j) is added to (kLPF-hLPF) multiplied by coefficient r, and (kHPF-hHPF) is further added. ) Multiplied by coefficients β and q. Since hHPF is “0”, (kHPF−hHPF) at the pixel position (i, j) is a positive value. Therefore, in the interpolation method according to the present embodiment, the interpolated signal level is the position of the white circle (o), and is interpolated with high accuracy with respect to the true value of the k signal. In the present embodiment, (kLPF−hLPF) multiplied by coefficient r and (kHPF−hHPF) multiplied by coefficients β and q are added to h (i, j) as shown in FIG. Even when there is no correlation between signals, pixel interpolation can be performed with high accuracy.

18 and 19, the case where there is no correlation between the color signals has been described. Next, the case where there is a negative correlation between the color signals will be described. FIG. 20 shows a case where there is a negative correlation between the k signal and the h signal. The sum of the difference Δ (ac) between the curve a and the chain line c and the difference Δ (db) between the curve d and the curve b at the pixel position (i, j) is the difference between the high frequency components (kHPF−hHPF). Is equal to a non-correlation value βq (kHPF−hHPF) proportional to. Pixel interpolation by the bilinear interpolation method has a signal level indicated by a white triangle mark (Δ), and an interpolation error occurs with respect to a true value as in FIG. Further, in the case of a negative correlation, as indicated by a white square mark (□) in pixel interpolation using only a color correlation change by the method disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2001-197512). Furthermore, the interpolation error increases. In the interpolation method according to the present embodiment, pixel interpolation can be realized with high accuracy as indicated by white circles (◯).
As described above, highly accurate pixel interpolation can be performed in any case where there is a positive correlation between color signals, no correlation, or a negative correlation.

The pixel interpolation calculation process will be specifically described below. FIG. 21 is a flowchart showing a procedure of processing for calculating an interpolation value in the saturation coefficient calculation means 9, the decorrelation value calculation means 12, and the calculation means 10. As described in the flowchart, the calculation of the interpolation value includes the following six processes.
Step S1: Processing for obtaining a G signal (GonR) at the R pixel position.
Step S2: A process for obtaining a G signal (GonB) at the B pixel position.
Step S3: Processing for obtaining an R signal (RonG) at the G pixel position.
Step S4: Processing for obtaining a B signal (BonG) at the G pixel position.
Step S5: Processing for obtaining the R signal (RonB) at the B pixel position.
Step S6: Processing for obtaining the B signal (BonR) at the R pixel position.
These six processes are
“Process for obtaining a pixel signal of k color (k = R, G, or B, where h is different from k) at a pixel position where a pixel signal of h color (h = R, G, or B) exists "
It can be said in general. Each of these six processes is performed for all pixel positions on the screen (within one frame).

  The processing of steps S1 to S6 is performed by, for example, the saturation coefficient calculating unit 9 and the non-correlation by a control unit (not shown) (the control unit 30a in FIG. 12 and the control unit 30b in FIG. This is executed by sequentially operating the value calculation means 12 and the calculation means 10.

  When all of the above six processes are completed, pixel signals of insufficient colors are gathered at all pixels at all pixel positions on one screen.

  First, the calculation process of step S1 will be described. In FIG. 4, attention is focused on the coordinates (i, j) of a pixel in which the G signal does not exist in advance (value is unknown). The LPF value GLPF of the G signal at coordinates (i, j) is calculated by the following equation (4), for example.

GLPF (i, j) = [{G (i−3, j) + G (i−1, j) + G (i + 1, j) + G (i + 3, j)} / 4
+ {G (i, j-3) + G (i, j-1) + G (i, j + 1) + G (i, j + 3)} / 4] / 2
... (4)

  The parentheses () in each signal mean the pixel coordinates.

  The LPF value GLPF of the G signal at the pixel position (i + 1, j) where the G signal exists in advance is calculated by the following equation (5).

GLPF (i + 1, j) = [{G (i-3, j) / 8 + G (i-1, j) / 4 + G (i + 1, j) / 4 + G (i + 3, j) / 4 + G (i + 5, j) / 8}
+ {G (i + 1, j-4) / 8 + G (i + 1, j-2) / 4 + G (i + 1, j) / 4 + G (i + 1, j + 2) / 4 + G (i + 1, j + 4) / 8}] / 2
... (5)

  Since the arrangement of G pixels is a repetition of the same pixel interval, the LPF value of the G signal can be calculated by the above equations (4) and (5). The LPF value of the G signal is calculated by the LPF 8g shown in FIG.

  Further, the HPF value GHPF of the G signal at the pixel position (i, j) where the G signal does not exist in advance is calculated by the following equation (6).

GHPF (i, j) = [{− G (i−3, j) + G (i−1, j) + G (i + 1, j) −G (i + 3, j)} + {− G (i, j−3) ) + G (i, j-1) + G (i, j + 1) -G (i, j + 3)}] / 2
... (6)

  The HPF value GHPF of the G signal at the pixel position (i + 1, j) where the G signal is present in advance is calculated by the following equation (7).

GHPF (i + 1, j) = [{− G (i−3, j) / 4−G (i−1, j) + 2.5G (i + 1, j) −G (i + 3, j) −G (i + 5, j) ) / 4} + {− G (i + 1, j−4) / 4−G (i + 1, j−2) + 2.5G (i + 1, j) −G (i + 1, j + 2) −G (i + 1, j + 4) / 4 }] / 2
... (7)

  Since the arrangement of the G pixels is a repetition of the same pixel interval, the HPF value of the G signal can be calculated by the above equations (6) and (7). The calculation of the HPF value of the G signal is calculated by the HPF 7 g in FIG. 1 and input to the decorrelation value calculation means 12.

  The LPF value and HPF value of the R signal are calculated by the following equations. First, in the j-th row where no R signal exists in any pixel, values are calculated from the upper and lower rows. The LPF value RLPF of the R signal at the pixel coordinates (i, j) is calculated by the following equation (8).

RLPF (i, j) = [[{R (i-3, j-1) + R (i-3, j + 1)} / 2+ {R (i-1, j-1) + R (i-1, j + 1) } / 2 + {R (i + 1, j-1) + R (i + 1, j + 1)} / 2+ {R (i + 3, j-1) + R (i + 3, j + 1)} / 2] / 4
+ [{R (i-1, j-3) + R (i + 1, j-3)} / 2+ {R (i-1, j-1) + R (i + 1, j-1)} / 2+ {R (i −1, j + 1) + R (i + 1, j + 1)} / 2+ {R (i−1, j + 3) + R (i + 1, j + 3)} / 2] / 4] / 2
(8)

  The LPF value RLPF of the R signal at the coordinates (i + 1, j) is calculated by the following equation (9).

RLPF (i + 1, j) = [[{R (i-1, j-1) + R (i-1, j + 1)} / 2+ {R (i + 1, j-1) + R (i + 1, j + 1)} / 2+ { R (i + 3, j-1) + R (i + 3, j + 1)} / 2] / 3
+ {R (i + 1, j-3) + R (i + 1, j-1) + R (i + 1, j + 1) + R (i + 1, j + 3)} / 4] / 2
... (9)

  On the other hand, the LPF value RLPF of the R signal at the pixel position (i, j + 1) in the row where the R signal exists in advance, for example, j + 1 row, is calculated by the following equation (10).

RLPF (i, j + 1) = [{R (i-3, j + 1) + R (i-1, j + 1) + R (i + 1, j + 1) + R (i + 3, j + 1)} / 4
+ [{R (i-1, j-1) + R (i + 1, j-1)} / 2+ {R (i-1, j + 1) + R (i + 1, j + 1)} / 2+ {R (i-1, j + 3) ) + R (i + 1, j + 3)} / 2] / 3] / 2
(10)

  Furthermore, the LPF value RLPF of the R signal at the pixel position (i + 1, j + 1) in the row where the R signal exists in advance, for example, j−1 row, is calculated by the following equation (11).

RLPF (i + 1, j + 1) = [R (i−3, j + 1) / 8 + R (i−1, j + 1) / 4 + R (i + 1, j + 1) / 4 + R (i + 3, j + 1) / 4 + R (i + 5, j + 1) / 8 + R ( i + 1, j-3) / 8 + R (i + 1, j-1) / 4 + R (i + 1, j + 1) / 4 + R (i + 1, j + 3) / 4 + R (i + 1, j + 5) / 8] / 2
... (11)

  Since the array of R pixels is a repetition of (i, j), (i + 1, j), (i, j + 1), (i + 1, j + 1), the LPF of the R signal is expressed by the above equations (8) to (11). A value can be calculated. The LPF value of the R signal is calculated by the LPF 8r shown in FIG.

  Further, the HPF value RHPF of the R signal at the coordinates (i, j) is calculated by the following equation (12).

RHPF (i, j) = [− {R (i−3, j−1) + R (i−3, j + 1)} / 2+ {R (i−1, j−1) + R (i−1, j + 1) } / 2 + {R (i + 1, j-1) + R (i + 1, j + 1)} / 2- {R (i + 3, j-1) + R (i + 3, j + 1)} / 2
− {R (i−1, j−3) + R (i + 1, j−3)} / 2+ {R (i−1, j−1) + R (i + 1, j−1)} / 2+ {R (i− 1, j + 1) + R (i + 1, j + 1)} / 2- {R (i-1, j + 3) + R (i + 1, j + 3)} / 2] / 2
(12)

  The HPF value RHPF of the R signal at the coordinates (i + 1, j) is calculated by the following equation (13).

RHPF (i + 1, j) = [[-{R (i-3, j-1) + R (i-3, j + 1)} / 2/4 {R (i-1, j-1) + R (i- 1, j + 1)} / 2 + 2.5 {R (i + 1, j-1) + R (i + 1, j + 1)} / 2- {R (i + 3, j-1) + R (i + 3, j + 1)} / 2- {R ( i + 5, j-1) + R (i + 5, j + 1)} / 2/4]
+ [-R (i + 1, j-3) + R (i + 1, j-1) + R (i + 1, j + 1) -R (i + 1, j + 3)]] / 2
... (13)

  On the other hand, the HPF value RHPF of the R signal at the pixel position (i, j + 1) in the row where the R signal exists in advance, for example, j + 1 row, is calculated by the following equation (14).

RHPF (i, j + 1) = [{− R (i−3, j + 1) + R (i−1, j + 1) + R (i + 1, j + 1) −R (i + 3, j + 1)}
+ [-{R (i-1, j-3) + R (i + 1, j-3)} / 2/4 {R (i-1, j-1) + R (i + 1, j-1)} / 2 + 2 .5 {R (i-1, j + 1) + R (i + 1, j + 1)} / 2- {R (i-1, j + 3) + R (i + 1, j + 3)} / 2- {R (i-1, j + 5) + R (I + 1, j + 5)} / 2/4]] / 2
... (14)

  Further, the HPF value RHPF of the R signal at the pixel position (i + 1, j + 1) in the row where the R signal exists in advance, for example, j−1 row is calculated by the following equation (15).

RHPF (i + 1, j + 1) = [{− R (i−3, j + 1) / 4−R (i−1, j + 1) + 2.5R (i + 1, j + 1) −R (i + 3, j + 1) −R (i + 5, j + 1) ) / 4}
+ {-R (i + 1, j-3) / 4-R (i + 1, j-1) + 2.5R (i + 1, j + 1) -R (i + 1, j + 3) -R (i + 1, j + 5) / 4}] / 2
... (15)

  Since the array of R pixels is a repetition of (i, j), (i + 1, j), (i, j + 1), (i + 1, j + 1), the HPF of the R signal is obtained from the above equations (12) to (15). A value can also be calculated. The calculation of the HPF value of the R signal is calculated by the HPF 7r shown in FIG.

  Finally, regarding the LPF value and HPF value of the B signal, the arrangement of the B pixels is the same as the arrangement of the R pixels, except for the coordinate values. Therefore, similar to the formulas for calculating the LPF and HPF of the R signal shown in formulas (8) to (15), the detailed formulas are omitted because they can be calculated simply by changing their coordinates. The LPF value of the B signal is calculated by the LPF 8b shown in FIG. The calculation of the HPF value of the B signal is calculated by the HPF 7 b in FIG. 1 and input to the decorrelation value calculation means 12.

  The formulas for calculating the LPF and HPF shown above are formulas for calculating values for use in the formula (1), but are merely examples. For example, the number of pixels used in the formula for calculating the LPF value and the formula for calculating the HPF value As the coefficient, other values may be appropriately provided according to the size and resolution of the image.

  Next, a method for calculating the saturation c for calculating the saturation coefficient β will be described. In FIG. 2, attention is focused on the coordinates (i-1, j-1) of the pixel where the R signal exists. The saturation c at the pixel position (i−1, j−1) is calculated by the following equation (16), for example.

R ′ = R (i−1, j−1)
G ′ = (G (i−1, j−2) + G (i−2, j−1) + G (i, j−1) + G (i−1, j)) / 4
B ′ = (B (i−2, j−2) + B (i, j−2) + B (i−2, j) + B (i, j)) / 4
c = | G′−R ′ | + | G′−B ′ |
... (16)

  The saturation c at the pixel position (i, j) where the B signal exists is calculated by the following equation (17), for example.

R ′ = (R (i−1, j−1) + R (i + 1, j−1) + R (i−1, j + 1) + R (i + 1, j + 1)) / 4
G ′ = (G (i, j−1) + G (i−1, j) + G (i + 1, j) + G (i, j + 1)) / 4
B ′ = B (i, j)
c = | G′−R ′ | + | G′−B ′ |
... (17)

  The saturation c at the pixel position (i, j−1) where the R signal and the G signal are alternately arranged and where the G signal exists is calculated by the following equation (18), for example.

R ′ = (R (i−1, j−1) + R (i + 1, j−1)) / 2
G ′ = (G (i−1, j−2) / 4 + G (i + 1, j−2) / 4 + G (i, j−1) + G (i−1, j) / 4 + G (i + 1, j) / 4) / 2
B ′ = (B (i, j−2) + B (i, j)) / 2
c = | G′−R ′ | + | G′−B ′ |
... (18)

  The saturation c at the pixel position (i−1, j) where the B signal and the G signal are alternately arranged and where the G signal exists is calculated by the following equation (19), for example.

R ′ = (R (i−1, j−1) + R (i−1, j + 1)) / 2
G ′ = (G (i−2, j−1) / 4 + G (i, j−1) / 4 + G (i−1, j) + G (i−2, j + 1) / 4 + G (i, j + 1) / 4) / 2
B ′ = (B (i−2, j) + B (i, j)) / 2
c = | G′−R ′ | + | G′−B ′ |
... (19)

  Since the arrangement of each pixel is repeated at the same pixel interval, the saturation c at all pixel positions can be calculated by the above equations (16) to (19). When the saturation c is obtained, the saturation coefficient β is calculated based on, for example, the equation (2b). Thus, the saturation coefficient β calculated by the saturation coefficient calculation means 9 of FIG. 1 is input to the decorrelation value calculation means 12.

  The above equation for calculating the saturation c is merely an example, and the number of pixels and coefficients used may be appropriately set according to the size and resolution of the image. Further, the saturation c is calculated using only pixel signals that exist in advance, but may be calculated using pixel signals calculated by interpolation.

  The LPF value, the HPF value, and the saturation coefficient β for all the pixel positions of the R, G, and B signal images are obtained by the above-described equations (4) to (19). In step S1, the missing G signal at the position of the R pixel is calculated. The G signal at the position of the R pixel is calculated by the following equation (20) according to the equation (1). Since the R pixel exists at a position of (i + n, j + m) (n and m are odd numbers), the coordinate value is different from the equation (1).

G (i + n, j + m) = {R (i + n, j + m) + βq (GHPF (i + n, j + m) −RHPF (i + n, j + m))} + r (GLPF (i + n, j + m) −RLPF (i + n, j + m))
... (20)

  GHPF, GLPF, RHPF, RLPF, and β shown in equation (20) are the HPF value, LPF value, and saturation coefficient β calculated in equations (4) to (19). The constants q and r may be determined in advance so that the image is optimally interpolated. For example, pixel interpolation can be performed satisfactorily with q = 0.25 and r = 1, but is not limited to this value. FIG. 22 two-dimensionally shows the G signal gr obtained as a result of interpolation according to Expression (20) at the corresponding R pixel position.

  Next, the processing proceeds to step S2. Step S2 interpolates the missing G signal at the B pixel position. The G signal at the position of the B pixel is calculated by the following equation (21) according to the equation (1). Since the B pixel exists at the position (i + s, j + t) (where s and t are even numbers), the coordinate value differs from that of the equation (1).

G (i + s, j + t) = {B (i + s, j + t) + βq (GHPF (i + s, j + t) −BHPF (i + s, j + t))} + r (GLPF (i + s, j + t) −BLPF (i + s, j + t))
... (21)

FIG. 23 two-dimensionally shows the G signal gb obtained as a result of the interpolation by the equation (21) at the corresponding B pixel position. FIG. 23 also shows the result gr of the interpolation by equation (20).
The G signal at all pixel positions is obtained by the interpolation of the G signal at the R pixel position according to Expression (20) and the interpolation of the G signal at the B pixel position according to Expression (21).

  Next, the processing proceeds to step S3. Step S3 interpolates the missing R signal at the G pixel position. The R signal at the position of the G pixel is calculated by the following equations (22) and (23) according to equation (1). Since the G pixel exists at the position (i + s, j + m) (s is an even number, m is an odd number) and (i + n, j + t) (n is an odd number, t is an even number), the equation (1) is coordinated accordingly. The value will be different.

R (i + s, j + m) = {G (i + s, j + m) + βq (RHPF (i + s, j + m) −GHPF (i + s, j + m))} + r (RLPF (i + s, j + m) −GLPF (i + s, j + m))
... (22)

R (i + n, j + t) = {G (i + n, j + t) + βq (RHPF (i + n, j + t) −GHPF (i + n, j + t))} + r (RLPF (i + n, j + t) −GLPF (i + n, j + t))
... (23)

FIG. 24 two-dimensionally shows the R signal rg obtained as a result of interpolation according to the equations (22) and (23) at the corresponding G pixel positions.
In the equations (22) and (23), RLPF, RHPF, GLPF, GHPF, and β are the LPF value, HPF value, and saturation coefficient β shown in the equations (4) to (19). , GLPF and GHPF may be newly calculated using the interpolation values gr and gb calculated in step S1 and step S2. In this case, as shown in FIG. 1, the interpolation values gr and gb calculated by the calculation means 10 are once output to the two-dimensional memory 6g, temporarily stored and held, and then calculated again by the HPF 7g and LPF 8g. Become.

  Next, the processing proceeds to step S4. Step S4 interpolates the missing B signal at the position of the G pixel. The B signal at the position of the G pixel is calculated by the following equations (24) and (25) according to equation (1). Since the G pixel exists at the position (i + s, j + m) (s is an even number, m is an odd number) and (i + n, j + t) (n is an odd number, t is an even number), the equation (1) is coordinated accordingly. The value will be different.

B (i + s, j + m) = {G (i + s, j + m) + βq (BHPF (i + s, j + m) −GHPF (i + s, j + m))} + r (BLPF (i + s, j + m) −GLPF (i + s, j + m))
... (24)

B (i + n, j + t) = {G (i + n, j + t) + βq (BHPF (i + n, j + t) −GHPF (i + n, j + t))} + r (BLPF (i + n, j + t) −GLPF (i + n, j + t))
... (25)

FIG. 25 two-dimensionally shows the B signal bg obtained as a result of interpolation according to the equations (24) and (25) at the corresponding G pixel positions.
In this equation (24) and equation (25), BLPF, BHPF, GLPF, GHPF, and β are the above-mentioned LPF value, HPF value, and saturation coefficient β, but for GLPF and GHPF, step S1 and step A new calculation may be performed using the interpolation values gr and gb calculated in S2.

  Next, the processing proceeds to step S5. Step S5 interpolates the missing R signal at the B pixel position. The R signal at the position of the B pixel is calculated by the following equation (26) according to the equation (1). Since the B pixel exists at the position (i + s, j + t) (where s and t are even numbers), the coordinate value differs from that of the equation (1).

R (i + s, j + t) = {G (i + s, j + t) + βq (RHPF (i + s, j + t) −GHPF (i + s, j + t))} + r (RLPF (i + s, j + t) −GLPF (i + s, j + t))
... (26)

FIG. 26 two-dimensionally shows the R signal rb obtained as a result of the interpolation by the equation (26) at the corresponding B pixel position. FIG. 26 also shows the R signal rg obtained as a result of the interpolation according to the equations (22) and (23) at the corresponding G pixel positions. As a result of the interpolation according to the equations (22) and (23) and the interpolation according to the equation (26), R signals of all the pixels are obtained.
In Equation (26), RLPF, RHPF, GLPF, GHPF, and β are the above-described LPF value, HPF value, and saturation coefficient β. For GLPF and GHPF, the interpolation values calculated in steps S1 and S2 are used. You may newly calculate using gr and gb. Also, RLPF and RHPF may be newly calculated using the interpolation value rg calculated in step S3.

  Next, the processing proceeds to step S6. Step S6 interpolates the missing B signal at the position of the R pixel. The B signal at the position of the R pixel is calculated by the following equation (27) according to the equation (1). Since the R pixel exists at a position of (i + n, j + m) (n and m are odd numbers), the coordinate value is different from the equation (1).

B (i + n, j + m) = {G (i + n, j + m) + βq (BHPF (i + n, j + m) −GHPF (i + n, j + m))} + r (BLPF (i + n, j + m) −GLPF (i + n, j + m))
... (27)

FIG. 27 two-dimensionally shows the B signal br obtained as a result of interpolation according to the equation (27) at the corresponding R pixel position. FIG. 27 also shows the G pixel positions corresponding to the B signals bg obtained as a result of interpolation according to equations (24) and (25). As a result of the interpolation by the equations (24) and (25) and the interpolation by the equation (27), the B signals of all the pixels are prepared.
In Expression (27), BLPF, BHPF, GLPF, GHPF, and β are the above-described LPF value, HPF value, and saturation coefficient β, but for GLPF and GHPF, the interpolation values calculated in step S1 and step S2 are used. You may newly calculate using gr and gb. Also, BLPF and BHPF may be newly calculated using the interpolation value bg calculated in step S4.

  As described above, the color signal missing in each pixel is interpolated by the calculation from step S1 to step S6, and R, G, B signals of all the pixels are obtained.

  The high-pass filters 7r, 7g, and 7b, the low-pass filters 8r, 8g, and 8b, the saturation coefficient calculation unit 9, the decorrelation value calculation unit 12, and the calculation unit 10 described in the above example are performed by software, that is, by a programmed computer. It can also be realized. In that case, the process of step S1-S6 is performed as follows.

FIG. 28 is a flowchart showing the procedure of step S1. In step S1, since the missing G signal at the position of the R pixel is calculated, a pixel in which the R signal exists in advance is selected as the target pixel (step S10).
A G-color LPF value and an R-color LPF value at the selected target pixel position are calculated (steps S11 and S12), and a value obtained by multiplying the difference by a predetermined constant r is set as a low-frequency component difference (step S13). .

Next, the G-color HPF value and the R-color HPF value at the selected target pixel position are calculated (steps S14 and S15). Further, the saturation coefficient β is calculated using signals of the pixel of interest and the surrounding pixels (step S16), and the difference between the HPF value of the R color and the G color is multiplied by the saturation coefficient β and a predetermined constant q. A correlation value is calculated (step S17).
Then, the G color signal is calculated by adding the difference between the low frequency components and the decorrelation value to the R color signal at the target pixel position (step S18). The obtained G color signal value is output to the two-dimensional memory 6g (step S19). The above processing is repeatedly executed for all pixels in which the R signal is present in advance (step S20).
The processing of steps S2 to S6 is the same as that of step S1. That is, each procedure of steps S2 to S6 is shown as replacing “R” and “G” in the flowchart of FIG. 28 with the color (h) of the pixel of interest and the color (k) to be obtained by interpolation. .

The correspondence between each step shown in FIG. 28 and the members shown in FIGS. 1, 12, 13, and 16 is as follows. That is, the processing in step S10 corresponds to the processing in the selection means 21 and the control means (the control means 30a and 30b in FIGS. 12 and 13 form part of them, but the whole is not shown). The processing corresponds to the processing in the low-pass filter 8g and the selection unit 24k, the processing in step S12 corresponds to the processing in the low-pass filter 8r and the selection unit 24h, and the processing in step S13 includes the difference calculation unit 26 and the coefficient multiplication unit 28. The processing in step S14 corresponds to the processing in the high-pass filter 7g and the selection means 23k, the processing in step S15 corresponds to the processing in the high-pass filter 7r and the selection means 23h, and the processing in step S16 is Corresponding to the processing in the saturation coefficient calculation means 9, the processing in step S17 is Corresponding to the processing in the calculating means 25 and the coefficient multiplying means 27, the processing in step S18 corresponds to the processing in the adding means 29, and the processing in step S19 is the data from the computing means 10 to the two-dimensional memories 6r, 6g, 6b. Supports transfer and writing.
Control of each means in step S20 and each step is performed by a control means (the control means 30a and 30b in FIGS. 12 and 13 form a part thereof, but the whole is not shown).

  FIG. 29 is a flowchart showing the procedure of step S16. In step S16, the saturation coefficient β is calculated. First, the average value of each color signal is calculated using the signals of the target pixel and the surrounding pixels (step S30). The sum of the absolute value of the difference between the average value of the G signal and the average value of the R color and the absolute value of the difference between the average value of the G signal and the average value of the B color is obtained as saturation c (step S31). The value of the saturation c is compared with the threshold value Th1, and the saturation coefficient β is set (steps S32, S33, S34).

  The process shown in FIG. 29 corresponds to the process performed by the saturation coefficient calculation means 9, and the correspondence between the steps in FIG. 29 and the members shown in FIG. 16 is as follows. That is, the process in step S30 corresponds to the process in the pixel signal generation units 9r, 9g, and 9b, the process in step S31 corresponds to the process in the saturation calculation unit 9d, and the processes in steps S32 to S34 are performed in saturation. This corresponds to the processing in the coefficient generation means 9e.

Embodiment 2. FIG.
Next, an image signal processing apparatus according to the second embodiment will be described. The overall configuration of the imaging apparatus including the image signal processing apparatus according to the second embodiment is as shown in FIG. 1, but the configuration of the computing means 10 is different from that of the first embodiment. FIG. 30 shows the configuration of the computing means of the second embodiment. The calculation means shown in FIG. 30 is generally the same as the calculation means of FIG. 13, but includes a ratio calculation means 32 instead of the difference calculation means 26, and an addition means 33 and a multiplication means 34 instead of the addition means 29. It differs in having a combination of

  The ratio calculation means 32 obtains a ratio kLPF / hLPF between the output of the selection means 24k and the output of the selection means 24h.

The coefficient multiplication means 28 multiplies the output kLPF / hLPF of the ratio calculation means 32 by a predetermined coefficient r and outputs the product r (kLPF / hLPF).
The adding unit 33 adds the pixel value h output from the selecting unit 21 and the uncorrelated value βq (kHPF−hHPF) output from the uncorrelated value calculating unit 12 and adds the sum h + βq (kHPF−hHPF).
Is output.

The multiplication unit 34 outputs the output h + βq of the addition unit 33 (kHPF−hHPF).
And the output r (kLPF / hLPF) of the coefficient multiplication means 28
And the product {h + βq (kHPF−hHPF)} × r (kLPF / hLPF)
Is output.
The output of the multiplication unit 34 is used as the color component value (interpolation value) of the kth color of the interpolation target pixel.

  Also in the second embodiment, the LPFs 8r, 8g, and 8b and the selection unit 24h are used to select the low frequency components of the pixel signal having the h-th spectral sensitivity characteristic at a plurality of pixel positions in the vicinity of the interpolation target pixel position. The h signal low frequency component generating means for extracting or generating is configured, and the LPFs 8r, 8g, 8b and the selecting means 24k, the kth spectral sensitivity at a plurality of pixel positions in the vicinity of the interpolation target pixel position. A k signal low frequency component generating means for extracting or generating a low frequency component of the characteristic pixel signal is configured. Then, the h signal low frequency component generation means and the k signal low frequency component generation means, and the ratio calculation means 32 allow the low frequency component of the pixel signal of the kth spectral sensitivity characteristic and the vicinity of the interpolation target pixel position. A ratio calculating unit is configured to calculate a ratio of the pixel signal having the h-th spectral sensitivity characteristic to a low frequency component at a plurality of pixel positions in the region.

Among the above, the high-pass filters 7r, 7g, and 7b and the selection unit 23h change the change component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in the vicinity of the interpolation target pixel position ( H signal change component generation means for extracting or generating (high frequency components) is configured, and the high-pass filters 7r, 7g, 7b and the selection means 23k are used for the first pixel position in a plurality of pixel positions in the vicinity of the interpolation target pixel position. A k signal change component generating means for extracting or generating a change component (high frequency component) of the pixel signal having the spectral sensitivity characteristic of k is configured.
Further, the difference calculation means 25 and coefficient multiplication means 27 shown in FIG. 12 and the selection means 21, ratio calculation means 32, coefficient multiplication means 28, addition means 33 and multiplication means 34 shown in FIG.
The decorrelation between the kth signal and the hth signal determined based on the difference between the value obtained by the k signal change component generation means and the value obtained by the h signal change component generation means and the saturation coefficient. The sum of the value (βq (kHPF−hHPF)) and the h-th pixel signal (h) at the pixel position to be interpolated is obtained by the value obtained by the k signal low frequency component generation means and the h signal low frequency component generation means. Interpolation value calculation means for obtaining the kth pixel signal at the interpolation target pixel position by multiplying the ratio of the obtained values is configured.
The interpolation value calculation means of the example shown in the figure is a selection means 21 for selecting a pixel signal of one color (hth color) at the position of the interpolation target pixel, and a ratio calculation for obtaining a low frequency component ratio (kLPF / hLPF). And a non-correlation value βq (kHPF−hHPF) obtained as an output of the difference calculation means 25 and a pixel signal of one color (h-th color) at the position of the interpolation target pixel selected by the selection means 21. By multiplying the ratio of the low frequency components (kLPF / hLPF) obtained by the ratio calculation means 32 by the first predetermined coefficient (r) to obtain another color (first color) at the interpolation target pixel position. k color) pixel signal is obtained.
The non-correlation value βq (kHPF−hHPF) is, for example, the saturation coefficient β obtained by the saturation coefficient calculation means 9 to the difference (kHPF−hHPF) of the high frequency component obtained by the difference calculation means 25 as in the first embodiment. And the second predetermined coefficient (q).

  The interpolation value calculated by the interpolation value calculation means is stored in, for example, a two-dimensional memory (any one of 6r, 6g, 6b) for the pixel signal of the kth color, or is output from the output terminal 11.

  The interpolation calculation by the calculation means 10 is expressed by the following equation (28).

k (i, j)
= [H (i, j) + βq {kHPF (i, j) -hHPF (i, j)}]
Xr {kLPF (i, j) / hLPF (i, j)}
... (28)

  In equation (28), k (i, j) is a missing color signal at coordinates (i, j) on the image sensor 2 as in equation (1), and is a color signal to be interpolated. h (i, j) is a color signal existing in advance at the position (i, j). kHPF and hHPF are HPF values calculated by a predetermined calculation from pixels around the position of (i, j) of the k signal and h signal. kLPF and hLPF are LPF values calculated by another predetermined calculation from pixels around the (i, j) position of the k signal and h signal. q and r are predetermined constants. β is a coefficient determined based on the saturation of the signal at the position (i, j) and the surrounding pixel positions. When the saturation is high, the value of β increases, and when the saturation is low, β The value of is set to be small.

The meaning of the calculation formula shown in Formula (28) will be described with reference to FIGS. In these drawings, as in FIGS. 17 to 20, the signal levels and the positions of the pixels on the image sensor 2 are shown. For the sake of simplicity, only one line of the image sensor 2 is described, and the calculation is limited to one-dimensional direction. Described above is the arrangement of each color filter, where h is an h pixel, k is a k pixel, and parentheses () of each pixel are coordinates indicating the pixel position. Curve a is the true value of the k signal, and curve b is the true value of the h signal. On the curves a and b, the portions indicated by black circles (●) are the pixel signal values of the k signal and h signal output from the image sensor 2. Curves c and d are the LPF values of the k and h signals, respectively, and curves e and f are the HPF values of the k and h signals, respectively. With reference to these drawings, a method for performing pixel interpolation of the k signal at the pixel position (i, j) will be specifically described.
FIG. 31 shows a case where there is a positive correlation between the k signal and the h signal, FIG. 32 and FIG. 33 show a case where there is no correlation between the k signal and the h signal, and FIG. The case where there is a negative correlation between the signal and the h signal is shown.

First, a case where there is a positive correlation between the k signal and the h signal will be described with reference to FIG. The HPF values indicated by the curves e and f are depicted as overlapping because they have the same value when the changes in the k and h signals are similar.
In the bilinear method according to the prior art, the k signal at the pixel position (i−1, j) and (i + 1, j) is used, and the average value is set as the k signal at the pixel position (i, j). The signal level interpolated by the bilinear method is indicated by white triangles (Δ) in FIG. 31, but a true value to be obtained and an interpolation error are generated.

  On the other hand, in the interpolation method according to the present embodiment, the known value h (i, j) at the pixel position (i, j) is multiplied by the high-frequency component difference (kHPF−hHPF) by the saturation coefficient β and the coefficient q. A value obtained by performing post-addition and multiplying the addition result by multiplying the ratio kLPF / hLPF by the coefficient r becomes an interpolation signal k (i, j). kLPF / hLPF represents the ratio of LPF values, and this ratio represents the degree of correlation. The higher the degree of correlation, the closer to “1”.

  As described in connection with the first embodiment, there is a strong correlation between signal changes in a local region of an image. Therefore, the following equation (29) is established between the LPF value indicating a gradual change of the signal and each signal.

k (i, j): h (i, j) = kLPF (i, j): hLPF (i, j)
... (29)

  By transforming equation (29), the signal of k (i, j) at (i, j) with h pixels can be expressed by the following equation (30).

k (i, j) = h (i, j) × kLPF (i, j) / hLPF (i, j)
... (30)

  Equation (30) assumes that there is a strong correlation to signal changes in the local region of the image, and the above assumption is valid in most regions of the image. Therefore, accuracy is high in the region where the signal correlation has a high positive correlation. High pixel interpolation is possible. However, as in the first embodiment, a pixel interpolation error occurs in a region having no correlation such as an edge of an image or a region having a negative correlation.

  In FIG. 31, the ratio of the curves c and d at the pixel position (i, j) is kLPF / hLPF. The difference between the curves e and f at the pixel position (i, j) is (kHPF−hHPF). In the interpolation method according to the present embodiment, the value h (i, j) multiplied by (kHPF−hHPF) multiplied by the saturation coefficient β and the coefficient q at the pixel position (i, j) is added, and kLPF / Multiply hLPF multiplied by coefficient r. The interpolated signal k (i, j) calculated by the present method represented by Expression (28) is indicated by white circles (◯) in FIG. Pixel interpolation can be realized with high accuracy for true values. As shown in FIG. 31, when the change in signal between the k signal and the h signal is the same, (kHPF−hHPF) is close to “0”, and is proportional to the ratio between the change signals obtained from the LPF. The interpolation signal is calculated by multiplying the calculated value. By this method, pixel interpolation can be performed with high accuracy when the correlation between color signals is large.

  Next, a case where there is no correlation between color signals will be described. Since the correlation between each color is low at the edge of the image, an interpolation error occurs in the equation (30). Therefore, considering the fact that the correlation is low at the edge portion, the above problem is solved by inserting the HPF signal difference (kHPF−hHPF) into the equation (30) and using the interpolation method of the above equation (28). be able to. In Expression (28), (kHPF (i, j) −hHPF (i, j)) is a difference between signal components in the image edge portion, and when there is a strong correlation in the edge portion between changes in the k signal and the h signal. Becomes “0”, so if r = 1 in the equation (28), the equation is the same as the equation (30). When there is no correlation between the colors, (kHPF (i, j) −hHPF (i, j)) is related to a specific value of the signal of each color, and therefore high-precision pixel interpolation can be realized for each color. .

  32 and 33 show the signal levels and the positions of the pixels on the image sensor 2. FIG. In the signals shown in FIGS. 32 and 33, there is no correlation between changes in the k signal and the h signal.

First, a description will be given with reference to FIG. In the example of FIG. 32, since there is no change in the k signal, the HPF value kHPF of the k signal indicated by the curve e is “0”.
When interpolation is performed using only the color correlation change shown in Expression (30), the reference h signal changes even though the interpolation target k signal at the pixel position (i, j) has not changed. Therefore, pixel interpolation is performed to the signal level indicated by the white square mark (□), and an interpolation error occurs. However, how much the signal change between the k signal and the h signal is not correlated is obtained as a value of (kHPF−hHPF). Therefore, the value of (kHPF−hHPF) multiplied by the saturation coefficient β and the coefficient q is added to the value of h (i, j), and the addition result is multiplied by r (kLPF / hLPF). In the case of FIG. 32, kHPF is “0”, and (kHPF−hHPF) at the pixel position (i, j) is a negative value, so that a certain value is subtracted from the value of h (i, j). Become. The pixel interpolation signal level of the calculation according to the present embodiment (expression (30)) is indicated by white circles (◯). Interpolation can be performed with higher accuracy than the true value. Thus, even when there is no correlation between signals, pixel interpolation can be performed with high accuracy.

In contrast to FIG. 32, FIG. 33 shows another example in which the k signal changes and the h signal does not change and there is no correlation between the two color signals. When interpolation is performed using only the color correlation change according to Expression (28), the signal of h (i, j) is multiplied by the signal level ratio of kLPF / hLPF multiplied by the coefficient r. In this case, the signal level after pixel interpolation is the signal level indicated by white square marks (□) in FIG. Since the signal level indicated by the white square mark is a value at a position away from the true value of the k signal, it can be seen that an interpolation error has occurred.
On the other hand, in the pixel interpolation method according to the present embodiment, the h signal h (i, j) at the pixel position (i, j) is multiplied by (kLPF−hLPF) multiplied by the saturation coefficient β and the coefficient q, The addition result is multiplied by r (kLPF / hLPF). In the case of FIG. 33, hHPF is “0”, and (kHPF−hHPF) at the pixel position (i, j) is a positive value. In the interpolation method according to the present embodiment, the interpolated signal level is the position of a white circle (o), and is interpolated with high accuracy with respect to the true value of the k signal.

32 and 33 describe the case where there is no correlation between the color signals. Next, the case where there is a negative correlation between the color signals will be described. FIG. 34 shows a case where there is a negative correlation between the k signal and the h signal. Pixel interpolation by the bilinear interpolation method has a signal level indicated by a white triangle mark (Δ), and an interpolation error occurs with respect to a true value as in FIG. If the correlation is negative, pixel interpolation using only the color correlation change according to equation (30) further increases the interpolation error as indicated by the white square mark (□). In the interpolation method according to the present embodiment, pixel interpolation can be realized with high accuracy as indicated by white circles (◯).
As described above, highly accurate pixel interpolation can be performed in any case where there is a positive correlation between color signals, no correlation, or a negative correlation.

  The pixel interpolation calculation process is performed according to the procedure of the flowchart shown in FIG. 21 as in the first embodiment. When all of the six processes shown in the flowchart are finished, pixel signals of insufficient colors are prepared for all pixels at all pixel positions on one screen.

  The arithmetic processing in each procedure will be specifically described. First, the HPF and LPF values of the color signals R, G, and B are obtained by the calculations shown in the equations (4) to (15) as in the first embodiment.

  First, the calculation process of step S1 will be described. In step S1, the missing G signal at the position of the R pixel is calculated. The G signal at the position of the R pixel is calculated by the following equation (31) according to equation (28). Since the R pixel exists at a position of (i + n, j + m) (n and m are odd numbers), the coordinate value is different from the equation (28).

G (i + n, j + m) = {R (i + n, j + m) + βq (GHPF (i + n, j + m) −RHPF (i + n, j + m))} × r (GLPF (i + n, j + m) / RLPF (i + n, j + m))
... (31)

  The constants q and r may be determined in advance so that the image is optimally interpolated. For example, pixel interpolation can be performed satisfactorily even when q = 0.25 and r = 1. FIG. 22 shows the result gr of the interpolation by equation (31) at the corresponding R pixel position.

  Next, the processing proceeds to step S2. Step S2 interpolates the missing G signal at the B pixel position. The G signal at the position of the B pixel is calculated by the following equation (32) according to the equation (28). Since the B pixel exists at the position (i + s, j + t) (where s and t are even numbers), the coordinate value differs from that of the equation (28).

G (i + s, j + t) = {B (i + s, j + t) + βq (GHPF (i + s, j + t) −BHPF (i + s, j + t))} × r (GLPF (i + s, j + t) / BLPF (i + s, j + t))
... (32)

  FIG. 23 shows the result gb of the interpolation according to the equation (32) at the corresponding B pixel position. FIG. 23 also shows the result gr of the interpolation by equation (31). The G signal at all pixel positions is obtained by the interpolation of the G signal at the R pixel position according to Expression (31) and the interpolation of the G signal at the B pixel position according to Expression (32).

  Next, the processing proceeds to step S3. Step S3 interpolates the missing R signal at the G pixel position. The R signal at the position of the G pixel is calculated by the following equations (33) and (34) according to equation (28). Since the G pixel exists at the position (i + s, j + m) (s is an even number, m is an odd number) and (i + n, j + t) (n is an odd number, t is an even number), the equation (28) is coordinated accordingly. The value will be different.

R (i + s, j + m) = {G (i + s, j + m) + βq (RHPF (i + s, j + m) −GHPF (i + s, j + m))} × r (RLPF (i + s, j + m) / GLPF (i + s, j + m))
... (33)

R (i + n, j + t) = {G (i + n, j + t) + βq (RHPF (i + n, j + t) −GHPF (i + n, j + t))} × r (RLPF (i + n, j + t) / GLPF (i + n, j + t))
... (34)

FIG. 24 two-dimensionally shows the R signal rg obtained as a result of interpolation according to equations (33) and (34) at the corresponding G pixel position.
In equations (33) and (34), RLPF, RHPF, GLPF, GHPF, and β are the output values of the LPF, HPF, and saturation coefficient calculating means 9 described above, but for GLPF and GHPF, step S1 In addition, a new calculation may be performed using the interpolation values gr and gb calculated in step S2.

  Next, the processing proceeds to step S4. Step S4 interpolates the missing B signal at the position of the G pixel. The B signal at the position of the G pixel is calculated by the following equations (35) and (36) according to equation (28). Since the G pixel exists at the position (i + s, j + m) (s is an even number, m is an odd number) and (i + n, j + t) (n is an odd number, t is an even number), the equation (28) is coordinated accordingly. The value will be different.

B (i + s, j + m) = {G (i + s, j + m) + βq (BHPF (i + s, j + m) −GHPF (i + s, j + m))} × r (BLPF (i + s, j + m) / GLPF (i + s, j + m))
... (35)

B (i + n, j + t) = {G (i + n, j + t) + βq (BHPF (i + n, j + t) −GHPF (i + n, j + t))} × r (BLPF (i + n, j + t) / GLPF (i + n, j + t))
... (36)

FIG. 25 two-dimensionally shows the B signal bg obtained as a result of interpolation according to the equations (35) and (36) at the corresponding G pixel positions.
In this equation (35) and equation (36), BLPF, BHPF, GLPF, GHPF, and β are the output values of the LPF, HPF, and saturation coefficient calculating means 9 described above, but for GLPF and GHPF, the steps You may newly calculate using the interpolation values gr and gb calculated by S1 and step S2.

  Next, the processing proceeds to step S5. Step S5 interpolates the missing R signal at the B pixel position. The R signal at the position of the B pixel is calculated by the following equation (37) according to equation (28). Since the B pixel exists at the position (i + s, j + t) (where s and t are even numbers), the coordinate value differs from that of the equation (28).

R (i + s, j + t) = {G (i + s, j + t) + βq (RHPF (i + s, j + t) −GHPF (i + s, j + t))} × r (RLPF (i + s, j + t) / GLPF (i + s, j + t))
... (37)

FIG. 26 two-dimensionally shows the R signal rb obtained as a result of interpolation according to Expression (37) at the corresponding B pixel position. FIG. 26 also shows the R signal rg obtained as a result of interpolation according to the equations (33) and (34) at the corresponding G pixel positions. As a result of the interpolation by the equations (33) and (34) and the interpolation by the equation (37), the R signals of all the pixels are prepared.
In Equation (37), RLPF, RHPF, GLPF, GHPF, and β are the output values of the LPF, HPF, and saturation coefficient calculating means 9 described above, but GLPF and GHPF are calculated in steps S1 and S2. A new calculation may be performed using the interpolated values gr and gb. Also, RLPF and RHPF may be newly calculated using the interpolation value rg calculated in step S3.

  Next, the processing proceeds to step S6. Step S6 interpolates the missing B signal at the position of the R pixel. The B signal at the position of the R pixel is calculated by the following equation (38) according to equation (28). Since the R pixel exists at a position of (i + n, j + m) (n and m are odd numbers), the coordinate value is different from the equation (28).

B (i + n, j + m) = {G (i + n, j + m) + βq (BHPF (i + n, j + m) −GHPF (i + n, j + m))} × r (BLPF (i + n, j + m) / GLPF (i + n, j + m))
... (38)

FIG. 27 two-dimensionally shows the B signal br obtained as a result of interpolation according to the equation (38) at the corresponding R pixel position. FIG. 27 also shows the B signal bg obtained as a result of the interpolation according to the equations (35) and (36) at the corresponding G pixel position. As a result of the interpolation by the equations (35) and (36) and the interpolation by the equation (38), the B signals of all the pixels are prepared.
In Expression (38), BLPF, BHPF, GLPF, GHPF, and β are the output values of the LPF, HPF, and saturation coefficient calculating means 9 described above, but GLPF and GHPF are calculated in steps S1 and S2. A new calculation may be performed using the interpolated values gr and gb. Also, BLPF and BHPF may be newly calculated using the interpolation value bg calculated in step S4.

  As described above, the color signal missing in each pixel is interpolated by the calculation from step S1 to step S6, and R, G, B signals of all the pixels are obtained.

  The high-pass filters 7r, 7g, and 7b, the low-pass filters 8r, 8g, and 8b, the saturation coefficient calculation unit 9, the decorrelation value calculation unit 12, and the calculation unit 10 described in the above example are performed by software, that is, by a programmed computer. It can also be realized. In that case, the process of step S1-S6 is performed as follows.

FIG. 35 is a flowchart showing the procedure of step S1. In step S1, since the missing G signal at the position of the R pixel is calculated, a pixel in which the R signal exists in advance is selected as the target pixel (step S40).
A G-color LPF value and an R-color LPF value at the selected target pixel position are calculated (steps S41 and S42), and a value obtained by multiplying the ratio by a predetermined constant r is used as the low-frequency component ratio (step S43). .
Next, the G-color HPF value and the R-color HPF value at the selected target pixel position are calculated (steps S44 and S45). Further, the saturation coefficient β is calculated using the signals of the pixel of interest and the surrounding pixels (step S46), and the saturation coefficient β and a predetermined constant q are multiplied by the difference between the HPF values of the R color and the G color to calculate A correlation value is calculated (step S47).
Then, a non-correlation value is added to the R color signal at the target pixel position, and the G color signal is calculated by multiplying the low frequency component ratio (step S48). The obtained G color signal value is output to the two-dimensional memory 6g (step S49). The above processing is repeatedly executed for all pixels in which the R signal is present in advance (step S50).
The processing of steps S2 to S6 is the same as that of step S1. That is, each procedure of steps S2 to S6 is shown as replacing “R” and “G” in the flowchart of FIG. 35 with the color (h) of the pixel of interest and the color (k) to be obtained by interpolation. .

Correspondences between the steps shown in FIG. 35 and the members shown in FIGS. 1, 12, 16, and 30 are as follows. That is, the processing in step S40 corresponds to the processing in the selection means 21 and the control means (the control means 30a and 30b in FIGS. 12 and 30 form part of them, but the whole is not shown), and step S41. The processing in step S42 corresponds to the processing in the low-pass filter 8r and the selection unit 24h, and the processing in step S43 includes the ratio calculation unit 32 and the coefficient multiplication unit. 28 corresponds to the processing in the high-pass filter 7g and the selection means 23k, the processing in step S45 corresponds to the processing in the high-pass filter 7r and the selection means 23h, and the processing in step S46 is Corresponding to the processing in the saturation coefficient calculation means 9, the processing in step S47 is Corresponding to the processing in the difference calculating means 25 and the coefficient multiplying means 27, the processing in step S48 corresponds to the processing in the adding means 33 and the multiplying means 34, and the processing in step S49 is performed from the computing means 10 to the two-dimensional memories 6r, 6g. , 6b corresponds to data transfer and writing.
Control of each means in step S50 and each step is performed by a control means (the control means 30a and 30b in FIGS. 12 and 30 form a part thereof, but the whole is not shown).

  The process in step S46 of FIG. 35 can be performed as shown in FIG. 29 as in step S16 of FIG.

  In Embodiments 1 and 2, the order of generating the signals of each color is not limited to the order shown in FIG. 21, and the order of generating the signals may be changed. For example, step S1 and step S2, step S3 and step S4, step S5 and step S6 can be changed in order of calculation.

  In Embodiments 1 and 2, the arithmetic expression for performing two-dimensional filtering on the LPF and HPF is described. However, the correlation of the output signal around the interpolation target pixel is determined, and the correlation is determined to be strong. Alternatively, the output values of the HPF and LPF may be used by using only the output signals of the pixels arranged in the same direction.

  When interpolation is performed by the method described in the first and second embodiments, when there is a correlation between each signal (R, G, B), pixel interpolation using the correlation can be performed, such as an edge portion of an image. Even in a portion where there is no correlation, pixel interpolation with high accuracy can be performed, and blackening and white spots in the vicinity of the color boundary, which the method disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2001-197512) has, are included. The problem that the image degradation occurs is remarkably improved.

It is a block diagram which shows the structure of the imaging device provided with the pixel signal processing apparatus of Embodiment 1 of this invention. It is explanatory drawing which shows the color filter of three primary colors of R, G, and B arrange | positioned at a Bayer type. It is a figure which shows arrangement | positioning of R pixel on the imaging surface of an image sensor. It is a figure which shows arrangement | positioning of G pixel on the imaging surface of an image sensor. It is a figure which shows arrangement | positioning of B pixel on the imaging surface of an image sensor. It is a figure which shows the LPF value of R signal. It is a figure which shows the LPF value of G signal. It is a figure which shows the LPF value of B signal. It is a figure which shows the HPF value of R signal. It is a figure which shows the HPF value of G signal. It is a figure which shows the HPF value of B signal. 3 is a block diagram illustrating a configuration example of a decorrelation value calculation unit according to Embodiment 1. FIG. FIG. 3 is a block diagram illustrating a configuration example of a calculation unit according to the first embodiment. It is a figure which shows an example of the relationship between saturation c and saturation coefficient (beta). It is a figure which shows an example of the relationship between saturation c and saturation coefficient (beta). 3 is a block diagram illustrating a configuration example of a saturation coefficient calculation unit according to Embodiment 1. FIG. FIG. 3 is an explanatory diagram schematically illustrating the principle of pixel interpolation according to the first embodiment when there is a positive correlation between a k signal and an h signal. FIG. 3 is an explanatory diagram schematically illustrating the principle of pixel interpolation according to the first embodiment when there is no correlation between a k signal and an h signal. FIG. 3 is an explanatory diagram schematically illustrating the principle of pixel interpolation according to the first embodiment when there is no correlation between a k signal and an h signal. FIG. 4 is an explanatory diagram schematically illustrating the principle of pixel interpolation according to the first embodiment when there is a negative correlation between a k signal and an h signal. It is a flowchart which shows the interpolation procedure in Embodiment 1, 2 of this invention. It is a figure which shows the arrangement | sequence of G signal which carried out the pixel interpolation in R pixel position. It is a figure which shows the arrangement | sequence of G signal which interpolated the pixel in a B pixel position. It is a figure which shows the arrangement | sequence of the R signal which carried out the pixel interpolation in the G pixel position. It is a figure which shows the arrangement | sequence of the B signal which carried out pixel interpolation in the G pixel position. It is a figure which shows the arrangement | sequence of R signal which interpolated the pixel in a B pixel position. It is a figure which shows the arrangement | sequence of the B signal which carried out the pixel interpolation in the R pixel position. In Embodiment 1 of this invention, it is a flowchart which shows the procedure which interpolates G signal of R pixel position. 5 is a flowchart showing a procedure for calculating a saturation coefficient β in the first and second embodiments of the present invention. 6 is a block diagram illustrating a configuration example of a calculation unit according to Embodiment 2. FIG. It is explanatory drawing which shows typically the principle of the pixel interpolation of Embodiment 2 when there exists a positive correlation between k signal and h signal. It is explanatory drawing which shows typically the principle of the pixel interpolation of Embodiment 2 when there is no correlation between k signal and h signal. It is explanatory drawing which shows typically the principle of the pixel interpolation of Embodiment 2 when there is no correlation between k signal and h signal. It is explanatory drawing which shows typically the principle of the pixel interpolation of Embodiment 2 when there exists a negative correlation between k signal and h signal. In Embodiment 2 of this invention, it is a flowchart which shows the procedure which interpolates G signal of R pixel position.

Explanation of symbols

1 lens, 2 image sensor, 3 A / D converter, 4 frame memory, 5 demultiplexer, 2D memory for 6r R signal, 2D memory for 6g G signal, 2D memory for 6b B signal, 7r for R signal HPF, 8r R signal LPF, 7g G signal HPF, 8g G signal LPF, 7b B signal HPF, 8b B signal LPF, 9 saturation coefficient calculation means, 10 calculation means, 11 output terminal, 12 uncorrelated Value calculation means 21, 23h, 23k, 24h, 24k selection means, 25, 26 difference calculation means, 27, 28 coefficient multiplication means, 29 addition means, 30 control means, 32 ratio calculation means, 33 addition means, 34 multiplication means .

Claims (14)

Based on a set of pixel signals arranged on a two-dimensional plane and each having any one of the first to Nth spectral sensitivity characteristics, the hth (h is any one of 1 to N) In the pixel signal processing device that generates the pixel signal of the kth (k is any one of 1 to N except h) spectral signal at the interpolation target pixel position where the pixel signal of the spectral sensitivity characteristic exists. ,
H signal low frequency component generating means for generating a low frequency component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
H signal change component generation means for generating a change component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
K-signal low-frequency component generating means for generating low-frequency components of the pixel signal of the k-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
K signal change component generation means for generating a change component of the pixel signal of the kth spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A saturation coefficient calculation means for calculating a saturation coefficient based on the saturation of the pixel signal at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
The k-th signal and the h-th signal determined based on the difference between the value obtained by the k signal change component generation means and the value obtained by the h signal change component generation means and the saturation coefficient; , The difference between the value obtained by the k signal low frequency component generation means and the value obtained by the h signal low frequency component generation means, the h th pixel signal at the interpolation target pixel position, And interpolated value calculation means for obtaining the k-th pixel signal at the interpolation target pixel position. Based on a set of pixel signals arranged on a two-dimensional plane and each having any one of the first to Nth spectral sensitivity characteristics, the hth (h is any one of 1 to N) In the pixel signal processing device that generates the pixel signal of the kth (k is any one of 1 to N except h) spectral signal at the interpolation target pixel position where the pixel signal of the spectral sensitivity characteristic exists. ,
H signal low frequency component generating means for generating a low frequency component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
H signal change component generation means for generating a change component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
K-signal low-frequency component generating means for generating low-frequency components of the pixel signal of the k-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
K signal change component generation means for generating a change component of the pixel signal of the kth spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A saturation coefficient calculation means for calculating a saturation coefficient based on the saturation of the pixel signal at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
The kth signal and the hth signal determined based on the difference between the value obtained by the k signal change component generation means and the value obtained by the h signal change component generation means and the saturation coefficient. Obtained by the k signal low frequency component generation means and the h signal low frequency component generation means to the sum of the non-correlation value and the h th pixel signal at the interpolation target pixel position. An interpolation value calculating means for multiplying the ratio of the values to obtain the k-th pixel signal at the interpolation target pixel position.   The interpolation value calculation means is a value obtained by multiplying the difference between the value obtained by the k signal change component generation means and the value obtained by the h signal change component generation means by the saturation coefficient and a predetermined coefficient. The pixel signal processing apparatus according to claim 1, wherein the pixel signal processing apparatus has an uncorrelated value. The saturation coefficient calculation means is
4. The saturation coefficient is increased when the saturation of pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position is higher than when the saturation is low. The pixel signal processing device according to any one of the above. The saturation coefficient calculation means is
An average value of signals of pixels having respective spectral sensitivity characteristics is calculated for each of the first to Nth spectral sensitivity characteristics from pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position, and different spectral values are obtained. The pixel signal processing apparatus according to any one of claims 1 to 4, wherein the saturation is calculated from a magnitude of a difference between average values of pixel signals of sensitivity characteristics. The saturation coefficient calculation means is
From pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position, an average value of signals of all pixels, and signals of pixels having respective spectral sensitivity characteristics for each of the first to Nth spectral sensitivity characteristics Each of the average values is calculated, and the saturation is calculated from the magnitude of the difference between the average value of all the pixel signals and the average value of the signals of each pixel of the first to Nth spectral sensitivity characteristics. The pixel signal processing device according to claim 1.   7. The pixel signal processing apparatus according to claim 1, wherein the pixels having the first to N-th spectral sensitivity characteristics are three types of R, G, and B pixels. Based on a set of pixel signals arranged on a two-dimensional plane and each having any one of the first to Nth spectral sensitivity characteristics, the hth (h is any one of 1 to N) In the pixel signal processing method for generating a pixel signal of the kth spectral sensitivity characteristic (k is any one of 1 to N excluding h) at the pixel position to be interpolated where the pixel signal of the spectral sensitivity characteristic of ,
An h signal low frequency component generating step for generating a low frequency component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
An h signal change component generation step for generating a change component of a pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A k-signal low-frequency component generation step for generating a low-frequency component of the pixel signal of the k-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A k signal change component generation step of generating a change component of a pixel signal of the kth spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A saturation coefficient calculation step for calculating a saturation coefficient based on the saturation of pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
The k-th signal and the h-th signal determined based on the difference between the value obtained by the k signal change component generation step and the value obtained by the h signal change component generation step and the saturation coefficient; , A difference between the value obtained by the k signal low frequency component generation step and the value obtained by the h signal low frequency component generation step, the h th pixel signal at the interpolation target pixel position, And an interpolation value calculating step for obtaining the k-th pixel signal at the interpolation target pixel position. Based on a set of pixel signals arranged on a two-dimensional plane and each having any one of the first to Nth spectral sensitivity characteristics, the hth (h is any one of 1 to N) In the pixel signal processing method for generating a pixel signal of the kth spectral sensitivity characteristic (k is any one of 1 to N excluding h) at the pixel position to be interpolated where the pixel signal of the spectral sensitivity characteristic of ,
An h signal low frequency component generating step for generating a low frequency component of the pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
An h signal change component generation step for generating a change component of a pixel signal of the h-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A k-signal low-frequency component generation step for generating a low-frequency component of the pixel signal of the k-th spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A k signal change component generation step of generating a change component of a pixel signal of the kth spectral sensitivity characteristic at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
A saturation coefficient calculation step for calculating a saturation coefficient based on the saturation of pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position;
The kth signal and the hth signal determined based on the difference between the value obtained by the k signal change component generation step and the value obtained by the h signal change component generation step, and the saturation coefficient. Obtained by the k signal low frequency component generation step and the h signal low frequency component generation step to the sum of the non-correlation value and the h th pixel signal at the interpolation target pixel position. An interpolation value calculating step of multiplying a ratio of the values to obtain the k-th pixel signal at the interpolation target pixel position.   In the interpolation value calculating step, the difference between the value obtained in the k signal change component generation step and the value obtained in the h signal change component generation step is multiplied by the saturation coefficient and a predetermined coefficient. The pixel signal processing method according to claim 8 or 9, wherein a non-correlation value is used. The saturation coefficient calculation step includes
11. The saturation coefficient is increased when the saturation of pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position is higher than when the saturation is low. The pixel signal processing method according to any one of the above. The saturation coefficient calculation step includes
An average value of signals of pixels having respective spectral sensitivity characteristics is calculated for each of the first to Nth spectral sensitivity characteristics from pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position, and different spectral values are obtained. The pixel signal processing method according to any one of claims 8 to 11, wherein the saturation is calculated from a magnitude of a difference between average values of pixel signals of sensitivity characteristics. The saturation coefficient calculation step includes
From pixel signals at a plurality of pixel positions in a region in the vicinity of the interpolation target pixel position, an average value of signals of all pixels, and signals of pixels having respective spectral sensitivity characteristics for each of the first to Nth spectral sensitivity characteristics Each of the average values is calculated, and the saturation is calculated from the magnitude of the difference between the average value of all the pixel signals and the average value of the signals of each pixel of the first to Nth spectral sensitivity characteristics. The pixel signal processing method according to any one of claims 8 to 11.   14. The pixel signal processing method according to claim 8, wherein the pixels having the first to Nth spectral sensitivity characteristics are R, G, and B pixels.

Images