JP5195957B2 - Color filter and image sensor - Google Patents

Description

  The present invention relates to a color filter, an image sensor, and an image processing apparatus used in an image pickup apparatus using a solid-state image pickup device such as a digital camera.

There is a single-plate color image pickup device having an image processing device in which a color filter is attached to each pixel of a single-plate solid-state image pickup device and all colors are aligned at all pixel positions using a mosaic image of a color photographed by the image pickup device.
In such an imaging apparatus using a single-plate solid-state imaging device, only a single spectral sensitivity can be obtained even if an image is taken as it is. For this reason, a method is generally employed in which different color filters are attached to a large number of pixels and arranged in a specific pattern to perform color photographing. The captured image can obtain only one color for each pixel, and an image having a mosaic shape with respect to the color is generated. However, by performing interpolation using color information obtained from surrounding pixels, it is possible to generate an image in which all the colors are aligned in all the pixels. Such interpolation processing is called color separation processing or demosaic processing.

  In a single-plate color imaging apparatus, a Bayer array shown in US Pat. No. 3,971,065, “Bryce E. Bayer.“ COLOR IMAGING ARRAY ”” is known as a commonly used color filter array. As shown in FIG. 1, this arrangement is characterized in that the G filters are arranged in a checkered pattern and are present at double the density of the R and B filters. Therefore, for G having a large amount of information and a strong correlation with luminance, a pixel interpolation method is often used in which colors are aligned for all pixels first, and R and B colors remaining as a reference are aligned.

For example, in the method shown in US Patent No. 4,716,455, “Ozawa, Akiyama, Satoh, Nagahara and Mimura,“ Chrominance Signal Interpolation Device for a Color Camera ”,” the ratio of the low frequency components of each color in the local region is almost constant. Based on the assumption that there is a pixel, all pixels are pre-arranged with G, and the average of the ratio of R and G in the neighboring pixels and the average of the ratio of B and G are multiplied by G at the target pixel position to estimate the unknown color component. .
Further, in such an interpolation method, it is known that the resolution in the horizontal and vertical directions can be increased by utilizing the fact that the G filters are arranged in a checkered pattern.
For example, in Japanese Patent No. 2931520 “single-plate color video camera color separation circuit”, horizontal and vertical correlation values of interpolated pixel positions are obtained, processing means suitable for cases where the horizontal correlation is strong, A process has been proposed in which two interpolated values obtained by processing means suitable for a case where the correlation is strong are mixed with the correlation value.

With these technologies, it is possible to align all RGB colors at all pixel positions with high resolution using a single-plate color imaging device.
However, since each color is sampled discretely, if the captured image contains high-frequency components that exceed the Nyquist frequency, aliasing (folding of the high-frequency components back into the low-frequency region) occurs, and a different color is estimated. There was a problem that.

This phenomenon called false color is very conspicuous because a large aliasing occurs in a specific spatial frequency band when a regularly arranged color filter array is used. In addition, the false color once generated cannot be removed using a frequency filter because it cannot be distinguished whether it is a return of a high-frequency component or a low-frequency component originally present.
Therefore, in the conventional single-plate color imaging device, it is necessary to place an optical low-pass filter in front of the imaging element in order to reduce false colors, and to remove high-frequency components of the captured image in advance.
However, since the optical low-pass filter does not have a steep cutoff with respect to the Nyquist frequency, there is a problem that if a false color is completely suppressed, a frequency band lower than the Nyquist frequency is also cut.
In the Bayer array, the Nyquist frequency of R and B is lower than that of G, and using an optical low-pass filter that matches the Nyquist frequency of R and B causes degradation of the resolution of G.
In practice, since it is necessary to suppress degradation of resolution, it is difficult to completely suppress false colors. Furthermore, the insertion of an optical low-pass filter has hindered downsizing and cost reduction of the imaging device.

In addition, it is known that color reproduction and dynamic range can be improved by using a filter array in which the number of color filters is increased to four or more compared to the Bayer array consisting of three colors of R, G, and B. Yes.
As shown in FIG. 2 (a), a larger number of filters that pass only a narrow range of light wavelength bands use fewer filters that pass a wider range of light wavelength bands as shown in FIG. 2 (c). Compared to, it is suitable for color reproduction.
As shown in FIG. 2 (b), the use of several types of filters that pass through the same wavelength band with different transmittances expands the dynamic range rather than using the filter shown in FIG. 2 (c). Can do.
Nevertheless, the reason why the Bayer array is widely used is that the smaller the number of pixels for one color, the lower the resolution for that color and the easier it is to generate false colors.

In order to solve the above problems, inventions have been made to reduce the regularity of the color filter array and reduce the occurrence of false colors.
To be precise, inventions have been made for problems in which false colors are biased to a specific spatial frequency band to make them visually noticeable and difficult to remove.
Similarly, the reduction of false colors described in the present invention means that the generation of false colors biased to a specific spatial frequency band is suppressed.

In the pseudo-random Bayer array proposed by FillFactory (acquired by Cypress in 2004), G is arranged in a checkered pattern, and R and B are arranged pseudo-randomly in the remaining pixel positions. A three-color G checkered pseudorandom array is shown.
Japanese Patent Application Laid-Open No. 2000-316169 discloses a six-color random arrangement under the condition that the pixel is adjacent to a six-color filter at four sides or four corners of the target pixel position.
5 colors in EP Publication No. 0,804,037, (A2) “Mutze, Ulrich, Dr.“ Process and system for generating a full color image or multispectral image from the image data of a CCD image sensor with a mosaic color filter. ”” Shows an array with a 3x3 repeating pattern and a pseudo-random pattern.
What is common to these arrangements is that the arrangement includes randomness, and the latter two are characterized by the fact that the number of filters exceeding three colors is used.
The randomness of the arrangement provides an effect that the false color is dispersed in various spatial frequency bands and becomes inconspicuous, and an improvement in dynamic range and color reproduction can be expected by increasing the number of filters.

However, in the pseudo-random Bayer array of FillFactory, the R and B arrays are pseudo-random, but the repetition period is short, so the false color occurrence position in the spatial frequency domain is only slightly different from the Bayer array, which is substantially Since the effect of reducing false colors is not seen and the arrangement is three colors, the color reproduction and dynamic range are equivalent to the Bayer arrangement.
The latter two are more random than the former, so the effect of reducing false colors can be expected. However, since all colors are arranged randomly without distinction, they are disadvantageous in terms of resolution compared to Bayer and pseudo-random Bayer arrays. It becomes.
In general, in color separation processing, the color interpolated at a high resolution is used as a reference,
Other color interpolation methods are used. Therefore, when G is present in Ichimatsu and compared with the Bayer array or pseudo-random Bayer array that can be used for the correlation processing shown in Patent No. 2931520, Japanese Patent Laid-Open No. 2000-316169 and EP Publication Nos. 0,804,037, (A2 ) Has a problem that it is very difficult to generate a reference color, and the frequency band that can be reproduced for each pixel position is greatly different.
In addition, since the latter two increase the number of colors compared to the Bayer array, it is clear that the number of pixels per color decreases and the resolution further decreases.

In general, pixel thinning and addition are performed to read out signals from the solid-state imaging device at high speed. However, since thinning and addition are performed in a periodic pattern, if a random filter array is used, There arises a problem that the filter arrangement of this is changed from the original, or pixels of different colors are randomly added.
As described above, adding randomness to the filter array and increasing the number of colors can be achieved by reducing the resolution, reducing the resolution, reducing the number of colors, and increasing the number of colors. Cause problems.

JP 2005-160044 A US Patent No. 3,971,065 US Patent No. 4,716,455 EP Publication No. 0,804,037, (A2) JP 2000-316169

  There have been inventions to suppress the occurrence of false colors by randomly arranging color filters used in single-chip color image sensors, and to improve color reproduction and dynamic range by increasing the number of filter colors. There is a problem that the resolution is lower than that of the conventional arrangement represented by No. 1 and there is a problem in thinning / addition readout.

  The present invention solves the above-described problem, and even when a filter having a larger number of colors is used than in the Bayer array, a decrease in resolution and generation of false colors can be sufficiently suppressed, and a color filter that can cope with thinning / addition readout. An object of the present invention is to provide an imaging device and an image processing apparatus.

  In order to solve the above-described problems of the prior art and achieve the above-described object, the color filter according to the first invention has any one of a plurality of types of filters having different spectral characteristics assigned to each pixel position. A color filter, wherein the plurality of types of filters have four colors, the first color is arranged in a checkered pattern, and the second color is horizontally and vertically at a pixel position where the first color does not exist. Arranged every other line, the third color and the fourth color are randomly arranged at pixel positions where the first color and the second color do not exist.

  The color filter of the second invention is a color filter in which any one of a plurality of types of filters having different spectral characteristics is assigned to each pixel position, and the plurality of types of filters have five colors. The first color is arranged in a checkered pattern, and the second color and the third color are randomly arranged at pixel positions every other line horizontally and vertically at the pixel position where the second color and the third color do not exist, and the fourth color And the fifth color is randomly arranged at pixel positions where the first color, the second color, and the third color do not exist.

  The color filter of the third invention is a color filter in which any one of five or more types of filters having different spectral characteristics is assigned to each pixel position, and the first color among the colors of the plurality of filters is It is arranged every other line in the horizontal and vertical directions, the second color is arranged every other line in the horizontal and vertical directions, and is arranged on a line different from the first color, and part or all of the remaining colors are The first color and the second color are randomly arranged at pixel positions that do not exist.

  The color filter of the fourth invention is a color filter in which any one of a plurality of types of filters having different spectral characteristics is assigned to each pixel position, and the plurality of filters have six colors, The first color is arranged every other line horizontally and vertically, the second color is arranged every other line horizontally and vertically, and arranged on a line different from the first, and the third color and the fourth color are arranged. Are randomly arranged at pixel positions every other line horizontally and vertically at the pixel positions where the first color and the second color do not exist, and the fifth color and the sixth color are the first color and the second color, respectively. , The second color, the third color, and the fourth color are randomly arranged in pixel positions.

  An image pickup device of a fifth invention is an image pickup device having a color filter in which any one of a plurality of types of filters having different spectral characteristics is assigned to each pixel position, and the color of the plurality of types of filters is 4 The first color is arranged in a checkered pattern, the second color is arranged every other line horizontally and vertically at the pixel position where the first color does not exist, the third color and the fourth color Colors are randomly arranged at pixel positions where the first color and the second color do not exist.

  In the image processing apparatus of the ninth invention, any one of a plurality of types of filters having different spectral characteristics is assigned to each pixel position, and one type of the predetermined filter among the plurality of types of filters is a checkered pixel. Input means for inputting image data from an image sensor having a color filter arranged at a position and a part or all of the remaining filters are randomly arranged at pixel positions where the predetermined filter is not located; and the input means A first image is generated by interpolating the pixel value of the first color at the pixel position where the predetermined color does not exist in the image data input by using the first color existing around the pixel value. Pixel values of the second color other than the first color are interpolated using the first color and the second color existing in the local region including the target pixel position, and second Images of And a second interpolation means for forming.

  In the color filter array shown in FIG. 3, the color C1 is arranged in a checkered pattern, and the colors C2, C3, C4, and C5 are randomly arranged at positions where C1 does not exist, and have a predetermined size. This area is a rectangle of 15 × 15 pixels, the existence frequency of C1 is 1/2 as a ratio, the existence frequencies of colors C2, C3, C4, and C5 are each 1/8, and the error of the existence frequency is ± 1/50.

In the present embodiment, in the color filter array, as a first modification, the colors included in the filter array are four colors, the first color C1 is arranged in a checkered pattern, and the second color C2 is It is arranged every other line horizontally and vertically at the pixel position where the color C1 does not exist, and the third color C3 and the fourth color C4 are randomly arranged at the pixel position where the color C1 and the color C2 do not exist. Also good.
In the filter arrangement determined in this way, C1 and C2 are regularly arranged, and C3 and C4 are randomly arranged.
The color C1 has about 1/2 the number of pixels of the entire image sensor, and the color C2 has about 1/4 the number of pixels.
Colors C3 and C4 have only about 1/4 pixel count for the two colors, so if there are almost the same number of colors C3 and C4, for example, colors C3 and C4 each have only about 1/8 pixel count. Although the resolution is low and false colors are likely to occur, the occurrence of false colors can be reduced by arranging them randomly.

In addition, the color filter array of the first modification of the present embodiment is, for example, a filter array having a high correlation between the spectral sensitivities of the color C3 and the color C4.
In the color filter array defined in this way, for example, if color C1 is G, color C2 is R, color C3 is B, and color C4 is B 'which is close to B in spectral sensitivity, B is arranged in the Bayer array. It is considered that the colors B and B ′ are randomly arranged at the positions where they are placed.

Hereinafter, this arrangement is referred to as a four-color G checkered random arrangement. A 4-color G checkered random array is shown in FIG.
Most of the techniques currently used for thinning out and addition readout methods for solid-state imaging devices are based on the Bayer array, and the array obtained as a result of the readout is still a Bayer array. .

FIG. 5 shows an example in which signals are read out from the solid-state imaging device at high speed by combining thinning readout and addition readout.
In this example, four out of eight vertical lines are thinned out, and at the same time, the same colors are added by skipping one line in the horizontal direction, and the same colors are added by skipping three lines in the vertical direction.

When these conventional readout methods are applied to a four-color G checkered random array, it is clear that the thinning-out readout results in a four-color G checkered random array even after readout.
Even after the thinning-out reading is performed, it is more preferable to determine the arrangement in advance so as to satisfy the above-described condition regarding the existence frequency.

In addition reading, the colors G and R have the same conditions as the Bayer array, but the colors B and B ′ are added and output without distinction.
For example, when adding 4 pixels, (1) 4 pixels of color B and 0 pixels of color B ′ are added, (2) 3 pixels of color B, and color B ′ When one pixel is added, (3) When two pixels of color B and two pixels of color B ′ are added, (4) One pixel of color B and one pixel of color B ′ When three pixels are added, (5) there are five ways of adding 0 pixels of color B and 4 pixels of B ′.
That is, even in a single color region, the pixel value obtained at the position B in the Bayer array differs depending on the difference in the number of colors B and B ′ to be added.
However, due to the calculation characteristics in the color separation processing, when a color obtained by blending the pixel values of the colors B and B ′ in proportion to the frequency of the color described in the color filter array of the present embodiment is a new color B ″. The array obtained by addition readout can be regarded as an RGB ″ Bayer array.
In the color separation processing of the Bayer array, it is known that the colors R and B can be estimated by including the high frequency components with reference to the color G, as long as the low frequency components are known.
For example, if the pixel value of color G at a certain pixel position p is G (p), the low frequency components of G and B are Glow (p), and Blow (p), the pixel value B (p) of B is given by the following formula (1 ) Can be used for estimation.

The low frequency component is calculated as an average of the colors G and B existing in the vicinity of the pixel position p. In most parts of the RGB image, there is a strong positive correlation between the color R, G and B pixel values.
Therefore, as shown in FIG. 6, when the pixels existing in the local area are plotted against the space where the horizontal axis is the pixel value of color G and the vertical axis is the pixel value of color B, the pixel is within a limited narrow area. Pixels are distributed.
This distribution can be approximated using a linear regression line passing through the origin and the center of the distribution, that is, a straight line represented by the equation (1).
Here, it is considered that the same calculation as the calculation of the low frequency component performed for the color B of the Bayer array is performed on the array obtained by the addition reading.
This calculation is obtained by averaging the results of adding colors B and B ′ within a certain local range.

When considered as a calculation for the array before addition, it is merely an average of the colors B and B ′ existing in the vicinity of the pixel position p.
Since the spectral sensitivities of colors B and B 'are close, the same incident light pattern can be expected to show almost the same frequency characteristics, and if the local area is sufficiently large, colors B and B' are added. It can be expected that the ratio of the number of pixels to be approximated to the color existence frequency described above.
With this feature, the low-frequency component of the color B ″ in the Bayer array of the color RGB ″ can be approximated using the array obtained by addition reading.
If the low-frequency component of color B '' is known, the pixel value of B '' can be estimated using G as the reference as shown in Equation (1), so the array obtained by additive readout is the Bayer array of RGB '' Can be used instead of
When thinning readout and addition readout are combined, they can be used instead of the RGB ″ Bayer array for the same reason.

As a second modification, the color filter array of the present embodiment includes five colors included in the filter array, the first color C1 is arranged in a checkered pattern, and the second color C2 and the third color C3 is randomly placed at every other pixel position horizontally and vertically at the pixel position where C1 does not exist, and the fourth color C4 and the fifth color C5 are randomly placed at pixel positions where C1, C2, and C3 do not exist Placed in.
In the filter array thus determined, the color reproduction and dynamic range can be further improved by increasing the number of colors of the filter array.
In the second modification, the color filter array may have a high correlation between the spectral sensitivities of C2 and C3 and a high correlation between the spectral sensitivities of C4 and C5.
In the color filter array determined in this way, for example, if C1 is G, C2 is R, C3 is R 'close to R, C4 is B, and C5 is B' close to B, In the Bayer array, R and R ′ are randomly arranged at the position where R is arranged, and B and B ′ are randomly arranged at the position where B is arranged.

Hereinafter, this arrangement is referred to as a five-color G checkered random arrangement.
A five-color G checkered random array is shown in FIG.
Even when addition reading or thinning-out reading is performed for a 5-color G checkered random array, a new color that blends R and R 'pixel values in proportion to the color existence frequency described in claim 1-1 is newly added. Color R '' and a color blended with the pixel values of B and B 'in proportion to the frequency of the color described in the above color filter array is a new color B''. For the same reason, the array after reading can be handled as a Bayer array of R ″ GB ″.

Hereinafter, an example of an imaging device (digital video camera) in which the present invention is mounted will be described.
FIG. 8 is a block diagram showing the configuration of the digital video camera system 100 according to the present embodiment.
As shown in FIG. 8, a digital video camera system 100 includes a lens 101, an aperture 102, a CCD image sensor 103, a correlated double sampling circuit 104, an A / D converter 105, a DSP block 106, a timing generator 107, and a D / A converter 108. , A video encoder 109, a video monitor 110, a CODEC 111, a memory 112, a CPU 113 and an input device 114.
Here, the input device 114 refers to operation buttons such as a recording button on the camera body. The DSP block 106 is a block having a signal processing processor and an image RAM, and the signal processing processor can perform pre-programmed image processing on the image data stored in the image RAM. ing. Hereinafter, the DSP block is simply referred to as DSP.

  Incident light that passes through the optical system and reaches the CCD 103 first reaches each light receiving element on the CCD imaging surface, is converted into an electrical signal by photoelectric conversion at the light receiving element, and is denoised by the correlated double sampling circuit 104. After being digitized by the A / D converter 105, it is temporarily stored in the image memory in the DSP 106.

In the state during imaging, the timing generator 107 controls the signal processing system so as to maintain image capture at a constant frame rate. A stream of pixels is also sent to the DSP 106 at a constant rate, and after appropriate image processing is performed there, the image data is sent to the D / A converter 108 and / or the CODEC 111.
The D / A converter 108 converts the image data sent from the DSP 106 into an analog signal, the video encoder 109 converts it into a video signal, and the video signal can be monitored on the video monitor 110. Reference numeral 110 serves as a camera finder in this embodiment.
The CODEC 111 encodes image data sent from the DSP 106, and the encoded image data is recorded in the memory 112. Here, the memory 112 may be a recording device using a semiconductor, a magnetic recording medium, a magneto-optical recording medium, an optical recording medium, or the like.
The above is the description of the entire system of the digital video camera of the present embodiment. In the present embodiment, the present invention is realized by image processing in the DSP 106. The image processing portion will be described in detail below.

As described above, the image processing unit of the present embodiment is actually realized by the DSP 106. Therefore, in the configuration of the present embodiment, the operation of the image processing unit is realized in the DSP 106 such that the arithmetic unit sequentially executes the operations described in the predetermined program code with respect to the stream of the input image signal. Has been.
In the following description, each processing unit in the program will be described as a functional block, and the order in which each processing is executed will be described with a flowchart. However, the present invention may be configured by mounting a hardware circuit that realizes processing equivalent to the functional blocks described below, in addition to the program form described in the present embodiment.
Assume that the on-chip color filter of the CCD 103 uses the filter array of the present invention. An image in the temporary storage state has only one color for each pixel. The DSP 106 processes this image using an image processing program incorporated in advance, and generates image data having all colors in all pixels.

In this embodiment, a case will be described in which image processing is performed on a mosaic image obtained from the five-color G checkered random array shown in FIG.
First, G pixel values at all pixel positions are calculated using G pixel values obtained in a checkered pattern.
Subsequently, G is used as a reference, and R, R ′, B, and B ′ are interpolated at all pixel positions.
In this way, MS-SyncNR is applied to an image in which all colors are aligned at all pixel positions to remove false colors.

  FIG. 9 is a block diagram of this embodiment. RR'GBB 'mosaic image obtained from 5 colors G checkered random array is input, and G pixel interpolation unit 201, R interpolation unit 202, R' interpolation unit 203, B interpolation unit 204, B 'interpolation unit 205 all pixel positions Thus, an RR′GBB ′ interpolated image 1 206 which is an image in which all colors are interpolated is obtained. Further, the RMS-SyncNR unit 207, the R'MS-SyncNR unit 208, the BMS-SyncNR unit 209, and the B'MS-SyncNR unit 210 are applied to the RR'GBB 'interpolated image 206, and the final output RR A 'GBB' interpolated image 211 is obtained.

[G interpolation section]
The G interpolation unit 201 interpolates G pixel values at all pixel positions.
Since there are more pixels than other colors, an interpolation result having a high resolution can be obtained even by using a simple interpolation method such as Bicubic interpolation.
Here, as an example of an interpolation method that takes advantage of the feature that G is arranged in a checkered pattern, the interpolation method described in Patent No. 2931520 “Color Separation Circuit of Single-Plate Color Video Camera” will be described.

FIG. 10 is a flowchart showing a processing procedure of the G interpolation unit 201 of FIG.
The procedure will be described below based on this flowchart.
The G interpolation unit 201 performs processing in the loop at all pixel positions by the loop 302.
One pixel processed in one loop is referred to as a target pixel here.
In step ST301, the G interpolation unit 201 reads a pixel value near the target pixel position in the mosaic image.
Next, in step ST303, the G interpolation unit 201 determines whether the color of the filter in the target pixel is G.
Next, when it is determined in step ST303 that the color of the filter is G, the G interpolation unit 201 sets the pixel value of the target pixel as the G pixel value in step ST304.
Next, when it is determined in step ST303 that the filter color is not G, the G interpolation unit 201 proceeds to step ST305.
Next, in step ST305, the G interpolation unit 201 calculates the horizontal gradient GradH by the following equation (2), and calculates the horizontal interpolation GH by the following equation (3). M (x, y) represents the pixel value of the mosaic image at the target pixel position (x, y).

Similarly, in step ST306, the G interpolation unit 201 calculates a vertical gradient GradV by the following equation (4), and calculates a vertical interpolation GV by the following equation (5).

  Using these GradH, GradV, GH, and GV, in step ST307, the pixel value G (x, y) of G of the pixel of interest is interpolated by Expression (6).

  When step ST304 or step ST307 is completed, the interpolation of the G pixel value at the target pixel is completed, so the next loop processing is performed for processing the next target pixel position. When the loop processing is completed for all the pixels, the loop 302 is exited and the processing of the G interpolation unit 201 is completed.

[R interpolation unit 202, R 'interpolation unit 203, B interpolation unit 204, B' interpolation 205]
The R interpolation unit 202, the R ′ interpolation unit 203, the B interpolation unit 204, and the B ′ interpolation unit 205 are the same processes except for the target colors, and thus the R interpolation unit 202 will be described as an example here. . For the interpolation units 203, 204, and 205 relating to R ′, B, and B ′, the notation of R used in the description of the R interpolation unit 202 may be read as R ′, B, and B ′.

FIG. 11 is a flowchart showing a processing procedure of the R interpolation unit 202 of FIG. The procedure will be described below based on this flowchart.
The R interpolation unit 202 performs processing in the loop at all pixel positions by the loop 403. One pixel processed in one loop is referred to as a target pixel here.
The R interpolation unit 202 reads the G pixel value near the target pixel calculated by the G interpolation unit 201 of FIG. 9 in step ST401.
Further, the R interpolation unit 202 reads a pixel value in the vicinity of the target pixel position in the RR′GBB ′ mosaic image in step ST402.
Next, the R interpolation unit 202 calculates the low frequency component Glow of the G pixel in the target pixel in step ST404, and calculates the low frequency component Rlow of the R pixel in the target pixel in step ST405.
The R interpolation unit 202 uses these Glow and Rlow, and in step ST406, interpolates the pixel value R (x, y) of R of the pixel of interest by Expression (7).

  When step ST406 is completed, the interpolation of the R pixel value at the target pixel is completed, so the next loop process is performed for the next target pixel process. When the loop processing is completed for all pixels, the loop 403 is exited, and the processing of the R interpolation unit 202 is completed.

FIG. 12A is a flowchart showing details of the processing procedure of step ST404 in FIG.
This process is a process of averaging the G pixels included in the local region, and constitutes a low-pass filter using an FIR filter.
First, in step ST501, the R interpolation unit 202 sets a variable Glow to 0 and initializes it.
Subsequently, the R interpolation unit 202 performs processing in the loop at all pixel positions in the local region including the target pixel by the loop 502.
The local region used here is the same as the region having the predetermined size described above.
One pixel processed in the local region in one loop is referred to as a local attention pixel here.
In step ST503, the R interpolation unit 202 adds G (s, t) multiplied by WG (s, t) to Glow, and sets the result as a new Glow.
Here, G (s, t) represents the G pixel value at the local pixel of interest position (s, t), and WG (s, t) represents the weighting coefficient.
WG (s, t) may be any coefficient that is a low-pass filter, and is selected so that the sum of the coefficients is 1.

  When the processing in step ST503 is completed, the processing in the local attention pixel is completed, so the next loop processing is performed for processing the next local attention pixel position. When the loop processing is completed for all local pixels, the loop 502 is exited, and the processing of step ST404 of the R interpolation unit 202 is completed.

FIG. 12B is a flowchart showing details of the processing procedure of step ST405 of FIG.
This process is a process of averaging the R pixels included in the local region, and constitutes a low-pass filter using an FIR filter.
First, in step ST601, the R interpolation unit 202 initializes the variable Rlow to 0.
Subsequently, the R interpolation unit 202 performs processing in the loop at all pixel positions in the local region including the target pixel by the loop 602.
The local region used here is the same as the region having the predetermined size described above.
In step ST603, the R interpolation unit 202 determines whether or not the color of the filter in the local attention pixel is R.
When it is determined in step ST603 that the filter color is R, the R interpolation unit 202 adds M (s, t) multiplied by WR (s, t) to Rlow in step ST604, The result is a new Rlow.
Here, M (s, t) represents the pixel value of the mosaic image at the local pixel of interest position (s, t), and WR (s, t) represents the weighting coefficient.
WR (s, t) may be any coefficient that becomes a low-pass filter, and is selected so that the sum of the coefficients is 1.
However, since the R pixels are randomly arranged, different WR (s, t) are used according to the arrangement of R in the local region.
In addition, WR (s, t) is preferably selected so as to be as close as possible to the characteristics of the low-pass filter constituted by WG (s, t).

  Since the processing for the local attention pixel is completed when step ST604 is completed, the next loop processing is performed for the processing of the next local attention pixel position. When the loop processing is completed for all local pixels, the loop 602 is exited, and the processing of step ST405 of the R interpolation unit 202 is completed.

[R MS-SyncNR unit 207, R'MS-SyncNR unit 208, B MS-SyncNR unit 209, B'MS-SyncNR unit 210]
Since the RMS-SyncNR unit 207, the R'MS-SyncNR unit 208, the BMS-SyncNR unit 209, and the B'MS-SyncNR unit 210 are the same processes except for the target colors, here the RMS- The explanation will be made taking SyncNR207 as an example. For the MS-SyncNR units 208, 209, and 210 related to R ′, B, and B ′, the notation of R used in the description of R MS-SyncNR207 may be read as R ′, B, and B ′.

FIG. 13 is a block diagram showing details of the RMS-SyncNR unit 207 of FIG.
The configuration shown in FIG. 13 calculates an output obtained by removing noise from R by inputting two channels of G and R. Since a false color is removed if a component having no R correlation with G is removed, noise removal for G is not performed in the present invention. However, in practice, it is more appropriate to perform noise removal of R according to the present invention after some noise removal from G.
The RMS-SyncNR unit 207 includes two multi-resolution conversion units 701 and 702, a multi-resolution inverse conversion unit 717, and correction units 711, 712, and 713 that are one less than the number of multi-resolution layers (3 in FIG. 13). The The first multi-resolution conversion unit 701 converts the input image of the G channel into multi-resolution image data, and as a result, stores the image signals of each layer of multi-resolution in the corresponding memories 703, 704 705, 706.
Similarly, the second multi-resolution conversion unit 702 converts the input image of the R channel into multi-resolution image data, and as a result stores the image signals of each layer of multi-resolution in the corresponding memories 707, 708, 709, 710. .

  Three correction units 711, 712, and 713 correspond to layers other than the lowest resolution layer, respectively, and input G and R channel images of each layer to correct each pixel mixed with noise. The removed R channel image is generated and stored in the corresponding image memory 714, 715, 716.

Hereinafter, the correction units 711, 712, and 713 will be described in detail.
Since the correction units 711, 712, and 713 have the same configuration and perform the same operation, the correction unit 711 will be described here as an example.
FIG. 14 is a block diagram illustrating an example of the configuration of the correction unit 711.
As illustrated in FIG. 14, the correction unit 711 includes sampling processing units 801 and 802, a variance calculation processing unit 803, a clipping processing unit 804, a division processing unit 805, a covariance calculation processing unit 806, and a multiplication processing unit 807.

  The sampling processing unit 801 samples (extracts) a plurality of G channel pixel values from neighboring pixels at a predetermined position set corresponding to the target pixel position from the G channel image of the corresponding layer, and a multiplication processing unit 807, This is supplied to the variance calculation processing unit 803 and the covariance calculation processing unit 806. The sampling processing unit 802 samples (extracts) a plurality of R channel pixel values from neighboring pixels at a predetermined position set corresponding to the target pixel position from the R channel image of the corresponding layer, and performs a covariance calculation process Supplied to the unit 806. Note that the sampling processing units 801 and 802 extract the G pixel value and the R pixel value at the same predetermined position set for the target pixel position.

  The variance calculation processing unit 803 calculates a variance value around the G pixel of interest based on the sampled G pixel value, and supplies the calculated variance value to the clipping processing unit 804.

  The covariance calculation processing unit 806 calculates the covariance values based on the sampled G and R pixel values and supplies them to the division processing unit 805. When the variance value of the G channel samples is smaller than a predetermined threshold, the clipping processing unit 804 performs clipping with the threshold and supplies the result to the division processing unit 805. The division processing unit 805 divides the covariance value supplied from the covariance calculation processing unit 806 by the variance value supplied from the clipping processing unit 804, and the ratio of the covariance value to the variance value (covariance value / variance value). ) To the multiplication processing unit 807. Note that the processing of the clipping processing unit 804 is processing for avoiding 0% because the subsequent division processing unit 805 calculates (covariance value / variance value). The multiplication processing unit 807 estimates the R channel pixel value from which noise has been removed from the target pixel by multiplying (covariance value / variance value) by the G channel pixel value at the target pixel position, and outputs it.

Here, details of the processing of the correction units 711, 712, and 713 will be described.
The correction units 711, 712, and 713 correct the pixel values by a pixel value estimation method using inter-channel correlation. More specifically, the pixel value estimation method using the correlation between channels is based on the assumption that there is a linear relationship between two channels (for example, G and R) when focusing on the local region. The estimated value of a certain pixel position R is calculated by linear regression calculation.

For example, in the local region around the target pixel in the image, a sample of pixel values of the C ₁ channel (for example, G) (pixel values of a plurality of pixels at a predetermined position corresponding to the position of the target pixel) {C ₁₁ , C ₁₂ , C ₁₃ ,..., C _1N } and C ₂ channel (for example, R) pixel value samples (pixel values of a plurality of pixels at predetermined positions corresponding to the position of the target pixel) {C ₂₁ , C ₂₂ , C ₂₃ ,..., C _2N } is obtained, the luminance estimation value C _2C ′ of the target pixel position of the C ₂ channel is obtained by assuming that there is a linear relationship between the _two. From the luminance value C _1C of the C ₁ channel, it can be estimated by the following equation (8).

Here, M _C1 is the expected value of the C ₁ channel in the local region, M _C2 is the expected value of the C ₂ channel, V _C1C2 is the covariance value of the C ₁ and C ₂ channels, and V _C1C1 is C ₁ The variance value of the channel.

Further, the covariance value V _C1C2 and the variance value V _C1C1 are obtained by the following equations (9) and (10), respectively.

In Expressions (9) and (10), w _i is a predetermined weight coefficient.

  Therefore, noise can be removed by calculating the above-described equation (8). However, as shown in FIG. 13, the RMS-SyncNR unit 207 is provided with multi-resolution conversion units 701 and 702, and the correction unit 711 performs the above-described two-channel processing on the image separated for each band. The pixel value is corrected using the correlation.

  By the way, in the image of a plurality of layers generated by the multi-resolution conversion, the direct current component of the image is concentrated on the layer of the lowest resolution, so that the expected value of the local pixel is 0 in the other layers. As a result, when multi-resolution processing is used, the above-described equations (8) to (10) are replaced by the following equations (11) to (13).

  Accordingly, the variance calculation processing unit 803 in FIG. 14 substantially calculates the variance value by Equation (13), the covariance calculation processing unit 806 calculates the covariance value by Equation (12), and further performs multiplication processing. The unit 807 corrects (corrects) the pixel value using Expression (11).

  Furthermore, in calculating the covariance values and variance values of Equations (12) to (13), the number of multiplication processes is reduced by using an approximate function to reduce multiplication processing with a large processing load by computer calculation processing. For example, it may be calculated by an approximation function such as the following formulas (14) to (15).

  Further, the multi-resolution inverse transform unit 717 receives the R channel image from which noise has been removed from each layer and the R channel image of the lowest resolution layer as input, and integrates and outputs the image to the same resolution as the original image.

As described above, in the present embodiment, there are four or more color filters, some colors are regularly arranged, and the remaining colors are used in a single-plate color image sensor that is randomly arranged. An array of filters is defined, and an image processing apparatus using a color filter having this filter array is configured.
The color filter of this embodiment has a larger number of colors than the Bayer arrangement conventionally used, and can improve color reproduction and dynamic range.
Further, the color filter of this embodiment can obtain a resolution equivalent to the Bayer array due to the presence of the regularly arranged color filters, and at the same time, the random array and the RMS-SyncNR unit 207, R shown in FIG. By using the 'MS-SyncNR unit 208, the B MS-SyncNR unit 209, and the B'MS-SyncNR unit 210, generation of false colors can be effectively reduced.
That is, in the mosaic image obtained by the color filter of the present embodiment, C1 occupies half of the total number of pixels, and thus has a higher resolution than other colors. Therefore, by first aligning C1 at all pixel positions and using that as a reference to estimate (interpolate) high-frequency components of other colors, it is possible to obtain an interpolation result with high resolution for all colors.

In the present embodiment, the image obtained by applying image processing to the mosaic image obtained from the color filter is suppressed from generating false colors biased to a specific spatial frequency band, and has various spatial frequencies. False colors are scattered in the band little by little. For this reason, it is hardly perceived by humans, and can be effectively removed using the RMS-SyncNR unit 207, the R'MS-SyncNR unit 208, the BMS-SyncNR unit 209, and the B'MS-SyncNR unit 210.
The processing using the RMS-SyncNR unit 207, the R'MS-SyncNR unit 208, the BMS-SyncNR unit 209, and the B'MS-SyncNR unit 210 is based on the same concept as the pixel value estimation method using inter-channel correlation. By performing processing to estimate the correlation of other channel signals as high as possible with respect to the reference channel (for example, G) signal, the uncorrelated component between channels is reduced, and noise mixed in chroma components This is an image processing method that can eliminate the phenomenon of color unevenness and color misregistration that are components. The image obtained in the present embodiment has the same characteristics as an image including color unevenness. An image in which false colors exist in various spatial frequency bands is in a state in which the correlation between channels is low, and the RMS-SyncNR unit 207, the R'MS-SyncNR unit 208, and the B MS-SyncNR unit By increasing the correlation using the B'MS-SyncNR unit 210, false colors can be removed.

  Note that the color filter of this embodiment can be regarded as an array equivalent to the Bayer array when thinning readout and addition readout are performed.

Second Embodiment
The second embodiment is an example of the inventions of the second and fourth aspects.
The color filter array of this embodiment is obtained by replacing the point where the color C1 is arranged in a checkered pattern in the color filter array described in the first embodiment with a regular arrangement of two colors C1a and C1b.
As described in Japanese Patent Laid-Open No. 2005-160044, the new color C1c created based on C1a and C1b from the regular arrangement of C1a and C1b is interpolated on all the pixels where C1a and C1b exist Can do.
The arrangement in which C1c is arranged instead of C1a and C1b is equal to the filter arrangement described in the first embodiment.
That is, the color filter of this embodiment has the advantages of the color filter described in the first embodiment, but can further improve color reproduction and dynamic range by increasing the number of colors of the filter by one.

FIG. 15 is a diagram for explaining the configuration of the color filter of the present embodiment.
As shown in FIG. 15, the color filter of this embodiment comprises a checkered arrangement with two colors G and G ′, and the remaining four colors are arranged randomly.

FIG. 16 is a diagram for explaining the demosaic process when the color filter of the present embodiment shown in FIG. 15 is used.
As shown in FIG. 16, in contrast to FIG. 9, the image processing apparatus of this embodiment uses a (G + G ′) interpolation unit a201 instead of the G interpolation unit 201, and R (R ′, B, B ′ ) The same processing as the interpolation unit 202 (203, 204, 205) is added for G and G ′ (G interpolation unit a202, G ′ interpolation unit a203), and as a result, instead of 5 planes, 6 plane images RR′GG′BB ′ interpolation image 1a208 Is generated.
In the subsequent noise removal processing, noise due to random arrangement does not occur in the G image and G ′ image, so the noise removal processing is passed through, and the four images R, R′B, and B ′ are obtained in the same manner as in FIG. On the other hand, MS-SyncNR processing is performed.

Note that the interpolation processing algorithm disclosed in Japanese Patent Laid-Open No. 2005-160044 can be used to realize the (G + G ′) interpolation unit a201. This publication describes a process for interpolating C3 = (C1 + C2) from the two colors C1 and C2 of the array constituting the checkered arrangement with two colors to all pixels.
Also in this embodiment, the block configuration shown in FIG. 8 and the flowchart shown in FIG. 9 described in the first embodiment can be applied.

  As a modification of the color filter of this embodiment, the configuration shown in FIG. 17 may be used. In the color filter, a checkered arrangement is formed with two colors G and G ′, R is arranged every other pixel horizontally and vertically, and the remaining two colors are arranged randomly.

The color filter of this embodiment can further improve color reproduction and dynamic range by replacing the color C1 with C1a and C1b, for example.
In addition, it is preferable that the correlation between the spectral sensitivities of the colors C1a and C1b is higher than the correlation between the spectral sensitivities of the other colors. That is, as described in Japanese Patent Application Laid-Open No. 2005-160044, when creating a new color C1c based on C1a and C1b from the regular arrangement of C1a and C1b, the correlation between the spectral sensitivities of C1a and C1b Higher is preferable.

In addition, in image processing for generating a color image in which all the colors are matched in all pixels from the mosaic image obtained by the image sensor including the color filter of the present embodiment, a new color C1c is used by using the colors C1a and C1b. Is calculated.
When the color C1a is arranged at the target pixel position, the image processing unit estimates (interpolates) the pixel value of the color C1b at the target pixel position using the colors C1a and C1b existing in the local region including the target pixel position. )
When the color C1b is arranged at the target pixel position, the image processing unit interpolates the pixel value of the color C1a at the target pixel position using the colors C1a and C1b existing in the local region including the target pixel position.
The image processing unit calculates the color C1c of the checkered pixel position from the pixel values of C1a and C1b obtained at the checkered pixel position, and calculates the pixel value of the color C1c at the pixel position where the color C1c does not exist Is interpolated using the color C1c existing in, and colors CX other than the color C1c are interpolated using the colors C1c and CX existing in the local region including the target pixel position.
Also in the present embodiment, as shown in FIG. 16, as in the first embodiment, the RMS-SyncNR unit a209, the R′MS-SyncNR unit a210, the B MS-SyncNR unit a211 and the B′MS-SyncNR A part a212 is provided.

As described above, in this embodiment, a new color C1c created based on the colors C1a and C1b from the regular arrangement of the colors C1a and C1b can be interpolated on all the pixels where C1a and C1b exist.
Also in this embodiment, the same effect as the first embodiment can be obtained.

The present invention is not limited to the embodiment described above.
That is, those skilled in the art may make various modifications, combinations, subcombinations, and alternatives regarding the components of the above-described embodiments within the technical scope of the present invention or an equivalent scope thereof.

  100 digital video camera system, 101 lens, 102 aperture, 103 CCD image sensor, 104 correlated double sampling circuit, 105 A / D converter, 106 DSP block, 107 timing generator, 108 D / A converter, 109 video encoder, 110 video Monitor, 111 CODEC, 112 memory, 113 CPU, 114 input device, 201 G interpolation unit, 202 R interpolation unit, 203 R 'interpolation unit, 204 B interpolation unit, 205 B' interpolation unit, 206 RR'GBB 'interpolation image, 207 R MS-SyncNR unit, 208 R 'MS-SyncNR unit, 209 B MS-SyncNR unit, 210 B' MS-SyncNR unit, 211 RR'GBB 'interpolated image, 701 Multi-resolution conversion unit (G pixel), 702 multiplexing Resolution converter (R pixel), 703 Layer 0 image (G pixel), 704 Layer 1 image (G pixel), 705 Layer 2 image (G pixel), 706 Layer 3 image (G pixel), 707 Layer 0 image (R Pixel), 708 Layer 1 image (R pixel), 709 Layer 2 image (R pixel), 710 Layer 3 image (R pixel), 711 Correction unit (Layer 0), 712 Correction unit (Layer 1), 71 3 Correction unit (Layer 2), 714 Layer 0 correction image (R pixel), 715 Layer 1 correction image (R pixel), 716 Layer 2 correction image (R pixel), 717 Multi-resolution inverse conversion unit (R pixel), 801 Sampling processing unit, 802 sampling processing unit, 803 variance calculation processing unit, 804 clipping processing unit, 805 division processing unit, 806 covariance calculation processing unit, 807 multiplication processing unit

Claims (2)

A color filter in which any one of a plurality of types of filters having different spectral characteristics is assigned to each pixel position,
The plurality of types of filters have four colors, the first color is arranged in a checkered pattern, and the second color is arranged every other line horizontally and vertically at pixel positions where the first color does not exist. A color filter in which the third color and the fourth color are randomly arranged at pixel positions where the first color and the second color do not exist. An image sensor having a color filter in which any one of a plurality of types of filters having different spectral characteristics is assigned to each pixel position,
The plurality of types of filters have four colors, the first color is arranged in a checkered pattern, and the second color is arranged every other line horizontally and vertically at pixel positions where the first color does not exist. The image sensor in which the third color and the fourth color are randomly arranged at pixel positions where the first color and the second color do not exist.

Images