JP2015216521A - Color information compensation device and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a color information compensation device for compensating color information in which a captured image that is captured via a color filter is missed.SOLUTION: A color information compensation device 1 includes: vibration component extraction means 10 for each azimuth which extracts a vibration component of a captured image for each azimuth; azimuth correlation strength calculation means 20 which calculates an azimuth correlation strength; an image compensation means 30 for each azimuth/band which generates a compensation image with respect to a G color for each azimuth/band; edge component extraction means 40 for each azimuth which extracts an edge component for each azimuth; high frequency component addition coefficient calculation means 50 which calculates a coefficient corresponding to the edge component; compensation image pixel value calculation means 60 which adds a pixel value of the compensation image for each azimuth/band in accordance with the azimuth correlation strength and the coefficient; G image generation means 70 which replaces a pixel value of a compensation target pixel with the pixel value that is calculated by the compensation image pixel value calculation means 60 and compensates color information of the G color; and R image generation means 90 and B image generation means 90B for compensating color information of other colors than the G color.

Description

  The present invention relates to a color information complementing apparatus that complements missing color information of each color in a captured image captured through a color filter and a program thereof.

In general, a single-plate imaging device used in a digital camera or the like acquires an image signal (captured image) by photoelectrically converting light received through a color filter. At this time, in general, each color such as Bayer (Bayer) sequence shown in Figure 14 (e.g., R, G, B) color filters _{F C} that filters for each pixel is used.

If capturing the image through the color filter F _C of the Bayer array, only it is obtained color signals for each pixel. That is, the image signal captured via a color filter F _C, as shown in FIG. 14, for the G signal, a color filter F _C of R, color signals missing color information of the pixel corresponding to the position B . Similarly, the R signal, G color filters F _C, the color information of the pixel corresponding to the B position, for B signals, the color filter F _C R, the color information of the pixel corresponding to the position G, respectively The missing color signal.

Therefore, in order to display and process such a color signal as a color image, a technique for complementing (demosaicing) the missing color signal is required.
The most basic technique for complementing this color signal is a technique for bilinear complementing the color signal (pixel value) of the missing pixel using the color signal (pixel value) of the non-missing pixel adjacent to the pixel. Is.

For example, Patent Document 1 discloses a technique for obtaining a pixel value of a complement target pixel based on a median value of neighboring pixels of the complement target pixel.
Further, in Patent Document 2, when complementing the pixel value of the complement target pixel, the degree of correlation with respect to the direction of the surrounding pixel of the same color as the complement target pixel is obtained, and the pixel of the complement target pixel is calculated at a ratio according to the degree of correlation. A method for obtaining a value is disclosed.

JP 2006-262167 A JP 2010-104019 A

  As in the method described in Patent Document 1, in the method of calculating the pixel value of the pixel to be complemented by bilinear interpolation, the pixel value between the pixels is smoothed, so that the spatial frequency component is not properly reproduced. There is a problem that the image is blurred. Furthermore, in the method described in Patent Document 1, since the pixel value of the pixel to be complemented is obtained from the intermediate value of each color, there is a problem that a false color that is a color that does not exist in the actual subject is likely to occur. is there.

  Further, as in the method described in Patent Document 2, the method of performing bilinear interpolation according to the degree of correlation with respect to the direction of surrounding pixels refers only to the correlation of a specific color. Therefore, this method has a problem that it cannot accurately estimate the azimuth correlation of the entire color information of the color image and cannot reproduce a high-quality image.

  In view of such a problem, the present invention appropriately obtains the azimuth correlation and frequency characteristics of the color information of the image when complementing the missing color information of the captured image captured through the color filter, It is an object of the present invention to provide a color information complementing apparatus capable of improving the reproducibility of a spatial frequency component and generating a perceptually natural color image and a program thereof.

  In order to solve the above problems, a color information complementing apparatus according to the present invention is a color information complementing apparatus that supplements color information lacking each color from a captured image captured by a single-plate imaging device using a color filter. Azimuth-specific vibration component extraction means, azimuth correlation strength calculation means, azimuth / band-specific image complement means, azimuth-specific edge component extraction means, high-frequency component addition coefficient calculation means, complementary image pixel value calculation means, identification A color complementing unit and a plurality of other color complementing units for color information other than the specific color. The other color complementing unit includes a difference unit, a difference complementing unit, and an adding unit.

In such a configuration, the color information complementing device extracts the horizontal and vertical vibration components (spatial frequency components) for each pixel of the captured image by the orientation-specific vibration component extraction unit, so that all the pixels of the entire captured image are extracted. , The vibration component for each direction is extracted.
The color information complementing device calculates the strength of the azimuth correlation for each pixel based on the correlation of the vibration components extracted by the directional vibration component extraction means by the azimuth correlation strength calculation means.

  Thereby, the color information complementing apparatus uses not only the specific color information but also color information other than the specific color included in the captured image, for each pixel of the captured image, in the horizontal direction and the vertical direction. It is possible to specify how much intensity is correlated. For example, when it is specified that there is a correlation only in the horizontal direction, the pixel is not correlated with a pixel in the vertical direction, and the color information complementing device uses the pixel in the horizontal direction when complementing the color information. It will be sufficient to complement.

  Further, the color information complementing device complements the pixel value of the pixel to be complemented with respect to a specific color of the captured image for each band of the low frequency component and the high frequency component for each of the horizontal direction and the vertical direction by the image complementing unit for each direction and band. Then, an image classified by direction and band is generated. As a result, the color information complementing device, in the complementing target pixel, the pixel value when there is a correlation of the low frequency component only in the horizontal direction, the pixel value when there is a correlation of the high frequency component only in the horizontal direction, and the vertical direction only. The pixel value when there is a correlation between low frequency components and the pixel value when there is a correlation between high frequency components only in the vertical direction can be obtained individually.

In addition, the color information complementing apparatus extracts edge components for each direction of the horizontal direction, vertical direction, and diagonal direction for each pixel of the captured image by the edge component extraction unit for each direction.
Then, the color information complementing device corrects the high-frequency component of the image for each direction / band based on the size of the edge component extracted by the edge-specific edge component extraction unit by the high-frequency component addition coefficient calculation unit, for example, The high frequency component addition coefficient is calculated by a predetermined calculation so that the coefficient increases as the size of the edge component increases.
Then, the color information complementing device calculates the pixel value of the azimuth / band image supplemented by the azimuth / band-specific image complementing unit according to the azimuth correlation strength and the high-frequency component addition coefficient. Addition is performed to calculate the pixel value of the pixel to be complemented.

  The pixel value of the pixel to be complemented is a pixel value when there is a correlation only in the horizontal direction and a pixel value when there is a correlation only in the vertical direction, depending on the strength of the azimuth correlation, that is, according to the ratio. Therefore, the value includes a vibration component (spatial frequency component) corresponding to the orientation correlation. In addition, for the pixel value of the complement target pixel, by increasing the ratio of addition for pixels with a large edge component, it is possible to suppress an increase in noise at a flat portion with a small edge component, and at a portion with a large edge component. High spatial frequency can be reproduced.

  Then, the color information complementing device replaces the pixel value of the pixel to be complemented for the specific color with the pixel value calculated by the complementary image pixel value calculating unit in the captured image by the specific color complementing unit. In other words, the color information complementing device complements the pixel value of the pixel to be complemented for a specific color with the color information corresponding to the azimuth correlation and the band characteristics in the captured image.

In addition, the color information complementing device supplements the color information from the captured image for a color other than the specific color by the other color complementing unit.
That is, the color information complementing device calculates a difference value between the pixel value of the pixel and the pixel value of the complementary image corresponding to the pixel for each pixel of the target color in the captured image by the difference unit of the other color complementing unit. To do. Thus, the difference value is a value in which the azimuth correlation and the band characteristic are added.
Then, the color information complementing device complements the difference value of the complement target pixel from the difference value of the surrounding pixels of the complement target pixel with respect to the target color by the difference complement means of the other color complement means.
Further, the color information complementing device adds the pixel value of the complementary image corresponding to the pixel for each pixel of the image whose difference value is complemented by the difference complementing unit by the adding unit of the other color complementing unit. As a result, the pixel value of the target color is calculated. Further, the pixel value of the target color generated in this way corresponds to the azimuth correlation and the band characteristic of the specific color.

The present invention has the following excellent effects.
According to the present invention, the pixel value of the pixel to be complemented is obtained by performing weighted addition for each pixel of the complemented image complemented according to the direction and the band based on the local properties (direction and edge components) of the image. Therefore, it is possible to suppress the occurrence of visually noticeable distortion in the flat portion of the image, while reproducing the high spatial frequency in the edge portion of the image and suppressing the generation of false colors.
Therefore, according to the present invention, it is possible to generate a visually natural color image by accurately complementing color information from a captured image lacking color information captured through a color filter.

It is a block block diagram which shows the structure of the color information complementation apparatus which concerns on embodiment of this invention. It is a block block diagram which shows the structure of the vibration component extraction means according to direction of FIG. (A) is a figure which shows each example of the linear filter which extracts a horizontal vibration component in the vibration component extraction means classified by direction of FIG. 2, (b) is a linear filter which extracts a vertical vibration component. It is explanatory drawing for demonstrating typically the calculation content in the azimuth | direction correlation intensity | strength calculation means of FIG. It is a block block diagram which shows the structure of the image complementing means classified by direction and zone | band of FIG. 5 is a linear filter for complementing the captured image, in which (a) is a horizontal low-pass filter, (b) is a horizontal high-pass filter, (c) is a vertical low-pass filter, (D) is a figure which shows each example of a vertical high-pass filter. It is a block block diagram which shows the structure of the edge component extraction means according to direction of FIG. 7 is a linear filter for detecting edge components, in which (a) is an E direction edge detection filter, (b) is an NE direction detection filter, (c) is an N direction detection filter, (D) is an NW direction detection filter, (e) is a W direction detection filter, (f) is an SW direction detection filter, (g) is an S direction detection filter, and (h) is a diagram showing examples of an SE direction detection filter. It is. It is a block block diagram which shows the structure of the high frequency component addition coefficient calculation means of FIG. It is a block block diagram which shows the structure of the complementary image pixel value calculation means of FIG. FIG. 2 is an explanatory diagram for explaining an example of complementing pixels at B and G pixel positions in the R image generation unit of FIG. 1, wherein (a) is an example of complementing pixels at a B pixel position; These are explanatory drawings for demonstrating the example which complements the pixel of G pixel position. It is a flowchart which shows operation | movement of the color information complementation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the characteristic of the color information complementation apparatus which concerns on embodiment of this invention, Comprising: (a) is the figure which extracted a part (for 5 pixels) of a Bayer arrangement | sequence image, (b) is a color signal. It is a figure for demonstrating the correlation of these. It is explanatory drawing for demonstrating the missing state of each color in the captured image imaged with the color filter of the Bayer arrangement.

Embodiments of the present invention will be described below with reference to the drawings.
[Configuration of color information complementer]
First, the configuration of the color information complementing apparatus 1 according to the embodiment of the present invention will be described with reference to FIG.
The color information complementing apparatus 1 shown in FIG. 1 complements color information lacking each color from a captured image captured by a single-plate imaging device using a color filter. Here, the color arrangement of the color filter is a Bayer arrangement, and each color is three primary colors of R (red), G (green), and B (blue). That is, the color information complementing apparatus 1 inputs the captured image (Bayer array image) through a color filter F _C shown in FIG. 14, G signal (G image), R signal (R image), and B signals An RGB image is generated by complementing missing color information in each of the (B images).

  This color information complementing apparatus 1 includes an orientation-specific vibration component extraction means 10, an orientation correlation strength calculation means 20, an orientation / band-specific image complement means 30, an orientation-specific edge component extraction means 40, and a high-frequency component addition coefficient calculation means. 50, complementary image pixel value calculating means 60, G image generating means 70, R pixel extracting means 80, B pixel extracting means 80B, R image generating means 90, B image generating means 90B, and RGB image generating Means 100.

The direction-specific vibration component extraction means 10 extracts vibration components for each horizontal direction and vertical direction for each pixel of the Bayer array image that is the input captured image. The vibration component is a variation in brightness (luminance) and is a spatial frequency component (AC [AC] component) of an image.
Here, with reference to FIG. 2 (refer to FIG. 1 as appropriate), a detailed configuration of the vibration component extracting means 10 for each direction will be described. As shown in FIG. 2, the directional vibration component extraction means 10 includes horizontal vibration component extraction means 11 (11 ₁ , 11 ₂ ) and vertical vibration component extraction means 12 (12 ₁ , 12 ₂ ).

  The horizontal vibration component extracting unit 11 extracts a horizontal vibration component for each pixel of the Bayer array image. The horizontal vibration component extraction unit 11 applies a linear filter that captures a local horizontal vibration component to the Bayer array image, and performs a two-dimensional convolution process to extract the horizontal vibration component. Here, the local linear filter is a filter that covers the local region including the minimum configuration composed of at least all the colors (RGB in the case of a Bayer array image) in the color arrangement configuration of the input captured image. .

Specifically, the horizontal vibration component extraction unit 11 extracts a horizontal vibration component using a linear filter (horizontal vibration component extraction filter) illustrated in FIG. As shown in (a-1) and (a-2) of FIG. 3A, here, two filters having different spatial frequency scales (the number of filter taps and the corresponding spatial frequency) are used in the horizontal direction. As described above, by using the filter of at least two rows, the horizontal vibration component extracting unit 11 extracts the horizontal vibration component from the luminance component of the local region including the arrangement positions of all the colors of RGB in the Bayer array image. Can do. Incidentally, the horizontal vibration component extracting means 11 _1, using the filter _{F H1} in Figure 3 (a) (a-1), the horizontal vibration component extracting means 11 _2, 3 in (a) of (a-2) A filter F _H2 is used.

Then, the horizontal vibration component extracting means 11 _1, an image signal obtained by extracting the horizontal vibration component (the horizontal vibration component image H _1), and outputs the azimuth correlation intensity calculation unit 20.
The horizontal vibration component extracting means 11 _2, an image signal obtained by extracting the horizontal vibration component (the horizontal vibration component image H _2), and outputs the azimuth correlation intensity calculation unit 20.

  The vertical vibration component extraction unit 12 extracts a vertical vibration component for each pixel of the Bayer array image. This vertical vibration component extraction means 12 applies a linear filter that captures local vertical vibration components to the Bayer array image only in the direction of the vibration component to be extracted from the horizontal vibration component extraction means 11. By performing the convolution process, the vertical vibration component is extracted.

Specifically, the vertical vibration component extraction unit 12 extracts a vertical vibration component using a linear filter (vertical vibration component extraction filter) illustrated in FIG. As shown in (b-1) and (b-2) of FIG. 3B, here, two filters having different spatial frequency scales (number of filter coefficients) are used in the vertical direction. As described above, by using at least two columns of filters, the vertical vibration component extraction unit 12 extracts the vertical vibration component from the luminance component of the local region including the arrangement positions of all the colors of RGB in the Bayer array image. Can do. The vertical vibration component extracting means 12 _1, using the filter _{F V1} in FIG. 3 (b) (b-1), the vertical vibration component extracting means 12 _2, 3 (b), the (b-2) A filter F _V2 is used.

Then, vertical vibration component extracting means 12 _1, an image signal obtained by extracting the vertical vibration component (the vertical vibration component image V _1), and outputs the azimuth correlation intensity calculation unit 20.
The vertical vibration component extracting means 12 _2, an image signal obtained by extracting the vertical vibration component (the vertical vibration component image V _2), and outputs the azimuth correlation intensity calculation unit 20.
In this example, the directional vibration component extraction unit 10 includes two horizontal vibration component extraction units 11 and two vertical vibration component extraction units 12, but at least one or more of them may be provided. . Further, the horizontal vibration component extraction means 11 and the vertical vibration component extraction means 12 are the same number.

In addition, when the vibration component extraction unit 10 by direction includes a plurality of horizontal vibration component extraction units 11 and vertical vibration component extraction units 12, a filter having a different spatial frequency scale is used in each direction. As a result, the azimuth-specific vibration component extraction unit 10 can obtain horizontal and vertical vibration components at a plurality of different resolutions, and reduces errors when the azimuth correlation strength calculation unit 20 described later calculates the azimuth correlation of the vibration components. Can be made.
Returning to FIG. 1, the description of the configuration of the color information complementing apparatus 1 will be continued.

The azimuth correlation strength calculation means 20 calculates the strength of the local azimuth correlation for each pixel of the Bayer array image based on the correlation between the horizontal vibration component and the vertical vibration component extracted by the directional vibration component extraction means 10. Is.
That is, the azimuth correlation strength calculating means 20 has a high degree of correlation in the vertical direction if the horizontal vibration component is greater than the vertical vibration component, and the horizontal direction if the vertical vibration component is greater than the horizontal vibration component. A parameter (azimuth nonuniformity parameter) indicating that the degree of correlation is high is calculated.

Specifically, the azimuth correlation strength calculating unit 20 extracts the horizontal vibration component image H _n (here, H ₁ , H ₂ ) extracted by the horizontal vibration component extracting unit 11 (see FIG. ₂ ) and the vertical vibration component extraction. By performing the following equation (1) from the vertical vibration component image V _n (here, V ₁ , V ₂ ) extracted by the means 12 (see FIG. 2), the azimuth nonuniformity is obtained for each pixel. The parameter λ is calculated.

Here, (i, j) indicates the pixel position of i row and j column of the image. | H _n (i, j) | is the absolute value of the pixel value at the pixel position (i, j) of the horizontal vibration component image H _n , and | V _n (i, j) | is the vertical vibration component image V _n . The absolute values of the pixel values at the pixel position (i, j) are respectively shown.
Further, p is an arbitrary real number greater than or equal to “0” for enhancing the strength of the azimuth correlation (azimuth nonuniformity). Although it is desirable to use a value of about “1” or “2” for this p, it is not limited thereto. N is the number of spatial frequency scales, and N = 2 in this embodiment. The number of spatial frequency scales is predetermined according to the number of horizontal vibration component extraction means 11 (the same number of vertical vibration component extraction means 12).

In this equation (1), a small constant may be added to the denominator on the right side or the denominator and numerator to avoid the influence of division by “0”. Alternatively, when the denominator is “0”, no division is performed, and the calculation result may be “0.5”.
Further, in order to reduce noise, the azimuth correlation strength calculation means 20 spatially smoothes an image using the value of the azimuth nonuniformity parameter λ (i, j) obtained by the equation (1) as a pixel value. May be used.

  According to the equation (1), the azimuth nonuniformity parameter λ is λ = 0 when only the horizontal vibration component is extracted, λ = 1 when only the vertical vibration component is extracted, the horizontal direction and When the vibration components in the vertical direction are equal, λ = 0.5. Further, the azimuth nonuniformity parameter λ is a real value in the range of 0.5 <λ ≦ 1 if the horizontal direction is smaller than the vibration component in the vertical direction, and 0 ≦ λ <0.5 if it is larger. .

  The azimuth correlation strength calculation unit 20 outputs the calculated azimuth nonuniformity parameter λ for each pixel to the complementary image pixel value calculation unit 60. Here, the azimuth correlation strength calculation means 20 uses the azimuth nonuniformity parameter λ based on the horizontal direction as the azimuth correlation and the azimuth nonuniformity parameter (1-λ) based on the vertical direction as the azimuth correlation. , And output to the complementary image pixel value calculation means 60.

  Here, with reference to FIG. 4 (refer to FIGS. 1 to 3 as appropriate), the calculation contents of the equation (1) performed by the azimuth correlation strength calculating means 20 will be schematically described. In FIG. 4, the level display on the right side of each image indicates the pixel value of each image.

The azimuth correlation intensity calculating means 20 extracts, as a horizontal vibration component, the horizontal vibration component image H ₁ that has been extracted by the filter F _H1 in the horizontal vibration component extracting means 11 _1, the filter F _H2 in the horizontal vibration component extracting means 11 ₂ horizontal vibration component image H ₂ which is is input.
Further, the azimuth correlation intensity calculating means 20, as a vertical vibration component, a vertical vibration component image V _1, which is extracted by the filter F _V1 in the vertical vibration component extracting means 12 _1, the vertical vibration component extracting means 12 ₂ filter F _V2 a vertical vibration component image V ₂ extracted by is input.
In FIG. 4, horizontal vibration component images | H ₁ |, | H ₂ | and vertical vibration component images | V ₁ |, | V ₂ | are respectively represented as horizontal vibration component images H ₁ , H ₂ , and vertical vibration components. It is an image that takes an absolute value for each pixel of the images V ₁ and V ₂ .

Then, the azimuth correlation strength calculation means 20 {| V ₁ | / (| H ₁ | + | V ₁ |)} for each pixel with respect to the horizontal vibration component image H ₁ and the vertical vibration component image V ₁ . Alternatively, by calculating {V ₁ ² / (H ₁ ² + V ₁ ² )}, an intensity distribution image Λ ₁ indicating the spatial distribution of the strength of the azimuth correlation (azimuth nonuniformity parameter λ ₁ ) is generated. .
In addition, the azimuth correlation strength calculation unit 20 {| V ₂ | / (| H ₂ | + | V ₂ |)} for each pixel with respect to the horizontal vibration component image H ₂ and the vertical vibration component image V ₂ . Alternatively, by calculating {V ₂ ² / (H ₂ ² + V ₂ ² )}, an intensity distribution image Λ ₂ indicating the spatial distribution of the strength of the azimuth correlation (azimuth nonuniformity parameter λ ₂ ) is generated. .
Then, the azimuth correlation strength calculating means 20 averages the intensity distribution image Λ ₁ and the intensity distribution image Λ ₂ for each pixel, and obtains an intensity distribution image Λ indicating the spatial distribution of the averaged azimuth nonuniformity parameter λ. Generate.

As described above, the azimuth correlation strength calculating means 20 calculates the azimuth nonuniformity parameters (λ ₁ , λ ₂ ) for each spatial frequency scale and averages them, thereby calculating the azimuth correlation from a plurality of images having different resolutions. Can be sought. As a result, the azimuth correlation strength calculation means 20 can reduce the error and obtain a highly accurate azimuth correlation as compared with the case of only a single spatial frequency scale.
Returning to FIG. 1, the description of the configuration of the color information complementing apparatus 1 will be continued.

  The azimuth / band-specific image complementing means 30 complements the pixel values of the pixels of a specific color in the Bayer array image, which is the input captured image, for each of the low-frequency component and high-frequency component bands in the horizontal and vertical directions. Thus, an image classified by direction and band is generated. Here, the specific color is a color in which the pixel position of the pixel to be complemented in the horizontal direction with respect to the specific color matches the pixel position of the pixel to be complemented in the vertical direction. That is, in the case of a Bayer array image, the specific color is G (green) arranged every other pixel in the horizontal and vertical directions.

  Here, with reference to FIG. 5 (refer to FIG. 1 as appropriate), a detailed configuration of the image complementing unit 30 for each direction / band will be described. As shown in FIG. 5, the azimuth / band-specific image complementing means 30 includes a horizontal low-frequency component complementing means 31, a horizontal high-frequency component complementing means 32, a vertical low-frequency component complementing means 33, and a vertical high-frequency component complementing means 34. .

The horizontal low-frequency component complementing means 31 averages the G pixel values adjacent in the horizontal direction at the R and B pixel positions of the Bayer array image, and complements the G pixel value at the R and B pixel positions. Is.
More specifically, the horizontal low-frequency component complementing means 31 uses a horizontal low-pass filter F _HL having a coefficient of “1/2, 0, 1/2” as shown in FIG. By filtering the Bayer array image in the horizontal direction, the G pixel value is complemented at the R and B pixel positions.
The horizontal low-frequency component complementing unit 31, a horizontal low-frequency component complementary image G _HL complemented the G in the horizontal direction, and outputs the complemented image pixel value computing unit 60.

The horizontal high-frequency component complementing means 32 calculates a high-frequency component at the R and B pixel positions of the Bayer array image from the pixel value of the pixel and the pixel value of the pixel at a position two pixels away in the horizontal direction. It complements the G pixel value at the B pixel position.
More specifically, the horizontal high-frequency component complementing means 32 is a horizontal high frequency band having a coefficient of “−1/4, 0, 2/4, 0, −1/4” as shown in FIG. By filtering the Bayer array image in the horizontal direction using the filter F _HH , the G pixel value is complemented at the R and B pixel positions.
The horizontal high frequency component complementing unit 32 outputs a horizontal high frequency component supplemented image _{GHH in} which G is complemented in the horizontal direction to the complementary image pixel value calculating unit 60.

The vertical low frequency component complementing means 33 averages the G pixel values adjacent in the vertical direction at the R and B pixel positions of the Bayer array image, and complements the G pixel value at the R and B pixel positions. Is.
More specifically, the vertical low-frequency component complementing means 33 uses a vertical low-pass filter F _VL having a coefficient of “1/2, 0, 1/2” as shown in FIG. By filtering the Bayer array image in the vertical direction, the G pixel value is complemented at the R and B pixel positions.
The vertical low-frequency component complementing unit 33, a vertical low-frequency component complementary image G _VL complemented the G in the vertical direction, and outputs the complemented image pixel value computing unit 60.

The vertical high frequency component complementing means 34 calculates a high frequency component from the pixel value of the pixel at the R and B pixel positions of the Bayer array image and the pixel value of the pixel at a position two pixels away in the vertical direction. It complements the G pixel value at the B pixel position.
More specifically, the vertical high frequency component complementing means 34 is a vertical high frequency band having a coefficient of “−1/4, 0, 2/4, 0, −1/4” as shown in FIG. By filtering the Bayer array image in the vertical direction using the filter F _VH , the G pixel value is complemented at the R and B pixel positions.
The vertical high-frequency component complementing unit 34 outputs a vertical high-frequency component supplemented image G _VH supplemented with G in the vertical direction to the complementary image pixel value calculating unit 60.

Note that the filters used by the azimuth / band-specific image complementing means 30 are not limited to the coefficients and filter lengths shown in FIGS. That is, the low-pass filter (horizontal low-pass filter F _HL , vertical low-pass filter F _VL ) may be a smoothing filter that smoothes the pixel values of adjacent pixels, and the high-pass filter (horizontal high-pass filter F _HH , The vertical high-pass filter F _VH ) may be a high-pass filter that detects a change in pixel value.
Returning to FIG. 1, the description of the configuration of the color information complementing apparatus 1 will be continued.

  The orientation-specific edge component extraction unit 40 extracts edge components for each pixel in the horizontal direction, the vertical direction, and the oblique direction from the input Bayer array image. This directional edge component extraction means 40 has eight directions (E direction [right direction], NE direction [upper right direction], N direction [upward direction], NW direction [upper left direction], W direction [left direction], and SW direction. [Left lower direction], S direction [lower direction], SW direction [lower right direction]) are extracted edge components existing between adjacent pixels.

  Here, with reference to FIG. 7 (refer to FIG. 1 as appropriate), a detailed configuration of the orientation-specific edge component extraction unit 40 will be described. As shown in FIG. 7, the edge component extraction unit 40 by direction includes an E direction edge component extraction unit 41, an NE direction edge component extraction unit 42, an N direction edge component extraction unit 43, and an NW direction edge component for each direction. Extraction means 44, W direction edge component extraction means 45, SW direction edge component extraction means 46, S direction edge component extraction means 47, and SE direction edge component extraction means 48 are provided.

The E-direction edge component extraction unit 41 extracts an edge component in the E direction (right direction) from the pixel values of eight pixels around the pixel at the R and B pixel positions of the Bayer array image.
More specifically, E direction edge component extraction unit 41 uses the E direction edge detection filter F _E coefficients as shown in FIG. 8 (a), filtering the Bayer array image, by absolute values , E direction edge component _ME is extracted.
The E direction edge component extraction means 41 outputs the extracted E direction edge component _ME to the high frequency component addition coefficient calculation means 50.

  Further, NE direction edge component extraction means 42, N direction edge component extraction means 43, NW direction edge component extraction means 44, W direction edge component extraction means 45, SW direction edge component extraction means 46, S direction edge component extraction means 47, Similarly to the E-direction edge component extraction unit 41, the SE-direction edge component extraction unit 48 determines each direction (N from the pixel values of 8 pixels around the pixel at the R and B pixel positions of the Bayer array image. Direction, NW direction, W direction, SW direction, S direction, and SE direction) are extracted.

More specifically, the NE direction edge component extraction means 42 filters the Bayer array image using the NE direction edge detection filter F _NE as shown in FIG. The edge component M _NE is extracted.
Further, N direction edge component extraction unit 43 using the N direction edge detection filter F _N as shown in FIG. 8 (c), filtering the Bayer array image, by absolute values, N direction edge component M _N To extract.
Moreover, NW direction edge component extraction unit 44 with the NW direction edge detection filter F _NW as shown in FIG. 8 (d), filtering the Bayer array image, by absolute values, NW direction edge component M _NW To extract.

Further, W direction edge component extraction unit 45 by using the W direction edge detection filter F _W as shown in FIG. 8 (e), filtering the Bayer array image, by absolute values, W direction edge component M _W To extract.
Further, SW direction edge component extraction unit 46 by using the SW direction edge detection filter F _SW as shown in FIG. 8 (f), filtering the Bayer array image, by absolute values, SW direction edge component M _SW To extract.
Further, S direction edge component extraction unit 47 by using the S-direction edge detection filter F _S as shown in FIG. 8 (g), filtering the Bayer array image, by absolute values, S direction edge component M _S To extract.

Further, SE direction edge component extraction unit 48 using the SE direction edge detection filter F _SE as shown in FIG. 8 (h), filtering the Bayer array image, by absolute values, SE direction edge component M _SE To extract.
The NE direction edge component extraction means 42, the N direction edge component extraction means 43, the NW direction edge component extraction means 44, the W direction edge component extraction means 45, the SW direction edge component extraction means 46, the S direction edge component extraction means 47, The SE direction edge component extracting means 48 extracts the extracted edge components (M _NE , M _N , M _NW , M _W , M _SW , M _S , M _SE ) in the respective directions, and the high frequency component addition coefficient calculating means 50. Output to.

As described above, the edge-specific edge component extraction unit 40 can extract edge components from the Bayer array image by using the direction-specific filters shown in FIGS.
Returning to FIG. 1, the description of the configuration of the color information complementing apparatus 1 will be continued.

  The high-frequency component addition coefficient calculating unit 50 complements the G pixel values at the R and B pixel positions of the Bayer array image from the edge components for each direction (eight directions) extracted by the direction-specific edge component extraction unit 40. Is a coefficient for correcting the high frequency component (high frequency component addition coefficient w).

  Here, with reference to FIG. 9 (refer to FIG. 1 as appropriate), a detailed configuration of the high-frequency component addition coefficient calculation means 50 will be described. As shown in FIG. 9, the high frequency component addition coefficient calculation means 50 includes a maximum value detection means 51 and a coefficient conversion means 52.

The maximum value detecting means 51 detects the maximum value of the edge component extracted by the edge-specific edge component extracting means 40.
That is, as shown in the following formula (2), the maximum value detecting unit 51 detects the edge components (M _E , M _NE , M _N , M _NW , M M) extracted by the direction-specific edge component extracting unit 40. _{For W 1} , M _SW , M _S , M _SE ), the maximum edge component value M _MAX (i, j) is determined for each pixel position (i, j) of the Bayer array image. Note that (i, j) indicates the pixel position of i row and j column of the image.

  The maximum value detection means 51 outputs the detected maximum value of the edge component to the coefficient conversion means 52 for each pixel.

The coefficient conversion means 52 converts the maximum value of the edge component detected by the maximum value detection means 51 into a high frequency component addition coefficient. The high-frequency component addition coefficient is a coefficient for correcting the high-frequency component when complementing the G pixel values at the R and B pixel positions of the Bayer array image in the complementary image pixel value calculation unit 60 described later.
The coefficient conversion means 52 converts the maximum value M _MAX of the edge component into a real value high frequency component addition coefficient in the range of 0 to 1. Here, the coefficient conversion means 52 uses the two predetermined threshold values thr1 and thr2, and converts the edge component for each pixel position (i, j) of the Bayer array image by the conversion equation shown in the following equation (3). converting the maximum value _M MAX (i, j) to the high-frequency component addition coefficient w (i, j).

The threshold values thr1 and thr2 do not need to be fixedly set in advance, and may be appropriately adjusted from the outside. Alternatively, the threshold value thr1 may be fixed to “0”, and only the threshold value thr2 may be adjusted within the range of the maximum value that 0 <thr2 ≦ the edge component can take.
The coefficient conversion unit 52 outputs the converted high-frequency component addition coefficient to the complementary image pixel value calculation unit 60 for each pixel.
Returning to FIG. 1, the description of the configuration of the color information complementing apparatus 1 will be continued.

The complementary image pixel value calculation means 60 complements the direction (horizontal direction / vertical direction) and the band (low frequency component / high frequency component) by the azimuth / band-specific image complementation means 30 for each pixel to be complemented for the specific color (G). The pixel values of the obtained image are added according to the strength of the azimuth correlation and the high frequency component addition coefficient, and the pixel value of the complementary image corresponding to the azimuth and the band is calculated.
In other words, the complementary image pixel value calculating unit 60 includes the horizontal low-frequency component complementary image G _HL , the horizontal high-frequency component complementary image G _HH , the vertical low-frequency component complementary image G _VL, and the vertical that are complemented by the azimuth / band-specific image complementing unit 30. The high-frequency component complement image G _VH is complemented with respect to G according to the azimuth nonuniformity parameter λ calculated by the azimuth correlation strength calculation means 20 and the high-frequency component addition coefficient w calculated by the high-frequency component addition coefficient calculation means 50. A pixel value for each target pixel is calculated.

Here, the pixel values at the positions of the pixel i row j column of the complementary images G _HL , G _HH , G _VL, and G _VH supplemented by the azimuth / band image complementing means 30 are represented as G _HL (i, j), G _{When HH} (i, j), G _VL (i, j), and G _VH (i, j) are used, the complementary image pixel value calculation means 60 uses the following equation (4) to calculate the strength and high-frequency component of the azimuth correlation. A pixel value G (i, j) corresponding to the addition coefficient is calculated.

This equation (4) can be calculated by the detailed configuration of the complementary image pixel value calculating means 60 shown in FIG. That is, multiplication means 61 for multiplying λ (i, j) and pixel value G _HL (i, j) and multiplication means 62 for multiplying λ (i, j) and pixel value G _HH (i, j). And multiplication means 63 for multiplying the multiplication result of the multiplication means 62 and the high frequency component addition coefficient w (i, j), and (1-λ (i, j)) and the pixel value G _VL (i, j). Multiplying means 64 for multiplying, multiplying means 65 for multiplying (1−λ (i, j)) and pixel value G _VH (i, j), the multiplication result of the multiplying means 65 and the high frequency component addition coefficient w (i , J) and multiplication means 66 for multiplying, and addition means 67 for adding the multiplication results of the multiplication means 61, 63, 64, 66, respectively.
The complementary image pixel value calculating unit 60 associates the pixel value of G with the pixel position of the complement target pixel with respect to G, and sends the G pixel value to the G image generating unit 70, the R image generating unit 90, and the B image generating unit 90B. Output.

The G image generation unit (specific color complementing unit) 70 calculates the pixel value of the pixel to be complemented for the specific color (G) in the Bayer array image that is the input captured image by the complementary image pixel value calculation unit 60. It replaces with pixel values and complements the color information of the specific color (G).
That is, the G image generation means 70 uses the pixel value of the Bayer array image as it is for the pixel value at the G pixel position of the Bayer array image. The G image generation means 70 uses the G pixel value calculated by the complementary image pixel value calculation means 60 for the pixel values at the R and B pixel positions of the Bayer array image.
Thereby, the G image generation means 70 can complement pixels other than the G pixel position in the Bayer array image with the G pixel value in consideration of the direction corresponding to the vibration component and the edge component for each band. it can.
The G image generation unit 70 outputs the generated G image to the RGB image generation unit 100.

  The R pixel extraction unit 80 extracts pixels other than the specific color (G) from the Bayer array image that is the input captured image. Here, the R pixel extraction unit 80 extracts the pixel value of the pixel corresponding to the R pixel position in the Bayer array image, and outputs the pixel value to the R image generation unit 90.

  The R image generating means (other color complementing means) 90 complements color information for colors other than the specific color (G) of the layer array image. The R image generation unit 90 complements the color information of the complement target pixel that is the G and B pixel positions of the Bayer array image with R as the target color. Here, the R image generation unit 90 includes a difference unit 91, a difference complement unit 92, and an addition unit 93.

For each pixel of the target color (R) extracted by the R pixel extraction unit 80, the difference unit 91 calculates a difference value between the pixel value of the pixel and the pixel value of the pixel of the specific color (G) corresponding to the pixel. Is to be calculated.
That is, the difference means 91 is the same as the pixel position calculated from the pixel value at the R pixel position of the Bayer array image extracted by the R pixel extraction means 80, and is calculated (complemented) by the complementary image pixel value calculation means 60. By subtracting the pixel value, a difference value (R−G) between the pixel values of R and G is calculated.
The difference unit 91 outputs the calculated difference value between R and G to the difference complement unit 92.

The difference complementing means 92 complements the difference value of the complement target pixel from the difference values of the peripheral pixels whose difference values have already been calculated in the complement target pixel for the target color (R). That is, the difference complementing unit 92 estimates (calculates) a difference value of (RG) for the G and B pixel positions other than the R pixel in the Bayer array image.
The difference complementing means 92 outputs the (RG) difference value at the R pixel position in the Bayer array image and the (RG) difference value complemented at the G and B pixel positions to the adding means 93.

Note that as a method of complementing the difference value of the pixel to be complemented from the already calculated difference value of the surrounding pixels, a general complementing method may be used.
For example, as shown in FIG. 11A, the difference complementing unit 92, when the B pixel position of the Bayer array image is the position of pixel i row j column, as shown in the following equation (5), An average value of four (R−G) difference values Δ at the upper left, lower left, upper right, and lower right of the obtained B pixel position is obtained, and the difference value Δ (i, j) at the B pixel position (i, j) is obtained. ).

  Further, as shown in FIG. 11B, when the G pixel position of the Bayer array image is set to the position of pixel i row j column as shown in FIG. An average value of four (R−G) difference values Δ on the upper, lower, left, and right sides with respect to the obtained G pixel position is obtained, and the difference value Δ (i, j) at the G pixel position (i, j) is complemented.

The adding unit 93 adds the pixel value of the pixel of the specific color (G) image generated by the G image generating unit 70 corresponding to the pixel for each pixel of the image whose difference value is complemented by the difference complementing unit 92. Thus, an image of the target color (R) is generated.
That is, as shown in the following formula (7), the adding unit 93 adds the G image to the difference value Δ (i, j) of the position of the pixel i row j column of the Bayer array image generated by the difference complementing unit 92. By adding the pixel values G (i, j) at the same pixel position generated by the generating unit 70, the pixel values R (i, j) are calculated, and an R image is generated.

The adding means 93 outputs, to the RGB image generating means 100, an R image obtained by calculating for each pixel of the Bayer array image and complementing the R pixel value according to the equation (7).
As described above, the R image generation unit 90 uses the difference unit 91 to determine, from the pixel value at the R pixel position, the G pixel value in consideration of the direction corresponding to the vibration component corresponding to the pixel position and the edge component for each band. Therefore, the difference value also becomes a value corresponding to the azimuth correlation of the vibration component and the edge component. Therefore, the difference complementing unit 92 also determines the (RG) difference value of the pixel position estimated by the complementation from the surrounding (RG) difference value at the pixel position other than the R pixel position. And a value corresponding to the edge component.
Then, the R image generation unit 90 adds the G pixel value in which the azimuth correlation and the edge component are similarly considered to the addition value 93 to the (R−G) difference value in which the azimuth correlation and the edge component are considered. Thus, an R image composed of R pixel values corresponding to the azimuth correlation of the vibration component and the edge component is generated.

  The B pixel extracting unit 80B and the B image generating unit (other color complementing unit) 90B only change the target color to be processed from R to B with respect to the R pixel extracting unit 80 and the R image generating unit 90. Since the configuration and processing contents are the same, description thereof is omitted.

The RGB image generation unit 100 combines the G image generated by the G image generation unit 70, the R image generated by the R image generation unit 90, and the B image generated by the B image generation unit 90B to generate RGB. An image is generated.
That is, the RGB image generation unit 100 generates a color image using each pixel value corresponding to the same pixel position of each of the R, G, and B images as a color pixel value at the pixel position. Note that the data format of the RGB (color) image may be a general image format, and thus the description thereof is omitted here.

  By configuring the color information complementing apparatus 1 as described above, the color information complementing apparatus 1 targets all colors obtained as captured images (Bayer array images) by the orientation-specific vibration component extracting means 10. The vibration component can be extracted. For this reason, the color information complementing apparatus 1 uses the azimuth correlation strength calculation unit 20 to effectively use all the color components and obtain the azimuth correlation of the entire image more accurately than when obtaining the azimuth correlation from a single color component. You can often ask.

  In addition, the color information complementing apparatus 1 can obtain a plurality of vibration components having different spatial frequency scales in the horizontal direction and the vertical direction by the azimuth correlation strength calculating unit 20. Therefore, the color information complementing apparatus 1 can obtain the azimuth correlation at a plurality of resolutions of different captured images by the azimuth correlation strength calculation means 20, and can obtain a high-precision azimuth correlation that is resistant to noise and has few errors. it can.

  Further, the color information complementing apparatus 1 can detect the flat portion and the edge portion of the image by extracting the edge component by the edge component extracting means 40 by direction. Therefore, the color information complementing apparatus 1 can add the high-frequency component at the edge portion of the image without adding an unnecessary high-frequency component at the flat portion of the image by the complementary image pixel value calculating unit 60, and the flat portion of the image. It is possible to reproduce a high spatial frequency at the edge portion while suppressing an increase in noise at the edge portion.

As described above, the color information complementing apparatus 1 supplements the color information by the highly accurate azimuth correlation and the edge component, so that generation of false colors can be suppressed and the spatial frequency component can be accurately reproduced. A natural color image can be generated.
The color information complementing apparatus 1 can operate a computer (not shown) with a program (color information complementing program) for causing each computer to function as each unit configured as described above.

[Operation of color information complementer]
Next, the operation of the color information complementing apparatus 1 according to the embodiment of the present invention will be described with reference to FIG. 12 (refer to FIG. 1, FIG. 2, FIG. 5, FIG. 7, FIG. 9 for the configuration as appropriate).

(Directional correlation [azimuth nonuniformity] estimation operation)
First, the color information complementing apparatus 1 uses a plurality of filters with different spatial frequency scales for each captured image (Bayer array image) for each pixel in the horizontal direction and the vertical direction for each pixel. Are extracted (step S1).

In other words, the color information complementing apparatus 1 uses the horizontal vibration component extraction means 11 (11 ₁ , 11 ₂ ) to extract image signals (horizontal vibration component images H ₁ , H ₂₎ that have been extracted with the filters having different spatial frequency scales. ) Is generated.
In addition, the color information complementing apparatus 1 uses the vertical vibration component extraction means 12 (12 ₁ , 12 ₂ ) to extract image signals (vertical vibration component images V ₁ , V ₂ ) with the filters having different spatial frequency scales. ) Is generated.

Then, the color information complementing apparatus 1 calculates the azimuth nonuniformity from the plurality of vibration components for each pixel in the horizontal direction and the vertical direction generated in step S1 by the azimuth correlation strength calculation unit 20 according to the equation (1). The parameter λ is calculated (step S2).
Thereby, the color information complementing apparatus 1 can estimate the strength of the azimuth correlation for each pixel from the color information of the entire captured image.

(High-frequency component weight calculation operation)
Next, the color information complementing apparatus 1 extracts edge components for each direction of the captured image (Bayer array image) for each pixel by the edge component extraction unit 40 for each direction (step S3).
That is, the color information complementing apparatus 1 includes an E-direction edge component extraction unit 41, an NE-direction edge component extraction unit 42, an N-direction edge component extraction unit 43, an NW-direction edge component extraction unit 44, a W The direction edge component extraction means 45, the SW direction edge component extraction means 46, the S direction edge component extraction means 47, and the SE direction edge component extraction means 48 extract eight direction edge components for each pixel.

Then, the color information complementing apparatus 1 uses the high frequency component addition coefficient calculation means 50 to determine the high frequency component weight in the pixel based on the size (maximum value) of the edge component for each pixel extracted in step S3. A component addition coefficient w is calculated (step S4).
That is, the color information complementing apparatus 1 detects the maximum value of the edge component in the eight directions by the maximum value detection unit 51 of the high frequency component addition coefficient calculation unit 50, and the coefficient conversion unit 52 detects 0 or more according to the maximum value. A high frequency component addition coefficient w in a range of 1 or less is calculated.

(Specific color [G] pixel interpolation operation)
Next, the color information complementing device 1 converts the pixel value of the pixel of the specific color (G) in the captured image (Bayer array image) into the orientation (horizontal / vertical direction) and band by the image complementing unit 30 by direction / band. Supplementation is performed for each (low frequency component / high frequency component) (step S5).

That is, the color information complementing device 1 uses the horizontal low-frequency component complementing means 31 to set the G pixel value at the R and B pixel positions to be complemented for G as a specific color to the other G in the horizontal direction. By smoothing and complementing from the pixel value, a horizontal low-frequency component complemented image _GHL is generated.
Further, the color information complementing apparatus 1 calculates a high-frequency component at the R and B pixel positions from the pixel value of the pixel and the pixel value of the pixel located two pixels away in the horizontal direction by the horizontal high-frequency component complementing unit 32. Then, by complementing the G pixel values at the R and B pixel positions, the horizontal high-frequency component complemented image _GHH is generated.

Further, the color information complementing device 1 uses the vertical low-frequency component complementing unit 33 to set the G pixel value at the R and B pixel positions to be complemented for G as a specific color to the other G in the vertical direction. By smoothing and complementing from pixel values, a vertical low-frequency component complement image _GVL is generated.
Further, the color information complementing device 1 calculates the high frequency component at the R and B pixel positions from the pixel value of the pixel and the pixel value of the pixel located two pixels away in the vertical direction by the vertical high frequency component complementing unit 34. Then, by complementing the G pixel values at the R and B pixel positions, the vertical high-frequency component complement image G _VH is generated.

  As a result, the color information complementing apparatus 1 for each pixel to be complemented for G has a pixel value when the correlation of the G pixel is strongest in the horizontal direction and a correction value (pixel) of the high frequency component when the correlation is strongest in the horizontal direction. Value), the pixel value when the correlation is strongest in the vertical direction, and the correction value (pixel value) of the high frequency component when the correlation is strongest in the vertical direction.

Then, the color information complementing apparatus 1, the complementary image pixel value calculating means 60, the complementary image _G HL generated in step _S5, the G _{HH, G VL} and _{G VH,} for each complement target pixel, the equation (4 ) To add the pixel values according to the strength of the azimuth correlation (azimuth nonuniformity parameter λ) calculated in step S2 and the high-frequency component addition coefficient w calculated in step S4, so that the pixel value of the complementary image Is calculated (step S6).

  Then, the color information complementing apparatus 1 uses the G image generation unit 70 to replace the pixel value of the complementation target pixel for G with the pixel value of the complementation image calculated in step S6 in the captured image (Bayer array image). A G image is generated (step S7).

  Accordingly, the color information complementing apparatus 1 uses the pixel value of the G pixel captured as the Bayer array image for the pixel at the G pixel position of the Bayer array, and the direction correlation strength and high frequency for the complement target pixel for G. By using the G pixel value obtained by the component addition coefficient, a G image in which the spatial frequency component is accurately reproduced can be generated.

(Target color [R, B] pixel interpolation operation)
Next, the color information complementing apparatus 1 generates an R image and a B image from the captured image using R and B as target colors by the following operation. In addition, since the complementary operation of the R pixel and the B pixel is the same process except for the target color, in the following description, the complementary operation of the R pixel will be mainly described, and contents corresponding to the B pixel will be appropriately described. Corresponding colors and means are shown in parentheses.

  First, the color information complementing apparatus 1 extracts the pixel value of the pixel corresponding to the pixel position of R (B) in the captured image (Bayer array image) by the R pixel extraction unit 80 (B pixel extraction unit 80B) ( Step S8).

  Then, the color information complementing apparatus 1 uses the difference value 91 of the R image generation means 90 (B image generation means 90B) for each R (B) pixel extracted in step S8, A difference value from the pixel value of the G pixel corresponding to the pixel calculated in S6 is calculated (step S9).

  Then, the color information complementing apparatus 1 uses the difference complementing unit 92 of the R image generating unit 90 (B image generating unit 90B) to calculate the neighboring pixels of the same color whose difference values have already been calculated in the pixel to be complemented for R (B). From the difference value, the difference value of the pixel to be complemented is complemented (step S10), as shown in the examples of the equations (5) and (6).

Further, the color information complementing apparatus 1 performs, for each pixel of the image in which the difference value is complemented in step S10 by the adding unit 93 of the R image generating unit 90 (B image generating unit 90B), in step S7 corresponding to the pixel. The pixel values of the pixels of the generated G image are added to generate an R (B) image (step S11).
As a result, the color information complementing apparatus 1 generates the R (B) image using the pixel value of the G pixel corresponding to the strength of the azimuth correlation and the high frequency component addition coefficient, thereby accurately determining the spatial frequency component. A reproduced R (B) image can be generated.

(RBG image generation operation)
Then, the color information complementing apparatus 1 combines the G image generated in step S7 with the R image and B image generated in step S11 by the RGB image generation unit 100 to generate an RGB image (step S12). ).

With the above operation, the color information complementing apparatus 1 can complement the missing color information with high accuracy at each pixel position of the captured image.
As described above, the color information complementing device 1 finally adds each complemented image complemented based on the local properties (directionality and edge component) of the captured image by weighting and adding the weights of the local properties. Since the complement value is determined, it is possible to reproduce a high spatial frequency at the edge portion of the image and generate a natural color image with few false colors while suppressing the occurrence of visually noticeable distortion in the flat portion of the image. Can do.

The configuration and operation of the color information complementing apparatus 1 according to the embodiment of the present invention have been described above, but the present invention is not limited to this embodiment.
For example, here, a captured image captured by a color filter having a color arrangement of three primary colors of RGB has been described as an example, but the color filter is not limited to these three primary colors, and other three colors. It doesn't matter.

Here, a plurality of (11 ₁ , 11 ₂ ) horizontal vibration component extracting means 11 are provided and operated in parallel here, but one horizontal vibration component extracting means 11 sequentially serially in Bayer array image units. It is good also as extracting a vibration component using a different spatial frequency scale. The same applies to the vertical vibration component extraction means 12.

In order to realize high-speed processing, it is preferable that a plurality of horizontal vibration component extraction means 11 and a plurality of vertical vibration component extraction means 12 are provided and operated in parallel.
In order to realize high-speed processing, E direction edge component extraction means 41, NE direction edge component extraction means 42, N direction edge component extraction means 43, NW direction edge component extraction means 44, W direction edge component extraction means 45, It is preferable to provide a plurality of SW direction edge component extraction means 46, S direction edge component extraction means 47, and SE direction edge component extraction means 48, respectively, for parallel operation.

[Supplemental explanation]
As described above, the color information complementing apparatus 1 uses the four types of complementary images (G _HL , G _HH , G _VL, and G _VH ) for each azimuth and each band as azimuth correlation strength (azimuth nonuniformity parameter λ ) And the size of the edge component (high frequency component addition coefficient w), the G image is complemented by weighted addition. Here, the horizontal high-frequency component complementing means 32 and the vertical high-frequency component complementing means 34 of the azimuth / band-specific image complementing means 30 use pixel values of other colors (for example, R) in order to generate a G complementary image. To complement.
Therefore, finally, referring to FIG. 13 (refer to FIG. 1 as appropriate), the technical meaning of complementing the G pixel value with other colors (R, B) will be described, and the orientation nonuniformity parameter The technical meaning of λ and the high frequency component addition coefficient w will be described.

Here, as shown in FIG. 13A, the R pixel position of the Bayer array image is set to the position of the pixel i row j column, and attention is paid to the horizontal five pixels for which the pixel position is to be complemented. Note that L _{(i, j)} means a signal level of i rows and j columns.
In the complementary processing of only the G signal, it is impossible to restore a frequency component higher than the G sampling frequency of the Bayer array. For this reason, a correction value α of a high frequency component higher than the sampling frequency of the G signal is required for the complementary value (hereinafter referred to as G value) of the G signal at the pixel position (i, j).
Here, the G value can be expressed by the following equation (8).

Here, if the true value of the G value at the pixel position (i, j) (unknown in the Bayer array image) is LG _{(i, j)} , the correction value α is ideally expressed by the following equation (9). Is done.

However, since the true value L _{G (i, j)} of the G value is unknown, it cannot be used.
Therefore, as shown in FIG. 13B, assuming that there is a correlation between the color signals (here, between the G signal and the R signal), as shown in the following equation (10), the Bayer array The correction value α is approximated using the R signal of the image.

  Here, β is a change parameter (gain parameter) between the G signal high-frequency component and the R signal high-frequency component, and is described in the document “JF Hamilton Jr. , 5629734. ”, and“ H. Malvar, L. He, R. Cutler, High-quality linear interpolation for demosaicing of Bayer-patterned color images, Proceedings of ICASSP III (2004) 485-488. ” Β = 1/2 is used.

However, there are two problems when the correction value α shown in Expression (10) is used.
The first problem is that, since the high frequency component is small in the flat portion of the image, the noise component of the image is dominant in the correction value α, and the image quality may deteriorate.
Usually, the signal level L _{(i, j)} of the captured image is the sum of the signal level L ′ _{(i, j)} not including noise and the noise level N _{(i, j)} as shown in the following equation (11). Represented as:

  When this equation (11) is substituted into the equation (10) when β is ½, the following equation (12) is obtained.

The first term of the equation (12) is a high frequency component calculated from a signal not including noise, and the second term is a noise component.
As described above, in the case of a flat image including a noise component, the correction value α is dominated by the noise component of the second term and is added to restore the G value as in the equation (8). The image quality will be degraded.

As a second problem, when a steep edge portion exists between a pixel adjacent to one pixel and a pixel adjacent to two pixels at a pixel position (i, j) that is a pixel to be complemented, Since the approximate expression 10) does not hold, the image quality may deteriorate.
For example, when a sharp vertical edge portion exists between the (j + 1) column and the (j + 2) column, L _{(i, j-1)} and L _{(i, j + 1)} have equivalent signal levels. , L _{(i, j−2)} and L _{(i, j)} and L _{(i, j + 2)} have very different signal levels. Therefore, the equation (9) cannot be approximated by the equation (10).
As described above, when obtaining the complementary value of G at the R (B is the same) pixel position, if the edge component existing between the pixel adjacent to one pixel and the pixel adjacent to two pixels is large, the equation (8) On the contrary, the image may be deteriorated by adding the correction value α.

  On the other hand, when there is a steep edge portion between the complement target pixel (i, j) and the pixel adjacent to one pixel, the approximate expression of the expression (10) is established. Therefore, the correction value α in the equation (8) is added according to the edge component existing between the complement target pixel and the pixel adjacent to one pixel.

However, when there is an edge in the vertical direction as described above, the complementary value of G is not obtained from the pixel value of the pixel in the horizontal direction, but the vertical direction with respect to the pixel position (i, j) that is the pixel to be complemented. What is necessary is just to obtain | require a complement value from the pixel value of the pixel position (i-2, j) of (i + 2, j).
Therefore, the color information complementing apparatus 1 obtains an orientation non-uniformity parameter λ indicating orientation correlation in the orientation correlation strength calculation means 20 and complements with a horizontal component in accordance with the value of λ in the complement image pixel value calculation means 60. The added image and the image complemented in the vertical direction are weighted and added.

However, since the azimuth correlation strength calculation means 20 obtains only the correlation in the horizontal direction and the vertical direction, the case where the edge component exists in the oblique direction is not taken into consideration.
Therefore, the color information complementing apparatus 1 further obtains the high frequency component addition coefficient w according to the size of the edge component for each direction (eight directions) by the high frequency component addition coefficient calculation means 50, and the complementary image pixel value calculation means 60. In FIG. 5, only the complementary images ( _GHH and _GVH ) supplemented by the high-pass filter are multiplied by the high-frequency component addition coefficient w among the complementary images.
As a result, the color information complementing device 1 suppresses image degradation at an oblique edge portion or a flat portion of the image with respect to the complementary images (G _HL and G _VL ) supplemented by the low-pass filter while suppressing high-frequency. The component can be corrected.

DESCRIPTION OF SYMBOLS 1 Color information complementing device 10 Direction-specific vibration component extraction means 11 Horizontal vibration component extraction means 12 Vertical vibration component extraction means 20 Direction correlation strength calculation means 30 Direction / band-specific image complement means 31 Horizontal low-frequency component complement means 32 Horizontal high-frequency component complement Means 33 Vertical low-frequency component complementing means 34 Vertical high-frequency component complementing means 40 Edge-specific edge component extraction means 50 High-frequency component addition coefficient calculating means 60 Supplemental image pixel value calculating means 70 G image generating means (specific color complementing means)
80 R pixel extracting means 80B B pixel extracting means 90 R image generating means (other color complementing means)
90B B image generating means (other color complementing means)
91 Difference means 92 Difference complement means 93 Addition means 100 RGB image generation means

Claims (7)

A color information complementing device that complements missing color information for each color from a captured image captured by a single-plate imaging device using a color filter,
A vibration component extracting means for each orientation for extracting a vibration component for each orientation of the horizontal direction and the vertical direction for each pixel of the captured image;
Based on the correlation of the vibration component for each direction extracted by this vibration component extraction unit for each direction, the direction correlation strength calculation unit for calculating the strength of the direction correlation for each pixel;
Complement the pixel value of the pixel to be complemented for a specific color of the captured image for each azimuth in the horizontal direction and the vertical direction for each band of the low frequency component and the high frequency component, and generate an image for each azimuth / band. Means,
Edge component extraction means for each orientation that extracts edge components for each orientation of the horizontal direction, vertical direction and oblique direction for each pixel of the captured image;
High-frequency component addition coefficient calculating means for calculating a high-frequency component addition coefficient for correcting the high-frequency component of the azimuth / band-specific image based on the size of the edge component extracted by the orientation-specific edge component extraction means;
The pixel value of the azimuth / band image supplemented by the azimuth / band image complementing unit is added according to the strength of the azimuth correlation and the high-frequency component addition coefficient, and the pixel value of the complement target pixel is calculated. A complementary image pixel value calculating means for calculating;
In the captured image, a specific color complementing unit that replaces the pixel value of the complement target pixel for the specific color with the pixel value calculated by the complementary image pixel value calculating unit, and complements the color information of the specific color;
Other color complementing means for each target color that complements color information for colors other than the specific color from the captured image,
The other color complementing means includes:
In the captured image, for each pixel of the target color, a difference unit that calculates a difference value between a pixel value of the pixel and a pixel value of the complementary image corresponding to the pixel;
Difference complementing means for complementing the difference value of the complement target pixel from the difference value of the surrounding pixels of the complement target pixel for the target color;
For each pixel of the image in which the difference value is complemented by the difference complementing unit, an adding unit that adds the pixel value of the complementary image corresponding to the pixel to generate an image of the target color;
A color information complementing device comprising: The direction-specific vibration component extracting means includes:
Horizontal vibration component extraction means for extracting a horizontal vibration component by performing convolution processing on the captured image in the horizontal direction using a linear filter;
Vertical vibration component extraction means for extracting a vertical vibration component by performing convolution processing on the captured image in the vertical direction using a linear filter;
The color information complementing device according to claim 1, further comprising: For each of the linear filters having different spatial frequency scales for the convolution process in the horizontal direction, a plurality of the horizontal vibration component extraction means are provided,
3. The color information complementing apparatus according to claim 2, wherein the number of the vertical vibration component extraction units is the same as the number of the horizontal vibration component extraction units for each linear filter having a different spatial frequency scale for the convolution process in the vertical direction. .   The orientation-specific edge component extraction unit is configured to perform the edge component for each of the pixels of the captured image according to the eight directions of the right direction, the upper right direction, the upper direction, the upper left direction, the left direction, the lower left direction, the lower direction, and the lower right direction. The color information complementing device according to claim 1, wherein the color information complementing device is extracted. The azimuth / band-specific image complementing means is:
Horizontal low-frequency component complementing means for generating a horizontal low-frequency component-complemented image by averaging pixel values adjacent in the horizontal direction of the complement target pixel;
Horizontal high frequency component complementing means for calculating a high frequency component from a pixel value of the pixel to be complemented and a pixel value of a pixel located two pixels away in the horizontal direction of the pixel to generate a horizontal high frequency component complement image,
Vertical low frequency component complementing means for generating a vertical low frequency component complemented image by averaging pixel values adjacent to each other in the vertical direction of the complement target pixels;
Vertical high frequency component complementing means for calculating a high frequency component from a pixel value of the pixel to be complemented and a pixel value of a pixel located two pixels away in the vertical direction of the pixel to generate a vertical high frequency component complementary image;
The color information complementing device according to any one of claims 1 to 4, further comprising:   2. The captured image according to claim 1, wherein the captured image is an image captured using a Bayer array color filter of three primary colors, wherein the specific color is green, and the target colors are red and blue. The color information complementing device according to claim 5.   A color information complementing program for causing a computer to function as the color information complementing apparatus according to any one of claims 1 to 6.