JP4585045B1 - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

When a color image is generated from a color mosaic image obtained using a single-plate image sensor, a processing load is reduced and a high-quality color image is generated.
A low-frequency image generation unit that generates GrL, GbL, RL, and BL signals in a low-frequency band from a color mosaic image; an intermediate-frequency image generation unit that generates GrM and GbM signals in a medium-frequency band; By subtracting the average value of GrL and GbL from each of BL and RL, the sum of the low frequency color difference signal generating means 14, BL, GrL, GbL, RL for generating the first color difference UL and the second color difference VL is obtained. Low frequency luminance signal generating means 15 for generating luminance signal YL, medium frequency luminance signal generating means 16 for generating medium frequency luminance signal YM by the sum of GrM and GbM, YL and YM are synthesized, and luminance signal Y is A luminance synthesis unit 18 to generate, a color space conversion unit 19 to generate color information for each pixel using Y, UL, and VL, and the like are provided.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program of a digital imaging apparatus using a solid-state imaging device.

  2. Description of the Related Art Conventionally, in an imaging apparatus such as a digital camera, a subject image is formed on a solid-state imaging device via a lens, and this solid-state imaging device (for example, a CCD: Charged Coupled Device or a CMOS: Complementary Metal-Oxide Semiconductor). An image processing apparatus and an image processing method are known in which a subject image is photoelectrically converted by an imaging device to generate an image signal.

  As a single-plate image sensor, a plurality of photoelectric conversion elements are provided in a matrix and a color filter is provided in front of the image sensor, and image processing is performed by applying signal processing to the pixel signals of each color output through the color filter. An image processing apparatus and an image processing method to be generated are known.

  In a so-called single-plate image sensor using only one image sensor, specific color filters corresponding to each pixel on the image sensor, such as red (R), green (G), and blue (B) colors, are used. A filter is provided. In the present application, a pixel provided with an R color filter is an R pixel (red pixel), a pixel provided with a G color filter is G pixel (green pixel), and a pixel provided with a B color filter is B pixel ( Blue pixels).

  A color filter called a primary color Bayer array is used as a representative color filter in a single-plate image sensor. The color filter of the primary color Bayer arrangement is such that rows in which R and G are alternately arranged in the horizontal direction and rows in which G and B are alternately arranged in the horizontal direction are alternately arranged in the vertical direction. A plurality of basic blocks each having two vertical pixels (2 × 2 pixels) as one unit (one unit) are periodically arranged. Each basic block has a structure in which two green pixels are arranged on one diagonal and a red pixel and a blue pixel are arranged on the other diagonal. In the present application, in the horizontal direction, G located next to R is referred to as Gr, and G located next to B is referred to as Gb.

  In some image processing apparatuses, a luminance signal and a color difference signal are generated from an image signal output via a single-plate image sensor, and an image signal of a color image is generated from these signals.

  Specifically, in a single-plate image sensor, each pixel has only single color information, but in order to display a color image, all the values of red (R), green (G), and blue (B) are each pixel. It is necessary. For this reason, in image processing using a single-plate image sensor, demosaic processing is performed based on a color mosaic image in which each pixel has only one of R, G, and B components. Here, the demosaic process is an interpolation operation using the luminance information of other insufficient colors collected from the surrounding pixels for the single color information of each pixel of the color mosaic image, so that each pixel is R, This is processing for generating a color image having all of the G and B components (so-called color interpolation processing) (see, for example, Patent Documents 1, 2, 3, 4, and 5).

  Furthermore, as a demosaicing method, a luminance image signal having luminance information for each pixel is generated from the imaging signal, and the change state of the image signal is detected by using the luminance image signal, and the detection result is reflected in the demosaic image. By doing so, an image processing method for increasing the resolution is known (see, for example, Patent Document 6).

  Further, in general, an image processing method proposed for the purpose of high image quality determines the correlation between horizontal and vertical directions by determining the behavior of adjacent pixels adjacent to the target pixel (the state of change in the image signal). Similarity) is calculated, and the signal components generated by the horizontal component extraction filter and the vertical component extraction filter are switched or combined according to the calculation result to interpolate the color signal components.

JP-T-2004-534429 JP 2000-270294 A JP-A-9-84031 Japanese Patent Laid-Open No. 10-108209 JP 11-275373 A JP 2003-87810 A

In recent years, the processing load required for image processing has increased with the increase in resolution of still images and the increase in definition of HD (High Definition) moving images.
That is, when the above-described demosaic processing is realized by hardware, there is a problem that the circuit scale increases as the processing amount increases, leading to an increase in cost and impairing productivity. Further, when generating a high-definition HD moving image, there is a possibility that the power consumption increases and the battery driving time decreases, or the temperature rises due to heat generation and the reliability decreases.

  In addition, when the demosaic processing described above is implemented using software in a DSP (Digital Signal Processor), the processing capacity required of the DSP increases as the amount of computation increases, and high-definition HD video is generated. When handling, there was a possibility that processing capacity might be insufficient.

  Accordingly, the present invention provides an image processing apparatus and an image processing method capable of reducing a processing load and generating a high-quality color image when generating a color image from a color mosaic image obtained using a single-plate image sensor. An object of the present invention is to provide an image processing program. Furthermore, an image processing apparatus, an image processing method, and an image processing program that can reduce image quality degradation such as false colors and zipper artifacts in demosaic processing are provided.

  The invention according to claim 1, which has been made to achieve this object, is output via an image sensor having a Bayer array color filter of R (red), G (green), and B (blue), and each pixel is output. An image processing apparatus for generating a color image in which each pixel has color information of a plurality of colors using a color mosaic image having color information of a single color, wherein G pixels positioned beside R in the Bayer array When the pixel of G located next to Gr and B is represented as Gb, the color mosaic image is filtered by applying a low-pass filter, and the R signal, Gr signal, Gb signal, and B of the low-frequency band are filtered. A low-frequency image generating means for generating a signal, a medium-frequency image generating means for generating a Gr signal and a Gb signal in a medium frequency band by filtering the color mosaic image by applying a band filter, and the low-frequency image generation hand An average value of the Gr signal and the Gb signal is obtained using the low-frequency signal obtained through the signal, and the first color difference signal is generated by subtracting the average value from the B signal, and the average value is calculated from the R signal. The low frequency color difference signal generating means for generating the second color difference signal by subtracting the low frequency luminance signal by obtaining the sum of the R signal, Gr signal, Gb signal and B signal obtained by the low frequency image generating means Medium frequency luminance signal generation means for obtaining a sum of the Gr signal and Gb signal obtained by the low frequency luminance signal generating means for generating the medium frequency image and the medium frequency image generating means, and generating a medium frequency luminance signal in association with the sum Means for synthesizing the low frequency luminance signal and the medium frequency luminance signal to generate a luminance signal Y, and the luminance signal generated by the luminance synthesizing means and the low frequency chrominance signal generating means. Generated in the first By using the color difference signals and second color difference signals, characterized in that it comprises a color space conversion means for generating color information for each pixel of the color image.

  According to the image processing apparatus of claim 1, low frequency image generation means for generating R, Gr, Gb, and B signals in the low frequency band from the color mosaic image, and Gr signals and Gb signals in the medium frequency band are generated. Medium frequency image generating means for generating a first color difference signal and a second color difference signal by subtracting the average value of the low frequency bands Gr and Gb from the B signal and the R signal in the low frequency band, respectively. Generating means, low frequency luminance signal generating means for generating a low frequency luminance signal by calculating the sum of R, Gr, Gb, B in the low frequency band, generating a medium frequency luminance signal by summing Gr and Gb in the medium frequency band The medium frequency luminance signal generating means, the low frequency luminance signal and the medium frequency luminance signal are combined to generate a luminance signal, and the luminance signal generated by the luminance combining means and the low frequency color difference signal generating means are generated. Brightness Color space conversion means for generating color information for each pixel using the signal, the first color difference signal, and the second color difference signal, etc., so that a color can be obtained from the color mosaic image obtained via the single-plate image sensor. When an image is generated, the processing load can be reduced, a high-quality color image can be generated, and further, image quality degradation that occurs during color interpolation such as false colors and zipper artifacts can be reduced.

  According to the image processing apparatus of the first aspect, the interpolation processing related to the demosaicing process is simultaneously performed in the low-frequency image generation unit and the medium-frequency image generation unit, and the pixel signal when the luminance signal and the color difference signal are generated Since generation is performed and the demosaicing interpolation process is not necessary in the subsequent steps up to the color space conversion means, the processing load can be reduced when generating a color image from the color mosaic image. Further, in the low frequency luminance signal, the false color due to the luminance change component from the medium frequency to the high frequency is small, zipper artifacts at the color boundary are suppressed, and the R signal and the B signal are not used in the medium frequency luminance signal. Therefore, no artifacts at the color boundary occur.

  Furthermore, a luminance signal with less zipper artifacts and a color difference signal with fewer false colors can be generated by simple processing (addition / subtraction) from the interpolation signals of the respective colors generated by the low frequency image generation means and the medium frequency image generation means.

  Further, a high-resolution luminance signal can be obtained by synthesizing the low-frequency luminance signal and the medium-frequency luminance signal.

Next, the invention according to claim 2 is the image processing apparatus according to claim 1, further comprising second low-frequency color difference signal generation means instead of the low-frequency color difference signal generation means, and the second low-frequency color difference. The signal generation means has a positive (+) or negative (-) sign of a difference value obtained by subtracting the Gb signal from the Gr signal of the low frequency signal obtained via the low frequency image generation means, and B A low frequency signal Gr signal, Gb signal, B signal, and R signal obtained through the low frequency image generation means by multiplying the signal by the positive or negative sign of the difference value obtained by subtracting the R signal from the signal Is expressed as GrL, GbL, BL, RL, the first color difference signal is expressed as UL, and the second color difference signal is expressed as VL. When the multiplication result is positive (+), The first color difference signal UL is calculated using an arithmetic expression of UL = BL-GrL. The second color difference signal VL is calculated using an arithmetic expression of VL = RL−GbL, and when the result of the multiplication is negative (−), the first color difference signal UL is set to UL = BL−GbL. The calculation is performed using an arithmetic expression, and the second color difference signal VL is calculated using an arithmetic expression of VL = RL−GrL.

According to the image processing apparatus of the second aspect, it is possible to suppress the false color without impairing the saturation. There are two types of phenomena called false colors. One is a color fringe (chromatic aberration) appearing at the edge of the image, which is removed by a low frequency filter. The other is a phenomenon in which high-frequency fold-back moire (interference fringes) are different for each color against fine repetitive patterns. Appears and colors a wide area of the shooting screen. In order to suppress false colors, the amount of false colors is estimated by the absolute value of (GrL-GbL), and the first color difference signal and the second color difference signal are calculated by the absolute value of (GrL-GbL). Means for generating by subtraction are known. However, according to this method, (GrL-GbL) becomes large when the image includes a high-frequency signal in addition to the false color occurrence portion, so that the portion of the high-frequency region where no false color is generated due to suppression of color moire There is a risk of degrading the saturation.
Therefore, the inventors of the present invention have found a means for generating the first color difference and the second color difference without reducing color moire and without losing the saturation. Specifically, the direction of moire (interference fringes) is estimated in association with a positive or negative sign obtained by an arithmetic expression of (GrL−GbL) × (BL−RL), and the first is determined according to the estimation result. By switching the arithmetic expression for generating the color difference signal and the second color difference signal, the occurrence of color moire can be reduced without deteriorating the saturation.

  Next, according to a third aspect of the present invention, in the image processing apparatus according to the first or second aspect, a difference between the Gr signal and the Gb signal generated via the low frequency image generation means is obtained. The false color generation amount estimation means for generating a false color generation estimation amount signal, and the false color generation so that the absolute value approaches 0 for each of the first color difference signal and the second color difference signal. False color suppression means for adding and subtracting the estimated amount signal to generate a third color difference signal and a fourth color difference signal, and the color space conversion means replaces the first color difference signal and the second color difference signal. The color information is generated using the third color difference signal and the fourth color difference signal.

  According to the image processing apparatus of claim 3, a false color generation amount for generating a false color generation estimation amount signal by obtaining a difference between the Gr signal and the Gb signal generated through the low frequency image generation means. A false color generation estimation amount signal is added to or subtracted from the estimation means, the first color difference signal, and the second color difference signal so that the absolute value approaches 0, and a third color difference signal and a fourth color difference signal are obtained. And a color space conversion unit that generates color information using the third color difference signal and the fourth color difference signal instead of the first color difference signal and the second color difference signal. By doing so, a high-quality color image with fewer false colors can be generated.

  Next, according to a fourth aspect of the present invention, in the image processing device according to any one of the first to third aspects, when the sampling frequency is represented as Fs, the passband of the low frequency signal is 0 to 0. It is configured to be approximately (1/4) Fs.

According to the image processing apparatus according to claim 4, the passband of the low-frequency signal, 0 substantially (1/4) by being configured to be Fs, and have your to the generation of the low frequency image signal color Intrusion of high frequency components and luminance high frequency components can be suppressed, and false colors can be suppressed.

  Next, according to a fifth aspect of the present invention, in the image processing device according to any one of the first to fourth aspects, when the sampling frequency is expressed as Fs, the passband of the medium frequency signal is centered. It is approximately (1/4) Fs, and (0) Fs and (1/2) Fs are preferably configured so that the passing amount is approximately 0. Thereby, the medium frequency luminance signal and the low frequency luminance signal can be synthesized to generate a luminance signal with good resolution.

  Next, according to a sixth aspect of the present invention, in the image processing device according to any one of the first to fifth aspects, the low-frequency image generating means is configured to have the R, Gr, Gb, and B pixel positions. A low frequency signal of a color associated with each color is generated, and low frequency signals of all other colors except the associated color are generated.

  According to the image processing apparatus of the sixth aspect, the low-frequency signal of the color associated with each of the R, Gr, Gb, and B pixel positions is generated, and the associated color is excluded. By generating low-frequency signals of all the colors, a plurality of color components can be provided for each pixel.

  Next, according to a seventh aspect of the present invention, in the image processing apparatus according to any one of the first to sixth aspects, the medium frequency image generating means is configured to have the R, Gr, Gb, and B pixel positions. It is characterized by generating medium frequency signals of Gr and Gb.

  According to the image processing device of the seventh aspect, it is possible to generate the G signal in the middle frequency band for each pixel by generating the medium frequency signal of Gr and Gb at the pixel positions of Gr, Gb, R, and B. Can improve the resolution.

  Next, according to an eighth aspect of the present invention, each pixel outputs color information of a single color that is output through an image pickup device having a Bayer array color filter of R (red), G (green), and B (blue). An image processing method for generating a color image in which each pixel has color information of a plurality of colors using a color mosaic image having Gr and B pixels in the Bayer array. When the G pixel located horizontally is represented as Gb, the color mosaic image is filtered by applying a low-pass filter to generate an R signal, a Gr signal, a Gb signal, and a B signal in a low-frequency band. Obtained through a frequency image generation procedure, a medium frequency image generation procedure for filtering the color mosaic image by applying a bandpass filter to generate a Gr signal and a Gb signal in the middle frequency band, and the low frequency image generation procedure Low frequency signal The first color difference signal is generated by subtracting the average value from the B signal, and the second color difference signal is obtained by subtracting the average value from the R signal. A low-frequency luminance signal generation procedure for generating a low-frequency luminance signal by obtaining a sum of the R signal, Gr signal, Gb signal, and B signal obtained in the low-frequency image generation procedure A medium frequency luminance signal generating procedure for obtaining a medium frequency luminance signal in association with the sum, and obtaining the sum of the Gr signal and Gb signal obtained by the medium frequency image generating procedure, and the low frequency luminance signal And the medium frequency luminance signal to generate a luminance signal; a luminance signal generated by the luminance combining procedure; a first color difference signal generated by the low frequency color difference signal generating procedure; With the second color difference signal Used, characterized by using a color space conversion procedure for generating color information for each pixel of the color image.

  According to the image processing method of the eighth aspect, similarly to the first aspect of the invention, the processing load is reduced when generating a color image from the color mosaic image obtained through the single-plate image sensor. In addition, a high-quality color image can be generated, and further, image quality degradation caused by color interpolation such as false colors and zipper artifacts can be reduced.

Next, the invention according to claim 9 is the image processing method according to claim 8, wherein the second low-frequency color difference signal generation procedure is used instead of the low-frequency color difference signal generation procedure. The signal generation procedure includes a positive (+) or a negative (−) sign of a difference value obtained by subtracting the Gb signal from the Gr signal of the low frequency signal obtained through the low frequency image generation procedure, and B A low frequency signal Gr signal, a Gb signal, a B signal, and an R signal obtained through the low frequency image generation procedure are multiplied by a positive or negative sign of a difference value obtained by subtracting the R signal from the signal. Is expressed as GrL, GbL, BL, RL, the first color difference signal is expressed as UL, and the second color difference signal is expressed as VL. When the multiplication result is positive (+), The first color difference signal UL is calculated using an arithmetic expression of UL = BL-GrL. The second color difference signal VL is calculated using an arithmetic expression of VL = RL−GbL, and when the result of the multiplication is negative (−), the first color difference signal UL is set to UL = BL−GbL. The calculation is performed using an arithmetic expression, and the second color difference signal VL is calculated using an arithmetic expression of VL = RL−GrL.

  According to the image processing method of claim 9, as in the invention of claim 2, the image processing method is associated with a positive or negative sign obtained by an arithmetic expression of (GrL−GbL) × (BL−RL). Estimate the direction of moire (interference fringes) and switch the calculation formula for generating the first color difference signal and second color difference signal according to the estimation result, reducing the occurrence of color moire without losing saturation it can.

  A tenth aspect of the present invention is the image processing method according to the eighth or ninth aspect, wherein a difference between the Gr signal and the Gb signal generated through the low frequency image generation procedure is obtained. The false color generation amount estimation procedure for generating the false color generation estimation amount signal and the false color generation so that the absolute value approaches 0 for each of the first color difference signal and the second color difference signal. A false color suppression procedure that adds and subtracts an estimated amount signal to generate a third color difference signal and a fourth color difference signal, and the color space conversion procedure uses the first color difference signal and the second color difference signal. Instead, the color information is generated using the third color difference signal and the fourth color difference signal.

  According to the image processing method of the tenth aspect, similarly to the third aspect of the invention, a high-quality color image with fewer false colors can be generated.

  Next, according to an eleventh aspect of the present invention, in the image processing method according to any one of the eighth to tenth aspects, when the sampling frequency is expressed as Fs, the passband of the low frequency signal is 0 to 0. It is approximately (1/4) Fs.

According to the image processing method of the eleventh aspect, similarly to the fourth aspect of the invention, it is possible to suppress the intrusion of the color high frequency component and the luminance high frequency component into the low frequency image signal, thereby suppressing the false color.

  Next, according to a twelfth aspect of the present invention, in the image processing method according to any one of the eighth to eleventh aspects, when the sampling frequency is expressed as Fs, the passband of the medium frequency signal is centered. It is approximately (1/4) Fs, and (0) Fs and (1/2) Fs are characterized in that the passing amount is approximately 0.

  According to the image processing method of the twelfth aspect, similarly to the fifth aspect of the invention, it is possible to synthesize a medium frequency luminance signal and a low frequency luminance signal to generate a luminance signal with good resolution.

  Next, an invention according to claim 13 is the low-frequency image generation procedure of the image processing method according to any one of claims 8 to 12, for each of the R, Gr, Gb, and B pixel positions. A low frequency signal of a color associated with each color is generated, and low frequency signals of all other colors except the associated color are generated.

  According to the image processing method of the thirteenth aspect, similarly to the sixth aspect of the invention, a plurality of color components can be provided for each pixel.

  Next, an invention according to a fourteenth aspect is the medium frequency image generation procedure of the image processing method according to any one of the eighth to thirteenth aspects, for each pixel position of the Gr, Gb, R, and B. A medium frequency signal of Gr and Gb is generated.

  According to the image processing method of the fourteenth aspect, as in the seventh aspect of the invention, the G signal in the middle frequency band can be generated for each pixel, and the resolution can be improved.

  Next, the invention according to claim 15 is the output of an image sensor having a Bayer array color filter of three colors of R (red), G (green), and B (blue). An image processing program for generating a color image in which each pixel has color information of a plurality of colors using a color mosaic image having Gr and B pixels in the Bayer array When the G pixel located horizontally is represented as Gb, the color mosaic image is filtered by applying a low-pass filter to generate an R signal, a Gr signal, a Gb signal, and a B signal in a low-frequency band. Through a frequency image generation step, a medium frequency image generation step of filtering the color mosaic image by applying a band filter to generate a Gr signal and a Gb signal in a medium frequency band, and the low frequency image generation step Using the obtained low frequency signal, an average value of the Gr signal and the Gb signal is obtained, and the average value is subtracted from the B signal to generate a first color difference signal, and the average value is subtracted from the R signal. A low frequency color difference signal generation step for generating a second color difference signal and a sum of the R signal, Gr signal, Gb signal, and B signal obtained in the low frequency image generation step are obtained to generate a low frequency luminance signal. A medium frequency luminance signal generating step for obtaining a sum of the Gr signal and the Gb signal obtained in the low frequency luminance signal generating step and the medium frequency image generating step, and generating a medium frequency luminance signal in association with the sum; A luminance synthesis step for synthesizing the low frequency luminance signal and the medium frequency luminance signal to generate a luminance signal; a luminance signal generated in the luminance synthesis step and the low frequency color difference signal generation step; And a color space conversion step of generating color information for each pixel of the color image using the first color difference signal and the second color difference signal generated in the step. .

  According to the image processing program of the fifteenth aspect, as in the first aspect of the invention, the processing load is reduced when generating a color image from the color mosaic image obtained through the single-plate image sensor. In addition, a high-quality color image can be generated, and further, image quality degradation caused by color interpolation such as false colors and zipper artifacts can be reduced.

Next, an invention according to claim 16 is the image processing program according to claim 15, further comprising a second low frequency color difference signal generation step instead of the low frequency color difference signal generation step, and the second low frequency color difference. A signal generation step includes a positive (+) or a negative (−) sign of a difference value obtained by subtracting the Gb signal from the Gr signal of the low frequency signal obtained through the low frequency image generation step , and B A low frequency signal Gr signal, Gb signal, B signal, R signal obtained through the low frequency image generation step by multiplying the signal by the positive or negative sign of the difference value obtained by subtracting the R signal from the signal Is expressed as GrL, GbL, BL, RL, the first color difference signal is expressed as UL, and the second color difference signal is expressed as VL. When the multiplication result is positive (+), The first color difference signal UL is set to UL = BL-GrL. In addition to calculating using the arithmetic expression, the second color difference signal VL is calculated using an arithmetic expression of VL = RL−GbL, and when the multiplication result is negative (−), the first color difference signal is calculated. The signal UL is calculated using an arithmetic expression of UL = BL-GbL, and the second color difference signal VL is calculated using an arithmetic expression of VL = RL-GrL.

  According to the image processing program of the sixteenth aspect, similarly to the second aspect of the invention, the image processing program is associated with a positive or negative sign obtained by an arithmetic expression of (GrL−GbL) × (BL−RL). Estimate the direction of moire (interference fringes) and switch the calculation formula for generating the first color difference signal and second color difference signal according to the estimation result, reducing the occurrence of color moire without losing saturation it can.

Next, an invention according to claim 17 is the image processing program according to claim 15,
A false color generation amount estimation step of generating a false color generation estimation amount signal by obtaining a difference between the Gr signal and the Gb signal generated through the low frequency image generation step, the first color difference signal, and the A false color suppression step of adding and subtracting the false color generation estimation amount signal so that the absolute value approaches 0 for each of the second color difference signals, and generating a third color difference signal and a fourth color difference signal; And the color space conversion step generates the color information using the third color difference signal and the fourth color difference signal instead of the first color difference signal and the second color difference signal. To do.

  According to the image processing program of the seventeenth aspect, similarly to the third aspect of the invention, a high-quality color image with fewer false colors can be generated.

  Next, according to an eighteenth aspect of the present invention, in the image processing program according to any one of the fifteenth to seventeenth aspects, when the sampling frequency is expressed as Fs, the passband of the low frequency signal is 0. It is about (1/4) Fs.

According to the image processing program of the eighteenth aspect, similarly to the fourth aspect of the invention, the intrusion of the color high frequency component and the luminance high frequency component can be suppressed in the low frequency image signal, and the false color can be suppressed.

  Next, the invention according to claim 19 is the image processing program according to any one of claims 15 to 18, wherein when the sampling frequency is expressed as Fs, the passband of the medium frequency signal is centered. It is approximately (1/4) Fs, and (0) Fs and (1/2) Fs are characterized in that the passing amount is approximately 0.

  According to the image processing program of the nineteenth aspect, similar to the fifth aspect, the medium frequency luminance signal and the low frequency luminance signal can be synthesized to generate a luminance signal with good resolution.

  Next, an invention according to a twentieth aspect is the low frequency image generation step of the image processing program according to any one of the fifteenth to nineteenth aspects, for each of the R, Gr, Gb, and B pixel positions. A low frequency signal of a color associated with each color is generated, and low frequency signals of all other colors except the associated color are generated.

  According to the image processing program of the twentieth aspect, similarly to the sixth aspect of the invention, a plurality of color components can be provided for each pixel.

  Next, the invention according to claim 21 is the medium frequency image generation step of the image processing program according to any one of claims 15 to 20, for each pixel position of the Gr, Gb, R, B. A medium frequency signal of Gr and Gb is generated.

  According to the image processing program of the twenty-first aspect, similarly to the seventh aspect of the invention, the G signal in the middle frequency band can be generated for each pixel and the resolution can be improved.

  An image processing apparatus, an image processing method, and an image processing program according to the present invention provide a low-frequency image generation unit and a medium-frequency image generation unit that simultaneously perform interpolation processing related to demosaic processing and generate a luminance signal and a color difference signal. Since the pixel signal is generated and no interpolation processing is required in the subsequent steps up to the color space conversion means, the processing load when generating a color image from a color mosaic image can be reduced. In addition, the low frequency luminance signal has few false colors due to the luminance change component from the medium frequency to the high frequency, suppresses zipper artifacts at the color boundary, and the medium frequency luminance signal does not use the R signal and the B signal. As a result, artifacts at the color boundary do not occur.

  In addition, the image processing apparatus, the image processing method, and the image processing program according to the present invention have a moire (interference fringe) pattern associated with a positive or negative sign obtained by an arithmetic expression of (GrL−GbL) × (BL−RL). By estimating the direction and switching the arithmetic expressions for generating the first color difference signal and the second color difference signal according to the estimation result, the occurrence of color moire can be reduced without deteriorating the saturation.

  Furthermore, a luminance signal with less zipper artifacts and a color difference signal with fewer false colors can be generated by simple processing (addition / subtraction) from the interpolation signals of the respective colors generated by the low frequency image generation means and the medium frequency image generation means. Then, by synthesizing the low-frequency luminance signal and the medium-frequency luminance signal, a high-resolution luminance signal can be obtained.

  In addition, the image processing apparatus, the image processing method, and the image processing program of the present invention obtain a difference between the Gr signal and the Gb signal in the low-frequency image and generate a false color generation estimation amount signal, and the first color difference signal A false color generation estimation amount signal is added to or subtracted from each of the second color difference signals so that the absolute value approaches 0 to generate a third color difference signal and a fourth color difference signal, and then the first color difference signal By generating color information using the third color difference signal and the fourth color difference signal instead of the signal and the second color difference signal, a high-quality color image with fewer false colors can be generated.

Further, the image processing apparatus and image processing method of the present invention, the image processing program by the passband of the low-frequency signal is 0 substantially (1/4) Fs, the low-frequency image signal color high-frequency component and intensity Intrusion of high frequency components can be suppressed, false color can be suppressed, and the pass band of the medium frequency signal is approximately (1/4) Fs at the center, and passes at (0) Fs and (1/2) Fs. When the amount is substantially 0, a medium-frequency luminance signal and a low-frequency luminance signal can be synthesized to generate a luminance signal with good resolution.

  In addition, the image processing apparatus, the image processing method, and the image processing program of the present invention generate low-frequency signals of colors corresponding to the pixel positions of Gr, Gb, R, and B, respectively, and By generating low-frequency signals of all other colors except the selected color, it is possible to provide color components of a plurality of colors for each pixel, and Gr and Gb at the pixel positions of R, Gr, Gb, and B By generating a medium frequency signal, a G signal in the medium frequency band can be generated for each pixel, and the resolution can be improved.

1 is a block diagram illustrating a configuration of an imaging device including an image processing device according to a first embodiment of the present invention. It is explanatory drawing of the low frequency image generation in the image processing apparatus of the said 1st Embodiment. FIG. 3 is an explanatory diagram of medium frequency image generation in the image processing apparatus of the first embodiment. FIG. 6 is a characteristic diagram of a low-pass filter when generating a low-frequency image in the image processing apparatus according to the first embodiment. FIG. 4 is a characteristic diagram of a bandpass filter when generating a medium frequency image in the image processing apparatus according to the first embodiment. It is a flowchart showing the procedure of the image processing method of the 1st Embodiment of this invention. It is a figure showing the structure of the 2nd low frequency color difference signal production | generation means in the image processing apparatus of the 2nd Embodiment of this invention. FIG. 10 is an explanatory diagram of second low-frequency color difference signal generation in the image processing apparatus according to the second embodiment, in which a low-frequency color difference signal UL generated by subtracting an average value of GrL and GbL from BL and RL, and It is a characteristic view of VL. FIG. 11 is an explanatory diagram of second low-frequency color difference signal generation in the image processing apparatus of the second embodiment, and includes a low-frequency color difference signal UL generated by subtracting either GrL or GbL from each of BL and RL; It is a characteristic view of VL. It is explanatory drawing of the effect of the 2nd low frequency color difference signal production | generation means in the image processing apparatus of the said 2nd Embodiment, Comprising: It is a characteristic view expressed by subtracting GbL from GrL. It is explanatory drawing of the effect of the 2nd low frequency color difference signal production | generation means in the image processing apparatus of the said 2nd Embodiment, Comprising: It is a characteristic view expressed by subtracting RL from BL.

(First embodiment)
Next, an image processing apparatus, an image processing method, and an image processing program according to the first embodiment of the present invention will be described with reference to the drawings.

  As illustrated in FIG. 1, the imaging apparatus 300 includes an imaging unit 100 that guides an optical image to the image sensor 2 and outputs a digital image signal, and a plurality of pixels for each pixel based on the digital image signal output from the imaging unit. An image processing apparatus 200 that generates color image data including the color information of the image data.

  The imaging unit 100 photoelectrically converts and amplifies the optical image formed through the imaging optical system 1 to guide the captured optical image to the image sensor, converts the optical image into a digital signal, and outputs the digital signal. An image sensor 2 is used.

  The imaging optical system 1 includes an imaging lens 1a that collects subject light and guides it to the image sensor 2, and an aperture that can be opened and closed, and adjusts the amount of incident light incident through the imaging lens 1a. 1b, etc.

The image sensor 2 has an imaging element 2a that converts an optical image formed through the imaging optical system 1 into an analog electric signal in association with the amount of received light, and a variable gain that amplifies the analog electric signal output from the imaging element 2a. An amplifier (AGC: Automatic Gain Control) 2b, an analog-to-digital converter (ADC) 2c that converts an analog electric signal output from the variable gain amplifier 2b into a digital signal, and the like are included. The imaging element 2a has a plurality of photoelectric conversion elements arranged in a matrix, and a Bayer array of three colors R (red), G (green), and B (blue) associated with the photoelectric conversion elements on the front surface thereof. A color filter (not shown) is arranged, and the amount of light that has passed through each color filter unit is converted into an electrical signal. A color mosaic image having only one color information for each pixel is generated by the electrical signal output from the imaging unit 100.

  Next, the image processing apparatus 200 filters the color mosaic image by applying a low-pass filter to the mosaic image storage unit 10 that stores the color mosaic image input via the imaging unit 100, and performs image Gr in the low-frequency band. Low-frequency image generation means 11 for generating a signal, Gb signal, R signal, and B signal, and a medium-frequency image generation means for generating a medium frequency band Gr signal and Gb signal by filtering the color mosaic image by applying a band filter 12, etc.

  The mosaic image storage means 10 is configured to store an R pixel signal, a Gr pixel signal, a Gb pixel signal, and a B pixel signal in association with the Bayer array shown in FIG. Yes.

  As shown in FIG. 2, the low-frequency image generation unit 11 performs interpolation generation by filtering the low-frequency pixel signal of the target pixel by applying a low-pass filter to surrounding pixels (5 × 5 pixel range). 2B is a coefficient arrangement diagram of the low-pass filter when the Gr low-frequency signal GrL is generated by interpolation. FIG. 2C is a low-pass filter coefficient generation diagram when the Gb low-frequency signal GbL is generated by interpolation. FIG. 2D is a coefficient arrangement diagram, and FIG. 2D is a coefficient arrangement diagram of the low-pass filter when the low frequency signal RL of R is generated by interpolation. FIG. It is a coefficient arrangement | positioning figure of a low-pass filter. At this time, a 5 × 5 filter coefficient array centered on the target pixel is used, the coefficients shown in FIGS. 2B to 2D are arranged in this filter coefficient array, and each pixel signal is weighted. Can be obtained.

  Specifically, as shown in FIG. 2B, the Gr low frequency signal GrL at the Gr pixel position is generated by applying a low-pass filter to the surrounding Gr signal and the Gr signal of the target pixel. A low frequency signal GrL at the R pixel position is generated by applying a low-pass filter to the surrounding Gr signal. A low-frequency signal GrL at the B pixel position is generated by applying a low-pass filter to the surrounding Gr signal. A low frequency signal GrL at the pixel position of Gb is generated by applying a low-pass filter to the surrounding Gr signal.

  Also, as shown in FIG. 2C, the Gb low frequency signal GbL at the Gr pixel position is generated by applying a low-pass filter to the surrounding Gb signal. A low frequency signal GbL at the R pixel position is generated by applying a low-pass filter to the surrounding Gb signal. The low-frequency signal GbL at the B pixel position is generated by applying a low-pass filter to the surrounding Gb signal. The low-frequency signal GbL at the pixel position of Gb is generated by applying a low-pass filter to the surrounding Gb signal and the Gb signal of the target pixel.

  Further, as shown in FIG. 2D, an R low frequency signal RL at the Gr pixel position is generated by applying a low pass filter to the surrounding R signal. A low-frequency signal RL at the R pixel position is generated by applying a low-pass filter to the surrounding R signal and the R signal of the target pixel. A low-frequency signal RL at the B pixel position is generated by applying a low-pass filter to the surrounding R signal. A low-frequency signal RL at the Gb pixel position is generated by applying a low-pass filter to the surrounding R signal.

Further, as shown in FIG. 2E, the B low-frequency signal BL at the Gr pixel position is generated by applying a low-pass filter to the surrounding B signal. A low-frequency signal BL at the R pixel position is generated by applying a low-pass filter to the surrounding B signal. The low-frequency signal BL at the B pixel position is generated by applying a low-pass filter to the surrounding B signal and the B signal of the target pixel. A low-frequency signal BL at the Gb pixel position is generated by applying a low-pass filter to the surrounding B signal .

  FIG. 4 is a characteristic diagram of the low-pass filter used in the low-frequency image generating means, and represents the frequency characteristic of each signal at each pixel position. 4A to 4D, the fv axis is the vertical spatial frequency of the pixel array (sampling frequency Fs), the fh axis is the horizontal spatial frequency of the pixel array (sampling frequency Fs), and the vertical axis is the output value. (Signal passing amount). Moreover, it is distributed in the order of RL, GrL, GbL, and BL from the top to the bottom of the vertical axis direction.

  FIG. 4A shows the frequency characteristic of RL at the R pixel position, the frequency characteristic of GrL at the Gr pixel position, the frequency characteristic of GbL at the Gb pixel position, and the frequency characteristic of BL at the B pixel position. FIG. 4B shows GrL frequency characteristics at the R pixel position, RL frequency characteristics at the Gr pixel position, BL frequency characteristics at the Gb pixel position, and GbL frequency characteristics at the B pixel position. FIG. 4C shows GbL frequency characteristics at the R pixel position, BL frequency characteristics at the Gr pixel position, RL frequency characteristics at the Gb pixel position, and GrL frequency characteristics at the B pixel position. FIG. 4D shows the BL frequency characteristics at the R pixel position, the GbL frequency characteristics at the Gr pixel position, the GrL frequency characteristics at the Gb pixel position, and the RL frequency characteristics at the B pixel position.

  As shown in FIGS. 4A to 4D, the pass band of each low-frequency signal is configured to be in the range of 0 to approximately (1/4) Fs in the fv direction and the fh direction.

  Next, as shown in FIG. 3, the medium frequency image generation unit 12 filters the medium frequency pixel signal for each pixel of interest by applying a bandpass filter to surrounding pixels (5 × 5 pixel range). Generate interpolation. FIG. 3A is a coefficient arrangement diagram of the band filter when the Gr medium frequency signal GrM is generated by interpolation. FIG. 3B is a band filter when the Gb medium frequency signal GbM is generated by interpolation. FIG.

  Specifically, as shown in FIG. 3A, the Gr medium frequency signal GrM at the Gr pixel position is generated by applying a bandpass filter to the surrounding Gr signal and the Gr signal of the target pixel. A medium frequency signal GrM at the R pixel position is generated by applying a bandpass filter to the surrounding Gr signal. A medium frequency signal GrM at the B pixel position is generated by applying a bandpass filter to the surrounding Gr signal. A medium frequency signal GrM at the Gb pixel position is generated by applying a bandpass filter to the surrounding Gr signal.

  Also, as shown in FIG. 3B, the Gb medium frequency signal GbM at the Gr pixel position is generated by applying a bandpass filter to the surrounding Gb signal. A medium frequency signal GbM at the R pixel position is generated by applying a bandpass filter to the surrounding Gb signal. A medium frequency signal GbM at the B pixel position is generated by applying a bandpass filter to the surrounding Gb signal. A medium frequency signal GbM at the pixel position of Gb is generated by applying a bandpass filter to the surrounding Gb signal and the Gb signal of the target pixel.

  FIG. 5 is a characteristic diagram of the band-pass filter used in the medium frequency image generating means, and represents the frequency characteristic of each signal at each pixel position. 5A to 5C, the fv axis is the vertical spatial frequency of the pixel array (sampling frequency Fs), the fh axis is the horizontal spatial frequency of the pixel array (sampling frequency Fs), and the vertical axis is the output value. (Signal passing amount). Further, RM, GrM, GbM, and BM are distributed in this order from the top to the bottom of the vertical axis direction.

  FIG. 5A shows RM frequency characteristics at the R pixel position, GrM frequency characteristics at the Gr pixel position, GbM frequency characteristics at the Gb pixel position, and BM frequency characteristics at the B pixel position. FIG. 5B shows GrM frequency characteristics at the R pixel position, RM frequency characteristics at the Gr pixel position, BM frequency characteristics at the Gb pixel position, and GbM frequency characteristics at the B pixel position. FIG. 5C shows GbM frequency characteristics at the R pixel position, BM frequency characteristics at the Gr pixel position, RM frequency characteristics at the Gb pixel position, and GrM frequency characteristics at the B pixel position. Note that the frequency characteristics of GrM at the Gb pixel position, GbM at the Gr pixel position, RM at the B pixel position, BM at the R pixel position, and the like have zero passing amounts at all spatial frequencies. It doesn't appear inside.

  As shown in FIGS. 5A to 5C, the passbands of GrM and GbM are (1/4) Fs at the center in the fv direction and the fh direction, and (0) Fs and (1/2) ) Fs is configured such that the passing amount is zero.

  Next, as shown in FIG. 1, the image processing apparatus 200 generates a false color generation estimated amount signal (FCS) using the low frequency signal obtained via the low frequency image generation unit 11. Generation amount estimation means 13, low frequency color difference signal generation means 14 for generating a first color difference signal (UL) and a second color difference signal (VL), low frequency luminance signal generation means for generating a low frequency luminance signal (YL) 15, etc.

  In addition, the image processing apparatus 200 includes a medium frequency luminance signal generating unit 16 that generates a medium frequency luminance signal (YM) using the medium frequency signal obtained through the middle frequency image generating unit 12, and a first color difference signal. A false color generation estimated amount signal (FCS) is added to or subtracted from each of (UL) and the second color difference signal (VL) to generate a third color difference signal (U) and a fourth color difference signal (V). The low frequency luminance signal (YL) obtained via the false color suppression means 17 and the low frequency luminance signal generation means 15 and the medium frequency luminance signal (YM) obtained via the medium frequency luminance signal generation means 16 are obtained. Luminance synthesizing means 18 for generating a luminance signal (Y) by combining, color information for each pixel is generated using the luminance signal (Y), the third color difference signal (U), and the fourth color difference signal (V). Color space conversion means 19, CPU (Central Pr cessing Unit) equipped with a 20, ROM (Read Only Memory) 21 or the like, CPU 20 controls each process in the image processing apparatus 200 and the imaging apparatus 300 according to a control program stored in the ROM 21.

The false color generation amount estimation means 13 calculates the difference between the GrL and GbL signals generated by the low frequency image generation means 11 and generates the false color generation estimation amount signal FCS, as expressed in (Equation 1). .
FCS = abs (GrL−GbL) / 2 (1)
At this time, if necessary, an upper limit may be set for the false color generation estimated amount signal FCS generated by the false color generation amount estimating means 13, or an appropriate coefficient may be applied to make it variable. .

The low frequency chrominance signal generation unit 14 obtains an average value of the low frequency signal GrL and the low frequency signal GbL generated by the low frequency image generation unit 11 as expressed in (Expression 2) and (Expression 3), The average value is subtracted from the low frequency signal BL to generate a first color difference UL, and the average value is subtracted from the low frequency signal RL to generate a second color difference VL.
UL = BL− (GrL + GbL) / 2 (Formula 2)
VL = RL− (GrL + GbL) / 2 (Formula 3)

The low frequency luminance signal generation means 15 is represented by (Equation 4), the low frequency signal RL generated by the low frequency image generation means 11, the low frequency signal GrL, the low frequency signal GbL, and the low frequency signal. The low frequency luminance signal YL is generated by obtaining the sum with BL.
YL = RL + GrL + GbL + BL (Formula 4)

The medium frequency luminance signal generation unit 16 obtains the sum of the medium frequency signal GrM and the medium frequency signal GbM generated by the medium frequency image generation unit 12 as shown in (Equation 5), and multiplies this by L. A medium frequency luminance signal YM is generated.
YM = (GrM + GbM) × L (Formula 5)
In (Expression 5), L is a coefficient for estimating and adding signals in the medium frequency band of R and B at the medium frequency of G, and is adjusted between about 1 to 4 as necessary.

As shown in (Expression 6) and (Expression 7), the false color suppression means 17 has an absolute value of 0 for each of the first color difference signal (UL) and the second color difference signal (VL). The false color generation estimated amount signal (FCS) is added or subtracted so as to approach, and the third color difference signal (U) and the fourth color difference signal (V) are generated. U = sign (UL) * max (0, abs (UL ) -FCS) (Formula 6)
V = sign (VL) * max (0, abs (VL) −FCS) (Expression 7)

As shown in (Equation 8), the luminance synthesizing unit 18 uses the low frequency luminance signal (YL) generated by the low frequency luminance signal generating unit 15 and the medium frequency luminance generated by the medium frequency luminance signal generating unit 16. The luminance signal (Y) is generated by combining the signal (YM).
Y = YL + YM (Formula 8)
At this time, if necessary, the overall luminance frequency characteristic may be varied by multiplying YM by an arbitrary coefficient.

The color space conversion means 19 is generated by the luminance signal (Y) generated by the luminance synthesis means 18 and the false color suppression means 17 as expressed in (Equation 9), (Equation 10), and (Equation 11). Color information (R, G, B) for each pixel of the color image is generated using the third color difference signal (U) and the fourth color difference signal (V).
R = Y−U + 3 * V (Formula 9)
G = Y−U−V (Formula 10)
B = Y + 3 * U−V (Formula 11)
At this time, if necessary, the data may be output in the YUV format without being converted into the RGB color space, or may be output after being converted into another color space.

  Next, the procedure of the image processing method will be described based on FIG. This procedure is executed by the CPU 20 giving a command signal to each functional unit based on a program stored in the ROM 21. Further, S in FIG. 6 represents a step.

  First, this procedure starts when a start signal is input to the image processing apparatus 200 and the imaging apparatus 300 by the user.

  Next, in S100, an image signal (color mosaic image) is read into the image processing device 200 via the imaging unit 100, and is associated with the Bayer array using the mosaic image storage unit 10 for each of the R, Gr, Gb, and B signals. Store, and then move to S110 and S120.

  Next, in S110, the low-frequency image generation unit 11 is used to filter the color mosaic image by applying a low-pass filter to generate a low-frequency band RL signal, GrL signal, GbL signal, and BL signal, and then S130, The process moves to S140 and S150.

  Next, in S130, the false color generation amount estimation means 13 is used to determine the difference between the GrL and GbL signals of the low frequency signal generated in S110 and generate the false color generation amount estimation signal FCS.

  In S140, the low-frequency color difference signal generation unit 14 is used to obtain an average value of the GrL signal and the GbL signal of the low-frequency signal generated in S110, and the first color difference is obtained by subtracting the average value from the BL signal. A signal UL is generated, and a second color difference signal VL is generated by subtracting the average value from the RL signal.

  In S150, the low frequency luminance signal generation means 15 is used to obtain the sum of the RL signal, GrL signal, GbL signal, and BL signal generated in S110 to generate the low frequency luminance signal YL.

  Next, the process proceeds from S130 and S140 to S170. In S170, the false color suppression means 17 is used, and the false color is set so that the absolute value approaches 0 for each of the first color difference signal UL and the second color difference signal VL. The generated estimated amount signal FCS is added or subtracted to generate a third color difference signal U and a fourth color difference signal V, and then the process proceeds to S190.

  On the other hand, in S120, the medium frequency image generation means 12 is used to filter the color mosaic image by applying a band filter to generate a GrM signal and a GbM signal in the medium frequency band, and then the process proceeds to S160.

  Next, in S160, the medium frequency luminance signal generating means 16 is used to obtain the sum of the GrM signal and GbM signal generated in S120, and the medium frequency luminance signal YM is generated in association with this sum, and then in S180. Move.

  Next, in S180, the luminance synthesizing unit 18 is used to generate the luminance signal Y by synthesizing the low frequency luminance signal YL generated in S150 and the medium frequency luminance signal YM generated in S160, and then the process proceeds to S190. .

  Next, in S190, using the color space conversion means 190, the color difference signals (U, L) generated in S170 and the luminance signal Y generated in S180 are converted into color information (R, G, B for each pixel of the color image). ), And then this process ends.

  The low-frequency image generation step of the present invention corresponds to S110, the medium-frequency image generation step of the present invention corresponds to S120, the false color generation amount estimation step of the present invention corresponds to S130, and the low-frequency image generation step of the present invention corresponds to S120. The color difference signal generation step corresponds to S140, the low frequency luminance signal generation step of the present invention corresponds to S150, the medium frequency luminance signal generation step of the present invention corresponds to S160, and the false color suppression step of the present invention corresponds to S170. The luminance composition step of the present invention corresponds to S180, and the color space conversion step of the present invention corresponds to S190.

  As described above, according to the image processing apparatus 200, the image processing method, and the image processing program in the first embodiment, the interpolation process related to the demosaic process is simultaneously performed in the low frequency image generation unit 11 and the medium frequency image generation unit 12. At the same time, pixel signals are generated when the luminance signal Y and the color difference signals U and V are generated, and no interpolation processing is required in the subsequent steps up to the color space conversion means 19, so that the color mosaic image can be converted into color. The processing load can be reduced when generating an image.

  Further, as shown in FIGS. 2 and 3, the low-pass filter for generating the low-frequency image by interpolation and the band-pass filter for generating the intermediate frequency image by interpolation can be realized with a filter size of 5 × 5 pixel range. Therefore, the memory capacity for holding the peripheral pixels of the target pixel can be reduced, and the amount of calculation can be reduced.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS. Since the basic configuration of the second embodiment is the same as that of the first embodiment, portions that are different from the first embodiment will be described in detail, and the same reference numerals are given to the components that are common to the first embodiment. To omit the detailed description.

  The second embodiment differs from the first embodiment only in the configuration of the low-frequency chrominance signal generating means 14 in the first embodiment, and the low-frequency chrominance signal generating means in the second embodiment is given reference numeral 14A. The second low-frequency color difference signal generation unit 14A will be described.

  As shown in FIG. 7, the second low-frequency color difference signal generation unit 14 </ b> A includes the first color difference signal UL and the second color difference signal UL based on the image signals GrL, GbL, RL, and BL generated by the low-frequency image generation unit 11. The color difference signal VL is generated.

  Specifically, as shown in FIG. 7A, the second low-frequency color difference signal generation unit 14A subtracts GbL from the image signal GrL generated by the low-frequency image generation unit 11 to obtain a positive or negative difference value. The first subtraction means 14b to calculate and the first sign acquisition means 14c to acquire the positive or negative sign of the difference value obtained by the first subtraction means 14b, the image signal generated by the low frequency image generation means 11 Second subtracting means 14d for subtracting RL from BL to calculate a positive or negative difference value and second sign acquiring means 14e for acquiring a positive or negative sign of the difference value obtained by the second subtracting means 14d Depending on the positive or negative sign as a result of multiplication by multiplying the positive or negative sign obtained by the first sign obtaining means 14c and the positive or negative sign obtained by the second sign obtaining means 14e. First color difference signal and second color difference signal The XOR operation means 14f for outputting the first and second command signals for switching the arithmetic expressions, and the first one for switching the arithmetic expressions for the first color difference signal and the second color difference signal according to the command signals obtained by the XOR operation means 14f. First color difference for calculating the first color difference signal UL using an arithmetic expression switched by the first color difference signal calculation switching means 14g, the second color difference color difference signal switching means 14h, and the first color difference signal calculation switching means 14g. The signal calculation means 14j, the second color difference signal calculation means 14k for calculating the second color difference signal VL using the arithmetic expression switched by the second color difference signal calculation switching means 14h, and the like are configured.

The XOR operation means 14f outputs a first command signal for instructing the first arithmetic expression when the multiplication result is positive and outputs a second instruction for instructing the second arithmetic expression when the multiplication result is negative. The second command signal is output.
The first color difference signal calculation switching unit 14g outputs GrL to the first color difference signal calculation unit 14j when the first command signal is input from the XOR calculation unit 14f, and the second color difference signal calculation switching unit 14g outputs the second color difference signal calculation unit 14f. When a command signal is input, GbL is output to the first color difference signal calculation means 14j.

  On the other hand, the second color difference signal calculation switching unit 14h outputs GbL to the second color difference signal calculation unit 14k when the first command signal is input from the XOR calculation unit 14f, and the second color difference signal calculation switching unit 14h receives the first command signal from the XOR calculation unit 14f. When the second command signal is input, GrL is output to the second color difference signal calculating means 14k.

  The first color difference signal calculation unit 14j calculates the first color difference signal UL by subtracting the GrL signal or GbL signal output from the BL signal via the first color difference signal calculation switching unit 14g. Specifically, the first color difference signal calculating unit 14j outputs the first color difference signal UL as UL = BL−GrL when the GrL signal is output via the first color difference signal calculation switching unit 14g. When the GbL signal is output through the arithmetic expression and the first color difference signal calculation switching means 14g is output, the first color difference signal UL is calculated by the arithmetic expression of UL = BL-GbL.

  The second color difference signal calculation unit 14k calculates the second color difference signal VL by subtracting the GbL signal or GrL signal output from the RL signal via the second color difference signal calculation switching unit 14h. Specifically, the second color difference signal calculation unit 14k outputs the second color difference signal VL as VL = RL−GbL when the GbL signal is output via the second color difference signal calculation switching unit 14h. When the GrL signal is output through the arithmetic expression and is output via the second color difference signal arithmetic switching means 14h, the second color difference signal VL is calculated according to the arithmetic expression of VL = RL-GrL.

With the above-described configuration, the arithmetic expressions for calculating the first color difference signal UL and the second color difference signal VL are switched according to the positive or negative arithmetic sign obtained by the arithmetic expression of (GrL−GbL) × (BL−RL). It is done.
That is, when a positive sign is obtained from the arithmetic expression (GrL−GbL) × (BL−RL), the first color difference signal UL is calculated by the arithmetic expression UL = BL−GrL, The color difference signal VL is calculated by an arithmetic expression of VL = RL−GbL.

  Also, with the above-described configuration, when a negative sign is obtained from the arithmetic expression of (GrL-GbL) × (BL-RL), the first color difference signal UL is calculated by the arithmetic expression of UL = BL-GbL. Then, the second color difference signal VL is calculated by an arithmetic expression of VL = RL−GrL.

  According to the low-frequency color difference signal generation unit 14A according to the second embodiment described above, moire (interference fringes) is associated with a positive or negative sign obtained by an arithmetic expression of (GrL−GbL) × (BL−RL). The generation of color moire can be reduced without impairing the saturation by switching the calculation formulas for generating the first color difference signal and the second color difference signal according to the estimation result. Hereinafter, the operation and effect of the second low-frequency color difference signal generation unit 14A according to the second embodiment will be described.

  FIG. 8 is a characteristic diagram of UL and VL obtained by calculating the first color difference UL and the second color difference VL according to (Expression 2) and (Expression 3) in the first embodiment. In (Expression 2) and (Expression 3), when calculating UL and VL, the average of GrL and GbL is subtracted from each of BL and RL.

  In FIG. 8, the fv axis represents the spatial frequency in the vertical direction of the pixel array, the fh axis represents the spatial frequency in the horizontal direction of the pixel array, and the vertical axis represents the output value. 8A is the UL characteristic at the R pixel position, FIG. 8B is the VL characteristic at the R pixel position, FIG. 8C is the UL characteristic at the Gr pixel position, and FIG. The VL characteristic in the pixel position of Gr is represented.

  In the primary color RGB Bayer array, the color information is modulated at a frequency of (fh, fV) = (± fs / 2, ± fs / 2). Therefore, color information can be obtained by extracting and demodulating this frequency component. It is done. At this time, fs / 2 is a Nyquist frequency.

  Ideally, when generating color information, it is better not to pick up the frequencies of (fh, fV) = (0, ± fs / 2), (± fs / 2, 0). As can be seen, since this has a passband of about 0.5 at the frequency of (fh, fV) = (0, ± fs / 2), (± fs / 2, 0), this portion is false color. As a result, an area that is not originally colored is colored.

  If the color difference signals U and V are generated using (Equation 6) and (Equation 7) in the first embodiment, a false color generation estimated amount signal (FCS) generated by the false color generation amount estimation means 13 is generated. ) Is highly correlated with the false color generated at the frequencies of (fh, fV) = (0, ± fs / 2), (± fs / 2, 0), but the image contains many high frequencies. In some cases, the false color generation estimation amount signal becomes a large value even in a place where no false color is generated, and the saturation of the color image may be lowered.

  Therefore, the inventor of the present application, as means for reducing false colors without reducing the saturation of the color image, as shown in the second low-frequency color difference signal generation means 14A, Inventing a method of using either one of the color difference signals VL without using the average value of GrL and GbL.

That is, if one of GrL and GbL is used when generating the first color difference signal UL and the second color difference signal VL, the first arithmetic expression using (Expression 12) and (Expression 13) There is a second arithmetic expression using (Expression 14) and (Expression 15). In the present embodiment, the first arithmetic expression and the second arithmetic expression are obtained by combining the positive or negative sign obtained by (GrL-GbL) and the positive or negative sign obtained by (BL-RL). Switch.
"First formula"
UL = BL-GrL (Formula 12)
VL = RL-GbL (Formula 13)
"Second formula"
UL = BL-GbL (Formula 14)
VL = RL-GrL (Formula 15)

  As a result of observing the occurrence of false colors with CZP (Circular Zone Plate) for each of the first and second arithmetic expressions, the first arithmetic expression occurs with (fh, fV) = (± fs / 2, 0). False colors (vertical stripes) decrease, and false colors (horizontal stripes) generated at (fh, fV) = (0, ± fs / 2) increase. Further, the second arithmetic expression is opposite to the first arithmetic expression, and false colors (horizontal stripes) generated at (fh, fV) = (0, ± fs / 2) are reduced, and (fh, fV) = False color (vertical stripes) generated at (± fs / 2, 0) increases. Then, the inventor of the present application indicates whether the false color generation area is close to (fh, fV) = (± fs / 2, 0) or (fh, fV) = (0, ± fs / 2), that is, It has been found that if the direction of the vertical stripe and the horizontal stripe can be determined, the false color can be suppressed by switching the calculation method of the color difference signal based on the direction determination.

  Next, FIG. 9 shows frequency characteristics of the color difference signals in the R pixel and the Gr pixel obtained by applying the first arithmetic expression. In FIG. 9, the fv axis represents the spatial frequency in the vertical direction of the pixel array, the fh axis represents the spatial frequency in the horizontal direction of the pixel array, and the vertical axis represents the output value. 9A is the UL characteristic at the R pixel position, FIG. 9B is the VL characteristic at the R pixel position, FIG. 9C is the UL characteristic at the Gr pixel position, and FIG. The VL characteristic in the pixel position of Gr is represented.

In FIG. 9, both UL and VL have a signal passing amount of 0 at a frequency of (fh, fV) = (± fs / 2, 0), and no false color is generated at this frequency. At a frequency of (fV) = (0, ± fs / 2), since the signal passing amount is 1, a false color is generated. Further, when the second arithmetic expression is used, the characteristics are obtained by switching fh and fv in FIG.

  Therefore, based on the frequency characteristics of FIG. 9, if the false color generation area is close to which of (± fs / 2, 0) (0, ± fs / 2), that is, if the direction of vertical stripes and horizontal stripes can be determined, that direction By switching the arithmetic expression of the color difference signal based on the determination, false colors can be suppressed.

Next, means for determining the direction of vertical stripes and horizontal stripes will be described with reference to FIGS.
In FIG. 10, the fv axis represents the spatial frequency in the vertical direction of the pixel array, the fh axis represents the spatial frequency in the horizontal direction of the pixel array, and the vertical axis represents the output value. 10 (a) shows the output characteristics of (GrL-GbL) at the Gr pixel position, FIG. 10 (b) shows the output characteristics of (GrL-GbL) at the R pixel position, and FIG. The output characteristics of (GrL-GbL) at the pixel position, and FIG. 10D show the output characteristics of (GrL-GbL) at the pixel position of Gb.

  As shown in FIG. 10B, at the R pixel position, if the calculation result of (GrL-GbL) is positive (+), it can be estimated as horizontal stripes, and if it is negative (-), it can be estimated as vertical stripes. Thus, it was found that at the pixel position of B, if the calculation result of (GrL-GbL) is positive (+), it can be estimated as vertical stripes, and if it is negative (-), it can be estimated as horizontal stripes. However, as shown in FIGS. 10 (a) and 10 (d), at the Gr and Gb pixel positions, the direction of the vertical stripes and the horizontal stripes cannot be determined by the sign of the calculation result of (GrL−GbL). It was.

  On the other hand, as shown in FIG. 11A, at the pixel position of the Gr pixel position, if the calculation result of (BL−RL) is positive (+), it can be estimated as vertical stripes, and if it is negative (−), it can be estimated as horizontal stripes. As shown in (d), it was found that at the pixel position of the Gb pixel position, if the calculation result of (BL−RL) is positive (+), it can be estimated as horizontal stripes, and if it is negative (−), it can be estimated as vertical stripes. However, as shown in FIG. 11B and FIG. 11C, in the R pixel position and the B pixel position, the direction determination between the vertical stripes and the horizontal stripes cannot be made according to the sign of the calculation result of (BL-RL). I found out.

  Therefore, as shown in FIG. 7B, vertical stripes and horizontal stripes are associated with the codes obtained from the calculation results of (GrL-GbL) and (BL-RL) at the pixel positions of Gr, R, B, and Gb. The direction can be determined, and if the numerical value obtained from the calculation result of (GrL−GbL) × (BL−RL) is positive (+), it can be determined as vertical stripes, and negative (−) as horizontal stripes.

  As mentioned above, although one Example of this invention was described, this invention is not limited to the said Example, Various aspects can be taken. For example, in FIG. 2 and FIG. 3, the low-pass filter and the band-pass filter are in the 5 × 5 pixel range, but a wider pixel range may be used if there is a margin in the calculation amount and the circuit scale. Thereby, steeper characteristics can be obtained.

  The image processing apparatus, the image processing method, and the image processing program according to the present invention are suitable for generating a color image having a plurality of color components for each pixel from a color mosaic image obtained through a single-plate image sensor. It is.

  DESCRIPTION OF SYMBOLS 1 ... Imaging optical system, 1a ... Imaging lens, 1b ... Iris (diaphragm), 2 ... Image sensor, 2a ... Imaging element, 2b ... Variable gain amplifier (AGC: Automatic Gain Control), 2c ... A / D converter, DESCRIPTION OF SYMBOLS 10 ... Mosaic image storage means, 11 ... Low frequency image generation means, 12 ... Medium frequency image generation means, 13 ... False color generation amount estimation means, 14 ... Low frequency color difference signal generation means, 14A ... Second low frequency color difference signal generation Means 15 ... Low frequency luminance signal generating means 16 ... Medium frequency luminance signal generating means 17 ... False color suppressing means 18 ... Luminance synthesizing means 19 ... Color space converting means 20 ... CPU (Central Processing Unit), 21 ... ROM (Read Only Memory), 100 ... imaging unit, 200 ... image processing apparatus, 300 ... imaging apparatus.

Claims (21)

R (red) G (green) B (blue) A plurality of pixels using a color mosaic image that is output through an image sensor having a color filter with a Bayer array of three colors and each pixel has color information of a single color. An image processing apparatus for generating a color image having color information of a color,
In the Bayer array, when the G pixel located beside R is represented as Gr, and the G pixel located beside B is represented as Gb,
Low-frequency image generation means for filtering the color mosaic image by applying a low-pass filter to generate an R signal, a Gr signal, a Gb signal, and a B signal in a low-frequency band;
A medium frequency image generating means for filtering the color mosaic image by applying a band filter to generate a Gr signal and a Gb signal in a medium frequency band;
Using the low frequency signal obtained through the low frequency image generation means, obtain an average value of the Gr signal and the Gb signal, subtract the average value from the B signal to generate a first color difference signal, Low-frequency color difference signal generating means for generating a second color difference signal by subtracting the average value from the R signal;
Low frequency luminance signal generating means for generating a low frequency luminance signal by obtaining the sum of the R signal, Gr signal, Gb signal, and B signal obtained by the low frequency image generating means;
A medium frequency luminance signal generating unit that obtains a sum of the Gr signal and the Gb signal obtained by the medium frequency image generating unit and generates a medium frequency luminance signal in association with the sum;
Luminance synthesis means for synthesizing the low frequency luminance signal and the medium frequency luminance signal to generate a luminance signal Y;
Color information for each pixel of the color image is generated using the luminance signal generated by the luminance synthesizing unit and the first color difference signal and the second color difference signal generated by the low frequency color difference signal generating unit. Color space conversion means;
An image processing apparatus. In place of the low-frequency chrominance signal generating means, a second low-frequency chrominance signal generating means,
The second low-frequency color difference signal generating means is
The positive (+) or negative (−) sign of the difference value obtained by subtracting the Gb signal from the Gr signal and the R signal from the B signal of the low frequency signal obtained through the low frequency image generating means. Multiply by the positive or negative sign of the difference value obtained by subtraction,
The Gr signal, Gb signal, B signal, and R signal of the low frequency signals obtained through the low frequency image generation means are respectively represented as GrL, GbL, BL, and RL, and the first color difference signal is represented by UL and the first signal. When the second color difference signal is expressed as VL,
When the result of the multiplication is positive (+), the first color difference signal UL is calculated using an arithmetic expression of UL = BL-GrL, and the second color difference signal VL is calculated as VL = RL-GbL. Calculated using the following equation:
When the result of the multiplication is negative (−), the first color difference signal UL is calculated using an arithmetic expression of UL = BL−GbL, and the second color difference signal VL is calculated as VL = RL−GrL. Calculate using the following formula:
The image processing apparatus according to claim 1.
A false color generation amount estimation unit that determines a difference between the Gr signal and the Gb signal generated through the low frequency image generation unit and generates a false color generation estimation amount signal;
The false color generation estimation amount signal is added to or subtracted from the first color difference signal and the second color difference signal so that the absolute value approaches 0, and the third color difference signal and the fourth color difference signal are obtained. False color suppression means to generate,
The color space conversion means generates the color information using the third color difference signal and the fourth color difference signal instead of the first color difference signal and the second color difference signal.
The image processing apparatus according to claim 1. In the low-frequency image generation means, when the sampling frequency is represented as Fs, the low-frequency signal has a pass band of 0 to approximately (1/4) Fs.
The image processing apparatus according to claim 1. In the medium frequency image generating means, when the sampling frequency is expressed as Fs, the center band of the pass band of the medium frequency signal is approximately (1/4) Fs, and (0) Fs and (1/2) In Fs, the passage amount is configured to be substantially zero.
The image processing apparatus according to claim 1. The low-frequency image generation means generates low-frequency signals having colors associated with the R, Gr, Gb, and B pixel positions, and all other colors except for the associated colors. Produces a low frequency signal of
The image processing apparatus according to claim 1. The medium frequency image generating means generates medium frequency signals of Gr and Gb at the pixel positions of R, Gr, Gb, and B.
The image processing apparatus according to claim 1. R (red) G (green) B (blue) A plurality of pixels using a color mosaic image that is output through an image sensor having a color filter with a Bayer array of three colors and each pixel has color information of a single color. An image processing method for generating a color image having color information of a color,
In the Bayer array, when the G pixel located beside R is represented as Gr, and the G pixel located beside B is represented as Gb,
A low-frequency image generation procedure for filtering the color mosaic image by applying a low-pass filter to generate an R signal, a Gr signal, a Gb signal, and a B signal in a low-frequency band;
A medium frequency image generation procedure for filtering the color mosaic image by applying a band filter to generate a Gr signal and a Gb signal in a medium frequency band;
Using the low frequency signal obtained through the low frequency image generation procedure, an average value of the Gr signal and the Gb signal is obtained, and the first color difference signal is generated by subtracting the average value from the B signal, A low-frequency chrominance signal generation procedure for generating a second chrominance signal by subtracting the average value from the R signal;
A low-frequency luminance signal generation procedure for generating a low-frequency luminance signal by calculating the sum of the R signal, Gr signal, Gb signal, and B signal obtained in the low-frequency image generation procedure;
A medium frequency luminance signal generation procedure for obtaining a sum of the Gr signal and the Gb signal obtained in the medium frequency image generation procedure, and generating a medium frequency luminance signal in association with the sum;
A luminance synthesis procedure for synthesizing the low frequency luminance signal and the medium frequency luminance signal to generate a luminance signal;
Color information for each pixel of the color image is generated using the luminance signal generated in the luminance synthesis procedure, the first color difference signal and the second color difference signal generated in the low frequency color difference signal generation procedure. Color space conversion procedure;
An image processing method using
Instead of the low frequency color difference signal generation procedure, using a second low frequency color difference signal generation procedure,
The second low frequency chrominance signal generation procedure includes:
The positive (+) or negative (-) sign of the difference value obtained by subtracting the Gb signal from the Gr signal of the low frequency signal obtained through the low frequency image generation procedure, and the R signal from the B signal Multiply by the positive or negative sign of the difference value obtained by subtraction,
The Gr signal, Gb signal, B signal, and R signal of the low frequency signals obtained through the low frequency image generation procedure are represented as GrL, GbL, BL, and RL, respectively, and the first color difference signal is represented as UL. When the second color difference signal is expressed as VL,
When the result of the multiplication is positive (+), the first color difference signal UL is calculated using an arithmetic expression of UL = BL-GrL, and the second color difference signal VL is calculated as VL = RL-GbL. Calculated using the following equation:
When the result of the multiplication is negative (−), the first color difference signal UL is calculated using an arithmetic expression of UL = BL−GbL, and the second color difference signal VL is calculated as VL = RL−GrL. Calculate using the following formula:
The image processing method according to claim 8.
A false color generation amount estimation procedure for generating a false color generation estimation amount signal by obtaining a difference between the Gr signal and the Gb signal generated through the low frequency image generation procedure;
The false color generation estimation amount signal is added to or subtracted from the first color difference signal and the second color difference signal so that the absolute value approaches 0, and the third color difference signal and the fourth color difference signal are obtained. A false color suppression procedure to generate,
Use
The color space conversion procedure generates the color information using the third color difference signal and the fourth color difference signal instead of the first color difference signal and the second color difference signal.
The image processing method according to claim 8 or 9. In the low-frequency image generation procedure, when the sampling frequency is expressed as Fs, the passband of the low-frequency signal is 0 to approximately (1/4) Fs.
The image processing method according to claim 8. In the intermediate frequency image generation procedure, when the sampling frequency is expressed as Fs, the pass band of the intermediate frequency signal is approximately (1/4) Fs at the center, and (0) Fs and (1/2) In Fs, the passing amount is substantially 0.
The image processing method according to claim 8. In the low frequency image generation procedure, for each of the R, Gr, Gb, and B pixel positions, a low frequency signal of a color corresponding to each of the pixel positions is generated, and all the other colors except for the corresponding color are generated. Generate color low frequency signal,
The image processing method according to claim 8. In the intermediate frequency image generation procedure, an intermediate frequency signal of Gr and Gb is generated for each of the R, Gr, Gb, and B pixel positions.
The image processing method according to claim 8. R (red) G (green) B (blue) A plurality of pixels using a color mosaic image that is output through an image sensor having a color filter with a Bayer array of three colors and each pixel has color information of a single color. An image processing program for generating a color image having color information of a color,
In the Bayer array, when the G pixel located beside R is represented as Gr, and the G pixel located beside B is represented as Gb,
A low-frequency image generation step of filtering the color mosaic image by applying a low-frequency filter to generate an R signal, a Gr signal, a Gb signal, and a B signal in a low frequency band;
A medium frequency image generating step of filtering the color mosaic image by applying a band filter to generate a Gr signal and a Gb signal in a medium frequency band;
Using the low frequency signal obtained through the low frequency image generation step, an average value of the Gr signal and the Gb signal is obtained, and the first color difference signal is generated by subtracting the average value from the B signal. A low-frequency color difference signal generation step of generating a second color difference signal by subtracting the average value from the R signal;
A low-frequency luminance signal generating step for generating a low-frequency luminance signal by obtaining the sum of the R signal, Gr signal, Gb signal, and B signal obtained in the low-frequency image generation step;
A medium frequency luminance signal generating step for obtaining a sum of the Gr signal and the Gb signal obtained in the medium frequency image generating step and generating a medium frequency luminance signal in association with the sum;
A luminance combining step of generating a luminance signal by combining the low frequency luminance signal and the medium frequency luminance signal;
Color information for each pixel of the color image is generated using the luminance signal generated in the luminance synthesis step, the first color difference signal and the second color difference signal generated in the low frequency color difference signal generation step. A color space conversion step;
An image processing program for causing a computer to execute. In place of the low frequency color difference signal generation step, comprising a second low frequency color difference signal generation step,
The second low frequency color difference signal generating step includes:
The positive (+) or negative (−) sign of the difference value obtained by subtracting the Gb signal from the Gr signal of the low frequency signal obtained through the low frequency image generation step , and the R signal from the B signal. Multiply by the positive or negative sign of the difference value obtained by subtraction,
The Gr signal, Gb signal, B signal, and R signal of the low frequency signals obtained through the low frequency image generation step are represented as GrL, GbL, BL, and RL, respectively, and the first color difference signal is represented as UL. When the second color difference signal is expressed as VL,
When the result of the multiplication is positive (+), the first color difference signal UL is calculated using an arithmetic expression of UL = BL-GrL, and the second color difference signal VL is calculated as VL = RL-GbL. Calculated using the following equation:
When the result of the multiplication is negative (−), the first color difference signal UL is calculated using an arithmetic expression of UL = BL−GbL, and the second color difference signal VL is calculated as VL = RL−GrL. Calculate using the following formula:
The image processing program according to claim 15.
A false color generation amount estimation step for determining a difference between the Gr signal and the Gb signal generated through the low frequency image generation step and generating a false color generation estimation amount signal;
The false color generation estimation amount signal is added to or subtracted from the first color difference signal and the second color difference signal so that the absolute value approaches 0, and the third color difference signal and the fourth color difference signal are obtained. A false color suppression step to be generated;
The color space conversion step generates the color information using the third color difference signal and the fourth color difference signal instead of the first color difference signal and the second color difference signal.
The image processing program according to claim 15 or 16. In the low frequency image generation step, when the sampling frequency is expressed as Fs, the pass band of the low frequency signal is 0 to approximately (1/4) Fs.
The image processing program according to any one of claims 15 to 17. In the intermediate frequency image generation step, when the sampling frequency is expressed as Fs, the pass band of the intermediate frequency signal is approximately (1/4) Fs at the center, and (0) Fs and (1/2) In Fs, the passing amount is substantially 0.
The image processing program according to any one of claims 15 to 18. In the low-frequency image generation step, a low-frequency signal having a color associated with each of the R, Gr, Gb, and B pixel positions is generated, and all the other colors except for the associated color are generated. Generate color low frequency signal,
The image processing program according to any one of claims 15 to 19. In the medium frequency image generation step, medium frequency signals of Gr and Gb are generated for each of the R, Gr, Gb, and B pixel positions.
The image processing program according to any one of claims 15 to 20.

Images