JP5032914B2 - Image processing apparatus, image processing program, and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing program, and an image processing method for processing an RGB Bayer array image.

  An image pickup apparatus that picks up an optical image and obtains an electrical color image has a configuration of a three-plate image pickup device capable of obtaining signals of three colors (for example, RGB three primary colors) at one pixel position, and one pixel. It is roughly divided into those having a single-plate image pickup device configuration that can obtain only one of the three color signals for each position.

  Digital cameras that are currently on the market generally have a single-plate image sensor configuration. On the other hand, digital video cameras currently on the market are in a state where a three-plate image sensor configuration and a single-plate image sensor configuration are mixed.

  When the imaging device has a three-plate imaging device configuration, three color signals can be obtained at each pixel position of the captured image at the time of imaging. The price increases because of the increase.

  On the other hand, when the image pickup apparatus has a single-plate image pickup device configuration, there is an advantage that the structure is simple as compared with the three-plate image pickup device configuration. However, in order to obtain three types of color signals, R, G, B for each pixel It is necessary to arrange the filters in a mosaic pattern (see the Bayer array as shown in FIG. 5 of the present invention), and only a single color signal can be obtained at each pixel position of the captured image. Therefore, in an image sensor having such a Bayer array single-plate image sensor configuration, three color signals per pixel are obtained by interpolating the missing color signals at each pixel position using the color signals at the surrounding pixel positions. Like to get.

  By the way, the Bayer array as described above has 2 × 2 pixels as a basic unit of the pixel array, and two G filters in one diagonal direction in the 2 × 2 pixels and an R filter in the other diagonal direction. And B filters are arranged one pixel at a time. Therefore, the sampling density of the G pixel is twice the sampling density of the R pixel and the sampling density of the B pixel. When imaging is performed using imaging elements having different sampling densities depending on color signals, in an imaging region having a high spatial frequency such as an edge, a high spatial frequency that can be expressed by a G signal is R signal and B The signal is folded back to the low frequency side. Therefore, it is known that when interpolation processing is performed for each color signal, false colors (colors that do not exist originally) may occur in the edge portion and the vicinity thereof.

  The simplest means for reducing such false colors is to reduce the characteristics of the lens or the optical low-pass filter to a spatial frequency that can reproduce the sampling interval of the R signal or the B signal of the Bayer array, and the optical image of the spatial frequency. Is imaged on the image sensor. However, if this means is employed, only a captured image having a spatial frequency that is half the reproducible spatial frequency can be obtained, resulting in a blurred image with low resolution. Therefore, the optical system is generally designed so that the resolution of the optical image formed on the image sensor has a spatial frequency characteristic that does not cause moire due to aliasing distortion at the G signal sampling interval.

  Various techniques for reducing false colors using the optical system designed in this way have been proposed.

  For example, Japanese Patent Application Laid-Open No. 8-237672 describes a technique for performing an interpolation process in which interpolation positions of three color (R, G, and B) signals are set to intermediate positions between pixels in both the horizontal direction and the vertical direction. Yes. This technique pays attention to the fact that R, G, and B frequency characteristics can be approximated at half the Nyquist frequency when interpolation processing is performed on the position. Although the technique described in this publication reduces false color in this way, false color suppression in an edge region including a frequency component close to the Nyquist frequency has not been sufficient. Furthermore, the technique described in the publication interpolates pixels at half-pixel positions both horizontally and vertically, and it is inevitable that high-frequency components are attenuated. However, the frequency characteristic of this interpolation filter is the Nyquist frequency. In order to request that the frequency characteristics of R, G, and B be approximated at half the frequency, the frequency band cannot be provided to the limit of the Nyquist frequency, and the resolution has been lowered.

  Further, as another technique for reducing false colors, which is different from the method of implementing interpolation processing of each color signal as described above in JP-A-8-237672 with a filter having uniform characteristics in an image, there is a local region of an image. Proposed a method of switching the interpolation filter to adapt to at least one direction dependency among the G signal similarity of G, the similarity between color signals of G signal and R signal, and the similarity between color signals of G signal and B signal Has been.

  For example, Japanese Patent Laid-Open No. 7-59098 discloses a horizontal similarity and a vertical similarity between four G signals around a pixel position of the missing G signal (the present invention). 19 to FIG. 21 according to FIG. 19, vertical similarity = | G1−G2 | and horizontal similarity = | G3−G4 |) are calculated, and the calculated similarity and a predetermined threshold value are calculated. And G pixel interpolation candidates {(G1 + G2) / 2, (G3 + G4) / 2, which are obtained by three linear interpolation methods using G signals as shown in FIGS. 19 to 21 according to the present invention, One G pixel interpolation candidate is selected from (G1 + G2 + G3 + G4) / 4} as an interpolation value of the missing G signal.

  However, in this technique, since the similarity between G signals that is twice the pixel interval is used, if the G pixel value changes at a frequency higher than ½ of the Nyquist frequency in the horizontal direction or the vertical direction. There is a high possibility that an interpolation method selection determination error will occur, and the generation of a false color due to the determination error may be a problem.

  Further, the technique described in Japanese Patent Application Laid-Open No. 11-275373 discloses an R signal or a B signal at a pixel position for generating an interpolated G signal, and four peripheral G signals (as shown in FIG. 21 according to the present invention). G1, G2, G3, G4) and the similarity {vertical similarity = | X−G1 | + | X−G2 |, horizontal similarity = | X−G3 | + | X−G4 | X is an R signal or a B signal)}, and one of vertical interpolation, horizontal interpolation, and interpolation obtained by weighted averaging of these two interpolations is selected. When this technique is used, there is an advantage that false colors in the achromatic edge region can be effectively reduced, but there is a problem that the saturation may be lowered in a region with high saturation.

  The false color is detected more conspicuously in the achromatic edge region. The achromatic region is a region in which the R signal, the G signal, and the B signal have substantially the same value, and means a region in which the ratio of the R signal to the G signal and the ratio of the B signal to the G signal are approximately 1. ing. The false color is generated when the ratio of the R signal, the G signal, and the B signal is different from 1, which is the ratio that should be originally obtained, as a result of performing the interpolation process. Is.

  The technique described in the above-mentioned Japanese Patent Application Laid-Open No. 11-275373 selects an interpolation from among a plurality of interpolation candidates that makes an R signal to G signal ratio or a B signal to G signal ratio close to 1. Since this is a technology, the occurrence of false colors in the achromatic region is inevitably suppressed.

  However, when this technique is used, the ratio of the R signal to the G signal in the chromatic color region in which the ratio of the R signal to the G signal or the ratio of the B signal to the G signal is originally different from 1 is similar to the achromatic color region. In other words, since the interpolation that brings the ratio of the B signal to the G signal close to 1 is selected, there is a problem that the saturation that should be originally reproduced is reduced in the chromatic color region.

  This problem will be described with a simple example. That is, here, an area where the color in the horizontal direction is the same and the color changes in the vertical direction is considered.

  Assume that a certain line including an R signal in a Bayer array has a constant value of 200 for all R signals and 10 for all G signals. Further, the values of the G signal positioned on the line above the target line all take a constant value of 5, and the values of the G signal positioned on the line below the target line all take a constant value of 45. . Note that the B signals in the lines above and below the line of interest are not considered in order to simplify the description.

When the technique described in Japanese Patent Laid-Open No. 11-275373 is applied to such a specific numerical example, the vertical similarity is | 200-5 | + | 200-45 | = 350, and the horizontal similarity is Becomes | 200−10 | + | 200−10 | = 380, and it is determined that the vertical similarity is higher than the horizontal similarity. Therefore, although the vertical interpolation should be selected originally, the vertical interpolation is selected, and the G signal interpolation value obtained at the R signal position of the line of interest is (5 + 45) / 2 = 25. In this case, the color difference RG of the line of interest is 200-25 = 175, which is a smaller value than the color difference RG = 200-10 = 190 that should be reproduced originally, so that the saturation is lowered. Will end up.
JP-A-8-237672 JP 7-59098 A JP 11-275373 A

  In the conventional technology as described above, a missing color signal is interpolated based on the surrounding color signal for a Bayer array image of one color of each pixel imaged by a single-plate image sensor, and a plurality of color signals are obtained for each pixel. When generating a color image having a false color at the edge portion, a decrease in resolution occurs in an attempt to reduce the false color, or pixel correlation is used to suppress the false color or resolution decrease When applying the adaptive interpolation method, image quality degradation may occur due to an adaptation error. Thus, the prior art has been insufficient for suppressing the generation of false colors, the reduction in resolution, and the reduction in saturation.

  The present invention has been made in view of the above circumstances, and when interpolating a missing color signal with respect to an RGB Bayer array image, while sufficiently suppressing false color and resolution reduction that occur in association with the interpolation processing. An object of the present invention is to provide an image processing apparatus, an image processing program, and an image processing method that can prevent further reduction in saturation.

In order to achieve the above object, an image processing apparatus according to a first invention processes an image including a pixel in which at least one color signal is missing among a plurality of color signals, and An image processing apparatus for generating an image including pixels interpolated with at least a first color signal, wherein the position in the image is (x, y) (where x is an integer indicating a pixel position in the horizontal direction, y Is an integer indicating the pixel position in the vertical direction) and has a second color signal different from the first color signal at the position (x0, y0) where the pixel having the first color signal is missing. When the pixel is X (x0, y0), M types of the first color signals interpolated in this manner of M (M is an integer of 2 or more) with respect to X (x0, y0). Pixel interpolation candidate calculating means for calculating Gt (x0, y0) (where t = 1,..., M); And band limiting means for calculating an XL (x0, y0) which has been band-limited using the same type color signals near to said X (x0, y0), the first M types calculated by the pixel interpolation candidate calculating means Based on one color signal Gt, M types of color difference candidates, which are color difference candidates X (x0, y0) −Gt (x0, y0) or color difference candidates XL (x0, y0) −Gt (x0, y0), are calculated. A peripheral position (of the same type t as the first color signal Gt (x0, y0) used when calculating the color difference candidate calculation means and the color difference candidate X (x0, y0) -Gt (x0, y0) ( Based on the first color signal Gt (x0 + n, y0 + m) calculated in (x0 + n, y0 + m) (here, n and m are arbitrary integers which are not simultaneously 0), the peripheral position (x0 + n, y0 + m) A plurality of X color difference candidates X (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) and the color difference candidate X (x0, y0) −Gt (x0, y0) or when calculating the color difference candidate XL (x0, y0) −Gt (x0, y0). Gt calculated at the peripheral position (x0 + n, y0 + m) of the same type t as the first color signal Gt (x0, y0) used (where n and m are any integers that are not 0 simultaneously). Based on (x0 + n, y0 + m), a plurality of XL color difference candidates XL (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) calculated for the peripheral position (x0 + n, y0 + m) and the color difference candidates XL (x0, Based on y0) −Gt (x0, y0), M types of color difference similarities are calculated, and based on the calculated M types of color difference similarities, M types of the first color signals Gt (x0, p types of the above y0) first color signal Gp (x0, y0) (here, 1 ≦ p ≦ M) or Type of the first color signal Gq (x0, y0) (here, 1 ≦ q ≦ M) based on the calculated color difference candidate X (x0, y0) -Gp ( x0, y0) or color difference candidate XL (x0 , Y0) −Gq (x0, y0) as the color difference, and the color difference X (x0, y0) −Gp (x0, x0, x0, y0) selected from the above X (x0, y0) by the optimum color difference selecting means. calculating means for calculating the first color signal G (x0, y0) for the position (x0, y0) by subtracting y0) or the color difference XL (x0, y0) −Gq (x0, y0). It is equipped.

  According to the first aspect of the present invention, when a missing color signal is interpolated with respect to an RGB Bayer array image, a color image that suppresses a false color generated in association with the interpolation process and maintains a saturation that should be originally reproduced. Can be obtained.

The image processing apparatus according to a second aspect of the invention is the image processing apparatus according to the first aspect of the invention, wherein the color difference at a position where the color difference is not calculated by the optimum color difference selecting unit is calculated by the optimum color difference selecting unit. A color difference interpolation unit that performs an interpolation process based on a color difference of the same color around the color difference, a color difference calculated by the optimum color difference selection unit or the color difference interpolation unit, and the first color signal at the same position as the color difference. And a pixel calculation means for calculating a pixel including a plurality of color signals based on the pixel.

  According to the second invention, when the missing color signal is interpolated with respect to the RGB Bayer array image, the false color generated by the interpolation processing is suppressed, and the saturation that should be reproduced is kept high. A resolution color image can be obtained.

Further, an image processing apparatus according to a third invention is the image processing apparatus according to the first invention, wherein a color difference interpolation means for calculating an interpolated color difference for the predetermined position based on a color difference of the same color around the predetermined position; based on the pixels having the first color signal around the predetermined position, said predetermined interpolation means place in pairs to calculate the interpolated said first color signal, interpolation calculated in the predetermined position Pixel calculation means for calculating pixels including a plurality of color signals at the predetermined position based on the color difference and the interpolated first color signal is further provided.

  According to the third invention, when the missing color signal is interpolated with respect to the RGB Bayer array image, the false color generated by the interpolation processing is suppressed, and the saturation that should be reproduced is kept high. A color image with a resolution can be obtained, and a smoother edge can be reproduced by reducing the backlash of the edge boundary. Furthermore, the number of interpolation pixels can be changed, and an image with a predetermined resolution can be created by selecting a band-limited interpolation filter coefficient corresponding to the number of interpolation pixels.

  An image processing apparatus according to a fourth invention is the image processing apparatus according to the first invention, wherein the optimum color difference selection means calculates the color difference similarity by using the color difference candidates X (x0, y0) −Gt (x0, y0). And a plurality of color difference candidates X (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) of the same color X and the same type t as the color difference candidates X (x0, y0) −Gt (x0, y0) And a plurality of color difference candidates X ^ (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) of a color X ^ different from the color difference candidate X (x0, y0) -Gt (x0, y0) and of the same type t. It is calculated based on the similarity.

  According to the fourth aspect, it is possible to select the color difference calculated by G pixel interpolation that matches the edge direction of the image.

  An image processing apparatus according to a fifth aspect of the present invention is the image processing apparatus according to the first aspect, wherein the optimum color difference selection means determines the color difference similarity as the color difference candidate XL (x0, y0) -Gt (x0, y0). And a plurality of color difference candidates XL (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) of the same color X and the same type t as the color difference candidates XL (x0, y0) -Gt (x0, y0) And a plurality of color difference candidates X ^ L (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) of a color X ^ different from the color difference candidate XL (x0, y0) -Gt (x0, y0) and of the same type t. The similarity is calculated based on the similarity.

  According to the fifth aspect, it is possible to select the color difference calculated by G pixel interpolation that matches the edge direction of the image.

  An image processing apparatus according to a sixth invention is the image processing apparatus according to the fourth or fifth invention, wherein the similarity is a sum of absolute differences between color difference candidates of the same color X and the same type t, The optimum color difference selection means includes selection means for selecting one color difference candidate corresponding to one of the color difference similarities that gives the minimum value among the M types of color difference similarities, and the color difference candidates selected by the selection means Is a color difference.

  According to the sixth aspect of the invention, it is possible to determine the optimum color difference and G pixel interpolation while suppressing the circuit scale when performing hardware.

  An image processing apparatus according to a seventh invention is the image processing apparatus according to the first invention, wherein the color difference selected by the optimum color difference selection means is XL (x0, y0) -Gt (x0, y0). The color difference is compared with a predetermined threshold value to determine whether the color difference is a high color difference, and G () for the position (x0, y0) determined to be a high color difference by the high color difference determination unit. change means for changing x0, y0) to Gt (x0, y0).

  According to the seventh aspect of the invention, it is possible to suppress the occurrence of jaggy in a high color difference and oblique edge region.

  An image processing apparatus according to an eighth invention is the image processing apparatus according to the first invention, wherein X (x0, y0) at a position (x0, y0) and X ( x0 + u, y0 + v) (where u and v are arbitrary integers that are not 0 at the same time), a saturation state determination means for determining a saturation state for the position (x0, y0), and the saturation state determination When it is determined that the saturation state is detected by the means, the color difference selected from the M kinds of color difference candidates by the optimum color difference selection means is fixed to one kind of color difference determined in advance based on the saturation state. And immobilization means.

  According to the eighth invention, the saturation state of the R pixel or the B pixel is determined, and a plurality of selectable color difference candidates are narrowed down to one predetermined type, so that the peripheral G pixel by blooming in the R pixel or the B pixel. Even when charges are leaked unevenly, the occurrence of unnatural artifacts can be suppressed.

An image processing apparatus according to a ninth aspect is the image processing apparatus according to any one of the first to eighth aspects, wherein the first color signal is a G component and the second color signal is a B component or an R component.

An image processing apparatus according to a tenth aspect is the image processing apparatus according to any one of the first to ninth aspects, wherein the image is an RGB Bayer array image.

An image processing apparatus according to an eleventh aspect is the image processing apparatus according to the ninth aspect, wherein the pixel interpolation candidate calculation means calculates an average of two G pixels adjacent above and below the X pixel as the M types of G components. A first interpolation calculating means for calculating as one kind of G component signal of the signals, and an average of two G pixels adjacent to the left and right of the X pixel, and one kind of G of the M kinds of G component signals. A second interpolation calculating means for calculating as a component signal, and a third for calculating an average of four G pixels adjacent to the top, bottom, left and right of the X pixel as one type of G component signal among the M types of G component signals. Interpolation calculation means of
It is comprised with having.

An image processing apparatus according to a twelfth invention is the image processing apparatus according to the eleventh invention, wherein the optimum color difference selection means includes a G pixel around a G pixel missing position indicating a position where a pixel having a G component signal is missing. The G pixel fluctuation amount calculating means for calculating the fluctuation amount of the pixel value and calculating a weighting coefficient based on the fluctuation amount, and the G component signal calculated based on the G component signal calculated by the third interpolation calculating means. Multiplier means for multiplying the similarity between the color difference candidate at the G pixel missing position and the surrounding color difference candidates by the weighting factor.

An image processing device according to a thirteenth invention is the image processing device according to the twelfth invention , wherein X (x0, y0) at the position (x0, y0) and X ( x0 + u, y0 + v) (where u and v are arbitrary integers that are not 0 at the same time), a saturation state determination means for determining a saturation state for the position (x0, y0), and the saturation state determination And a weighting factor changing unit that changes the weighting factor calculated by the G pixel variation calculation unit to a minimum value when it is determined that the unit is saturated.

  According to the thirteenth invention, the saturation state of the R pixel or the B pixel is determined, and a plurality of selectable color difference candidates are narrowed down to one predetermined type, so that the peripheral G pixel by blooming in the R pixel or the B pixel. Even when charges are leaked unevenly, the occurrence of unnatural artifacts can be suppressed.

An image processing program according to a fourteenth aspect of the invention causes a computer to process an image including a pixel in which at least one color signal of a plurality of color signals is missing, and at least a first of the plurality of color signals. An image processing program for generating an image including a pixel in which a color signal is interpolated, wherein the computer determines a position in the image (x, y) (where x is an integer indicating a pixel position in the horizontal direction, y is an integer indicating the pixel position in the vertical direction), and a second color different from the first color signal at the position (x ₀ , y ₀ ) where the pixel having the first color signal is missing. pixels having a signal when the _{X (x 0, y 0)} , the X (x _0, y ₀₎ with respect to, M kinds (M is an integer of 2 or more) M type interpolated by the interpolation manner the first color signal _{G t (x 0, y 0} ) ( here, t = 1, , The band is calculated and a pixel interpolation candidate calculating a M), the X and (x _0, y ₀₎ X has been band-limited using the same type color signals near against _L (x _0, y ₀₎ limits Color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) or color difference based on the step and the M types of first color signals G _t calculated in the pixel interpolation candidate calculation step A color difference candidate calculation step for calculating M types of color difference candidates consisting of candidates X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ), and the color difference candidates X (x ₀ , y ₀ ) −G _t. the (x _0, y ₀₎ of the first color signal used for calculating the G _t (x _0, y ₀₎ and the same type t, the peripheral position _{(x 0 + n, y 0} + m) ( where Based on the first color signal G _t (x ₀ + n, y ₀ + m) calculated in (n and m are arbitrary integers that are not 0 at the same time) A plurality of X color difference candidates X (x ₀ + n, y ₀ + m) −G _t (x ₀ + n, y ₀ + m) calculated for the side position (x ₀ + n, y ₀ + m), and the color difference candidates Based on X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) or the color difference candidate X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) is calculated. The peripheral position (x ₀ + n, y ₀ + m) of the same type t as the first color signal G _t (x ₀ , y ₀ ) used here (where n and m are not 0 simultaneously) Based on G _t (x ₀ + n, y ₀ + m) calculated in (any integer), a plurality of color difference candidates X _L of X _L calculated for the peripheral position (x ₀ + n, y ₀ + m) ( Based on x ₀ + n, y ₀ + m) −G _t (x ₀ + n, y ₀ + m) and the color difference candidate X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ), M Calculate the color difference similarity of the type Based on the M types of color difference similarity issued, first color signal M types of the _{G t (x 0, y 0} ) 1 single type p of the first color signal G _p (x ₀ of the, y ₀ ) (where 1 ≦ p ≦ M) or one type q of the first color signal G _q (x ₀ , y ₀ ) (where 1 ≦ q ≦ M) Optimal color difference selection step of selecting candidate X (x ₀ , y ₀ ) −G _p (x ₀ , y ₀ ) or color difference candidate X _L (x ₀ , y ₀ ) −G _q (x ₀ , y ₀ ) as a color difference And the color difference X (x ₀ , y ₀ ) −G _p (x ₀ , y ₀ ) or color difference X _L (x ₀ , y ₀ ) selected from the above X (x ₀ , y ₀ ) by the optimum color difference selection step. ) -G _q (x ₀ , y ₀ ) is subtracted to perform the calculation step of calculating the first color signal G (x ₀ , y ₀ ) for the position (x ₀ , y ₀ ). for Is a program.

  According to the fourteenth aspect, substantially the same effect as the first aspect can be achieved.

An image processing method according to a fifteenth aspect of the invention processes an image including a pixel in which at least one of the plurality of color signals is missing, so that at least the first color signal of the plurality of color signals is obtained. An image processing method for generating an image including interpolated pixels, wherein a position in the image is (x, y) (where x is an integer indicating a horizontal pixel position, and y is a vertical pixel. A pixel having a second color signal different from the first color signal at the position (x ₀ , y ₀ ) where the pixel having the first color signal is missing. When (x ₀ , y ₀ ), M types of the first colors interpolated in this manner of M (M is an integer of 2 or more) with respect to X (x ₀ , y ₀ ) (herein, t = 1, ..., M ) signal _{G t (x 0, y 0} ) and a pixel interpolation candidate calculating a upper X (x _0, y ₀₎ and band limiting step of calculating the band-limited by the _{X L (x 0, y 0} ) by using the same type color signals near relative, calculated by the pixel interpolation candidate calculating step M Based on the type of the first color signal G _t , the color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) or the color difference candidate X _L (x ₀ , y ₀ ) −G _t ( The color difference candidate calculation step for calculating M types of color difference candidates consisting of x ₀ , y ₀ ) and the color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) are used. Peripheral position (x ₀ + n, y ₀ + m) of the same type t as the first color signal G _t (x ₀ , y ₀ ) (where n and m are any integers that are not 0 simultaneously) X calculated for the peripheral position (x ₀ + n, y ₀ + m) based on the first color signal G _t (x ₀ + n, y ₀ + m) calculated in step Color candidate X (x ₀ + n, y ₀ + m) −G _t (x ₀ + n, y ₀ + m) and the color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) Or the first color signal G _t (x ₀ , y ₀ ) used in calculating the color difference candidate X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ). ) And G _t (x ₀ + n, y ₀ + m) calculated at the peripheral position (x ₀ + n, y ₀ + m) (where n and m are not any integers that are not 0 at the same time). based on), the peripheral position _{(x 0 + n, y 0} + multiple color difference X _L calculated for m) candidate _{X L (x 0 + n,} y 0 + m) -G t (x 0 + n, y 0 + M) and the color difference candidates X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ), M types of color difference similarities are calculated, and the calculated M types of color difference similarities are calculated. based on the, on the M types (Here, 1 ≦ p ≦ M) first color signal _{G t (x 0, y 0} ) in the first one type p of the color signal _{G p (x 0, y 0} ) or one Color difference candidate X (x ₀ , y ₀ ) −G _p (x ₀ , x, calculated based on the first color signal G _q (x ₀ , y ₀ ) (here 1 ≦ q ≦ M) of type q y ₀ ) or a color difference candidate X _L (x ₀ , y ₀ ) −G _q (x ₀ , y ₀ ) as an optimum color difference selection step, and the optimum color difference selection from X (x ₀ , y ₀ ) By subtracting the color difference X (x ₀ , y ₀ ) −G _p (x ₀ , y ₀ ) or the color difference X _L (x ₀ , y ₀ ) −G _q (x ₀ , y ₀ ) selected by the step. And calculating the first color signal G (x ₀ , y ₀ ) for the position (x ₀ , y ₀ ).

According to the fifteenth aspect, substantially the same effect as in the first aspect can be achieved.

  According to the image processing apparatus, the image processing program, and the image processing method of the present invention, when the missing color signal is interpolated with respect to the RGB Bayer array image, the false color generated due to the interpolation processing and the resolution reduction are sufficiently obtained. It is possible to prevent further reduction in saturation while suppressing the color saturation.

  Embodiments of the present invention will be described below with reference to the drawings.

[Embodiment 1]
FIGS. 1 to 7 and FIGS. 19 to 29 show Embodiment 1 of the present invention, and FIG. 1 is a block diagram showing the overall configuration of the image processing apparatus.

  As shown in FIG. 1, this image processing apparatus includes an imaging unit 101, a G interpolation color difference calculation unit 102, a color difference interpolation processing unit 103, an RGB calculation unit 104, and a compression recording unit 105. .

  Although not shown, the imaging unit 101 includes a lens, an IR cut filter, an optical low-pass filter, a single-plate imaging device (hereinafter simply referred to as “imaging device”), an amplifier, and A / D conversion. And an image sensor controller. Here, the lens is used to form an optical image of the subject on the image sensor. The IR cut filter is for cutting light in the infrared region from the light beam passing through the lens. The optical low-pass filter is an optical device that limits the band so that the resolution of the optical image formed on the image pickup device by the lens has a spatial frequency characteristic that does not cause over-permissible moire due to aliasing distortion at the G signal sampling interval. It is a filter. The image pickup device is a single-plate image pickup device having a Bayer array color filter as shown in FIG. 5, and is composed of, for example, a CCD or a CMOS. FIG. 5 is a diagram illustrating a Bayer array of color signals captured by the single-plate image sensor of the imaging unit 101. The amplifier amplifies the signal output from the image sensor in an analog manner. The A / D converter converts the analog signal amplified by this amplifier into a digital signal. The image pickup device controller drives the image pickup device while controlling the image pickup device, and outputs a digital signal output from the A / D converter after the image pickup to the G interpolation color difference calculation unit 102 according to the color component as described later. To do.

  In such a configuration, the light imaged on the single-plate image sensor through the lens, the IR cut filter, and the optical low-pass filter is photoelectrically converted by the color filter-equipped pixels arranged as shown in FIG. The electric signal of each pixel thus photoelectrically converted is amplified by an amplifier, converted into a digital signal by an A / D converter, and output as digital color signals Rs, Gs, Bs. The image sensor controller divides the color signals Rs, Gs, and Bs output from the A / D converter into Rs, Bs, and Gs, respectively, and outputs them to the G interpolation color difference calculation unit 102. Further, the image pickup device controller has a noise reduction processing function and a white balance processing function for the color signals Rs, Gs, and Bs, and these processing is performed on Rs, Gs, and Bs output to the G interpolation color difference calculation unit 102. Is given.

  The G interpolation color difference calculation unit 102 calculates Gi that is an interpolation G pixel corresponding to the pixel position of the input color signals Rs and Bs, and color difference signals R-Gi and B-Gi at the pixel position. Then, the G-interpolation color difference calculation unit 102 converts the color difference signals R-Gi and B-Gi into a two-dimensional array as shown in FIG. 7 and outputs the two-dimensional array of color difference signals to the color difference interpolation processing unit 103 in the raster scan order. To do. FIG. 7 is a diagram showing a two-dimensional array of R-Gi, B-Gi signals (an example of X-Gi signals) output from the G interpolation color difference calculation unit 102. Further, the G interpolation color difference calculation unit 102 outputs the G signal of the two-dimensional array to the RGB calculation unit 104 in the raster scan order as a two-dimensional array as shown in FIG. 6 for the G signal. FIG. 6 is a diagram showing a two-dimensional array of G signals output from the G interpolation color difference calculation unit 102. The G interpolation color difference calculation unit 102 will be described in more detail later.

  The chrominance interpolation processing unit 103 is a chrominance interpolation unit, and is missing in all pixels in the two-dimensional array as shown in FIG. The color difference BG is interpolated using the color difference R-Gi or B-Gi relating to the same surrounding color, and the color difference RG at all pixel positions and the color difference BG at all pixel positions obtained by the interpolation are interpolated. Are output to the RGB calculation unit 104, respectively.

  The RGB calculation unit 104 is an RGB pixel calculation unit, and includes two types of color difference signals RG and BG input from the color difference interpolation processing unit 103 and a G signal input from the G interpolation color difference calculation unit 102. , RGB signals are calculated, color matching processing and γ correction processing are performed on the calculated RGB signals to calculate Rγ, Gγ, Bγ signals, and the calculated Rγ, Gγ, Bγ signals are compressed. Output to the recording unit 105.

  The compression recording unit 105 converts the Rγ, Gγ, and Bγ signals input from the RGB calculation unit 104 into Y, U, and V signals, and further converts the Y, U, and V signals into a high-efficiency compression code such as JPEG or MPEG. The compressed data is converted to compressed data, and the compressed data is stored in a recording medium (for example, a flash memory, a hard disk, a magnetic tape, an optical disk, etc.).

  Next, FIG. 2 is a block diagram illustrating a configuration of the G interpolation color difference calculation unit 102. The configuration of the G interpolation color difference calculation unit 102 shown in FIG. 2 is a basic form of the configuration of the G interpolation color difference calculation unit 102 of another embodiment described later.

  The G interpolation color difference calculation unit 102 includes a memory 201, a memory 202, a vertical interpolation G calculation unit 203, a horizontal interpolation G calculation unit 204, a four-pixel average G calculation unit 205, a low-pass filter 206, and a subtraction unit 207. A subtractor 208, a subtractor 209, a memory 210, a memory 211, a memory 212, a peripheral similarity calculator 213, a peripheral similarity calculator 214, a peripheral similarity calculator 215, and a G variation. An amount calculation unit 216, a memory 217, a multiplication unit 218, a determination unit 219, a color difference selection unit 220, a memory 221, a memory 222, and a subtraction unit 223 are included.

  The color signals Rs and Bs output from the imaging unit 101 are obtained in the memory 201, and the color signal Gs is obtained in the memory 202 in order to obtain a delay until the pixels that can perform the two-dimensional interpolation processing of the missing G pixel position are completed. Are stored for a predetermined number of lines. In the example shown in FIG. 2, the number of lines stored in the memory 201 and the memory 202 is at least three lines.

  Here, although the R pixel or the B pixel is arranged at the position of the missing G pixel, these two color signals are collectively described as X or X pixel, and will be described hereinafter. .

  The vertical interpolation G calculation unit 203 calculates G pixel interpolation candidates by the interpolation formula Gv = (G1 + G2) / 2 using the vertical neighboring G pixels as shown in FIG. Interpolation candidate calculation means and first interpolation calculation means. FIG. 19 is a diagram showing positions on the Bayer array of adjacent G signals in the vertical direction used as vertical interpolation of the missing G signal positions.

  The horizontal interpolation G calculation unit 204 calculates a G pixel interpolation candidate by using an interpolation formula Gh = (G3 + G4) / 2 using the neighboring G pixels in the horizontal direction as shown in FIG. Interpolation candidate calculation means and second interpolation calculation means. FIG. 20 is a diagram showing the positions on the Bayer array of the adjacent G signals in the left-right direction used as the horizontal interpolation of the missing G signal positions.

  The 4-pixel average G calculation unit 205 calculates G pixel interpolation candidates by using the interpolation formula Ga = (G 1 + G 2 + G 3 + G 4) / 4 using the neighboring G pixels in the four vertical and horizontal directions as shown in FIG. 21, and outputs them to the subtraction unit 209. G pixel interpolation candidate calculation means and third interpolation calculation means. FIG. 21 is a diagram showing positions on the Bayer array of adjacent G signals in the vertical and horizontal directions used as adjacent 4-pixel interpolation of missing G signal positions.

  Further, the X pixel located at the same position as the missing G pixel stored in the memory 201 is input to the subtracting units 207 and 208 and further passed to the subtracting unit 209 as an XL pixel via the low pass filter 206 serving as a band limiting unit. Entered.

Here, as shown in FIG. 27, the low-pass filter 206 has a missing G pixel X (i, j) to be processed, and the same type X pixel X (i, j + 2), X (i) positioned above, below, left and right. , J−2), X (i−2, j), and X (i + 2, j), the XL pixel at the position (i, j) is calculated. FIG. 27 is a diagram showing the positions on the Bayer array of the color signals X used for configuring the low-pass filter in the low-pass filter 206. At this time, the frequency characteristic of the low-pass filter is as shown by a curve fc in FIG. The low-pass filter is as follows.
XL (i, j) = [alpha] X (i, j) + [beta] {X (i, j + 2) + X (i, j-2)
+ X (i−2, j) + X (i + 2, j)}
Here, α and β are weight values for approximating the spatial frequency characteristics of the XL pixel to the spatial frequency characteristics of Ga, which is calculated by averaging the four surrounding pixels of the missing G pixel, in a 45-degree oblique direction.

  The color difference signals X-Gv, X-Gh, and XL-Ga calculated by the subtraction units 207, 208, and 209, which are color difference candidate calculation means, are stored in the memories 210, 211, and 212, respectively.

  The memories 210, 211, and 212 are provided in order to obtain a delay that can perform the similarity calculation processing by the peripheral similarity calculation units 213, 214, and 215 in the subsequent stage. In the present embodiment, the peripheral similarity calculation units 213, 214, and 215 calculate the similarity using a neighborhood area that includes three rows and three columns of color difference signals, as shown in FIGS. For this reason, the number of lines stored in the memories 210, 211, and 212 is at least five.

  FIG. 22 is a diagram showing the position on the Bayer array of the color difference signals used for calculating the peripheral similarity among the color differences calculated by the vertical interpolation at the missing G signal position, and FIG. 23 is the horizontal direction at the missing G signal position. FIG. 24 is a diagram showing the position on the Bayer array of color difference signals used for calculating the peripheral similarity among the color differences calculated by interpolation; FIG. 24 is a calculation of the peripheral similarity of the color differences calculated by adjacent 4-pixel interpolation at the missing G signal position. It is a figure which shows the position on the Bayer arrangement | sequence of the color difference signal used for FIG.

  In this way, when the memory 210, 211, 212 stores the color difference signals sufficient to execute the color difference peripheral similarity calculation process, the memory 210, 211, 212 stores the color difference signals X-Gv, X-Gh, XL-. Ga is output to the peripheral similarity calculation units 213, 214, and 215, respectively.

The peripheral similarity calculation unit 213 is an optimum color difference selection unit, a selection unit, and calculates the color difference peripheral similarity Sv (k, l) at the missing G pixel position (k, l) as shown in FIG. Color difference signal (X-Gv) k, l (where "(X-Gv) k, l" is abbreviated as "X (k, l) -Gv (k, l)") The same applies hereinafter.) And eight color differences (X-Gv) k-2, l-2, (X-Gv) k, l-2, (X-Gv) k + 2, l-2, (X-Gv) k-2, l, (X-Gv) k + 2, l, (X-Gv) k-2, l + 2, (X-Gv) k, l + 2, (X-Gv ) Sv1 (k, l) based on k + 2, l + 2, (X ^ -Gv) k-1, l-1, (X ^ -Gv) k + 1, l-1, (X ^ It is defined and calculated as the sum of -Gv) k-1, l + 1 and (X ^ -Gv) k + 1, l + 1.
Sv (k, l) = Sv1 (k, l) + Sv2 (k, l)
Sv1 (k, l) = {| (X-Gv) k, l-2- (X-Gv) k, l |
+ | (X−Gv) k, l + 2− (X−Gv) k, l |} × WvcX
+ {| (X-Gv) k-2, l-2- (X-Gv) k-2, l |
+ | (X-Gv) k-2, l + 2- (X-Gv) k-2, l |
+ | (X−Gv) k + 2, l−2− (X−Gv) k + 2, l |
+ | (X−Gv) k + 2, l + 2− (X−Gv) k + 2, l |} × WvaX
Sv2 (k, l) = {| (X ^ -Gv) k-1, l-1- (X ^ -Gv) k-1, l + 1 |
+ | (X ^ −Gv) k + 1, l−1− (X ^ −Gv) k + 1, l + 1 |} × WvX ^
Here, the weights WvcX, WvaX and WvX ^ used for the calculation are
WvcX + 2 × WvaX = WvX ^
Are in a relationship.

Similarly, the peripheral similarity calculation unit 214 is an optimum color difference selection unit and selection unit, and the color difference peripheral similarity Sh (k, l) at the missing G pixel position (k, l) is shown in FIG. , Center color difference signal (X-Gh) k, l and its surrounding eight color differences (X-Gh) k-2, l-2, (X-Gh) k, l-2, (X-Gh) k + 2, l-2, (X-Gh) k-2, l, (X-Gh) k + 2, l, (X-Gh) k-2, l + 2, (X-Gh) k, Sh + (k, l) based on l + 2, (X-Gh) k + 2, l + 2, (X ^ -Gh) k-1, l-1, and (X ^ -Gh) k + 1 , l−1, (X ^ −Gh) k−1, l + 1, and (X ^ −Gh) k + 1, l + 1, and Sh2 (k, l).
Sh (k, l) = Sh1 (k, l) + Sh2 (k, l)
Sh1 (k, l) = {| (X-Gh) k-2, l- (X-Gh) k, l |
+ | (X−Gh) k + 2, l− (X−Gh) k, l |} × WhcX
+ {| (X-Gh) k-2, l-2- (X-Gh) k, l-2 |
+ | (X-Gh) k + 2, l-2- (X-Gh) k, l-2 |
+ | (X-Gh) k-2, l + 2- (X-Gh) k, l + 2 |
+ | (X−Gh) k + 2, l + 2− (X−Gh) k, l + 2 |} × WhaX
Sh2 (k, l) = {| (X ^ -Gh) k-1, l-1- (X ^ -Gh) k + 1, l-1 |
+ | (X ^ -Gh) k-1, l + 1- (X ^ -Gh) k + 1, l + 1 |} × WhX ^
Here, WhcX, WhaX and WhX ^ which are weights used for calculation are
WhcX + 2 × WhaX = WhX ^
Are in a relationship.

Further, the peripheral similarity calculation unit 215 is an optimum color difference selection unit and selection unit, and the color difference peripheral similarity Sa (k, l) at the missing G pixel position (k, l) is shown in FIG. Central color difference signal (XL-Ga) k, l and eight color differences (XL-Ga) k-2, l-2, (XL-Ga) k, l-2, (XL-Ga) k + 2, l-2, (XL-Ga) k-2, l, (XL-Ga) k + 2, l, (XL-Ga) k-2, l + 2, (XL-Ga) k, l +2, (XL-Ga) k + 2, l + 2, Sa1 (k, l), (X ^ L-Ga) k-1, l-1, and (X ^ L-Ga) k + 1, l-1, (X ^ L-Ga) k-1, l + 1, and (X ^ L-Ga) K + 1, l + 1 based on Sa2 (k, l) calculate.
Sa (k, l) = Sa1 (k, l) + Sa2 (k, l)
Sa1 (k, l) = {| (XL-Ga) k-2, l-2- (XL-Ga) k, l |
+ | (XL-Ga) k, l-2- (XL-Ga) k, l |
+ | (XL-Ga) k + 2, l-2- (XL-Ga) k, l |
+ | (XL-Ga) k-2, l- (XL-Ga) k, l |
+ | (XL-Ga) k + 2, l- (XL-Ga) k, l |
+ | (XL-Ga) k-2, l + 2- (XL-Ga) k, l |
+ | (XL-Ga) k, l + 2- (XL-Ga) k, l |
+ | (XL-Ga) k + 2, l + 2- (XL-Ga) k, l |} × Wax
Sa2 (k, l) = {| (X ^ L-Ga) k-1, l-1- (X ^ L-Ga) k + 1, l + 1 |
+ | (X ^ L-Ga) k-1, l + 1- (X ^ L-Ga) k + 1, l-1 |} × Wax
Here, WaX and WaX ^ which are weights used for calculation are
4 x WaX = WaX ^
Are in a relationship.

  Of the three color difference peripheral similarities Sv (k, l), Sh (k, l) and Sa (k, l) calculated in this way, Sv (k, l) and Sh (k, l) Is directly determined, and Sa (k, l) is an optimum color difference selection means and is multiplied by the G variation amount weighting coefficient w (k, l) stored in the memory 217 by the multiplication section 218 as the multiplication means, respectively, and then determined. Input to the unit 219.

  Here, the G variation amount weighting coefficient w (k, l) will be described.

The G variation amount calculation unit 216 is an optimum color difference selection unit and a G pixel variation amount calculation unit, and among the Gs signals stored in the memory 202, a missing G pixel position (i, j) as shown in FIG. ) Is read, and the G fluctuation amount (i, j) is calculated as shown below. FIG. 25 is a diagram showing the position of the G signal used by the G fluctuation amount calculation unit 216 on the Bayer array.
G variation (i, j) = Dc (i, j) / P
+ {Dr1 (i, j) + Dr2 (i, j) + Dr3 (i, j) + Dr4 (i, j)} / Q
Here, P is a weighting coefficient for the fluctuation amount in the central portion, Q is a weighting coefficient for the fluctuation amount in the peripheral portion, and is an arbitrary constant satisfying P> 0 and Q> 0. And
Dc (i, j) = | Gs (i-1, j) -Gs (i, j-1)
+ Gs (i + 1, j) -Gs (i, j + 1) |
+ | Gs (i-1, j) + Gs (i, j-1)
−Gs (i + 1, j) −Gs (i, j + 1) |
+ | Gs (i-1, j) -Gs (i, j-1)
−Gs (i + 1, j) + Gs (i, j + 1) |
Dr1 (i, j) = | Gs (i-2, j-1) -Gs (i-1, j-2)
+ Gs (i, j-1) -Gs (i-1, j) |
+ | Gs (i-2, j-1) + Gs (i-1, j-2)
−Gs (i, j−1) −Gs (i−1, j) |
+ | Gs (i-2, j-1) -Gs (i-1, j-2)
-Gs (i, j-1) + Gs (i-1, j) |
Dr2 (i, j) = | Gs (i, j-1) -Gs (i + 1, j-2)
+ Gs (i + 2, j-1) -Gs (i + 1, j) |
+ | Gs (i, j-1) + Gs (i + 1, j-2)
−Gs (i + 2, j−1) −Gs (i + 1, j) |
+ | Gs (i, j-1) -Gs (i + 1, j-2)
−Gs (i + 2, j−1) + Gs (i + 1, j) |
Dr3 (i, j) = | Gs (i, j + 1) -Gs (i + 1, j)
+ Gs (i + 2, j + 1) -Gs (i + 1, j + 2) |
+ | Gs (i, j + 1) + Gs (i + 1, j)
−Gs (i + 2, j + 1) −Gs (i + 1, j + 2) |
+ | Gs (i, j + 1) -Gs (i + 1, j)
-Gs (i + 2, j + 1) + Gs (i + 1, j + 2) |
Dr4 (i, j) = | Gs (i-2, j + 1) -Gs (i-1, j)
+ Gs (i, j + 1) -Gs (i-1, j + 2) |
+ | Gs (i-2, j + 1) + Gs (i-1, j)
−Gs (i, j + 1) −Gs (i−1, j + 2) |
+ | Gs (i-2, j + 1) -Gs (i-1, j)
-Gs (i, j + 1) + Gs (i-1, j + 2) |
It is.

  Based on the G variation amount (i, j) calculated in this way, the G variation amount calculation unit 216 calculates a weight coefficient w (i, j) using a function having a shape as shown in FIG. . FIG. 26 is a diagram showing the relationship between the G fluctuation amount calculated by the G fluctuation amount calculation unit 216 and the weighting coefficient for the peripheral similarity calculated using the adjacent four-pixel interpolation. The function shown in FIG. 26 sets the weight to 1 when the G fluctuation amount is equal to or greater than the predetermined threshold Th, and is proportional to the G fluctuation amount when the G fluctuation amount is less than the threshold Th (that is, , The weight becomes smaller as the G variation becomes smaller). Such a relationship of the weight coefficient w (i, j) with respect to the G fluctuation amount is calculated based on a predetermined calculation formula or converted by referring to a lookup table stored in advance. ing.

  The weighting coefficient w (i, j) calculated by the G variation calculation unit 216 is output to the memory 217 and stored.

  As described above, when the G variation amount is less than the predetermined threshold Th by multiplying the peripheral similarity Sa of the color difference XL−Ga obtained by using the 4-pixel interpolation Ga by a weighting coefficient w of 1 or less (that is, w In the case of <1>, Sa × w becomes a value smaller than that of Sa alone, so that the possibility that Sa × w becomes the minimum value in the determination by the determination unit 219 increases. That is, in a flat region where the amount of G fluctuation is small, the possibility of selecting vertical interpolation G or horizontal interpolation G is reduced, and 4-pixel interpolation G is selected with a higher probability, and the influence of noise is cancelled. It is like that.

  The determination unit 219 serving as the optimum color difference selection unit calculates the values of the three color difference peripheral similarities Sv (k, l), Sh (k, l), and Sa (k, l) × w (k, l) as described above. In comparison, one G interpolation method that gives the minimum color difference peripheral similarity is selected, and a selection signal corresponding to the selected interpolation method is output to the color difference selection unit 220. When there are a plurality of minimum color difference peripheral similarities, the determination unit 219 determines, for example, Sa (k, l) × w (k, l), Sh (k, l), Sv (k, l). The order of priority is determined in this order, and one G interpolation method is selected along this priority.

  The color difference selection unit 220 is an optimum color difference selection unit, a selection unit, and based on the selection signal input from the determination unit 219, the color difference candidate of one missing G pixel position (k, l) corresponding to the selection signal That is, the color difference candidate (X−Gv) k, l stored in the memory 210, the color difference candidate (X−Gh) k, l stored in the memory 211, and the color difference candidate (XL) stored in the memory 212 -Ga) Either k or l is selected and input, and is output as a color difference to the color difference interpolation processing unit 103. More specifically, the color difference selection unit 220 sets (X−Gv) k, l when the color difference peripheral similarity Sv (k, l) is the minimum, and sh (k, l) when the Sh (k, l) is the minimum. (X−Gh) k, l, and when Sa (k, l) × w (k, l) is minimum, (XL−Ga) k, l is subtracted by the subtracting unit 223 and the color difference interpolation processing unit. 103, respectively. The color difference signals are output in the raster scan order from the upper left to the lower right in the two-dimensional array as shown in FIG.

  The color difference signal output from the color difference selection unit 220 to the color difference interpolation processing unit 103 is divided into R-Gi and B-Gi in the color difference interpolation processing unit 103, and the pixel position where R-Gi is missing is the peripheral. Interpolation is performed using R-Gi, and similarly, pixel positions where B-Gi is missing are interpolated using neighboring B-Gi.

  Further, the memory 221 is for temporarily saving the X pixel value existing as the imaging pixel, and the timing at which any one of the color difference signals X-Gv, X-Gh, or XL-Ga described above is selected and output. It is for adjusting to.

Subsequently, the subtraction unit 223 performs subtraction as shown below based on the color difference X-Gv, X-Gh, or XL-Ga output from the color difference selection unit 220 and X at the same pixel position as this color difference. Subtracting means that performs processing and calculates a Gi signal at the position of the missing G pixel.
When color difference X-Gv is selected: Gi = X- (X-Gv)
When color difference X-Gh is selected: Gi = X- (X-Gh)
When color difference XL-Ga is selected: Gi = X- (XL-Ga)

  The memory 222 is for temporarily saving Gs pixels existing as image pickup pixels, and is for adjusting the timing at which Gi for the above-mentioned missing G pixel position is calculated and output.

  Thus, a two-dimensional array of G signals having no omission as shown in FIG. 6 is output to the RGB calculation unit 104 in the raster scan order from the upper left to the lower right.

  Next, FIG. 3 is a block diagram illustrating a configuration of the color difference interpolation processing unit 103.

  The color difference interpolation processing unit 103 includes a color difference selection unit 301, a memory 302, a memory 303, an interpolation calculation unit 304, and an interpolation calculation unit 305.

  The color difference signal X-Gi from the G interpolation color difference calculation unit 102 is input to the color difference selection unit 301 of the color difference interpolation processing unit 103. When the color difference signal X-Gi is input, the color difference selection unit 301 inputs the color difference signal X-Gi to the memory 302 when the color difference signal X-Gi is R-Gi, and when the color difference signal X-Gi is B-Gi. Separately output to the memory 303.

  When the color difference signal R-Gi is input from the color difference selection unit 301, the memory 302 stores it. At this time, the memory 302 has a memory capacity that can store the number of lines necessary for interpolation processing of the missing pixel portion of the color difference signal R-Gi.

  Similarly, when the color difference signal B-Gi is input from the color difference selection unit 301, the memory 303 stores it. At this time, the memory 303 has a memory capacity that can store the number of lines necessary for interpolation processing of the missing pixel portion of the color difference signal B-Gi.

  Thus, when the interpolation process can be started, pixel data necessary for the interpolation process is output from the memories 302 and 303 to the interpolation calculation units 304 and 305, respectively.

ここに、Ｎｏは、上述した３×３行列内における非欠落画素Ｒ−Ｇiの数である。なお、欠落位置におけるＲ−Ｇiの値はゼロである。 The interpolation calculation unit 304 performs interpolation of the color difference signal RG by using, for example, the following linear interpolation formula, generates the color difference signal RG at all pixel positions, and outputs the color difference signal RG to the RGB calculation unit 104. To do.

Here, No is the number of non-missing pixels R-Gi in the 3 × 3 matrix described above. Note that the value of R-Gi at the missing position is zero.

ここに、Ｎｏは、上述した３×３行列内における非欠落画素Ｂ−Ｇiの数である。なお、欠落位置におけるＢ−Ｇiの値はゼロである。 Similarly, the interpolation calculation unit 305 interpolates the color difference signal BG by using, for example, a linear interpolation formula as shown below, generates the color difference signal BG at all pixel positions, and generates an RGB calculation unit. To 104.

Here, No is the number of non-missing pixels B-Gi in the 3 × 3 matrix described above. Note that the value of B-Gi at the missing position is zero.

  Next, FIG. 4 is a block diagram illustrating a configuration of the RGB calculation unit 104.

  The RGB calculation unit 104 includes a memory 401, an addition unit 402, an addition unit 403, a color matrix processing unit 404, a γ correction unit 405, a γ correction unit 406, and a γ correction unit 407. .

  The G signal from the G interpolation color difference calculation unit 102 is input and stored in the memory 401 of the RGB calculation unit 104. This memory 401 is for holding the G signal until the color difference signal RG and the color difference signal BG at the same pixel position as the G signal are input from the color difference interpolation processing unit 103 ( That is, it functions as a delay).

  Further, the color difference signal RG from the color difference interpolation processing unit 103 is input to the addition unit 402 of the RGB calculation unit 104. Then, the adding unit 402 reads out the G signal at the same position as the input color difference signal RG from the memory 401, adds the color difference signal RG and the G signal, generates an R signal, and generates a color matrix. The data is output to the processing unit 404.

  Further, the color difference signal BG from the color difference interpolation processing unit 103 is input to the addition unit 403 of the RGB calculation unit 104. Then, the adder 403 reads out the G signal at the same position as the input color difference signal BG from the memory 401, adds the color difference signal BG and the G signal, generates a B signal, and generates a color matrix. The data is output to the processing unit 404.

  Then, the memory 401 outputs the R signal calculated by the adding unit 402 and the G signal at the same pixel position as the B signal calculated by the adding unit 403 to the color matrix processing unit 404.

  By performing such processing, the R signal, the G signal, and the B signal at each pixel position are restored.

  When the restored R signal, G signal, and B signal are input, the color matrix processing unit 404 converts them into signals in a predetermined color space such as an sRGB space. Then, the color matrix processing unit 404 outputs the converted R signal to the γ correction unit 405, the converted B signal to the γ correction unit 406, and the converted G signal to the γ correction unit 407, respectively.

  If the converted R, B, and G signals input from the color matrix processing unit 404 are, for example, 12-bit signals, the γ correction units 405, 406, and 407 respectively perform γ correction on these signals to obtain 8 bits. R γ signal, B γ signal, and G γ signal converted to γ are generated and output to the compression recording unit 105.

  The subsequent processing by the compression recording unit 105 is as described above.

  Next, FIG. 29 is a flowchart showing processing performed by the G interpolation color difference calculation unit 102 having the configuration shown in FIG.

  The G-interpolation color difference calculation unit 102 is input with the single-plate Bayer array image captured by the imaging unit 101 and stored in the memory in the raster scan order. Then, as described above, the G interpolation color difference calculation unit 102 stores the Rs signal and the Bs signal in the memory 201 and the Gs signal in the memory 202 in the single-plate Bayer array image input in the raster scan order. Only to store each one.

  As described above, in a state where the predetermined amount of data is stored in the memories 201 and 202, the interpolation candidate value and the color difference candidate value of the missing G signal are stored in the memory. The number of lines × the number of line pixels = the number of N pixels is predetermined. The initial value (in this embodiment, the initial value of the number of lines = 5) is set (step S2900).

  Next, an average value Gv (i, j) of vertically adjacent G signals as shown in FIG. 19 is calculated as an interpolation candidate for the G signal missing position (i, j), and the G signal missing position (i , J) calculates a color difference (X−Gv) i, j based on X (i, j) (where X represents an R signal or B signal) and the interpolation candidate Gv (i, j), and the memory 210 is stored (step S2901).

  Subsequently, for the G signal missing position (i, j), an average value Gh (i, j) of the G signals adjacent to the left and right as shown in FIG. 20 is calculated as an interpolation candidate, and the G signal missing position (i , J) calculates a color difference (X−Gh) i, j based on X (i, j) (where X represents an R signal or B signal) and the interpolation candidate Gh (i, j), and the memory It stores in 211 (step S2902).

  Further, for the G signal missing position (i, j), an average value Ga (i, j) of G signals adjacent vertically and horizontally as shown in FIG. 21 is calculated as an interpolation candidate, and the G signal missing position (i , J) and X (i, j), X (i-2, j), X (i + 2, j), X (i, j−) 2) and X (i, j + 2) are used to perform a band limiting process (low-pass filter) that approximates the frequency characteristic in the oblique 45 degree direction to the characteristic of the interpolation filter when calculating the average value Ga (i, j). The color difference (XL−Ga) i, j is calculated based on the calculated XL (i, j) and the interpolation candidate Ga (i, j) and stored in the memory 212 (step S2903).

  Then, the G variation amount is calculated based on the G signal of the surrounding 12 pixels with respect to the G signal missing position (i, j), converted into a weighting coefficient w (i, j) corresponding to the G variation amount, The data is stored in the memory 217 (step S2904).

  At this time, it is determined whether or not the processing for N pixels has been completed (step S2905).

  If the processing for N pixels has not been completed yet, the process returns to step S2901, and (X-Gv) i, j, (X-Gh) i, j, (XL-Ga) i, The process of calculating j and w (i, j) and storing them in the memory is continued.

  On the other hand, if it is determined in step S2905 that the processing for N pixels has been completed, N is set to 1 (step S2906).

  Then, the color difference (X−Gv) calculated by the same interpolation type (X is the same color and G type is the same Gv) of the G signal missing position (k, l) stored in the memory. k, l and the same kind of color difference (X−Gv) k + n, l + m in the eight directions nearby (where n = −2, 0, 2, m = −2, 0, 2 and n And m are not equal to 0 at the same time), and the peripheral similarity Sv1 (k, l) is calculated based on the above, and the peripheral different color difference of the color difference (X−Gv) k, l, that is, the color is X Color difference (X ^ −Gv) k + n, l + m (where n = −1, 1, m = −1, 1) of different colors X ^ and G having the same Gv Based on this, the peripheral similarity Sv2 (k, l) is calculated, and these two peripheral similarities Sv1 (k, l) and Sv2 (k, l) are added to obtain the peripheral similarity Sv (k, l). Calculate (step S2907).

  Subsequently, the color difference (X−Gh) calculated by the same interpolation type (X is the same color and G type is the same Gh) of the G signal missing position (k, l) stored in the memory. ) K, l and its adjacent eight-direction color difference (X−Gh) k + n, l + m (where n = −2, 0, 2, m = −2, 0, 2 and n And m are not equal to 0 at the same time) and the peripheral similarity Sh1 (k, l) is calculated based on the above, and the peripheral different color difference of the color difference (X−Gh) k, l, that is, the color is X Color difference (X ^ −Gh) k + n, l + m (where n = −1, 1, m = −1, 1) of different colors X ^ and G having the same Gh Based on this, the peripheral similarity Sh2 (k, l) is calculated, and these two peripheral similarities Sh1 (k, l) and Sh2 (k, l) are added to obtain the peripheral similarity Sh (k, l). Calculate (step S2908).

  Further, the color difference (XL−Ga) calculated by the same interpolation type (XL is the same color and G type is the same Ga) of the G signal missing position (k, l) stored in the memory. k, l and the same kind of color difference (XL−Ga) k + n, l + m in the eight directions nearby (where n = −2, 0, 2, m = −2, 0, 2 and n m is not 0 at the same time.) and the peripheral similarity Sa1 (k, l) is calculated based on the above, and the peripheral different color difference of the color difference (XL−Ga) k, l, that is, the color is different from XL. Color difference (X ^ L-Ga) k + n, l + m of color X ^ L and G of the same type Ga (where n = -1,1, m = -1,1) The peripheral similarity Sa2 (k, l) is calculated based on the above, and the two peripheral similarities Sa1 (k, l) and Sa2 (k, l) are added to obtain the peripheral similarity Sa (k, l). Is calculated in memory 217. Paid has been that G variation weighting factor w (k, l) and multiplying (step S2909).

  Each of the peripheral similarity Sv (k, l), Sh (k, l), and Sa (k, l) × w (k, l) calculated in this way has a higher similarity as the value is smaller. It is the quantity which shows. In the subsequent process, a process of selecting one color difference candidate having the maximum similarity (minimum similarity value) is performed.

  That is, when the three peripheral similarity Sv (k, l), Sh (k, l), and Sa (k, l) × w (k, l) are calculated as described above, first, the peripheral similarity is calculated. Sv (k, l) is compared with the peripheral similarity Sh (k, l) (step S2910).

  If it is determined that the peripheral similarity Sv (k, l) is smaller than the peripheral similarity Sh (k, l), the peripheral similarity Sv (k, l) and the peripheral similarity Sa ( k, l) × w (k, l) is compared (step S2912).

  On the other hand, if it is determined in step S2910 that the peripheral similarity Sv (k, l) is equal to or higher than the peripheral similarity Sh (k, l), the peripheral similarity Sh (k, l) and the peripheral similarity are further increased. The degree Sa (k, l) × w (k, l) is compared (step S2911).

  If it is determined in step S2912 that the peripheral similarity Sv (k, l) is smaller than the peripheral similarity Sa (k, l) × w (k, l), the G signal missing position ( The color difference of k, l) is determined as (X−Gv) k, l and stored in the memory (step S2913).

  Further, when it is determined in step S2912 that the peripheral similarity Sv (k, l) is equal to or higher than the peripheral similarity Sa (k, l) × w (k, l), or in the above-described step S2911 When it is determined that the similarity Sh (k, l) is equal to or higher than the peripheral similarity Sa (k, l) × w (k, l), the color difference of the G signal missing position (k, l) is expressed as (XL -Ga) k, l is determined and stored in the memory (step S2914).

  Further, if it is determined in step S2911 that the peripheral similarity Sh (k, l) is smaller than the peripheral similarity Sa (k, l) × w (k, l), the G signal missing position (k , L) is determined as (X−Gh) k, l and stored in the memory (step S2915).

  Thus, after performing any of the processing of step S2913, step S2914, or step S2915, G (k, l) is obtained by subtracting the determined color difference (k, l) from X (k, l). The calculated G is stored in the memory (step S2916).

  Thereafter, it is determined whether or not the processing for the total number of pixels as the output image has been completed (step S2917).

  If it is determined that the processing of the total number of pixels has not been completed yet, the process returns to the above-described step S2901, and is unnecessary (X-Gv), (X-) stored in the memory. Gh), (XL−Ga), w, Sv, Sh, and Sa are replaced with newly calculated values, and the above-described processing is continuously performed.

  On the other hand, if it is determined in step S2917 that the process for the total number of pixels has been completed, the G interpolation color difference calculation process ends.

  Here, the reason why the G interpolation color difference calculation unit 102 is configured as described above will be described in detail.

  The three color difference candidates (X-Gv), (X-Gh), and (XL-Ga) are each vertically selected by selecting one that has the maximum peripheral color difference similarity (the minimum is the similarity value). Almost all are assigned to edge regions, horizontal edge regions, and other regions including diagonal edges. The reason is that, assuming that the color difference signals in the local area are spatially similar (that is, the color change is small), it is based on Gt interpolated in the same direction as the edge direction included in the local area. When the calculated multiple color differences X-Gt and the calculated multiple color differences X-Gt ′ based on Gt ′ interpolated in different directions are compared, the former approximates the G signal inherent in the subject. Therefore, the color difference becomes closer to the color difference originally possessed by the subject, and the color difference similarity in the surrounding area becomes higher. The reason why (XL-Ga) instead of (X-Ga) is used for the color difference bearing the oblique edge will be described below with reference to FIG. FIG. 28 is a diagram illustrating frequency characteristics in a 45-degree oblique direction with respect to the horizontal direction of the captured image, the interpolation pixel, and the like.

  In FIG. 28, a curve fa indicates a spatial frequency characteristic in an oblique 45 ° direction when an image formed on the image sensor via the lens and the optical low-pass filter is set to 0 ° in the horizontal pixel arrangement direction. ing. A curve fb shows the spatial frequency characteristics of Ga, which is created by averaging the top, bottom, left, and right G around the missing G pixel position in a 45-degree oblique direction. Further, the curve fc shows the spatial frequency characteristic in the oblique 45 degree direction of XL which is band-limited with respect to X by using the same kind of surrounding X.

  When the color difference (X-Ga) is used for the oblique edge, first, the frequency characteristics in the oblique direction of X and Ga are different as shown by the curves fa and fb in FIG. A color difference including many colors will be generated.

  Second, since Gi generated at the position of the missing G pixel is Gi = X− (X−Ga) = Ga, the frequency characteristic in the oblique direction becomes the curve fb in FIG. Since the frequency characteristic of the G pixel is the curve fa in FIG. 28 and is different from the curve fb, pixel values with low contrast and pixel values with high contrast appear alternately in the diagonal edge region, and the diagonal edge boundary appears. The problem is that it does not connect smoothly.

  On the other hand, when the color difference (XL−Ga) is used, the frequency characteristics of XL and Ga in the oblique 45 degree direction are the curve fc and the curve fb in FIG. The generation of false colors in the edge region can be further suppressed.

  Further, Gi generated at the position of the missing G pixel is Gi = X− (XL−Ga) = Ga + (X−XL), the first term has frequency characteristics as shown by the curve fb in FIG. Since the second term is a high-frequency component excluding the curve fc of FIG. 28 that is band-limited from the curve fa of FIG. 28 that is the original signal band, Gi that is the addition of these is a frequency characteristic approximated to the curve fa of FIG. Will get. That is, the pixel value is the same as that of the G pixel on the image sensor in the oblique edge region, and the oblique edge region can be reproduced more naturally.

  Therefore, the reproducibility of the oblique edge can be improved, and further, it is not necessary to prepare a separate G interpolation for the oblique edge, so that the circuit scale can be reduced.

  Incidentally, Gi generated at the missing G pixel position by (X−Gv) assigned to the vertical edge region or (X−Gh) assigned to the horizontal edge region is Gi = X− (X−Gv) = Gv, Or Gi = X- (X-Gh) = Gh. Since Gv and Gh each store frequency characteristics orthogonal to the edges, the pixel values are the same as those of the G pixels, and unnatural artifacts do not occur in the vertical and horizontal edge regions, and clean edges can be obtained. Can be reproduced.

  Further, the G-interpolation color difference calculation unit 102 maximizes the peripheral color difference similarity from the three color difference candidates (X-Gv), (X-Gh), and (XL-Ga) created by the three interpolation candidates at the missing G pixel position ( Although an example in which one of the minimum similarity values is selected has been shown, the same processing can be performed even if four or more color difference candidates are created when an increase in circuit scale is allowed. Needless to say, there are. That is, XL in the above (XL-Ga) is a signal whose band is limited to approximate the frequency characteristic of 45 degrees obliquely as shown by the curve fc in FIG. It is also possible to create XLθ that is band-limited so as to approximate the frequency characteristic in the θ-degree direction to the frequency characteristic in the same direction of Ga, and add it as a color difference candidate (XLθ-Ga). In this case, the same color X pixels necessary to calculate XLθ are {X (i, j), X (i, j-2), X (i, j + 2), X (i-2, j), It is considered that it is necessary to design a low-pass filter using not only X (i + 2, j)} but also surrounding peripheral X-color pixels. By doing so, it is possible to further reduce the false color with respect to an oblique edge other than the oblique 45 degree direction.

  According to the first embodiment, a missing G pixel that is an undetermined parameter when calculating a color difference is considered to be the most probable based on the premise that the color difference signal is locally highly correlated. One of the G pixel interpolation candidates can be selected. As a result, since the optimum missing G pixel is interpolated at the local edge of the image, generation of false colors can be suppressed, and further, the circuit scale can be reduced while suppressing the resolution deterioration of the edge portion. Can be small.

[Embodiment 2]
8 to 18 show Embodiment 2 of the present invention, and FIG. 8 is a block diagram showing the overall configuration of the image processing apparatus. In the second embodiment, parts that are the same as those in the first embodiment are given the same reference numerals and description thereof is omitted, and only differences are mainly described.

  The second embodiment further improves the image quality degradation due to the color difference determination error in the minute edge region and the difference in the noise level of the R, G and B signals, and additionally changes the image size to be recorded. It has become possible.

  The image processing apparatus of the present embodiment is configured as shown in FIG. 8, and the difference from the first embodiment shown in FIG. 1 is that the color difference interpolation processing unit 103 is replaced with an interpolation processing unit 801. This difference is caused by a difference in the interpolated G, RG, and BG pixel positions output to the RGB calculation unit 104. That is, in the first embodiment, these pixel positions are the same positions as the pixels imaged by the image sensor, but in the second embodiment, the pixel positions can be generated at arbitrary positions with respect to the pixel positions of the image sensor. In particular, when the resolution of the image after interpolation is the same as the resolution of the image sensor, the image is generated at a position shifted by 1/2 pixel both horizontally and vertically. Along with this, the G signal output from the G interpolation color difference calculation unit 102 is input to the interpolation processing unit 801 instead of the RGB calculation unit 104. In addition to the color differences RG and BG, the interpolation processing unit 801 further outputs a G signal to the RGB calculation unit 104. Further, the recorded image size is input to the interpolation processing unit 801.

  FIG. 9 is a block diagram illustrating a configuration of the interpolation processing unit 801.

  The interpolation processing unit 801 includes a color difference selection unit 301, a memory 901, a memory 902, a memory 903, an interpolation calculation unit 904, an interpolation calculation unit 905, an interpolation calculation unit 906, an interpolation coefficient calculation unit 907, A control unit 908 and a memory 909 are included.

  The interpolation processing unit 801 includes a G signal having a two-dimensional array structure as shown in FIG. 6 and a color difference X− having a two-dimensional array structure as shown in FIG. Gi signal is input.

  The color difference X-Gi signal from the G interpolation color difference calculation unit 102 is input to the color difference selection unit 301, separated into R-Gi and B-Gi, and stored in the memories 901 and 902, respectively. Here, the memories 901 and 902 store R-Gi and B-Gi corresponding to the number of lines necessary for interpolation processing to a predetermined position, respectively.

  When the interpolation process can be started, the pixels necessary for the interpolation process are respectively input to the interpolation calculation units 904 and 905 serving as the color difference interpolation unit, and are interpolated based on the interpolation filter coefficients output from the control unit 908. Color differences RG and BG as pixels are calculated and output to the RGB calculation unit 104.

  On the other hand, the G signal from the G interpolation color difference calculation unit 102 is stored in the memory 903. The memory 903 stores G corresponding to the number of lines necessary for interpolation processing to a predetermined position.

  When the interpolation process can be started, pixels necessary for the interpolation process are input to the interpolation calculation unit 906 that is a G interpolation unit, and the interpolation pixel G is determined based on the interpolation filter coefficient output from the control unit 908. Calculated and output to the RGB calculation unit 104.

  When the recording image size determined by the user is input from a system controller (not shown), the control unit 908 causes the interpolation coefficient calculation unit 907 to apply an interpolation filter corresponding to the recording image size and its filter coefficient. Is instructed.

  The interpolation coefficient calculation unit 907 calculates an interpolation filter coefficient, and stores the calculated interpolation filter coefficient in the memory 909 according to the interpolation pixel position.

  The control unit 908 reads out the filter coefficient stored in the memory 909 according to the pixel position to be interpolated, and outputs it to the interpolation calculation units 904, 905, and 906.

  In the above description, the example in which the filter coefficient is created when the recording image size is input has been described. However, filter coefficients corresponding to a plurality of types of recording image sizes may be stored in the memory 909 in advance. In this case, the interpolation coefficient calculation unit 907 is not necessary. Then, the control unit 908 reads the coefficient lookup table stored in the memory 909 based on the recorded image size designated by the user, and supplies the coefficient lookup table to the interpolation calculation units 904, 905, and 906.

  The interpolation calculation processing in the interpolation calculation units 904, 905, and 906 is for an 8 × 8 pixel region as shown in FIGS. 10 is a diagram showing an example of the G signal interpolation processing unit in the interpolation processing unit 801. FIG. 11 is a diagram showing an example of the R-Gi signal interpolation processing unit in the interpolation processing unit 801. FIG. FIG. 10 is a diagram illustrating an example of a B-Gi signal interpolation processing unit in a unit 801. The interpolation calculation units 904, 905, and 906, as an example of designating an image size having the same resolution as that of the image sensor, are arranged at 1/2 pixel positions in both the horizontal and vertical directions (in FIG. 10 to FIG. The pixel is created at the position shown.

  As an example of the interpolation filter used in this interpolation calculation process, a convolution filter such as a Lanczos filter can be cited. The filtering process at this time is a process of calculating an interpolation pixel by applying a one-dimensional filter in the vertical direction to the result of the one-dimensional filter processed in the horizontal direction.

  Although a one-dimensional filter has been described here, a two-dimensional filter may be used instead. FIGS. 13 and 15 to 18 are diagrams illustrating examples of an 8 × 8 tap two-dimensional filter. FIG. 13 is a diagram showing filter coefficients for the G signal, and FIGS. 15 to 18 are diagrams showing interpolation filter coefficients for the R-Gi signal or the B-Gi signal. These four filter coefficients shown in FIGS. 15 to 18 are switched and used in accordance with the interpolation pixel position and the positional relationship of the color difference R-Gi or B-Gi to be processed.

  Here, the interpolation filter coefficient fij for the G signal and the interpolation filter coefficient f * ij for the color difference may be the same coefficient fij = f * ij, or different filter coefficients that are more band-limited for the color difference. It is possible to use f * ij ≠ fij.

  In the above description, the number of taps of the interpolation filter is 8 × 8 taps. However, N × N taps (where N is a multiple of 4) may be used as a trade-off between the hardware scale and the degree of freedom in design of the filter characteristics. . In this case, as a matter of course, the G signal and the color difference signal may have different tap numbers. In this case, the G signal may be a filter having a tap number in which N is a multiple of 2.

  In particular, the frequency characteristics of the interpolation filter for the G signal, whose recorded image size is the same resolution as that of the image sensor, is a low-pass filter whose response is zero or almost zero at the Nyquist frequency (NF) as shown by the solid line in FIG. Will be used. FIG. 14 is a diagram showing the frequency characteristics of the interpolation filter used in the interpolation processing unit 801.

  Also, when enlarging an image, a plurality of interpolation pixel positions are provided between the pixels, and the interpolation filter coefficient is changed according to the interpolation pixel position, whereby pixel interpolation having the same frequency characteristics as the interpolation filter is performed. Can be realized.

  On the other hand, when reducing the image, in order to suppress aliasing distortion that occurs in response to pixel decimation, interpolation is performed using an interpolation filter coefficient having a low-pass characteristic that cuts at the Nyquist frequency corresponding to the sampling interval of the reduced image. This can be realized by creating the interpolation pixels (G, RG, BG) while changing the interpolation filter coefficient according to the pixel position. The broken lines in FIG. 14 described above indicate, as an example, the filter characteristics of the frequency band when both horizontal and vertical are reduced to ½. In this case, the interpolated pixel position is ½ pixel position in both horizontal and vertical positions (indicated by a cross in the center in FIGS. 13 and 15 to 18), similarly to the positions illustrated in FIGS. 10 to 12. Created. Naturally, the interpolated pixels are created at twice the pixel interval of the image sensor in both the horizontal and vertical directions.

  The interpolation pixels (G, RG, BG) synchronized in the interpolation processing unit 801 in this way are converted into γ by the RGB matrix calculation unit 104 by color matrix processing and a non-linear tone curve as in the first embodiment. Corrected Rγ, Gγ, and Bγ signals are created, compressed by the compression recording unit 105, and recorded on a predetermined recording medium such as a flash memory, a hard disk, or a magnetic tape.

  According to the second embodiment, it is possible to suppress generation of a false color and a decrease in saturation as in the first embodiment, and to create a resized image. The circuit scale can be reduced as compared with the conventional technology. Here, the image resizing process generally performed is a case in which a separate resizing process is performed on an RGB image having the same number of pixels as the number of pixels of the image sensor output from the RGB calculation unit 104 of the first embodiment. Pointing.

  Furthermore, according to the second embodiment, the points of the first embodiment described below can be further improved.

  First, in the first embodiment, when a color difference selection error occurs in a minute edge region in a specific direction due to noise or the like of the image sensor, the G pixel value generated at the missing G pixel position and the original imaging The G pixel value at the G pixel position on the element is greatly different, and may appear as a deterioration pattern having a frequency component in the vicinity of the Nyquist frequency such that the edge is spilled. Such a deterioration pattern can be suppressed when the characteristic of the interpolation filter of the second embodiment (FIG. 14) becomes zero response at the Nyquist frequency, and the edge can be reproduced clearly.

  Second, the RGB pixels input to the G interpolation color difference calculation unit 102 generally have lower R and B pixel sensitivities than the G pixels. In such a case, in order to obtain white balance, the R and B pixels are amplified with a gain value larger than that of the G pixel, so that the noise of the R and B pixels becomes larger than that of the G pixel. become. When the technique of the first embodiment is used under such conditions, the G pixel value at the missing G pixel position is generated using the R or B pixel value including a lot of noise. The noise is larger than the pixel value. Even in such a case, according to the technique of the second embodiment, since the noise level is averaged by the interpolation filter, the SN ratio can be improved.

  Third, gain variation may occur between odd-numbered G pixels and even-numbered G pixels depending on the characteristics of the image pickup device itself. Such gain variation has a component near the Nyquist frequency particularly in a flat portion. Appears as a step. Even in such a case, according to the second embodiment, this step can be suppressed by using the above-described interpolation filter, so that a smooth gradation change can be obtained in the flat portion.

  As described above, according to the second embodiment, the occurrence of false color is suppressed at the edge portion as in the first embodiment, and the generation of an unnatural pattern at the edge boundary or flat portion is further suppressed. And an image of an arbitrary size designated by the user can be created.

[Embodiment 3]
30 to 32 show the third embodiment of the present invention, and FIG. 30 is a block diagram showing the configuration of the G interpolation color difference calculation unit. In the third embodiment, parts that are the same as those in the first and second embodiments are given the same reference numerals, description thereof is omitted, and only differences are mainly described.

  The third embodiment makes it possible to improve image quality under conditions where the configuration of the first embodiment described above does not function optimally. Here, the region under the condition where the configuration of the first embodiment does not function optimally includes, for example, a blooming occurrence region of the image sensor, a correlation between R or B and G is low, and particularly R or B is greater than G. Area, etc.

  Here, the blooming generation area is a state where the amount of charge generated in each pixel exceeds the storable charge amount when a light stronger than the allowable amount is incident on each pixel of the image sensor, and the saturation amount. This is a region where the above charges leak to the surrounding pixels. For example, when light having a strong red wavelength is incident, the R pixel is saturated, and charge leaks to the surrounding G and B pixels. Particularly problematic in such a blooming occurrence region, due to the structure of the image sensor, the charges leaking to the Gr pixel adjacent to the horizontal direction and the Gb pixel adjacent to the vertical direction of the saturated R pixel are not equal. Is the case. In this case, as shown in FIG. 31, the G signal has a pattern of Nyquist frequency both horizontally and vertically. Here, FIG. 31 is a diagram illustrating a state in which electric charge leaks unevenly to the peripheral G pixels in the blooming generation region for the X pixels. In FIG. 31, X corresponds to the R pixel, X ^ corresponds to the B pixel, GX corresponds to the Gr pixel, GX ^ corresponds to the Gb pixel, and the difference in hatching density between GX and GX ^ indicates the gradation level associated with blooming. Shows the difference.

In such a case, when the three types of color difference values are calculated using the technique of the first embodiment, the three types of color difference values are affected by the difference in the amount of charge leakage between the horizontal direction and the vertical direction. As a specific example, when the amount of charge leaking in the vertical direction is larger than the amount of charge leaking in the horizontal direction, the magnitude relationship between the three color difference values is:
(X-Gv) <(XL-Ga) <(X-Gh)
And so on. In this case, the three color difference peripheral similarities Sv (k, l), Sh (k, l), and Sa (k, l) × w (k, l) are related in the area where the similarity is calculated. It is determined by random noise of the image sensor and an uncertain difference in the amount of charge leaked, and is not uniquely determined. Here, the G variation amount weight coefficient w (k, l) for Sa (k, l) has a large G variation amount in a region where the difference between GX and GX ^ is large, and the weight coefficient value is 1. In addition, the priority of Sa (k, l) cannot be increased.

  In such a region where the three color difference peripheral similarity is not uniquely determined, and the color difference value is greatly different, a pattern in which the color is changed irregularly in a synchronized RGB image (subjectively dirty pattern) Will appear.

  Further, even when the charge accumulated in the X pixel is not saturated, the edge is inclined in the diagonal direction under the condition that the correlation between X and G is low such that X takes a large value with respect to G. If present, the following conditions may occur:

That is, as described in the first embodiment, the color difference peripheral similarity Sa (k, l) × w (k, l) is the highest at the diagonal edge, and the color difference (XL−Ga) in this case is the highest. And X are used to calculate the interpolation value Gi at the missing G pixel position as follows.
Gi = X- (XL-Ga) = Ga + (X-XL) = Ga + .DELTA.X
Here, ΔX corresponds to the high frequency component of X.

  The above Gi calculation formula can be assumed that when the correlation between X and G is high, the unknown high-frequency component predicted value ΔG of the G signal at the missing G pixel position is equal to ΔX, that is, ΔG = ΔX. For this reason, blunting of the edge in the oblique direction is suppressed, and reproducibility is improved. However, when the correlation between X and G is low, the above assumption does not hold and ΔG ≠ ΔX, and therefore, in the region where X is sufficiently larger than G, the unknown high-frequency component of the G signal that should be corrected originally There is a high possibility that ΔX becomes larger than necessary with respect to the predicted value ΔG, and in this case, jaggy appears as an oblique edge.

  In the third embodiment, such a point is further improved with respect to the G interpolation color difference calculation unit 102 of the first embodiment described above.

  As shown in FIG. 30, the G interpolation color difference calculation unit 102 of the third embodiment replaces the G fluctuation amount calculation unit 216 with the G fluctuation amount calculation unit 216 in comparison with the G interpolation color difference calculation unit 102 shown in FIG. In place of the amount calculation unit 3001, a saturation region determination unit 3002, a high color difference determination unit 3003, a 4-pixel average G calculation unit 3004, and a G interpolation selection unit 3005 are added, and the memory 217 is omitted.

  The memory 221 is also connected to the saturation region determination unit 3002. The memory 222 and the saturation region determination unit 3002 are connected to the G fluctuation amount calculation unit 3001. The G fluctuation amount calculation unit 3001 is connected to the multiplication unit 218. The memory 212 is connected to the G interpolation selection unit 3005 via the high color difference determination unit 3003. The determination unit 219 is also connected to the high color difference determination unit 3003. The memory 222 is connected to the G interpolation selection unit 3005 via the 4-pixel average G calculation unit 3004. The subtraction unit 223 is connected to the G interpolation selection unit 3005. The G signal is output from the memory 222 or the G interpolation selection unit 3005 to the RGB calculation unit 104.

  Next, the difference between the operation of the present embodiment and the first embodiment will be described.

  The saturation region determination unit 3002 serving as a saturation state determination unit performs the following saturation region determination using X (R or B) as shown in FIG. 31 stored in the memory 221 as an input.

As an example of the determination condition of the saturated region, for a 5 × 5 pixel region as shown in FIG. 31, based on threshold values THsatX and THsatX ^ in which the white balance coefficient is considered in the saturation level value of the image sensor,
Min {X (k, l), X (k-2, l), X (k, l-2)
, X (k + 2, l), X (k, l + 2)} ≧ THsatX
Or Min {X ^ (k-1, l-1), X ^ (k + 1, l-1)
, X ^ (k + 1, l + 1), X ^ (k-1, l + 1)} ≧ THsatX ^
In the case of satisfying the above, it may be determined that X (k, l) is in a saturated region (saturated state).

  Here, the saturation level value of the image sensor does not need to be the maximum gradation value, and may be set to a gradation value with a sufficiently low blooming occurrence probability. The threshold THsatX is THsatX = saturation level value × white balance coefficient X, and is a value clipped with the maximum gradation value.

  This determination result by the saturated region determination unit 3002 is output to the G fluctuation amount calculation unit 3001.

  The G variation amount calculation unit 3001 is an optimum color difference selection unit and a G pixel variation amount calculation unit, and is similar to that described in the first embodiment described above, and the missing G pixel as shown in FIG. The G fluctuation signal (k, l) is calculated by reading the Gs signal arranged around the position (k, l), and the weight coefficient w (k, l) as shown in FIG. Corresponding to k, l), a predetermined calculation formula or a lookup table is used. However, the G fluctuation amount calculation unit 3001 of the present embodiment further determines that the weight coefficient w when X (k, l) is determined to be in the saturation region based on the saturation region determination result. (K, l) is forcibly changed to 0 (minimum value) (that is, it is a weighting factor changing means and functions as a fixing means). By performing such processing, since Sa (k, l) × w (k, l) = 0, the color difference of the pixel X (k, l) located in the saturation region is determined by the determination unit 219. (XL-Ga) is forcibly selected, and the color difference selection in the saturation region can be uniquely determined. As a result, the occurrence of color spots as described above can be suppressed.

  In the above description, the case of selecting one of the three color difference candidates has been described as an example. However, the fourth (XLθ−Ga) or more color difference candidates as described in the first embodiment are also used. The above-described technique can also be applied in the same manner even when prepared. Even in this case, the weight coefficient may be set to 0 so that one color difference (XL-Ga) is selected.

  In the above description, as a means for forcibly selecting the color difference (XL−Ga), a means for setting the weighting coefficient w (k, l) to 0 has been described. However, the present invention is not limited to this. For example, the saturated region determination result is input to the determination unit 219, and the color difference (XL-Ga) is forcibly selected regardless of the similarity of the plurality of color difference candidates (that is, the determination unit 219 is used as a fixing unit). ), The same result can be obtained.

  Further, the high color difference determination unit 3003 serving as the high color difference determination unit inputs the color difference (XL−Ga) stored in the memory 212, and determines a predetermined threshold THc (an example of the threshold THc is the maximum floor). And a value of about 1/10 of the tone value). Then, the high color difference determination unit 3003 has a color difference (XL−Ga) larger than the threshold value THc, that is, (XL−Ga)> THc, and the selection signal output from the determination unit 219 has a color difference (XL− In the case of a signal for selecting Ga), a change request signal (ON / OFF) for changing the G interpolation is output to the G interpolation selection unit 3005 serving as a changing means.

  On the other hand, the 4-pixel average G calculation unit 3004 generates G (k−1, l), G (k, l−1) adjacent to the pixel X (k, l) in the vertical and horizontal directions from the G signal stored in the memory 222. ), G (k + 1, l), and average value of G (k, l + 1) are created as the interpolation Ga of the missing G pixel position (k, l), and the created interpolation Ga is output to the G interpolation selection unit 3005.

  The G interpolation selection unit 3005 outputs the G signal interpolation value Gi of the missing G pixel position (k, l) as an output from the subtraction unit 223 and the missing G pixel position (k as an output from the 4-pixel average G calculation unit 3004). , L) Ga is selected based on a change request signal that is an input from the high color difference determination unit 3003. That is, the G interpolation selection unit 3005 selects Ga calculated by the 4-pixel average G calculation unit 3004 when the change request is on, and calculates by the subtraction unit 223 when the change request is off. Select the G interpolation value.

  As a result, in a region having an X signal larger than the G signal and at an oblique edge portion, the G signal interpolation value Gi = Ga, and the high frequency component ΔX of X is not added, thereby suppressing the occurrence of jaggy. Can do.

When the color difference (XL−Ga) ≦ THc, the correlation between the X signal and the G signal is high, or the X signal is small relative to the G signal, and the influence of ΔX can be ignored. The interpolation value Gi of the missing G pixel position (k, l) at the oblique edge portion
Gi = X- (XL-Ga) = Ga + .DELTA.X
By doing so, it is possible to improve the resolution of the oblique edge in the region where the correlation between the X signal and the G signal is high.

  In the threshold determination, the determination may be made by comparing the absolute value or even power (for example, square, fourth power, etc.) of the color difference (XL−Ga) with the threshold THc. Here, by taking the absolute value or even power, it is determined even when the X signal is smaller than the G signal. Therefore, the presence or absence of the correlation between the X signal and the G signal is more strictly determined as a threshold value. It becomes possible to judge. However, when the X signal is smaller than the G signal, ΔX is also smaller than the G signal, so the degree of influence on the G signal is small.

  For the color difference (X−Gv) or (X−Gh) calculated using vertical interpolation or horizontal interpolation, the G signal interpolation value Gi at the missing G pixel position (i, j) is Gi = X− ( X−Gv) = Gv, or Gi = X− (X−Gh) = Gh, and since there is no ΔX term, the above-described problem occurs when these two color differences are selected. There is no. Therefore, when the color difference (X-Gv) or (X-Gh) is selected, it is not necessary to perform the high color difference determination and change of the interpolation value.

  Further, in the above description, the case where one of three color difference candidates is selected has been described as an example. However, the fourth (XLθ−Ga) or more color difference candidates as described in the first embodiment are used. Even when prepared, the above-described technique can be similarly applied. Even in this case, when the second term ΔX exists in the G signal interpolation value Gi at the missing G pixel position (i, j) as shown by Gi = Ga + ΔX = Ga + (X−XLθ). Can suppress the occurrence of jaggies for edges other than vertical and horizontal by setting the interpolation value Gi to Gi = Ga at the position determined to have a high color difference.

  Next, FIG. 32 is a flowchart showing processing performed by the G interpolation color difference calculation unit 102 having the configuration shown in FIG.

  The G-interpolation color difference calculation unit 102 is input with the single-plate Bayer array image captured by the imaging unit 101 and stored in the memory in the raster scan order. Then, as described above, the G interpolation color difference calculation unit 102 stores the Rs signal and the Bs signal in the memory 201 and the Gs signal in the memory 202 in the single-plate Bayer array image input in the raster scan order. Only to store each one.

  As described above, in a state where the predetermined amount of data is stored in the memories 201 and 202, the interpolation candidate value and the color difference candidate value of the missing G signal are stored in the memory. The number of lines × the number of line pixels = the number of N pixels is predetermined. The initial value (in this embodiment, the initial value of the number of lines = 5) is set (step S3200).

  Next, an average value Gv (i, j) of vertically adjacent G signals as shown in FIG. 19 is calculated as an interpolation candidate for the G signal missing position (i, j), and the G signal missing position (i , J) calculates a color difference (X−Gv) i, j based on X (i, j) (where X represents an R signal or B signal) and the interpolation candidate Gv (i, j), and the memory It stores in 210 (step S3201).

  Subsequently, for the G signal missing position (i, j), an average value Gh (i, j) of the G signals adjacent to the left and right as shown in FIG. 20 is calculated as an interpolation candidate, and the G signal missing position (i , J) calculates a color difference (X−Gh) i, j based on X (i, j) (where X represents an R signal or B signal) and the interpolation candidate Gh (i, j), and the memory It stores in 211 (step S3202).

  Further, for the G signal missing position (i, j), an average value Ga (i, j) of G signals adjacent vertically and horizontally as shown in FIG. 21 is calculated as an interpolation candidate, and the G signal missing position (i , J) and X (i, j), X (i-2, j), X (i + 2, j), X (i, j−) 2) and X (i, j + 2) are used to perform a band limiting process (low-pass filter) that approximates the frequency characteristic in the oblique 45 degree direction to the characteristic of the interpolation filter when calculating the average value Ga (i, j). The color difference (XL−Ga) i, j is calculated based on the calculated XL (i, j) and the interpolation candidate Ga (i, j), and stored in the memory 212 (step S3203).

  At this point, it is determined whether or not the processing for N pixels has been completed (step S3204).

  If the processing for N pixels has not been completed yet, the process returns to step S3201, and (X-Gv) i, j, (X-Gh) i, j, (XL-Ga) i, Continue to calculate j and store it in memory.

  On the other hand, if it is determined in step S3204 that the processing for N pixels has been completed, N is set to 1 (step S3205).

  Then, the G fluctuation amount is calculated based on the G signal of the surrounding 12 pixels with respect to the G signal missing position (k, l), and is further saturated based on the X signal of the surrounding 5 pixels and the X ^ signal of the surrounding 4 pixels. Is calculated and converted into a weighting coefficient w (k, l) corresponding to the G fluctuation amount and the saturation state (step S3206).

  The same interpolation type (X is the same color and G type is the same for the G signal missing position (k, l) stored in the memory by performing the processing loop from step S3201 to step S3204 described above. Color difference (X−Gv) k, l calculated by the following Gv) and the same kind of color difference (X−Gv) k + n, l + m in the eight directions in the vicinity thereof (where n = −2, 0, 2 and m = −2, 0, 2 and n and m are not 0 at the same time.), The peripheral similarity Sv1 (k, l) is calculated, and the color difference (X− Gv) Different color difference around k, l, that is, color difference X (X ^ -Gv) k + n, l + m (here, color X ^ which is different from X and G type is the same) , N = -1,1, m = -1,1) to calculate the peripheral similarity Sv2 (k, l), and these two peripheral similarities Sv1 (k, l) and Sv2 k, l) and adding to the calculating the surrounding similarity Sv (k, l) (step S3207).

  Subsequently, the color difference (X−Gh) calculated by the same interpolation type (X is the same color and G type is the same Gh) of the G signal missing position (k, l) stored in the memory. ) K, l and its adjacent eight-direction color difference (X−Gh) k + n, l + m (where n = −2, 0, 2, m = −2, 0, 2 and n And m are not equal to 0 at the same time) and the peripheral similarity Sh1 (k, l) is calculated based on the above, and the peripheral different color difference of the color difference (X−Gh) k, l, that is, the color is X Color difference (X ^ −Gh) k + n, l + m (where n = −1, 1, m = −1, 1) of different colors X ^ and G having the same Gh Based on this, the peripheral similarity Sh2 (k, l) is calculated, and these two peripheral similarities Sh1 (k, l) and Sh2 (k, l) are added to obtain the peripheral similarity Sh (k, l). Calculate (step S3208).

  Further, the color difference (XL−Ga) calculated by the same interpolation type (XL is the same color and G type is the same Ga) of the G signal missing position (k, l) stored in the memory. k, l and the same kind of color difference (XL−Ga) k + n, l + m in the eight directions nearby (where n = −2, 0, 2, m = −2, 0, 2 and n m is not 0 at the same time.) and the peripheral similarity Sa1 (k, l) is calculated based on the above, and the peripheral different color difference of the color difference (XL−Ga) k, l, that is, the color is different from XL. Color difference (X ^ L-Ga) k + n, l + m of color X ^ L and G of the same type Ga (where n = -1,1, m = -1,1) The peripheral similarity Sa2 (k, l) is calculated based on the above, and the two peripheral similarities Sa1 (k, l) and Sa2 (k, l) are added to obtain the peripheral similarity Sa (k, l). And further, step S32 G variation weighting factor w (k, l) calculated by 6 and multiplied (step S3209).

  Each of the peripheral similarity Sv (k, l), Sh (k, l), and Sa (k, l) × w (k, l) calculated in this way has a higher similarity as the value is smaller. It is the quantity which shows. In the subsequent process, a process of selecting one color difference candidate having the maximum similarity (minimum similarity value) is performed.

  That is, when the three peripheral similarity Sv (k, l), Sh (k, l), and Sa (k, l) × w (k, l) are calculated as described above, first, the peripheral similarity is calculated. Sv (k, l) is compared with the peripheral similarity Sh (k, l) (step S3210).

  If it is determined that the peripheral similarity Sv (k, l) is smaller than the peripheral similarity Sh (k, l), the peripheral similarity Sv (k, l) and the peripheral similarity Sa ( k, l) × w (k, l) is compared (step S3212).

  On the other hand, if it is determined in step S3210 that the peripheral similarity Sv (k, l) is equal to or higher than the peripheral similarity Sh (k, l), the peripheral similarity Sh (k, l) and the peripheral similarity are further increased. The degree Sa (k, l) × w (k, l) is compared (step S3211).

  In step S3212 described above, if it is determined that the peripheral similarity Sv (k, l) is smaller than the peripheral similarity Sa (k, l) × w (k, l), the G signal missing position ( The color difference of k, l) is determined as (X−Gv) k, l and stored in the memory (step S3213).

  If it is determined in step S3212 that the peripheral similarity Sv (k, l) is equal to or higher than the peripheral similarity Sa (k, l) × w (k, l), or in the above-described step S3211 When it is determined that the similarity Sh (k, l) is equal to or higher than the peripheral similarity Sa (k, l) × w (k, l), the color difference of the G signal missing position (k, l) is expressed as (XL -Ga) k, l is determined and stored in the memory (step S3214).

  Furthermore, when it is determined in step S3211 described above that the peripheral similarity Sh (k, l) is smaller than the peripheral similarity Sa (k, l) × w (k, l), the G signal missing position (k , L) is determined as (X−Gh) k, l and stored in the memory (step S3215).

  Thus, after performing any of the processing of step S3213, step S3214, or step S3215, G (k, l) is obtained by subtracting the determined color difference (k, l) from X (k, l). The calculated G is stored in the memory (step S3216).

  Further, the average value Ga of the four surrounding pixels G (k-1, l), G (k + 1, l), G (k, l-1), G (k, l + 1) around the G signal missing position (k, l). Is calculated (step S3217).

  Then, it is determined whether or not the color difference (k, l) determined at the G signal missing position (k, l) is (XL−Ga) k, l and the value is larger than the threshold value THc (step). S3218).

  Here, if it is determined that the color difference (k, l) is (XL−Ga) k, l and the value is larger than the threshold value THc, G (k, l) stored in the memory. ) Is replaced with the peripheral 4-pixel average value Ga (step S3219).

  If it is determined in step S3218 that the color difference (k, l) is not (XL−Ga) k, l, or the value of the color difference (k, l) is less than or equal to the threshold value THc, or the process of step S3219 is performed. In that case, it is then determined whether or not the processing of the total number of pixels as the output image has been completed (step S3220).

  If it is determined that the processing for the total number of pixels has not been completed yet, the process returns to step S3201 described above, and is no longer necessary (X-Gv), (X- Gh), (XL−Ga), Sv, Sh, and Sa are replaced with newly calculated values, and the above-described processing is continuously performed.

  On the other hand, if it is determined in step S3220 that the process for the total number of pixels has been completed, the G interpolation color difference calculation process is terminated.

  Needless to say, the technique of the third embodiment can be combined with the technique of the second embodiment described above without any problem.

  According to the third embodiment, the same effects as those of the first embodiment described above can be obtained, and deterioration of image quality under a wider imaging condition than that of the first embodiment can be suppressed to restore a high-quality color image. Can do.

  In each of the above-described embodiments, the image processing apparatus that generates a color image in which sampling positions are synchronized from an RGB Bayer array image has been mainly described. However, an image processing method for performing similar processing on a general-purpose processing circuit or the like. Alternatively, the image processing program may be executed by a computer to perform the same processing.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

  The present invention can be suitably used for an image processing apparatus, an image processing program, and an image processing method for processing RGB Bayer array images.

1 is a block diagram showing an overall configuration of an image processing apparatus according to Embodiment 1 of the present invention. FIG. 3 is a block diagram illustrating a configuration of a G interpolation color difference calculation unit according to the first embodiment. The block diagram which shows the structure of the color difference interpolation process part of the said Embodiment 1. FIG. FIG. 3 is a block diagram illustrating a configuration of an RGB calculation unit according to the first embodiment. In the said Embodiment 1, the figure which shows the Bayer arrangement | sequence of the color signal imaged with the single plate image sensor of the imaging part. In the said Embodiment 1, the figure which shows the two-dimensional arrangement | sequence of G signal output from the G interpolation color difference calculation part. In the said Embodiment 1, the figure which shows the two-dimensional arrangement | sequence of the R-Gi and B-Gi signal (an example of an X-Gi signal) output from the G interpolation color difference calculation part. The block diagram which shows the whole structure of the image processing apparatus in Embodiment 2 of this invention. The block diagram which shows the structure of the interpolation process part of the said Embodiment 2. FIG. The figure which shows an example of the interpolation process unit of G signal in the interpolation process part of the said Embodiment 2. FIG. The figure which shows an example of the interpolation process unit of the R-Gi signal in the interpolation process part of the said Embodiment 2. FIG. The figure which shows an example of the interpolation process unit of the B-Gi signal in the interpolation process part of the said Embodiment 2. FIG. The figure which shows the 2 * interpolation filter coefficient of 8x8 tap with respect to G signal in the interpolation process part of the said Embodiment 2. FIG. The diagram which shows the frequency characteristic of the interpolation filter used in the interpolation process part of the said Embodiment 2. FIG. The figure which shows the 2nd interpolation filter coefficient of the 1st 8x8 tap with respect to the R-Gi signal or B-Gi signal in the interpolation process part of the said Embodiment 2. FIG. The figure which shows the 2nd 8 * 8 tap two-dimensional interpolation filter coefficient with respect to the R-Gi signal or B-Gi signal in the interpolation process part of the said Embodiment 2. FIG. The figure which shows the 3rd 8x8 tap two-dimensional interpolation filter coefficient with respect to the R-Gi signal or B-Gi signal in the interpolation process part of the said Embodiment 2. FIG. The figure which shows the 4th 8 * 8 tap two-dimensional interpolation filter coefficient with respect to the R-Gi signal in the interpolation process part of the said Embodiment 2, or a B-Gi signal. The figure which shows the position on the Bayer arrangement | sequence of the adjacent G signal of the up-down direction used as vertical direction interpolation of the missing G signal position in the said Embodiment 1. FIG. The figure which shows the position on the Bayer arrangement | sequence of the adjacent G signal of the left-right direction used as horizontal direction interpolation of the missing G signal position in the said Embodiment 1. FIG. The figure which shows the position on the Bayer arrangement | sequence of the adjacent G signal of the up-down and left-right direction used as adjacent 4 pixel interpolation of the missing G signal position in the said Embodiment 1. FIG. In the said Embodiment 1, the figure which shows the position on the Bayer arrangement | sequence of the color difference signal used for the periphery similarity calculation of the color differences calculated by the vertical direction interpolation in the missing G signal position. FIG. 4 is a diagram illustrating a position on a Bayer array of color difference signals used for calculating a peripheral similarity among color differences calculated by lateral interpolation at a missing G signal position in the first embodiment. FIG. 4 is a diagram illustrating a position on a Bayer array of color difference signals used for calculating a peripheral similarity among color differences calculated by adjacent four-pixel interpolation at a missing G signal position in the first embodiment. The figure which shows the position on the Bayer arrangement | sequence of G signal used by the G fluctuation amount calculation part in the said Embodiment 1. FIG. In the said Embodiment 1, the diagram which shows the relationship between G variation | change_quantity calculated in a G variation | change_quantity calculation part, and the weighting coefficient with respect to the periphery similarity calculated using adjacent 4 pixel interpolation. The figure which shows the position on the Bayer arrangement | sequence of the color signal X used in order to comprise a low-pass filter in the said Embodiment 1. FIG. The figure which shows the frequency characteristic of 45 degrees diagonal with respect to horizontal directions, such as a captured image and an interpolation pixel, in the said Embodiment 1. FIG. 3 is a flowchart showing processing performed by a G interpolation color difference calculation unit having the configuration shown in FIG. 2 of the first embodiment. The block diagram which shows the structure of the G interpolation color difference calculation part of Embodiment 3 of this invention. In the said Embodiment 3, the figure which shows the state which the electric charge leaked unevenly to the periphery G pixel in the blooming generation | occurrence | production area | region with respect to X pixel. The flowchart which shows the process performed by the G interpolation color difference calculation part of the structure shown in FIG. 30 of the said Embodiment 3. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Imaging part 102 ... G interpolation color difference calculation part 103 ... Color difference interpolation process part (color difference interpolation means)
104... RGB calculation unit (RGB pixel calculation means)
105: Compression recording unit 201, 202 ... Memory 203 ... Vertical interpolation G calculation unit (G pixel interpolation candidate calculation means, first interpolation calculation means)
204 .. Horizontal interpolation G calculation unit (G pixel interpolation candidate calculation means, second interpolation calculation means)
205... 4 pixel average G calculation unit (G pixel interpolation candidate calculation means, third interpolation calculation means)
206: Low pass filter (band limiting means)
207, 208, 209 ... subtracting section (color difference candidate calculating means)
210, 211, 212 ... memory 213, 214, 215 ... peripheral similarity calculation unit (optimum color difference selection means, selection means)
216... G variation amount calculation unit (optimum color difference selection means, G pixel variation amount calculation means)
217 ... Memory 218 ... Multiplication unit (optimum color difference selection means, multiplication means)
219... Determination unit (optimum color difference selection means, fixing means)
220 ... Color difference selection section (optimum color difference selection means, selection means)
221, 222 ... Memory 223 ... Subtraction unit (subtraction means)
301 ... Color difference selection unit 302, 303 ... Memory 304, 305 ... Interpolation calculation unit 401 ... Memory 402, 403 ... Addition unit 404 ... Color matrix processing unit 405, 406, 407 ... γ correction unit 801 ... Interpolation processing unit 901, 902 903: Memory 904, 905: Interpolation calculation unit (color difference interpolation means)
906: Interpolation calculation unit (G interpolation means)
907: Interpolation coefficient calculation unit 908 ... Control unit 909 ... Memory 3001 ... G variation amount calculation unit (optimum color difference selection unit, G pixel variation amount calculation unit, fixing unit, weighting factor change unit)
3002 ... Saturation region determination unit (saturation state determination means)
3003 ... High color difference determination unit (high color difference determination means)
3004... 4-pixel average G calculation unit 3005... G interpolation selection unit (changing means)

Claims (15)

Image processing for processing an image including a pixel in which at least one color signal is missing among a plurality of color signals and generating an image including a pixel in which at least a first color signal is interpolated among the plurality of color signals A device,
The position in the image is represented as (x, y) (where x is an integer indicating the pixel position in the horizontal direction and y is an integer indicating the pixel position in the vertical direction), and the pixel having the first color signal is missing When a pixel having a second color signal different from the first color signal at the position (x0, y0) is X (x0, y0), for this X (x0, y0), Pixel interpolation candidates for calculating M types of first color signals Gt (x0, y0) (here, t = 1,..., M) interpolated in M types (M is an integer of 2 or more). A calculation means;
Band limiting means for calculating XL (x0, y0) band-limited to the above X (x0, y0) using peripheral similar color signals;
Based on the M types of first color signals Gt calculated by the pixel interpolation candidate calculation means, color difference candidates X (x0, y0) -Gt (x0, y0) or color difference candidates XL (x0, y0) -Gt Color difference candidate calculating means for calculating M types of color difference candidates of (x0, y0);
Peripheral position (x0 + n, y0 + m) of the same type t as the first color signal Gt (x0, y0) used in calculating the color difference candidate X (x0, y0) −Gt (x0, y0) (here In addition, based on the first color signal Gt (x0 + n, y0 + m) calculated for n and m, which are arbitrary integers that are not 0 at the same time, X calculated for the peripheral position (x0 + n, y0 + m) Based on the plurality of color difference candidates X (x0 + n, y0 + m) −Gt (x0 + n, y0 + m) and the color difference candidates X (x0, y0) −Gt (x0, y0),
Alternatively, the peripheral position (x0 + n, y0 + m) of the same type t as the first color signal Gt (x0, y0) used in calculating the color difference candidate XL (x0, y0) −Gt (x0, y0). Based on Gt (x0 + n, y0 + m) calculated in (here, n and m are arbitrary integers that are not 0 at the same time), a plurality of XL color differences calculated for the peripheral position (x0 + n, y0 + m) Based on the candidate XL (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) and the color difference candidate XL (x0, y0) -Gt (x0, y0), M types of color difference similarities are calculated,
Based on the calculated M kinds of color difference similarities, p kinds of the first color signals Gp (x0, y0) (here, out of M kinds of the first color signals Gt (x0, y0)) 1 ≦ p ≦ M) or q kinds of first color signals Gq (x0, y0) (here, 1 ≦ q ≦ M), color difference candidates X (x0, y0) −Gp (x0, y0) or optimum color difference selection means for selecting a color difference candidate XL (x0, y0) -Gq (x0, y0) as a color difference;
Subtracting the color difference X (x0, y0) -Gp (x0, y0) or the color difference XL (x0, y0) -Gq (x0, y0) selected by the optimum color difference selection means from the X (x0, y0). Calculating means for calculating the first color signal G (x0, y0) for the position (x0, y0),
An image processing apparatus comprising: A color difference interpolation unit that performs a process of interpolating a color difference at a position where a color difference is not calculated by the optimal color difference selection unit based on a color difference of the same color around the position calculated by the optimal color difference selection unit;
Pixel calculation for calculating a pixel including a plurality of color signals based on the color difference calculated by the optimum color difference selection unit or the color difference interpolation unit and the pixel having the first color signal at the same position as the color difference. Means,
The image processing apparatus according to claim 1, further comprising: A color difference interpolation means for calculating an interpolated color difference for the predetermined position based on a color difference of the same color around the predetermined position;
Based on the pixels having the first color signal around the predetermined position, and interpolation means for calculating an interpolated the first color signal in pair to the predetermined position,
Based on the first color signals the interpolated color difference calculated in the predetermined position and the interpolation pixel calculation means for calculating fraud and mitigating risk pixel including a plurality of color signals in the predetermined position,
The image processing apparatus according to claim 1, further comprising: The optimum color difference selection means calculates the color difference similarity as follows:
The color difference candidate X (x0, y0) -Gt (x0, y0) and a plurality of color difference candidates X (x0 + n) of the same color X and the same type t as the color difference candidate X (x0, y0) -Gt (x0, y0) , Y0 + m) −Gt (x0 + n, y0 + m), and
Similarity based on a plurality of color difference candidates X ^ (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) of a color X ^ different from the color difference candidates X (x0, y0) -Gt (x0, y0) and of the same type t. When,
The image processing apparatus according to claim 1, wherein the image processing apparatus is calculated on the basis of the above. The optimum color difference selection means calculates the color difference similarity as follows:
The color difference candidate XL (x0, y0) -Gt (x0, y0) and a plurality of color difference candidates XL (x0 + n) of the same color X and the same type t as the color difference candidate XL (x0, y0) -Gt (x0, y0) , Y0 + m) −Gt (x0 + n, y0 + m), and
Similarity based on a plurality of color difference candidates X ^ L (x0 + n, y0 + m) -Gt (x0 + n, y0 + m) of a color X ^ different from the color difference candidates XL (x0, y0) -Gt (x0, y0) and of the same type t Degree,
The image processing apparatus according to claim 1, wherein the image processing apparatus is calculated on the basis of the above. The similarity is the sum of absolute differences between color difference candidates of the same color X and the same type t,
The optimum color difference selecting means includes a selecting means for selecting one color difference candidate corresponding to one of the color difference similarities giving the minimum value among the M kinds of color difference similarities, and the color difference selected by the selecting means 6. The image processing apparatus according to claim 4, wherein the candidate is a color difference. When the color difference selected by the optimum color difference selection means is XL (x0, y0) −Gt (x0, y0), it is determined whether or not the color difference is a high color difference by comparing the color difference with a predetermined threshold value. High color difference determination means;
Changing means for changing G (x0, y0) to Gt (x0, y0) for the position (x0, y0) determined to be a high color difference by the high color difference determining means;
The image processing apparatus according to claim 1, further comprising: X (x0, y0) at the position (x0, y0) and X (x0 + u, y0 + v) at positions around the position (x0, y0) (where u and v are not 0 at the same time) ) And a saturation state determination means for determining a saturation state for the position (x0, y0),
When it is determined that the saturation state is determined by the saturation state determination unit, the color difference selected from the M types of color difference candidates by the optimum color difference selection unit is determined as one type of color difference determined in advance based on the saturation state. Immobilization means for immobilizing to,
The image processing apparatus according to claim 1, further comprising: 9. The image processing apparatus according to claim 1, wherein the first color signal is a G component, and the second color signal is a B component or an R component. The image processing apparatus according to claim 1, wherein the image is an RGB Bayer array image. The pixel interpolation candidate calculation means includes:
  First interpolation calculating means for calculating an average of two G pixels adjacent above and below the X pixel as one kind of G component signals of the M kinds of G component signals;
  A second interpolation calculating means for calculating an average of two G pixels adjacent to the left and right of the X pixel as one kind of G component signals of the M kinds of G component signals;
  Third interpolation calculation means for calculating an average of four G pixels adjacent to the X pixel in the vertical and horizontal directions as one G component signal of the M types of G component signals;
  The image processing apparatus according to claim 9, wherein the image processing apparatus is configured to include: The optimum color difference selecting means is:
  G pixel fluctuation amount calculating means for calculating a fluctuation amount of a G pixel value around a G pixel missing position indicating a position where a pixel having a G component signal is missing, and calculating a weighting coefficient based on the fluctuation amount; ,
  Multiplying means for multiplying the similarity between the color difference candidate at the G pixel missing position and the surrounding color difference candidate calculated based on the G component signal calculated by the third interpolation calculating means by the weight coefficient;
  The image processing apparatus according to claim 11, wherein the image processing apparatus is configured to include: X (x0, y0) at the position (x0, y0) and X (x0 + u, y0 + v) at positions around the position (x0, y0) (where u and v are not 0 at the same time) ) And a saturation state determination means for determining a saturation state for the position (x0, y0),
  A weight coefficient changing means for changing the weight coefficient calculated by the G pixel fluctuation amount calculating means to a minimum value when it is determined that the saturation state is determined by the saturation state determining means;
  The image processing apparatus according to claim 12, further comprising: An image including a pixel obtained by interpolating at least a first color signal of a plurality of color signals by causing a computer to process an image including a pixel in which at least one of the color signals is missing. An image processing program for generating
The position in the image is represented as (x, y) (where x is an integer indicating the pixel position in the horizontal direction and y is an integer indicating the pixel position in the vertical direction), and the pixel having the first color signal is missing When a pixel having a second color signal different from the first color signal at the position (x ₀ , y ₀ ) is X (x ₀ , y ₀ ), this X (x ₀ , y _{0 0} ), M types of the first color signals G _t (x ₀ , y ₀ ) (where t = 1,...) Interpolated in M types (M is an integer of 2 or more). , M) for calculating pixel interpolation candidates;
A band limiting step of calculating the band-limited by the _{X L (x 0, y 0} ) by using the same type color signals near to said _{X (x 0, y 0)} ,
Based on the M types of first color signals G _t calculated in the pixel interpolation candidate calculation step, color difference candidates X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) or color difference candidates X _L A color difference candidate calculating step of calculating M types of color difference candidates of (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ );
Surroundings of the same type t as the first color signal G _t (x ₀ , y ₀ ) used in calculating the color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) The first color signal G _t (x ₀ + n, y ₀ + m) calculated at the position (x ₀ + n, y ₀ + m) (where n and m are arbitrary integers that are not 0 simultaneously) A plurality of X color difference candidates X (x ₀ + n, y ₀ + m) −G _t (x ₀ + n, y ₀ + m) calculated for the peripheral position (x ₀ + n, y ₀ + m), Based on the color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ),
Alternatively, the same type t as the first color signal G _t (x ₀ , y ₀ ) used in calculating the color difference candidate X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ). , Based on G _t (x ₀ + n, y ₀ + m) calculated at the peripheral position (x ₀ + n, y ₀ + m) (where n and m are arbitrary integers that are not 0 simultaneously) near position _{(x 0 + n, y 0} + m) a plurality of color difference candidates X _L of the calculated X _L against _{(x 0 + n, y 0} + m) -G t (x 0 + n, y 0 + m) and, the Based on the color difference candidates X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ), M types of color difference similarities are calculated,
Based on the calculated M kinds of color difference similarities, the first color signal G _p (x ₀ ) of one kind p among the M kinds of first color signals G _t (x ₀ , y ₀ ). , Y ₀ ) (where 1 ≦ p ≦ M) or one type q of the first color signal G _q (x ₀ , y ₀ ) (where 1 ≦ q ≦ M) Optimal color difference selection for selecting a color difference candidate X (x ₀ , y ₀ ) −G _p (x ₀ , y ₀ ) or a color difference candidate X _L (x ₀ , y ₀ ) −G _q (x ₀ , y ₀ ) as a color difference Steps,
Color difference X (x ₀ , y ₀ ) −G _p (x ₀ , y ₀ ) or color difference X _L (x ₀ , y ₀ ) − selected from the above X (x ₀ , y ₀ ) by the optimum color difference selection step A calculation step of calculating the first color signal G (x ₀ , y ₀ ) for the position (x ₀ , y ₀ ) by subtracting G _q (x ₀ , y ₀ );
An image processing program for executing An image including a pixel in which at least one color signal of the plurality of color signals is missing is processed to generate an image including a pixel in which at least the first color signal of the plurality of color signals is interpolated. An image processing method for
The position in the image is represented as (x, y) (where x is an integer indicating the pixel position in the horizontal direction and y is an integer indicating the pixel position in the vertical direction), and the pixel having the first color signal is missing When a pixel having a second color signal different from the first color signal at the position (x ₀ , y ₀ ) is X (x ₀ , y ₀ ), this X (x ₀ , y _{0 0} ), M types of the first color signals G _t (x ₀ , y ₀ ) (where t = 1,...) Interpolated in M types (M is an integer of 2 or more). , M) for calculating pixel interpolation candidates;
A band limiting step of calculating the band-limited by the _{X L (x 0, y 0} ) by using the same type color signals near to said _{X (x 0, y 0)} ,
Based on the M types of first color signals G _t calculated in the pixel interpolation candidate calculation step, color difference candidates X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) or color difference candidates X _L A color difference candidate calculating step of calculating M types of color difference candidates of (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ );
Surroundings of the same type t as the first color signal G _t (x ₀ , y ₀ ) used in calculating the color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ) The first color signal G _t (x ₀ + n, y ₀ + m) calculated at the position (x ₀ + n, y ₀ + m) (where n and m are arbitrary integers that are not 0 simultaneously) A plurality of X color difference candidates X (x ₀ + n, y ₀ + m) −G _t (x ₀ + n, y ₀ + m) calculated for the peripheral position (x ₀ + n, y ₀ + m), Based on the color difference candidate X (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ),
Alternatively, the same type t as the first color signal G _t (x ₀ , y ₀ ) used in calculating the color difference candidate X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ). , Based on G _t (x ₀ + n, y ₀ + m) calculated at the peripheral position (x ₀ + n, y ₀ + m) (where n and m are arbitrary integers that are not 0 simultaneously) near position _{(x 0 + n, y 0} + m) a plurality of color difference candidates X _L of the calculated X _L against _{(x 0 + n, y 0} + m) -G t (x 0 + n, y 0 + m) and, the Based on the color difference candidates X _L (x ₀ , y ₀ ) −G _t (x ₀ , y ₀ ), M types of color difference similarities are calculated,
Based on the calculated M kinds of color difference similarities, the first color signal G _p (x ₀ ) of one kind p among the M kinds of first color signals G _t (x ₀ , y ₀ ). , Y ₀ ) (where 1 ≦ p ≦ M) or one type q of the first color signal G _q (x ₀ , y ₀ ) (where 1 ≦ q ≦ M) Optimal color difference selection for selecting a color difference candidate X (x ₀ , y ₀ ) −G _p (x ₀ , y ₀ ) or a color difference candidate X _L (x ₀ , y ₀ ) −G _q (x ₀ , y ₀ ) as a color difference Steps,
Color difference X (x ₀ , y ₀ ) −G _p (x ₀ , y ₀ ) or color difference X _L (x ₀ , y ₀ ) − selected from the above X (x ₀ , y ₀ ) by the optimum color difference selection step A calculation step of calculating the first color signal G (x ₀ , y ₀ ) for the position (x ₀ , y ₀ ) by subtracting G _q (x ₀ , y ₀ );
An image processing method comprising:

Images