JP5454156B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, in a digital camera, in order to acquire color information with a single-plate image sensor, for example, a color filter array based on a Bayer pattern is arranged on the image sensor to detect a plurality of color components (see, for example, Patent Document 1).

USP 3,971,065

  By the way, since a single-plate color image sensor using a color filter array can detect only one color component for each pixel, an interpolation process is required to create a color image having RGB information at each pixel. In addition, the obtained signal includes a noise component, and the influence of noise becomes large particularly in high-sensitivity shooting.

  An object of the present invention is to provide an image processing apparatus, an imaging apparatus, and an image processing program capable of performing interpolation processing of a color filter array image affected by noise with high accuracy.

  The image processing apparatus of the present invention is obtained by detecting one color component of a plurality of color components in each pixel by a color image sensor in which a plurality of color filters having different transmission characteristics are arranged on the front surface of the image sensor. In the image processing apparatus for performing data interpolation of the color filter image, photographing condition acquisition means for obtaining photographing conditions in the color image sensor, band division setting means for changing frequency band division according to the photographing conditions, An extraction unit that extracts a luminance and a plurality of color difference components by dividing a frequency band of a color filter image; and a color conversion unit that performs color conversion to a predetermined color component using the luminance and the plurality of color difference components. And

  Further, an imaging apparatus according to the present invention includes the image processing apparatus according to the present invention.

  The image processing program of the present invention detects one color component of a plurality of color components in each pixel by a color image sensor in which a plurality of color filters having different transmission characteristics are arranged on the front surface of the image sensor. An image processing program applied to an image processing apparatus that performs data interpolation of an obtained color filter image, the step of acquiring shooting conditions in the color image sensor, and changing a frequency band division according to the shooting conditions A step of dividing the frequency band of the color filter image to extract luminance and a plurality of color difference components, and a step of performing color conversion to a predetermined color component using the luminance and the plurality of color difference components. It is characterized by that.

  According to the image processing device, the imaging device, and the image processing program of the present invention, it is possible to perform interpolation processing of a color filter array image affected by noise with high accuracy.

It is a block diagram which shows the structure of the digital camera which concerns on embodiment. It is a figure for demonstrating the structure of the color filter array which concerns on embodiment. It is a block diagram which shows the structure of the image process part which concerns on embodiment. It is a figure for demonstrating image processing. It is a graph which shows the spectral quantum efficiency of a color image sensor. It is a graph which shows the spectral quantum efficiency of a color image sensor. It is a graph which shows the spectral quantum efficiency of a color image sensor. It is a graph which shows the filter characteristic of each parameter of a low-pass filter. It is a graph which shows the interpolation error of each color of RGB of Raw data. It is a graph which shows the interpolation precision by a color difference. It is a graph which shows the interpolation precision including the influence by noise. It is a figure for demonstrating the structure of the color filter array which concerns on embodiment. It is a graph which shows the result of having evaluated the interpolation precision by the dimension of the color difference. It is a graph which shows the noise after the interpolation seen from the color difference. It is a graph which shows the interpolation error seen by the color difference.

  A digital camera as an image processing apparatus (imaging apparatus) according to a first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the digital camera according to the first embodiment. The digital camera 1 forms a subject image on an image sensor 6 having a color filter array 4 on the incident surface by subject light incident through the photographing lens 2 and converts the subject image into an electrical signal by the image sensor 6. To do. Here, the color filter array 4 and the image sensor 6 constitute a color image sensor. For example, as shown in FIG. 2A, the color filter array 4 includes a green (G) color filter for the upper left pixel and the lower right pixel that is the diagonal of the four pixels, and an upper right pixel. A red (R) color filter and a Bayer color filter array (Bayer CFA) in which blue (B) color filters are arranged in an array with respect to the lower left pixel. In this embodiment, instead of the color filter array 4, for example, as shown in FIG. 2B, a green (G) color filter is used for the upper left pixel of the four pixels, and an upper right pixel is used. On the other hand, a red (R) color filter, a blue (B) color filter for the lower left pixel, and an emerald (E) color filter for the lower right pixel arranged in an array. In some cases, a color image sensor equipped with the above is used. Details will be described later.

  The electrical signal output from the image sensor 6 is input to the image processing unit 8. The image processing unit 8 also includes shooting conditions in the image sensor 6 acquired by the shooting condition acquisition unit 9, for example, the amount of image noise, the resolution of the shooting lens 2, image focusing information, image saturation, and image sensor 6. Is input. In the image processing unit 8, an image processing program read from a memory (not shown) or the like is executed, so that interpolation or the like is performed using parameters described later selected based on the shooting conditions input from the shooting condition acquisition unit 9. Image processing is performed. The image processed by the image processing unit 8 is recorded in the recording unit 10.

  FIG. 3 is a block diagram illustrating a configuration of the image processing unit 8 according to the first embodiment. The image processing unit 8 includes a white balance processing unit 12, a first gradation processing unit 14, an interpolation processing unit 16, a first color conversion processing unit 18, a second gradation processing unit 20, a second color conversion processing unit 22, an edge An enhancement processing unit 24 and a compression processing unit 26 are provided.

  The white balance processing unit 12 performs white balance processing so that the output of each color channel on the color image sensor is substantially the same when an achromatic subject is photographed. This processing may be performed, for example, on the assumption that the brightest area in the image is an achromatic color, and so that the RGB values of the area are the same. A corresponding gain may be set. Further, a gain may be set so that the photographing scene is determined and the face area of the person becomes skin color. The white balance processing unit 12 multiplies the RGB of each pixel by a white balance coefficient.

となるような特性の階調変換を行う。定数γは、γ＝２．２程度が良い。なおγ＝１として実質的に階調処理を行わないようにしても良い。 Next, in the first gradation processing unit 14, the output pixel value y is set to the input pixel value x called gamma processing.

Gradation conversion with such characteristics is performed. The constant γ is preferably about γ = 2.2. Note that the gradation processing may not be substantially performed with γ = 1.

Next, the interpolation processing unit 16 interpolates a CFA (Color Filter Array) image to generate a color image in which RGB is aligned for each pixel. As an interpolation method, Eric Dubois, “Frequency-Domain Methods for Demosaicking of Bayer-Sampled Color Images,” IEEE Signal Process. Lett., Vol.12, pp.847,2005 and Keigo Hirakawa and Patrick J. Wolfe, “Spatio- Spectral color Filter Array Design for Optimal image Recovery, "IEEE Tras. image process., vol. 17, pp. 1876, the luminance signal by dividing the CFA image in a frequency band as in the 2008 method described L, the color difference signals C ₁ , C ₂ components are extracted. Here, when an image obtained by the Bayer CFA is subjected to Fourier transform, when frequency analysis is performed, L components appear in the low frequency region and C ₁ and C ₂ components appear in the high frequency region. Here, L, C ₁ and C ₂ are respectively
L = (2G + B + R) / 4
C ₁ = (2G−B−R) / 4
C ₂ = (BR) / 4
It is represented by

Here, the luminance component L appears as a low frequency component in the Bayer CFA image, the color difference component C ₁ is modulated in a frequency range centered on (−1/2, −1/2), and the color difference component C ₂ is (0, − 1/2) and (−1/2, 0) are modulated and distributed in a frequency range centered on (−1/2, 0). The L, C ₁ , and C ₂ components are extracted using a low-pass filter or a band-pass filter, and a Bayer CFA image is interpolated to obtain a color image with RGB for each pixel. In this embodiment, a low-pass filter process (LPF process) that extracts L, C ₁ , and C ₂ components using a low-pass filter will be described as an example.

First, the color difference C ₁ component distributed around the (−1/2, −1/2) frequency when the Bayer CFA image is subjected to frequency analysis is extracted. For this purpose, the pixel value at the position of the coordinates (x, y) of the Bayer CFA image is multiplied by (−1) ^{x + y,} and the component distributed around the frequency of (−1/2, −1/2) is (0, 0). Is modulated so as to be distributed around the center frequency. Then, LPF processing is performed to extract a low-frequency component to obtain a C ₁ (x, y) component.

Next, a color difference C _2a component distributed around the frequency (−1/2, 0) when the Bayer CFA image is subjected to frequency analysis is extracted. Therefore, the pixel value at the position of the coordinates (x, y) of the Bayer CFA image is multiplied by (−1) ^x, and the component distributed around the frequency of (−1/2, 0) is set as the frequency of (0, 0). Modulate to distribute in the center. Then, LPF processing is performed to extract a low frequency component and set it as a C _2a (x, y) component.

Next, the color difference C _2b component distributed around the frequency (0, −1/2) when the Bayer CFA image is subjected to frequency analysis is extracted. For this purpose, the pixel value at the position of the coordinates (x, y) of the Bayer CFA image is multiplied by (−1) ^y, and the component distributed around the frequency of (0, −1/2) is set as the frequency of (0, 0). Modulate to distribute in the center. Then, LPF processing is performed to extract the low-frequency component to obtain a C _2b (x, y) component.

Next, a luminance L component when the Bayer CFA image is subjected to frequency analysis is extracted. The pixel value at the position of the coordinate (x, y) of the C ₁ component is multiplied by (−1) ^{x + y} to return to a component distributed around the frequency of (−1/2, −1/2), and C ′ ₁ ( x, y) component. Also, the pixel value at the position of the coordinates (x, y) of the C _2a component is multiplied by (−1) ^x to return to a component distributed around the frequency of (−1/2, 0), and C ′ _2a (x , Y) component. Also, the pixel value at the position of the coordinates (x, y) of the C _2b component is multiplied by (−1) ^y to return to the component distributed around the frequency (0, −1/2), and C ′ _2b (x , Y) component.

Next, C ′ ₁ , C ′ _2a , and C ′ _2b are subtracted from the Bayer CFA image to extract the L component (see the following equation).
L (x, y) = CFA (x, y) −C ′ ₁ (x, y) −C ′ _2a (x, y) −C ′ _2b (x, y)

  The L component may be extracted by directly applying an LPF to the CFA image. In this case, the color difference and the luminance are not extracted from the respective regions separated in the frequency band, but a partially overlapping region may be extracted for both the color difference and the luminance. The LPF may not be a linear filter, and may be extracted using a non-linear filter that depends on a difference in pixel values, such as an ε-filter, a σ-filter, or a bilateral filter. In this case, it is possible to extract luminance and color difference not only by the difference in band but also by the difference in amplitude.

_Next, determine the _{C 2} from _{C 2a} and _{C 2b} (see Equation 2).

としてαとβを定めてＣ_２を求めても良い。 This depends on the distribution of the power spectrum of the luminance component, that is, whether the region has vertical edges or horizontal edges, or the distribution of color difference components.

C ₂ may be obtained by determining α and β.

Next, using the L, C ₁ , and C ₂ components of each pixel obtained by the complement processing unit 16, the first color conversion processing unit 18 obtains the RGB value of each pixel from Equation 4.

Here, the CFA image is modulated and the color difference component is extracted by LPF processing. However, the CFA image is subjected to band pass filter processing and high pass filter processing to extract the color difference component, and then the extracted component is modulated. It may be a low frequency component. The pass band of the filter may be changed for each component of L, C ₁ , C _2a , and C _2b . Further, since the passbands of the filters suitable for the respective RGB components after the color conversion of Equation 4 may be different, the passbands of the filters for obtaining the respective RGB after the color conversion may be set. In this case, the complement process is required three times. Further, although Bayer CFA interpolation processing is shown here, the method of extracting a luminance / color difference component by dividing a CFA image in a frequency band is not limited to Bayer CFA, and can be applied to any CFA image. Different CFAs have different frequency bands in which color difference components are distributed, and different conversion matrices for converting luminance color difference components to RGB components. In addition, the interpolation here is performed only by LPF processing, but the CFA image may be subjected to Fourier transform or DCT transform to extract a luminance color difference band, or may be subjected to wavelet transform to consider local spatial information. Thus, the luminance color difference band may be extracted.

となるような特性の階調変換を行う。なお、γ=１の場合はこの処理をスキップできる。 Next, the second gradation processing unit 20 performs inverse conversion of the gradation processing performed in the first gradation processing unit 14. That is, the output pixel value y is equal to the input pixel value x.

Gradation conversion with such characteristics is performed. If γ = 1, this process can be skipped.

Next, the second color conversion processing unit 22 converts the RGB data of the color image sensor into RGB data R ₀ , G ₀ , B ₀ of the output color space (see Equation 6).

となるような特性で入力ＲＧＢを出力Ｒ'Ｇ'Ｂ'に階調変換を行う。 Note that c _ij is a constant determined from the spectral sensitivity, light source, and output color space of the image sensor 6. An example of the output color space standard is IEC 61966-2-1 sRGB. Further, gradation processing is performed according to the definition of the output color space. For example, in IEC 61966-2-1 sRGB, the output pixel value y is different from the input pixel value x.

The gradation conversion is performed from the input RGB to the output R′G′B ′ with such characteristics.

となる。 Further, color conversion from RGB to YCbCr may be performed for edge enhancement and compression processing. If this color conversion is R'G'B '

It becomes.

  Next, the edge enhancement processing unit 24 performs edge enhancement processing on the luminance component Y. Further, the compression processing unit 26 performs compression using a compression method such as JPEG, JPEG2000, or JPEG XR.

Here, the interpolation processing unit 16 optimizes the low-pass filter (hereinafter referred to as LPF) used to extract C ₁ , C _2a , and C _{2b according} to the shooting subject, shooting conditions, and characteristics of the color image sensor. A quality interpolation image is obtained. Consider the image processing shown in FIG. That is, an image obtained by adding an error due to spectral sensitivity of the image sensor (color image sensor) and an error due to color conversion to information on a subject visible to human eyes is an image 1, and an image obtained by adding an error due to interpolation to image 1 2 and an image obtained by adding an error due to noise to the image 3 and comparing these, the optimum interpolation parameter can be determined.

In this embodiment, five types of color image sensors TypeC 3ch (hereinafter referred to as C3ch), TypeN 3ch (hereinafter referred to as N3ch), TypeC 4ch (hereinafter referred to as C4ch), TypeN 4ch (hereinafter referred to as N4ch). ), TypeN2 4ch _{(hereinafter,} referred to as N 2 4ch.) is used. C3ch and N3ch are color image sensors including a Bayer CFA as shown in FIG. 2A, FIG. 5A is a graph showing spectral quantum efficiency of C3ch, and FIG. 5B is N3ch. Further, C4ch, N4ch and _N 2 4ch is a color image sensor having a 4-color CFA as shown in FIG. 2 (b), 6 (a) is C4ch, FIG. 6 (b) N4ch, 7 Is a graph showing the spectral quantum efficiency of N ₂ 4ch.

  In addition, as a subject shown in FIG. 4, an image P4 Standard Image (hereinafter referred to as an image P4), which is a standard spectral image of the Miyake Laboratory of Chiba University (Yoichi Miyake, “Introduction to Spectral Image Processing”, University of Tokyo Press 2006), is used. The interpolation of the image P4 is performed while changing the LPF parameters. FIG. 8 is a graph showing the filter characteristics of each order of the LPF parameter ftn0 (= 1 to 31). When the LPF is ftn0 = 1, the LPF extracts a frequency band from about 0 to about 0.2 as shown in graph A of FIG. Similarly, when the LPF is ftn0 = 31, the LPF extracts a frequency band from about 0 to about 0.5 as shown in the graph B of FIG. Graph C is ftn0 = 3, Graph D is ftn0 = 5, Graph E is ftn0 = 7, Graph F is ftn0 = 9, Graph G is ftn0 = 11, and Graph H is When ftn0 = 13, the graph I shows the filter characteristics when ftn0 = 15.

First, the interpolation accuracy of each color component of the Raw data will be examined. Specifically, Raw data (image data that has not been subjected to color conversion processing) that has undergone interpolation processing through the image sensor, and Raw data that has not been subjected to interpolation processing or color conversion processing (only through interpolation processing and The difference with the image data that has not been subjected to color conversion processing, that is, the interpolation error that does not consider the error due to color conversion shown in FIG. FIG. 9 is a graph showing the interpolation error (PSNR) of each RGB color of the Raw data in the color image sensor C3ch. Note that the interpolation error of each color of RGB of the Raw data in the color image sensor other than C3ch also shows the same tendency as in C3ch. Table 1 shows ftn0 where the interpolation error of green (G) is the smallest and the interpolation error (PSNR) at that time.

  As shown in Table 1, the optimum LPF characteristics vary depending on the characteristics of the color image sensor. Specifically, the G interpolation error in the color image sensor C3ch is the smallest compared to other color image sensors. This is because, as shown in FIG. 5A, since the overlap between the C3ch spectral quantum efficiency colors is large, the correlation is strong and the color difference is small. Similarly, not only in spectral quantum efficiency but also in spectral sensitivity, if the overlap between the colors is large, the color difference becomes small. Therefore, the color image sensor TypeC has the highest interpolation accuracy where ftn0 is small and the color difference band is narrow compared to TypeN.

Next, the interpolation accuracy of each color image sensor after color conversion of Raw data will be examined. For example, the difference between the image data (image 1 shown in FIG. 4) that has undergone color conversion through the image sensor and the image data (image 2 shown in FIG. 4) that has undergone interpolation processing and color conversion through the image sensor (image 2). chrominance _CIE94 indicated by 4 _1-2) comparing. Similarly, the difference (the color difference CIE94 _0-1 shown in FIG. 4) between the “information of the subject visible to the human eye” that does not pass through the image sensor shown in FIG. 4 and the image 1 shown in FIG. 4 is compared. FIG. 10 is a graph showing the interpolation accuracy by the color difference CIE94 (ΔE ^* ₉₄ ) of the color image sensor N3ch. The smaller the value of the color difference CIE ₉₄ (ΔE ^* ₉₄ ), which is the vertical axis of the graph shown in FIG. 10, the smaller the interpolation error (the higher the interpolation accuracy). Incidentally, the color image sensor other than N3ch, shows the same tendency as N3ch also interpolation accuracy by the color difference _{CIE94 1-2 (ΔE} ^* _94). Further, Table 2, in each color image sensor, interpolation accuracy by the color difference shows the color difference _{CIE94 1-2} when Ftn0, and a maximum.

  As shown in Table 2, the optimum LPF characteristics vary depending on the characteristics of the color image sensor. Table 1 shows that C3ch has the best interpolation accuracy, but Table 2 shows that N3ch has the best interpolation accuracy. This is because the error is amplified by the color conversion process, the amplification degree differs for each color image sensor, and Type N has a characteristic with a low amplification degree. In addition, since it is more appropriate to evaluate a digital image by a color difference (CIE94) than an interpolation error (PSNR) of Raw data, hereinafter, interpolation accuracy is evaluated by the color difference.

Next, the interpolation accuracy of each color image sensor considering the influence of noise (error due to noise shown in FIG. 4) will be examined. Specifically, assuming that the light source is CIE D65, the pixel pitch is 4 μm, and the baseline noise is 0, the model of H. Kuniba and RS Berns, Journal of Electronic Imaging, vol. 18 (2), 023002 (2009) is used. An image (hereinafter referred to as an image P) is created, and four types of image processing shown in FIG. 4 are performed, and each is compared. That is, the difference between the image 1 and the image 2 shown in FIG. 4 (color difference CIE94 _1-2 shown in FIG. 4), the difference between the image 2 and the image 3 shown in FIG. 4 (color difference CIE94 _2-3 shown in FIG. 4), The difference between the image 1 shown in FIG. 4 and the image 3 (color difference CIE94 _1-3 shown in FIG. 4), “information on the subject visible to the human eye” and the image 1 without using the image sensor shown in FIG. The differences (formula difference CIE94 _0-1 shown in FIG. 4) are compared.

FIG. 11 is a graph showing the interpolation accuracy (according to the color difference CIE94) including the influence of noise at the ISO sensitivity 100 using the color image sensor N3ch. The color image sensors other than N3ch also show the same tendency as N3ch. Further, Table 3 shows that each color image sensor has no noise and ftn0 at which the interpolation accuracy is maximized when affected by noise at ISO sensitivities 25, 100, 400, and 1600, and the color difference CIE94 _1-3 at that time. Is shown.

As shown in Table 3, the optimum LPF characteristics vary depending on the characteristics of the color image sensor. Chrominance _{CIE94 1-2} containing no influence of noise, Ftn0 but also increases too large or too small, _{CIE94 2-3} including the influence of noise becomes monotonously increases as Ftn0 increases, the interpolation error The value of ftn0 that minimizes the image error considering both the influence of noise differs depending on the magnitude of noise (ISO sensitivity). That is, the higher the ISO sensitivity, the smaller the value of ftn0 that minimizes the image error. This is because noise has a great influence on the color difference, and by reducing the value of ftn0 that minimizes the image error and narrowing the color difference band, the color difference is smoothed and the noise amplitude is reduced. Further, the amount of noise in the image not only increases due to the magnitude of ISO sensitivity, but may increase because dark current noise increases due to an increase in exposure time. Also, the amount of noise may increase due to a decrease in scene illuminance.

Similar evaluation was performed on different images (image P5 Fruit described in Miyake Laboratory Standard Spectral Image of Chiba University (Yoichi Miyake “Introduction to Spectral Image Processing”, University of Tokyo Press 2006)). In each color image sensor, Table 4 shows ftn0 at which the interpolation accuracy is maximized when there is no noise and noise is affected by ISO sensitivities 25, 100, 400, and 1600, and the color difference CIE94 _1-3 at that time.

  In the image P5, the focused area is narrow, the value of ftn0 is increased, and the interpolation error does not increase even if the luminance band is narrowed. Therefore, compared with the value of ftn0 for the image P (see Table 3), the value of ftn0 for the image P5 is larger. Thus, it is desirable to change the LPF parameter ftn0 depending on the subject to be photographed. Further, when the depth of field is shallow, it is desirable to increase the value of ftn0 (that is, widen the band of color difference components) and narrow the band of luminance. Note that the depth of field changes according to the resolution of the photographic lens 2. For example, the longer the focal length of the photographic lens 2, the smaller the F value at the time of photographing, and the shallower the subject distance. In addition, the value of ftn0 is reduced in the region focused on the subject to widen the luminance band, and the value of ftn0 is increased in the region not focused on the subject. It is desirable to narrow down. Whether or not the subject is in focus is determined based on information from the autofocus sensor, standard deviation of pixel values, and the like. When the resolution of the taking lens 2 is low, it is desirable to increase the value of ftn0 and widen the luminance band. Also, an optical low-pass filter is usually arranged in front of the single color image sensor in which the CFA is arranged to suppress the generation of moire. The value of ftn0 is changed by the characteristic of this optical low-pass filter. Also good. For example, when an optical low-pass filter is not used, an image with high resolution can be obtained. Therefore, it is better to widen the luminance band than when an optical low-pass filter is used.

Further, the same evaluation was performed on different images (image P1 Chart described in the standard spectral image of Miyake Laboratory, Chiba University (Yoichi Miyake “Introduction to Spectral Image Processing”, University of Tokyo Press 2006)). In each color image sensor, Table 5 shows ftn0 at which the interpolation accuracy is maximized when there is no noise and noise is affected by ISO sensitivities 25, 100, 400, and 1600, and the color difference CIE94 _1-3 at that time.

  Since the image P1 has many monochrome areas and little color difference information, the value of ftn0 for the image P1 is smaller than the value of ftn0 for the image P (see Table 3). That is, the luminance band is widened. Thus, when the saturation of the image is low, such as when the subject is close to an achromatic color, it is desirable to reduce the value of ftn0 and widen the luminance band. Alternatively, the image may be divided into a plurality of regions, and for each region, it may be determined whether or not the subject in the region is close to an achromatic color, and the luminance band may be changed.

  According to the digital camera according to the first embodiment, interpolation processing of a CFA image affected by noise can be performed with high accuracy.

Next, a digital camera as an image processing apparatus (imaging apparatus) according to a second embodiment of the present invention will be described with reference to the drawings. Since the configuration and image processing of the digital camera according to the second embodiment are the same as the configuration and image processing of the digital camera 1 according to the first embodiment, description thereof will be omitted. In this embodiment, instead of the color filter array 4 in which the color filters as shown in FIG. 2A are arranged in an array, for example, among the four pixels as shown in FIG. A blue (B) color filter for the upper left and lower right pixels, a red (R) color filter for the upper right pixels, and a green (G) color filter for the lower left pixels are arranged in an array. A color image sensor having a B Bayer color filter array (B-Bayer CFA) may be used. In addition, when Fourier-transforming the image obtained by B-Bayer CFA, L, C ₁ and C ₂ are respectively
L = (2B + G + R) / 4
C ₁ = (2B−G−R) / 4
C ₂ = (G−R) / 4
It is represented by

For example, as shown in FIG. 12B, a red (R) color filter for the upper left and lower right pixels of the four pixels, and a green (G) color filter for the upper right pixels, A color image sensor including an R Bayer color filter array (R-Bayer CFA) in which blue (B) color filters are arranged in an array with respect to the lower left pixel may be used. In addition, when the image obtained by R-Bayer CFA is Fourier-transformed, L, C ₁ and C ₂ are respectively
L = (2R + G + B) / 4
C ₁ = (2R-GB) / 4
C ₂ = (B−G) / 4
It is represented by

In this embodiment, a color image sensor C3ch having a spectral quantum efficiency as shown in the graph of FIG. 2A and an N3ch having a spectral quantum efficiency as shown in the graph of FIG. 2B are used. Further, as the color filter array provided in the color image sensors C3ch and N3ch, the color filter array 4, which is a Bayer CFA, the B-Bayer CFA, and the R-Bayer CFA are used interchangeably. Also, the image P4 is used as the subject shown in FIG. 4 and CIE D65 is used as the light source, and the interpolation of the image P4 is performed while changing the value of the parameter ftn0 of the LPF. Specifically, performance of image interpolation P4 values of ftn0 for band _{C 1,} while changing independently a value of ftn0 for band _{C 2.}

First, the interpolation accuracy of each color image sensor after color conversion of Raw data is examined. For example, image data (image 1 shown in FIG. 4) that has undergone color conversion processing via an image sensor (color image sensor), and image data that has undergone interpolation processing and color conversion processing (image 2 shown in FIG. 4) via the image sensor. (The color difference CIE94 _1-2 shown in FIG. 4) is compared. Similarly, the difference (the color difference CIE94 _0-1 shown in FIG. 4) between the “information of the subject visible to the human eye” that does not pass through the image sensor shown in FIG. 4 and the image 1 shown in FIG. 4 is compared. FIG. 13 is a graph showing a result of evaluating the interpolation accuracy in the color difference dimension in the color image sensor N3ch provided with the Bayer CFA. Incidentally, _{E 1} is ΔE ^* ₉₄ = 1.6, _{E 2} is ΔE ^* ₉₄ = 1.8, _{E 3} is ΔE ^* ₉₄ = 2.0, _{E 4} is ΔE ^* ₉₄ = 2.2 shown in FIG. 13, E ₅ is a graph when ΔE ^* ₉₄ = 2.4. Further, the color image sensor C3ch provided with the Bayer CFA also shows the same tendency as that of the N3ch. Further, Table 6 shows that when each of the color image sensors C3ch and N3ch includes a Bayer CFA, a B-Bayer CFA, and an R-Bayer CFA, the value of C ₁ ftn0 and the C ₂ ftn0 where the interpolation accuracy is maximized. It shows the values, and the color difference _{CIE94 1-2} at that time.

As shown in FIG. 13 and Table 6, the optimal C ₁ ftn0 value is greater than the optimal C ₂ ftn0 value. That is, it is desirable to make the band of C ₁ wider than the band of C ₂ . This, _C 1 to BayerCFA, B-BayerCFA _C 1 of, R-BayerCFA each _{C 1} of densely sampled in which the color components (in BayerCFA G, B-BayerCFA In B, the R-BayerCFA R), that while including a high frequency component, because it contains no color _{components C} 2 of _{BayerCFA, B-BayerCFA C 2,} R-BayerCFA each _{C 2} of are densely sampled, _{C 2} a band of _{C 1} It is desirable to make it wider than the bandwidth.

のように、三刺激値ＸＹＺに変換する。添字ｎのついている値は白色での値になる。カラーイメージセンサーＴｙｐｅＣ及びＴｙｐｅＮの色変換マトリクスＭ_Ｃ及びＭ_Ｎはそれぞれ、

となっており、これから色変換でのノイズ増幅率はＬ^＊，ａ^＊，ｂ^＊（Ｌ^＊ａ^＊ｂ^＊色空間の各座標）毎にＴｙｐｅＣ及びＴｙｐｅＮで、

となる。ＴｙｐｅＣでａ^＊方向のノイズ増幅が特に大きい。また、輝度（Ｙ）への寄与はＧチャンネルが大きく、ＹはＬ^＊，ａ^＊，ｂ^＊のいずれにも寄与するので緑（Ｇ）を精度良く補間出来るＢａｙｅｒＣＦＡの補間誤差が小さいと考えられる。なお、Ｌ^＊，ａ^＊，ｂ^＊は、

で定義されている。Ｒチャンネルのノイズ増幅が大きく、ａ^＊，ｂ^＊の定義に基づくとＺよりもＸの方が視覚にとって重要であろうにもかかわらず、補間誤差はＲ−ＢａｙｅｒＣＦＡの方がＢ−ＢａｙｅｒＣＦＡよりも大きい。これは、Ｒ−ＢａｙｅｒＣＦＡの補間誤差そのものが大きいためであると考えられる。色差で最適なパラメータでのＲＧＢそれぞれのＲａｗデータの補間誤差を表７に示す。

Further, as shown in Table 6, the optimum LPF characteristics vary depending on the characteristics of the color image sensor. Although the interpolation accuracy of Raw data before color conversion processing is higher in Type C than in Type N, Table 6 shows that Type N has the best interpolation accuracy. This is because the error is amplified by the color conversion process, and the amplification factor of Type C is larger than that of Type N. Here, the output (R, G, B) of the color image sensor uses a color conversion matrix,

As shown in FIG. 5, the tristimulus values XYZ are converted. The value with the subscript n is the value in white. The color conversion matrices M _C and M _N of the color image sensors Type C and Type _N are respectively

From now on, the noise amplification factor in color conversion is Type C and Type N for each L ^* , a ^* , b ^* (each coordinate in L ^* a ^* b ^* color space).

It becomes. The noise amplification in the a ^* direction is particularly large with Type C. Further, the contribution to the luminance (Y) is large in the G channel, and Y contributes to all of L ^* , a ^* , and b ^* , and therefore it is considered that the interpolation error of Bayer CFA that can accurately interpolate green (G) is small. . L ^* , a ^* , b ^* are

Defined in Even though the R channel noise amplification is large and X may be more important to vision than Z based on the definition of a ^* , b ^* , the interpolation error is greater for R-Bayer CFA than for B-Bayer CFA. large. This is considered because the interpolation error itself of R-Bayer CFA is large. Table 7 shows the interpolation error of each RGB raw data with the optimum parameters for color difference.

As shown in Table 7, Type C of B-Bayer CFA has better interpolation accuracy than Type C of R-Bayer CFA. In Type C, the accuracy of a ^* is important from the viewpoint of error amplification in color conversion. That is, the accuracy of X and Y is important, and this is related to the accuracy of R and G 1. On the other hand, Type-N of B-Bayer CFA has an interpolation accuracy of B, G is slightly worse, and R is worse than Type-N of R-Bayer CFA. In Type N, the accuracy of a ^* and b ^* is almost as important from the viewpoint of error amplification in color conversion. This is related to the accuracy of all R, G and B. It is considered that the total interpolation error is determined by the balance between the RGB interpolation accuracy and the error amplification in the color conversion.

  Next, the influence of the noise on the interpolated image when the sensor data (photon shot noise) is included in the Raw data was examined. That is, the difference between the image after interpolation in the simulation not including noise and the image after interpolation in the simulation including noise was evaluated by color difference. In addition, the difference between the image when all the RGB channels were acquired for each pixel and the image after interpolation in the simulation including noise was also evaluated by the color difference. The latter will evaluate both interpolation error and noise. As simulation parameters, the pixel pitch is 4 μm, the color image sensor uses C3ch and N3h, baseline noise is 0, and ISO sensitivity 25, 100, 400, 1600 is used.

  Sensor noise modulated by interpolation increases as ftn0 increases. When ftn0 is increased, the band of the color difference component at the time of interpolation is increased, so that the high frequency component of the color difference is restored. The color difference component is more affected by the sensor noise than the luminance component, and the high frequency component is mainly used. Therefore, it is considered that the influence of sensor noise is more easily affected by increasing the color difference band in the interpolation. Narrowing the color difference band (decreasing ftn0) smoothes the color difference and seems to have a noise reduction effect.

であり、ａ^＊成分が大きい。ａ^＊成分は、概ねｒｅｄ−ｇｒｅｅｎの方向であり、ＢａｙｅｒＣＦＡでは、Ｃ_２＝（−Ｒ＋Ｂ）／４よりもＣ_１＝（−Ｒ＋２Ｇ−Ｂ）／４成分にノイズが含まれやすいと考えられる。 Macbeth ColorChecker's average noise fluctuation in the L ^* a ^* b ^* color space is ISO 1600.

And a ^* component is large. The a ^* component is generally in the direction of red-green, and in Bayer CFA, it is considered that the C ₁ = (− R + 2G−B) / 4 component is more likely to contain noise than C ₂ = (− R + B) / 4.

FIG. 14 is a graph showing noise after interpolation in the color image sensor N3ch provided with the Bayer CFA, as viewed from the color difference (CIE94 _2-3 ) at an ISO sensitivity of 100. Incidentally, _{E 10} shown in FIG. 14 ΔE ^* ₉₄ = _{1.2, E 11} is ΔE ^* ₉₄ = _{1.4, E 12} is ΔE ^* ₉₄ = _{1.6, E 13} is a ΔE ^* ₉₄ = 1.8 It is a graph of time. As shown in FIG. 14, (widen the band of _{C 1,} namely weaken the smoothing of _{C 1)} which ftn0 the increasing of _{C 1} and abruptly noise increases. On the other hand, the _B-BayerCFA, C 2 = - for (R + G) / 4, ( widen the band of _{C 2,} namely weaken the smoothing of _{C 2)} which ftn0 the increase of _{C 2} and, suddenly noise is large Become. In R-Bayer CFA, the asymmetry of the C ₁ and C ₂ bands of noise amount is not so strong.

In an image after interpolation in a simulation including noise, the higher the ISO sensitivity, the stronger the influence of noise. Therefore, the smaller the ftn0, the smaller the color difference of the image after interpolation as the ISO sensitivity increases. In particular, a large effect of smoothing the red-green noise of the image sensor and ftn0 is small _{C 1} by BayerCFA. FIG. 15 is a graph showing the interpolation error of the color image sensor N3ch provided with the Bayer CFA, as seen by the color difference (CIE94 _1-3 ) at the ISO sensitivity of 100. Incidentally, _{E 20} is ΔE ^* ₉₄ = _{2.4, E 21} is ΔE ^* ₉₄ = _{2.6, E 22} is ΔE ^* ₉₄ = _{2.8, E 23} is ΔE ^* ₉₄ = 3.0 shown in FIG. 15, E ₂₄ is ΔE ^* ₉₄ = _{3.2, E 25} is a graph of the time of ΔE ^* ₉₄ = 3.4. Table 8 BayerCFA, Table 9 shows the R-BayerCFA, Table 10 the minimum error ΔE ^* ₉₄ ΔE ^* ₉₄ value ftn0 and at that time to give in consideration of the noise at the B-BayerCFA.

The higher the ISO sensitivity is, the smaller ftn0 is, and it is better to reduce the resolution of the color difference, and Type N has ΔE ^* ₉₄ smaller than Type C. However, in ISO 1600, Type C is smaller in CIE94 _1-3 than Type N.

  According to the digital camera according to the second embodiment, interpolation processing of a CFA image affected by noise can be performed with high accuracy.

  In each of the above-described embodiments, a digital camera has been described as an example of an image processing apparatus. However, the present invention is also applied to other image processing apparatuses that can perform image processing, such as a personal computer. Can do.

  In each of the above-described embodiments, interpolation processing using Bayer CFA is described. However, complementary CFA, four-color RGBE filters, and various CFA listed in the above-mentioned Hirakawa and Wolfe (2008). The present invention can also be applied to the interpolation processing used.

  In each of the above-described embodiments, an image processing program for causing the digital camera 1 to execute image processing is read from a memory (not shown) or the like and executed. In other words, the image processing program used in each of the above-described embodiments uses a color image sensor in which a color filter array 4 having a plurality of different transmission characteristics is arranged on the front surface of the image sensor 6 to generate a plurality of color components in each pixel. An image processing program applied to the digital camera 1 as an image processing apparatus that performs data interpolation of a color filter image obtained by detecting one of the color components, and obtains photographing conditions in the color image sensor. A step of changing the frequency band division according to the shooting conditions, a step of dividing the frequency band of the color filter image to extract luminance and a plurality of color difference components, and a predetermined process using the luminance and the plurality of color difference components And a step of performing color conversion to the color components. Therefore, according to the image processing program used in each of the above-described embodiments, interpolation processing of a CFA image affected by noise can be performed with high accuracy.

A simulation image was created using the image P4 using the digital camera according to the second embodiment. Looking at low-sensitivity images, Type N has a lighter false color due to interpolation. In high-sensitivity images, the influence of noise is greater than the false color resulting from interpolation. In addition, coloring is seen around Y and C circles of Kodak Gray Scale. This is because the ftn0 of ftn0 and _{C 2} of C ₁ interpolation parameters are different, considered blurring amount of _{C 1} and _{C 2} are colored shifted color balance differently. If C ₁ ftn0 and C ₂ ftn0 are set independently, the separation of the overall luminance and color difference is improved, but when viewed partially, it may cause false colors. From the viewpoint of interpolation, it is only necessary to set the spectral sensitivity in consideration of the correlation between the channels so that the interpolation parameters C ₁ ftn0 and C ₂ ftn0 are the same.

As described above, the CFA image was evaluated by an end-to-end system in consideration of all processes from input to output. The evaluation index was evaluated based on the color difference dimension, that is, approximately perceptually uniform color space. The evaluation with the Raw data indicated that the color conversion was not taken into consideration and therefore it was not appropriate. In addition, in order to perform evaluation including sensor noise and improve the reproduction image quality, it was shown that it is better to narrow the frequency band of the color difference component during interpolation as the noise increases. Among the image sensors that were evaluated this time, type C and Type N with Bayer CFA, Type C and Type N with R-Bayer CFA, Type C and Type N with B-Bayer CFA, the image sensor with excellent interpolation image quality has Bayer CFA. It can be said that it is TypeN. This is considered that the interpolation error and the amplification by noise color conversion have a great influence. However, when the ISO sensitivity is increased with Bayer CFA, the color difference from the ideal image in the optimum interpolation parameter is larger in Type N. Also, if green (G) is sampled more than red (R) or blue (B), the interpolation error viewed from the color difference becomes smaller. Interpolating green (G) with higher accuracy has a good effect on any of L ^* a ^* b ^* .

  Furthermore, as the overall image quality, the color reproduction error of each image sensor must be considered. In addition, since the smoothing of the color difference in the interpolation is affected by both the reduction of the color difference noise amplitude and the low frequency component, the overall image quality is determined by the balance between them.

According to Example 1, it has been found that higher interpolation accuracy can be obtained as a whole by setting the bands of C ₁ and C ₂ independently. At this time, it was found that C ₁ should have a wider band than C ₂ . However, if the bands of C ₁ and C ₂ are different, a false color is generated around the edge of the color due to a mismatch of the respective color difference resolutions.

In addition, when evaluated by color difference, it was found that Type N, which has a small noise amplification in color conversion, has a smaller interpolation error than Type C. Also, when considering the photon shot noise, noise is increased in the high-frequency components of the color difference, the noise component has been found that the image quality is suppressed is improved by setting a narrow band of C ₁ and C _2. In addition, since a lot of noise is included in the red-green component (particularly in the case of Type C), it has been found that in the case of Bayer CFA, the noise can be effectively suppressed by setting the C ₁ band narrow. Further, it was found that the color difference component determined by CFA determines which color difference component band can be set narrower to improve the image quality. In particular, when noise increases, Type N has more interpolation errors and noise errors. Type N has less color noise but more luminance noise than Type C. Therefore, in Type N, it is better not to set the C ₁ and C ₂ bands as narrow as Type C in order not to increase luminance noise, which may hinder the reduction of color difference noise.

From the above, it was found that it is better to use an image sensor in which the correlation between channels is high, but the interpolation accuracy is good, but the amplification in the color conversion of the interpolation error is large, so the correlation between channels is not so large. In addition, the color difference band varies depending on the sampling density of the color components included in the band. Therefore, in Bayer CFA, interpolation accuracy increases by setting a wide band of C ₁ including G. However, if the bands of C ₁ and C ₂ are different, a false color is generated around the color boundary.

In high-sensitivity shooting, the image quality is improved by narrowing the color difference band. Noise generally has a large red-green component, and in Bayer CFA, C ₁ noise is larger than C ₂ noise. However, as the color difference components of BayerCFA, band _{C 1} is larger than _{C 2.} Therefore, when noise of a predetermined amount or more is included, it is preferable to set the bands of C ₁ and C ₂ to be substantially the same. Further, it was found that R-Bayer CFA and B-Bayer CFA tend to be biased, and Bayer CFA is most excellent.

For example, a light source such as a light bulb having a low color temperature has a short wavelength energy, so that noise of the B component among RGB components acquired from the CFA image increases. In this case, it is expected _{that C 2} of the noise in BayerCFA increases. Therefore, the pass band of the filter may be set according to the photographing light source.

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... Shooting lens, 4 ... Color filter, 6 ... Image sensor, 8 ... Image processing part, 9 ... Shooting condition acquisition part, 10 ... Recording part, 12 ... White balance processing part, 14 ... 1st floor Tone processing unit, 16 ... interpolation processing unit, 18 ... first color conversion processing unit, 20 ... second gradation processing unit, 22 ... second color conversion processing unit, 24 ... edge enhancement processing unit, 26 ... compression processing unit.

Claims (14)

Data interpolation of the color filter image obtained by detecting one color component of a plurality of color components at each pixel by a color image sensor in which a plurality of color filters having different transmission characteristics are arranged in front of the image sensor. In the image processing apparatus to perform,
Shooting condition acquisition means for acquiring shooting conditions in the color image sensor;
Band division setting means for changing the frequency band division according to the photographing conditions;
Extracting means for dividing the frequency band of the color filter image to extract luminance and a plurality of color difference components;
Color conversion means for performing color conversion into a predetermined color component using the luminance and the plurality of color difference components;
An image processing apparatus comprising: The shooting condition is a noise amount of an image,
The image processing apparatus according to claim 1, wherein the band division setting unit sets the frequency band from which the color difference component is extracted narrower as the amount of noise increases.   The image processing apparatus according to claim 2, wherein the noise amount is estimated according to photographing sensitivity.   The image processing apparatus according to claim 2, wherein the amount of noise is estimated according to an exposure time.   The image processing apparatus according to claim 2, wherein the noise amount is estimated according to scene illuminance. The photographing condition is a resolution of the photographing lens,
The image processing apparatus according to claim 1, wherein the band division setting unit sets the frequency band for extracting the color difference component wider as the resolution is lower.   The image processing apparatus according to claim 6, wherein the resolution is estimated according to a depth of field. The shooting condition is focusing information of an image,
The image processing apparatus according to claim 1, wherein the band division setting unit sets the frequency band for extracting the color difference component to be narrower as the image is in focus. The shooting condition is image saturation,
The image processing apparatus according to claim 1, wherein the band division setting unit sets the frequency band for extracting the color difference component to be narrower as the saturation is lower. The photographing condition is a spectral sensitivity of the image sensor,
The image processing apparatus according to claim 1, wherein the band division setting unit narrows the frequency band for extracting the color difference component as the overlap between the spectral sensitivity channels increases. The photographing condition is information of each color component in each pixel detected by the color image sensor,
The image processing apparatus according to claim 1, wherein the band division setting unit sets a wide frequency band for extracting the color difference component including information of the color component detected more frequently. The shooting condition is information of each color component and the amount of noise of an image in each pixel detected by the color image sensor,
The band division setting means sets the frequency band for extracting the color difference component including the information of the color component detected more frequently, and sets the frequency band for extracting the color difference component having a larger amount of noise narrower. The image processing apparatus according to claim 1.   An imaging device including the image processing device according to claim 1. Data interpolation of the color filter image obtained by detecting one color component of a plurality of color components at each pixel by a color image sensor in which a plurality of color filters having different transmission characteristics are arranged in front of the image sensor. An image processing program applied to an image processing apparatus to perform,
Acquiring shooting conditions in the color image sensor;
Changing the division of the frequency band according to the shooting conditions;
Dividing the frequency band of the color filter image to extract luminance and a plurality of color difference components;
Performing color conversion to a predetermined color component using the luminance and the plurality of color difference components;
An image processing program comprising:

Images