JP2000278702A - Interpolation processor and recording medium storing interpolation processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To surely improve the decision accuracy of similarity by removing the low band part of spatial frequency from a color image to be interpolated and mutually comparing the color information of different colors in the prescribed color image to decide similarity. SOLUTION: An interpolation processing part removes the low band part of spatial frequency from image data to be interpolated and calculates color information mainly consisting of a high band part. Namely the absolute value of the difference between red color information and green color information in an image mainly consisting of a high band part is calculated for a pixel having red color information, the absolute value of the difference between blue color information and green color information in the image mainly consisting of the high band part is calculated for a pixel having blue color information, the different color information data in the image mainly consisting of the high band part are mutually compared, and the vertical similarity and horizontal similarity of the pixel to be interpolated are calculated. Then the evaluation values of vertical and horizontal similarity are calculated by using the vertical and horizontal similarity of plural pixels close to each other and then red and blue are interpolated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention determines the degree of similarity in a plurality of directions with respect to a color image composed of color information of a plurality of predetermined colors, and determines a pixel based on the result of the determination. An interpolation processing device that performs an interpolation process by calculating an interpolation amount corresponding to color information of a missing color for each color;
And a recording medium storing an interpolation processing program for causing a computer to execute the interpolation processing.

[0002]

2. Description of the Related Art Electronic cameras have three colors (for example, RG
There is a type in which image data of a color image is generated by an image sensor in which color filters of B, red, green, and blue) are arranged at predetermined positions (for example, a Bayer arrangement). In such an electronic camera, only one color information is generated for each pixel. Therefore, interpolation processing is necessary to obtain color information of all colors on a pixel-by-pixel basis.

[0003] As a method of such interpolation processing, the spatial similarity of an interpolation target pixel to be subjected to interpolation processing is determined, and the amount of interpolation is determined using color information of pixels adjacent in the direction of strong similarity. Conventionally, a calculation method has been considered.

For example, US Pat. No. 5,373,322, US Pat. No. 5,382,976, and US Pat. No. 5,629,734 disclose spatial interpolation pixels. Techniques for determining similarity by chrominance gradients have been disclosed. FIG. 6 is a diagram illustrating the conventional interpolation processing disclosed in these US patent specifications. FIG. 6 shows color information corresponding to each pixel of the image pickup device (configured such that three color filters of RGB are arranged in a Bayer array). G11, G13,... R12, R14,...
.., R56 and the like indicate red (R) color information, and B2
1, B23,..., B67, etc. indicate blue (B) color information.

In the technique disclosed in US Pat. No. 5,373,322, when calculating the amount of green interpolation of a pixel having R34, information HDiff on the horizontal color gradient and information on the color gradient in the vertical direction are used. HDiff = | ((R32 + R36) / 2) -R34 | VDiff = | ((R14 + R54) / 2) -R34 |, and when HDiff <VDiff, it is determined that the similarity in the horizontal direction is strong, and VDiff is determined. If <HDiff, it is determined that the similarity in the vertical direction is strong.

Hereinafter, for the sake of simplicity, strong similarity in the vertical direction is referred to as vertical similarity, strong similarity in the horizontal direction is referred to as horizontal similarity, and two-way similarities are very similar. This is called an intermediate similarity. According to the technique disclosed in U.S. Pat. No. 5,382,976, when calculating a green interpolation amount of a pixel having R34, information G on a horizontal color gradient is obtained.
Information on diff-hor and vertical color gradient Gdiff-ver
Gdiff-hor = | G33-G35 | If> Threshold∩Gdiff-ver <Threshold, it is determined that the images are vertically similar.

In the technique disclosed in US Pat. No. 5,629,734, when calculating the amount of green interpolation of a pixel having R34, information DH on the horizontal color gradient and information on the similarity in the vertical direction are obtained. DV is calculated as follows: DH = | −R32 + 2 × R34−R36 | + | G33−
G35 | DV = | -R14 + 2 × R34-R54 | + | G24-
G44 |, it is determined that horizontal similarity is obtained when DH <DV, and vertical similarity is determined when DV <DH.

By the way, as shown in FIG.
In a case where the pixel pitch in the vertical direction and the horizontal direction is defined as a pixel pitch P in an imaging device in which color filters of colors are arranged in a Bayer array, the information on each color gradient described above is arranged at a 2P interval. Of the color information of each pixel. Therefore, according to the technology disclosed in the above-mentioned U.S. Patent Specification, for a color image (a color image having a high spatial frequency) that changes more finely than 2P in the vertical direction and the horizontal direction, a direction having a strong similarity is determined. The determination cannot be made with high accuracy, and the interpolation amount cannot be accurately calculated.

Accordingly, the present applicant has proposed an invention capable of accurately calculating an interpolation amount even for a color image having a high spatial frequency, as disclosed in Japanese Patent Application Nos. 10-70454 and 10-1.
No. 51278. In these inventions,
Since similarity can be determined using color information of different colors, similarity can be determined by comparing color information at one pixel interval (interval of P). Therefore, according to these inventions, for a color image which is close to an achromatic color and has low saturation (a color image having a similar hue and a waveform in which the spatial frequencies of RGB are very similar), even if the spatial frequency is high, the interpolation amount Can be accurately calculated.

In most cases, a natural image is substantially close to an achromatic color.
The interpolation amount can be calculated with considerably high accuracy.

[0011]

By the way, in a color image including a portion where the hue changes suddenly or a portion with high saturation (a portion showing a waveform in which each spatial frequency of RGB is different), different colors are used even if they are adjacent to each other. If the pixel has information, there is a possibility that a large difference occurs in the value of the color information. Therefore, even if similarity determination is performed on such a color image using color information of different colors, the similarity determination may not be correctly performed. There is a possibility that a false color that cannot originally exist in an image may occur.

Therefore, the inventions according to the first to fourth aspects suppress the influence of the characteristics of the color image when determining the similarity, and perform the similarity determination with high accuracy to suppress the generation of false colors. It is an object of the present invention to provide an interpolation processing device capable of performing the above. According to a fifth aspect of the present invention, an interpolation process capable of suppressing the influence of the characteristics of a color image at the time of determining similarity and performing similarity determination with high accuracy to suppress the occurrence of false colors. It is an object of the present invention to provide a recording medium that records an interpolation processing program that can be executed by a computer.

[0013]

Means for solving the problem will be described below in association with embodiments described later.
The interpolation processing device according to claim 1 performs an interpolation process for calculating an interpolation amount corresponding to color information of a missing color for each pixel on a color image including color information of a plurality of predetermined colors. In the interpolation processing device, a high-frequency main image generating means for removing a predetermined amount of the low-frequency component of the spatial frequency of the color image and generating a high-frequency main image mainly including the high-frequency component of the spatial frequency (the processing in FIGS. 2S2 and S3) Is performed, and the color information of the high-frequency main image generated by the high-frequency main image generation means is referred to, and an interpolation target pixel to be subjected to the interpolation processing is determined. A similarity that determines the strength of similarity of the interpolation target pixel in a plurality of directions by comparing color information of different colors among the color information of the interpolation target pixel and the color information of a pixel located in the vicinity of the interpolation target pixel. Sex judgment A stage (corresponding to the interpolation processing unit 20 of FIG processing by the state of performing the FIG 2S4 and S5 1),
An interpolation amount calculating unit (corresponding to the interpolation processing unit 20 in FIG. 1 in a state where the processing in FIG. 2 is performed) in accordance with the result of the determination by the similarity determining unit. It is characterized by having.

By the way, the high-frequency main image generated by the high-frequency main image generating means has an edge portion of the color image to be subjected to the interpolation processing emphasized, and is a color image close to an achromatic color and low in saturation (having a similar color tone). Color image). That is, according to the first aspect of the present invention, similarity determination is performed by comparing color information of different colors with a color image that is close to an achromatic color and has low saturation, from which low-frequency components of the spatial frequency have been removed. Therefore, the accuracy of the similarity determination can be reliably improved compared to the case where the similarity determination is performed on the color image before the low frequency component of the spatial frequency is removed.

The high-frequency main image generating means may generate the high-frequency main image for the whole area of the color image to be subjected to the interpolation processing, and may generate a part of the color image (for example, the interpolation target pixel and the interpolation target pixel). A high-frequency main image may be generated for a pixel located in the vicinity of the target pixel). Further, the high-frequency main image generation means includes a process of generating a high-frequency main image targeting pixels located in a relatively wide area, and a process of generating a high-frequency main image targeting pixels located in a relatively narrow area. The process for generation may be switched.

Here, the process of generating a high-frequency main image for pixels located in a relatively wide area is suitable for a portion where color information changes smoothly and is located in a relatively narrow area. The process of generating a high-frequency main image for a pixel is suitable for a portion where color information changes drastically.

Further, as one mode of a method of calculating the interpolation amount by the interpolation amount calculating means, an interpolation target pixel and a pixel located in the vicinity of the interpolation target pixel are determined in accordance with the strength of the similarity determined by the similarity determining means. When calculating the interpolation amount of the pixel to be interpolated, a pixel (for example, a pixel located in a direction having a high similarity) to be referred to when calculating the interpolation amount is selected, and the interpolation amount is calculated using the color information of the pixel. There is a way to do it.

According to a second aspect of the present invention, in the interpolation processing apparatus according to the first aspect, the high-frequency main image generating means removes at least half of a low-frequency component of the spatial frequency of the color image. Generating a high-frequency main image (Equation 15 to
In Equation 17, the coefficient k is a value that satisfies “k> 0.5”. According to a third aspect of the present invention, in the interpolation processing apparatus according to the first or second aspect, the high-frequency main image generating means leaves a small amount of low-frequency components of the spatial frequency of the color image high, It is characterized in that a region-based image is generated (corresponding to a value in which the coefficient k satisfies “1> k” in Equations 15 to 17).

According to a fourth aspect of the present invention, in the interpolation processing apparatus according to any one of the first to third aspects, the similarity determining means is generated by the high-frequency main image generating means. Referring to the color information of the high-frequency main image, the color information of the interpolation target pixel is compared with the color information of a pixel located in the vicinity of the interpolation target pixel, and the color information of a different color is compared. , The similarity of a pixel located in the vicinity of the interpolation target pixel is calculated in the same manner as the interpolation target pixel (corresponding to S4 in FIG. 2), and the similarity of the interpolation target pixel and the interpolation are calculated. Using the similarity of a plurality of pixels located near the target pixel, the degree of similarity of the interpolation target pixel in a plurality of directions is determined (FIG. 2).
(Corresponding to S5).

According to a fifth aspect of the present invention, there is provided a recording medium for causing a computer to function as a high-frequency main image generating unit, a similarity determining unit, and an interpolation amount calculating unit according to any one of the first to fourth aspects. Is recorded.

[0021]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a functional block diagram of an electronic camera corresponding to the first embodiment. An electronic camera corresponding to the first embodiment is described in claim 1.
Further, the present invention corresponds to an electronic camera having a function of an interpolation process performed by the interpolation processing device according to the present invention.

In FIG. 1, an electronic camera 10 includes a control unit 11, a photographing optical system 12, an OLPF (Optical Low Pass Filter) 13, an image sensor 14,
It has a / D conversion unit 15, an image buffer memory 16, a gradation conversion unit 17, an image processing unit 18, and a recording unit 19. The image processing unit 18 includes an interpolation processing unit (1 dedicated to the interpolation processing).
Chip microprocessor) 20.

In FIG. 1, in order to simplify the explanation,
Although only the interpolation processing unit 20 is described in the image processing unit 18, a functional block for performing other image processing may be provided in the image processing unit 18. In FIG. 1, the control unit 11
Is connected to the image sensor 14, the A / D converter 15, the image buffer memory 16, the gradation converter 17, the image processor 18, and the recorder 19.

The optical image acquired by the photographing optical system 12 is O
This is provided to the image sensor 14 via the LPF 13. The output of the image sensor 14 is quantized by the A / D converter 15 and stored in the image buffer memory 16 as image data of a color image to be interpolated. The image data stored in the image buffer memory 16 is stored in an image processing unit 18.
, Is subjected to an interpolation process by an interpolation processing unit 20, and is recorded via a recording unit 19.

In this embodiment, the OLPF 13 is not provided, and the optical image acquired by the photographing optical system 12 is
4 may be given directly. In the present embodiment, the image data to be subjected to the interpolation processing is the image processing unit 1
Before being supplied to 8, gradation conversion (log conversion, γ conversion, etc.) may be performed by the gradation conversion unit 17. FIG. 2 is an operation flowchart of the interpolation processing unit 20 according to the first embodiment.

FIG. 3 is a diagram showing an array of color information of image data to be subjected to interpolation processing in the first embodiment. In FIG. 3, the position at which the color information is arranged is represented by a coordinate system composed of “a horizontal axis X having a positive right direction” and a vertical axis Y having a positive downward direction. Information is represented by G [X, Y], and red or blue color information is represented by RB [X, Y].

That is, assuming that the red color information is R [X, Y] and the blue color information is B [X, Y], the pixel located at an arbitrary coordinate [i, j] is the red color information. If so, RB [i, j] is R
When a pixel corresponding to [i, j] and located at an arbitrary coordinate [i, j] has blue color information, RB [i, j] corresponds to B [i, j]. Also, in FIG. 3, “i
When + j = even number holds, a pixel having green color information is located at the coordinates, and when i + j = odd number holds,
A pixel having red or blue color information is located at the coordinates. Further, assuming that a pixel having red color information is located at a coordinate where “i + j = odd and i = even”, a blue color is provided at a coordinate where “i + j = odd and i = odd”. A pixel having information will be located.

The operation of the first embodiment will be described below. In the first embodiment, the interpolation processing unit 2 will be described with reference to FIG.
The following description focuses on the operation at 0 (particularly, green interpolation processing).
In FIG. 2, the processing of S1 to S6 corresponds to green interpolation processing. First, the interpolation processing unit 20 performs a vertical prediction value Gt and a horizontal prediction value Gy of a green interpolation amount for a pixel in which green color information is missing (corresponding to a pixel to be subjected to green interpolation processing). Is calculated (S1).

Here, the vertical predicted value corresponds to a value applied as an interpolation amount when the vertical similarity is strong, and the horizontal predicted value is an interpolation value when the horizontal similarity is strong. It corresponds to the value applied as a quantity. For example, the interpolation processing unit 20
Calculates the vertical prediction value Gt and the horizontal prediction value Gy of the green interpolation amount by the following << vertical prediction value calculation processing >> and << horizontal prediction value calculation processing >>.

<< Calculation Processing of Vertical Prediction Value >> Interpolation Processing Unit 2
0 indicates the coordinates of a pixel (pixel having red or blue color information) in which green color information is missing in FIG. 3 [i, j].
(Where “i + j = odd”), the color information RB [i, j] of the pixel and the color information RB [i, j-2], RB of the pixel located near the pixel [((RB [i, j] -RB [i, j-2])> between [i, j + 2], RB [i, j-2], RB [i, j + 2] th2) ∩ ((RB [i, j] -RB [i, j + 2])> th2)｝ ∪ ｛(RB [i, j] -RB [i, j-2]) <-th2) ∩ ( RB [i, j] -RB [i, j + 2] <-th2)｝... If condition 1 is satisfied, the vertical predicted value Gt [i, j] of the green interpolation amount
Gt [i, j] = (G [i, j-1] + G [i, j + 1]) / 2 + (2RB [i, j] -RB [i, j-2] -RB [i, j + 2]) / 4 Calculated by Equation 1. In addition, when the above condition 1 is not satisfied, the interpolation processing unit 20 calculates the vertical predicted value Gt [i, j] of the green interpolation amount as Gt [i, j] = (G [i, j−1] · t1 + G [i, j + 1] · t2) / (t1 + t2) Calculated by Equation 2.

However, in condition 1, th2 is a threshold, and in equation 2, t1 and t2 are: t1 = | RB [i, j] -RB [i, j + 2] | +1 ... Equation 3 t2 = | RB [i, j] -RB [i, j-2] | +1..., Which satisfies Expression 4. In Condition 1 and the conditional expressions described below, ∩ indicates and, and ∪ indicates or.

By the way, "+1" in equations 3 and 4
Is a term added to prevent the divisor (t1 + t2) in Equation 2 from becoming zero. Here, since it is assumed that the color information changes in the range of 0 or more and 255 or less, “+1” is added, but the value of such a term is limited to “+1”. It is not something to be done. << Process of Calculating Horizontal Predicted Value >> In FIG. 3, the interpolation processing unit 20 sets the coordinates of a pixel (pixel having red or blue color information) where green color information is missing to [i, j] (where , [I + j = odd number]), the color information RB [i, j] of the pixel and the color information RB [i−2, j], RB [i of the pixel located near the pixel + 2, j], RB
[((RB [i, j] -RB [i-2, j])> th2) ∩ ((RB [i, j j] -RB [i + 2, j])> th2)｝ ∪ ｛(RB [i, j] -RB [i-2, j]) <-th2) ∩ ((RB [i, j] -RB [i + 2, j]) <-th2)｝]... When Condition 2 is satisfied, the predicted value Gy [i, j] of the green interpolation amount in the horizontal direction
Is Gy [i, j] = (G [i-1, j] + G [i + 1, j]) / 2 + (2RB [i, j] -RB [i-2, j] -RB [i + 2, j]) / 4 Calculated by Equation 5. Further, when the above-described condition 2 is not satisfied, the interpolation processing unit 20 calculates the horizontal predicted value Gy [i, j] of the green interpolation amount as Gy [i, j] = (G [i−1, j] · y1 + G [i + 1, j] · y2) / (y1 + y2) ... Calculated by Equation 6.

However, in condition 2, th2 is a threshold, and in equation 6, y1 and y2 are: y1 = | RB [i, j] -RB [i + 2, j] | +1 ... Eq. 7 y2 = | RB [i, j] -RB [i-2, j] | +1..., Which satisfies Expression 8. The term “+1” in Equations 7 and 8 is a term added to prevent the divisor (y1 + y2) in Equation 6 from becoming zero. Here, since it is assumed that the color information changes in the range of 0 or more and 255 or less, “+1” is added, but the value of such a term is limited to “+1”. It is not something to be done.

Next, the interpolation processing section 20 calculates a low frequency component of the spatial frequency of the image data to be subjected to the interpolation processing (S2). For example, the interpolation processing unit 20 in FIG.
The pixel located at an arbitrary coordinate [i, j] is green color information G [i, j].
, The low frequency component of the spatial frequency at that pixel
LG [i, j] is replaced by LG [i, j] = (G [i, j] + G [i-1, j-1] + G [i-1, j + 1] + G [i + 1, j-1] + G [i + 1, j + 1] + G [i-2, j] + G [i + 2, j] + G [i, j-2] + G [i, j + 2 ]) / 9... Calculated by Equation 9 and when the pixel located at an arbitrary coordinate [i, j] has red color information R [i, j], the low frequency component LR [i of the spatial frequency at that pixel , j] is LR [i, j] = (R [i, j] + R [i-2, j] + R [i + 2, j] + R [i, j-2] + R [i, j + 2] + R [i-2, j-2] + R [i + 2, j-2] + R [i-2, j + 2] + R [i + 2, j + 2]) / 9... Calculated by Expression 10, and when a pixel located at an arbitrary coordinate [i, j] has blue color information B [i, j], the low frequency component LB [i, j] of the spatial frequency at that pixel LB [i, j] = (B [i, j] + B [i-2, j] + B [i + 2, j] + B [i, j-2] + B [i, j + 2 ] + B [i-2, j-2] + B [i + 2, j-2] + B [i-2, j + 2] + B [i + 2, j + 2]) / 9 ... 11

As an arithmetic expression for calculating the low frequency component of the spatial frequency, instead of the above-mentioned expressions 9 to 11,
Expressions 12 to 14 shown below may be used. LG [i, j] = (G [i, j] + (G [i-1, j-1] + G [i-1, j + 1] + G [i + 1, j-1] + G [i + 1, j + 1]) / 4) / 2 Equation 12 LR [i, j] = (R [i, j] + (R [i-2, j] + R [i + 2, j ] + R [i, j-2] + R [i, j + 2]) / 4) / 2 Equation 13 LB [i, j] = (B [i, j] + (B [i-2, j] + B [i + 2, j] + B [i, j-2] + B [i, j + 2]) / 4) / 2 Equation 14 Incidentally, it is calculated using Equations 9 to 11. The low-frequency component of the spatial frequency reflects color information of a pixel located in a relatively wide area. Therefore, the interpolation processing unit 20 calculates Equations 9 to 11
Is used, it is possible to appropriately calculate the low frequency component of the spatial frequency even in a portion where the color information changes drastically.

On the other hand, the low frequency component of the spatial frequency in the portion where the color information changes smoothly can be calculated by using only the color information of the pixels located in a relatively narrow area.
Therefore, the interpolation processing unit 20 can efficiently calculate the low-frequency component of the spatial frequency in a portion where the color information changes smoothly by using Expressions 12 to 14.

Next, the interpolation processing section 20 calculates the color information of the high-frequency main image by removing the low frequency component of the spatial frequency from the image data to be subjected to the interpolation processing (S3). For example, the interpolation processing unit 20 uses the low-frequency component of the spatial frequency calculated in S2, and in FIG. 3, arbitrary coordinates [i, j]
, Has green color information G [i, j], the color information DG [i, j] of the high-frequency main image at that pixel is given by DG [i, j] = G [i, j] − k · LG [i, j]... Calculated by Expression 15, and when the pixel located at an arbitrary coordinate [i, j] has red color information R [i, j], the high-frequency main image of the pixel The color information DR [i, j] is calculated by DR [i, j] = R [i, j] -k · LR [i, j]..., And the pixel located at an arbitrary coordinate [i, j] Has blue color information B [i, j], the color information DB [i, j] of the high-frequency main image at the pixel is expressed as DB [i, j] = B [i, j] -k · LB [ i, j] ... Calculated by Expression 17.

However, in Equations 15 to 17, the coefficient k of the low frequency component of the spatial frequency is a value that satisfies “1>k> 0.5”. That is, the color information of the high-frequency main image calculated by these arithmetic expressions is a value in which a small amount of the low-frequency component of the spatial frequency is removed while a half or more is removed. In the following, in order to simplify the description, when a pixel located at an arbitrary coordinate [i, j] has red or blue color information, the color information of the high-frequency main image at that pixel is converted to DRB [i, j]. j]. That is, when a pixel located at an arbitrary coordinate [i, j] has red color information, DRB [i, j] corresponds to DR [i, j], and DRB [i, j] corresponds to an arbitrary coordinate [i, j]. When the located pixel has blue color information, DRB [i, j] corresponds to DB [i, j].

Next, the interpolation processing section 20 compares the color information of different colors of the high-frequency main image, and determines whether or not each pixel lacks green color information (interpolation target pixel to be subjected to green interpolation processing). The vertical similarity Ct and the horizontal similarity Cy
Is calculated (S4).

For example, in FIG. 3, the interpolation processing unit 20 sets the coordinates of the pixel lacking green color information (the pixel having red or blue color information) to [i, j] (where “i + j = Odd number ”), the vertical similarity Ct of the pixel
[i, j] and the similarity Cy [i, j] in the horizontal direction are expressed as Ct [i, j] = a1 || DG [i, j-1] -DG [i, j + 1] | + a2 ( | DRB [i, j] -DG [i, j-1] | + | DRB [i, j] -DG [i, j + 1] |) / 2 Equation 18 Cy [i, j] = a1 · | DG [i-1, j] -DG [i + 1, j] | + a2 ・ (| DRB [i, j] -DG [i-1, j] | + | DRB [i, j] -DG [i + 1, j] |) / 2 ... Calculated by Expression 19. However, in Expressions 18 and 19, a1 is a positive constant, and a2 is 0 or a positive constant.

That is, the interpolation processing unit 20 calculates the absolute value of the difference between the red color information of the high-frequency main image and the green color information of the high-frequency main image for the pixel having the red color information, and The pixel having the color information calculates the absolute value of the difference between the blue color information of the high-frequency main image and the green color information of the high-frequency main image, thereby comparing the color information of different colors of the high-frequency main image. It calculates the vertical similarity Ct and the horizontal similarity Cy of the interpolation target pixel.

The smaller the values of the vertical similarity Ct and the horizontal similarity Cy, the stronger the similarity. Next, the interpolation processing unit 20 uses the vertical similarity Ct and the horizontal similarity Cy of a plurality of pixels adjacent to each other,
Evaluation value TCt of similarity in the vertical direction and evaluation value T of similarity in the horizontal direction
Cy is calculated (S5).

For example, in FIG. 3, the interpolation processing unit 20 sets the coordinates of the pixel lacking green color information (the pixel having red or blue color information) to [i, j] (where “i + j = Is odd), the evaluation value TCt [i, j] of the pixel in the vertical direction and the evaluation value TCy [i, j] of the similarity in the horizontal direction are calculated as TCt [i, j] = Ct [ i, j] + (Ct [i-1, j-1] + Ct [i-1, j + 1] + Ct [i + 1, j-1] + Ct [i + 1, j + 1]) /4...Equation 20 TCy [i, j] = Cy [i, j] + (Cy [i-1, j-1] + Cy [i-1, j + 1] + Cy [i + 1, j- 1] + Cy [i + 1, j + 1]) / 4...

The smaller the value of the vertical similarity evaluation value TCt and the value of the horizontal similarity evaluation value TCy, the stronger the similarity. Next, the interpolation processing unit 20 calculates the vertical similarity evaluation value TCt and the horizontal similarity evaluation value T
A direction with a strong similarity is determined based on Cy, and a green interpolation amount is determined according to the result of the determination (S6).

For example, in FIG. 3, the interpolation processing unit 20 sets the coordinates of the pixel lacking green color information (the pixel having red or blue color information) to [i, j] (where “i + j = If the pixel has an odd number, a value TCt [i, j] between the vertical similarity evaluation value TCt [i, j] and the horizontal similarity evaluation value TCy [i, j] of the pixel is obtained. ] -TCy [i, j]> th0 If condition 3 is satisfied, it is determined that horizontal similarity is obtained, and the horizontal predicted value Gy [i, j] of the green interpolation amount calculated in S1 is replaced with the green interpolation amount. Let G [i, j] (G [i, j] = Gy [i, j]).

The interpolation processing unit 20 also calculates the above-described vertical similarity evaluation value TCt [i, j] and the horizontal similarity evaluation value TCt.
TCy [i, j] -TCt [i, j]> th0... Condition 4 is satisfied between y [i, j] and the vertical interpolation similarity, and the green interpolation amount calculated in S1 Is defined as a green interpolation amount G [i, j] (G [i, j] = Gt [i, j]).

Further, when both the condition 3 and the condition 4 do not hold, the interpolation processing unit 20 determines that the similarity is intermediate and sets the green interpolation amount G [i, j] to G [i, j] = (Gt [i, j] + Gy [i, j]) / 2...

Next, the interpolation processing section 20 performs red interpolation processing and blue interpolation processing (S7). For example, as the red interpolation processing and the blue interpolation processing, the following known methods can be applied. Interpolation processing unit 2
0 calculates the difference (R−G) between the red color information R and the green interpolation amount G for pixels having red color information, and calculates the difference (R−G) for pixels lacking red color information. By applying the average value of (RG) of a plurality of pixels close to the pixel, (RG) is obtained for all the pixels. Then, the interpolation processing unit 20 calculates the red interpolation amount by adding G (green color information or green interpolation amount) of each pixel to (RG) of each pixel thus obtained. I do.

The interpolation processor 20 calculates the amount of blue interpolation in the same manner as the amount of red interpolation. As described above, in the first embodiment, the low frequency component of the spatial frequency is removed from the image data to be subjected to the interpolation processing (the image data supplied to the interpolation processing unit 20 via the image buffer memory 16). Thus, a high-frequency main image is generated, and similarity is calculated by comparing color information of different colors of the high-frequency main image. That is, in the first embodiment, an image to be subjected to the interpolation processing is replaced with a color image that is close to achromatic and has low saturation (a color image having a similar hue), and color information of different colors is compared. Similarity can be determined.

Therefore, according to the first embodiment, the similarity can be determined with high accuracy even for a color image including a portion where the hue changes suddenly or a portion with high saturation.
Further, in the first embodiment, as shown in Expressions 15 to 17, by generating a high-frequency main image in which a low-frequency component of a spatial frequency is removed by half or more and a small amount is left, a high-frequency main image is generated. The characteristics of the original image can be reflected. Furthermore, in a general color image, in a plurality of pixels close to each other, the direction of strong similarity tends to be the same, and the similarity tends to be continuous. In the first embodiment, as shown in Expressions 20 and 21, Then, as the similarity evaluation value of each pixel, the similarity of a plurality of neighboring pixels is used, and the continuity of the similarity is reflected. Therefore, the accuracy of the similarity determination is further improved.

The operation of the second embodiment will be described below.
It should be noted that the second embodiment corresponds to an electronic camera having the functions of the imaging device according to claims 1 to 4. FIG. 4 is a diagram illustrating an array of color information of image data to be subjected to an interpolation process in the second embodiment.

In FIG. 4, similarly to FIG. 3, the positions at which the color information is arranged are defined by "a horizontal axis X whose right direction is positive" and "a vertical axis Y whose downward direction is positive". Green coordinate information is represented by G [X, Y], and red or blue color information is represented by
Expressed by RB [X, Y]. That is, assuming that red color information is R [X, Y] and blue color information is B [X, Y], arbitrary coordinates [i, j]
RB [i, j] when the pixel located at
Corresponds to R [i, j], and if the pixel located at an arbitrary coordinate [i, j] has blue color information, RB [i, j] corresponds to B [i, j] .

In FIG. 4, two pixels having the same color information are arranged in the horizontal direction. Further, in FIG. 4, one pixel has a vertically long shape of “horizontal: vertical = 1: 2”,
A square is formed by two pixels arranged in the horizontal direction.
That is, the array shown in FIG. 4 has two pixels (for example, coordinates [-2, -1] and coordinates [-
1, -1]) in a Bayer arrangement. In FIG. 4, for simplicity of description, of two pixels forming such a pair,
The horizontal axis component X of a pixel located on the left side (hereinafter, referred to as a left pixel) is an even number, and the horizontal axis component X of a pixel located on the right side (hereinafter, a right pixel) is an odd number.

The difference between the second embodiment and the first embodiment is that the arrangement of the color information of the image data to be subjected to the interpolation processing is different, and the hardware of the second embodiment is different from the first embodiment. Since the configuration is the same as that of the electronic camera corresponding to the first embodiment shown in FIG. 1, the description is omitted here. The rough operation of the interpolation processing unit 20 in the second embodiment is the same as the operation of the interpolation processing unit 20 in the first embodiment shown in FIG.

Hereinafter, referring to S1 to S6 of FIG.
The operation of the green interpolation processing of the interpolation processing unit 20 in the embodiment will be described. First, the interpolation processing unit 20 calculates a vertical predicted value Gt and a horizontal predicted value Gy of a green interpolation amount for a pixel in which green color information is missing (S1). For example, in FIG. 4, if the coordinates of the pixel where the green color information is missing are [i, j], the interpolation processing unit 20 calculates the vertical predicted value Gt [i, j] of the green interpolation amount as Gt [i , j] = (G [i, j-1] + G [i, j + 1]) / 2... calculated by Equation 23, and when “i = even number” (the two pairs forming the pair as described above) Of the pixels, when the left pixel is a pixel to be interpolated), the horizontal predicted value Gy [i, j] of the green interpolation amount is calculated as Gy [i, j] = (G [i-1, j] + G [ i + 2, j]) / 2... calculated by Equation 24, when “i = odd number” (when the right pixel is a pixel to be interpolated among two pixels forming a pair as described above), green Is calculated by the following equation 25: Gy [i, j] = (G [i−2, j] + G [i + 1, j]) / 2

Here, of the pixels having green color information located in the vicinity of the pixel to be interpolated, the average of the color information of two pixels located in the vertical direction (or the horizontal direction) of the pixel to be interpolated. The value is the vertical prediction of the green interpolation amount (or
(Horizontal predicted value), the vertical predicted value and the horizontal predicted value of the green interpolation amount are calculated according to the << vertical predicted value calculation process >> and << horizontal predicted value of the first embodiment. Calculation processing >>
It may be calculated according to the following.

Next, the interpolation processing section 20 calculates a low frequency component of the spatial frequency of the image data to be subjected to the interpolation processing (S2). For example, the interpolation processing unit 20 replaces the average value of the color information of each pair as described above as the color information of the left pixel of the pair, and calculates the low frequency component of the spatial frequency using the replaced color information.

That is, assuming that the coordinates of the left pixel are [i, j] (where “i = even number”), the interpolation processing unit 20
If the left pixel has green color information G [i, j], the color information
G [i, j] is replaced by G [i, j] = (G [i, j] + G [i + 1, j]) / 2 Expression 26, and the left pixel has red color information R [i, j], the color information R [i, j] is replaced by R [i, j] = (R [i, j] + R [i + 1, j]) / 2... Has blue color information B [i, j], the color information B [i, j] is expressed as B [i, j] = (B [i, j] + B [i + 1, j]) / 2...

As described above, when the color information of all the left pixels is replaced, when the left pixel has green color information G [i, j], the interpolation processing unit 20 determines that the spatial frequency of the left pixel is low. Let the domain component LG [i, j] be LG [i, j] = (G [i, j] + G [i-2, j-1] + G [i-2, j + 1] + G [i + 2, j-1] + G [i + 2, j + 1] + G [i-4, j] + G [i + 4, j] + G [i, j-2] + G [i, j +2]) / 9 ... Calculated by Equation 29, and substitute the value of LG [i, j] as the low-frequency component LG [i + 1, j] of the spatial frequency at the right pixel paired with the left pixel.

When the left pixel has the red color information R [i, j], the interpolation processing unit 20 converts the low frequency component LR [i, j] of the spatial frequency of the left pixel into LR [i, j]. ] = (R [i, j] + R [i-4, j] + R [i + 4, j] + R [i, j-2] + R [i, j + 2] + R [i- 4, j-2] + R [i + 4, j-2] + R [i-4, j + 2] + R [i + 4, j + 2]) / 9 ... The low frequency component LR [i + 1, j] of the spatial frequency at the right pixel paired with the left pixel is defined as LR [i, j].

Further, when the left pixel has blue color information R [i, j], the interpolation processing unit 20 converts the spatial frequency low-frequency component LB [i, j] of the left pixel into LB [i, j]. ] = (B [i, j] + B [i-4, j] + B [i + 4, j] + B [i, j-2] + B [i, j + 2] + B [i- 4, j-2] + B [i + 4, j-2] + B [i-4, j + 2] + B [i + 4, j + 2]) / 9 ... The low frequency component LB [i + 1, j] of the spatial frequency at the right pixel paired with the left pixel is defined as LB [i, j].

Here, the low frequency component of the spatial frequency is calculated according to the equations 9 to 11 of the first embodiment, but the low frequency component of the spatial frequency is calculated by the equation of the first embodiment. 12
１４ 14 may be calculated. Next, the interpolation processing unit 2
In the case of 0, the color information of the high-frequency main image is calculated by removing the low-frequency component of the spatial frequency from the image data to be subjected to the interpolation processing, as in the first embodiment (S3).

That is, the interpolation processing unit 20 calculates the color information of the high-frequency main image by using Expressions 15 to 17 of the first embodiment. Next, the interpolation processing unit 20 calculates the vertical similarity Ct and the horizontal similarity Cy for each pixel where the green color information is missing by comparing the color information of different colors of the high-frequency main image. (S4).

For example, in FIG. 4, if the coordinates of the pixel where the green color information is missing are [i, j], the interpolation processing unit 20
Is similar to the first embodiment with respect to the vertical similarity Ct [i, j] of the pixel, as in the first embodiment: Ct [i, j] = a1 || DG [i, j-1] -DG [i , j + 1] | + a2 ・ (| DRB [i, j] -DG [i, j-1] | + | DRB [i, j] -DG [i, j + 1] |) / 2 ... 18 Further, when “i = even number” (when the left pixel is a pixel to be interpolated),
The horizontal similarity Ct [i, j] is calculated as Cy [i, j] = a1 · | DG [i-1, j] -DG [i + 2, j] | + a2 · (| DRB [i, j ] -DG [i-1, j] | + | DRB [i, j] -DG [i + 2, j] |) / 2 Calculated by Expression 32 and when “i = odd number” (right pixel Is the pixel to be interpolated), the horizontal similarity Ct [i, j] is calculated as Cy [i, j] = a1 || DG [i-2, j] -DG [i + 1, j] | + a2 · (| DRB [i, j] -DG [i-2, j] | + | DRB [i, j] -DG [i + 1, j] |) / 2 Calculated by Expression 33. However, in Expressions 18, 32 and 33, a1 is a positive constant, and a2 is 0 or a positive constant.

That is, the interpolation processing unit 20 calculates the absolute value of the difference between the red (or blue) color information of the high-frequency main image and the green color information of the high-frequency main image, as in the first embodiment. By calculating, the color information of different colors of the high-frequency main image is compared, and the vertical similarity Ct and the horizontal similarity Cy are compared.
Can be calculated. The smaller the values of the vertical similarity Ct and the horizontal similarity Cy, the stronger the similarity.

Next, the interpolation processing unit 20 uses the vertical similarity Ct and the horizontal similarity Cy of a plurality of pixels adjacent to each other to calculate a vertical similarity evaluation value TCt and a horizontal similarity TCt. A similarity evaluation value TCy is calculated (S5). For example, in FIG. 4, assuming that the coordinates of the pixel where the green color information is missing are [i, j], the interpolation processing unit 20 determines that “i = even number” (the case where the left pixel is the interpolation target pixel). , The vertical similarity evaluation value TCt [i, j] and the horizontal similarity evaluation value TCy [i, j] are represented by TCt [i, j] = Ct [i, j] + (Ct [i− 1, j-1] + Ct [i-1, j + 1] + Ct [i + 2, j-1] + Ct [i + 2, j + 1]) / 4 Equation 34 TCy [i, j ] = Cy [i, j] + (Cy [i-1, j-1] + Cy [i-1, j + 1] + Cy [i + 2, j-1] + Cy [i + 2, j +1]) / 4... Calculated by Equation 35, and when “i = odd number” (when the right pixel is the pixel to be interpolated), the evaluation value TCt [i,
j] and the evaluation value TCy [i, j] of the similarity in the horizontal direction are TCt [i, j] = Ct [i, j] + (Ct [i-2, j-1] + Ct [i-2, j + 1] + Ct [i + 1, j-1] + Ct [i + 1, j + 1]) / 4 Equation 36 TCy [i, j] = Cy [i, j] + (Cy [i -2, j-1] + Cy [i-2, j + 1] + Cy [i + 1, j-1] + Cy [i + 1, j + 1]) / 4 ... Calculated by Expression 37.

The smaller the value of the vertical similarity evaluation value TCt and the value of the horizontal similarity evaluation value TCy, the stronger the similarity. Next, the interpolation processing unit 20
Similarly to the embodiment, the direction of strong similarity is determined based on the similarity evaluation value TCt in the vertical direction and the similarity evaluation value TCy in the horizontal direction, and a green interpolation amount is determined according to the determination result. It is determined (S6).

That is, the interpolation processing unit 20 determines the amount of green interpolation based on the conditions 3, 4 and 22 in the first embodiment. As described above, in the second embodiment, the color information of the image data to be subjected to the interpolation processing is the color information of FIG.
, The image to be subjected to the interpolation processing is replaced with a color image that is close to achromatic and has low saturation (a color image having a similar hue). Similarly, similarity can be determined by comparing color information of different colors.

Therefore, according to the second embodiment, the first
Similar to the embodiment, similarity determination can be performed with high accuracy even for a color image including a portion where the hue changes suddenly or a portion with high saturation. Further, according to the second embodiment, similarly to the first embodiment, the characteristics of the original image can be reflected in the high-frequency main image, and the similarity evaluation value of each pixel can be set as a value close to the vicinity. Since the continuity of similarity is reflected by using the similarity of a plurality of pixels located, the accuracy of similarity determination is further improved.

In each of the above-described embodiments, the evaluation value of the similarity in the vertical direction (or the evaluation value of the similarity in the horizontal direction)
Is calculated using the vertical similarity (or horizontal similarity) of a plurality of pixels adjacent to each other, and the vertical similarity evaluation value (or the horizontal similarity evaluation value) ), The vertical similarity (or horizontal similarity) of a single pixel may be applied.

Further, in each of the above-described embodiments, the vertical predicted value and the horizontal predicted value of the green interpolation amount are calculated in S1, and the predicted value corresponding to the direction with the strong similarity is used as the interpolation amount in S6. Alternatively, after determining a direction having a strong similarity without calculating the predicted value, the interpolation amount corresponding to the direction may be calculated by the same calculation as the predicted value. Here, the interpolation amount calculated after determining the direction of strong similarity is an interpolation amount (first interpolation amount) calculated using the color information (equivalent to DG, DR, DB) of the high-frequency main image. Component) and the color information (equivalent to k · LG, k · LR, k · LB) of the image composed of the low frequency components of the spatial frequency. ) May be calculated by combining

When the interpolation amount is calculated in this manner, the first component and the second component of the interpolation amount may be calculated by the same operation, but may be calculated by different operations suitable for each component. It may be calculated. For example, the first interpolation amount
The component is an average value of color information of pixels located in the vicinity regardless of the direction of strong similarity, and the second component of the interpolation amount is an average value of color information of pixels located in the direction of strong similarity. It may be.

Further, in each of the above-described embodiments, the green interpolation amount when it is determined that the similarity is intermediate is the average value of the vertical predicted value and the horizontal predicted value as shown in Expression 22, but
When calculating the interpolation amount without calculating the predicted value, the average value of the green color information of the neighboring pixels may be used. Hereinafter, the operation of the third embodiment will be described.

FIG. 5 is a functional block diagram of the third embodiment. The third embodiment is equivalent to executing an interpolation process by a personal computer using a recording medium on which the interpolation process program according to claim 5 is recorded. 5, components having the same functions as those in the functional block diagram shown in FIG. 1 are denoted by the same reference numerals, and the description of the configuration will be omitted.

The difference between the electronic camera 30 shown in FIG. 5 and the electronic camera 10 shown in FIG.
And the image processing unit 32 are provided instead of the control unit 11 and the image processing unit 18, and an interface unit 33 is newly provided. In FIG. 5, the personal computer 40 has a CPU 41, an interface unit 42, a hard disk 43, and a memory 44.
Are connected to the interface unit 42, the hard disk 43, and the memory 44 via a bus.

The personal computer 40 has
An interpolation processing program (an interpolation processing program for executing the interpolation processing in the same manner as the interpolation processing unit 20 of the first and second embodiments) recorded on a recording medium such as a CD-ROM is installed in advance. Shall be. That is, the hard disk 43 stores such an interpolation processing program in an executable state.

The operation of the third embodiment will be described below with reference to FIG. First, in the electronic camera 30, image data generated in the same manner as the electronic camera 10 shown in FIG. 1 is supplied to the image processing unit 32. The image processing unit 32 performs image processing other than the interpolation processing on the image data.
Image data subjected to image processing is recorded in the form of an image file.

The image file is supplied to the personal computer 40 via the interface unit 33. The CPU 41 in the personal computer 40
When the image file is obtained via the interface unit 42, the above-described interpolation processing program is executed.

That is, in the third embodiment, the same interpolation processing as in the above-described embodiments can be performed by the personal computer 40.

[0080]

As described above, according to the first to fourth aspects of the present invention, the low frequency components of the spatial frequency are removed from the color image to be subjected to the interpolation processing, so that the color image having a saturation close to an achromatic color is obtained. Since similarity determination is performed by comparing color information of different colors with respect to a low color image, similar similarity determination is performed on a color image before the low frequency component of the spatial frequency is removed. Compared with, the accuracy of similarity determination can be reliably improved.

In particular, the invention according to claim 2 generates a high-frequency main image from which at least half of the low-frequency component of the spatial frequency has been removed. Since the high-frequency main image left in a small amount is generated, the characteristics of the original image can be reflected on the high-frequency main image. According to the invention described in claim 4, similarity determination is performed using not only the similarity of the pixel to be interpolated but also the similarity of a pixel located in the vicinity of the pixel to be interpolated. The continuity of similarity can be reflected.

Therefore, according to the second to fourth aspects of the present invention, the accuracy of similarity determination can be further improved. Therefore, according to the first to fourth aspects of the present invention, similarity can be accurately determined without being affected by the characteristics of a color image, and generation of a false color can be suppressed. In the invention according to claim 5,
Functions similar to those of the interpolation processing device according to any one of claims 1 to 4 can be realized by a computer.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of an electronic camera.

FIG. 2 is an operation flowchart of an interpolation processing unit.

FIG. 3 is a diagram illustrating an array of color information of image data to be subjected to an interpolation process in the first embodiment.

FIG. 4 is a diagram illustrating an array of color information of image data to be subjected to an interpolation process according to a second embodiment.

FIG. 5 is a functional block diagram of a third embodiment.

FIG. 6 is a diagram illustrating a conventional interpolation process.

DESCRIPTION OF SYMBOLS 10, 30 Electronic camera 11, 31 Control unit 12 Imaging optical system 13 OLPF 14 Image sensor 15 A / D conversion unit 16 Image buffer memory 17 Gradation conversion unit 18, 32 Image processing unit 19 Recording unit 20 Interpolation Processing units 33, 42 Interface unit 40 Personal computer 41 CPU 43 Hard disk 44 Memory

Claims (5)

[Claims] 1. An interpolation processing apparatus for performing an interpolation process for calculating an interpolation amount corresponding to color information of a missing color for each pixel on a color image including color information of a plurality of predetermined colors. A high-frequency main image generating means for removing a predetermined amount of a low-frequency component of the spatial frequency of the color image and generating a high-frequency main image mainly composed of the high-frequency component of the spatial frequency; The color information of the interpolation target pixel to be subjected to the interpolation processing and the color information of the pixels located in the vicinity of the interpolation target pixel by referring to the color information of the high-frequency main image thus obtained. A similarity determining unit that determines the strength of similarity in a plurality of directions in the interpolation target pixel by comparing the interpolation target pixel, and calculates an interpolation amount of the interpolation target pixel according to a determination result by the similarity determination unit. You And an interpolation amount calculating means. 2. The interpolation processing apparatus according to claim 1, wherein the high-frequency main image generating unit generates a high-frequency main image by removing at least half of a low-frequency component of a spatial frequency of the color image. An interpolation processing device characterized by the above-mentioned. 3. The interpolation processing device according to claim 1, wherein the high-frequency main image generation unit leaves a small amount of low-frequency components of a spatial frequency of the color image,
An interpolation processing device for generating a high-frequency main image. 4. The interpolation processing apparatus according to claim 1, wherein the similarity determination unit is configured to output color information of the high-frequency main image generated by the high-frequency main image generation unit. , And calculates the similarity of the interpolation target pixel by comparing color information of different colors among the color information of the interpolation target pixel and the color information of pixels located in the vicinity of the interpolation target pixel. In addition, the similarity of the pixel located near the interpolation target pixel is calculated in the same manner as the interpolation target pixel, and the similarity of the interpolation target pixel and the similarity of a plurality of pixels located near the interpolation target pixel are calculated. An interpolation processing apparatus for determining the degree of similarity of the interpolation target pixel in a plurality of directions using 5. The computer according to claim 1, wherein
A machine-readable recording medium that records an interpolation processing program for functioning as a high-frequency main image generation unit, a similarity determination unit, and an interpolation amount calculation unit according to any one of the above.