KR100637272B1 - Advanced Color Interpolation Considering Cross-channel Correlation - Google Patents

Abstract

The present invention relates to a color interpolation method in consideration of the correlation between channels, by applying a color filter array (CFA) on the surface of a charge coupled device (CCD), depending on the pixel among R (red), G (green), B (blue) In a color interpolation method of a CCD in which two channel values that are insufficient after selectively obtaining one value are interpolated using values of neighboring pixels, a first process of performing interpolation of the G channel, and a B channel at the R position And a third process of performing interpolation and R channel interpolation at the B position, and a third process of calculating interpolated images of the R channel and the B channel at the G position.

Color, color, channel, interpolation, correlation, CCD, CIS, edge

Description

Color interpolation considering channel correlation {Advanced Color Interpolation Considering Cross-channel Correlation}

1A and 1B are diagrams for explaining a Bayer format of a general CFA,

2 to 4 are diagrams for explaining a channel interpolation method according to the prior art,

5 is a flowchart illustrating a color interpolation process considering channel correlations according to an embodiment of the present invention;

FIG. 6 is a flowchart of a G channel interpolation process in FIG. 5;

7A and 7B are diagrams for explaining G channel interpolation,

8 is a flowchart of an A-channel interpolation process at position C in FIG. 5,

9A and 9B are diagrams for explaining A channel interpolation at a C position,

FIG. 10 is a flowchart of a B and R channel interpolation process at a G position in FIG. 5;

11A and 11B illustrate an interpolation process of B and R channels at a G position.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a color interpolation method, and more specifically, to each pixel of a charge coupled device (CCD) or a CMOS imaging sensor (CIS), red (R), green (G), and blue ( A color image is obtained through a sensor covered by the filter of B), and the color interpolation method considering inter-channel correlation to remove visible errors is performed by interpolating lost color channel values in consideration of inter-channel correlation. .

In general, digital cameras or camcorders use CCD or CIS instead of film.

The CCD or CIS is a device that converts a brightness value applied to a sensor through a lens in one pixel into a digital signal. In other words, the value accepted by the sensor is the brightness value, which accepts a black and white image instead of the color image we see with our eyes. To obtain a color image, a color image may be obtained by obtaining R, G, and B values at all points using a sensor in which R, G, and B filters are applied to each pixel. At this time, the reason for using R, G, B is because the three primary colors of the light, because the cone band in the human eye is also the band of the band mainly react.

As described above, in order to obtain a high quality color image, the number of pixels of a CCD or CIS three times that of a black and white image is required. Since the price of the sensor is an expensive device that greatly influences the camera's pricing, broadcast equipment uses expensive 3CCDs (receive R, G, and B in each of the three CCDs to obtain the primary colors), but ordinary users This is an unattainable price and is not widely used because of the additional technology going inside for 3CCD.

In general, CFA (Color Filter Array) is applied to one CCD surface to obtain one value among R, G, and B according to the pixel, and then two missing channel values are used by interpolating the values of neighboring pixels. CFA typically uses Bayer Format. At this time, the quality of the obtained image depends on which interpolation method is used.

Interpolating these lost color channel values is the first step in a variety of steps in digital cameras and other video processing. Depending on how good the results are in this process, subsequent processes may yield better results with simpler processes. By the way, the existing interpolation methods are simple considering the processing time and memory of the system, but there are a lot of prominent errors in the resulting image.

1A and 1B are diagrams for explaining Bayer formats most commonly used among various CFAs, and it can be seen that any of four types are continuously connected. In terms of processing, the indexes are only pushed one by one, but in the same way.

Color interpolation techniques restore the G channel first from pixels with only R or B channel values, and then fill the empty values in order of B, R, R, and B channels. Restore At this time, there are two different cases of restoring the B channel or the R channel. Sometimes R or B channel values are filled in the pixel where only the G channel existed. In some cases, B is filled in the pixel where R channel exists or R is filled in B.

2 to 4 are diagrams for explaining a channel interpolation method according to the prior art.

2 is a method of Nearest Neighbor Replication.

1) Reconstruction of G channel (pixels with only R values or pixels with only B values)

In A ₂₂ , the G channel value is substituted with the closest value as shown in Equation 1 below.

[ Equation 1 ]

G ₂₂ = G ₂₁

For simplicity, R and B are replaced with A depending on the situation. That is, A is a representative character which may be R or B. In this case, the counterpart channel (B is interested in R and B is interested in R is R).

2) A channel reconstruction (pixels with only R values or pixels with only B values)

In C ₃₃ , the A channel value is calculated by substituting the nearest point of the upper left side as shown in Equation 2 below.

[ Equation 2 ]

A ₃₃ = A ₂₂

FIG. 3 is a diagram for describing bilinear interpolation.

1) Restoration of G channel (pixel with R value only or pixel with B value only)

The G channel value at A ₃₃ is obtained as in Equation 3 below.

[ Equation 3 ]

Average the G-values of the surrounding four pixels.

2) Restoring A channel (pixel with R value only or pixel with B value only)

The channel value of A in C ₂₂ is obtained as shown in Equation 4 below.

[ Equation 4 ]

Average the A values of the four pixels around the diagonal line.

3) Restoring A channel (pixel with G value only)

The channel A value in G ₂₃ is obtained as shown in Equation 5 below.

[ Equation 5 ]

The A channel value at G ₃₂ is obtained as shown in Equation 6 below.

[ Equation 6 ]

Average two directions in which the same channel as the channel to be restored is located.

FIG. 4 is a view for explaining a smooth hue transition interpolation according to the prior art, and may be divided into a linear exposure space and a logarithmic exposure space.

The former case is based on the assumption that the ratio between the channels, that is, the division of the two channels, is constant. The latter works on the assumption that the difference between the two channels, that is, the subtraction, is constant. The relation of ratios has a lot of division operations and increases the amount of computation. This section only introduces the difference relationship. The subtraction operation can be processed in the same processing cycle as the addition operation, so there is an improvement in performance over the bilinear method and there is no significant difference in the calculation amount.

The assumption here is that GB or GR, that is, the difference between the G and other channels, is constant.

1) Reconstruction of G channel (pixels with only R values or pixels with only B values)

The G channel value at A ₃₃ is obtained as shown in Equation 7 below.

[ Equation 7 ]

The average value of the difference image G-A ' at four neighboring pixels is added to the A value present in the current pixel.

2) Restoring A channel (pixel with R value only or pixel with B value only)

The channel A value in C ₂₂ is obtained as shown in Equation 8.

[ Equation 8 ]

The G's use the ones already interpolated in ①. The difference image values of four pixels around a diagonal line are averaged as in 1).

3) Restoring A channel (pixel with G value only)

The channel A value in G ₂₃ is obtained as in Equation 9 below.

[ Equation 9 ]

The channel A value in G ₃₂ is obtained as in Equation 10 below.

[ Equation 10 ]

G already uses those interpolated in 1) above. Like bilinear, average two directions of the same channel as the channel you want to reconstruct.

As described above, in order to reduce the hardware load of the conventional system, the above-mentioned neighboring neighbor replication method and bilinear interpolation method are mainly used.

It is the form of the patents that have been made until recently, modified by their own system without applying the above-described method.

The problem with either method is that systems are simple enough to actually use, but they don't eliminate unobtrusive false color errors or moire effects. These images occur mainly at edges (edges, outlines or boundaries of objects), which are important information in the image, and thus the image quality looks worse.

False Color Error is a phenomenon where a color that is completely different from the original color is interpolated, and pixels in various places appear to be noticeably bounced out around.

The moire effect is often seen in checkered patterns, a phenomenon often seen when watching TV. People wearing checkered clothes or ties can see colorful colors like rainbows. There are studies to improve this one after another, but the results were not satisfactory.

The present invention is to solve the above-mentioned problems of the prior art, the purpose of which is red (R), green (G), blue for each pixel of the CCD (Charge Coupled Device) or CIS (CMOS Imaging Sensor) A color image is obtained through the sensor covered with the filter of (B), and the interpolation of the lost color channel value is taken into account in consideration of the correlation between the channels, thereby providing a color interpolation method considering the inter-channel correlation to remove visible errors. Is in.

In order to achieve the object of the present invention, the color interpolation process considering the inter-channel correlation according to the present invention is coated with a color filter array (CFA) on a charge coupled device (CCD) surface, depending on the pixel R (red), G (green) In the color interpolation method of a CCD in which two channel values which are insufficient after obtaining one value of B (blue) are interpolated by using a neighboring pixel value, the first process of performing interpolation of the G channel, and And a third process of performing interpolation of the B channel at the R position and the R channel interpolation at the B position, and a third process of calculating the interpolated image of the R channel and the B channel at the G position.

When described in detail with reference to the accompanying drawings, the operation according to an embodiment of the present invention made as follows.

FIG. 5 is a flowchart illustrating a color interpolation process considering channel correlation according to an embodiment of the present invention. First, G channel interpolation, (in the case of a pixel having only R value or a pixel having only B value) (S110) and an R position In the B-channel interpolation and the B position, and performs the R channel interpolation (S120), and performs the R and B channel interpolation at the G position (S130).

The interpolated image is calculated through the above process.

FIG. 6 is a detailed flowchart of a G channel interpolation process in FIG. 5. First, a processing channel is converted. That is, it generates 'D _A ' (S111).

The D _A generation method is described below with reference to FIG. 7A.

Interpolating at a position A ₃₃ is described as an example.

First, for the sake of explanation, 'T' is Top, 'B' is Bottom, 'L' is Left, 'R' is Right and A is one of R and B. Set as representative. D (Difference) is a channel of the difference image.

That is, D _R = G-R or D _B = G-B . In this case, D _AT means D _A (GR or GB) in the upward direction from R or B.

In the G-channel interpolation method, it is important to process everything in the difference image channel rather than the existing R, G, and B channels. Therefore, in the case of the position A ₃₃ is calculated as in Equation 11 (S112).

[ Equation 11 ]

This is represented as the relationship between the existing A channel and the newly created D channel, as shown in FIG. 7B.

In order to interpolate the G channel, it is necessary to determine whether the pixel of interest belongs to an edge, and if so, to determine in which direction the edge exists.

In order to know the edge presence and direction, we use the absolute value of the gradient to simplify the system.

First, α = D _AT - D _AB | and β = D _AL - D _AR | are calculated (S112).

That is, the difference value of the difference value of an up-down direction and the left-right value is calculated beforehand.

The processing method for determining the interpolation method according to the situation is as follows.

Two thresholds are used in the present invention. Called Th 1 and Th 2 respectively. Th 1 is related to the difference between α and β, and Th 2 is related to the smaller of α and β. According to the relationship between the thresholds and α, β is classified as follows (S113).

In the Pseudo Code method,

Same as

Here, Th 1 is 2% of the maximum brightness value of the image pixel, and Th 2 is 8% of the maximum brightness value of the image pixel.

Ⓐ edge direction interpolation (S114)

Most of the edges are in the vertical or horizontal direction. The basis of the algorithm is to average over the smaller of α and β. If you get the smaller of the two using Get_min () above, it is more likely that the direction is the edge direction without crossing the edge, so you can use the average value of the D channel in that direction.

If α is small, Equation 12

[ Equation 12 ]

If β is small, Equation 13

[ Equation 13 ]

Use

Here, the reason for using the lowercase letter g is to distinguish between the pixel where G originally exists and the interpolated pixel. The uppercase letter is the original pixel and the lowercase letter is the interpolated pixel.

Ⓑ Error prevention interpolation (S115)

If the smaller of α, β exceeds the threshold Th 2, it is likely not appropriate in both the transverse and longitudinal directions. So you need to take proper care to prevent splashing. This is the case where very few occur in the whole image, but this is the most prominent problem when an error occurs due to mishandling. Here, the 'Suppression' function is used, as shown in Equation 14 below.

[ Equation 14 ]

Here, the Suppression function selects the value having the smallest difference between two values of the difference image channels positioned in four directions of up, down, left, and right. A total of six cases will come out. So if you choose two interpolations with the smallest difference, you can reduce the error a lot. Another option is to modify the median to simplify the system.

The bilinear interpolation method corresponds to the largest number of pixels in the actual process. If the difference between α and β is minute, the excitation is a flat point, and thus an interpolated G channel is calculated as shown in Equation 15 (S116).

[ Equation 15 ]

FIG. 8 is a flowchart illustrating interpolation of the B channel at the R position and the R channel at the B position according to an exemplary embodiment of the present invention. First, the processing channel is converted. That is, it generates 'D _A ' (S111).

For example, the case of C ₂₂ is as follows. That is, since the G channel has already interpolated, the G value is set to exist. Then there are G values in C ₂₂ and G values in A. In order to obtain the A value in C ₂₂ , it is necessary to convert to the difference image channel in the diagonal direction where the A values exist. The result is shown in FIG. 9B and the following equation (16).

[ Equation 16 ]

As in the case of interpolating the G channel, α and β are defined as follows (S122).

Here, α is for judging the edge from the upper left to the lower right direction, and β is for judging the edge from the upper right to the lower left direction. Set it to the upper left and right upper edges. Since the remaining methods are the same as the interpolation method of the G channel described above, the detailed description thereof will be omitted. In addition, the interpolation method is determined in the same way as the G channel interpolation method (S123).

Ⓐ edge direction interpolation (S124)

When α is small, the following equation (17) is used.

[ Equation 17 ]

If β is small, the following equation (18) is used.

[ Equation 18 ]

Use

In this case, since the lower-case alpha is used, the pixel in which the original A exists is distinguished from the interpolated pixel. The uppercase letter is the original pixel and the lowercase letter is the interpolated pixel.

Ⓑ Error prevention interpolation (S125)

If the G channel is well interpolated, there will be no significant difference in the result even if the A channel interpolation is simple. Therefore, error prevention interpolation may be excluded in the R and B channel interpolation. If necessary, the Suppression function is used as shown in Equation 19.

[ Equation 19 ]

In this case, the Suppression function selects a value having the smallest difference between two values of difference image channels positioned in four diagonal directions. At this time, a total of six cases will come out. Therefore, if two interpolations are made with the smallest difference, the error can be greatly reduced. Another option is to modify the median to simplify the system.

Ⓒ For bilinear interpolation, this is the case with the most pixels in the actual process. If the difference between the values of α and β is minute, the excitation is a flat point and is obtained as shown in Equation 20 (S126).

[ Equation 20 ]

FIG. 10 is a flowchart illustrating an A channel (B or R channel) interpolation process at a G position according to an embodiment of the present invention. All of them are calculated. Here, the process of finding the r and b values in G and finishing the interpolation. The interpolation process is the same as the A channel interpolation process at the C position of FIG. 8, but uses a smaller difference value in the up, down, left, and right directions instead of a diagonal line.

The case of G ₂₂ is as follows. In the above processes, R, G, and B are interpolated to the pixels on the top, bottom, left, and right sides to complete the process. In order to obtain the A value from the G ₂₂ , the A value should be converted into a difference image channel in the interpolated left and right directions interpolated with the A values. When A and C are replaced, the same method is used to interpolate. The result is shown in FIG. 11B and Equation 21. (S131) (S132)

[ Equation 21 ]

Here, α and β must be obtained as in the interpolation of the G channel, and are defined as in Equation 22 below.

[ Equation 22 ]

Here, α is for judging the edge in the vertical direction, and β is for judging the edge in the left and right directions. The remaining methods are the same as the G channel interpolation method described above. The interpolation method decision is the same as the method presented for G channel interpolation.

In other words, the processing method using the coefficients is an edge interpolation, an error prevention interpolation, and a bilinear interpolation method (S133).

First, the edge direction interpolation method (S134)

If α is small, Equation 23

[ Equation 23 ]

Use If β is small, Equation 24

[ Equation 24 ]

Use

In this case, the reason for using the lowercase α is to distinguish between the pixel where A originally exists and the interpolated pixel. The uppercase letter is the original pixel, and the lowercase letter is the interpolated pixel.

Second, ⓑ the error prevention interpolation method (S135)

If the G channel is well interpolated, there will be no significant difference in the result even if the A channel interpolation is simple. Therefore, error prevention interpolation may be excluded in the R and B channel interpolation. If necessary, use the Suppression function as shown in Equation 25 below.

[ Equation 25 ]

Here, the Suppression function selects the value having the smallest difference between two values of the difference image channels positioned in four directions of up, down, left, and right. A total of six cases will come out. So if you choose two interpolations with the smallest difference, you can reduce the error a lot. Another option is to modify the median to simplify the system.

Third, in the? Bilinear interpolation method (S136), the most pixels in the actual processing correspond to this case. If the difference between the values of α and β is minute, the excitation is a flat point and is calculated through the following equation (26).

[ Equation 26 ]

Although a preferred embodiment according to the present invention has been described above, various modifications are possible according to the interpolation method, and those skilled in the art may various modifications and modifications without departing from the claims of the present invention. It is understood that examples may be practiced.

As described above, the color interpolation method considering the inter-channel correlation according to the present invention obtains a color image through a sensor in which R, G, and B filters are applied to each pixel of the CCD or CIS. By interpolating the lost color channel values, there is an effect of eliminating visible errors.

Claims (13)

Color filter array (CFA) is applied to the surface of charge coupled device (CCD) to obtain one of R (red), G (green), and B (blue) depending on the pixel. In the color interpolation method of a CCD which interpolates the value of, Converting a processing channel to generate a channel D _{A of the difference} image; Calculating the number of edge plate steps using the absolute value of the gradient; And performing the interpolation of the G channel by calculating interpolated G channel values by performing interpolation of edge direction on the upper left or right upper edge, error prevention interpolation on the sloped edge, and bilinear interpolation on the flat area according to the number of edge plate steps. A first process of doing; A second process of performing B channel interpolation at the R position and R channel interpolation at the B position; And A third step of calculating an interpolated image for the R channel and the B channel at the G position; Color interpolation method considering the correlation between the channels, characterized in that made. delete The method of claim 1, The G channel value g interpolated through the edge direction interpolation is represented by If α is small

If β is small

(Where α = │ D _AT − D _AB │, β = │ D _AL − D _AR │) Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 1, The G channel value g interpolated through the error prevention interpolation is represented by

(In this case, the 'Suppression' function selects the value having the smallest difference between two values of the difference image channels positioned in four directions of up, down, left, and right. Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 1, The G channel value g interpolated through the bilinear interpolation is represented by

Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 1, The second process may include converting a processing channel to generate a channel D _{A of the difference} image; Calculating the number of edge plate steps using the absolute value of the gradient; And According to the number of edge plate steps, the inter-channel correlation characterized in that it comprises the step of calculating the interpolated A channel by performing the interpolation of the edge direction on the upper left or right edge, the error prevention interpolation on the sloped edge, bi-linear interpolation on the flat area Color interpolation method considering relations. The method of claim 6, The A channel value a interpolated through the edge interpolation is represented by the following equation. If α is small

, If β is small

(Where α = │ D _ATL - D _ABR │, β│ D _ABL - D _ATR │) Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 6, The A channel value (a) interpolated through the error prevention interpolation is represented by

(In this case, the 'Suppression' function selects the value with the smallest difference between two values of difference image channels positioned in four diagonal directions.) Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 6, The A channel value (a) interpolated through the bilinear interpolation is expressed by

Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 1, The third process may include generating a channel D _{A of a difference} image by converting a processing channel; Calculating edge plate number (upper left, right, left and right α, β generation) using the absolute value of gradient; And Calculating interpolated A-channel values by performing edge interpolation if the upper left or the right upper edge, error preventing interpolation if the sloped edge, and bilinear interpolation if the flat region is determined according to the number of edge plate steps. Color interpolation method considering the correlation between The method of claim 10, The G channel value g interpolated through the edge direction interpolation is represented by If α is small

, If β is small

(Where α = │ D _AT − D _AB │, β = │ D _AL − D _AR │) Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 10, The A channel value (a) interpolated through the error prevention interpolation is represented by

(In this case, the 'Suppression' function selects the value of the difference between two values of the difference image channels located in four directions of up, down, left, and right. Color interpolation method considering the correlation between the channels, characterized in that calculated through. The method of claim 10, The A channel value (a) interpolated through the bilinear interpolation is expressed by

Color interpolation method considering the correlation between the channels, characterized in that calculated through.

Images