KR20080078262A - Region adaptive directional demosaicing method - Google Patents

Abstract

A region adaptive directional color interpolation method is provided to interpolate colors of G, B, and R channels in G, B, and R pixels for reducing color interpolation errors and obtaining the high resolution interpolation image. A region adaptive directional color interpolation method comprises the steps of: receiving image information of each pixel through a Bayer CFA(Color Filter Array)(210); dividing a region into flat, normal edge, and pattern edge regions(220); calculating a horizontal value difference image value to generate a difference image channel(230); minimizing an error by using a DLLMMSE(Difference Local Linear Minimum Mean Square Error) filter(240); carrying out a region adaptive color interpolation(250); and outputting the interpolated image(260). The step(250) has steps of; setting a proper vertical and horizontal difference image value to calculate the total of weighting values(251), interpolating a color of a G channel in a pixel having R and B values(252), interpolating a color of a B channel in a pixel having an R value and interpolating a color of an R channel in a pixel having a B value(253), and interpolating the color of the R and B channels in the pixel having the G value to interpolate the color of all pixels(254).

Description

Region-adaptive directional color interpolation method {REGION ADAPTIVE DIRECTIONAL DEMOSAICING METHOD}

1 is an illustration of Bayer pattern of a general color filter array (CFA)

2 is an exemplary view showing the characteristics of a flat region and an edge region in a color interpolation operation

3 is an exemplary diagram showing an area of a pixel for distinguishing a general edge area and a pattern edge area;

FIG. 4 is a flowchart illustrating operations for region division in a region adaptive directional color interpolation method according to an embodiment of the present invention. FIG.

5 is an exemplary diagram of a mask used for G channel interpolation in a flat region of a pixel having an R value or a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention.

6 is an exemplary diagram of a mask used for G channel interpolation in a general edge region of a pixel having an R value or a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention.

7 illustrates an example of a mask used for G channel interpolation in a pattern edge region of a pixel having an R value or a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention.

FIG. 8 is an exemplary diagram illustrating a reference pixel used for interpolating B channel values in a pixel having an R value and interpolating an R channel value in a pixel having a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention.

9 is an exemplary diagram illustrating a reference pixel used for interpolating R and B channel values of a pixel having a G value in a region adaptive directional color interpolation method according to an embodiment of the present invention.

10 is an operation flowchart of a region adaptive directional color interpolation method according to an embodiment of the present invention.

11 is a block diagram of a color interpolation apparatus according to an embodiment of the present invention.

12 is an exemplary view comparing an image using a conventional color interpolation method with an image using a color interpolation method according to an embodiment of the present invention.

13 is an exemplary view comparing an image using a conventional color interpolation method with an image using a color interpolation method according to an embodiment of the present invention.

14 is an exemplary view comparing an image using a conventional color interpolation method with an image using a color interpolation method according to an embodiment of the present invention.

15 is an exemplary view illustrating an enlarged comparison of an image using a conventional color interpolation method and an image using a color interpolation method according to an embodiment of the present invention.

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a color interpolation method in a digital photographing apparatus using an image sensor such as a single charge coupled device (CCD) or a CMOS imaging sensor (CIS), and more particularly, to an area adaptive color interpolation method.

Generally, digital cameras and camcorders use CCD or CMOS (Complementary Metal Oxide Semiconductor) instead of film. CCD or CMOS is a device that converts the brightness value applied to the sensor through the lens in one pixel into a digital signal. In other words, the value accepted by the image sensor is the brightness value, which accepts a black and white image instead of the color image we see with our eyes. To obtain a color image, a color image may be obtained by obtaining R, G, and B values at all points (pixels) using a sensor in which R, G, and B filters are applied to each pixel. At this time, the reason for using R, G, B is because the three primary colors of the light, because the cone band in the human eye is also the band of the band mainly react.

As described above, in order to obtain a high quality color image, three times the number of pixels of a CCD or CMOS is required compared to a black and white image. Since the price of the sensor is an expensive device that greatly influences the camera's pricing, broadcast equipment uses expensive 3CCDs (receive R, G, and B in each of the three CCDs to obtain the primary colors), but ordinary users This is an unattainable price and is not widely used because of the additional technology going inside for 3CCD.

In the case of Single CCD, one color information is stored among the color information of several channels. Therefore, to obtain the complete image information, the color information of another channel not stored in the pixel must be estimated (interpolated) from the surrounding pixel information of the pixel. . At this time, the quality of the obtained image depends on which interpolation method is used. Depending on how good the results are in this process, subsequent processes may yield better results with simpler processes.

Color interpolation algorithm is a field of image processing that has been studied a lot recently. The existing method is relatively simple because it takes into account the processing time of the system or the hardware implementation. Among these color interpolation methods, a relatively simple method can be thought of as smooth hue transition interpolation considering the correlation between colors and edge adaptive color interpolation considering the direction of edges. ) There is a law. Let's first look at the flat color change interpolation.

[Flat color change interpolation]

The conventional flat color change interpolation method performs color interpolation using the assumption that the difference between two channels, that is, the subtracted value, is constant in a certain region. In other words, the correlation between colors can be considered by obtaining G-B and G-R and interpolating them.

1) Restoration of G channel (pixels with only R values or pixels with only B values)

1 is an exemplary diagram of a Bayer pattern of a general color filter array (CFA). First, 'T' stands for Top, 'B' stands for Bottom, 'L' stands for Left, 'R' stands for Right, and R for simplicity. And B are replaced with A depending on the situation. That is, A is a representative character which may be R or B. In this case, the counterpart channel (B is the counterpart channel when interested in R, and R is the counterpart channel when interested in B) is denoted by C. D (Difference) is a channel of the difference image. That is, D _R = GR or D _B = GB. D _AT also means D _A (GR or GB) in the upward direction from R or B.

A ₃₃ with reference to FIG. 1 This section describes how to interpolate G channel values. First, the difference image D _AT , D _AB , D _AL , and D _AR of four pixels around the top, bottom, left, and right of the pixel to be interpolated are calculated, and the average value of the four difference image values is calculated to find the A present in the pixel to be interpolated. Add to the value to get the value of G ₃₃ . The above process is represented by Equation 1 and Equation 2 below.

2) Restoring A channel (pixel with R value only or pixel with B value only)

C ₂₂ with reference to FIG. 1 The following describes how to interpolate A channel values. First, calculate the difference image D _ATL , D _ATR , D _ABL , and D _ABR of four neighboring pixels in the diagonal direction of the pixel to be interpolated, and calculate the average value of the four difference image values from the G value of the pixel to be interpolated. Subtracting it gives the value of A ₂₂ . When the above process is represented by Equation 3, Equation 3 and Equation 4 below.

In Equations 3 and 4, g ₁₁ , g ₁₃ , g ₃₁ , g ₃₃ , and g ₂₂ use interpolated values in the process of restoring the G channel.

3) Restoring A channel (pixel with G value only)

A method of interpolating A channel values in G ₂₃ will be described with reference to FIG. 1. First, calculate D _AT and D _AB values of the two pixels above and below the A channel value among the pixels adjacent to the interpolation pixel, calculate the average value of the two difference image values, and subtract A from the G value of the pixel to be interpolated. You can get the value of ₂₃ . When the above process is represented by Equation 5, Equation 5 and 6 are as follows.

In the case of interpolating the A channel value in G ₃₂ , the difference image D _AL and D _AR of two left and right pixels having the A channel value among the pixels adjacent to the interpolation pixel are calculated, and the average value of the two difference image values is calculated. The value of A ₃₂ can be obtained by subtracting the G value of the pixel to be interpolated. Equation (7) and Equation (8).

In Equations 5, 6, 7, and 8, g ₁₃ , g ₃₃ , g ₃₁ , and g ₃₃ use interpolated values in the process of restoring the G channel.

[Edge Adaptive Color Interpolation]

The edge adaptive color interpolation method is a method of interpolating not to cross the edge in consideration of the direction of the edge during color interpolation in order to solve the problem of not considering the edge information in the conventional method. Color interpolation is performed on the assumption of smooth color transition.

1) Restoration of G channel (pixels with only R values or pixels with only B values)

In A ₃₃ of FIG. 1, the G channel value is obtained as follows. First, the vertical difference value of D _A defined in Equation 1 is set to ΔV and the horizontal difference value is set to ΔH, and is defined as in Equation 9 below.

The edges in the vertical and horizontal directions may be considered by comparing the values of ΔV and ΔH with each other and performing interpolation in a direction in which the values are small.

① Vertical interpolation

When ΔV is smaller than ΔH, color interpolation is performed in the vertical direction. In this case, g ₃₃ is represented by Equation 10 below.

② Horizontal interpolation

When ΔH is smaller than ΔV, color interpolation is performed in the vertical direction. In this case, g ₃₃ is represented by Equation 11 below.

③ Flat Area Interpolation

If the difference between ΔV and ΔH is smaller than the threshold, then the interpolation is performed in the same manner as the flat color change interpolation method because it is a flat region. The threshold value is determined according to the image. In this case, g ₃₃ is represented by Equation 12 below.

2) A channel reconstruction (pixels with only R values or pixels with only B values)

The channel A value in C ₂₂ is calculated as follows. The values of the G channel are all present by interpolating the equations (10, 11, 12). Then, G values exist in the R channel and G values exist in the B channel. To find the B value from R, calculate the difference between the B value and the G value in the diagonal direction to the G value in the R channel, and then subtract it by averaging it appropriately. Therefore, it can be expressed as Equation 13 below.

Where g already uses interpolated values using equations (10), (11), (12). The average value of the difference image values (D _ATL , D _ABR , D _ABL , D _ATR ) of four pixels around the diagonal direction is obtained and subtracted from the G value of the pixel to be interpolated to obtain an A ₂₂ value.

3) A channel reconstruction (pixels with G values only)

A method of interpolating A channel values in G ₂₃ will be described with reference to FIG. 1. First, calculate D _AT and D _AB values of two pixels above and below the A channel value among the pixels adjacent to the interpolation pixel, calculate the average value of the two difference image values, and subtract A from the G value of the pixel to be interpolated. You can get the value of ₂₃ . The above process is represented by the following equations (14) and (15).

In the case of interpolating the A channel value in G ₃₂ , the difference image D _AL and D _AR of two left and right pixels having the A channel value among the pixels adjacent to the interpolation pixel are calculated, and the average value of the two difference image values is calculated. The value of A ₃₂ can be obtained by subtracting the G value of the pixel to be interpolated. The equations are the same as equations (16) and (17).

In Equations 14, 15, 16, and 17, g ₁₃ , g ₃₃ , g ₃₁ , and g ₃₃ use interpolated values in the process of restoring the G channel. g already uses interpolated values using equations (10), (11), (12). Like flat color change interpolation, average the two directions in which the channel is located, such as the channel you want to reconstruct.

[Problems of Traditional Methods]

When interpolating color information of other channels that do not come into the input using a single CCD, the existing methods do not consider edge information effectively and do not consider the correlation between colors. It has a problem that the color error (False Color Error) or the moire effect (Moire Effect) that occurs mainly in the outlines or boundaries of the) can not be removed well. In addition, in consideration of only the flat region and the edge region, there is a problem such as a moiré effect in the pattern edge region.

An object of the present invention is to provide a method for obtaining a high resolution interpolation image while minimizing occurrence of errors that may occur during color interpolation by using an algorithm adaptive to a region and effectively considering edge directionality.

In order to achieve this, the present invention provides a method for directional color interpolation in a region adaptive direction in a digital photographing apparatus using an image sensor, wherein each pixel is arranged on a color filter array (CFA) to perform color adaptive interpolation. Is divided into a flat region, a general edge region, and a pattern edge region, and after estimating difference image values to minimize errors by generating difference image channels in vertical and horizontal directions, the vertical and horizontal directions at positions to be interpolated. Interpolating G (Green) channel values using the weighted sum by calculating the weighted sum of the difference image values of, and using the interpolated G channel values to obtain an edge direction indication function and reference around the pixels to be interpolated. Interpolate the B (Blue) channel value in the pixel with R (Red) value by using the weighted average by applying the edge direction display function as weight to the difference image value of the pixel Interpolating an R channel value from a pixel having a B value, obtaining an edge direction display function using the interpolated G channel value, and displaying the edge direction display function on a difference image value of a reference pixel around the pixel to be interpolated. It is characterized by including the process of interpolating the R channel and the B channel in the pixel having a G value by using the weighted average.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, specific details such as specific components are shown, which are provided to help a more general understanding of the present invention, and the specific details may be changed or changed within the scope of the present invention. It is self-evident to those of ordinary knowledge in Esau. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

False Color Error is a phenomenon where literally interpolated colors are completely different from the original colors, and pixels in various places appear not to be smooth with the surroundings. Also, the moiré effect is mostly seen in checkered patterns, which are often seen when watching TV. People wearing checkered clothes or ties can see colorful colors like rainbows. If you consider the correlation between each channel of color and the direction of the edges according to the area, you can minimize the color error or ripple effect.

The present invention divides a region into a flat region, a general edge region, and a pattern edge region for use of an algorithm adaptive to the region. Based on this, the correlation between the channels of the color is well considered, and the area adaptive color interpolation considering the direction of the edge according to the area is performed. In order to minimize the color errors, DLLMMSE (Directional Local Linear Minimum Mean Square Error) filter is used.

[Divide region]

Conventional color interpolation schemes have advantages and disadvantages in each region. Accordingly, the present invention proposes a technique of dividing a region and taking a proper manner in order to take advantage and exclude disadvantages.

2 is an exemplary diagram illustrating characteristics of a flat region and an edge region in a color interpolation operation. 2 shows the color interpolation operation of the G channel. In the flat region, the horizontal interpolation result is similar to the vertical interpolation result, but in the edge region, the horizontal interpolation result is the same as the horizontal interpolation result. This shows an example of what is different.

1) flat area

In the case of the flat region, since the surrounding pixels have similar characteristics, reasonable results can be obtained without considering the orientation. Therefore, when the current pixel exists in the flat region, the horizontal interpolated result and the vertical interpolated result have similar values. The difference between the horizontal value and the vertical value is calculated using Equation 18 below, and the flat area is determined using the difference.

In Equation 18, G _H5 is a result value interpolated in the horizontal direction, G _v5 is a result value interpolated in the vertical direction, and G _flat is a difference value between the result interpolated in the horizontal direction and the vertical interpolated result.

Where G _flat For regions where <Th _flat , the interpolated results are determined to be similar. That is, this area is determined as a flat area. Th _flat is a flat threshold value for determining a flat area, and generally has a value between 1 and 15.

1) edge area

In the case of the edge region, since the surrounding pixels have different characteristics, the directionality can reduce the error. The edge region can be divided into an edge region having a general edge region and a pattern. In the case of general edge regions, reasonable results can be obtained through conventional edge adaptive color interpolation. However, in the case of edges with patterns, it is difficult to estimate the exact value using conventional edge adaptive color interpolation. Therefore, we propose a method of improving the quality of the image by distinguishing these areas and taking a reasonable method.

3 is an exemplary diagram illustrating a region of a pixel for distinguishing a general edge region and a pattern edge region. In the case of the edge region, as shown in (a) of FIG. 3, it may be determined whether the region is a general edge region or a pattern edge region using a region composed of nine pixels of up, down, left, and right crisscross shapes of the reference pixel. Referring to FIG. 3B, when the horizontal color difference value and the vertical color difference value are considered, it can be seen that the difference value between the two values is large in the normal edge region. However, it can be seen that the difference between the two values is relatively small in the pattern edge region. The color difference value in the horizontal direction and the color difference value in the vertical direction are represented by Equations 19 and 20 below.

In Equations 19 and 20, Diff _H and Diff _V are color difference values in horizontal and vertical directions, respectively.

4 is a flowchart illustrating an operation of distinguishing regions in a region adaptive directional color interpolation method according to an embodiment of the present invention. Referring to FIG. 4, the image information is first received through the Bayer CFA, and in step 501, the size of the G _flat and the Th _flat are compared, and if the G _flat is smaller than the Th _flat , the flat region is determined. Conversely, if G _flat is greater than or equal to Th _flat , go to Step 502 | Compares Diff _V - Diff _H | and TH _edge (edge threshold) | If the value of Diff _V - Diff _H is less than TH _edge , it is regarded as an edge region having a pattern, and | If the value of Diff _V - Diff _H | is greater than or equal to TH _edge , it is determined as a general edge area.

[G-channel color interpolation in pixels with R or B values]

The local statistics have different characteristics according to the area of color interpolation, and the values of the statistics have advantages and disadvantages according to the mask type. Therefore, it is necessary to form the mask and color interpolation according to the effective division in the area classification process.

 The G-R signals in the vertical and horizontal directions used in the present invention may be represented by Equations 21, 22, and 23 below.

,

는 각각 수직, 수평 방향의 G-R 값이고, 상기 수학식 23에서

,

는 각각 수평, 수직 방향 보간으로 인해 발생하는 오류 및 센서의 잡음이다.

,

를 최소화하기 위해서 기존의 잡음 제거에 많이 사용되는 필터인 LLMMSE(Local Linear Minimum Mean Square Error)를 적용하면 하기의 수학식 24와 같은 형태를 가지게 된다. In Equation 21, 22, 23

,

Are GR values in the vertical and horizontal directions, respectively,

,

Are errors caused by horizontal and vertical interpolation, respectively, and sensor noise.

,

In order to minimize the LLMMSE (Local Linear Minimum Mean Square Error), a filter commonly used for removing noise, the following equation (24) is obtained.

는 LLMMSE 필터를 이용하여 추정된 차영상 값이다.

,

는 국부 통계치로 각각

의 평균과 분산이다.

은 오류의 분산으로 제안되는 알고리즘에서는 평탄영역에서의 분산을 구함으로써 그 값을 추정할 수 있다. 여기서 국부 통계치

의 평균과 분산은 에지를 가로질러서 그 값이 추정되지 않도록 가중치를 사용하여 그 값을 계산한다.

는 하기의 수학식 25로 나타내고,

는 하기의 수학식 26으로 나타낸다. In Equation 24

Is the difference image value estimated using the LLMMSE filter.

,

Are local statistics,

Is the mean and the variance.

In the algorithm proposed as the variance of the error, the value can be estimated by obtaining the variance in the flat region. Where local statistics

The mean and the variance of are computed using weights so that their value is not estimated across the edge.

Is represented by the following Equation 25,

Is represented by the following equation (26).

는 문턱값으로써 G-R 영상의 표준편차인

을 사용한다. 보간될 위치에서 상기 수학식 24를 통해서 얻어진 수직, 수평방향의 값

,

을 효과적으로 선택하기 위해서 제안되는 알고리즘에서는 가중치합을 통해서 그 값을 구한다.In Equations 25 and 26, N is defined differently according to an area in the algorithm proposed as a mask. In Equation 27

Is the threshold, which is the standard deviation of the GR image.

Use Values in the vertical and horizontal directions obtained through Equation 24 at the position to be interpolated

,

In order to effectively select, the proposed algorithm finds its value through the weighted sum.

,

은 각각 수평, 수직 방향 신호의 가중치 값이며, 다음과 같은 특성을 가진다. 신호의 분산에는 반비례하며, 주위 유사 픽셀의 개수에는 비례한다. 또한 두 가중치의 합은 1이 되어야 한다.

,

는 하기의 수학식 29를 통하여 계산한다. In Equation 28

,

Are weight values of horizontal and vertical signals, respectively, and have the following characteristics. It is inversely proportional to the variance of the signal and proportional to the number of similar pixels around it. In addition, the sum of the two weights should be 1.

,

Is calculated through Equation 29 below.

는 각각 수평, 수직방향으로 보간된 G 신호의 분산, G-R 신호의 분산, 수평, 수직 방향으로 보간되었을 때 주위 유사픽셀의 개수이다. G-R 신호의 분산은 상기 수학식 26과 동일하다. G 신호의 분산은 Bayer로부터 입력받은 원신호 G도 포함하여 계산한다. 따라서 G 신호의 평균, 분산, 가중치는 하기의 수학식 30, 31, 32와 같다. In Equation 29

Are the variance of the G signal interpolated in the horizontal and vertical directions, the variance of the GR signal, and the number of surrounding similar pixels when interpolated in the horizontal and vertical directions, respectively. The variance of the GR signal is the same as in Equation 26 above. The variance of the G signal is calculated including the original signal G received from Bayer. Therefore, the mean, variance, and weight of the G signal are as shown in Equations 30, 31, and 32 below.

을 사용한다. 또한 상기 수학식 29에서 주위 유사 픽셀의 개수를 계산함에 있어서는 픽셀간의 L1-norm을 사용하여 그 유사도를 계산한다. 먼저 G 신호의 거리와 G-R 신호의 거리를 마스크 내부 픽셀에 대해서 하기의 수학식 33과 같이 계산한다. In Equation 32, T _g is a threshold and is a standard deviation of a G image.

Use In addition, in calculating the number of neighboring similar pixels in Equation 29, the similarity is calculated using L1-norm between the pixels. First, the distance of the G signal and the distance of the GR signal are calculated for Equation 33 below with respect to the pixel inside the mask.

Next, H is defined as the number of pixels when the distance is smaller than a specific threshold Th _g and Th _c based on the distance calculated as in Equation 34 below.

, Th_c는

을 사용한다. 상기의 과정을 통해서 R 위치에서의 G-R 값을 추정할 수 있다. 이렇게 구한 G-R 값을 사용하여하기의 수학식 35처럼 현재 R 위치에서의 G 값을 추정할 수 있다. In Equation 34, the threshold Th _g is

, Th _c is

Use Through the above process, the GR value at the R position can be estimated. Using the obtained GR value, the G value at the current R position can be estimated as shown in Equation 35 below.

In Equation 35, R is an R value at a position to be interpolated, and g is an G value to be interpolated.

The procedure for obtaining the G-B value at the B position can also be obtained using the same method as described above.

1) flat area

는 컬러 보간 방향에 따른 오류보다는 잡음에 의한 영향이 더 크다고 할 수 있다. 그러므로 평탄 영역에서는 국부 통계치

,

을 계산하기 위한 마스크를 정방형으로 사용함으로써 잡음제거에 더 효율적으로 사용할 수 있게 된다. Error signal in the flat area

It can be said that the effect of noise is greater than that of the color interpolation direction. Therefore, in the flat area, local statistics

,

By using a mask to calculate the square, it can be used more efficiently for noise reduction.

5 is an exemplary diagram of a mask used for G channel interpolation in a flat region of a pixel having an R value or a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention. Masks are available with masks larger than 5x5 size.

,

의 값을 계산할 때도 동일한 정방 마스크를 사용하게 된다. The value obtained through this has the value that noise is effectively removed from the flat area. The estimated GR values in the vertical and horizontal directions are finally obtained through the weighted sum. At this time

,

The same square mask is used to calculate the value of.

2) general edge area

는 컬러 보간 방향에 따른 오류가 잡음에 의한 영향보다 더 크다고 할 수 있다. 그러므로 에지 영역에서는 국부 통계치

,

을 계산하기 위한 마스크를 열십자 모양으로 사용함으로써 컬러 보간 방향에 따른 오류를 좀 더 효율적으로 제거하게 된다. 오류 신호에 대한 잡음에 의한 영향은 컬러 보간 방향과는 관계없이 나타나므로 국부 통계치 계산에 정방 마스크가 사용될 경우 잡음의 영향으로 보간 방향에 따른 오류가 효율적으로 제거 되지 못하며 또한 가중치 함수의 값이 잘못 계산될 가능성이 있다. Error signal in the edge area

It can be said that the error along the color interpolation direction is greater than the effect by noise. Therefore, in the edge area, local statistics

,

By using the mask to calculate the shape of the cross, it is more efficient to remove the error according to the color interpolation direction. Since the influence of noise on the error signal is independent of the color interpolation direction, when the square mask is used for the calculation of local statistics, the influence of the noise does not effectively eliminate the error along the interpolation direction, and the value of the weight function is calculated incorrectly. There is a possibility.

,

의 값을 계산할 때도 동일한 형태의 마스크를 사용하게 된다. 6 is an exemplary diagram of a mask used for G channel interpolation in a general edge region of a pixel having an R value or a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention. At this time

,

The same type of mask is used to calculate the value of.

3) pattern edge area

The area where the performance difference of the color interpolation algorithm is greatest is the pattern area. In this area, it is difficult to confirm the directivity by comparing the difference between the vertical and horizontal values. In areas where it is difficult to estimate the direction of these edges, a more effective method of estimating the direction is needed. In the pattern area, it is helpful to determine the direction at the estimated position by fully considering the statistics of the ambient values.

FIG. 7 is an exemplary diagram of a mask used for G channel interpolation in a pattern edge region of a pixel having an R value or a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention. The shape of the mask used in the pattern edge area is the same square mask as in the flat area.

[B-channel color interpolation of pixels with R values and R-channel color interpolation of pixels with B values]

FIG. 8 is an exemplary diagram illustrating a reference pixel used for interpolating B channel values in a pixel having an R value and interpolating an R channel value in a pixel having a B value in a region adaptive directional color interpolation method according to an embodiment of the present invention. . A case of C ₂₂ will be described with reference to FIG. 8 as follows. That is, since the G channel has already interpolated, it is assumed that all G values exist. Then there are G values in C ₂₂ and G values in A. In order to obtain an A value at C ₂₂ , the A value must be converted to a difference image channel as shown in FIG. 8 in a diagonal direction in which the A values exist. The difference image channel may be calculated using Equation 36 below.

Unlike the interpolation of the G channel in Equation 36, since the information of the fully interpolated G channel is available, the interpolation according to the region is not performed, and the weighted average reflecting the edge information is used instead. This is because most of the brightness information of the image is in the G channel, so if the interpolation of the G channel is perfect, the R and B channel can be almost completely interpolated by interpolating the R and B channels using the information of the G channel. to be.

At this time, the edge direction display function used for interpolation is shown in Equation 37 below.

을 사용한다. In Equation 37, T _g is a threshold and is a standard deviation of a G image.

Use

Interpolating the A channel with the weighted average in the surrounding four directions is as shown in Equation 38 below.

는 색 보간에 사용되는 차영상의 위치에서의 에지 방향표시 함수이다.In Equation 38

Is an edge direction indicating function at the position of the difference image used for color interpolation.

[R-channel and B-channel color interpolation in pixels with G values]

FIG. 9 is an exemplary diagram illustrating a reference pixel used for interpolating R and B channel values of a pixel having a G value in a region adaptive directional color interpolation method according to an embodiment of the present invention. In most conventional methods, interpolation is performed using only up, down, left and right position information depending on the position of G, or using four-way position information using information interpolated in the previous process. However, in the present invention, the size of the mask is extended as shown in FIGS. 9A and 9B to perform interpolation using six pieces of position information, thereby using more accurate information for interpolation and providing sufficient ambient information. Will be used. Looking at the case of G ₃₃ with reference to Figure 9 as follows. In case of interpolating A, six difference image values are used by extending the mask to 5x3 as shown in FIG. 9 (b), and in case of interpolating C, the mask is extended to 3x5 as shown in (a) of FIG. Difference image values are used. If the positions of A and C are reversed, the reverse is assumed, and the mask is interpolated to 3x5 for A and 5x3 for C. In this case, likewise, B interpolation at the R position and R interpolation at the B position, a weighted average of six values are used, and the edge direction display function used at this time is the same as in Equation 37. If we calculate the weighted average of the six differences

Equation 39 is the case of FIG. 9A, and Equation 40 is the case of FIG. 9B. Color interpolated images are obtained through the above-described processes.

10 is a flowchart illustrating a method of region adaptive directional color interpolation according to an embodiment of the present invention. 10A is a flowchart illustrating an operation of the region adaptive directional color interpolation method according to an embodiment of the present invention. Referring to FIG. 10A, in step 210, image information is input for each pixel through a Bayer CFA, and in step 220, an area is classified into a flat area, a general edge area, and a pattern edge area. Next, in step 230, the difference image value in the vertical and horizontal directions is calculated to generate the difference image channel, and in step 240, the DLLMMSE filter is used to minimize the error. In operation 250, region adaptive color interpolation is performed, and in operation 260, the interpolated image is output.

10B is a flowchart illustrating a detailed operation of the region adaptive color interpolation operation. Referring to (b) of FIG. 10, color interpolation generally performs G channel first, and then interpolates the R and B channels using the information of the interpolated G channel.

In FIG. 10B, steps 251 and 252 are processes of interpolating the G channel. First, in step 251, the difference image values in the vertical and horizontal directions are selected in the difference image channel to calculate the weighted sum, and in step 252, the color of the G channel is interpolated in the pixel having the R value or the B value. In step 253, the color of the B channel is interpolated in the pixel having the R value, and the color of the R channel is interpolated in the pixel having the B value. In step 254, the colors of all the pixels are interpolated by interpolating the colors of the R and B channels in the pixel having the G value.

11 is a block diagram of a color interpolation apparatus according to an embodiment of the present invention. 11A is a block diagram of a color interpolation apparatus according to an embodiment of the present invention. Referring to FIG. 11A, a color interpolation apparatus according to an exemplary embodiment of the present invention may include an area separator 110, a difference image channel forming unit 120, a weight calculator 130, and a color beam. It includes the liver 140.

The area divider 110 divides an area of a pixel into a flat area, a general edge area, and a pattern edge area. The operation of dividing the region is the same as that of FIG. 4 described above, and thus a detailed description thereof will be omitted.

The difference image channel forming unit 120 includes a difference image channel D _R obtained by subtracting _R from a G value or a difference image channel D obtained by subtracting B from a G value. Form _B ). Which of the difference image channels D _R and D _B is to be used depends on the color component of the pixel to be interpolated and the type of color component to be interpolated. For example, if you want to interpolate an R value, form a difference image by subtracting the G and R values from each of the surrounding pixels of the pixel to be interpolated; if you want to interpolate the B value, G from each of the peripheral pixels of the pixel to be interpolated A difference image is obtained by subtracting the value from the B value. In addition, if the G value is to be interpolated, a difference image is formed by subtracting the G value and the color value of the color component of the pixel to be interpolated from each of the peripheral pixels of the pixel to be interpolated.

The weight calculator 130 calculates a weight by selecting an appropriate mask according to the area of the pixel to be interpolated when interpolating the G channel value, and interpolates colors among pixels around the pixel to be interpolated when interpolating the R channel value and the B channel value. Calculate the weight by selecting the appropriate pixel used in.

The color interpolator 140 performs color interpolation on each pixel by using the difference image value formed by the difference image channel forming unit 110 and the weights of pixels used for color interpolation calculated by the weight calculator 130. do.

FIG. 11B is a block diagram illustrating a configuration of the color interpolation unit 140 in detail. Referring to FIG. 11B, the color interpolator 140 includes a first color interpolator 141, a second color interpolator 142, and a third color interpolator 143.

The first color interpolator 141 interpolates the G value in the pixels having no G value by using the difference image value and the weight of the peripheral pixel. The second color interpolator 142 interpolates an R value in a pixel having a B value and interpolates an R value in a pixel having a B value. The third color interpolator 143 interpolates the R and B values in the pixel having the G value.

In the area adaptive directional color interpolation method according to the present invention, the resolution is improved in the high frequency region compared to the conventional color interpolation method, and the color noise and the sensor noise are reduced in the flat region. Effectively removed. Specifically, the following experimental results can be confirmed in detail.

[Experiment result]

The experimental image was made of Bayer CFA, which is the most commonly used image for color interpolation, and was used as experimental data. 12, 13, 14, and 15 are exemplary views illustrating comparison between an image using a conventional color interpolation method and an image using a color interpolation method according to an embodiment of the present invention. 12 (a), 13 (a), and 14 (a) are interpolated images by a conventional color interpolation method. In (a) of FIG. 12, it is possible to check the occurrence of the moiré effect diagonally in the window, and in FIG. 13 (a), it is possible to check the occurrence of color errors in the waves and color interpolation performance in a rope having a thin edge. have. In addition, in FIG. 14A, the fringe effect in the fence region where the edges appear repeatedly, the color error in the lighthouse, and the like can be confirmed. FIG. 15 (a) is an image of which the performance of the algorithm can be best confirmed, and is a partially enlarged image of the pattern region of FIG. 14 (a). In comparison with the conventional method, it can be seen that the wave pattern effect in the window shown in FIG. 12A does not occur in FIG. 12B, and FIG. 13A in FIG. 13B. Color errors in the waves seen in the) are reduced a lot, and you can also see that the ropes interpolate well without color errors. In addition, in FIG. 14B, it can be confirmed that the wave pattern effect and color error in the region where the edge shown in FIG. In order to check the performance in the pattern edge region, when comparing FIG. 15A and FIG. 15B, the interpolation method proposed by the present invention is used in the pattern edge region to interpolate without a moiré effect and color error. It can be confirmed that. 12, 13, 14, and 15, the interpolation method proposed in the present invention is well preserved considering the edge well, and color is effectively generated without almost causing color error and moiré effect that are problematic in color interpolation. You can check that you have interpolated.

As described above, the operation of the region-adaptive directional color interpolation method according to an embodiment of the present invention can be performed. Meanwhile, in the above description of the present invention, specific embodiments have been described, but various modifications do not depart from the scope of the present invention. It can be done without. Therefore, the scope of the present invention should not be defined by the described embodiments, but by the claims and equivalents of the claims.

The region-adaptive directional color interpolation method according to the present invention uses a region-adaptive algorithm and effectively considers the directionality of edges to minimize the occurrence of errors that can occur during color interpolation, thereby providing a higher resolution interpolation image with higher resolution than the conventional method. You can get it.

Claims (18)

In the region adaptive directional color interpolation method in a digital photographing apparatus using an image sensor, Dividing each pixel into a flat area, a general edge area, and a pattern edge area on a color filter array (CFA) to perform color interpolation adaptive to the area; After generating the difference image channel in the vertical and horizontal directions to estimate the difference image value to minimize the error, calculate the weighted sum of the difference image values in the vertical and horizontal directions to interpolate, and use the weighted sum to obtain the G (Green) the process of interpolating channel values, An edge direction indication function is obtained using the interpolated G channel value, and the edge direction indication function is weighted to a difference image value of a reference pixel around the pixel to be interpolated to have an R (Red) value using a weighted average. Interpolating the B (Blue) channel value at the pixel, and interpolating the R channel value at the pixel having the B value, An edge direction indication function is obtained using the interpolated G channel value, and the edge direction indication function is weighted to a difference image value of a reference pixel around the pixel to be interpolated, and a weighted average is used to R at a pixel having a G value. Color interpolation method comprising the step of interpolating the channel and the B channel. The process of claim 1, wherein each pixel is divided into a flat area, a general edge area, and a pattern edge area. When the difference between the result value of the reference pixel interpolated in the horizontal direction and the result value interpolated in the vertical direction is smaller than the preset flat threshold value, the flat area is set. If the difference between the horizontal interpolation result of the reference pixel and the vertical interpolation result is greater than or equal to the predetermined flat threshold value, the horizontal color difference value of the reference pixel and the vertical color difference value If the difference is greater than the preset edge threshold by checking the difference between the set to the normal edge area, And when the difference between the color difference value in the horizontal direction and the color difference value in the vertical direction is smaller than the preset edge threshold value, setting the pattern edge region. The color interpolation method according to claim 2, wherein the color difference value in the vertical direction and the color difference value in the horizontal direction are calculated using the following equations (41) and (42).

In Equations 41 and 42, Diff _H is a color difference value in a horizontal direction, Diff _V is a color difference value in a vertical direction, and R is an R channel value of a reference pixel, where a pixel position is (3,3). Suppose that R1, R2, R3, and R4 are R channel values at positions (3,1), (1,3), (3,5), and (5,3), and the G1, G2, G3, G4 is the G channel value at (3,2), (2,3), (3,4) and (4,3). 3. The method of claim 2, wherein the predetermined flat threshold that determines the flat region has a value between 1 and 15. The color interpolation method according to claim 1, wherein the vertical and horizontal difference image channels are generated using the following equations (43) and (44).

In Equations 43 and 44,

Is the difference image value in the vertical direction,

Is the difference image value in the horizontal direction, i and j are pixel coordinates, G (i, j) is a G channel value of (i, j) coordinates, and R (i, j) is a (i, j) coordinate. R channel value of. The method of claim 1, wherein estimating a difference image value that minimizes an error is performed by applying a Local Linear Minimum Mean Square Error (LLMMSE) filter to minimize errors caused by horizontal and vertical interpolation and noise of a sensor. Color interpolation method characterized in that the form as shown in equation (45).

In Equation 45,

Is the difference image value estimated using the LLMMSE filter,

Above

Is the average of

Above

Is the dispersion of

Is the variance of the error. The method of claim 6, wherein

The average and the variance of the color interpolation method using a weight so that the value is not estimated across the edge as in Equations 46, 47 and the weight is calculated as in Equation 48.

In Equation 46, 47, 48

Above

Is the average of

Above

Where w is a weight, and in Equations 46 and 47, N is a mask applied according to the area of the corresponding pixel.

Is the threshold, which is the standard deviation of the GR image.

Use The color interpolation method according to claim 1, wherein the weighted sum of the difference image values in the vertical and horizontal directions at the position to be interpolated is calculated using Equation 49 below.

In Equation 49,

Is a weighted sum difference image value,

Is the difference image value in the horizontal direction,

Is the difference image value in the vertical direction,

Is the weight value of the horizontal signal,

Is the weight value of the vertical signal. The color interpolation method of claim 8, wherein the weight values of the horizontal and vertical signals are represented by Equations 50 and 51. 10.

In Equation 50, 51

Is the variance of the interpolated G signal in the horizontal direction,

Is the variance of the interpolated G signal in the vertical direction,

Is the variance of the GR signal interpolated in the horizontal direction,

Is the variance of the interpolated GR signal in the vertical direction,

Is the number of surrounding similar pixels when interpolated in the horizontal direction,

The number of similar pixels around when interpolated in the vertical direction. 10. The color interpolation method of claim 9, wherein the mean, variance, and weight of the G signal interpolated in the horizontal direction are as shown in Equations 52, 53, and 54 below.

In Equations 52, 53, and 54, w is a weight, and in Equations 52 and 53, N is a mask applied according to a region of a corresponding pixel, and in Equation 54, T _g is a threshold value. Standard deviation

Use 10. The method of claim 9, wherein the similarity is calculated by using L1-norm between the pixels when calculating the number of similar pixels around the pixel. If the calculated distance is smaller than the predetermined threshold value, the color interpolation method, characterized in that the number of pixels defined as H as shown in Equation 56 below.

In Equation 56, the threshold Th _g is

, Th _c is

Using. The method of claim 1, wherein the G channel value is interpolated using Equation 57 below.

In Equation 57, R is an R channel value at a position to be interpolated, and g is a G channel value to be interpolated. The method of claim 1, wherein the interpolation of the G-channel is performed by using a square mask having a size of 5x5 or more in the flat region and the pattern edge region, and a cross-shaped cross in the vertical and horizontal directions in the general edge region. Color interpolation method characterized in that to perform interpolation using a mask. The method of claim 1, wherein in the process of interpolating a B channel value in a pixel having an R value and interpolating an R channel value in a pixel having a B value, four peripheral pixels of a diagonal of a pixel to be interpolated are referred to for color interpolation. Color interpolation method characterized by setting in pixels. The method of claim 1, wherein in the process of interpolating the R channel and the B channel in a pixel having a G value, six peripheral pixels included in an area having a size of either 3x5 or 5x3 are colored based on the pixel to be interpolated. Color interpolation method, characterized in that it is set to the pixel referred to the interpolation. 2. The method of claim 1, wherein the channels of the difference image are generated to obtain a weight using a fully interpolated G channel value, and the B channel value is interpolated at a pixel having an R value using the weight and the R channel at a pixel having a B value. The process of interpolating the values is calculated using Equation 58 below.

In Equation 58, D is a difference image value of a corresponding pixel located at each coordinate,

Is a weighting value applied to each difference image value as an edge direction display function at the position of the difference image used for color interpolation. The process of claim 1, wherein the channels of the difference image are generated to obtain weights using the fully interpolated G channel values, and the process of interpolating the R channel values and the B channel values from the pixels having the G values using the weights is 5 × 3 size. The color interpolation method is calculated using Equation 59 below when using the mask, and is calculated by Equation 60 below when using a mask having a 3x5 size.

In Equations 59 and 60, a ₃₃ is an A channel value to be interpolated, and its coordinate is assumed to be (3, 3). D is a difference image value of the corresponding pixel located at each coordinate, and w is a weight applied to each difference image value as an edge direction indicating function of the corresponding pixel located at each coordinate. 18. The color interpolation method according to claim 16 or 17, wherein the edge direction display function at the position of the difference image used for the color interpolation is calculated through Equation 61 below.

In Equation 61, w is an edge direction display function, and T _g is a threshold value, which is a standard deviation of a G image.

Using.

Images