KR102051367B1 - Processing apparatus for wrgb cfa image and processing method therefor - Google Patents

Abstract

The processing apparatus for the WRGB CFA image includes: a brightness compensator configured to compensate for brightness of an RGB channel; A correlation error compensator configured to compensate for a color correlation error of the W channel by using color correlation between the W channel and the RGB channel; And a G channel restoring unit for restoring a G channel at a W channel position by using the image compensated by the correlation error compensating unit, wherein the correlation error compensating unit comprises an image whose brightness is compensated by the brightness compensating unit. Alternatively, one of the images whose brightness is not compensated for may be used.

Description

PROCESSING APPARATUS FOR WRGB CFA IMAGE AND PROCESSING METHOD THEREFOR}

The present invention relates to a processing apparatus for an image obtained from a camera using a WRGB CFA (Color Filter Array) and a processing method thereof.

Today's typical digital cameras consist of a sensor that acquires brightness information, such as a CMOS or CCD, and a color filter array (CFA). In recent years, mobile camera sensors for mobile devices have continued to put more pixels in a small sensor area, and as the technologies are developed, the negative effect of each pixel is also reduced. Sensitivity attenuation is directly related to deterioration of image quality such as noise and loss of information in low light. In order to solve this problem, sensors using white-RGB (WRGB) CFAs have been manufactured instead of Bayer pattern RGB CFAs. The WRGB CFA is a W pixel with one G pixel having all the color bands passed, compared to the RGB Bayer CFA of a typical Bayer pattern with two G pixels and one R and B pixel in a 2 × 2 Unit Array. Replaced by As a result, the camera sensitivity is greatly improved, and especially in low light environment, the camera sensor using WRGB CFA is continuously developed.

One problem with using the WRGB CFA, however, is that W pixels have high sensitivity across all color bands, but do not actually have the color information needed to produce RGB images through demosaicing. Therefore, in order to demosaise the WRGB CFA image to obtain a complete RGB image, a technique of generating a Bayer pattern by estimating an accurate G channel at the W pixel position is required. Failure to estimate the correct G channel information can lead to serious image degradation issues such as color distortion, artifacts, and loss of information. In addition, since data of one pixel of a mosaic image obtained through CFA is generally up to 8 bits, there is a case where all data obtained with high sensitivity of the W channel is not contained and saturated.

 The present invention is a method for solving the technical problem as described above, by defining the correlation between the W channel and the RGB channel to compensate for errors such as saturation and noise of the W channel, robust to the edge region and accurate G channel recovery It is an object of the present invention to provide a processing apparatus for the WRGB CFA image and a processing method thereof.

In addition, the present invention provides a processing apparatus for a WRGB CFA image, and a method of processing the same, by which a brighter CFA image can be obtained by using the W channel, which can be viewed as a brightness component in which the bands of the RGB channels are combined, as brightness information. There is also that purpose.

The processing apparatus for the WRGB CFA image of the present invention includes: a brightness compensator for compensating brightness of an RGB channel; A correlation error compensator configured to compensate for a color correlation error of the W channel by using color correlation between the W channel and the RGB channel; And a G channel restoring unit for restoring a G channel at a W channel position by using the image compensated by the correlation error compensator. Preferably, the correlation error compensator uses one of an image whose brightness is compensated by the brightness compensator or an image whose brightness is not compensated.

In detail, the brightness compensator may include: a first regression analyzer configured to perform regression analysis using the input WRGB CFA image and the processed RGB image; And an inference device for compensating the brightness of the RGB channel using the analysis result by the first regression analyzer.

In addition, the image processing by the first regression analyzer compensates for the color correlation error of the W channel by using the color correlation between the W channel and the RGB channel, and uses the image compensated for the color correlation error to determine the position of the W channel. Restoring the G channel, and performing demosaicing, compensating for the brightness of the R channel, the G channel, and the B channel.

The first regression analyzer may calculate a correlation coefficient by using patches of the input WRGB CFA image and brightness compensated R, G, and B channels, respectively. In addition, it is preferable that the calculation of the correlation coefficient by the said 1st regression analyzer considers the pixel orientation of the said patch.

Advantageously, the correlation error compensator comprises: an interpolator for interpolating each of an R channel, a G channel, and a B channel; A second regression analyzer for performing regression analysis using the output interpolated by the interpolator; A model estimator for estimating a model of the W channel by the interpolated R channel, the interpolated G channel, and the interpolated B channel, using the analysis result of the second regression analyzer; A first order correlation error compensator that calculates and compensates a first order correlation error value by subtracting a model of the W channel estimated by the model estimator from the W channel information before estimation; And a second correlation error compensator for compensating the W channel by calculating a second correlation error value using the output of the first correlation error compensator.

In addition, the second correlation error compensator may compare the inclination value obtained by adding the absolute value of the difference value of the surrounding W pixels with respect to the center W pixel to a predetermined setting value, or the inclination value exceeds the setting value, When the primary correlation error value is equal to or greater than the value of the W channel before estimation, the secondary correlation error value is calculated as '0'. The second correlation error compensator calculates the second correlation error value by performing median filtering on the first correlation error value when the slope value is equal to or less than the set value. do. In addition, when the value of the W channel before the estimation is a saturated value, the second correlation error compensator preferably calculates the second correlation error value to the same value as the first correlation error value.

The G channel reconstructing unit may include: an edge information extractor extracting edge information around a corresponding W pixel; A G channel estimator for estimating a G channel in a W channel using an output of the edge information extractor; A W channel estimator that estimates the W channel by using the output of the G channel estimator to estimate the W pixel at the G pixel corresponding to the edge extracted by the edge information extractor around the W pixel; And a G channel reconstructor for reconstructing a G channel at a W channel position by using the outputs of the G channel estimator and the W channel estimator. In addition, the G channel reconstructor allocates a sum to a difference value between the W channel and the G channel estimated by the G channel estimator, and a difference value between the W channel and the G channel estimated by the W channel estimator, and adds the weighted sums. And subtracting the sum value from the W channel to output the restored G channel.

The processing method for the WRGB CFA image of the present invention includes the steps of: (a) compensating for brightness of an RGB channel; (b) compensating for the Color Correlation Error of the W channel using the color correlation between the W channel and the RGB channel; And (c) reconstructing the G channel of the W channel position by using the image compensated by the step (b). In addition, the step (b) is characterized in that using one of the image whose brightness is compensated by the step (a) or the image is not compensated for brightness.

In addition, the step (a), (a-1) performing a regression analysis using the input WRGB CFA image and the processed RGB image; And (a-2) compensating for the brightness of the RGB channel using the analysis result of step (a-1).

In addition, the image processing according to the step (a-1), by using the color correlation between the W channel and the RGB channel to compensate for the color correlation error (Color Correlation Error) of the W channel, to compensate for the image Reconstructing the G channel at the W channel position, performing demosaicing, and compensating for the brightness of the R channel, the G channel, and the B channel. In the step (a-1), the correlation coefficient may be calculated by using the patch of the input WRGB CFA image and the luminance-compensated R, G, and B channels, respectively. Preferably, the calculation of the correlation coefficient according to the step (a-1) is characterized in that it takes into account the pixel orientation of the patch.

In addition, the step (b), (b-1) interpolating each of the R channel, G channel and B channel (Interpolation); (b-2) performing a regression analysis using the output interpolated by step (b-1); (b-3) estimating a model of the interpolated R channel, the interpolated G channel, and the W channel by the interpolated B channel using the analysis result of step (b-2); (b-4) calculating and compensating a first order correlation error value by subtracting the model of the W channel estimated by step (b-3) from the W channel information before estimation; And (b-5) compensating for the W channel by calculating a second order correlation error value using the output of step (b-4).

In addition, in the step (b-5), the inclination value of the sum of the absolute values of the difference values of the peripheral W pixels on the basis of the center W pixel is compared with a predetermined setting value, or the inclination value exceeds the setting value. When the primary correlation error value is equal to or greater than the value of the W channel before estimation, the secondary correlation error value is calculated as '0'. In the step (b-5), when the slope value is less than or equal to the set value, median filtering is performed on the first correlation error value to calculate the second correlation error value. Preferably, in the step (b-5), when the value of the W channel before the estimation is a saturated value, the second correlation error value is calculated using the same value as the first correlation error value.

In addition, step (c) may include (c-1) extracting edge information around the corresponding W pixel; (c-2) estimating the G channel in the W channel using the output of the step (c-1); (c-3) Using the output of step (c-2), the W pixel at the G pixel corresponding to the edge extracted by step (c-1) around the corresponding W pixel is estimated, and W Estimating a channel; And (c-4) restoring the G channel of the W channel position by using the outputs of the steps (c-2) and (c-3).

Preferably, in step (c-4), the difference between the W channel and the G channel estimated in the step (c-2) and the difference between the W channel and the G channel estimated in the step (c-3). After the weights are assigned and added, the summed values are subtracted from the W channel to output the restored G channel.

According to the processing apparatus for the WRGB CFA image of the present invention and the processing method thereof, the correlation between the W channel and the RGB channel can be defined to compensate for errors such as saturation and noise of the W channel, and is robust and accurate to the edge region of the G channel. Restoration is possible.

In addition, according to the present invention, a brighter CFA image may be obtained by using the W channel, which is viewed as a brightness component in which all the bands of the RGB channels are combined, as the brightness information.

1A is a pattern diagram of an image of a typical WRGB CFA.
1B is an explanatory diagram of edge directionality.
2 is a block diagram of a processing apparatus for WRGB CFA image according to an embodiment of the present invention.
3 is a configuration diagram of a brightness compensator.
4 is an explanatory diagram for estimating a regression correlation coefficient matrix (c).
5 is a block diagram of a correlation error compensator according to an exemplary embodiment of the present invention.
6 is a block diagram of a G channel recovery unit according to an embodiment of the present invention.

Hereinafter, a processing apparatus for a WRGB CFA image and a processing method thereof according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings. The following examples of the present invention are intended to embody the present invention, but not to limit or limit the scope of the present invention. From the detailed description and examples of the present invention, those skilled in the art to which the present invention pertains can easily be interpreted as belonging to the scope of the present invention.

First, a processing apparatus 1000 for a WRGB CFA image of the present invention and a processing method thereof will be described below.

The present invention is the most commonly used RGB with raw data of W (White), R (Red), G (Green), B (Blue) channels obtained from a camera using WRGB CFA (Color Filter Array) In order to be applied to demosaicing of CFA, a Bayer image is generated by estimating and reconstructing the G of the W position, and the brightness of the corresponding image is improved by using the W channel. The present invention is divided into a process of obtaining a WRGB CFA image of improved brightness using a W channel, a process of compensating a correlation error between the W channel and the RGB channel, and a process of restoring G at the W position.

 First, the W, R, G, and B channels are interpolated to obtain an RGB image and a W image having the same size. YCbCr transforms the RGB image and replaces the Y component, which is the brightness component, with the interpolated W. The enhanced brightness RGB image is then regressed by the linear combination of the corresponding pixels in the original WRGB CFA image and its surrounding pixels. In this case, regression analysis is performed by dividing a patch for each edge direction, and the R, G, and B brightness of the WRGB CFA image is improved by using correlation coefficient matrices obtained therefrom.

Next, in the WRGB CFA image of the enhanced brightness, a correlation error that may exist between the W channel and the RGB channel is compensated for. Multiple linear regression analysis is used to define a linear model between the W and RGB channels and find the error between the W channel and the actual W generated by the linear model. The error is corrected using median filtering and thresholding, and the corrected error is subtracted into the W channel to compensate for the correlation error.

Finally, the Bayer image is obtained by restoring G at the W position. The reconstruction process obtains the primary G position G by interpolation using the G channel in the diagonal direction according to the edge direction information and Laplacesian filtering using the W channel. The model is estimated by W and weighted sum of the difference between W and G at each position is subtracted to W finally.

The main features of the present invention are as follows. In the present invention, regression correlation coefficient matrices are obtained to improve R, G and B channel brightness through W before demosaicing. Therefore, the brightness enhancement process through the W component does not affect the demosaicing technique and improves the brightness in such a way that the noise amplification is less than the simple gain change. The error in the W channel itself restored to the G channel is first compensated for. This reduces the error between the W and RGB channels, which can occur due to information loss due to saturation of the W channel, noise, and the spectrum of light entering the image sensor. In addition, the edge information and the value of the pixels around the W pixel are evenly used to perform a robust and more accurate G channel reconstruction in the edge area.

FIG. 1A is a pattern diagram of an image of a general WRGB CFA, and FIG. 1B is an explanatory diagram of edge orientation.

A processing technique for a WRGB CFA image according to the prior art will be described with reference to FIGS. 1A and 1B.

The basic technique of G channel reconstruction for WRGB CFA image demosaicing begins with the assumption that the W channel is equal to the simple sum of each RGB channel. Therefore, by calculating the average of each RGB channel around W within the arbitrarily defined range as shown in [Equation 1], multiplying the ratio of G channels in their sum by W to obtain the G channel value Gw at the W position. .

However, since W is equal to the simple sum of RGB and is often not applicable in practice, this approach causes inaccurate G channel value estimation. In addition, since the value of each RGB channel is obtained as the average of the pixels around W, the demosaicing result is blurred in the edge region. Other basic techniques based on filtering exist, but recovering inaccurate RGB values in the edge region is the same.

Conventionally, techniques for adaptively reconstructing the G channel in the edge region have been studied. The basic concept of this study is to extract the edge strength by [Equation 2], and to extract the edge direction by [Equation 3] to [Equation 5], through which the constant edge It only uses the information of the pixels in the direction.

In general, the sensitivity of the G channel among the RGB is high, and it is located in the diagonal direction of the W channel. Therefore, the diagonal direction of the diagonal direction is obtained through the edge strength, and the G channel and the W channel of the corresponding direction are used for reconstruction. However, this method is generally performed in a 5 × 5 array centered on W, and it is difficult to be precise because it uses the pixels in the edge direction extracted in the array without considering the weight.

 In addition, even if the weight is taken into consideration, the weight may be obtained by using a mixture of edge strengths of different channels. The problem with this approach is that it does not take into account the difference in sensitivity between each channel. For example, even if the same value is distributed around the corresponding pixel based on the entire RGB image, the edge intensity obtained from the W pixel on the mosaic pattern is different from the edge intensity obtained from the B pixel. This is because there are four G pixels around each W pixel and one R and B pixel, but there are four W pixels and one R and G pixel around the B pixel. Because it is different. Therefore, using this as a weight component instead of calculating it for each channel may cause errors, especially when the result is sensitive to sensor sensitivity such as low light.

Hereinafter, an operation of the processing apparatus 1000 for the WRGB CFA image according to an exemplary embodiment of the present invention will be described in detail.

The processing apparatus 1000 for the WRGB CFA image according to an exemplary embodiment of the present invention may be implemented by including a processor such as a digital signal processor (DSP), an MCU, an MPU, a CPU, and the like.

2 is a block diagram of a processing apparatus 1000 for a WRGB CFA image according to an exemplary embodiment of the present invention.

As can be seen from FIG. 2, the processing apparatus 1000 for the WRGB CFA image according to an exemplary embodiment of the present invention may include a brightness compensator 100, a correlation error compensator 200, and a G channel reconstructor ( 300).

The brightness compensator 100 compensates for the brightness of the RGB channel. That is, the brightness compensator 100 improves the R, G, and B pixel brightness of the WRGB CFA image using the W channel. In addition, the correlation error compensator 200 compensates for the color correlation error of the W channel by using the color correlation between the W channel and the RGB channel. In addition, the G channel reconstructor 300 restores the G channel at the W channel position by using the image compensated by the correlation error compensator 200.

The correlation error compensator 200 uses one of an image whose brightness is compensated by the brightness compensator 100 or an image whose brightness is not compensated. That is, the processing apparatus 1000 of the present invention compensates for the correlation error by compensating the brightness by the brightness compensator 100 when the illuminance is equal to or less than a predetermined value, and separate brightness compensation when the illuminance exceeds the predetermined value. Compensation error is compensated for.

That is, the brightness compensator 100 may have a high utilization value in the low light image and selectively omit the image in which brightness enhancement is not required, thereby performing the correlation error compensation and the G channel reconstruction step as the original CFA image. In the case of the reconstructed RGB Bayer pattern, a complete RGB image can be obtained by using a conventional demosaicing technique.

3 shows a configuration diagram of the brightness compensator 100.

As can be seen from FIG. 3, the brightness compensator 100 includes a first regression analyzer 110 and an inference device 120.

The first regression analyzer 110 performs a regression analysis using the input WRGB CFA image and the processed RGB image. In addition, the reasoning unit 120 outputs the WRGB image by compensating the brightness of the RGB channel using the analysis result of the first regression analyzer 110.

In the WRGB CFA, the W channel has high brightness because it accepts light energy for the wavelengths of all RGB channels. Therefore, the W channel may be regarded as having luminance information of the RGB image, and has a brighter value than the luminance estimated by the RGB channel information. Therefore, a brighter image can be obtained when the W channel is used as luminance information in a low light environment. In this method, the W, R, G, and B components are interpolated during the demosaicing process, and the brightness component Y is replaced with the interpolated W through YCbCr conversion of the RGB image. However, in this case, since the process of converting the brightness through W is performed later in the demosaicing, it is difficult to implement the WRGB CFA image when it is used in the demosaicing technique for the Bayer image. The proposed invention can compensate for the brightness of R, G, and B components by using W information on the WRGB CFA pattern so that no additional computation occurs in the demosaicing process. The process is divided into a regression analysis process by the first regression analyzer 110 and an inference process by the inference unit 120. The regression analysis process may include correlation error compensation by the first correlation error compensation module 111, G channel recovery by the G channel recovery module 112, demosaicing by the demosaicing module 113, and correlation coefficient estimation module. It is divided into the correlation coefficient estimation by 115. The correlation error compensation and the G channel reconstruction process will be described later, and the demosaicing process uses a conventional technique.

The regression analysis uses pairs of RGB color images reconstructed through the WRGB CFA image, demosaicing and brightness compensation. First, in the WRGB CFA, regression analysis is performed using R, G, and B information having a 3 × 3 patch and brightness corresponding to brightness corresponding to pixels corresponding to R, G, and B components.

4 is an explanatory diagram for estimating the regression correlation coefficient matrix c.

That is, the first regression analyzer 110 of the present invention estimates the regression correlation coefficient matrix c that satisfies Equation 6 using the WRGB CFA patch and the brightness compensated R channel as shown in FIG. 4.

In this case, the regression correlation coefficients are obtained according to the R, G, and B color channels, respectively, and the edge direction is used based on the WRGB pattern. Edge directionality is estimated in a total of nine modes in consideration of eight directions and cases where there is no direction in the vertical, horizontal and diagonal directions. The regression correlation coefficient matrix collects WRGB CFA patches corresponding to the same color channel and edge direction and estimates them globally. A total of 27 regression correlation coefficient matrices are obtained by considering the color channel (3) and the edge direction (9). .

For the above operation, the first regression analyzer 110, the first correlation error compensation module 111, G channel recovery module 112, demosaicing module 113, brightness compensation module 114 and correlation coefficient Estimation module 115 is included.

The image processing by the first regression analyzer 110 compensates for the color correlation error of the W channel by using the color correlation between the W channel and the RGB channel by the first correlation error compensation module 111. After reconstructing the G channel of the W channel position using the image whose color correlation error is compensated by the G channel reconstruction module 112, and performing demosaicing by the demosaicing module 113, the brightness compensation module ( 114) to compensate for the brightness of the R, G and B channels. In addition, the correlation coefficient estimation module 115 calculates a correlation coefficient by using patches of the input WRGB CFA image and brightness compensated R, G, and B channels, respectively. The calculation of the correlation coefficient by the correlation coefficient estimation module 115 preferably takes into account the pixel orientation of the patch.

More specific operations of the first correlation error compensation module 111 and the G channel recovery module 112 are similar to those of the correlation error compensation unit 200 and the G channel recovery unit 300.

In the inference process, the inference process 120 estimates the edge directionality of the input WRGB CFA using 8 pixels around the current pixel. In addition, the inference unit 120 calls the regression correlation coefficient matrix according to the color channel and the edge direction to perform weighted summation with the 3 × 3 patch centering on the current pixel to obtain the WRGB CFA with improved brightness of the R, G, and B pixels. Output

The signal coming through the sensor is typically 8-bit data, so any light that exceeds it will saturate to the highest 8-bit 255. The W channel has high sensitivity because all color bands have the intensity of light passed through them, but saturation occurs well in such a high illumination environment and the information in the area is lost. In addition, each of the RGB bands separately obtained and combined with the W channel causes an error in a specific band. Incorrect G channel recovery may be performed even in a region where correlation between W and RGB is degraded due to noise. Therefore, in order to solve this problem, the correlation error compensator 200 compensates the correlation error between W and RGB in the W channel.

5 is a block diagram of the correlation error compensator 200 according to an exemplary embodiment of the present invention.

As can be seen from Figure 5, the correlation error compensator 200 according to an embodiment of the present invention, the interpolator 210, the second regression analyzer 220, the model estimator 230, the first order correlation Error compensator 240 and second order correlation error compensator 250.

The interpolator 210 serves to interpolate each of the R channel, the G channel, and the B channel. The second regression analyzer 220 uses the output interpolated by the interpolator 210 to perform a regression analysis. The model estimator 230 estimates a model of the W channel by the interpolated R channel, the interpolated G channel, and the interpolated B channel, using the analysis result of the second regression analyzer 220.

The model of the W channel estimated by the model estimator 230 may be represented by Equation 7 below.

,

,

는 상관 계수이다. 이때의 Rw, Gw, Bw는 보간기(210)에 의해 RGB 각 채널을 바이큐빅(Bicubic) 등의 기법으로 보간하여 추정할 수 있다. 아울러, 제 2 회귀 분석기(220)는 상관 계수를 구하기 위해 [수학식 8]과 같이 LSM(Least Square Minimize)을 이용한 다중선형회귀분석(multi variable linear regression) 방식을 사용할 수 있다.Rw, Gw, and Bw are the RGB values of the W position, respectively.

,

,

Is the correlation coefficient. At this time, Rw, Gw, and Bw may be estimated by interpolating each RGB channel by a technique such as bicubic. In addition, the second regression analyzer 220 may use a multi-variable linear regression method using a Least Square Minimize (LSM) method as shown in Equation 8 to obtain a correlation coefficient.

,

,

를 이용하여 정의된 모델에 의해 생성된 W 채널을 W_E라 하면 이를 원래의 W 채널에 빼주는 것으로 1차 상관 오차 보상기(240)는, 1차상관 오차값을 구할 수 있다. 즉, 1차 상관 오차 보상기(240)는, 추정 전 W 채널 정보로부터 모델 추정기(230)에 의해 추정된 W 채널의 모델을 빼는 것에 의해 1차 상관 오차값을 산출하여 보상한다. From this

,

,

When the W channel generated by the model defined by using W _E is subtracted from the original W channel, the first order correlation error compensator 240 may obtain a first correlation error value. That is, the primary correlation error compensator 240 calculates and compensates the primary correlation error value by subtracting the model of the W channel estimated by the model estimator 230 from the W channel information before estimation.

However, since W _E is also not obtained using the exact RGB values, the correlation error thus obtained requires correction. W _E is obtained from the interpolated RGB value rather than the exact RGB value of the W position. It can be regarded as a low pass filtered signal, and due to the correlation error of subtracting W _E from the original W channel, Generated high frequency components are included. Therefore, this high frequency component can be regarded as interpolation error, and should be corrected. Therefore, in order to correct this interpolation error in the correlation error obtained by the primary correlation error compensator 240, the secondary correlation error compensator 250 performs median filtering and thresholding.

 The high frequency component may be divided into portions having an edge region and a high frequency pattern. First, the interpolation error of the region corresponding to the edge is generally sparse. Therefore, median filtering can reduce the error in the area. However, in the case where the W channel is saturated, the original W channel has no information, and only the correlation error has information. Therefore, performing median filtering on that region produces an output that blurs the reconstructed G channel. Accordingly, the secondary correlation error compensator 250 may solve this problem by selectively performing median filtering except for the saturation region.

 In addition, since the corresponding pattern is clearly shown in the portion having the high frequency pattern, the compensation method of filtering the correlation error from the median and subtracting it from W may adversely affect the pattern. Accordingly, the second order correlation error compensator 250 obtains a gradient value in the W pixel as shown in Equation 9 and does not compensate for the correlation error in the high frequency pattern region having a threshold value or more. Take it.

In addition, the secondary correlation error compensator 250 does not compensate even if the W channel becomes negative as a result of the compensation. In this process, the secondary correlation error compensator 250 subtracts the correlation error whose interpolation error is corrected from the W channel as shown in [Equation 10] and [Equation 11] to finally obtain a correlation error between the W channel and the RGB channel. Obtain the compensated W channel W '.

The operation of the second order correlation error compensator 250 described above is as follows.

The secondary correlation error compensator 250 serves to compensate for the W channel by calculating a secondary correlation error value using the output of the primary correlation error compensator 240. In addition, the second correlation error compensator 250 compares the inclination value obtained by adding the absolute value of the difference value of the surrounding W pixels with respect to the center W pixel to a predetermined setting value, or the inclination value exceeds the setting value, When the primary correlation error value is equal to or greater than the value of the W channel before the estimation, the secondary correlation error value is calculated as '0'. In addition, when the gradient value is less than or equal to the set value, the secondary correlation error compensator 250 performs median filtering on the primary correlation error value to calculate the secondary correlation error value. In addition, when the value of the W channel before the estimation is a saturated value, the second correlation error compensator 250 calculates the second correlation error value to the same value as the first correlation error value.

The G channel reconstructor 300 reconstructs the G channel through the W channel and the neighboring pixel values compensated for by the correlation error obtained by the correlation error compensator 200.

6 shows a configuration diagram of the G channel restoration unit 300 according to an embodiment of the present invention.

As can be seen from FIG. 6, the G channel reconstructor 300 according to an exemplary embodiment of the present invention includes an edge information extractor 310, a G channel estimator 320, a W channel estimator 330, and a G channel. Restorer 340.

The edge information extractor 310 extracts edge information around the corresponding W pixel. That is, the edge information extractor 310 obtains the edge strength and the edge direction by using Equations 2 to 5 of the conventional technique.

The G channel estimator 320 first estimates the G channel in the W channel using the output of the edge information extractor 310. In detail, the G channel estimator 320 linearly interpolates the G pixels around the edge direction, and the high frequency component lost due to the interpolation adds the amount of change between the center W pixel and the peripheral W pixels around the edge direction based on the Laplacian filtering. First estimate G at pixel location. The basic equation is the same as [Equation 12], and the following equations assume the edge direction is D1.

In this case, G _{E (i, j)} is a G channel value primarily estimated at the W pixel position. However, since the second term in [Equation 12] is the change between the values one pixel away in the diagonal direction, not the change with the surrounding W value, the large change amount is different from the actual value when W at the edge is an edge. Reflected, resulting in artifacts around the edges. Therefore, in order to solve this problem, the adaptive weight is applied using the inverse of the G position edge strength between the W values as shown in [Equation 13].

By the way, the second term of Equation (13) represents the amount of change in W. Since what we want to add is the amount of change in G, we need to normalize the term. This applies by multiplying the average of the surrounding G values by the ratio of W pixels in the form of coefficients. Therefore, the final G _{E (i, j)} can be obtained as shown in [Equation 14].

In addition, the W channel estimator 330 uses the output of the G channel estimator 320 to estimate the W pixels at the G pixels corresponding to the edges extracted by the edge information extractor around the corresponding W pixels, thereby providing a W channel. Estimate

That is, the W channel estimator 330 estimates the W value at the positions of the G pixels on both sides of the W pixel in the edge direction. In this case, W _{G, which} is an estimated W value at the G pixel position, is obtained from Equation 8 through the G value at the pixel position and the R and B values around the W pixel as shown in Equations 15 and 16. Obtain the coefficient by applying it.

The G channel reconstructor 340 reconstructs the G channel at the W channel position by using the outputs of the G channel estimator 320 and the W channel estimator 330. In addition, the G channel reconstructor 340 adds weights to the difference value between the W channel and the first estimated G channel, and the difference value between the estimated W channel and the G channel, and then adds a weighted value from the W channel. By subtracting, the restored G channel is output.

That is, the G channel reconstructor 340 is variable at the edge by weighting the difference between W and G _E and the difference between W _G and G according to the distance using the obtained values as shown in [Equation 17]. The G and G _w at the W channel position can be finally reconstructed by using the (adaptive) and color information around the W.

Hereinafter, a processing method for WRGB CFA image according to an exemplary embodiment of the present invention will be described.

Since the processing method for the WRGB CFA image according to the preferred embodiment of the present invention uses the above-described processing apparatus 1000 for the WRGB CFA image, the feature of the processing apparatus 1000 for the WRGB CFA image is not described. Of course, it includes all of them.

In addition, the processing method for the WRGB CFA image according to an embodiment of the present invention may be implemented by a computer program executed by a processor such as a digital signal processor (DSP), MCU, MPU, CPU, and the like.

The processing method for the WRGB CFA image according to the preferred embodiment of the present invention includes the steps of compensating the brightness of the RGB channel (S10); Compensating for a color correlation error of the W channel by using color correlation between the W channel and the RGB channel (S20); And reconstructing the G channel of the W channel position by using the image compensated by the step S20 (S30).

In addition, the step S20 is characterized by using one of the image whose brightness is compensated by the step S10 or the image whose brightness is not compensated.

In addition, step S10, the step of performing a regression analysis using the input WRGB CFA image and the image-processed RGB image (S11); And compensating for brightness of the RGB channel by using the analysis result of the step S11 (S12).

In addition, the image processing according to step S11, by using the color correlation between the W channel and the RGB channel, to compensate for the color correlation error (Color Correlation Error) of the W channel, using the image of the color correlation error is compensated for the W channel Restoring the G channel of the position and performing demosaicing, and compensating for the brightness of the R channel, the G channel, and the B channel. In addition, in the step S11, it is preferable to calculate the correlation coefficient by using the patch of the input WRGB CFA image and each of the R-channel, G-channel, and B-channel with brightness compensation.

In addition, calculation of the correlation coefficient by step S11 needs to consider the pixel orientation of a patch.

In detail, step S20 may include interpolating each of the R channel, the G channel, and the B channel (S21); Performing a regression analysis using the output interpolated by step S21 (S22); Estimating a model of the W channel by the interpolated R channel, the interpolated G channel, and the interpolated B channel, using the analysis result of step S22 (S23); Calculating and compensating a first order correlation error value by subtracting the model of the W channel estimated in step S23 from the W channel information before the estimation (S24); And compensating for the W channel by calculating a second order correlation error value using the output of step S24 (S25).

In step S25, the slope value, which is the sum of the absolute values of the difference values of the surrounding W pixels with respect to the center W pixel, is compared with a predetermined setting value, and the slope value exceeds the setting value, or the first correlation error value is before the estimation. When the value is greater than or equal to the value of the W channel, the second order correlation error value is calculated as '0'. In addition, in step S25, when the slope value is less than or equal to the set value, it is preferable to perform median filtering on the first order correlation error value to calculate the second order correlation error value. In operation S25, when the value of the W channel before the estimation is a saturated value, the second correlation error value may be calculated using the same value as the first correlation error value.

Step S30 may include extracting edge information around the corresponding W pixel (S31); Firstly estimating the G channel in the W channel using the output of step S31 (S32); Estimating the W channel at the G pixel corresponding to the edge extracted by the step S31 around the W pixel using the output of the step S32, and estimating the W channel (S33); And restoring the G channel of the W channel position using the output of the steps S32 and S33 (S34).

Specifically, in step S34, by adding weights to the difference between the W channel and the first estimated G channel and the estimated difference between the W channel and the G channel, the sum is subtracted from the W channel. And outputting the restored G channel.

The present invention proposes a technique of generating a Bayer image of a general RGB CFA by converting the W channel of the image obtained through the White-RGB CFA into a G channel value and improving the brightness using the W channel. According to the present invention, by compensating for the correlation error between the W channel and the RGB channel, problems such as information loss, color distortion, and artifacts due to saturation, which may occur due to the characteristics of the W channel, can be solved. The reconstruction technique, which utilizes both (adapted) interpolation and the color information of the surrounding pixels, can be used to reconstruct the exact G channel with significantly reduced artifacts and errors. In addition, according to the present invention, the CFA image RGB brightness is improved by using the high intensity of the W channel so that the noise is less than the method of amplifying the image brightness through the analog gain in the sensor in a manner that does not affect demosaicing. You can get a video.

1000: image processing device
100: brightness compensation unit 200: correlation error compensation unit
300: G channel recovery unit
110: first regression analyzer 120: reasoner
210: interpolator 220: second regression analyzer
230: model estimator 240: first order correlation error compensator
250: second order correlation compensator
310: edge information extractor 320: G channel estimator
330: W channel estimator 340: G channel decompressor
111: first correlation error compensation module 112: G channel recovery module
113: demosaicing module 114: brightness compensation module
115: correlation coefficient estimation module

Claims (24)

delete A processing apparatus for WRGB CFA imaging,
A brightness compensator for compensating brightness of the RGB channel;
A correlation error compensator configured to compensate for a color correlation error of the W channel by using color correlation between the W channel and the RGB channel; And
And a G channel restoring unit for restoring a G channel of a W channel position by using the image compensated by the correlation error compensator.
The correlation error compensator,
Using one of an image whose brightness is compensated by the brightness compensator or an image whose brightness is not compensated,
The brightness compensator,
A first regression analyzer performing regression analysis using the input WRGB CFA image and the processed RGB image; And
And an inference device for compensating for the brightness of the RGB channel by using the analysis result of the first regression analyzer. The method of claim 2,
Image processing by the first regression analyzer,
Compensate for the color correlation error of the W channel by using the color correlation between the W channel and the RGB channel, restore the G channel at the W channel position using an image compensated for the color correlation error, and perform demosaicing. And compensating for the brightness of the channel, the G channel and the B channel. The method of claim 3,
The first regression analyzer,
And calculating a correlation coefficient by using patches of the input WRGB CFA image and brightness compensated R, G, and B channels, respectively. The method of claim 4, wherein
The calculation of the correlation coefficient by the first regression analyzer,
And consider the pixel orientation of the patch. A processing apparatus for WRGB CFA imaging,
A brightness compensator for compensating brightness of the RGB channel;
A correlation error compensator configured to compensate for a color correlation error of the W channel by using color correlation between the W channel and the RGB channel; And
And a G channel restoring unit for restoring a G channel of a W channel position by using the image compensated by the correlation error compensator.
The correlation error compensator,
Using one of an image whose brightness is compensated by the brightness compensator or an image whose brightness is not compensated,
The correlation error compensator,
An interpolator for interpolating each of an R channel, a G channel, and a B channel;
A second regression analyzer for performing regression analysis using the output interpolated by the interpolator; And
And a model estimator for estimating a model of the W channel by the interpolated R channel, the interpolated G channel, and the interpolated B channel, using the analysis result of the second regression analyzer. The method of claim 6,
The correlation error compensator,
A first order correlation error compensator that calculates and compensates a first order correlation error value by subtracting a model of the W channel estimated by the model estimator from the W channel information before estimation; And
And a second correlation error compensator configured to compensate for the W channel by calculating a second correlation error value using the output of the first correlation error compensator. The method of claim 7, wherein
The secondary correlation error compensator,
The inclination value which adds the absolute value of the difference value of the surrounding W pixels with respect to the center W pixel is compared with a predetermined setting value, and the inclination value exceeds the setting value, or the first correlation error value is estimated before W And if the value is greater than or equal to the channel value, the second correlation error value is calculated as '0'. The method of claim 8,
The secondary correlation error compensator,
And when the slope value is less than or equal to the set value, performing median filtering on the first correlation error value to calculate the second correlation error value. The method of claim 8,
The secondary correlation error compensator,
And when the value of the W channel before the estimation is a saturated value, calculating the second correlation error value to the same value as the first correlation error value. A processing apparatus for WRGB CFA imaging,
A brightness compensator for compensating brightness of the RGB channel;
A correlation error compensator configured to compensate for a color correlation error of the W channel by using color correlation between the W channel and the RGB channel; And
And a G channel restoring unit for restoring a G channel of a W channel position by using the image compensated by the correlation error compensator.
The correlation error compensator,
Using one of an image whose brightness is compensated by the brightness compensator or an image whose brightness is not compensated,
The G channel recovery unit,
An edge information extractor for extracting edge information around the W pixel;
A G channel estimator using the output of the edge information extractor to estimate the G channel in the W channel;
A W channel estimator that estimates the W channel by using the output of the G channel estimator to estimate the W pixel at the G pixel corresponding to the edge extracted by the edge information extractor around the W pixel; And
And a G channel reconstructor for reconstructing a G channel at a W channel position using the outputs of the G channel estimator and the W channel estimator. The method of claim 11,
The G channel decompressor,
A sum of weights is added to the difference between the W channel and the G channel estimated by the G channel estimator and the difference between the W channel and the G channel estimated by the W channel estimator, and then subtracted from the W channel. Thereby outputting the restored G channel. delete In the processing method for WRGB CFA video,
(a) compensating for the brightness of the RGB channel;
(b) compensating for the Color Correlation Error of the W channel using the color correlation between the W channel and the RGB channel; And
(c) reconstructing a G channel of a W channel position by using the image compensated by the step (b);
In step (b),
Using either the image whose brightness is compensated by the step (a) or the image whose brightness is not compensated,
In step (a),
(a-1) performing regression analysis using the input WRGB CFA image and the processed RGB image; And
(a-2) compensating for the brightness of the RGB channel by using the analysis result of step (a-1). The method of claim 14,
The image processing according to the step (a-1),
Compensate for the color correlation error of the W channel by using the color correlation between the W channel and the RGB channel, restore the G channel at the W channel position by using the image compensated for the color correlation error, and demosaicing. And then compensating for the brightness of the R channel, the G channel, and the B channel. The method of claim 15,
Step (a-1),
And calculating a correlation coefficient by using patches of the input WRGB CFA image and brightness compensated R, G, and B channels, respectively. The method of claim 16,
The calculation of the correlation coefficient by the step (a-1),
Taking into account the pixel orientation of the patch. In the processing method for WRGB CFA video,
(a) compensating for the brightness of the RGB channel;
(b) compensating for the Color Correlation Error of the W channel using the color correlation between the W channel and the RGB channel; And
(c) reconstructing a G channel of a W channel position by using the image compensated by the step (b);
In step (b),
Using either the image whose brightness is compensated by the step (a) or the image whose brightness is not compensated,
In step (b),
(b-1) interpolating each of the R channel, the G channel, and the B channel;
(b-2) performing a regression analysis using the output interpolated by step (b-1); And
(b-3) estimating a model of the W channel by the interpolated R channel, the interpolated G channel, and the interpolated B channel, using the analysis result of step (b-2). How to. The method of claim 18,
In step (b),
(b-4) calculating and compensating a first order correlation error value by subtracting the model of the W channel estimated by step (b-3) from the W channel information before estimation; And
(b-5) using the output of step (b-4), calculating a second order correlation error value to compensate for the W channel. The method of claim 19,
Step (b-5) is,
A slope value obtained by adding the absolute value of the difference value of the surrounding W pixels with respect to the center W pixel is compared with a predetermined setting value, and the slope value exceeds the setting value, or the first correlation error value is estimated before W If the value is greater than or equal to the channel value, the second correlation error value is calculated as '0'. The method of claim 20,
Step (b-5) is,
And when the slope value is less than or equal to the set value, performing median filtering on the first correlation error value to calculate the second correlation error value. The method of claim 20,
Step (b-5) is,
And if the value of the W channel before the estimation is a saturated value, calculating the second correlation error value to the same value as the first correlation error value. In the processing method for WRGB CFA video,
(a) compensating for the brightness of the RGB channel;
(b) compensating for the Color Correlation Error of the W channel using the color correlation between the W channel and the RGB channel; And
(c) reconstructing a G channel of a W channel position by using the image compensated by the step (b);
In step (b),
Using either the image whose brightness is compensated by the step (a) or the image whose brightness is not compensated,
In step (c),
(c-1) extracting edge information around the corresponding W pixel;
(c-2) estimating the G channel in the W channel using the output of the step (c-1);
(c-3) Using the output of step (c-2), the W pixel at the G pixel corresponding to the edge extracted by step (c-1) around the corresponding W pixel is estimated, and W Estimating a channel; And
(c-4) restoring the G channel of the W channel position by using the outputs of the steps (c-2) and (c-3). The method of claim 23,
Step (c-4) is,
The difference between the W channel and the G channel estimated in the step (c-2) and the difference between the W channel and the G channel estimated in the step (c-3) are added to each other by weighting, and thereafter, from the W channel. Outputting the reconstructed G channel by subtracting the sum value.

Images