KR101803021B1 - Demosaicking method based on taylor series and adaptive weight selection - Google Patents

Abstract

A demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention includes the steps of: (a) interpolating missing green pixel values based on the weighted average of the green pixel values of four directions estimated using a second order approximation of the Taylor series; (b) refining the green pixel values; and (c) estimating missing red and blue pixel values at blue and red pixel positions and estimating missing red and blue pixel values at a green pixel position. In the second approximation of the Taylor series, a second derivative of the green pixel value is calculated based on the average of two second derivatives of the red and blue pixel values. Accordingly, the present invention can avoid wrong direction selection and achieve high efficiency.

Description

TECHNICAL FIELD [0001] The present invention relates to a demosaicing method based on Taylor series and adaptive weight selection,

The present invention relates to a demosaicing method based on Taylor series and adaptive weight selection.

In most digital cameras, images are acquired by a single image sensor whose surface is covered with a color filter array (CFA) to minimize the price and size of the camera, since the image sensor is the most expensive part of the camera [ One]. A single image sensor processes only one color channel at each pixel location. Therefore, the obtained image has a mosaic pattern. In order to recover three full color images from the mosaicked image, an interpolation method is needed to reconstruct the missing color values. This algorithm is called demosaicing or CFA interpolation [2].

In order to accurately recover the missing color channels, demosaicing methods using color differences between the green (G) channel and the red / blue (R / B) channels have been studied. Some methods first process the edge detection and then interpolate the missing colors along the detected edge direction, rather than across the detected edge. Adaptive color plane interpolation (ACPI) is one of the early demosaicking methods using edge-sensing interpolation [3]. The ACPI generates three predictors, and then selects one of them through an edge classifier consisting of a Laplace second order term for the R or B plane and a gradient term for the G plane. Effective color interpolation (ECI) uses color difference between G and R / B channels to interpolate missing values [4]. Using the concept of ECI, an extended ECI (EECI) using pixel differences as weights has been proposed [5]. Adaptive homogeneity-directed (AHD) methods use color homogeneity in CIELAB space [6] and adaptive filtering methods are applied to CFA demosaicing [7]. A demosaicking method (VCD) using color difference changes has been proposed to detect reliable edges. In this way, the missing G values fall into two categories: the plane region and the edge region. To determine whether the pixel is at the edge, a gradient change in the NxN window is calculated. Directional filtering and a posteriori decision method (DFPD) used the sum of the gradients in 5 × 5 windows in the horizontal and vertical directions. By combining DFPD and EECI, a color demosaicing (CDDFW) method with directional filtering and weighting has been proposed [10]. The high-order interpolation (HOI) method uses a Taylor series for interpolation and a weighted median filter to select predictive parameters [11]. Effective demosaicking based on subband correlation (EDUSC) proposed a discrete wavelet transform to classify edge pixels [12]. In an effective demosaicking method based on edge properties (EDAEP), the authors added fixed weights to the ECI method [13]. [14; 15], the authors proposed a demosaicing method using edge intensity filters and multi-scale gradients. In [16], the authors proposed a demosaicing method using geometric duality and extended direction distinction. Several methods have been used in the frequency domain of CFA images. The rule approach to demosaicing (RAD) uses frequency characteristics to evaluate luminance components and provides spatially adaptive regularization [17]. Dubois introduced a luminance-chrominance demultiplexing algorithm [18] and then used a least-squares strategy to design the optimal filter [19]. 20].

In [21], Kiku et al. Proposed residual interpolation as an alternative to chrominance interpolation. This method performs interpolation in the residual domain, the residuals being the difference between the observed pixel value and the experimentally estimated pixel value. The authors assumed that if image interpolation is performed in a domain with a smaller Laplace energy, its accuracy is improved. In reconstructing natural looking images, one key challenge is to model and respect the statistics of natural images. Chang et al. Proposed a modeling strategy-based demosaicing method [22]. This approach starts with joint modeling of color images, which supports simultaneous representation of inter-channel correlation and structural information in the image. The inter-channel correlation is then searched by measuring the channel difference signals in the gradient domain, while the structural information is searched by non-local low-order regularization.

The recently introduced demosaicing methods consider inter-channel correlation to locally select the best interpolation edge direction. These methods provide satisfactory results, except when the local geometry can not be estimated from neighboring pixels or when the channel correlation is low. [23], Duran and Buades proposed a demosaicing method that includes non-local image self-similarity to reduce interpolation defects when local geometry is ambiguous.

[24] Khashabi et al. Proposed a machine learning approach to demosaicing. The proposed method solves the demosaicing challenges by learning statistical models of images and noise from several hundred natural images. The authors showed that the machine learning approach has two advantages: (1) the model can be best trained in terms of PSNR or structural similarity, (2) the new CFA can be handled by holding the model .

[1] R. Lukac, K.N. Plataniotis, D. Hatzinakos, Color image zooming on the Bayer pattern, IEEE Trans. Circ. Syst. Video Technol. 15 (2005) 1475-1492. [2] B.K. Gunturk, J. Glotzbach, Y. Altunbask, R.W. Schafer, R.M. Mersereau, Demosaicking: color filter array interpolation, IEEE Signal Process. Mag. 22 (2005) 44-54. [3] J. F. Hamilton, J.E. Adams, Adaptive Color Plane Interpolation in Single Sensor Color Electronic Camera, U.S. Pat. Patent 5 629 734, 1997. [4] S.-C. Pei, I.-K. Full, Effective color interpolation in CCD color filter arrays using signal correlation, IEEE Trans. Circ. Syst. Video Technol. 13 (2003) 503-513. [5] L. Chang, Y.-P. Full, Effective use of spatial and spectral correlations for color filter array demosaicking, IEEE Trans. Consum. Electron. 50 (2004) 355-365. [6] K. Hirakawa, T.W. Parks, Adaptive homogeneity-directed demosaicing algorithm, IEEE Trans. Image Process. 14 (2005) 360-369. [7] N.X. Lian, L. Chang, Y.P. Tan, V. Zagorodnov, Adaptive filtering for color filter array demosaicking, IEEE Trans. Image Process. 16 (2007) 2515-2525. [8] K.-H. Chung, Y.-H. Chan, Color demosaicing using variance of color differences IEEE Trans. Image Process. 15 (2006) 2944-2955. [9] D. Menon, S. Andriani, G. Calvagno, Demosaicing with directional filtering and a posteriori decision, IEEE Trans. Image Process. 16 (2007) 132-141. [10] Z. Dengwen, X. Xiaoliu, D. Weiming, Color demosaicking with directional filtering and weighting, IET Image Process. 6 (2012) 1084-1092. [11] J.S.J. Li, S. Randhawa, Color filter array demosaicking using high-order interpolation techniques with a weighted median filter for sharp color edge preservation, IEEE Trans. Image Process. 18 (2009) 1946-1957. [12] C.-Y. Water, W.-C. Kao, Effective demosaicing using subband correlation, IEEE Trans. Consum. Electron. 55 (2009) 199-204. [13] W.-J. Chen, P.-Y. Chang, Effective demosaicking algorithm based on edge properties for color filter arrays, Digit. Signal Process. 22 (2012) 163-169. [14] I. Pekkucuksen, Y. Altunbasak, Edge strength filter based color filter array interpolation, IEEE Trans. Image Process. 21 (2012) 393-397. [15] I. Pekkucuksen, Y. Altunbasak, Multiscale gradients-based color filter array interpolation, IEEE Trans. Image Process. 22 (2013) 157-165. [16] J. Kim, G. Jeon, J. Jeong, Demosaicking using geometric duality and dilated directional differentiation, Opt. Commun. 324 (2014) 194-201. [17] D. Menon, G. Calvagno, Regularization approaches to demosaicking, IEEE Trans. Image Process. 18 (2009) 2209-2220. [18] E. Dubois, Frequency-domain methods for demosaicking of Bayer-sampled color images, IEEE Signal Process. Lett. 12 (2005) 847-850. [19] B. Leung, G. Jeon, E. Dubois, Least-squares luma-chroma demultiplexing algorithm for Bayer demosaicking, IEEE Trans. Image Process. 20 (2011) 1885-1894. [20] G. Jeon, E. Dubois, Demosaicking of noisy Bayer-sampled color images with least-squares luma-chroma demultiplexing and noise level estimation, IEEE Trans. Image Process. 22 (2013) 146-156. [21] D. Kiku, Y. Monno, M. Tanaka, M. Okutomi, Beyond color difference-residual interpolation for color image demosaicking, IEEE Trans. Image Process. 25 (2016) 1288-1300. [22] K. Chang, P. Ding, B. Li, Color image demosaicking using inter-channel correlation and nonlocal self-similarity, Signal Process: Image Commun. 39 (2015) 264-279. [23] J. Duran, A. Buades, Self-similarity and spectral correlation adaptive algorithm for color demosaicking, IEEE Trans. Image Process. 23 (2014) 4031-4040. [24] D. Khashabi, S. Nowozin, J. Jancsary, A.W. Fitzgibbon, Joint demosaicing and denoising via non-parametric random fields, IEEE Trans. Image Process. 23 (2014) 4968-4981. [25] Z. Sadeghipoor, Y.M. Lu, S. Susstrunk, Correlation-based joint acquisition and demosaicing of visible and near-infrared images, in: Proc. IEEE ICIP, September 2011, 2011, pp. 3165-3168. [26] L. Zhang, X. Wu, A. Buades, X. Li, Color demosaicking by local directional interpolation and non-local adaptive thresholding, J. Electron. Imag. 20 (2011) 023016. [27] S. Andriani, H. Brendel, T. Seybold, J. Goldstone, Beyond the Kodak image set: a new reference set of color image sequences, in: Proc. IEEE Int. Conf. Image Process. (ICIP), September 2013, 2013, pp. 2289-2293. [28] X. Zhang, B.A. Wandell, A spatial extension of CIELAB for digital color image reproduction, J. Soc. Inf. Display 5 (1997) 61-67. [29] L. Zhang, L. Zhang, X. Mou, and D. Zhang, FSIM: A Feature Similarity Index for Image Quality Assessment, IEEE Trans. Image Process. 8 (2011) 2378-2386.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and apparatus for approximating missing green pixel values using pixel values of neighboring pixels in four directions based on the Taylor series and instead of estimating the edge direction through limited candidate directions, Based on the Taylor series and the adaptive weight selection that can avoid false direction selections and achieve high efficiencies by estimating pixel values based on the Taylor series and the adaptive weight selection.

According to an aspect of the present invention, there is provided a demosaicing method based on Taylor series and adaptive weight selection,

(a) interpolating missing green pixel values based on a weighted average of four directional green pixel values estimated using a second approximation of the Taylor series;

(b) refining the green pixel values; And

(c) estimating missing red / blue pixel values at the blue / red pixel locations and estimating missing red / blue pixel values at the green pixel locations,

The second derivative of the green pixel value at the second order approximation of the Taylor series,

Is calculated based on the average of the two second derivatives of the red and blue pixel values.

In the demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention,

에 기반하여 계산되고,

, ≪ / RTI >

,

,

및

는 각각 북쪽(N), 남쪽(S), 동쪽(E), 및 서쪽(W) 방향으로 추정된 녹색(G) 픽셀값들이고,

는 적응적인 가중치이며,

,

,

및

는 상기 방향의 선택을 추정하기 위한 색차일 수 있다.In the above,

,

,

And

Are green (G) pixel values estimated in the north (N), south (S), east (E), and west (W)

Is an adaptive weight,

,

,

And

May be a color difference for estimating the selection of the direction.

,

,

및

는 각각,Also, in the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention,

,

,

And

Respectively,

,에 의해 계산되고,

, ≪ / RTI >

Where F may represent certain red (R) / blue (B) pixel values around the location (i, j).

Also, in the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention,

,

,

및

는 각각,remind

,

,

And

Respectively,

에 의해 계산되고,

Lt; / RTI >

,

,

, 및

는 각각 위치 (i,j)에서 북쪽, 남쪽, 서쪽 및 동쪽 방향의 녹색 픽셀값들이며,In the above,

,

,

, And

Are the green pixel values in the north, south, west and east directions at position (i, j), respectively,

이고,

ego,

이며,

Lt;

이고,

ego,

이며,

Lt;

이고,

ego,

이며,

Lt;

이고,

ego,

일 수 있다.

Lt; / RTI >

Also, in the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention,

The green pixel values in step (b)

에 기반하여 리파인(refine)되고,

Is refined,

,

,

, 및

는 각각 북쪽(N), 남쪽(S), 동쪽(E) 및 서쪽(W) 방향에서의 색차로서,In the above,

,

,

, And

Are color differences in the north (N), south (S), east (E) and west (W)

에 의해 계산되고,

Lt; / RTI >

,

,

, 및

는 재구성된 G 픽셀을 사용하는 색차로서,In the above,

,

,

, And

Is a color difference using a reconstructed G pixel,

에 의해 계산될 수 있다.

Lt; / RTI >

는,

에 의해 계산될 수 있다.Further, in a demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, the adaptive weight

Quot;

Lt; / RTI >

Also, in the demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, the missing red / blue pixel values at the blue / red pixel positions in step (c)

에 의해 계산되고,

Lt; / RTI >

,

,

,

및

,

,

,

는,

,

,

,

And

,

,

,

Quot;

에 의해 계산될 수 있다.

Lt; / RTI >

Also, in the demosaicking method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention, the missing red / blue pixel value at the green pixel position in the step (c)

에 의해 계산될 수 있다.

Lt; / RTI >

According to the demosaicking method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention, the missing green pixel values are calculated using the pixel values of the neighboring picked cells in four directions based on the Taylor series By approximating and estimating pixel values based on adaptive weights instead of estimating edge directions through limited candidate directions, it is possible to avoid wrong direction selection, achieve high efficiency with little computational burden, Can be improved.

Brief Description of the Drawings Figure 1 shows an exemplary 7 x 7 CFA Bayer pattern.
Fig. 2 shows pixel differences in a north direction in a specific red pixel; Fig.
3 shows an image color difference statistical model;
4 is a graph showing a candidate function for the four adaptive weighting, Figure 4a is ω ₁ (k), Figure 4b is ω ₂ (k), FIG. 4c is ω ₃ (k), FIG. 4d is ω ₄ (k). < / RTI >
FIG. 5A shows 25 images of a Zahra data set, FIG. 5B shows 18 images of a McM data set, and FIG. 5C shows 12 actual raw CFA images.
FIG. 6A shows a comparison of PSNR values for four candidate weighting functions; FIG. 6B shows the difference between? ₃ and the corresponding weighting functions; FIG.
7B is an AFD, FIG. 7C is a VCD, FIG. 7D is an HOI, and FIG. 7E is a view showing a part of an original McM # 5 image. 7F, 7G, 7D, and 7H illustrate images processed by the demosaicing method based on Taylor series and adaptive weight selection according to one embodiment of the present invention.
8B shows an AFD, FIG. 8C shows a VCD, FIG. 8D shows a HOI, FIG. 8E shows a view of a part of an original McM # 9 image, and FIG. 8F is an MSG, FIG. 8G is GD, and FIG. 8H is an image processed by a demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention.
9A is an original image, FIG. 9B is an AFD, FIG. 9C is a VCD, FIG. 9D is an HOI, FIG. 9E is an ESF, and FIG. 9F is a cross- MSG, Figure 9G, GD, and Figure 9H show images processed by TAD.
10 is a flow diagram of a demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The objectives, specific advantages and novel features of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

Prior to that, terms and words used in the present specification and claims should not be construed in a conventional and dictionary sense, and the inventor may properly define the concept of the term in order to best explain its invention Should be construed in accordance with the principles and the meanings and concepts consistent with the technical idea of the present invention.

It should be noted that, in the present specification, the reference numerals are added to the constituent elements of the drawings, and the same constituent elements are assigned the same number as much as possible even if they are displayed on different drawings.

Also, the terms "first", "second", "one side", "other side", etc. are used to distinguish one element from another, It is not.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the present invention, a detailed description of known arts which may unnecessarily obscure the gist of the present invention will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, an efficient demosaicing method based on adaptive direction selection using Taylor series and adaptive weights is proposed. The Taylor series is used to approximate the value to be interpolated using the values of neighboring pixels in four directions. Instead of estimating the edge direction through limited candidate directions, such as the conventional demosaicking methods, the demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention, We propose a demosaicing method based on adaptive direction and weight selection. The weights are used to evaluate the degree of similarity between neighboring pixels. Experimental results show that in both objective and subjective performance, the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention outperforms existing approaches.

The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention proposes an effective and adaptive demosaicing algorithm using adaptive direction estimation (TAD) employing Taylor series and weight selection do. The missing color values are estimated as a linearly weighted spatial average of neighboring color values estimated in the north, south, west and east directions using the Taylor series. In order to achieve high efficiency and avoid false directional selection, a demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention may be used to estimate the degree of similarity between the values to be interpolated and neighboring color values, Lt; RTI ID = 0.0 > a < / RTI > second derivative in the domain. There are two main contributions to the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention.

1. In the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, a missing green pixel is estimated as a weighted average value of four surrounding direction estimates. The four direction estimates use the Taylor series and the average of the two second derivatives in each direction is proposed to predict the second derivative in the Taylor series.

2. The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention uses an adaptive weight selection approach instead of the direction selection widely used in the conventional algorithm. Moreover, in the demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, a high weight is assigned to a small pixel difference and a small weight is assigned to a large pixel difference. Furthermore, in the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, four classical candidates are selected to obtain an optimal weight design.

A demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention introduces a large size local window for designing adaptive weights to evaluate fine details such as image detail and texture . In addition, the demosaicking method based on the Taylor series and adaptive weight selection according to an embodiment of the present invention may be used to refine estimates of color pixels for reconstructing major geometric structures and large- Lt; / RTI >

The invention In the embodiment  Taylor series and Adaptive  To select a weight Based  Demothic king method

Reconstruction of the green plane

Figure 1 shows a 7 x 7 Bayer pattern CFA. The G channel component retains more image detail in the Bayer CFA pattern. A good reconstruction quality of the G channel component may result in a good estimation of the R and B components. As can be seen in FIG. 1, any missing G component at the location of the R or B component is also located at a specific G component in the north (N), south (S), east (E), and west Surrounded by. Typically, assuming that the images gradually change in space and the same color values have strong correlation, the missing G values are weighted averages of the four directional G pixel values expressed as shown in equation (1).

,

,

및

는 각각 N, S, E, 및 W 방향으로 추정된 G 값들이고,

는 적응적인 가중치이다.

,

,

및

는 상기 방향의 선택을 추정하기 위한 색차로서, 이것들은 수학식 2 내지 수학식 5와 같이 2차 도함수에 의해 산출된다. 더 정확한 가중치들을 획득하기 위하여, 네 방향들인, 북쪽, 남쪽, 서쪽 및 동쪽에 있는 로컬 영역에서 픽셀 차이를 평가함으로써 이미지 상세가 추출된다. 도 2는 방향에 따른 픽셀 차이의 예를 도시한 것으로, 특정한 적색 픽셀에서 누락된 녹색 인텐시티가 추정될 수 있다. 도 2로부터, 11×11 로컬 윈도우에서 방향에 따른 픽셀 차이가 추정된다. 더욱이, 방향 정보를 유도하기 위하여 1차 도함수 대신에 2차 도함수가 사용되는데, 이것은 2차 도함수가 1차 도함수보다 이미지 상세에 더 민감하기 때문이다. 본 발명의 일 실시예에 의한 테일러 급수 및 적응적 가중치 선택에 기반한 디모자이킹 방법에서는 방향에 따른 픽셀 차이, 4개의 G 색차, 2개의 R 색차 및 2개의 B 색차를 평가하기 위하여 8개의 2차 도함수가 사용된다.In the above,

,

,

And

Are G values estimated in N, S, E, and W directions, respectively,

Is an adaptive weight.

,

,

And

Is a chrominance difference for estimating the selection of the direction, and these are calculated by the second derivative as shown in the equations (2) to (5). In order to obtain more accurate weights, image details are extracted by evaluating the pixel differences in the four directions, the local regions in the north, south, west and east. Fig. 2 shows an example of a pixel difference along a direction, in which a missing green intensity in a specific red pixel can be estimated. From Fig. 2, a pixel difference along the direction in an 11 x 11 local window is estimated. Moreover, a second order derivative is used instead of a first order derivative to derive the direction information because the second order derivative is more sensitive to image detail than the first order derivative. In the demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention, in order to evaluate the pixel difference along the direction, the four G chrominance, the two R chrominance, and the two B chrominance, Derivatives are used.

In the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, an adaptive weight is proposed to control the direction selection with this color difference.

Approximation with Taylor series

As shown in Equation (1), estimation of green pixel values in four directions is a key issue for reconstructing missing green pixel values. The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention employs a Taylor series to predict green pixel values in four directions. The general Taylor series is as follows.

The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention employs a quadratic approximation and the estimation of the green pixel values in the following four directions can be performed as follows:

,

,

, 및

는 각각 위치 (i,j)에서 북쪽, 남쪽, 서쪽 및 동쪽 방향의 녹색 픽셀값들이다.In the above,

,

,

, And

Are the green pixel values in the north, south, west and east directions at position (i, j), respectively.

The relationship of the three color differences is derived in Fig. In the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, it has been found that the three color differences have similar statistical properties in the local area. Therefore, it is assumed that the three color difference channels have a certain color difference in the local region.

To calculate the primary and secondary derivatives, it is assumed that the G pixel values and the R / B pixel values have a constant residual in the local region. Instead of using the unknown G pixel values, a demosaicking method based on the Taylor series and adaptive weight selection according to an embodiment of the present invention may be used to approximate the first derivatives and second derivatives of the G pixel values And uses specific R / B pixel values. For example, for the north direction, the first derivative is calculated as follows.

Where F denotes the specific R / B color pixel values around position (i, j).

The second derivative can render the image details more precisely. In order to estimate the local image feature, it is necessary to examine the feature in the local region in the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention. Thus, the second derivative of the G pixel value is calculated as the average of the two second derivatives of the R / B pixel values as follows.

Substituting Equations (8) and (9) into Equation (7), an approximate value of the north G pixel value can be obtained from Equation (10).

In a similar manner, the values of G _S , G _E and G _W can be obtained.

For example, for the south direction, the first order derivatives and the second order derivatives of the G pixel values are calculated as follows.

Further, for the west direction, the first derivative and the second derivative of the G pixel value are calculated as follows.

Further, for the east direction, the first derivative and the second derivative of the G pixel value are calculated as follows.

G plane Refinement (refinement)

When interpolating missing G color pixels, a large local window is used to estimate directional pixel differences to obtain adaptive weights. The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention uses an adaptive method instead of the directional estimation, so it is necessary to acquire more accurate direction information. Large local windows are known to be beneficial in evaluating image detail. On the other hand, in order to augment the large scale structure, the demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention also refines green pixels using a chrominance in a small local window. After restoring all the G pixel values, based on the assumption that the color values have a constant residual, the G plane is updated using Equation (11).

,

,

, 및

는 각각 N, S, E 및 W 방향에서의 색차를 나타내고, 하기와 같이 나타낼 수 있다.In the above,

,

,

, And

Represents the color difference in the N, S, E, and W directions, respectively, and can be expressed as follows.

,

,

, 및

는 하기와 같이 재구성된 G 픽셀을 사용하는 색차이다.

,

,

, And

Is a color difference using a reconstructed G pixel as follows.

Using equation (11), the G pixel value is updated.

Adaptive  Design of weights

It is assumed that the slow spatial variations fail at the edges, which are consequently blurred. To overcome this drawback, it is necessary to design an adaptive nonlinear weight that declines with the dissimilarity of pixel values. In general, large color differences should have large weights. Thus, it is possible to design an adaptive function which is a monotone decreasing function of the color difference. As described above, the adaptive function? (K) should have the following characteristics:? (0) = 1, and? (?) = 0. The demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention considers four functions as candidates for [omega] (k) and these are illustrated as follows:

A graph of these four functions is shown in FIG. The variable is color difference.

Reconstruction of the R / B plane

After reconstructing the G plane, specific R / B pixel values and reconstructed G pixel values are used to estimate the missing R / B plane. To recover the missing R / B values at a particular B / R position, Equation (18) is applied.

,

,

및

,

,

,

는 하기와 같이 계산된다.In the above, ,

,

,

And

,

,

,

Is calculated as follows.

For R / B pixels missing at a particular G pixel, the following values are calculated.

Since the R / B pixel values have strong correlation with the G pixel values, the G channel is first reconstructed and then the R / B channel is reconstructed based on the reconstructed G channel.

Finally, the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention is summarized as follows.

Referring to FIG. 12, in step S100, the missing green pixel values are interpolated based on the weighted average of the four-directional green pixel values estimated using the second approximation of the Taylor series. In this step, the missing G color channel is interpolated by Equation (1) and the entire G channel is reconstructed. In this step, the second derivative of the green pixel value in the second approximation of the Taylor series is calculated based on the average of the two second derivatives of the red and blue pixel values, as described in equation (9).

In step S102, the green channel is refined by updating the green pixels. The G channel estimated at this stage is refined by the equation (11).

In step S104, the missing red / blue pixel value at the blue / red pixel position is estimated, and the missing red / blue pixel value at the green pixel position is estimated. In this step, after the entire G channel is interpolated, the missing R / B values at a particular B / R position are estimated by equation (18), and the missing R / B pixels at a particular G pixel are Lt; / RTI >

The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention can be executed in the control unit of the image processing apparatus that processes the image received from the CFA.

Experiment result

The performance of the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention is evaluated and compared with several reference methods. In this simulation, three sets of images were used for the comparison test. The test sets include 25 Zahra data sets (768x512 size) [25], 18 McM data sets (500x500 size), and 12 actual raw CFA images (1824x1368 size) , All data sets are shown in Fig. The simulation is compared with the existing algorithms (AFD [7], VCD [8], HOI [11], ESF [14], MSG [15], and GD [ Was performed using standard test 500x500 images from the McM data set to demonstrate the superiority of the demosaicing method based on series and adaptive weight selection. Objective performance was evaluated in terms of PSNR, SSIM and CPU time in addition to subjective performance, a widely adopted criterion in the literature. The experiment was performed on the Intel Core 2 Duo CPU E8500 @ 3.16 GHz.

Weighting function selection

Figure 6 shows the CPSNR results for a comparison of four weighting functions using the 25 images (Zahra data set) shown in Figure 6a. FIG. 6B shows the CPSNR difference between? ₃ and the corresponding weighting functions. As can be seen, omega ₃ represents the best performance among the four weighting functions, and therefore, due to the best PSNR performance of omega ₃ , demosaicing based on the Taylor series and adaptive weighting selection according to an embodiment of the present invention In the method, ω ₃ is selected.

Objective performance analysis

The objective performance of the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention measured in terms of PSNR in each color plane (R, G and B) is presented. Table 1 provides the PSNR results compared to the different demosaicing methods. The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention, when compared to AFD, VCD, HOI, ESF, MSG and GD, respectively, from Table 1 shows that the average PSNR It is clear that it has the best PSNR with improvement. In the R and G channels, the demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention increases by about 1 dB with respect to the PSNR.

The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention also required a minimum of CPU time compared to all the existing methods as shown in Table 2. [ The demiginking method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention is 32.6%, 98.8%, 99.2%, 94.7%, 97.5% compared to AFD, VCD, HOI, ESF, MSG and GD, %, And 52.3%, respectively.

Subjective performance analysis

Since the reconstructed signal from the samples is different from the original continuous signal, the reconstructed images always have aliasing at the image edge. 5 and # 9 images, as shown in Figures 7 and 8, respectively, to illustrate the subjective performance evaluation of the resulting images in terms of visual effects. From FIG. 7, the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention has the best results in improved visual quality in terms of details of yellow and edge information compared to other methods . In Fig. 8, other methods distort the edge and cause very poor visual quality along the edge of the flower, in particular in some direction of the diagonal of 45 degrees and 135 degrees. However, the demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention shows improvement in visual quality as compared to other methods, as shown in FIG. 8h. From Figure 8, the demosaicking method based on Taylor series and adaptive weight selection according to an embodiment of the present invention results in improved visual quality with respect to the details of the edges in any direction compared to other methods.

메트릭[28]은 주관적인 성능 비교를 위한 양호한 평가자이다. S-CIELAB

척도는 S-CIELAB

의 공간 확장이고 원래의 이미지의 S-CIELAB

표현과 재구성된 이미지의 S-CIELAB

표현 간의 차를 결정하기 위하여 인간의 시가계를 고려하는 가장 효과적인 척도 중 하나이다. McM 데이터세트에 대한 S-CIELAB

메트릭의 관점에서, 본 발명의 일 실시예에 의한 테일러 급수 및 적응적 가중치 선택에 기반한 디모자이킹 방법은 표 3에 도시된 바와 같이 모든 다른 레퍼런스 이미지들보다 평균적으로 더 양호한 성능을 제공한다(0.2227, 0.3352, 0.1673, 0.4811, 0.2170, 0.2493). 본 발명의 일 실시예에 의한 테일러 급수 및 적응적 가중치 선택에 기반한 디모자이킹 방법의 성능을 강하게 입증하기 위하여, 세번째 메트릭인, FSIM 지수[29]가 사용된다. 1.0에 근접한 FSIM 결과는 더 양호한 이미지 품질을 나타낸다는 것이 주목된다. 표 4는 몇몇 디모자이킹 알고리즘들에 대한 McM 데이터세트에 대한 FSIM 결과를 도시한 것이다. 표 4로부터, 본 발명의 일 실시예에 의한 테일러 급수 및 적응적 가중치 선택에 기반한 디모자이킹 방법은 평균적으로 0.00058, 0.00081, 0.00041, 0.00122, 0.00056, 그리고 0.00058만큼 모든 다른 방법들을 능가하였음을 확인할 수 있다.Human visual system - based on S-CIELAB

Metrics [28] are good evaluators for subjective performance comparisons. S-CIELAB

The scale is S-CIELAB

And the original image of S-CIELAB

S-CIELAB of representation and reconstructed images

It is one of the most effective measures to consider human cigars to determine the difference between expressions. S-CIELAB for McM data sets

In terms of metrics, the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention provides better performance on average than all other reference images, as shown in Table 3 (0.2227 , 0.3352, 0.1673, 0.4811, 0.2170, 0.2493). In order to strongly demonstrate the performance of the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, the third metric, the FSIM index [29], is used. It is noted that FSIM results close to 1.0 exhibit better image quality. Table 4 shows the FSIM results for the McM data set for several demosaicing algorithms. From Table 4, it can be seen that the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention outperformed all other methods on average by 0.00058, 0.00081, 0.00041, 0.00122, 0.00056 and 0.00058 have.

In order to evaluate the performance of the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention, another visual measure, zipper effect ratio (ZER), is used. The result is in the range of 0 to 1, and the smallest ZER means the largest zipper effect. As can be seen in Table 5, the present invention shows the best ZER results except # 7.

Experiments were also performed on the twelve actual raw CFA images shown in Figure 5c. Table 6 shows the CPSNRs of these 12 original CFA images and the demosaicing method based on Taylor series and adaptive weight selection according to an embodiment of the present invention shows the best average CPSNR over other methods . Figure 9 shows a visual quality comparison of the # 1 actual raw CFA image.

conclusion

In the demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention, an Taylor series is used to estimate the four-directional color values, and an adaptive weight design is made to automatically select the edge direction And an adaptive demosaicing method has been proposed. It is possible to avoid erroneous prediction of the edge direction and reduce the computational burden, without estimating the direction using the candidate directions. The demosaicing method based on the Taylor series and the adaptive weight selection according to an embodiment of the present invention has superior performance when compared with the conventional methods in both objective performance and subjective performance.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It is clear that the present invention can be modified or improved.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

S100: Green pixel value interpolation step
S102: green channel refinement step
S104: Red / Blue pixel value estimation step

Claims (8)

(a) interpolating missing green pixel values based on a weighted average of four directional green pixel values estimated using a second approximation of the Taylor series;
(b) refining the green pixel values; And
(c) estimating missing red / blue pixel values at the blue / red pixel locations and estimating missing red / blue pixel values at the green pixel locations,
The second derivative of the green pixel value at the second order approximation of the Taylor series,
Is calculated based on an average of the two second derivatives of the red and blue pixel values,
The missing green pixel values,

, ≪ / RTI >
In the above,

,

,

And

Are green (G) pixel values estimated in the north (N), south (S), east (E), and west (W)

Is an adaptive weight,

,

,

And

Is a color difference for estimating a selection of the direction, the Taylor series and the adaptive weight selection. delete The method according to claim 1,
remind

,

,

And

Respectively,

, ≪ / RTI >
Where F is the specific red (R) / blue (B) pixel values around the location (i, j). The method of claim 3,
remind

,

,

And

Respectively,

Lt; / RTI >
In the above,

,

,

, And

Are the green pixel values in the north, south, west and east directions at position (i, j), respectively,

ego,

Lt;

ego,

Lt;

ego,

Lt;

ego,

A method of demosaicing based on the in, Taylor series and adaptive weight selection. The method of claim 4,
The green pixel values in step (b)

Is refined,
In the above,

,

,

, And

Are color differences in the north (N), south (S), east (E) and west (W)

Lt; / RTI >
In the above,

,

,

, And

Is a color difference using a reconstructed G pixel,

Gt; a < / RTI > demodulation method based on Taylor series and adaptive weight selection. The method of claim 5,
The adaptive weight

Quot;

Gt; a < / RTI > demodulation method based on Taylor series and adaptive weight selection. The method of claim 6,
The missing red / blue pixel values at the blue / red pixel locations in step (c)

Lt; / RTI >

,

,

,

And

,

,

,

Quot;

Gt; a < / RTI > demodulation method based on Taylor series and adaptive weight selection. The method of claim 7,
The missing red / blue pixel value at the green pixel position in step (c)

Gt; a < / RTI > demodulation method based on Taylor series and adaptive weight selection.

Images