KR102189923B1 - Method and apparatus for removing artifact in demosaicked image - Google Patents

Abstract

Generating a luminance image from the demosaiced image, generating a JND (Just Noticeable Difference) map from the luminance image, and generating a difference image between an image to which a low-pass filter (LPF) is applied to the luminance image and a luminance image , A method of removing artifacts from a demosaiced image comprising detecting a pixel in which an artifact has occurred based on a JND map and a difference image, and removing the artifact generated in the pixel through a clamping process is disclosed.

Description

Method and apparatus for removing artifacts from demosaiced image {METHOD AND APPARATUS FOR REMOVING ARTIFACT IN DEMOSAICKED IMAGE}

The present invention relates to a method and apparatus for removing artifacts from a demosaiced image, and more particularly, to a method and apparatus for detecting artifacts for each pixel in a demosaiced image and removing the detected artifacts.

With the development of technology, film cameras have been replaced by digital cameras, and this technology is installed in smart devices such as smartphones, and the use of this technology is increasing day by day. Here, most digital cameras digitize external light through CMOS (Complementary Metal-Oxide Semiconductor or CCD (Charge-Coupled Device)), and only digitize light intensity information during this process, so the color information is CFA (Color Filter Array). ) Is extracted through a separate means.

However, since this method can only extract one color information for each pixel, there is a disadvantage that only a Bayer image from which RGB values are disproportionately extracted can be obtained, rather than a complete Red-Green-Blue (RGB) value. do. Therefore, the digital camera needs a demosaicking process to restore an image having an appropriate RGB value from a Bayer image.

This demosaicing process generates demosaicing artifacts or noise in that the image is reconstructed using insufficient data. In order to obtain a high-quality image, a method to prevent it from occurring, and the edge direction effectively Research on a predictive interpolation method and a method of effectively removing artifacts as post-processing are ongoing.

An object of the present invention for solving the above problems is to provide a method for removing artifacts from a demosaiced image.

Another object of the present invention for solving the above problems is to provide an apparatus for removing artifacts from a demosaiced image.

The artifact removal method according to an embodiment of the present invention for achieving the above object includes: generating a luminance image from a demosaiced image, generating a JND (Just Noticeable Difference) map from the luminance image, and LPF on the luminance image. The steps of generating a difference image between the image to which the (Low-Pass Filter) is applied and the luminance image, detecting a pixel with artifacts based on the JND map and the difference image, and removing artifacts generated in the pixel through a clamping process Can include.

Here, generating the JND map from the luminance image includes generating luminance adaptation information for a masking effect of background luminance from the luminance image, generating contrast masking information including edge masking information and texture masking information from the luminance image. And generating a JND map based on the luminance adaptation information and contrast masking information.

Here, in the step of generating the JND map based on the luminance adaptation information and the contrast masking information, the JND map is generated from the luminance adaptation information and the contrast masking information through a non-linear addition model for masking (NAMM). It may include the step of generating.

Here, the step of generating contrast masking information including edge masking information and texture masking information from the luminance image includes generating a structure image and a texture image from a luminance image through a TV (Total Variation)-based decomposition method, a structure image, and It may include generating edge masking information and texture masking information based on a maximum luminance difference and a weight in the four directions of the texture image, and generating contrast masking information by summing the edge masking information and the texture masking information.

Here, the step of detecting a pixel in which an artifact has occurred based on the JND map and the difference image includes, for at least one pixel in the luminance image, the component value of the difference image corresponding to at least one pixel is JND If it is larger than the component value of the map, the step of detecting at least one pixel as a pixel in which an artifact has occurred.

Here, the step of removing the artifact generated in the pixel through the clamping process may include removing the artifact generated in the pixel using a plurality of reference pixels located within a preset range from the pixel.

Here, the preset range may be set differently according to the interpolated color component of the pixel.

Here, the step of removing artifacts generated in a pixel by using a plurality of reference pixels located within a preset range from the pixel includes the interpolated color component values of the pixel and the maximum color component values and the minimum color component values of the plurality of reference pixels. The comparison may include determining a final color component value of the pixel.

Here, the determining of the final color component value of the pixel includes determining the final color component value of the pixel as the maximum color component value when the interpolated color component value of the pixel is greater than the maximum color component value, and the interpolated color of the pixel When the component value is less than the minimum color component value, determining the final color component value of the pixel as the minimum color component value.

Here, the step of determining the final color component value of the pixel includes interpolating the final color component value of the pixel when the interpolated color component value of the pixel is less than or equal to the maximum color component value and greater than or equal to the minimum color component value. It may include determining the color component value as it is.

An artifact removal apparatus according to an embodiment of the present invention for achieving the above other object includes at least one processor and a memory in which at least one instruction executed through the at least one processor is stored, At least one command is executed to generate a luminance image from the demosaiced image, and is executed to generate a JND (Just Noticeable Difference) map from the luminance image, and an image and luminance in which a Low-Pass Filter (LPF) is applied to the luminance image. It may be executed to generate a difference image between images, to detect a pixel in which an artifact has occurred based on a JND map and a difference image, and to remove an artifact generated in the pixel through a clamping process.

Here, at least one command is executed to generate luminance adaptation information for a masking effect of background luminance from a luminance image, and is executed to generate contrast masking information including edge masking information and texture masking information from the luminance image, and It may be executed to generate a JND map based on the adaptation information and the contrast masking information.

Here, at least one command may be executed to generate a JND map from luminance adaptation information and contrast masking information through a non-linear addition model for masking (NAMM).

Here, at least one command is executed to generate a structure image and a texture image from a luminance image through a TV (Total Variation)-based decomposition method, and edge masking based on the maximum luminance difference and weight in four directions of the structure image and texture image. It may be executed to generate information and texture masking information, and may be executed to generate contrast masking information by summing edge masking information and texture masking information.

Here, the at least one command is at least one pixel when the component value of the difference image corresponding to at least one pixel is greater than the component value of the JND map corresponding to at least one pixel with respect to at least one pixel in the luminance image. May be executed to detect as a pixel in which an artifact has occurred.

Here, at least one command may be executed to remove artifacts generated in the pixel by using a plurality of reference pixels located within a preset range from the pixel.

Here, the preset range may be set differently according to the interpolated color component of the pixel.

Here, at least one command may be executed to determine a final color component value of the pixel by comparing the interpolated color component value of the pixel with the maximum and minimum color component values of the plurality of reference pixels.

Here, at least one command is executed to determine the final color component value of the pixel as the maximum color component value when the interpolated color component value of the pixel is greater than the maximum color component value, and the interpolated color component value of the pixel is the minimum color component value. If it is less than the component value, it can be executed to determine the final color component value of the pixel as the minimum color component value.

Here, at least one command is to determine the final color component value of the pixel as the interpolated color component value when the interpolated color component value of the pixel is less than or equal to the maximum color component value and greater than or equal to the minimum color component value. Can be implemented to

According to the present invention, artifacts from a demosaiced image can be detected and removed for each pixel.

According to the present invention, the image quality of an image can be improved by effectively removing artifacts from a demosaiced image.

1 is a conceptual diagram of a method for removing artifacts according to an embodiment of the present invention.
2 is a graph showing a discrimination threshold value for background luminance according to an embodiment of the present invention.
3 is a diagram illustrating a filter for calculating background luminance according to an embodiment of the present invention.
4A to 4D are diagrams showing an operator for calculating a maximum luminance difference in four directions according to an embodiment of the present invention.
5 is a diagram showing an image for each step according to an embodiment of the present invention.
6 is a diagram illustrating an image in which an artifact is detected according to an embodiment of the present invention.
7A and 7B are diagrams illustrating reference pixels for interpolation of R/B pixels according to an embodiment of the present invention.
8A and 8B are diagrams illustrating a reference pixel for interpolation of a G pixel according to an embodiment of the present invention.
9 is a block diagram of an apparatus for removing artifacts according to an embodiment of the present invention.
10 is a flowchart of a method for removing artifacts according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term "and/or" includes a combination of a plurality of related stated items or any of a plurality of related stated items.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. Should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate an overall understanding, the same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a method for removing artifacts according to an embodiment of the present invention.

Referring to FIG. 1, the artifact removal method according to an embodiment of the present invention can detect an artifact from a demosaiced image, and can remove the detected artifact by processing it using an adjacent reference pixel. In the description of the present invention, a method of detecting artifacts will be first described, and a method of removing the detected artifacts will be described later with reference to FIGS. 7 and 8.

According to an exemplary embodiment of the present invention, various processes may be performed on a luminance image of a demosaiced image in order to detect artifacts from a demosaiced image. In other words, since the method of detecting artifacts is performed based on a luminance image, a method of generating a luminance image from a demosaiced image may precede. Here, the luminance image L may be generated through luminance conversion as shown in Equation 1 in a demosaiced image composed of RGB components.

In Equation 1, L may denote a luminance component value, and R, G, and B may denote red, green, and blue component values, respectively.

Hereinafter, a method of detecting artifacts using a luminance image according to an embodiment of the present invention will be described.

According to an embodiment of the present invention, the artifact may be detected using JND (Just Noticeable Difference) information for the luminance image and the difference L _diff information between the image forcibly removing the artifact and the luminance image. Here, the JND information may mean information on a difference capable of recognizing a difference according to a human visual characteristic.

In other words, according to an embodiment of the present invention, it is possible to compare the JND value and the L _diff value for each pixel to determine whether an artifact that can be discriminated by human vision has occurred in the corresponding pixel, and in more detail, the L _diff value is A pixel larger than the JDN value can be determined as a pixel with artifacts.

Hereinafter, a method of generating JND information for a luminance image will be first described, and then a method of generating L _diff information will be described.

In an embodiment of the present invention, the JND information for a luminance image may be generated based on luminance adaptation (LA) information and contrast masking (CM) information for a luminance image. In other words, according to an embodiment of the present invention, JND information may be generated from LA information and CM information through a non-linear addition model for masking (NAMM) as shown in Equation 2.

는 NAMM에서 두 마스킹 요소가 겹치는 부분을 제거하기 위한 게인 감소 성분을 의미할 수 있으며, 0.3으로 설정될 수 있으나, 이에 한정되는 것은 아니다. In Equation 2

May mean a gain reduction component for removing a portion where the two masking elements overlap in the NAMM, and may be set to 0.3, but is not limited thereto.

In addition, when the image is viewed in the form of a matrix, i and j may refer to row and column numbers indicating the position of the corresponding pixel on the image, and these may be used with the same meaning in the description of the present invention.

2 is a graph showing a discrimination threshold value for background luminance according to an embodiment of the present invention.

Referring to FIG. 2, when measuring human color sensitivity, it can be seen that when the background luminance value is 127, the discrimination threshold is the lowest.

In an embodiment of the present invention, the LA information may mean information on a difference that can be distinguished by a human according to a background luminance value, that is, a masking effect of the background luminance in the human's perspective, and Equation 3 is shown in FIG. As represented by Equation 3, LA information for each pixel may be generated from a luminance image through Equation 3.

는 해당 픽셀에서 인접한 픽셀을 참조한 평균 휘도 값을 의미할 수 있으며, 이는 도 3과 함께 상세히 후술하겠다.In Equation 3

May mean an average luminance value referring to adjacent pixels in the corresponding pixel, which will be described in detail later with reference to FIG. 3.

3 is a diagram illustrating a filter for calculating background luminance according to an embodiment of the present invention.

를 산출할 수 있다.An exemplary embodiment of the present invention is an average luminance value based on a luminance image and a filter as shown in Equation 4

Can be calculated.

In Equation 4, B may mean a filter for calculating the background luminance of FIG. 3. In other words, when calculating an average luminance value for a pixel at a specific location, it may be calculated by applying the filter of FIG. 3 to each of the pixels within a certain distance from the pixel at a specific location.

In an embodiment of the present invention, the CM information may mean information including edge masking (EM) information and texture masking (TM) information.

In more detail, the CM information may be calculated based on edge masking information EM ^s for the structure image L ^s of the luminance image and texture masking information TM ^t for the texture image L ^t of the luminance image, and the structure image L ^s and The texture image L ^t may be generated from a luminance image through a TV (Total Variation)-based decomposition method, and the TV-based decomposition method may be optimized in a manner that satisfies Equation (5).

는 노름(norm)을 의미할 수 있고, s(L^s) 및 t(L^s, L^f)은 각각 수학식 6 및 수학식 7과 같이 산출될 수 있다.In Equation 5, L ^f is the luminance image L,

May mean a norm, and s(L ^s ) and t(L ^s , L ^f ) may be calculated as in Equations 6 and 7 respectively.

In the description of the present invention, the edge masking information EM ^s and the texture masking information TM ^t may be calculated as in Equation 8.

는 감소 성분으로 0.117로 설정될 수 있으나, 이에 한정되는 것은 아니다. SC^s 및 SC^t는 각각 구조 이미지 L^s 및 텍스처 이미지 L^t의 4방향에 대한 최대 휘도차를 의미할 수 있으며, 이는 도 4a 내지 도 4d와 함께 후술하겠다.In Equation 8, W _s and W _t may mean weights representing information contributed by EM ^s and TM ^t to CM, and may be set to 1 and 3, respectively, but are not limited thereto. Also,

May be set to 0.117 as a reduction component, but is not limited thereto. SC ^s and SC ^t may mean the maximum luminance difference in the four directions of the structure image L ^s and the texture image L ^t , respectively, which will be described later with reference to FIGS. 4A to 4D.

4A to 4D are diagrams showing an operator for calculating a maximum luminance difference in four directions according to an embodiment of the present invention.

In an embodiment of the present invention, SC ^s and SC ^t may mean the maximum luminance difference in the four directions of the structure image L ^s and the texture image L ^t , respectively, and may be calculated as shown in Equation 9.

In Equation 9, r may be s or t, and accordingly, SC ^s and SC ^t may be calculated. In addition, grad ^r,k may mean an amount of luminance change according to a direction, and may be calculated as in Equation 10.

In Equation 10, G ^k may mean an operator in four directions, and k may mean any one of 1, 2, 3, and 4 depending on the direction, and the four operators according to this are FIGS. 4A to 4D Can be the same as

According to an embodiment of the present invention, grad ^r,k in the direction showing the maximum change may be calculated as SC ^r by using the operators of FIGS. 4A to 4D.

In the description of the present invention, the CM information may be calculated by summing the edge masking information EM ^s and the texture masking information TM ^t respectively calculated according to the above-described process as shown in Equation 11.

Hereinafter, a method of generating L _diff information for a luminance image will be described.

In an embodiment of the present invention, the luminance difference image L _diff may be generated based on the blurred L _LPF by applying a low-pass filter (LPF) to the luminance image L and the luminance image L. Here, the luminance difference image L _LPF may mean information on an image forcibly removing demosaicing artifacts, and L _diff may mean information on the difference between an image forcibly removing demosaicing artifacts and a luminance image. have.

In other words, the luminance difference image L _diff may be calculated by taking an absolute value of the difference between the luminance image L and the blurred L _LPF as shown in Equation 12.

Further, in Equation 12, the L _LPF blurred by applying the low-pass toe filter to the luminance image L may be calculated as in Equation 13.

는 콘볼루션(convolution) 연산자를 의미할 수 있으며, h_G는 가우시안(Gaussian) LPF를 의미할 수 있고, 가우시안 LPF에는 수학식 14를 이용할 수 있다.In Equation 13

May denote a convolution operator, h _G may denote a Gaussian LPF, and Equation 14 may be used for a Gaussian LPF.

는 표준편차를 의미할 수 있으나, 이에 한정되지 않으며, 통상의 가우시안 필터를 이용하는 방법이 참조될 수 있다.In Equation 14, x and y may each mean a horizontal distance and a vertical distance from the center,

May mean a standard deviation, but is not limited thereto, and a method using a conventional Gaussian filter may be referred to.

In summary, according to an embodiment of the present invention, a pixel in which an L _diff value generated through the above-described process in a luminance image is greater than a JND value may be determined as a pixel in which an artifact occurs.

5 is a diagram showing an image for each step according to an embodiment of the present invention.

In FIG. 5, the upper left image may represent a luminance image generated from a demosaiced image, and the upper right image may represent a blurred image by applying LPF to the luminance image. In addition, the lower left image may represent a difference image between the blurred image and the luminance image, and the lower right image may represent JND information or a JND map for the luminance image.

6 is a diagram illustrating an image in which an artifact is detected according to an embodiment of the present invention.

Referring to FIG. 6, according to an embodiment of the present invention, an artifact may be detected for each pixel based on the lower left image L _diff and the lower right image JND information of FIG. 5, and a demosaicing artifact map may be generated as shown in FIG. 6. have. In other words, artifacts included in the demosaicing artifact map may be selected or detected as pixels having an L _diff value greater than a JND value.

Hereinafter, a method of removing detected artifacts according to an embodiment of the present invention will be described.

The artifact removal apparatus according to an embodiment of the present invention may remove a pixel in which the detected artifact has occurred by referring to an adjacent pixel. In other words, the apparatus for removing artifacts may determine a final color component for the interpolated color component of the pixel where the artifact has occurred through the clamping process based on the color component value of the reference pixel adjacent to the pixel where the artifact has occurred.

In more detail, the artifact removal device may detect a pixel in which an artifact has occurred from a demosaiced image, and a color component value interpolated through a demosaic process of the detected pixel and a maximum color component of adjacent reference pixels of the detected pixel. Artifacts can be removed by comparing the value and the minimum color component value to determine the final color component value of the detected pixel.

In other words, in an embodiment of the present invention, when the interpolated color component value of the detected pixel is greater than the maximum color component value of adjacent reference pixels, the final color component value of the detected pixel may be determined as the maximum color component value, If it is smaller than the minimum color component value of adjacent reference pixels, the final color component value of the detected pixel may be determined as the minimum color component value. Further, according to an embodiment of the present invention, when the interpolated color component value of the detected pixel is less than or equal to the maximum color component value of adjacent reference pixels, and is greater than or equal to the minimum color component value, the final color component value of the detected pixel Can be determined as it is as the interpolated color component value. This clamping process can be expressed as Equation 15.

In Equation 15, X'may mean the final color component value of the detected pixel, and X may mean the interpolated color component value of the detected pixel. Also, X _min may mean a minimum value among color components of adjacent reference pixels of the detected pixel, and X _max may mean a maximum value among color components of adjacent reference pixels of the detected pixel.

Here, the range of adjacent reference pixels may be set differently according to the interpolated color component of the pixel detecting the artifact, and a description thereof will be described later with reference to FIGS. 7 to 8.

7A and 7B are diagrams illustrating reference pixels for interpolation of R/B pixels according to an embodiment of the present invention.

Referring to FIG. 7A, when an interpolated color component of a pixel in which an artifact detected according to an embodiment of the present invention is R is R, pixels located in a range of 3x3 around R will be set as reference pixels for the clamping process. I can.

Referring to FIG. 7B, when the interpolated color component of a pixel in which an artifact detected according to an embodiment of the present invention is B is B, pixels located in a range of 3x3 around B will be set as reference pixels for the clamping process. I can.

8A and 8B are diagrams illustrating a reference pixel for interpolation of a G pixel according to an embodiment of the present invention.

According to an embodiment of the present invention, when the interpolated color component of a pixel in which the detected artifact has occurred is G, two cases may exist due to the Bayer pattern.

8A and 8B show reference pixels according to these two cases. Referring to this, when the interpolated color component of the pixel where the artifact detected according to the embodiment of the present invention is G is G, 5x5 around G Pixels of R and B components located in the range of may be set as reference pixels for the clamping process.

9 is a block diagram of an apparatus for removing artifacts according to an embodiment of the present invention.

Referring to FIG. 9, the apparatus 900 for removing artifacts according to an embodiment of the present invention may include at least one processor 910, a memory 920, and a storage device 930.

The processor 910 may execute a program command stored in the memory 920 and/or the storage device 930. The processor 910 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which the methods according to the present invention are performed. The memory 920 and the storage device 930 may be formed of a volatile storage medium and/or a nonvolatile storage medium. For example, the memory 920 may be composed of a read-only memory (ROM) and/or a random access memory (RAM).

The memory 920 may store at least one instruction executed through the processor 910. At least one command is a command that generates a luminance image from a demosaiced image, a command that creates a JND (Just Noticeable Difference) map from a luminance image, and the difference between the luminance image and the image to which the LPF (Low-Pass Filter) is applied A command for generating an image, a command for detecting a pixel in which an artifact has occurred based on a JND map and a difference image, and a command for removing an artifact generated in the pixel through a clamping process may be included.

Here, the command for generating the JND map from the luminance image is a command for generating luminance adaptation information for the masking effect of background luminance from the luminance image, and generating contrast masking information including edge masking information and texture masking information from the luminance image. It may include an instruction for generating a JND map based on the instruction and luminance adaptation information and contrast masking information.

Here, the command for generating the JND map based on the luminance adaptation information and the contrast masking information is a JND map from the luminance adaptation information and the contrast masking information through a non-linear addition model for masking (NAMM). It may contain commands to generate.

Here, the command for generating contrast masking information including edge masking information and texture masking information from the luminance image is a command for generating a structure image and a texture image from a luminance image through a TV (Total Variation)-based decomposition method, a structure image, and A command for generating edge masking information and texture masking information based on the maximum luminance difference and weight in the four directions of the texture image, and a command for generating contrast masking information by summing the edge masking information and the texture masking information may be included.

Here, the command for detecting a pixel in which an artifact has occurred based on the JND map and the difference image is JND in which the component value of the difference image corresponding to at least one pixel with respect to at least one pixel in the luminance image corresponds to at least one pixel. If it is larger than the component value of the map, a command for detecting at least one pixel as a pixel in which an artifact has occurred may be included.

Here, the command for removing the artifact generated in the pixel through the clamping process may include a command for removing the artifact generated in the pixel by using a plurality of reference pixels located within a preset range from the pixel.

Here, the preset range may be set differently according to the interpolated color component of the pixel.

Here, the command for removing artifacts generated in a pixel by using a plurality of reference pixels located within a preset range from the pixel is the interpolated color component value of the pixel, the maximum color component value and the minimum color component value of the plurality of reference pixels. It may include instructions to compare and determine the final color component value of the pixel.

Here, the instruction for determining the final color component value of the pixel includes an instruction for determining the final color component value of the pixel as the maximum color component value and the interpolated color of the pixel when the interpolated color component value of the pixel is greater than the maximum color component value. When the component value is less than the minimum color component value, an instruction for determining the final color component value of the pixel as the minimum color component value may be included.

Here, the command for determining the final color component value of the pixel is, when the interpolated color component value of the pixel is less than or equal to the maximum color component value and greater than or equal to the minimum color component value, the final color component value of the pixel is interpolated. It may include an instruction to determine the color component value as it is.

10 is a flowchart of a method for removing artifacts according to an embodiment of the present invention.

Referring to FIG. 10, the apparatus for removing artifacts according to an embodiment of the present invention may generate a luminance image from a demosaiced image as a post-processing of demosaicing (S1010). In other words, the artifact removal apparatus may generate a luminance image by performing luminance conversion on a demosaiced image of an RGB component.

The apparatus for removing artifacts according to an embodiment of the present invention may generate contrast masking information including edge masking information and texture masking information from a luminance image (S1020). Here, the edge masking information may be generated by applying an edge masking predictor to the structure image, and the texture masking information may be generated by applying a texture masking predictor to the texture image. In addition, a structure image and a texture image may be generated from a luminance image through a TV (Total Variation) based decomposition method. In addition, the contrast masking (CM) information may be generated by setting a weight on the edge masking (EM) information and the texture masking (TM) information and adding them.

The apparatus for removing artifacts according to an embodiment of the present invention may generate luminance adaptation information from a luminance image (S1030). Here, the luminance adaptation (LA) information may refer to information on a difference that can be distinguished by a human according to a background luminance value, that is, a masking effect of the background luminance in the human's perspective, and can be calculated by referring to adjacent pixels. I can. In the artifact removal method according to an embodiment of the present invention, steps S1020 and S1030 may be irrelevant even if the order is changed.

The apparatus for removing artifacts according to an embodiment of the present invention may generate a JND map based on contrast masking information and luminance adaptation information through a Non-linear Additivity Model for Masking (NAMM) (S1040). Here, the JND map (or JND information) may mean information on a difference that can recognize a difference according to a human visual characteristic.

The artifact removal apparatus according to an embodiment of the present invention may generate a difference image based on a luminance image and an image blurred through a low-pass filter (LPF) (S1050). Here, the LPF may include a Gaussian filter, the blurred image (L _LPF ) may mean an image forcibly removing the demosaicing artifact, and the difference image (L _diff ) may force the demosaicing artifact It may mean information on the difference between the removed image and the luminance image.

The apparatus for removing artifacts according to an embodiment of the present invention may detect a pixel in which an artifact has occurred based on a Just Noticeable Difference (JND) map and a difference image (S1060). In other words, in an embodiment of the present invention, when a component value of a difference image corresponding to at least one pixel with respect to at least one pixel in a luminance image is greater than a component value of a JND map corresponding to at least one pixel, at least one The pixels of can be detected as pixels with artifacts.

The artifact removing apparatus according to an embodiment of the present invention may remove the artifact based on the component value of the reference pixel through a clamping process (S1070). In other words, in an embodiment of the present invention, when the interpolated color component value of the pixel in which the artifact is detected is greater than the maximum color component value of the reference pixel, the final color component value of the pixel in which the artifact is detected may be determined as the maximum color component value. In addition, when it is smaller than the minimum color component value, the final color component value of the pixel in which the artifact is detected may be determined as the minimum color component value. In addition, if the interpolated color component value of the pixel in which the artifact is detected is less than or equal to the maximum color component value of the reference pixel, and greater than or equal to the minimum color component value, the interpolated color component value may be determined as the final color component value. have. Here, the reference pixel may be set differently according to the interpolated color component of the pixel in which the artifact is detected.

The operation according to an embodiment of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. In addition, the computer-readable recording medium may be distributed over a computer system connected through a network to store and execute a computer-readable program or code in a distributed manner.

In addition, the computer-readable recording medium may include a hardware device specially configured to store and execute program commands, such as ROM, RAM, and flash memory. The program instructions may include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

While some aspects of the invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or characteristic of a method step. Similarly, aspects described in the context of a method can also be represented by a corresponding block or item or a feature of a corresponding device. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer or electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In embodiments, the field programmable gate array may work with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by some hardware device.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. You will understand that you can do it.

Claims (20)

As a method of removing artifacts from a demosaiced image,
Generating a luminance image from the demosaiced image;
Generating a Just Noticeable Difference (JND) map from the luminance image;
Generating a difference image between the luminance image and an image to which a LPF (Low-Pass Filter) is applied to the luminance image;
Detecting a pixel in which an artifact has occurred based on the JND map and the difference image; And
Including the step of removing the artifact generated in the pixel through a clamping process,
The step of removing the artifact generated in the pixel through the clamping process,
And removing an artifact generated in the pixel by using a plurality of reference pixels located within a preset range from the pixel,
The step of removing an artifact generated in the pixel by using a plurality of reference pixels located within a preset range from the pixel,
And determining a final color component value of the pixel by comparing the interpolated color component value of the pixel with a maximum color component value and a minimum color component value of the plurality of reference pixels. The method according to claim 1,
Generating a JND map from the luminance image,
Generating luminance adaptation information for a masking effect of background luminance from the luminance image;
Generating contrast masking information including edge masking information and texture masking information from the luminance image; And
And generating the JND map based on the luminance adaptation information and the contrast masking information. The method according to claim 2,
Generating the JND map based on the luminance adaptation information and the contrast masking information,
Generating the JND map from the luminance adaptation information and the contrast masking information through a non-linear addition model for masking (NAMM). The method according to claim 2,
Generating contrast masking information including edge masking information and texture masking information from the luminance image,
Generating a structure image and a texture image from the luminance image through a TV (Total Variation) based decomposition method;
Generating the edge masking information and the texture masking information based on a maximum luminance difference and a weight in four directions of the structure image and the texture image; And
And generating the contrast masking information by summing the edge masking information and the texture masking information. The method according to claim 1,
The step of detecting a pixel in which an artifact has occurred based on the JND map and the difference image,
When a component value of the difference image corresponding to the at least one pixel with respect to at least one pixel in the luminance image is greater than a component value of the JND map corresponding to the at least one pixel, the at least one pixel is A method of removing artifacts comprising the step of detecting as a pixel in which the artifact has occurred. delete The method according to claim 1,
The preset range is,
The artifact removal method that is set differently according to the interpolated color component of the pixel. delete The method according to claim 1,
The step of determining the final color component value of the pixel,
Determining a final color component value of the pixel as the maximum color component value when the interpolated color component value of the pixel is greater than the maximum color component value; And
And determining a final color component value of the pixel as the minimum color component value when the interpolated color component value of the pixel is less than the minimum color component value. The method according to claim 1,
The step of determining the final color component value of the pixel,
When the interpolated color component value of the pixel is less than or equal to the maximum color component value and greater than or equal to the minimum color component value, determining a final color component value of the pixel as the interpolated color component value Including, artifact removal method. A device that removes artifacts from a demosaiced image,
At least one processor; And
A memory in which at least one instruction executed through the at least one processor is stored,
The at least one command,
Executed to generate a luminance image from the demosaiced image,
It is executed to generate a JND (Just Noticeable Difference) map from the luminance image,
It is executed to generate a difference image between the luminance image and an image to which LPF (Low-Pass Filter) is applied to the luminance image,
It is executed to detect a pixel in which an artifact has occurred based on the JND map and the difference image,
By comparing the interpolated color component value of the pixel, the maximum color component value and the minimum color component value of a plurality of reference pixels located within a preset range from the pixel through a clamping process, the final color component value of the pixel is determined. , The artifact removal apparatus executed to remove the artifact generated in the pixel. The method of claim 11,
The at least one command,
It is executed to generate luminance adaptation information for a masking effect of background luminance from the luminance image,
It is executed to generate contrast masking information including edge masking information and texture masking information from the luminance image,
The artifact removal apparatus, which is executed to generate the JND map based on the luminance adaptation information and the contrast masking information. The method of claim 12,
The at least one command,
The artifact removal apparatus, which is executed to generate the JND map from the luminance adaptation information and the contrast masking information through a non-linear addition model for masking (NAMM). The method of claim 12,
The at least one command,
It is executed to generate a structure image and a texture image from the luminance image through a TV (Total Variation)-based decomposition method,
It is executed to generate the edge masking information and the texture masking information based on the maximum luminance difference and weight in the four directions of the structure image and the texture image,
The artifact removal apparatus, which is executed to generate the contrast masking information by summing the edge masking information and the texture masking information. The method of claim 11,
The at least one command,
When a component value of the difference image corresponding to the at least one pixel with respect to at least one pixel in the luminance image is greater than a component value of the JND map corresponding to the at least one pixel, the at least one pixel is An artifact removal device that is executed to detect as a pixel in which an artifact has occurred. delete The method of claim 11,
The preset range is,
An apparatus for removing artifacts that is set differently according to the interpolated color component of the pixel. delete The method of claim 11,
The at least one command,
When the interpolated color component value of the pixel is greater than the maximum color component value, it is executed to determine a final color component value of the pixel as the maximum color component value,
And determining the final color component value of the pixel as the minimum color component value when the interpolated color component value of the pixel is less than the minimum color component value. The method of claim 11,
The at least one command,
When the interpolated color component value of the pixel is less than or equal to the maximum color component value and greater than or equal to the minimum color component value, the final color component value of the pixel is determined as the interpolated color component value. , Artifact removal device.

Images