KR102507165B1 - Image processing apparatus and image processing method thereof - Google Patents

Abstract

An image processing device is disclosed. The image processing apparatus performs noise removal processing on an input unit and an image input through the input unit, identifies a first pixel block based on a target pixel in the input image, and removes noise from the noise-removed image based on the target pixel. Two pixel blocks are identified, a gain value is obtained based on a pixel value included in the first pixel block and a pixel value included in the second pixel block, and the acquired gain value is applied to the second pixel block to reduce noise. and a processor performing enhancement processing on the removed image.

Description

Image processing apparatus and image processing method thereof {Image processing apparatus and image processing method thereof }

The present disclosure relates to an image processing device and an image processing method thereof, and more particularly, to an image processing device performing enhancement processing on an input image and an image processing method thereof.

Thanks to the development of electronic technology, various types of electronic devices are being developed and supplied. In particular, display devices used in various places such as homes, offices, and public places have been continuously developed in recent years.

Recently, the demand for high-resolution video service and real-time streaming service is greatly increasing. However, there is a problem in that lossy compression is unavoidable for transmission of high-resolution video and streaming video, which leads to a decrease in image quality.

The present disclosure is in accordance with the above-described needs, and an object of the present disclosure is to improve the sharpness and local contrast of edges reduced by image compression, noise removal processing, etc. An image processing device and an image processing method thereof are provided.

In order to achieve the above object, an image processing apparatus according to an embodiment of the present disclosure performs noise removal processing on an input unit and an image input through the input unit, and performs a first noise removal process based on a target pixel in the input image. A pixel block is identified, a second pixel block is identified based on the target pixel in the noise-removed image, and based on a pixel value included in the first pixel block and a pixel value included in the second pixel block and a processor that obtains a gain value and applies the obtained gain value to the second pixel block to perform enhancement processing on the noise-removed image.

In this case, the processor may obtain the gain value based on the variance of pixel values included in the first pixel block and the variance of pixel values included in the second pixel block.

Alternatively, the processor may obtain the gain value based on a gradient of pixel values included in the first pixel block and a gradient of pixel values included in the second pixel block.

In addition, when the first pixel block is included in the texture region of the input image, the processor determines the variance of pixel values included in the first pixel block and the variance of pixel values included in the second pixel block. The gain value is obtained based on, and when the first pixel block is included in an edge region of the input image, a gradient for a pixel value included in the first pixel block and a pixel included in the second pixel block The gain value may be obtained based on a gradient for the value.

The processor may further include a pixel value included in the second pixel block, a first average value of pixel values included in the first pixel block, and a second average value of pixel values included in the second pixel block. and based on the gain value, enhancement processing may be performed on the noise-removed image.

In addition, the processor subtracts the second average value from each pixel value included in the second pixel block, multiplies the gain value, and adds the first average value to enhance the noise-removed image processing can be performed.

The processor obtains a pixel value of the target pixel in the second pixel block, subtracts the second average value from the acquired pixel value, multiplies the gain value, and adds the first average value. Thus, enhancement processing may be performed on the noise-removed image.

Also, the processor may perform the enhancement process by applying at least one of unsharp masking and pixel value stretching.

The processor may further include a display, and the processor may control the display to display the enhanced image.

Meanwhile, an image processing method of an image processing apparatus according to an embodiment of the present disclosure includes performing noise removal processing on an input image, identifying a first pixel block based on a target pixel in the input image, Identifying a second pixel block based on the target pixel in the noise-removed image, obtaining a gain value based on a pixel value included in the first pixel block and a pixel value included in the second pixel block and performing enhancement processing on the noise-removed image by applying the obtained gain value to the second pixel block.

In this case, the obtaining of the gain value may include obtaining the gain value based on the variance of pixel values included in the first pixel block and the variance of pixel values included in the second pixel block. .

Alternatively, the obtaining of the gain value may include obtaining the gain value based on a gradient of pixel values included in the first pixel block and a gradient of pixel values included in the second pixel block.

The obtaining of the gain value may include, when the first pixel block is included in the texture region of the input image, the variance of the pixel value included in the first pixel block and the gain value included in the second pixel block. The gain value is obtained based on dispersion of pixel values, and when the first pixel block is included in an edge region of the input image, a gradient for pixel values included in the first pixel block and the second pixel The gain value may be obtained based on a gradient of a pixel value included in a block.

In addition, the performing of the enhancement process may include a pixel value included in the second pixel block, a first average value of pixel values included in the first pixel block, and a pixel value included in the second pixel block. Enhancement processing may be performed on the noise-removed image based on the second average value and the gain value.

In the performing of the enhancement process, the noise is removed by subtracting the second average value from each pixel value included in the second pixel block, multiplying the gain value, and then adding the first average value. Enhancement processing may be performed on the image.

The performing of the enhancement process may include obtaining a pixel value of the target pixel in the second pixel block, subtracting the second average value from the obtained pixel value, multiplying the gain value, and then multiplying the pixel value by the gain value. An enhancement process may be performed on the noise-removed image by adding the average value of 1 .

In the performing of the enhancement process, the enhancement process may be performed by applying at least one of unsharp masking and pixel value stretching.

The method may further include displaying the enhanced image.

In a non-transitory computer readable medium storing computer instructions that cause an electronic device to perform an operation when executed by a processor of an electronic device according to an embodiment of the present disclosure, the operation includes noise on an input image. performing a removal process; identifying a first pixel block based on a target pixel in the input image; identifying a second pixel block based on the target pixel in the noise-removed image; obtaining a gain value based on a pixel value included in a pixel block and a pixel value included in the second pixel block, and applying the acquired gain value to the second pixel block to obtain an image from which noise has been removed. It may include a step of performing enhancement processing for.

According to various embodiments of the present disclosure, it is possible to improve sharpness and/or local contrast of an edge reduced by image compression, noise removal processing, and the like.

1 is a diagram for explaining an implementation example of a display device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a configuration of a display device according to an exemplary embodiment of the present disclosure.
3 is a diagram for explaining a similar patch search method according to an embodiment of the present disclosure.
4A to 4C are diagrams for explaining a pixel value stretching method according to an exemplary embodiment of the present disclosure.
5A to 5C are diagrams for explaining a filtering method according to an embodiment of the present disclosure.
6A and 6B are block diagrams schematically illustrating an image processing method according to various embodiments of the present disclosure.
7 to 13b are drawings for explaining another embodiment of the present disclosure.
14 is a table for explaining quantitative effects according to another embodiment of the present disclosure.
15 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.
16 is a block diagram illustrating the configuration of an image processing device according to another embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

Embodiments of the present disclosure may apply various transformations and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of technology disclosed. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. Terms are only used to distinguish one component from another.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

The expression at least one of A and B should be understood to denote either "A" or "B" or "A and B".

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module and implemented by at least one processor (not shown), except for "modules" or "units" that need to be implemented with specific hardware. It can be.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram for explaining an implementation example of a display device according to an embodiment of the present disclosure.

1 is a diagram for explaining an implementation example of an image processing device according to an embodiment of the present disclosure.

The image processing device 100 may be implemented as a TV or set-top box as shown in FIG. 1, but is not limited thereto, and is not limited thereto, such as a smart phone, a tablet PC, a notebook PC, a head mounted display (HMD), a NED (Near Eye Display), LFD (large format display), Digital Signage, DID (Digital Information Display), video wall, projector display, camera, etc. can be applied without

The image processing device 100 may receive various compressed images or images of various resolutions. For example, the image processing device 100 may include Moving Picture Experts Group (MPEG) (eg, MP2, MP4, MP7, etc.), joint photographic coding experts group (JPEG), Advanced Video Coding (AVC), H.264 , H.265, High Efficiency Video Codec (HEVC), etc., to receive video in a compressed form. Alternatively, the image processing device 100 may receive any one of SD (Standard Definition), HD (High Definition), Full HD, and Ultra HD images.

According to an embodiment, when an input image is a compressed image, it includes noise due to compression. For example, block-discrete cosine transform (BDCT) is a widely used and effective compression method, but generates compression noise such as blocky, ringing, mosquito, and stair case, blurring, and weak texture loss. For example, noise such as blocky can cause discontinuities at the boundaries of blocks used for compression. In addition, ringing, mosquito noise, and blurring may occur around the edge due to quantization of high-frequency components.

Accordingly, when a compressed image is received, processing to remove compression noise is essential. Even in the process of removing compression noise, edges are blurred and local contrast is reduced. Accordingly, an enhancement process for restoring edge sharpness and local contrast after compression noise removal is additionally required.

However, if the sharpness of the edge is excessively increased, side effects such as image distortion or undershoot or overshoot may occur. Conversely, if the sharpness of the edge is not sufficiently restored, the sharpness of the high-definition level cannot be obtained. Accordingly, it is more important than anything else to calculate the optimal gain for restoring edge sharpness and local contrast.

Hereinafter, various embodiments for improving the sharpness and local contrast of an edge blurred in the process of compression or compression noise removal will be described.

2 is a block diagram illustrating a configuration of a display device according to an exemplary embodiment of the present disclosure.

According to FIG. 2 , the image processing device 100 includes an input unit 110 and a processor 120 .

The input unit 110 receives various types of content. For example, the input unit 110 is AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Bluetooth (Bluetooth), Zigbee (Zigbee), wired / wireless LAN (Local Area Network), WAN, Ethernet, IEEE 1394, HDMI , external device (eg, source device), external storage medium (eg, USB), external server (eg, USB) through communication methods such as USB, MHL, AES / EBU, optical, coaxial, etc. For example, a video signal may be input from a web hard drive or the like in a streaming or download method. Here, the video signal may be a digital signal, but is not limited thereto.

The processor 120 controls overall operations of the image processing device 100 .

According to an embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital image signals. However, it is not limited thereto. It is not a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), or a communication processor ( CP)) and ARM processors, or may be defined in terms of one or more of them In addition, the processor 140 is implemented as a System on Chip (SoC) with a built-in processing algorithm or a large scale integration (LSI). It may be, or it may be implemented in the form of a Field Programmable Gate Array (FPGA).

<Noise Removal Processing>

The processor 120 performs noise removal processing (or noise reduction processing) on the image input through the input unit 110 . In general, noise is generated during a process of compressing and transmitting an image. The reason for performing the noise removal process is that not only the quality of the image is degraded by noise, but also other image processing effects such as scaling, feature extraction, and resolution processing may be reduced.

According to an embodiment, the processor 120 may perform noise removal processing based on a method using non-local filtering and self-similarity. Most images have a feature called self-similarity, which is the basis of the fractal theory. That is, under the assumption that another region similar to one region is included in the image, after measuring the similarity of another region in the image, noise removal may be performed based on a pixel region having a high similarity.

For example, the processor 120 obtains a preset pixel area (or pixel block) in the image, for example, a preset patch (hereinafter referred to as a current patch), and patches similar to the current patch exist in the image. Suppose you do Patches similar to the current patch are collected in the image, and similar components are regarded as structure components and dissimilar components are regarded as noise components. That is, the processor 120 removes noise in an image by leaving signals with high similarity between patches and removing signals with low similarity. Using this method, compared to a smoothing filter by low pass filtering, it is possible to minimize the blur of the structure component and effectively remove noise.

3 is a diagram for explaining a similar patch search method according to an embodiment of the present disclosure.

In FIG. 3 , similar patches are searched within a preset area 320 based on the current patch 310 . As a cost function for determining similarity, mean square error (MSE), sum of absolute difference (SAD), median absolute deviation (MAD), correlation, and the like can be used. For example, when MSE is applied, similar patches may be obtained by calculating MSEs of acquired patches of a compressed image and searching for patches having a high similarity from the MSE point of view, that is, patches having a small MSE difference.

When the similar patches 331 to 336 are acquired, compression noise is removed by utilizing the self-similarity of the similar patches 330-1 to 330-6.

To remove noise using similar patches, NLM (A. Buades, B. Coll, and J.M. Morel, “A non-local algorithm for image denoising,” in Proc. IEEE Int. Conf. Comput. Vis., pp. 60-65, 2005.), BM3D (K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol 16, no. 8, pp. 2080?2095, Aug. 2007), CONCOLOR (J. Zhang, R. Xiong, C. Zhao, Y. Zhang, S. Ma, and W. Gao, “CONCOLOR: Constrained non -convex low-rank model for image deblocking,” IEEE Trans. Image Process., vol. 25, no. 3, pp. 1246-1259, 2016) can be used.

For example, when CONCOLOR is applied, similar components and dissimilar components are distinguished using singular value decomposition (SVD). CONCOLOR removes noise by maintaining similar components corresponding to the structure components of the patch and reducing the signal of dissimilar components corresponding to noise. On the other hand, in BM3D, 3D-DCT of patches is performed to distinguish between similar and dissimilar components between patches. Using SVD, similar and dissimilar components can be distinguished more accurately than 3D-DCT.

<Gain acquisition and enhancement processing according to an embodiment>

Next, the processor 120 obtains a gain for enhancement processing based on the denoised image and the input image.

According to an embodiment, the processor 120 may calculate a gain for restoring a blurred edge or degraded sharpness in a noise removal process.

In this case, first, it is assumed that the noise power around the edge is not greater than the power of the edge. This assumption is due to the generation process of compression noise. That is, this is because compression noise is generated in a process of performing descret cosine transform (DCT) in a patch unit to represent an edge during compression and then performing quantization on DCT coefficients. Second, it is assumed that the noise removal method using non-local filtering does not make the texture a perfectly flat surface. The perfectly flat region in the noise-removed image may be a region where the input image is already perfectly flat or almost flat, and thus the user cannot perceive the texture.

Based on these two assumptions, a method for finding an optimal gain according to an embodiment of the present disclosure may be performed in such a way that a metric to be preserved according to local characteristics of a compressed image is preserved even after noise is removed. For example, the gain may be calculated by calculating metrics such as average, variance, and gradient of the compressed image and preserving them.

Specifically, the processor 120 may identify a first pixel block based on a target pixel in the input image and identify a second pixel block based on the corresponding target pixel in the noise-removed image. According to an embodiment, the processor 120 may obtain a first pixel block and a second pixel block having a target pixel as a center pixel and having a predetermined size including neighboring pixels. Here, the predetermined size may be various sizes such as 3*3 or 5*5, but hereinafter, for convenience of description, a case in which a 3*3 pixel block is obtained will be assumed and described. Here, the pixel block may be implemented as a patch, for example.

The processor 120 may obtain a gain value based on a pixel value included in the first pixel block and a pixel value included in the second pixel block.

및

라고 하고, 제2 패치에서의 평균 및 분산을 각각

및

라 한다. According to an example, the processor 120 may calculate the average and variance in the first patch and calculate the average and variance in the second patch. Here, the mean and variance in the first patch are respectively

and

, and the mean and variance in the second patch are respectively

and

say

및 제2 패치에서의 분산

에 기초하여 게인 g를 획득할 수 있다. 여기서, 분산은 (편차)²의 평균을 의미하며, 편차는 제1 패치(또는 제2 패치)에 포함된 픽셀 값들의 평균과 각 픽셀 값의 차이가 될 수 있다. The processor 120 distributes in the first patch

and variance in the second patch

Gain g can be obtained based on Here, the variance means the average of (deviation) ² , and the deviation may be a difference between the average of pixel values included in the first patch (or the second patch) and each pixel value.

및

라 한다. 여기서, 그래디언트는 인접 픽셀 간 픽셀 값 차이를 의미하며, 최대 그래디언트는 제1 패치(또는 제2 패치)에 포함된 인접 픽셀 간 픽셀 값 차이 중에서 가장 큰 값이 될 수 있다. According to another example, the processor 120 may calculate the maximum gradient in the first patch and the maximum gradient in the second patch. Here, the maximum gradient in the first patch and the maximum gradient in the second patch are respectively

and

say Here, the gradient means a pixel value difference between adjacent pixels, and the maximum gradient may be the largest value among pixel value differences between adjacent pixels included in the first patch (or the second patch).

Meanwhile, it may be effective to calculate the gain based on different metrics according to the local characteristics of the image. For example, an image is divided into flat, edge, and texture regions.

In this case, preserving the maximum gradient of the edge is effective in preventing blur in the edge area, and since there is no pixel with a maximum gradient larger than the edge in a narrow area such as a patch, side effects such as overshoot do not occur. In the texture region, unlike the edge region, since signals having similar sizes are distributed in a complex manner, it may be effective to preserve dispersion. However, a separate restoration process may not be performed on the flat area.

가 선명도 복원을 위한 최적 게인이 되고, 텍스처 영역에서는

가 선명도 복원을 위한 최적 게인이 된다.Accordingly, a gain may be obtained based on variance in the texture area, and a gain may be obtained based on the maximum gradient in the edge area. Accordingly, in the edge area

becomes the optimal gain for sharpness restoration, and in the texture area

is an optimum gain for sharpness restoration.

Thereafter, the processor 120 applies the obtained gain value to the second pixel block to perform enhancement processing on the noise-removed image. In this case, the processor 120 may obtain an output pixel value of the target pixel by applying the acquired gain value to the target pixel included in the second pixel block. The processor 120 may obtain an output image by processing all pixel values of the input image in this way.

That is, the processor 120 obtains a gain in units of a preset pixel block (eg, a preset patch), applies the obtained gain to pixel values of target pixels included in the second pixel block, and stretches the target pixel. value can be obtained.

Specifically, the processor 120 includes a pixel value included in the second pixel block, an average value of pixel values included in the first pixel block (hereinafter referred to as a first average value), and a pixel value included in the second pixel block. A stretched target pixel value may be obtained based on an average value (hereinafter, referred to as a second average value) and an obtained gain value for .

According to an example, the processor 120 subtracts the first average value from the pixel values included in the second pixel block, that is, the second patch, multiplies the obtained pixel value by the gain value, and then calculates the second average value for the obtained pixel value. Values may be added to obtain an enhanced target pixel value. Accordingly, the corresponding patch can maintain the average of the first pixel block composed of the input pixel values and also maintain the characteristic metric in the input image.

4A to 4C are diagrams for explaining a pixel value stretching method according to an exemplary embodiment of the present disclosure.

For example, as shown in FIG. 4A , the first pixel block 410 is identified based on the target pixel 411 in the input image (eg, compressed image), and the target pixel 421 in the noise-removed image. ), the second pixel block 420 is identified. Here, the target pixel 421 of the noise-removed image may be a pixel at the same position as the target pixel 411 of the input image.

Subsequently, a first average value Pm of pixel values included in the first pixel block 410 and a second average value Pm' of pixel values included in the second pixel block 420 are obtained.

Subsequently, as shown in FIG. 4B , a third pixel block 430 is obtained by subtracting the second average value Pm' from each pixel value included in the second pixel block 420, and the corresponding pixel block 430 The fourth pixel block 440 is obtained by multiplying each value included in ) by the gain value g obtained according to an embodiment of the present disclosure.

Thereafter, the output pixel block 450 may be obtained by adding the first average value Pm of the pixel values included in the first pixel block 410 to each value included in the fourth pixel block 440 . .

This processing may be performed for each pixel value included in the pixel block as shown in FIGS. 4A and 4B, but only for the target pixel 411 when only the pixel value of the target pixel is used as the pixel value of the output image. processing can be performed. For example, after subtracting the second average value (Pm') from the target pixel value P ₀ '(421) included in the second pixel block and multiplying the gain value (g) by the value (431) (441) , the second average value Pm may be added to calculate the output pixel value 451 for the target pixel.

Meanwhile, the processor 120 may perform image processing on pixels located at the edges of the input image based on the mirrored pixel values. For example, assuming that P ₁ is a pixel located at a corner of the input image, a virtual patch 450 having P ₁ as the center of 451 is obtained by mirroring pixel values at the boundary of the first patch 410. The corresponding second patch 460 may be obtained by generating , and processing may be performed in the same form as described in FIGS. 4A and 4B.

On the other hand, since it is assumed that the compression noise has zero-mean, the gain value does not affect the noise component, so the performance reduction in terms of noise removal does not occur.

In addition, as described above, since only the value of the target pixel, that is, the center pixel, is used in the output image in the second patch, discrepancies with neighboring pixels or sudden characteristic changes do not occur.

In some cases, it is also possible to perform the Gaussian filtering process on the first pixel block 410 and the second pixel block 420, and then perform the process shown in FIGS. 4A and 4B. For example, as shown in FIG. 5A, the Gaussian filter may have a large weight at 0 on the x-axis and a smaller weight toward the +/- part. ), the center of the mask 50 may have a large weight, and the weight may decrease towards the edges of the mask 50. However, the numerical value shown in FIG. 5A is an example, and the filtering numerical value varies depending on the sigma value of the Gaussian function.

As shown in FIGS. 5B and 5C, the processor 120 performs filtering by applying the Gaussian mask 50 to each pixel value included in the first patch 410 and the second patch 420, and then, FIG. 4A And processing as shown in FIG. 4B may be performed.

6A and 6B are block diagrams schematically illustrating an image processing method according to various embodiments of the present disclosure.

As shown in FIG. 6A , a noise-removed image Idn is obtained by performing noise removal processing on the input image In, that is, the compressed image (S610).

Next, a first metric (Min) is obtained based on the input image (In) (S620), and a second metric (Mdn) is obtained based on the noise-removed image (S630). For example, a first variance (or gradient) of a pixel value included in a first pixel block identified in the input image In is obtained, and a second pixel block identified in the noise-removed image Idn is obtained. A second variance (or gradient) for the included pixel values may be obtained.

및 제2 픽셀 블럭에서의 분산

에 기초하여 게인

를 획득할 수 있다. 다른 예로, 제1 픽셀 블럭에서의 최대 그래디언트

및 제2 픽셀 블럭에서의 최대 그래디언트를 각각

에 기초하여 게인

을 획득할 수 있다. Subsequently, a gain value Grst is obtained based on the first metric (Min) and the second metric (Mdn) (S640). As an example, the variance in the first pixel block

and the variance in the second pixel block.

gain based on

can be obtained. As another example, the maximum gradient in the first pixel block

and the maximum gradient in the second pixel block, respectively.

gain based on

can be obtained.

을 적용하고, 에지 영역에는 게인

을 적용하여 인핸스 처리를 수행할 수 있다. Thereafter, enhancement processing is performed by applying the acquired gain value Grst to the noise-removed image Idn (S650). For example, the texture area has gain

, and gain in the edge region

It is possible to perform enhancement processing by applying .

Alternatively, as shown in FIG. 6B, the metric is recalculated (S660) based on the output image Iout according to the application of the gain value obtained in step S640, and the gain value is recalculated based on the recalculated metric. It is also possible to calculate an optimized gain value by repeating the process.

<Gain acquisition and enhancement processing according to another embodiment>

Meanwhile, according to another embodiment, the processor 120 may calculate a gain for restoring sharpness of a blurred edge or degraded local contrast in the process of compressing an image. In this case, an optimal gain according to the compression rate of the original video may be calculated and stored in a table form, and enhancement processing may be performed using the result. Hereinafter, a method of calculating an optimal gain will be described.

The processor 120 may calculate an optimal gain based on a maximum a posteriori (MAP) technique.

, 인핸스 처리된 영상을

라고 할 수 있다. According to an embodiment, it is assumed that compression artifact removal (CAR) is applied as a noise removal process and edge sharpness restoration (ESR) is applied as an enhancement process. In this case, x is the original image, y is the compressed image, and the denoised image is

, the enhanced image

can be said

The processor 120 may obtain an optimal gain using a maximum a posteriori (MAP) technique and an optimization technique. Here, the MAP technique is a method of finding optimal parameters by combining given observation results and prior knowledge. Using this MAP technique, the optimal gain according to the compression rate of the original video is calculated and stored in the form of a table, and enhancement processing can be performed using this.

을 수학식 1과 같이 모델링할 수 있다. Specifically, the processor 120 generates an image from which compression noise has been removed based on the MAP technique.

can be modeled as in Equation 1.

은 에지 블러 잡음이 발생하며, 이를 e_b라고 할 수 있다. Here, x denotes an original image, that is, an image before compression is performed, and e _b denotes edge blur noise in an image from which compression noise is removed and has a Gaussian distribution. For example, when an original image x is compressed as shown in FIG. 7 , noise due to the compression occurs in the compressed image y. In addition, when compression noise removal is performed on the compressed image y, the image on which the compression noise removal process has been performed

causes edge blur noise, which can be referred to as e _b .

은 하기와 같은 수학식 2로 모델링될 수 있다. According to the MAP technique, an image in which edges are restored by edge sharpness restoration processing

may be modeled by Equation 2 as follows.

Equation 2 may be transformed into Equation 3 as follows by the Bayesian rule.

Since the edge blur noise e _b has a Gaussian distribution, the first part of Equation 3 can be expressed as Equation 4 below.

는 가우시안 분포를 가지는 에지 블러 잡음의 variance이다. here,

is the variance of edge blur noise having a Gaussian distribution.

를 추정하기 위하여, 예를 들어 도 8a에 도시된 바와 같이 JEPG Qulity Factor를 변경함에 따른 σ_b값을 산출할 수 있다.

In order to estimate , a value of σ _b may be calculated by changing the JEPG Qulity Factor, for example, as shown in FIG. 8A.

는 도 8b에 도시된 바와 같이

와 동일한 방식으로 산출될 수 있다. 즉, 수학식 3의 두번째 부분은 variance 차이가 클수록 작은 결과 값을 가지도록 하는 함수 fv를 이용하여 하기와 같은 수학식 5로 나타내어질 수 있다. Meanwhile, in an image compressed with a low bit rate, blurred edges have a wider edge width. Accordingly, the variance of the edges before compression becomes greater than the variance of the edges after compression. Accordingly

As shown in Figure 8b

can be calculated in the same way as That is, the second part of Equation 3 can be expressed as Equation 5 below using a function fv that has a smaller result value as the difference in variance increases.

는 가우시안 분포를 가지는 에지 분산 에러의 분산을 나타내는다. 도 9는 본 개시의 일 실시 예에 따른 함수 fv의 형태를 나타내는 도면이다. here,

represents the variance of the edge variance error having a Gaussian distribution. 9 is a diagram illustrating a form of a function fv according to an embodiment of the present disclosure.

Accordingly, an equation for obtaining an optimal gain may be expressed as Equation 6 below.

Here, Ω denotes a set of pixels corresponding to an edge area in the image.

According to an embodiment, the processor 120 may perform unsharp masking (USM) processing by applying a gain.

Specifically, the processor 120 may perform USM processing based on Equation 7 below.

Here, k represents the amount of sharpness increase, i.e. gain, r is the blur radius used to create x _smooth , t is the minimum variance threshold to which unsharp masking is applied, and g(x) is the high-frequency component extracted by applying the smoothing filter. am. Here, r and t are fixed values, but k is a variable value.

That is, if the high-frequency component g(x) of the image is extracted, multiplied by the gain k, and then added, edge sharpness can be improved.

Meanwhile, since k is obtained iteratively in the above-described manner, Equation 7 becomes the following Equation 8.

Here, l represents the number of iterations.

Meanwhile, when USM is applied to a complex texture region in a compressed image, unwanted distortion may occur. This is because it is difficult to maintain a complex weak texture when compression is performed at a low bit rate. Conversely, an edge region having a simple structure and a large gradient can be sharpened and restored more reliably. Accordingly, a complex texture region can be detected using the Laplacian filter, and unsharp masking can be applied only to the edge region excluding the weak texture region using the edge weight map w _e as shown in Equation 9 below.

Here, the edge weight map w _e can be expressed as in Equation 10 below.

영상에 Laplacian filter를 적용 후, smoothing필터를 적용한 결과이다. 즉, L_smooth = │LaplacianFilter(

)│*b 이고, b는 blur kernel이다. Here, L _smooth is

This is the result of applying the smoothing filter after applying the Laplacian filter to the image. That is, L _smooth = │LaplacianFilter(

)│*b, and b is the blur kernel.

10 illustrates an edge weight map w _e according to an embodiment of the present disclosure.

Meanwhile, in Equation 10, EC(i,j) is an edge correlation value, and can be obtained as shown in Equation 11 below by applying step kernel S to the vertical and horizontal directions of the input image.

Here, T _hor and T _ver mean n pixel sets in vertical and horizontal directions centered on pixel (i, j).

11A shows an example of step kernel S according to an embodiment of the present disclosure, and FIG. 11B shows that the edge gradient of the compressed image decreases as the JEPG quality factor decreases.

는 undershoot과 overshoot의 부작용이 발생할 수 있다. 이에 따라 하기 수학식 12와 같이 현재 픽셀 주변의 최소 픽셀 값과 최대 픽셀 값을 획득하여, 픽셀 값이 해당 값들을 초과하거나, 미만이 되지 않도록 픽셀 값을 조정할 수 있다. Meanwhile, an image with unsharp masking applied to the edge

can cause side effects of undershoot and overshoot. Accordingly, as shown in Equation 12 below, the minimum pixel value and the maximum pixel value around the current pixel may be obtained, and the pixel value may be adjusted so that the pixel value does not exceed or fall below the corresponding values.

12 is an example of improving edge sharpness without overshoot and undershoot according to an embodiment of the present disclosure.

For example, as shown in FIG. 13A, when the original image 1310 is compressed, ringing, mosquito, and edge blur phenomena occur in the compressed image 1320 (1321). 13A to 13D show the input image (ie, the original image 1310) on the left side and the image processed in each step on the right side for convenience of description.

Subsequently, when noise removal processing is performed on the compressed image 1320, as shown in FIG. ).

If edge sharpness is improved by simply applying k calculated through the optimization technique of the present disclosure to an image 1330 subjected to noise removal processing, overshoot/undershoot occurs in the corresponding image 1340 as shown in FIG. 13C becomes (1341).

Accordingly, if the minimum pixel value and the maximum pixel value according to Equation 12 are used, as shown in FIG. 13D , edge sharpness can be restored in the corresponding image 1350 without overshoot/undershoot (1351).

Meanwhile, the processor 120 may obtain an optimal gain according to the compression ratio by applying an optimization technique based on Equations 6 and 12. Here, as an example of an optimization technique, a gradient decent technique can be used. The gradient decent method finds the direction in which the gradient descends most steeply from the current position, moves in that direction, and takes a new position. This method is repeated to find an optimization point.

When the optimal gain according to each compression rate is obtained, the processor 120 may store the obtained gain in the form of a lookup table. Then, the processor 120 may obtain a corresponding gain from the lookup table based on the compression rate of the input image, and perform enhancement processing based on the obtained gain.

14 is a table for explaining quantitative effects according to another embodiment of the present disclosure.

After JPEG compression with the quality factor (QF) of 10 and 20 for 8 images, the PSNR change is displayed. When QF is 10, PSNR (Peak Signal-to-noise ratio) improves by up to 2.81dB as a result of noise cancellation using CONCOLOR, and when QF is 20, PSNR improves by up to 2.95dB as a result of noise cancellation. After noise removal, as a result of edge sharpness restoration (CONCOLOR + edge sharpness restoration), the PSNR improved by up to 0.42dB at QF 10 and by up to 0.46dB at QF 20.

15 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.

According to FIG. 15, first, a noise removal process is performed on an input image (S1510).

Next, a first pixel block is identified based on the target pixel in the input image, and a second pixel block is identified based on the target pixel in the noise-removed image (S1520).

Subsequently, a gain value is obtained based on the pixel values included in the first pixel block and the pixel values included in the second pixel block (S1530).

Thereafter, enhancement processing is performed on the noise-removed image by applying the obtained gain value to the second pixel block (S1540).

Also, in step S1530, a gain value may be obtained based on the variance of pixel values included in the first pixel block and the variance of pixel values included in the second pixel block.

Also, in step S1530, a gain value may be obtained based on a gradient of pixel values included in the first pixel block and a gradient of pixel values included in the second pixel block.

Further, in step S1530, when the first pixel block is included in the texture region of the input image, a gain is obtained based on the variance of pixel values included in the first pixel block and the variance of pixel values included in the second pixel block. value can be obtained. Further, when the first pixel block is included in an edge region of the input image, a gain value may be obtained based on a gradient of pixel values included in the first pixel block and a gradient of pixel values included in the second pixel block. can

Further, in step S1540, the pixel values included in the second pixel block, the first average value of the pixel values included in the first pixel block, and the second average value and gain value of the pixel values included in the second pixel block Based on , enhancement processing may be performed on an image from which noise is removed.

Further, in step S1540, enhancement processing may be performed on the noise-removed image by subtracting the second average value from each pixel value included in the second pixel block, multiplying the gain value, and then adding the first average value. there is.

Further, in step S1540, a pixel value of a target pixel in the second pixel block is obtained, a second average value is subtracted from the obtained pixel value, a gain value is multiplied, and the first average value is added to obtain a noise-removed image. Enhancement processing can be performed on .

Also, in step S1540, enhancement processing may be performed by applying at least one of unsharp masking and pixel value stretching.

16 is a block diagram illustrating the configuration of an image processing device according to another embodiment of the present disclosure.

According to FIG. 16 , the image processing device 100' includes an input unit 110, a processor 120, a display 130, and a memory 140. Among the components shown in FIG. 16 , detailed descriptions of components overlapping those shown in FIG. 2 will be omitted.

The display 130 may be implemented in various forms such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a quantum dot (QD) display panel, and the like. can

The processor 120 stores a CPU, a ROM (or non-volatile memory) in which a control program for controlling the image processing device 100' is stored, and data input from the outside of the image processing device 100' or image processing It may include RAM (RAM, or volatile memory) used as a storage area corresponding to various tasks performed in the device 100'.

The CPU accesses at least one memory 140 and performs various operations using stored various programs, contents, data, and the like.

Here, the at least one memory 140 may be implemented as an internal memory such as a ROM or RAM included in the processor 120 or may be implemented as a separate memory from the processor 120 . In this case, the at least one memory 140 may be implemented in the form of a memory embedded in the image processing device 100' or in the form of a removable memory in the image processing device 100', depending on the purpose of storing data. . For example, data for driving the image processing device 100' is stored in a memory embedded in the image processing device 100', and data for an extended function of the image processing device 100' is image processed. It can be stored in a memory that can be attached to or detached from the device 100'. Meanwhile, in the case of memory embedded in the image processing device 100', it is implemented in the form of a non-volatile memory, volatile memory, hard disk drive (HDD) or solid state drive (SSD), etc., and is detachable from the display device 200. Possible memory may be implemented in the form of a memory card (eg, micro SD card, USB memory, etc.), an external memory (eg, USB memory) connectable to a USB port, and the like. According to an example, the memory 140 may be implemented as an internal memory of the processor 120, and in this case, it may be implemented as an N-line memory according to a hardware memory capacity limit of the processor 120.

In addition, the image processing device 100 ′ may further include an output unit (not shown) that outputs a sound signal and a user interface (not shown) that receives a user command.

According to various embodiments described above, it is possible to improve the sharpness and/or local contrast of an edge reduced by image compression, noise removal processing, and the like.

However, it goes without saying that various embodiments of the present disclosure can be applied to all electronic devices capable of image processing, such as image receiving devices such as set-top boxes and display devices such as TVs, as well as image processing devices.

Meanwhile, various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor 120 itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing the processing operation of the image processing device 100 according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. can When the computer instructions stored in the non-transitory computer readable medium are executed by the processor of the specific device, the processing operation in the audio output device 100 according to various embodiments described above is performed by the specific device.

A non-transitory computer readable medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of the non-transitory computer readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is commonly used in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: image processing device 110: input unit
120: processor

Claims (19)

input unit; and
performing noise removal processing on an image input through the input unit;
Identifying a first pixel block based on a target pixel in the input image;
Identifying a second pixel block based on the target pixel in the noise-removed image;
obtaining a gain value based on a pixel value included in the first pixel block and a pixel value included in the second pixel block;
A processor performing enhancement processing on the noise-removed image by applying the obtained gain value to the second pixel block;
the processor,
When the first pixel block is included in the texture region of the input image, based on a value obtained by dividing the variance of pixel values included in the first pixel block by the variance of pixel values included in the second pixel block Obtaining the gain value;
When the first pixel block is included in the edge region of the input image, based on a value obtained by dividing a gradient for pixel values included in the first pixel block by a gradient for pixel values included in the second pixel block An image processing device that obtains the gain value. delete delete delete According to claim 1,
the processor,
Based on a pixel value included in the second pixel block, a first average value of pixel values included in the first pixel block, a second average value of pixel values included in the second pixel block, and the gain value An image processing device that performs enhancement processing on the image from which the noise has been removed. According to claim 5,
the processor,
Image processing for performing enhancement processing on the noise-removed image by subtracting the second average value from each pixel value included in the second pixel block, multiplying the gain value, and then adding the first average value. Device. According to claim 5,
the processor,
The noise is removed by obtaining a pixel value of the target pixel in the second pixel block, subtracting the second average value from the obtained pixel value, multiplying the gain value, and then adding the first average value. An image processing device that performs enhancement processing on an image. According to claim 1,
the processor,
An image processing device that performs the enhancement processing by applying at least one of unsharp masking and pixel value stretching. According to claim 1,
It further includes a display;
the processor,
An image processing device that controls the display to display the enhanced image. In the image processing method of the image processing device,
performing noise removal processing on the input image;
identifying a first pixel block based on a target pixel in the input image;
identifying a second pixel block based on the target pixel in the noise-removed image;
obtaining a gain value based on a pixel value included in the first pixel block and a pixel value included in the second pixel block; and
Applying the obtained gain value to the second pixel block to perform enhancement processing on the noise-removed image;
The step of obtaining the gain value is,
When the first pixel block is included in the texture region of the input image, based on a value obtained by dividing the variance of pixel values included in the first pixel block by the variance of pixel values included in the second pixel block Obtaining the gain value;
When the first pixel block is included in an edge region of the input image, based on a value obtained by dividing a gradient for pixel values included in the first pixel block by a gradient for pixel values included in the second pixel block An image processing method of obtaining the gain value. delete delete delete According to claim 10,
The step of performing the enhancement process,
Based on a pixel value included in the second pixel block, a first average value of pixel values included in the first pixel block, a second average value of pixel values included in the second pixel block, and the gain value An image processing method for performing enhancement processing on an image from which the noise has been removed. According to claim 14,
The step of performing the enhancement process,
Image processing for performing enhancement processing on the noise-removed image by subtracting the second average value from each pixel value included in the second pixel block, multiplying the gain value, and then adding the first average value. method. ◈Claim 16 was abandoned when the registration fee was paid.◈ According to claim 14,
The step of performing the enhancement process,
The noise is removed by obtaining a pixel value of the target pixel in the second pixel block, subtracting the second average value from the obtained pixel value, multiplying the gain value, and then adding the first average value. An image processing method for performing enhancement processing on an image. ◈Claim 17 was abandoned when the registration fee was paid.◈ According to claim 10,
The step of performing the enhancement process,
The image processing method of performing the enhancement processing by applying at least one of unsharp masking and pixel value stretching. ◈Claim 18 was abandoned when the registration fee was paid.◈ According to claim 10,
The image processing method further comprising a; displaying the enhanced image. A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation comprising:
performing noise removal processing on the input image;
identifying a first pixel block based on a target pixel in the input image;
identifying a second pixel block based on the target pixel in the noise-removed image;
obtaining a gain value based on a pixel value included in the first pixel block and a pixel value included in the second pixel block; and
Applying the obtained gain value to the second pixel block to perform enhancement processing on the noise-removed image;
The step of obtaining the gain value is,
When the first pixel block is included in the texture region of the input image, based on a value obtained by dividing the variance of pixel values included in the first pixel block by the variance of pixel values included in the second pixel block Obtaining the gain value;
When the first pixel block is included in the edge region of the input image, based on a value obtained by dividing a gradient for pixel values included in the first pixel block by a gradient for pixel values included in the second pixel block A non-transitory computer readable medium for obtaining the gain value.

Images