KR20130105316A - Method and apparatus for noise reduction - Google Patents

Abstract

PURPOSE: A method for removing thermal noise of a video and an apparatus for the same are provided to remove the thermal noise generated in an image sensor when taking the video. CONSTITUTION: Noise is removed in a difference between a current frame and a previous frame in which temporal noise is reduced (S210). Motion information of the current frame is detected based on the difference in which noise is removed, and the temporal noise of the current frame is removed according to the motion information (S240). [Reference numerals] (AA) Current frame; (BB) Previous frame; (CC) Frame with its temporal noises removed; (S220) Calculate an average SAD value within W; (S230) Subtract an offset value from the average SAD value; (S250) Calculate a first motion detection coefficient; (S260) Calculate a second motion detection coefficient; (S270) Calculate a third motion detection coefficient; (S280) Remove temporal noises

Description

Method and Apparatus for Noise Reduction of Video {Method and Apparatus for Noise Reduction}

The present invention relates to a method and apparatus for removing noise in video.

Recently, digital images have been applied in various fields. Such digital images are degraded due to noise generated during image processing such as acquisition, storage, and transmission. This noise not only degrades the image quality but also decreases the efficiency of processes involved in compression, recognition, and the like. Therefore, noise reduction of digital images is one of the most important processes in an image processing system, and research is being conducted to remove noise and improve image quality while preserving signal characteristics.

Noise reduction techniques in digital images are classified into three categories: spatial noise reduction, temporal noise reduction, and 3D noise reduction.

Spatial noise reduction technique removes noise only in the space domain, ignoring processing in the time domain. The temporal noise cancellation technique removes noise by considering pixels in the time domain. Three-dimensional noise reduction (3DNR) is a method that combines the temporal and spatial methods to obtain their advantages.

U.S. Patent # 7705918

The present invention provides an apparatus and method for removing thermal noise generated in an image sensor during video shooting.

A temporal noise removing method of a video image according to an exemplary embodiment of the present invention may include removing noise from a difference between a current frame and a temporal noise canceled previous frame; And detecting motion information of the current frame based on the noise canceled difference, and removing temporal noise of the current frame according to the motion information.

The noise removing step of the difference image may include averaging difference values between corresponding pixels of the current frame and the previous frame within a local window; And calculating a second average difference value by subtracting an offset value set according to a noise level from the average difference value.

The temporal noise removing step may include: calculating a first motion detection coefficient for the center pixel of the local window of the current frame based on the constant set according to the offset value and the second average difference value; Calculating a second motion detection coefficient by a weighted sum of an average value and a minimum value of the first motion detection coefficients within the local window of the current frame; And calculating, in the local window of the current frame, the third motion detection coefficient of the current frame based on the second motion detection coefficient and the third motion detection coefficient of the previous frame.

The temporal noise removing step may further include removing temporal noise of the current frame by a weighted sum of the current frame and a temporal noise canceled previous frame using the third motion detection coefficient.

In the spatial noise removing method of a video image according to an exemplary embodiment of the present invention, within a local window of a current frame from which temporal noise is removed, an average of a difference value with respect to the center pixel in each of a plurality of directions set based on the center pixel is obtained. Calculating; Detecting a direction in which the average difference value is minimum; And averaging pixel values of the pixels in the detected direction.

The plurality of directions may include horizontal, vertical, diagonal, and reverse diagonal directions.

In accordance with another aspect of the present invention, there is provided a method of removing noise from a video image, the method including removing noise from a difference between a current frame and a temporal noise-removed previous frame; Detecting motion information of the current frame based on the noise canceled difference; Removing temporal noise of the current frame by a weighted sum of the current frame and the temporal noise canceled previous frame according to the motion information; And removing spatial noise based on an edge direction in the current frame from which the temporal noise has been removed.

The method may further include filtering the current frame and the temporal noise canceled previous frame by applying a mask having a variable size according to a noise level before the differential image noise removing step.

In accordance with another aspect of the present invention, an apparatus for temporal noise removal of a video image includes: a differential unit for removing noise from a difference between a current frame and a temporal noise canceled previous frame; And a coefficient calculator configured to detect motion information of the current frame based on the noise removed difference to remove temporal noise of the current frame.

The image difference unit may include: a first difference unit which averages a difference value between corresponding pixels of the current frame and the previous frame in a local window; And a second difference unit configured to calculate a second average difference value by subtracting an offset value set according to a noise level from the average difference value.

The coefficient calculating unit may include: a first coefficient calculating unit configured to calculate a first motion detection coefficient with respect to the center pixel of the local window of the current frame based on the constant set according to the offset value and the second average difference value; A second coefficient calculating unit for calculating a second motion detection coefficient by a weighted sum of the average value and the minimum value of the first motion detection coefficient within the local window of the current frame; And a third coefficient calculating unit configured to calculate a third motion detection coefficient of the current frame based on the second motion detection coefficient and the third motion detection coefficient of the previous frame within the local window of the current frame. can do.

The apparatus may further include a noise canceller configured to remove temporal noise of the current frame by a weighted sum of the current frame and a temporal noise-removed previous frame using a third motion detection coefficient of the current frame. have.

According to an embodiment of the present invention, a spatial noise removing apparatus of a video image includes a difference value for a center pixel in a plurality of directions set based on the center pixel within a second local window of a current frame from which temporal noise is removed. A direction difference unit for calculating an average; A minimum value detector for detecting a direction in which the average difference value is minimum; And a noise remover configured to remove spatial noise of the current frame by averaging pixel values of the pixels in the detected direction.

The plurality of directions may include horizontal, vertical, diagonal, and reverse diagonal directions.

The noise removing apparatus of a video image according to an exemplary embodiment of the present invention removes noise from a difference between a current frame and a temporal noise-removed previous frame and calculates motion information of the current frame based on the noise-reduced difference. A temporal noise canceller configured to remove temporal noise of the current frame by a weighted sum of the current frame and the temporal noise canceled previous frame; And a spatial noise remover configured to remove spatial noise based on an edge direction in the current frame from which the temporal noise is removed.

The noise removing device of the video image filters the current frame and the temporal noise-removed previous frame by applying a mask having a variable size according to a noise level before the difference between the current frame and the temporal noise-removed previous frame. The variable filter may further include.

The present invention can improve image quality by minimizing blur in the moving area while maximizing the noise removing effect in the flat area in the moving video image.

1 is a block diagram schematically illustrating a configuration of an apparatus for removing noise of a video image according to an embodiment of the present invention.
2 is a block diagram schematically illustrating a configuration of a variable filter according to an embodiment of the present invention.
3 shows an example of a mask configuration according to an embodiment of the present invention.
4 is a block diagram schematically illustrating a configuration of a temporal noise canceller according to an embodiment of the present invention.
5 is a diagram schematically illustrating a configuration of an image difference unit of FIG. 4.
6 illustrates an example of a corresponding local window of a current frame and a previous frame.
7 is a diagram schematically illustrating a configuration of a coefficient calculator of FIG. 4.
8 is a block diagram schematically illustrating a configuration of a spatial noise canceller according to an embodiment of the present invention.
9 is a diagram illustrating weight calculation of a conventional adaptive weight averaging (AWA) algorithm.
10 to 12 are diagrams for describing the spatial noise cancellation of FIG. 8.
13 is a flowchart schematically illustrating a method of removing noise of a video image according to an embodiment of the present invention.
14 is a flowchart schematically illustrating the filtering method of FIG. 13.
FIG. 15 is a flowchart schematically illustrating a method of removing temporal noise of FIG. 13.
FIG. 16 is a flowchart schematically illustrating a method of removing spatial noise of FIG. 13.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Like reference numbers in the drawings denote like elements. In the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components are not limited by these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. It is noted that the terms "comprises" and / or "comprising" used in the specification are intended to be inclusive in a manner similar to the components, steps, operations, and / Or additions.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used in a sense that can be commonly understood by those skilled in the art. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

1 is a block diagram schematically illustrating a configuration of an apparatus for removing noise of a video image according to an embodiment of the present invention.

Referring to FIG. 1, the noise canceling apparatus 1 of a video image of the present invention includes a variable filter 10, a motion adaptive temporal noise canceller 40, a memory 50, and a spatial noise canceller 60. .

The video image (or video) is input to the noise reduction device 1 in units of frames.

The variable filter 10 is an NxM low pass filter (LPF), where N and M are natural numbers, the magnitude of which varies in proportion to the noise level. The variable filter 10 filters the current frame and the previous frame. Here, the previous frame is a frame from which temporal noise is removed. The noise level may be a noise parameter which is a preset value or a value input by a user. Alternatively, the noise level may vary with sensor gain. It is assumed that the noise level of the current frame and the previous frame is the same.

The variable filter 10 includes a first variable filter 20 for low pass filtering the current frame and a second variable filter 30 for low pass filtering the previous frame. Prior to temporal and spatial noise cancellation, the current frame and previous frame are first low-pass filtered to remove noise to reduce errors in motion detection. When the noise is relatively small, the size of the variable filter 10 is used to minimize the problem that the motion blur occurs when removing the noise for motion detection. On the contrary, when there is a lot of noise, the size of the variable filter 10 may be used to sufficiently remove the noise for motion detection.

The motion adaptive temporal noise canceller 40 (hereinafter, referred to as a "temporal noise canceller") removes temporal noise based on the filtered motion information of the current frame. Moving object tracking between adjacent frames is needed to accurately remove noise from consecutively input frames. However, when the noise is severe, it is difficult to detect accurate motion information of the moving object. Therefore, the temporal noise removing unit 40 performs additional noise removing when detecting the motion information.

Specifically, the temporal noise remover 40 removes noise included in the difference between the current frame and the previous frame from which the temporal noise is removed. The motion detection coefficient representing the motion information of the current frame is calculated based on the noise-reduced difference to remove temporal noise of the current frame.

Typical temporal noise cancellation detects motion information between the current frame and the previous frame, and classifies the pixels into motion regions and non-motion regions. In the non-motion region, a temporal filter is applied to the pixels in the current frame and the previous frame along the time axis. In the motion domain, no noise is removed by the temporal filter to avoid motion blur. In contrast, the embodiment of the present invention performs temporal noise removal even in the motion region of the moving object, and minimizes blur in the motion region while maintaining the noise removal effect in the flat region as much as possible. To this end, the noise included in the difference is removed by adding a difference of an offset of a numerical value corresponding to the noise to the difference between temporally adjacent frames. The noise of the motion detection coefficient is removed to compensate for the motion detection coefficient.

The memory 50 stores temporal noise canceled frames. The stored frame is referred to as the previous frame upon temporal noise cancellation of the next input frame. The memory 50 may be a frame buffer as temporary storage means.

The spatial noise remover 60 removes the spatial noise based on the edge direction in the current frame from which the temporal noise is removed. The spatial noise removing unit 60 obtains a difference value in four main directions for the temporal noise canceled frame, and takes an average value for the direction in which the difference value is the smallest and performs spatial noise removal.

2 is a block diagram schematically illustrating a configuration of a variable filter according to an embodiment of the present invention. 3 shows an example of a mask configuration according to an embodiment of the present invention.

Referring to FIG. 2, the variable filter 10 includes a noise predictor 110, a mask selector 130, a first variable filter 20, and a second variable filter 30.

The noise predictor 110 receives a noise parameter and estimates a noise level based on the noise parameter. The noise parameter may be sensor gain of an image sensor (not shown). According to the sensor gain, the luminance level of the image data is amplified, and the higher the sensor gain, the higher the noise.

The mask selector 130 selects a filter mask corresponding to the predicted noise level. A large mask makes it easy to detect motion in a noisy image, but a lot of motion blur occurs. Smaller masks have less motion blur, but motion is difficult when there is a lot of noise. Thus, the mask selector 130 selects one of the plurality of masks according to the predicted noise level. As an example, the mask selector 130 may match a mask corresponding to each noise level in advance one-to-one and then select a mask matching the predicted noise level. As another example, the mask selector 130 may select a mask according to a result of comparing the predicted noise level with a threshold value. A plurality of thresholds may be set to find a range corresponding to the predicted noise level, and a one-to-one matched mask may be selected. 3, masks of 5x13, 5x9, 5x5, and 5x3 are illustrated by way of example. 3 is an exemplary mask, and the present invention is not limited thereto. Of course, the mask size can be set. The center of each mask corresponds to the position of the current pixel. The higher the noise level, the larger the mask size, and the smaller the noise level, the smaller the mask size. For example, you can choose a mask size of 5x3 for low noise (low noise level) and a mask size of 5x13 for high noise (high noise level). Masks of the first variable filter 20 and the second variable filter 30 are selected to have the same size.

The first variable filter 20 filters the current frame by applying the selected mask.

The second variable filter 30 filters the previous frame by applying the selected mask.

4 is a block diagram schematically illustrating a configuration of a temporal noise canceller according to an embodiment of the present invention. FIG. 5 is a diagram schematically illustrating a configuration of the difference part of FIG. 4. FIG. 7 is a diagram schematically illustrating a configuration of a coefficient calculator of FIG. 4.

Referring to FIG. 4, the temporal noise remover 40 may include an image difference unit 402, a coefficient calculator 404, and a time filter 406.

The image difference unit 402 removes the noise included in the difference by subtracting an offset value set according to the noise level from the difference between the current frame and the previous frame. Referring to FIG. 5, the image difference unit 402 includes a first difference unit 412 and a second difference unit 472. Here, the previous frame is a temporal noise canceled frame.

)의 대응하는 국부적 윈도우(W)를 각각 도시하고 있다. The first difference unit 412 calculates the SAD (i) value for each pixel. Equation (1) calculates the SAD (i) value at the current pixel of the current frame, that is, the i-th pixel position. The SAD (i) value is the average difference value between the current frame and the previous frame, in the local window W of size NxM, as shown in equation (1). m is the position of the pixels covered by the local window W, and the i th pixel is the center pixel among the pixels covered by the local window W. n (W) is the number of pixels that the local window W covers. 6 shows the current frame In and the previous frame (S) for calculating SAD (i) values.

Each corresponding local window W is shown.

...(1)

...(One)

)을 획득하여 잡음에 의한 움직임 검출 에러를 저감한다. 식 (2)에서 보는 바와 같이, 0보다 작은 값을 갖지 않는 한도에서 SAD(i) 값에서 오프셋 값을 빼고 다시 상수 c를 더해준다. 여기서, 오프셋 값은 잡음이 많고 적음에 따라 가변적으로 설정해 주는 상수 값으로 잡음이 많을 경우 오프셋 값은 큰 값으로 설정하고, 잡음이 적을 경우 오프셋 값은 작은 값으로 설정한다. 예를 들어, 잡음이 상대적으로 적은 3dB의 경우 오프셋 값은 15, 잡음이 많은 10dB의 경우 30으로 설정할 수 있다. SAD(i) 값과 오프셋 값과의 차이가 0이 되어 잡음 제거가 어려워지는 경우를 방지하기 위해, SAD(i) 값과 오프셋 값과의 차이와 0 중 큰 값에 c를 더해준다. 즉, c는 0에 가깝지만 0이 아닌 작은 값으로서, SAD(i) 값이 0이 되지 않도록 설정되는 값이다.The second difference unit 472 may determine a second average difference value obtained by subtracting an offset value corresponding to the noise level from the average difference value of the i-th pixel, that is, the SAD (i) value (

) To reduce the motion detection error due to noise. As shown in Equation (2), the offset value is subtracted from the SAD (i) value, and the constant c is added again unless it has a value less than zero. Here, the offset value is a constant value that is variably set according to a lot of noise and a small amount. The offset value is set to a large value when there is a lot of noise, and the offset value is set to a small value when there is little noise. For example, you can set the offset value to 15 for 3dB with relatively low noise and 30 for 10dB with noisy. To prevent the difference between the SAD (i) value and the offset value becoming 0, it is difficult to remove noise, and c is added to the difference between the SAD (i) value and the offset value and the larger of 0. That is, c is a small value which is close to 0 but not 0, and is set such that the SAD (i) value does not become 0.

...(2)

...(2)

The coefficient calculator 404 calculates a motion detection coefficient representing motion information of the current frame based on the noise-reduced difference value. Referring to FIG. 7, the coefficient calculator 404 may include a first coefficient calculator 414, a second coefficient calculator 434, and a third coefficient calculator 454.

)를 산출한다. 제1 계수 산출부(414)는 제2 평균 차분 값(

)을 오프셋 값에 따라 설정되는 상수를 이용하여 스케일링함으로써, i번째 픽셀의 제1 움직임 검출 계수(

)를 산출한다. 식 (3)에서 보는 바와 같이, 제1 계수 산출부(414)는 제1 움직임 검출 계수(

)를 0에서 1 사이의 값을 갖도록 제2 평균 차분 값(

)에 적절한 스케일 상수 a를 곱하여 i번째 픽셀의 제1 움직임 검출 계수(

)를 산출한다. 스케일 상수 a는 오프셋 값과 연동하여 설정한다. 즉, 오프셋 값이 큰 경우 제1 움직임 검출 계수(

)는 작은 값을 갖게 되므로 스케일 상수 a를 크게 설정하고, 오프셋 값이 작은 경우 움직임 검출 계수(

)는 오프셋 값이 클 때에 비해 큰 값을 갖게 되므로 스케일 상수 a를 작게 설정한다. 예를 들어, 3dB 잡음의 경우 오프셋 값은 15, 스케일 상수(a)는 0.025로 설정하고, 10dB 잡음의 경우 오프셋 값은 30, 스케일 상수(a)는 0.035로 설정할 수 있다. The first coefficient calculating unit 414 may generate the first motion detection coefficient of the i th pixel based on the constant and the second average difference value set according to the offset value.

). The first coefficient calculating unit 414 has a second average difference value (

) Is scaled by using a constant set according to an offset value, so that the first motion detection coefficient

). As shown in Equation (3), the first coefficient calculating unit 414 uses the first motion detection coefficient (

) So that the second average difference value (

) Is multiplied by an appropriate scale constant a to determine the first motion detection coefficient (

). The scale constant a is set in conjunction with the offset value. That is, when the offset value is large, the first motion detection coefficient (

) Has a small value, so set the scale constant a to be large, and if the offset value is small, the motion detection coefficient (

) Has a larger value than when the offset value is large, so set the scale constant a small. For example, in the case of 3dB noise, the offset value may be set to 15, the scale constant (a) may be set to 0.025, in the case of 10dB noise, the offset value may be set to 30 and the scale constant (a) may be set to 0.035.

...(3)

... (3)

)의 평균값을 이용한다. 제2 계수 산출부(434)는 현재 프레임의 국부적 윈도우(W) 내에서, 제1 움직임 검출 계수(

)를 가중치로 하는 제1 움직임 검출 계수(

)의 평균값 및 최소값의 가중합에 의해 제2 움직임 검출 계수(

)를 산출한다. 식 (4)에서 보는 바와 같이 국부적 윈도우(W) 영역 안에서, 제1 움직임 검출 계수(

)의 평균값(

)과 제1 움직임 검출 계수(

)의 최소값(

)을 산출한다. 그리고, 식 (3)에서 산출한 제1 움직임 검출 계수(

)로 가중 합을 취하여 제2 움직임 검출 계수(

)를 산출한다. 제1 움직임 검출 계수(

)의 평균값(

)의 가중치는 제1 움직임 검출 계수(

)이고, 제1 움직임 검출 계수()의 최소값(

)의 가중치는 (1 - 제1 움직임 검출 계수(

))이다. The second coefficient calculator 434 may include the first motion detection coefficient (i.

Use the average value of). The second coefficient calculating unit 434 may determine the first motion detection coefficient (s) in the local window W of the current frame.

First motion detection coefficient

By the weighted sum of the average and minimum values of

). As shown in equation (4), in the local window W region, the first motion detection coefficient (

Mean value of

) And the first motion detection coefficient (

) Minimum value (

). And, the first motion detection coefficient (calculated by equation (3)

Taking the weighted sum of the second motion detection coefficient (

). First motion detection coefficient (

Mean value of

) Is a first motion detection coefficient (

) And the first motion detection coefficient ( ) Minimum value (

) Is a weight of (1-first motion detection coefficient (

))to be.

...(4)

...(4)

)은 하기 식 (5)에서 보는 바와 같이 현재 프레임의 국부적 윈도우(W) 내에서, 각 픽셀들의 제1 움직임 검출 계수(

)를 평균한 값이다. medium(

) Is the first motion detection coefficient of each pixel in the local window (W) of the current frame as shown in equation (5)

) Is the average value.

...(5)

... (5)

)은 하기 식 (6)에서 보는 바와 같이 현재 프레임의 국부적 윈도우(W) 내 픽셀들의 제1 움직임 검출 계수(

)들 중 최소 값이다. Minimum value (

Is the first motion detection coefficient of the pixels in the local window (W) of the current frame as shown in equation (6)

) Is the minimum value.

...(6)

... (6)

) 및 이전 프레임의 제3 움직임 검출 계수(

)를 기초로, 현재 프레임의 제3 움직임 검출 계수(

)를 산출한다. 하기 식 (7)에서 보는 바와 같이, 현재 프레임의 국부적 윈도우(W) 내에서, 제2 움직임 검출 계수(

)의 최대 값(

)을 구한다. 그리고, 하기 식 (8)과 같이 이전 프레임의 제3 움직임 검출 계수(

)와 상수 b의 차와 0 중에 큰 값을 구하고, 이를 식 (7)에서 구한 제2 움직임 검출 계수(

)의 최대 값(

)과 비교하고, 이 중 큰 값을 제3 움직임 검출 계수(

)로 취한다. The third coefficient calculator 454 uses the motion information of the neighboring pixels and the motion information of the corresponding pixel of the previous frame as temporal motion information to minimize blur in the motion region. The third coefficient calculator 454 is configured to generate a second motion detection coefficient in the local window W of the current frame.

) And the third motion detection coefficient of the previous frame (

Based on the third motion detection coefficient of the current frame (

). As shown in Equation (7) below, in the local window W of the current frame, the second motion detection coefficient (

Value of)

). Then, as shown in Equation (8), the third motion detection coefficient of the previous frame (

) And a large value among the difference between the constant b and zero, and the second motion detection coefficient (

) Maximum value (

), And the larger of these values is compared with the third motion detection coefficient (

Take).

...(7)

... (7)

...(8)

...(8)

)는 시간 필터(406)의 계수로서 이용될 수 있다. 제3 움직임 검출 계수(

)는 0에서 1 사이의 값을 가지며, 움직임이 많은 영역에서는 1에 가까운 값을 갖고 움직임이 적은 영역에서는 0에 가까운 값을 갖는다. The third motion detection coefficient finally obtained by equation (8)

) May be used as a coefficient of the time filter 406. Third motion detection coefficient (

) Has a value between 0 and 1, and has a value close to 1 in a high motion region and a value close to 0 in a low motion region.

)를 가중치로 하여, 현재 프레임과 이전 프레임의 가중 합에 의해 현재 프레임의 시간적 잡음을 제거한다. 하기 식 (9)와 같이, 현재 프레임의 가중치는 제3 움직임 검출 계수이고, 시간적 잡음 제거된 이전 프레임의 가중치는 (1 - 제3 움직임 검출 계수)이다. 시간 필터(406)는 시간적 잡음 제거된 현재 프레임(

)을 출력한다. The temporal filter 406 stores the third motion detection coefficient of the current frame (

) Is weighted to remove temporal noise of the current frame by the weighted sum of the current frame and the previous frame. As shown in Equation (9), the weight of the current frame is the third motion detection coefficient, and the weight of the temporal noise-removed previous frame is (1-third motion detection coefficient). The temporal filter 406 is a temporal noise canceled current frame (

)

...(9)

... (9)

은 메모리(50)에 저장되어 있던 시간적 잡음 제거된 이전 프레임이고,

는 현재 입력된 프레임이고,

는 시간적 잡음 제거된 현재 프레임이다. 움직임이 많은 경우, 제3 움직임 검출 계수(

)는 1에 가까운 값을 갖고, 움직임 블러를 최소화하기 위해 현재 프레임(

)에는 높은 가중치를 주고, 이전 프레임(

)에는 낮은 가중치를 준다. 반대로 움직임이 적은 경우, 제3 움직임 검출 계수(

)는 0에 가까운 값을 갖고, 잡음 제거 효과를 크게 하기 위해 상대적으로 잡음이 많은 현재 프레임(

)에는 낮은 가중치를 주고, 이미 잡음 제거가 수행되어 메모리(50)에 저장되어 있던 이전 프레임(

)에는 높은 가중치를 준다.here,

Is the previous temporal noise canceled frame stored in the memory 50,

Is the currently entered frame,

Is the current frame from which temporal noise has been removed. If there is a lot of motion, the third motion detection coefficient (

) Has a value close to 1, and the current frame (

) Gives a high weight, and the previous frame (

) Gives a lower weight. On the contrary, when there is little motion, the third motion detection coefficient (

) Has a value close to zero, and the current frame (which is relatively noisy)

) Is given a low weight and noise reduction has been performed so that the previous frame (

) Gives a high weight.

8 is a block diagram schematically illustrating a configuration of a spatial noise canceller according to an embodiment of the present invention. 10 to 12 are diagrams for describing the spatial noise cancellation of FIG. 8.

9 is a diagram illustrating weight calculation of a conventional adaptive weight averaging (AWA) algorithm.

Adaptive Weight Averaging (AWA), which is a conventional 2DNR algorithm for removing spatial noise, has a center pixel (x _c ) and a peripheral pixel (x _i ) within a predetermined window (S) as shown in FIG. 9 (a) and Equation (10). ) Can be expressed as the sum of the weights w _i , and the weights w _i is a threshold Th at which the difference value (x _i -x _c │) of the center pixel x _c and the surrounding pixel x _i is constant. If it is less than or equal to 1, the value is set to 1. If it is greater than or equal to the threshold, it is set to 0.

...(10)

... (10)

Referring to FIG. 9 (b), in the case of an edge image having no noise, weights of AWA are relatively accurately obtained. However, when the noise is severe as shown in FIG. 9 (c), it is very difficult to determine the edge region. Therefore, the distribution of weights does not indicate the pattern of the pattern to which the region belongs, but only the random noise distribution. Therefore, the AWA does not properly remove the noise that is not removed from the 3DNR motion region, and the noise remains in the motion region or the edge region even after the AWA is performed.

The spatial noise removing unit 60 of the present invention for solving the above problems is to limit the shape of the pattern to four main directions of horizontal, vertical, diagonal, and reverse diagonal lines, and remove the spatial noise only in the limited direction (2DNR). It is a directional average filter that performs.

Referring to FIG. 8, the spatial noise remover 60 may include a direction difference unit 602, a minimum value detector 604, and an average unit 606.

)의 국부적 윈도우(W) 내에서, 중심 픽셀을 기준으로 설정된 복수의 방향마다 중심 픽셀에 대한 차분 값의 평균을 산출한다. 여기서 국부적 윈도우(W)는 시간적 잡음 제거부에서의 국부적 윈도우와 상이할 수 있으며, 본 실시예에서는 5x5 윈도우를 예로서 설명하겠다. The direction difference unit 602 is a current frame from which temporal noise has been removed.

In the local window (W) of), an average of difference values for the center pixel is calculated for each of a plurality of directions set based on the center pixel. Here, the local window W may be different from the local window in the temporal noise canceller, and in this embodiment, a 5x5 window will be described as an example.

As in the following equation (11), four difference values are obtained for each of the four main directions. Here, l is an index indicating a direction, and S _l is a set of peripheral pixels excluding the center pixel x _c in the l direction.

...(11)

... (11)

Referring to FIG. 10, the first direction is set in the vertical direction, the second direction is set in the diagonal direction, the third direction is set in the horizontal direction, and the fourth direction is set in the reverse diagonal direction. In the 5x5 window, the peripheral pixel set in the first direction is S ₁ , the peripheral pixel set in the second direction is S ₂ , the peripheral pixel set in the third direction is S ₃ , and the peripheral pixel set in the fourth direction is S ₄ . The mean difference value D ₁ , which is the average of the difference between the center pixel x _c and the surrounding pixels of the 5 × 5 window in each direction, is obtained.

The minimum value detector 604 detects the direction in which the average difference value D _l is minimum. As shown in the following equation (12), the smallest value D _min is detected among the average difference values D1, D2, D3, and D4 obtained in four directions. FIG. 11 shows that the average difference value D1 in the vertical direction is obtained as the smallest value. This indicates that the area of FIG. 10, that is, the local area belonging to the center pixel x _c and its surrounding 5x5 window, is likely to be a vertical edge.

...(12)

... (12)

는 중심 픽셀(x_c)과 주변 픽셀 집합(S_l)에 포함된 주변 픽셀들의 집합이고, n(

)는 이 집합의 픽셀 개수이다. 도 12는 세로 방향의 영역에 속하는 픽셀들에 대해서만 평균 값을 구하는 연산을 보여주고 있다. The averaging unit 606 removes spatial noise of the current frame by averaging pixel values of pixels in a direction in which the average difference value is minimum. At this time, the pixel values of both the center pixel and the surrounding pixels are averaged. In the following formula (13),

Is a set of peripheral pixels included in the center pixel x _c and the peripheral pixel set S _l , and n (

) Is the number of pixels in this set. 12 illustrates an operation of obtaining an average value only for pixels belonging to a vertical area.

...(13)

... (13)

13 is a flowchart schematically illustrating a method of removing noise of a video image according to an embodiment of the present invention. 14 is a flowchart schematically illustrating the filtering method of FIG. 13. FIG. 15 is a flowchart schematically illustrating a method of removing temporal noise of FIG. 13. FIG. 16 is a flowchart schematically illustrating a method of removing spatial noise of FIG. 13. Hereinafter, detailed descriptions of contents overlapping with those described with reference to FIGS. 1 to 12 will be omitted.

Referring to FIG. 13, the current frame and the previous frame are filtered to remove the primary noise (S100). Here, the previous frame is a temporal noise canceled frame. The current frame and the previous frame are filtered by an NxM mask, where N and M are natural numbers, which vary in size in proportion to the noise level. In this case, the noise level may be preset or a value input by the user. Alternatively, the noise level may vary with sensor gain. It is assumed that the noise level of the current frame and the previous frame is the same.

Referring to FIG. 14 together, a noise level is predicted from the noise parameter (S110). For example, the noise level can be predicted from the sensor gain value. The sensor gain may be provided as information on the imaging device from the control unit of the image processing system. A mask that is matched according to the predicted noise level is selected (S130). The mask may be preset for each noise level. The current frame and the previous frame are respectively filtered by the selected mask (S150). Prior to temporal and spatial noise removal, the current frame and the previous frame are first filtered to remove noise, thereby reducing errors in motion detection. In addition, the motion blur can be optimized by varying the mask size according to the noise.

Hereinafter, temporal noise cancellation and spatial noise cancellation for the filtered current frame and the previous frame will be described.

First, temporal noise of the current frame is removed using the motion information detected from the current frame and the previous frame (S200).

Referring to FIG. 15, noise is removed from the difference between the current frame and the previous frame from which temporal noise has been removed (S210). In detail, in the local window, an average difference value between corresponding pixels of the current frame and the previous frame is calculated (S220). The offset value set according to the noise level of the current frame is subtracted from the average difference value, and a second average difference value is calculated (S230). The offset value may be preset or input by a user.

Motion information of the current frame is detected based on the noise-removed difference value, and temporal noise of the current frame is removed according to the motion information (S240). The motion information of the current frame includes a motion detection coefficient indicating the degree of motion of each pixel. Specifically, the motion detection coefficient is calculated by the following steps.

First, a first motion detection coefficient of each pixel is calculated based on a constant and a second average difference value set according to an offset value (S250). The first motion detection coefficient of each pixel is calculated by scaling the second average difference value by a constant set according to the offset value to have a value between 0 and 1.

Next, in the local window of the current frame, the second motion detection coefficient is calculated by weighted sum of the average value and the minimum value of the first motion detection coefficient (S260). The weight of the average value of the first motion detection coefficients is the first motion detection coefficient, and the weight of the minimum value of the first motion detection coefficient is (1-first motion detection coefficient).

Next, in the local window of the current frame, the third motion detection coefficient of the current frame is calculated based on the second motion detection coefficient and the third motion detection coefficient of the previous frame (S270). In order to minimize motion blur, the maximum value of the second motion detection coefficient is taken in units of blocks, i.e., local window, and the difference between the third motion detection coefficient and the constant of the previous frame and the constant and the greater of zero and the second motion is obtained. The larger of the maximum values of the detection coefficients is taken as the third motion detection coefficient. The third motion detection coefficient has a value between 0 and 1, has a value close to 1 when there is a lot of motion, and has a value close to 0 when there is little motion.

The temporal noise of the current frame is removed by the weighted sum of the current frame and the temporal noise-removed previous frame using the third motion detection coefficient (S280). The weight of the current frame is the third motion detection coefficient, and the weight of the temporal noise canceled previous frame is (1-third motion detection coefficient).

Referring back to FIG. 13, spatial noise is removed based on an edge direction in the current frame from which temporal noise is removed (S300). In detail, referring to FIG. 16, in a local window of the current frame from which temporal noise has been removed, an average of difference values with respect to the center pixel is calculated in a plurality of directions set based on the center pixel (S310). Here, the plurality of directions may include horizontal, vertical, diagonal, and reverse diagonal directions. Then, the direction which is the smallest among the average difference values in each direction is detected (S330). Next, pixel values of the pixels in the detected direction are averaged (S350).

Conventional video noise reduction schemes adjust the noise reduction intensity by simply changing the slope value of the motion detection coefficient during temporal noise reduction, whereas in the embodiment of the present invention, an offset is applied to the motion detection coefficient and the motion detection coefficient is applied. By using the average value of, the spatial motion information by the motion information of the surrounding pixels and the temporal motion information by the motion information of the previous frame, the noise removal effect is increased while minimizing the motion blur due to the noise removal.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation and that those skilled in the art will recognize that various modifications and equivalent arrangements may be made therein. It will be possible. Therefore, the true scope of protection of the present invention should be defined only by the appended claims.

Claims (20)

Removing noise from the difference between the current frame and the temporal noise canceled previous frame; And
Detecting motion information of the current frame based on the noise-reduced difference, and removing temporal noise of the current frame according to the motion information. The method of claim 1, wherein the step of removing the noise of the difference,
In a local window, averaging a difference value between corresponding pixels of the current frame and the previous frame; And
And calculating a second average difference value by subtracting an offset value set according to a noise level from the average difference value. The method of claim 2, wherein the temporal noise removing step comprises:
Calculating a first motion detection coefficient for the center pixel of the local window of the current frame based on the constant set according to the offset value and the second average difference value;
Calculating a second motion detection coefficient by a weighted sum of an average value and a minimum value of the first motion detection coefficients within the local window of the current frame; And
Calculating, within a local window of the current frame, a third motion detection coefficient of the current frame based on the second motion detection coefficient and the third motion detection coefficient of the previous frame. Noise Reduction Method. The method of claim 3, wherein the temporal noise removing step comprises:
And removing temporal noise of the current frame by the weighted sum of the current frame and the temporal noise-removed previous frame using the third motion detection coefficient. Calculating, within a local window of the current frame from which temporal noise is removed, an average of difference values for the center pixel in a plurality of directions set with respect to the center pixel;
Detecting a direction in which the average difference value is minimum; And
And averaging pixel values of the pixels in the detected direction. The method of claim 5,
And the plurality of directions include horizontal, vertical, diagonal, and anti-diagonal directions. Removing noise from the difference between the current frame and the temporal noise canceled previous frame;
Detecting motion information of the current frame based on the noise canceled difference;
Removing temporal noise of the current frame by a weighted sum of the current frame and the temporal noise canceled previous frame according to the motion information; And
And removing spatial noise based on an edge direction in the current frame from which the temporal noise has been removed. The method of claim 7, wherein
The noise removing step may include: calculating a second average difference value by subtracting an offset value set according to a noise level from an average difference value between corresponding pixels of the current frame and the previous frame within a local window; Including,
The motion information detection step,
Calculating a first motion detection coefficient for the center pixel of the local window of the current frame based on the constant set according to the offset value and the second average difference value;
Calculating a second motion detection coefficient by a weighted sum of an average value and a minimum value of the first motion detection coefficients within the local window of the current frame; And
Calculating a third motion detection coefficient of the current frame based on the second motion detection coefficient and the third motion detection coefficient of the previous frame within the local window of the current frame. How to remove. The method of claim 7, wherein the spatial noise removing step,
Calculating an average of a difference value for the center pixel in a plurality of directions set based on the center pixel in the second local window of the current frame from which the temporal noise is removed;
Detecting a direction in which the average difference value is minimum; And
Averaging pixel values of the pixels in the detected direction. The method of claim 7, wherein before the differential image noise removing step:
And filtering the current frame and the temporal noise-removed previous frame by applying a mask having a variable size according to a noise level. A differential to remove noise from the difference between the current frame and the previous frame from which temporal noise has been removed; and
And a coefficient calculator configured to detect motion information of the current frame based on the noise canceled difference to remove temporal noise of the current frame. The method of claim 11, wherein the difference portion,
A first difference part which averages a difference value between corresponding pixels of the current frame and the previous frame within a local window; And
And a second difference unit configured to calculate a second average difference value by subtracting an offset value set according to a noise level from the average difference value. 2. The method of claim 12, wherein the coefficient calculation unit,
A first coefficient calculating unit configured to calculate a first motion detection coefficient for the center pixel of the local window of the current frame based on the constant set according to the offset value and the second average difference value;
A second coefficient calculating unit for calculating a second motion detection coefficient by a weighted sum of the average value and the minimum value of the first motion detection coefficient within the local window of the current frame; And
And a third coefficient calculator configured to calculate a third motion detection coefficient of the current frame based on the second motion detection coefficient and the third motion detection coefficient of the previous frame within the local window of the current frame. Temporal noise reduction device for video images. The method of claim 13,
A temporal noise of the video image, further comprising: a noise canceller configured to remove temporal noise of the current frame by a weighted sum of the current frame and a temporal noise-removed previous frame by using a third motion detection coefficient of the current frame Removal device. A direction difference unit for calculating an average of difference values for the center pixel in a plurality of directions set based on the center pixel in the second local window of the current frame from which temporal noise is removed;
A minimum value detector for detecting a direction in which the average difference value is minimum; And
And a noise remover configured to remove the spatial noise of the current frame by averaging pixel values of the pixels in the detected direction. 16. The method of claim 15,
And the plurality of directions include horizontal, vertical, diagonal, and anti-diagonal directions. Noise is removed from the difference between the current frame and the temporal noise canceled previous frame, and the weighted sum of the current frame and the temporal noise canceled previous frame according to the motion information of the current frame calculated based on the noise canceled difference. A temporal noise canceller which removes temporal noise of the current frame by means of a temporal noise canceller; And
And a spatial noise remover configured to remove spatial noise based on an edge direction in the current frame from which the temporal noise is removed. The method of claim 17, wherein the temporal noise canceller,
An image difference unit configured to calculate a second average difference value by subtracting an offset value set according to a noise level from an average difference value between corresponding pixels of the current frame and the previous frame within a local window;
A first coefficient calculating unit configured to calculate a first motion detection coefficient for the center pixel of the local window of the current frame based on the constant set according to the offset value and the second average difference value;
A second coefficient calculating unit for calculating a second motion detection coefficient by a weighted sum of the average value and the minimum value of the first motion detection coefficient within the local window of the current frame; And
And a third coefficient calculator configured to calculate a third motion detection coefficient of the current frame based on the second motion detection coefficient and the third motion detection coefficient of the previous frame within the local window of the current frame. Noise reduction device for video images. The method of claim 17, wherein the spatial noise removing unit,
In the second local window of the current frame from which the temporal noise is removed, an average of a difference value for the center pixel is calculated for each of a plurality of directions set based on the center pixel, and the pixels of the pixels in the direction in which the average difference value is minimum are calculated. Noise reduction device for video images that averages values. 18. The method of claim 17,
And a variable filter for filtering the current frame and the temporal noise-removed previous frame by applying a mask having a variable size according to a noise level before the difference between the current frame and the temporal noise-removed previous frame. Noise reduction device.

Images