KR102049644B1 - Hierarchical Noise Reduction Method for Efficient Coding of Video with Noise - Google Patents

Abstract

As a method to improve the compression efficiency of noise video, the noise removal order of frames can be determined according to the GOP layer and slice type so that the complexity of noise removal can be flexibly adjusted and the delay caused by noise removal is rarely provided. do. An image encoding method according to an embodiment of the present invention includes: identifying a GOP layer of frames; Removing noise for at least one of the frames with reference to the identified GOP layer; And encoding the frames.
As a result, the noise removal order of the frames is determined according to the GOP layer and the slice type, so that the complexity of the noise removal can be flexibly adjusted and the delay due to the noise removal is hardly generated.

Description

Hierarchical Noise Reduction Method for Efficient Coding of Video with Noise

The present invention relates to a method for removing noise, and more particularly, to a method for increasing video compression efficiency by performing noise removal on a video by installing the video encoder.

Noise is generated by acquiring and editing a video, and most video acquired by a camera includes noise. However, its random nature reduces the accuracy of the inter-frame prediction used in video compression. This greatly reduces the coding efficiency of the video, and there have been many studies to increase the coding efficiency by removing the noise of the video.

The first method is to remove noise with the preprocessor before encoding. This method can greatly improve the compression efficiency by encoding the noise-free image. However, there is a risk of damaging the original video signal when removing noise. It cannot be used in applications where noise must be preserved as an important feature of the image. Film grain noise, for example, must be preserved as an important feature in making motion pictures look like film.

In the second method, the preprocessor first removes the noise, models the removed noise, and sends the noise parameter to the decoder. The decoder synthesizes the noise with the noise parameter and generates the final output image in addition to the compressed video from which the noise is removed. In this method, high noise efficiency can be obtained by encoding noise-free video, and noise texture similar to the original video can be obtained through synthesized noise. However, the synthesized noise has a problem that does not match the original noise. In the case of a medical image, such a method cannot be applied because noise cannot be synthesized.

The third method is to efficiently compress the original of the noise video. This method focuses on the fact that noise interferes with motion prediction and the compression efficiency is lowered. Therefore, the noise is removed from only the reference image for motion prediction inside the encoder. The noise-reduced reference picture is used only for motion prediction and motion compensation to prevent the original picture from being damaged by noise removal. To match the encoder with the decoder, the decoder also includes the same blocks as the noise canceller in the encoder. This method improves the compression efficiency even though the original contains noise. However, the decoder as well as the encoder requires a noise canceller, so there is a problem in that the decoder needs to be modified. In addition, the decoder's noise canceller has the disadvantage of increasing the decoder complexity by up to 8 times compared to the existing decoder complexity. Excessive increase in decoder complexity is difficult to apply in practical applications.

Existing methods sequentially remove noise in order of video display to remove video noise. That is, after performing the n-th frame, noise removal is sequentially performed on the n + 1 th frame and the n + 2 th frame. Therefore, a delay occurs because noise cancellation must be performed in advance until the next P frame is generated by the GOP structure of the codec. In addition, since the noise removal must be performed for the entire frame, there is a limit to controlling the complexity of the noise removal.

The present invention has been made to solve the above problems, an object of the present invention is to improve the compression efficiency of the noise video, the noise removal order of the frame is determined according to the GOP layer and slice type, noise removal It is possible to flexibly control the complexity of and to provide a method in which the delay caused by the noise removal is hardly generated.

According to an embodiment of the present invention, a video encoding method includes: identifying a GOP layer of frames; Removing noise for at least one of the frames with reference to the identified GOP layer; And encoding the frames.

The noise removing step may be to determine a noise removing order of frames to remove noise by referring to the identified GOP layer.

The encoding step may be to determine the encoding order of the frames based on the GOP layer.

The video encoding method according to the present invention further includes determining a slice type of frames constituting the GOP, and the noise removing step further removes noise for at least one of the frames by further referring to the identified slice type. It may be.

The noise removal step may be to determine the noise removal order of the frames from which the noise is to be removed by further referring to the identified slice type.

The noise removing step may be to adaptively determine frames to remove noise based on the GOP state.

The GOP state may include noise variance of the frames constituting the GOP.

The number of GOP layers to remove noise may be proportional to the noise variance of the GOP.

The number of slice types to remove noise may be proportional to the noise variance of the GOP.

On the other hand, according to another embodiment of the present invention, an image encoder includes an input unit for receiving frames; And a processor for identifying a GOP layer of frames input through the input unit, removing noise for at least one of the frames with reference to the identified GOP layer, and encoding the frames.

On the other hand, according to another embodiment of the present invention, the method for removing noise includes identifying a GOP layer of frames; And removing noise for at least one of the frames with reference to the identified GOP layer.

On the other hand, according to another embodiment of the present invention, an apparatus for removing noise includes an input unit for receiving frames; And a processor for identifying a GOP layer of the frames input through the input unit and removing noise with respect to at least one of the frames with reference to the identified GOP layer.

As described above, according to the embodiments of the present invention, the noise removal order of the frames may be determined according to the GOP layer and the slice type, so that the complexity of the noise removal may be flexibly adjusted, and the delay due to the noise removal may be substantially reduced. Does not occur.

In addition, according to embodiments of the present invention, it is possible to flexibly adjust the intensity of noise removal, to improve the compression efficiency of the original noise image without decoder modification, and to reduce the fluctuation of image quality between frames when only the lower layer is removed. It becomes possible.

1 is a diagram illustrating a layer and encoding order between slices in a GOP;
2 is a general encoding method,
3 is an encoding method according to an embodiment of the present invention, and
4 is a block diagram illustrating a hardware structure of an encoder according to another embodiment of the present invention.

Hereinafter, with reference to the drawings will be described the present invention in more detail.

In an embodiment of the present invention, a hierarchical noise removal method for efficient encoding of a noise video is provided. It is a technique to increase the compression efficiency of noise video and can remove noise with low delay and low complexity.

To this end, in the embodiment of the present invention, by adaptively determining the noise removal order of the frame according to the GOP layer and slice type, it is possible to flexibly adjust the complexity of the noise removal and hardly cause the delay due to the noise removal.

In the embodiment of the present invention, only the encoder is modified without modification to the decoder, so that the compression efficiency of the noise video is improved.

In general, a video codec divides and encodes a video sequence into groups of pictures (GOPs). One GOP can be viewed as a set of frames, and each frame is encoded into various slice types. Representative slice types include I (Intra), P (Predictive), and B (Bi-Predictive) slices.

The I slice may use only intra prediction and is encoded with only information of the current frame. P slices are encoded using intra prediction and unidirectional prediction that references past I or P slices. B slices can use intra prediction and unidirectional prediction as well as bidirectional prediction that can simultaneously refer to information of past and future frames. 1 shows the hierarchy and encoding order between slices in a GOP.

In X _n of FIG. 1, X denotes a slice type and n denotes an encoding order. The arrow refers to the frame. For example, a B ₂ picture performs encoding by referring to an I ₀ and P ₁ picture.

As shown in FIG. 1, a frame in an upper layer performs encoding by referring to a frame in a lower layer. As a result, the frame in the lowest layer affects all the frames in the upper layer. Therefore, the higher the image quality, the better the compression efficiency of the frame in which the layer exists at a lower position. For this reason, a general encoder tends to encode a smaller quantization parameter as a frame located in a lower layer. This is because the smaller the quantization parameter is, the higher the quality is encoded.

Conventional methods remove video noise in the order in which they are displayed. Therefore, noise removal must be performed for every frame.

On the other hand, in the embodiment of the present invention, the noise removal order is determined according to the slice type and the layer of the current frame. That is, as described above, since the frames of the lower layer are used as important reference frames, the noise removal is performed from the frame belonging to the lower layer.

In the example of FIG. 1, noise removal of an I ₀ and P ₁ picture is performed first, and then noise removal is performed on a B ₂ picture. Next, noise reduction is performed on pictures B ₃ and B ₆ , and finally noise removal is performed on B ₄ , B ₅ , B ₇ , and B ₈ . This has the following advantages.

1) Delay due to video noise removal can be reduced. Existing methods of removing noise in frames in display order must first remove noise from the I ₀ to B ₈ pictures, and then remove the noise from the P ₁ picture. Therefore, in order to encode P ₁ , which is encoded second in the GOP, all 9 frames of noise must be removed. As a result, the conventional method causes a delay due to noise removal. On the other hand, in the embodiment of the present invention, very little delay occurs by removing the noise of P ₁ immediately after I ₀ .

2) The complexity of noise removal can be adjusted. Since the embodiment of the present invention has a hierarchical structure, it can operate without performing noise removal for all layers. That is, noise removal is performed only up to a certain layer according to the complexity required. For example, only the 0th layer can perform noise reduction. In this case, in FIG. 1, noise is removed only for the I ₀ and P ₁ pictures. Assuming that noise is removed only up to the first layer, noise is removed only up to the I ₀ , P ₁ , and B ₂ pictures. Therefore, the noise removal complexity can be flexibly adjusted by performing noise removal only 1/8 or 1/4 of the entire frame. Even if only one or two frames of noise are removed from the GOP, compression efficiency is improved by improving the most important reference picture first.

3) The intensity of noise reduction can be adjusted. Conventional methods adjust the noise removal intensity by noise dispersion or external input regardless of the slice type. However, in the embodiment of the present invention, it is possible to determine whether or not to remove the noise according to the slice type and the hierarchy, thereby more flexibly adjusting the noise removing intensity.

4) Compression efficiency can be improved. When the present method is used, noise of the reference frame is removed to enable accurate motion prediction, which may improve compression efficiency. In particular, the compression efficiency of the noise original image can be improved. This can be achieved by removing noise only at the lower layer and not performing noise removal at the upper layer. The encoder performs encoding by inputting a frame of a lower layer from which noise is removed and an original frame of a higher layer having noise. The frame from which the noise is removed increases compression efficiency, but the removed noise tends to have a low objective image quality, which is a difference between the encoded image and the original image due to the removed noise. On the other hand, since the frames of the upper layer are encoded using the reference image from which the noise is removed, the compression efficiency is greatly improved. Therefore, the overall compression efficiency of the original image is improved.

5) When only the lower layer removes the noise, it is possible to reduce the quality fluctuation between frames. As mentioned earlier, the lower layer frames use smaller quantization parameters. Therefore, an objective image quality difference occurs between the lower layer and the upper layer. If only the lower layer is removed, the objective image quality of the lower layer is decreased while the objective image quality of the upper layer is increased, thereby reducing the overall fluctuation of image quality between frames.

Hereinafter, an encoding method according to an embodiment of the present invention will be described in detail.

General encoding is performed in the structure as shown in FIG. First, noise reduction is performed on the input video. After receiving the noise-removed video as an input, the GOP structure or slice type is determined, and encoding is performed according to the determined slice type.

However, in the encoding method according to the embodiment of the present invention, the GOP structure is first determined as shown in FIG. 3 (S110), and then the noise removal order and the presence or absence of noise removal are determined according to the GOP layer and slice type of the current frame (S120). . The video from which the noise is removed and the original video are encoded (S130).

The GOP structure may be predetermined or may be adaptively selected. Assuming that the GOP structure is determined as shown in FIG. 1, the noise remover removes noise according to the layer and slice type of the current frame. For example, if it is determined that the video encoder removes noise only for P frames in the 0th frame layer, noise removal is performed only on the corresponding frame, and noise removal is not performed on the remaining frames.

The slice type and layer from which noise is removed between the GOPs is not always fixed. That is, one video sequence is composed of several GOPs, and the layer and slice type to which noise reduction is applied to each GOP may be different.

For example, if the noise variance of the i-th GOP is σ ² _gop (i), the noise cancellation may be applied to the GOP as follows. NR _layer (i) indicates to which _layer the i-th GOP performs noise removal. S _type (i) indicates a slice type for removing noise in the i-th GOP. If S _type (i) is not specified, noise reduction is performed for all slice types. In the equation below, the more noise in the GOP, the more noise is removed to the upper layer. Adaptive coverage is not limited to this method. T _X is an appropriate threshold, and the larger X, the higher the value.

NR _layer (i) = 0 and S _type (i) = I Slice if σ ² _gop (i) ≤ T ₁

NR _layer (i) = 0 if σ ² _gop (i) ≤ T ₂

NR _layer (i) = 1 if σ ² _gop (i) ≤ T ₃

NR _layer (i) = 2 if σ ² _gop (i) ≤ T ₄

NR _layer (i) = 3 if σ ² _gop (i) ≤ T ₅

When the frame to be removed is determined, noise removal is applied. Noise reduction is performed in two steps. The first step is a noise variance estimation step and the second step is a noise removal step based on the estimated noise variance.

Noise variance is an indicator of how much noise the current frame has. Therefore, accurate estimation of noise variance is very important in order to remove noise without losing detail of the image. If the noise of the current frame is estimated, noise removal is performed based on that.

There are various methods of removing noise using spatial and temporal features, and embodiments of the present invention do not depend on a specific noise removing method.

Finally, the encoding is performed on the noise canceled frame or the non-removed frame. For example, in the above example, if the NR _layer (i) is determined to be ₁ , the noise-free frames are encoded for the pictures I ₀ , P ₁ , and B ₂ of FIG. 1, and the original frames are encoded for the B ₃ to B ₈ pictures. Although the B ₃ -B ₈ picture is encoded using the noise-reduced reference frame, although the noise is not removed, the compression efficiency for the B ₃ -B ₈ picture coding is increased.

4 is a block diagram illustrating a hardware structure of an encoder according to another embodiment of the present invention. As shown in FIG. 4, the encoder according to the embodiment of the present invention includes an input unit 210, a processor 220, an output unit 230, and a storage unit 240.

The input unit 210 is a means for receiving image data from an external memory, and the processor 220 performs encoding on the input image according to the method shown in FIG. 3.

The output unit 230 outputs an image encoded by the processor 220 to an external memory. The storage unit 240 is an internal memory that provides storage space necessary for the processor 220 to perform encoding.

Up to now, the hierarchical noise removal method for efficient encoding of a noise video has been described in detail with reference to a preferred embodiment.

The prior art has changed the encoder and decoder to increase the compression efficiency of the original noise image. In addition, there is a limit in controlling the complexity and intensity of noise removal by sequentially performing noise removal on all frames. However, in the embodiment of the present invention, the noise reduction may be adaptively performed according to the GOP layer and the slice type, thereby improving compression efficiency with low latency and low complexity.

The technical idea of the present invention is not limited to the above-described video encoding method and apparatus. The technical idea of the present invention may be applied to a method and apparatus for adaptively removing noise from frames based on a GOP layer and a slice type of frames for other purposes than image encoding.

On the other hand, the technical idea of the present invention can be applied to a computer-readable recording medium containing a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present disclosure may be implemented in the form of computer readable codes recorded on a computer readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. In addition, the computer-readable code or program stored in the computer-readable recording medium may be transmitted through a network connected between the computers.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the specific embodiments described above, but the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

210: input unit
220: processor
230: output unit
240: storage unit

Claims (12)

Identifying a GOP layer of frames;
Removing noise for at least one of the frames with reference to the identified GOP layer; And
Encoding frames;
The noise removal step is
Based on the GOP state, adaptively determine the frames to remove noise,
GOP status is
And a noise variance of the frames constituting the GOP.
The method according to claim 1,
The noise removal step is
The method of encoding an image, characterized in that to determine a noise removal order of frames to remove noise with reference to the identified GOP layer.
The method according to claim 2,
The encoding step is
And a method of encoding the frames based on the GOP layer.
The method according to claim 1,
Determining a slice type of frames constituting the GOP;
The noise removal step is
Further referring to the identified slice type to remove noise for at least one of the frames.
The method according to claim 4,
The noise removal step is
And referring to the identified slice type, to determine a noise removing order of frames to remove noise.

delete delete The method according to claim 1,
The number of GOP layers to remove noise is
A video encoding method characterized by being proportional to the noise variance of a GOP.
The method according to claim 1,
The number of slice types to remove noise is
A video encoding method characterized by being proportional to the noise variance of a GOP.
An input unit for receiving frames;
A processor for identifying a GOP layer of frames input through the input unit, removing noise for at least one of the frames with reference to the identified GOP layer, and encoding the frames;
The processor,
Based on the GOP state, adaptively determine the frames to remove noise,
GOP status is
And a noise variance of the frames constituting the GOP.
Identifying a GOP layer of frames; And
And removing noise for at least one of the frames with reference to the identified GOP layer.
The noise removal step is
Based on the GOP state, adaptively determine the frames to remove noise,
GOP status is
And noise variance of the frames constituting the GOP.
An input unit for receiving frames;
And a processor for identifying a GOP layer of frames input through the input unit and removing noise for at least one of the frames with reference to the identified GOP layer.
The processor,
Based on the GOP state, adaptively determine the frames to remove noise,
GOP status is
And noise dispersion of the frames constituting the GOP.

Images