KR20120129629A - Method of motion estimation and compensation using in-loop preprocessing filtering - Google Patents

Abstract

PURPOSE: A motion estimation and compensation method is provided to compensate blur and deblur phenomenon by adding a preprocessing process inside a video compression encoder and a decoder. CONSTITUTION: A video encoder estimates variation degree of image information between a current frame and a reference frame(S110). The video encoder configures a pre-processing filter group including an LPF(Low Pass Filter) group and an adaptive filter group(S120). The video encoder applies the pre-processing filter group to the reference frame(S130). The video encoder performs motion estimation and compensation by using the pre-processed reference frame(S140). [Reference numerals] (AA) Start; (BB) End; (S110) Estimating variation degree of image information between a current frame and a reference frame; (S120) Configuring a pre-processing filter group; (S130) Applying the pre-processing filter group to the reference frame; (S140) Motion estimation and compensation

Description

Method of motion estimation and compensation using in-loop preprocessing filtering

The present invention relates to video encoding and decoding, and more particularly, to estimation of motion estimation and motion compensation by using preprocessing filtering in a loop in video encoding and decoding, thereby improving the efficiency of video compression. And a compensation method.

Starting with telephones and facsimiles, information media dealing with modern DMB, mobile phones, and Internet communication networks are becoming more and more large. Accordingly, interest and demand for high-speed multimedia data communication are increasing, and as the transmission speed of wired / wireless data transmission and reception is greatly improved, even video data having a large data size can be transmitted and received in real time through a wired / wireless communication network. It became possible.

Recently, with the advent of smart phones and smart TVs, the use of video dating through wired / wireless communication networks is exploding. Video data has a higher information transfer capability than plain text data, but has a large capacity, making it difficult to transmit, play, and store data in a network channel having limited bandwidth. In addition, since a large amount of moving picture information must be appropriately processed according to an application request, a system for processing moving pictures also requires high specifications.

In order to solve these problems, video encoding algorithms, which are techniques for compressing video data into small information, have been actively studied. Representative international standards related to video compression studied to date include ISO / IEC's MPEG series and ITU-T's H.26x series.

The video data is characterized by having temporal, spatial, and statistical redundancy. Temporal redundancy means redundancy between successive frames, and pixel values of successive frames have a very high correlation. Spatial redundancy means redundancy existing in a frame, and the brightness value of one pixel has a high correlation with the brightness value of neighboring pixels. Finally, statistical redundancy means redundancy between coded data, which means redundancy by probability distribution of brightness values of pixels.

For video encoding, by removing the three redundancies it is possible to compress a large amount of video data into a smaller amount of data. For example, motion estimation and compensation techniques remove temporal redundancy, and transform coding and quantization techniques remove spatial redundancy. Entropy coding techniques, such as variable length coding, are adopted and used in international standards for video codecs for the purpose of removing statistical redundancy.

Among the international standards organizations, the MPEG standard has three types of prediction: intra coded frame (I frame), predictive coded frame (P frame), and bi-directionally-predicitve coded frame (B frame). A coding scheme is used, and P frames and B frames achieve high compression efficiency by using motion estimation and compensation techniques.

An I frame is encoded using neighboring pixel values within the screen without reference to other frames. P frames are encoded with reference to past I frames or P frames. Motion compensation is performed using the predicted motion vector and the predictive block by estimating the motion information in units of blocks in the past I frame or P frame, and the temporal redundancy by encoding the differential signal and the differential motion vector of the block after motion compensation. Can achieve high compression ratio.

The B frame performs prediction by referring not only to past I frames and P frames, but also to future I frames and P frames. The B frame uses the motion estimation and compensation technique like the P frame. However, the compression rate is the highest because two reference frames are used and a better prediction performance is selected among them. However, the B frame does not become a reference frame for other frames. These I and P pictures are called reference frames.

Motion estimation and compensation methods for removing temporal redundancy can be broadly classified into a pixel recursive algorithm (PRA) and a block matching algorithm (BMA). Among these, most video codec technologies use block matching algorithms in consideration of the regularity of data flow, computational complexity, and hardware implementation. The block matching algorithm divides a frame into an arbitrary M × N sized block or a smaller sized Searching Block, and has a minimum error value within a predetermined search area of the reference frame in units of blocks. Refers to a procedure for finding a matching block. The value indicating the position of the matching block selected according to the block matching algorithm is called a motion vector (MV). In video encoding, only the difference value and the motion vector of the matching block and the current block are encoded to remove redundancy of data.

The block matching algorithm generally searches for a block that minimizes a matching error in order to search for a matching block. The matching error includes, for example, the absolute difference between all pixels constituting the search block of the current block and the reference frame. Sum of Absolute Difference (SAD) is used. The SAD value used for the BMA for the MxN size block is represented by Equation 1 below.

[Equation 1]

Here, F _n means the nth frame, F _n means the current frame and F _{n -1} means the previous frame adjacent to the current frame. (x, y) represents the coordinates of the pixel at the upper left corner of the block, and i and j represent the horizontal and vertical indexes of the pixels included in the block, respectively. mv _x and mv _y represent a horizontal motion vector and a vertical motion vector, respectively. SAD is a value obtained by adding up the absolute difference values of pixels at the same position in two blocks. As the SAD value is smaller, the two blocks are determined to be similar, and a block having the smallest SAD value in the search area is defined as a matching block.

Since images record three-dimensional information in a two-dimensional plane, it is difficult to estimate the same motion as in reality, and since the high computational amount is required to estimate the rotation, enlargement, and reduced movement, the video coding standard assumes parallel movement. Use motion estimation and compensation techniques. As methods for improving motion estimation and compensation efficiency, there are a spatial sampling compensation method and a temporal sampling compensation method. Images are spatially sampled in integer units, and subpixel motion estimation techniques have been proposed to compensate for information loss due to spatial sampling. In MPEG-2, motion estimation is performed by doubling the reference frame twice. In H.264 / AVC, motion estimation is performed by interpolating the reference frame four times. In addition, the conventional motion estimation technique shows a high prediction efficiency when estimating the motion of a static background or a slowly moving object, but the prediction efficiency is greatly decreased when the image changes rapidly. For example, blur or de-blur occurs when an object moves quickly or the camera's focus changes, which greatly increases the SAD value. Therefore, a motion estimation technique has been proposed to apply a preprocessing filter to a reference frame to compensate for the change and loss of image data due to temporal sampling. Algorithms for compensating temporal sampling processes include a blur compensation algorithm using a plurality of low pass filters and a focus change compensation algorithm using an adaptive filter. The algorithms apply a preprocessing filter to a reference frame to compensate for degradation between the current frame and the reference frame, and then perform motion estimation on the reference frame to which the preprocessing filter is applied.

Among the algorithms for compensating temporal sampling methods, many low-pass filters are used to compensate for the blur, and the motion estimation method is effective only for global blur because only one preprocessing filter is specified in one frame. When blur occurs locally in a real image or the image becomes clear, a method of improving motion prediction efficiency by compensating for image quality degradation with a low pass filter cannot be used.

Algorithms using adaptive filters can compensate for local blur and deblur respectively, but it requires a lot of computation to distinguish between blur and deblur areas, and each adaptive filter coefficient is further quantized. Must be sent to the decoder. Therefore, as the number of adaptive filters increases, the amount of additional data to be transmitted increases, thereby reducing the compression efficiency.

In addition, comparing the individual performances of the two filter sets, the low-pass filter set is less efficient than the adaptive filter set in the high resolution image. On the other hand, the adaptive filter set has a problem that the compression efficiency is lower than that of the low pass filter set in a compression environment having a large quantization parameter value.

An object of the present invention for solving the above problems, by using the statistical characteristics of the image data of the current frame and the previous frame to estimate the filter that can compensate for the change between the adjacent image and performing preprocessing filtering on the reference frame image The present invention provides a motion estimation and compensation method during video encoding that can maintain high performance even when information changes rapidly.

Another object of the present invention for solving the above problems is, by performing a pre-processing filtering on the reference frame, motion estimation during video decoding using a motion estimation and compensation method that can maintain high performance even if the image information changes rapidly And to provide a compensation method.

According to an aspect of the present invention, there is provided a motion estimation and compensation method for video encoding, comprising: estimating a degree of change of image information between a current frame and a reference frame by using a current frame and a reference frame, the estimated image information Constructing a preprocessing filter set including at least one of a low pass filter set and an adaptive filter set based on a degree of change of, applying the preprocessing filter set to the reference frame, performing a preprocessing process, and the preprocessing process A motion estimation and compensation method including performing motion estimation and compensation using the performed reference frame is provided.

Here, the estimating of the change of the image information may be configured to estimate the degree of blur of the image by estimating the degree of blur of the image.

The estimating of the change of the image information may be configured to calculate an average of motion vector magnitudes of all blocks in the reference frame and to estimate the degree of change of the image information based on the average.

The configuring of the preprocessing filter set may be configured to configure the preprocessing filter set by using a low pass filter set and an adaptive filter set when the estimated degree of change of the image information is large.

Here, the configuring of the preprocessing filter set may be performed by using the prefilter using the adaptive filter set when the estimated change of the image information is small and the cost function of the adaptive filter set does not exceed a predetermined cost function threshold. It can be configured to construct a filter set.

Here, in the configuring of the preprocessing filter set, if the change of the estimated image information is small and the cost function of the adaptive filter set exceeds a predetermined cost function threshold, a correlation coefficient between the adaptive filter and the low pass filter is included. If the average value of the correlation coefficient is greater than the predetermined cost function threshold value can be configured to configure the pre-processing filter set using only the low pass filter.

The low pass filter set may include Gaussian Low Pass Filters (GLPF) in the preprocessing filter set, wherein the low pass filter set is N (N is a natural number). At least two or more of a square filter of * N, a vertical filter of N * 1, a horizontal filter of 1 * N and a diagonal filter.

In the configuring of the preprocessing filter set, the adaptive filter set may include a least square error filter that minimizes a matching error with respect to a current frame and a reference frame, wherein the adaptive filter set Is composed of adaptive filters calculated on a frame encoded before the current frame when the video encoding is performed by single pass encoding, and a frame encoded before the current frame when the video encoding is multi-pass encoded. It may be composed of adaptive filters calculated at and adaptive filters calculated at the current frame.

The performing of the preprocessing process may be configured to selectively apply a pre-processing filter belonging to the preprocessing filter set for each region of the reference frame to perform the preprocessing process.

According to another aspect of the present invention, there is provided a motion estimation and compensation method for decoding an encoded video, the method comprising: obtaining configuration information of a preprocessing filter set including at least one of a low pass filter set and an adaptive filter set, Applying a preprocessing filter set configured by referring to the configuration information of the preprocessing filter set to a reference frame to perform a preprocessing process and performing motion compensation using the reference frame on which the preprocessing process is performed Provide a method.

Herein, in the obtaining of configuration information of the preprocessing filter set, configuration information of the preprocessing filter set may be included in a slice header and obtained.

In the performing of the preprocessing process, the preprocessing filter applied to the encoding of the reference frame may be identified from the configured preprocessing filter set, and the identified preprocessing filter may be selected from the configured preprocessing filter set for the reference frame. It can be configured to apply. In this case, the performing of the preprocessing process may be configured to perform a preprocessing process by selectively applying a preprocessing filter belonging to the preprocessing filter set for each region of the reference frame.

The motion estimation and compensation method using intra-loop preprocessing filtering according to the present invention as described above further performs a preprocessing process before motion estimation in a video compression encoder and decoder, thereby compensating for the blur and deblur phenomenon. The accuracy of compensation can be increased.

Therefore, the video compression efficiency can be improved by greatly reducing the bit rate in the encoding process. In particular, it can show an excellent performance in compressing sports images with a lot of movement of objects and frequently moving cameras.

1 is a flowchart illustrating an embodiment of a motion estimation and compensation method in video encoding according to the present invention.
2 is a conceptual diagram illustrating a concept of grading degree of change of an image according to an exemplary embodiment of the present invention.
3 is a conceptual diagram illustrating a method for constructing an adaptive filter set in the case of single pass coding according to an embodiment of the present invention.
4 is a flowchart illustrating a method of configuring a preprocessing filter set in an embodiment according to the present invention.
5 is a flowchart illustrating an embodiment of a motion estimation and compensation method in video decoding according to the present invention.
6 is a block diagram illustrating an embodiment of a video encoder to which an in-loop preprocessing filter is applied according to the present invention.
7 is a block diagram illustrating an embodiment of a video decoder to which an in-loop preprocessing filter is applied according to the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

A video encoding apparatus (Video Encoding Apparatus), a video decoding apparatus (Video Decoding Apparatus) to be described below is a personal computer (PC), notebook computer, personal digital assistant (PDA), portable multimedia player (PMP) It may be a user terminal such as a portable multimedia player (PSP), a PlayStation Portable (PSP), a wireless communication terminal, a smart phone, a TV, or a server terminal such as an application server or a service server. Communication device, such as a communication modem for communicating with various devices or a wired / wireless communication network, memory for storing various programs and data for inter- or intra-prediction for encoding or decoding an image, or for encoding or decoding an image, and executing a program Microprocessor for operation and control It can mean a variety of devices.

In addition, the image encoded in the bitstream by the video encoding apparatus is real-time or non-real-time through the wired or wireless communication network, such as the Internet, local area wireless communication network, wireless LAN network, WiBro network, mobile communication network, or the like, or a cable, universal serial bus (USB: Universal) It may be transmitted to an image decoding apparatus through various communication interfaces such as a serial bus, and may be decoded by the image decoding apparatus to restore and reproduce the image.

In general, a video may be composed of a series of pictures, and each picture may be divided into a predetermined area such as a frame or a block. When a region of an image is divided into blocks, the divided blocks may be classified into intra blocks and inter blocks according to an encoding method. An intra picture block refers to a block that is encoded by using an intra prediction coding method. An intra picture prediction encoding indicates a pixel of blocks previously encoded, decoded, and reconstructed in a current picture that performs current encoding. A prediction block is generated by predicting pixels of the current block using the prediction block, and a difference value with the pixels of the current block is encoded. An inter block refers to a block that is encoded using inter prediction coding. Inter prediction coding generates a prediction block by predicting a current block within a current picture by referring to one or more past or future pictures, and then generates a current block. This is a method of encoding the difference value with. Here, a frame referred to for encoding or decoding the current picture is referred to as a reference frame.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

Meanwhile, the following embodiments are described based on the H.264 / AVC codec to confirm the performance of the present invention. However, the technical idea of the present invention is a video compression to which a motion estimation and compensation method other than H.264 / AVC is applied. It is widely applicable to the technology. For example, high-efficiency video coding (HEVC), which is being standardized as the next video compression standard of H.264 / AVC, can be applied.

Motion estimation and compensation method according to the present invention

1 is a flowchart illustrating an embodiment of a motion estimation and compensation method in video encoding according to the present invention.

Referring to FIG. 1, an embodiment of the motion estimation and compensation method according to the present invention includes estimating a degree of change of image information between a current frame and a reference frame using a current frame and a reference frame (S110). Configuring a preprocessing filter set including at least one of a low pass filter set and an adaptive filter set based on a degree of change of information (S120), and performing a preprocessing process by applying the preprocessing filter set to the reference frame. And a step (S140) of performing motion estimation and compensation using the reference frame on which the preprocessing process is performed.

First, step S110 is a step of estimating the degree of change of the image information between the current frame and the reference frame using the current frame and the reference frame. Image characteristic information may be utilized.

For example, in the present embodiment, a method of estimating the degree of change of image information using two pieces of image information, namely, a blur intensity of a current frame and an average size of a motion vector of a reference frame, will be described as an example.

First, H. Tong, M. Li, H. Zhang, and C. Zhang, “Blur detection for digital images using wavelet transform,” in Proc. IEEE ICME, vol. 1, pp. 17-20, Jun. 2004. The method of the paper can be used. In the blur detection algorithm, the scales of Per and BlurExtent are used to estimate the blur. In the embodiment of the present invention, a weighted blur extent (WBE) is described below to measure the blur of an image by combining two variables. It can be used to transform. Such a calculation method of the weight blur scale may be expressed by Equation 2 below.

&Quot; (2) "

Here w denotes a weight value and T denotes a threshold value.

Second, the mean of motion vectors (MMV) of the motion vectors of all the blocks in the frame is used. The magnitude of the motion vector of each block is represented by Equation 3 below.

&Quot; (3) "

Here, MV represents the magnitude of the motion vector of the block. mv _x represents the horizontal motion vector component of the block, and mv _y represents the vertical motion vector component of the block. The average value of the motion vector magnitudes of all blocks is MMV and used as a measure of motion activity of the corresponding frame.

Thus, the two measures described above may be used to estimate the degree of change between the current frame and the reference frame. In addition, in the embodiment of the present invention, the degree of change of the image is graded to take the configuration of the preprocessing filter set to be described later according to the grade.

2 is a conceptual diagram illustrating a concept of grading degree of change of an image according to an exemplary embodiment of the present invention.

Referring to FIG. 2, an embodiment according to the present invention may be configured to grade two degree of scene changes (hereinafter referred to as DoSCs) by using the aforementioned two measures.

Here, MMV _{n - 1} 210 represents the average of the motion vector magnitude of the reference frame, and WBE _n 220 represents the degree of blur of the current frame. Where L1, L2, and L3 are the grades of DoSC's strength and are high in the order of L1, L2, and L3. γ ₁ , γ ₂ , μ ₁ , μ ₂ are thresholds for dividing the DoSC plane.

The range of the MMV may vary depending on the size of the search region of the reference frame. Therefore, μ ₁ = 0.5 × SR and μ ₂ = 0.125 × SR are determined using a search range (SR).

WBE can change its value depending on the size of the frame. Accordingly, γ ₁ and γ ₂ may be determined as in Equation 4 below.

&Quot; (4) "

Where W and H represent the width and height of the current frame, respectively, and M and N are the width and height of the frame defined for normalization. α ₁ and α ₂ are proportional constants. M, N, α ₁ , α ₂ values used in the experiment were set to 832, 480, 1.5 and 3.0, respectively.

Next, step S120 is a step of constructing a preprocessing filter set including at least one of a low pass filter set and an adaptive filter set based on the degree of change of the image information estimated in step S110.

The preprocessing filter set may be configured by selecting candidate filters that can best reflect the change of the image information by using the estimated grade of the change of the image information. The low pass filter set and the adaptive filter set constituting the preprocessing filter set are described as follows.

The first filter set is a low pass filter set. The strength of the low pass filter is determined according to the degree of change of the image information using two kinds of image information, the intensity of the blur of the current frame and the size of the motion vector of the reference frame.

The low pass filter set may be composed of an NxN square filter, an Nx1 vertical filter, a 1xN horizontal filter, and a diagonal lowpass filter.

A Gaussian function such as Equation 5 may be used to generate a low-pass filter set.

[Equation 5]

The Gaussian Low Pass Filter (GLPF) is a type such that the NxN filter is represented by GLPF _NxN , the Nx1 filter is represented by GLPF _Nx1 , and the 1xN filter is represented by GLPF _1xN .

The lowpass filter set is composed of at least two filter sets. In this example, the mean and variance of the Gaussian function are GLPF with N as 3 and the coefficient with Gaussian as 0 and variance as 1.5. The coefficient can be determined by two.

The second filter set is an adaptive filter (APF) set. The adaptive filter may be configured by calculating a least square error filter that minimizes matching error with respect to the current frame and the reference frame. When calculating the least squared error, it is calculated by adding a zero-order term as an offset term.

3 is a conceptual diagram illustrating a method for constructing an adaptive filter set in an embodiment according to the present invention.

Referring to FIG. 3, the adaptive filter coefficient (APF _{t −1} ) of the current frame 310 and the encoded previous frame (past or future frame of the current frame, future frames of the current frame in H.264 / AVC) In FIG. 3, since the coefficients of the adaptive filters (APF _{t- 2} or APF _{t- 3} ) calculated in the past frame of the current frame; 311, 312, and 313 have a high correlation, An adaptive filter set may be determined by selecting one of single pass coding and multiple pass coding (two embodiments are possible).

If single pass coding is performed, the adaptive filter set is composed of adaptive filters (APF _{t -2} or APF _{t -3} ) calculated in the previous frames 311, 312, 313 after encoding.

If multi-pass coding is performed, the adaptive filter set includes the adaptive filter (APF _{t -1} ) calculated for the current frame and the adaptive filters (APF _{t -2} or APF) calculated in the previous frames. _{t -3} ). That is, referring to FIG. 3, an adaptive filter set is constructed by using an adaptive pre-filter (APF) calculated in a previous frame, and the most recently calculated to construct an adaptive filter set. Use two APFs.

To calculate adaptive filter coefficients, see, eg, Y. Ye, G. Motta, and M. Karczewicz, “Enhanced adaptive interpolation filters for video coding,” in Proc. IEEE DCC, pp. 432-444, Mar. 2010. The method presented in the paper can be applied.

Since the number of additional bits for indicating the type of filter increases as the number of filters included in the preprocessing filter set increases, the number of mixed preprocessing filter sets is previously defined in the slice header and transmitted.

The preprocessing filter set may be adaptively configured as follows according to the degree of change of the image.

First, if it is determined that the image information feature of the current frame is significantly changed compared to the reference frame, the preprocessing filter set is configured by using both the low pass filter set and the adaptive filter set. The low pass filter set is defined according to the degree of image change and the intensity of the blur of the current frame, and added to the preprocessing filter set. In the adaptive filter set, the most recently calculated adaptive filter is added to the preprocessing filter set. When the number of predefined preprocessing filters is added, the filter set is completed.

Second, if it is judged that the image information characteristic of the current frame is less changed than the reference frame, and the adaptive filter is available because the cost function of the adaptive filter does not exceed the cost function threshold, only the adaptive filter set is used. To configure the set of preprocessing filters.

Third, if the image information characteristic of the current frame is less changed than the reference frame and the cost function of the adaptive filter is not available beyond the cost function threshold, the correlation coefficient between the adaptive filter and the lowpass filter is calculated and correlated. The mean value of the coefficients is compared with a predefined correlation coefficient threshold. In this case, if the mean value of the correlation coefficients of the adaptive filter and the low pass filter is larger than the correlation coefficient threshold, the mixed preprocessing filter is configured using only the low pass filter, otherwise the preprocessing process is omitted.

That is, the adaptive filter coefficient may affect the compression efficiency in the encoding process having a high quantization parameter. Therefore, the cost function of the adaptive filter coefficient is used to determine whether to use the adaptive filter. If the ratio-distortion function of the adaptive filter coefficients divided by the total number of blocks exceeds the cost function threshold λ, the adaptive filter is not used. λ is 30. In other words, if the rate-distortion function of the adaptive filter coefficients is normalized to the size of the frame, the adaptive filter is turned off.

Hereinafter, a method of configuring the preprocessing filter set using the above-described degree of change of the image will be described as a specific embodiment.

4 is a flowchart illustrating a method of configuring a preprocessing filter set in an embodiment according to the present invention.

That is, the flowchart described with reference to FIG. 4 is for explaining step S120 of FIG. 1 in more detail.

First, the number of possible preprocessing filter sets may be predefined as shown in Table 1 below. For example, six sets of preprocessing filters are defined.

Pretreatment filter set type MPF set Θ ₁ {GLPF _{3 × 3} , GLPF _{3 × 1} , GLPF _{1 × 3} , GLPF _{5 × 5} } Θ ₂ {GLPF _{5 × 5} , GLPF _{5 × 1} , GLPF _{1 × 5} , GLPF _{3 × 3} } Θ ₃ {GLPF _{3 × 3} , GLPF _{3 × 1} , GLPF _{1 × 3} , APF _{t -1} } Θ ₄ {GLPF _{5 × 5} , GLPF _{5 × 1} , GLPF _{1 × 5} , APF _{t -1} } Θ ₅ {APF _{t -1} , APF _{t -2} } Θ ₆ {GLPF _{3 × 3} , GLPF _{3 × 1} , GLPF _{1 × 3} }

If the DoSC value is L2 or L3, the mixed preprocessing filter is determined in the order shown in FIG. 4 using the low pass filter set and the adaptive filter set.

If the DoSC value is L1 and the adaptive filter is available, the mixed preprocessing filter is configured using only the adaptive filter set (S410).

If the DoSC value is L1 and no adaptive filter is available, the correlation coefficients of the adaptive and lowpass filters are calculated and the mean of the correlation coefficients is compared with the predefined correlation coefficient threshold. At this time, if the average value of the correlation coefficient between the adaptive filter and the low pass filter is greater than the correlation coefficient threshold, the mixed preprocessing filter is configured using only the low pass filter (S420), otherwise the preprocessing process is omitted (S430).

Next, step S130 is a step of performing a preprocessing process by applying the preprocessing filter set determined through the step S120 to the reference frame. In other words, after the pre-processing process is performed on the reference frame using the pre-processing filter set configured through step S120, the reference frame buffer is added to the reference frame buffer.

On the other hand, since the change of the image data between two images may be different for each region, the preprocessing filter that minimizes the RD-cost is determined in units of macroblocks, and the preprocessing filter is selectively selected from the preprocessing filter set for each region of a reference frame (eg, It can be configured to perform the preprocessing process by applying, in macroblock units or blocks of any size). In this case, in order to specify the type of preprocessing filter used in the encoding process, an additional bit for specifying a preprocessing filter applied on a region basis of a reference frame (for example, in a macroblock unit or an arbitrary block size) is transmitted to the decoder. Can be configured.

Finally, in step S140, motion estimation and compensation for the current frame are performed using the reference frame preprocessed through step S130. Motion estimation and compensation in step S140 may be a motion estimation and compensation method applied to conventional video encoding.

Meanwhile, as described above, the preprocessing filtering on the reference frame in the motion estimation and compensation method described with reference to FIG. 1 is related to the encoding process, and the preprocessing applied to the reference frame in the encoding process to perform the motion estimation and compensation also in the decoding process. The same filtering as filtering should be performed.

5 is a flowchart illustrating an embodiment of a motion estimation and compensation method in video decoding according to the present invention.

Referring to FIG. 5, according to an embodiment of the motion estimation and compensation method during video decoding, acquiring configuration information of a preprocessing filter set including at least one of a low pass filter set and an adaptive filter set (S510). ), Performing a preprocessing process by applying a preprocessing filter set configured by referring to the configuration information of the preprocessing filter set to a reference frame (S520), and performing motion compensation using the reference frame on which the preprocessing process is performed ( S530) can be configured to include.

The motion estimation and compensation method during video decoding described with reference to FIG. 5 may be applied during a process of decoding a video encoded through FIG. 1. That is, the configuration of the preprocessing filter set applied in the process of applying the motion estimation and compensation method according to the present invention described with reference to FIG. 1 and the type of the preprocessing filter applied for each reference frame (or for each region of the reference frame). A feature of the motion estimation and compensation method during video decoding described with reference to FIG. 5 is to apply the same preprocessing filter to the reference frame during the decoding process.

First, obtaining configuration information of a preprocessing filter set including at least one of a low pass filter set and an adaptive filter set (S510) is a step in which a decoder obtains configuration information of a preprocessing filter set applied in an encoding process from an encoder. The configuration information of the preprocessing filter set may be configured to be obtained from an encoder by being included in a slice header. The configuration information of the preprocessing filter set may include, for example, the type of the filter set, the filter of the filter if necessary, as illustrated in Table 1. In the case of a coefficient value, a Gaussian low pass filter, and the like, the mean and variance values of the Gaussian function may be included.

Meanwhile, the low pass filter set and the adaptive filter set constituting the preprocessing filter set are described in step S120 of the method according to the present invention applied to the video encoding described with reference to FIG.

Next, the performing of the preprocessing step (S520) is a step of performing the preprocessing filtering on the reference frame by applying the preprocessing filter set configured by referring to the configuration information of the preprocessing filter set obtained in step S510 to the reference frame. .

That is, the performing of the preprocessing process (S520) identifies the type of the preprocessing filter applied at the time of encoding the reference frame from the preprocessing filter set configured by referring to the configuration information of the preprocessing filter set acquired in the step S510, and grasps the identified preprocessing. A filter is selected from the configured preprocessing filter set and applied to the reference frame to perform preprocessing filtering. In this case, the preprocessing filtering on the reference frame may be performed by selectively applying a preprocessing filter belonging to the preprocessing filter set on a region-by-region basis (for example, in a macroblock unit or a block of an arbitrary size). Can be configured. Meanwhile, a bit for designating a type of preprocessing filter to be applied for each reference frame or region of a reference frame may be configured to be transmitted by being specified for each reference frame or region of a reference frame from an encoder.

Finally, step S530 is a step of performing motion compensation using a reference frame on which the preprocessing process is performed in step S520. That is, in step S530, motion estimation and compensation are performed on the current frame using the preprocessed reference frame through step S520. In step S530, motion estimation and compensation are performed on the conventional video encoding. Applied motion estimation and compensation methods may be applied.

In-loop pretreatment according to the invention Filtering  applied Encoder  And decoder

6 is a block diagram illustrating an embodiment of a video encoder to which an in-loop preprocessing filter is applied according to the present invention.

Referring to FIG. 6, the video encoder 600 according to the present invention includes a prediction branch unit 610 for branching intra-frame prediction and inter-frame prediction, and intra to perform intra-frame prediction. The prediction unit 620, a motion estimation and compensation unit 630 that performs motion estimation and compensation for inter-frame prediction, a transformer 614, a quantizer 642, an entropy encoder 643, and an inverse transformer 645. ), An inverse quantization unit 644, a reference frame buffer 650, and an in-loop preprocessing filter 660 for performing preprocessing filtering on the reference frame.

In this case, the components except for the preprocessing filter 660 in the loop play the same role as the components of the conventional video encoder, and thus detailed description thereof will be omitted. The motion estimation and compensation to which the in-loop preprocessing filter according to the present invention is applied is a motion for performing motion estimation and compensation using the reference frame and the current frame which have passed through the pre-loop preprocessing filter 660 and the in-loop preprocessing filter 660. The estimation and compensation unit 630 is made.

Specifically, the video encoder according to the present invention is a motion estimation and compensation method to which the preprocessing filter in the loop according to the present invention described with reference to FIG. 1 is performed in steps S110 to S130 in the preprocessing filter 660 in the loop. Is configured to be performed, and in the motion estimation and compensation unit 630, step S140 may be performed.

7 is a block diagram illustrating an embodiment of a video decoder to which an in-loop preprocessing filter is applied according to the present invention.

Referring to FIG. 7, the video decoder according to the present invention includes an entropy decoder 711, an inverse quantizer 712, an inverse transform unit 713, intra-frame prediction, and inter-frame prediction. The prediction branching unit 720 for branching the intra prediction, the intra prediction unit 730 for performing intra-frame prediction, the motion compensation unit 740 for performing motion compensation for inter-frame prediction, the reference frame buffer 760 and the reference frame. And a preprocessing filter 750 in a loop to perform preprocessing filtering on.

In this case, the components except for the preprocessing filter 750 in the loop play the same role as the components of the conventional video decoder, and thus, detailed description thereof will be omitted. The motion compensation to which the in-loop preprocessing filter is applied according to the present invention generates a prediction frame for the current frame based on the motion vector information by using the reference frames that have passed through the in-loop preprocessing filter 750 and the in-loop preprocessing filter 750. The motion compensator 740 is performed. In addition, the prediction frame generated by the motion compensator 740 is added to the difference frame generated through the entropy decoder 711, the inverse quantizer 712, and the inverse transform unit 713 to generate a current frame.

Specifically, in the video decoder according to the present invention, the motion estimation and compensation method to which the preprocessing filter is applied according to the present invention described with reference to FIG. 5 is performed in the preprocessing filter 750 in the loop (S510 and S520). It may be configured to perform the operation, and may be configured to perform the step (S530) in the motion compensation unit 740.

Meanwhile, although FIGS. 6 and 7 illustrate the preprocessing filter and the motion estimation and compensation unit in the loop as separate components, this is a representation for functional division and may be implemented as one component. . In addition, it will be appreciated by those skilled in the art that the preprocessing filter and the motion estimation and compensation unit in the loop may be implemented as software code that can be read and executed by some component or central processing unit in an application specific integrated circuit (ASIC) together with the entire video encoder. Will be self explanatory.

H.264 / AVC  Simulation result by reference software

Table 2 below shows the results of implementing the above-described embodiment in the H.264 / AVC reference software to confirm the performance of the motion estimation and compensation method according to the present invention.

In order to compare the performance, H.264 / AVC high profile was used as a reference. The BD-PSNR in the table shows the difference in the average PSNR compared to the H.264 / AVC high profile, and the BD-RATE shows the difference in the average bit-rate compared to the H.264 / AVC high profile.

Size Sequence BD-rate BD-PSNR CIF Foreman -1.768 0.076 Mobile -0.089 0.004 Football -5.369 0.270 Coastgaurd -3.482 0.146 CIF Average -2.677 0.124 WVGA RaceHorses -1.811 0.082 Bqmall -2.747 0.122 Bashketballdrill -4.150 0.163 Keiba3 -3.462 0.153 Nuts5 -3.788 0.121 WVGA Average -3.192 0.128 HD Raven -15.548 0.672 Night -5.098 0.197 City -11.745 0.394 Bigship -9.640 0.275 HD Average -10.508 0.384

Referring to Table 2, it can be confirmed that the bit rate reduction (negative BD-rate) and the image quality improvement (positive BD-PSNR) can be obtained at all resolutions and all sample sequences.

This result demonstrates that the present invention can improve the compression efficiency by achieving a higher bit rate reduction than the conventional method.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

600: video encoder
610: prediction branch
620: intra prediction unit
630: motion estimation and compensation unit
641: conversion unit
642: quantization unit
643: entropy encoder
644: inverse quantization
645: inverse transform unit
650: reference frame buffer
660: preprocessing filter in loop

Claims (18)

As a motion estimation and compensation method for video encoding,
Estimating a degree of change of image information between the current frame and the reference frame using the current frame and the reference frame;
Constructing a pre-filter set including at least one of a low pass filter set and an adaptive filter set based on the estimated degree of change of the image information;
Performing a preprocessing process by applying the preprocessing filter set to the reference frame; And
And performing motion estimation and compensation using a reference frame on which the preprocessing is performed. The method of claim 1,
The estimating of the change of the image information may include estimating the degree of blur of the image to estimate the degree of change of the image information. The method of claim 1,
And estimating a change in the image information comprises calculating an average of motion vector sizes of all blocks in the reference frame and estimating a change degree of the image information based on the average. The method of claim 1,
The configuring of the preprocessing filter set comprises configuring the preprocessing filter set by using a low pass filter set and an adaptive filter set when the estimated degree of change of the image information is large. The method of claim 1,
The configuring of the preprocessing filter set may include configuring the preprocessing filter set by using an adaptive filter set when the change of the estimated image information is small and the cost function of the adaptive filter set does not exceed a predetermined cost function threshold. Motion estimation and compensation method characterized in that the configuration. The method of claim 1,
The configuring of the preprocessing filter set may include calculating correlation coefficients of the adaptive filter and the low pass filter when the estimated change of the image information is small and the cost function of the adaptive filter set exceeds a predetermined cost function threshold. And if the mean value of the correlation coefficient is greater than a predetermined cost function threshold, constructing the preprocessing filter set using only a low pass filter. The method of claim 1,
And in the configuring of the preprocessing filter set, the low pass filter set comprises Gaussian Low Pass Filters (GLPF). The method of claim 7, wherein
The low pass filter set includes at least two of N (N is a natural number) * N square filter, N * 1 vertical filter, 1 * N horizontal filter, and diagonal filter. Motion estimation and compensation method. The method of claim 1,
And in the configuring of the preprocessing filter set, the adaptive filter set comprises a least square error filter that minimizes matching error with respect to a current frame and a reference frame. The method of claim 9,
The adaptive filter set
When the video encoding is performed by single pass encoding, it is composed of adaptive filters calculated on a frame encoded before the current frame.
If the video encoding is to be performed in the multi-pass coding, the motion estimation and compensation method comprising the adaptive filters calculated in the frame encoded before the current frame and the adaptive filters calculated in the current frame. The method of claim 1,
Performing the pretreatment process
And performing a preprocessing process by selectively applying a pre-processing filter belonging to the preprocessing filter set for each region of the reference frame. A motion estimation and compensation method for decoding an encoded video,
Obtaining configuration information of the preprocessing filter set including at least one of a low pass filter set and an adaptive filter set;
Performing a preprocessing process by applying a preprocessing filter set configured by referring to the configuration information of the preprocessing filter set to a reference frame; And
And performing motion compensation using the reference frame on which the preprocessing is performed. 13. The method of claim 12,
The low pass filter set constituting the preprocessing filter set,
A method for motion estimation and compensation, comprising Gaussian Low Pass Filters (GLPF). The method of claim 13,
The low pass filter set includes at least two of N (N is a natural number) * N square filter, N * 1 vertical filter, 1 * N horizontal filter, and diagonal filter. Motion estimation and compensation method. 13. The method of claim 12,
And the adaptive filter set constituting the preprocessing filter set comprises a least square error filter that minimizes matching error with respect to the current frame and the reference frame. 13. The method of claim 12,
And obtaining configuration information of the preprocessing filter set, wherein the configuration information of the preprocessing filter set is included in a slice header and obtained. 13. The method of claim 12,
The performing of the preprocessing step may include grasping a type of a preprocessing filter applied in encoding the reference frame from the configured preprocessing filter set, and applying the identified preprocessing filter from the configured preprocessing filter set to the reference frame. Characterized in the motion estimation and compensation method. The method of claim 17,
Performing the pretreatment process
And performing a preprocessing process by selectively applying a pre-processing filter belonging to the preprocessing filter set for each region of the reference frame.

Images