KR100240620B1 - Method and apparatus to form symmetric search windows for bidirectional half pel motion estimation - Google Patents

Abstract

A method of forming a bidirectional coded picture from two reference pictures is disclosed. The method includes performing a memory fetch from each reference picture, finding a best match macroblock at the telephone line boundary, interpolating the telephone line boundary macroblock to form a bidirectional macroblock, and the bidirectional Calculating half-pixel reference picture data from the macroblock. The surplus pixels are reduced from the best match macroblock to make the best match macroblock symmetric in terms of size, shape, and direction. This can be accomplished by setting an edge detector to represent excess pixels. The surplus pixels correspond to pixels that make the best matching macroblock different in size, shape, or direction.

Description

Method and apparatus for forming symmetric search window for bidirectional half-pixel motion estimation

FIELD OF THE INVENTION The present invention relates to the compression of digital visual images, and more particularly to temporal compression, i.e. reduction of redundancy between pictures by the MPEG2 standard. By using a motion vector, inter-image redundancy is reduced or eliminated. According to the invention, half-pel motion vectors for bi-directionally predictions are generated from symmetric past and future best match macroblocks which are symmetrical to each other. . Here, symmetry means that macroblocks have the same size, shape, and direction.

In the past decades, electronic communication systems have emerged around the world, greatly improving the way people exchange information. In particular, the performance of real-time video / audio systems has advanced rapidly in recent years. In order to provide services such as video-on-demand (VOD) or video conferencing to subscribers, a very wide band of network and width is required. In fact, in many cases the efficiency of such a system is limited by the network band.

To overcome the limitations of the network, compression systems have emerged. The compression system reduces the amount of video / audio data to transmit by removing redundancy in the picture sequence. On the receiving side, the picture sequence can be decompressed and displayed in real time.

The Moving Picture Experts Group (MPEG) standard is an example of such a video compression standard. In the MPEG standard, video compression is defined within and between pictures. In-picture video compression is achieved by converting a digital image in the time domain into a frequency domain by discrete cosine transform, quantization, variable length coding, and Huffman coding. Inter-picture video compression is achieved by a process called motion estimation, in which motion vectors are used to indicate the transition of a set of picture dliments (pels) from one picture to another.

The motion estimation is achieved by obtaining 16x16 macroblock data from the current picture and comparing this macroblock with all 16x16 macroblocks in the search window of the reference picture, that is, the search window of both reference pictures for bidirectional prediction. The center of the search window is around the location of the current 16x16 macroblock. The comparison is made by taking the accumulated absolute pel difference between the current macroblock and the reference macroblock, and selecting the reference macroblock associated with the smallest prediction difference as the best match macroblock.

The MPEG2 criterion uses bi-directional interpolation with past and future reference pictures, so that motion estimation for picture encoding can be calculated. In fact, this is the most efficient form of motion estimation since it has the highest degree of data compression. The estimation may be done by full pel values or half pel values. Half pixel values are formed by interpolating between telephone values. In order to form all possible half-pixel values for a 16x16 macroblock, a telephone value of the area of 18x18 and an additional telephone value of each macroblock side are needed.

The problem that occurs in half-pixel motion estimation is that if the best matching phone location of the reference data is located at the corner, for example, at the corner of the phone search window, the half-pixel values are added to the macroblock side. Is not available to form a value.

This problem occurs when in the bidirectional half-pixel motion estimation, the best match phone data from one reference picture is at the edge of the picture and the data from the remaining pictures is not at this same edge. In this case, a half-pel calculation process is provided with two sets of reference data having different sizes, shapes, and / or directions for the current macroblock. In this case, separate read control of the reference data is required to pad, align, and skip the data to form a bidirectional half-pixel value.

It is a primary object of the present invention to impart symmetry to unsymmetrical search windows. This is to ensure that the best matching macroblocks used to form the bidirectional interpolated image have the same number of pixels, the same shape, and the same direction.

Figure 1 shows the flow of an encoder 11 suitable for generalized MPEG2, which comprises a discrete cosine transformer 21, a quantizer 23, a variable length coder 25 and an inverse quantizer 29. ), An inverse discrete cosine transformer 31, a motion compensator 41, a frame memory 42, and a motion estimator 43. The data path includes an i-th image input 111, a difference data 112, a motion vector 113, an image output 121, a feedback image 131 for motion estimation and compensation, and a motion compensation image 101. Include. In FIG. 1, it is assumed that the i th picture is located in the frame memory or frame store 42, and the (i + 1) th picture is encoded using motion estimation.

2 shows I, P, and B pictures, examples of their display order and transmission order, and forward and backward motion prediction.

3 is a diagram illustrating a process of searching for a best matching block of a next or previous frame / picture from a motion estimation block of a current frame / picture.

4 is a diagram showing the movement of a block from the position of the reference picture to the current picture by the motion vector, and the block of the reference picture after adjustment using the motion vector.

5 shows the reference data needed to calculate all half-pixel values.

6 shows reference data required to calculate half-pixel values when none of the reference data is available.

7 shows reference data needed to calculate half-pixel values using bi-directional edge conditions with symmetrical past / future data.

* Explanation of symbols for main parts of the drawings

11: Encoder 21: Discrete Cosine Converter

23 quantizer 25 variable length coder

29 Inverse Quantizer 31 Inverse Discrete Cosine Converter

41: motion compensator 42: frame memory

43: motion estimator

According to the method and apparatus of the present invention, this object is achieved by forming a picture that is bidirectionally coded from two reference pictures. This method is initiated by performing a memory fetch of each reference picture and finding the best match macroblock at the telephone boundary of the search window. Excess pixels are excluded from the best match macroblock, thereby causing the best match macroblock to be symmetrical in size, shape, and direction. This is accomplished by setting an edge detector to display excess pixels. The inyo pixels correspond to pixels which cause the best matching macroblocks to differ in size, shape or direction.

Thus, according to the present invention, it is possible to always provide a half-pixel calculation circuit having reference data symmetrical from past and future images. That is, the data has the same size and direction compared to the current macroblock position. In particular, when the two macroblocks do not have the same size, the large macroblock is reduced to the size of the small macroblock.

The invention described herein relates to an improvement of a half-pixel motion estimation method for B or bi-directional interpolated pictures, described by Agnes Agai and Ronald S. US patent application Ser. No. 08 / 411,100, filed March 27, 1995, by Svec, improves the half-pixel motion estimation method for B pictures.

The present invention relates to encoders and encoding processes that are compliant with MPEG and HDTV. Encoding functions performed by the encoder include data input, motion estimation, macroblock mode generation, data reconstruction, entropy coding, and data output functions. Motion estimation and compensation is a time compression function. These are repetitive functions that require high computational power and include intensive reconstruction processes such as inverse discrete cosine transform, inverse quantization, and motion compensation.

In particular, the present invention relates to motion estimation, compensation, and prediction, and more particularly to the calculation of motion vectors. Moving compensation utilizes temporal redundancy by dividing the current picture into blocks, such as macroblocks, and then searching for neighboring blocks with similar content in previously transmitted pictures. Only the difference between the current block pixel derived from the reference picture and the predicted block pixel is actually compressed and then transmitted.

The simplest method of motion compensation and prediction is to record the luminance and chrominance, i.e. intensity and color, in all the pixels of an "I" image, and then change the luminance and chrominance in all the specific pixels of the image, That is, the change in intensity and color is recorded. However, this method is uneconomical in terms of transmission frequency, memory, processor performance, and processing time, because the objuct moves between images, i.e. the content of a pixel is a particular position in an image. This is because the image moves to another position in the subsequent image. A more advanced method is to predict the position of the pixel block in the subsequent or previous picture using the previous or subsequent picture using a motion vector or the like and describe the result as a "predicted picture" or "P" picture. In particular, the method provides the best match or best prediction as to where the pixel of the i-th picture or the macroblock of the pixel is to be located in the (i-1) or (i + 1) th picture. It is related to accomplishing. Predicting pixelblock positions in an intermediate or "B" picture using both subsequent pictures and previous pictures is a step forward.

The picture encoding order and the call transmission order are not necessarily the same as the picture display order. This is shown in FIG. In the I-P-B system, the input picture transmission order is different from the encoding order, and the input picture is temporarily stored in a buffer until used for encoding.

For convenience of explanation, a general flowchart of encoding suitable for MPEG shown in FIG. 1 is shown. In the above flow chart, the images of the i th picture and the (i + 1) th picture are processed to generate a motion vector. The motion vector predicts where the pixel macroblock is located in the previous and / or subsequent picture. The use of motion vectors instead of full images in MPEG and HDTV standards is a key feature of time compression. As shown in Fig. 1, the generated motion vector is used for position translation of the pixel macroblock from the i-th image to the (i + 1) th image.

As shown in Fig. 1, in the encoding process, the image of the i th picture and the (i + 1) th picture are processed by the encoder 11 to generate a motion vector, for example, the (i + 1) th and Subsequent pictures are encoded and sent in this format. The input image 111 of the subsequent picture is input to the motion estimation unit 43 of the encoder. The motion vector 113 is formed as an output of the motion estimation unit 43. This motion vector is used by the motion compensation unit 41 to retrieve macroblock data that is " reference " data from the past and / or future pictures to the output of the unit 41. One output of the motion compensation unit 41 is subtracted from the output of the motion estimation unit 43, and the result is input to the discrete cosine transformer 21. The output of the discrete cosine converter 21 is quantized in quantizer 23. The output of the quantizer 23 is divided into two outputs 121 and 131 so that the output 121 is a downstream element such as a run lingth encoder for subsequent data compression and pre-transmission processing. A second output 131 is reconstructed in the encoded pixel macroblock and stored in the frame memory 42. In the encoder shown, the second output 131 generates a lossy version of the differential macroblock via an inverse quantizer 29 and an inverse discrete cosine transformer 31. This data is summed with the output of the motion compensation unit 41 to convey the lost value of the original picture to the frame memory 42.

As shown in Fig. 2, there are three types of images. "Intra coded pictures" or "I" pictures are encoded and transmitted as a whole, for which a motion vector does not need to be defined. An "I" picture functions as a source of motion vectors. "Predicted pictures" or "P" pictures are formed using motion vectors from previous pictures, which serve as a source of motion vectors for later pictures. Finally, a "bidirectional coded picture" or "B" picture is formed from two different pictures, a past picture and a future picture by motion vectors, which do not function as a source of the umbic vector. Motion vectors are generated from "I" and "P" pictures and used to form "P" and "B" pictures.

One way in which motion estimation is performed is to search from the macroblock 211 of the i < th > picture over a predetermined area of the next picture, as shown in FIG. 3, to find the best match macroblock 213. FIG. By moving the macroblock in this manner, a macroblock pattern for the (i + 1) th image as shown in Fig. 4 is obtained. In this way, the i < th > picture is slightly changed by, for example, the motion vector and the difference data to generate an (i + 1) < th > The encoding target is the motion vector and the difference data, not the (i + 1) th image. The motion vector shifts the image position between the images, and the difference data represents a change in chrominance, luminance, and saturation, that is, a change in shading and illumination.

In Fig. 3, the matching search is started in the i-th picture at the same position as in the (i + 1) -th picture. A search window is generated in the i th picture. Search for the best match in this search window. Once found, the best matched motion vector for the macroblock is coded. The coding of the best match macroblock includes how best the match vectors are displaced in the next vector in the motion vector, i. Difference data, also called " prediction error ", is also coded, which represents the difference in color difference and luminance between the current macroblock and the best match reference macroblock.

According to the definition in the MPEG standard, a video image can be compressed into any of three picture types, i.e., P or B picture. The I picture is compressed by removing spatial redundancy in the picture. The P picture is compressed by removing the time redundancy for the previously encoded (compressed) picture. The B picture is also compressed by eliminating time redundancy only for the two pictures that were previously encoded. The B picture can be compressed by interpolation of both reference pictures. Thereby, the B picture can achieve the highest level of compression among the three picture types.

Bidirectional interpolation in B pictures is defined as follows:

x is referred to as a pixel from the reference image I, and y is referred to as an image from the reference image P.

Bidirectional interpolated reference pixels

(x + y) / 2

to be. However, the "/" operator here means rounding after division.

A pixel is defined as a positive integer of 8 bits from 0 to 255. Therefore, rounding means that if the most significant bit of the remainder is 1, 1 is added to the least significant bit of the quotient. Only the quotient is stored as the result of the operation, and the remaining values are discarded. This is easily implemented with hardware such as the right shift element and the increment element.

In motion picture encoding, temporal redundancy must be identified and eliminated. This is done through the motion estimation process. A comparison circuit is used to search for a close match for the current picture within the search window. Therefore, one motion estimation is required for each of the two reference pictures, and one motion estimation is required for the interpolated reference, so all three motion estimations are required for the image B.

In the MPEG criteria, motion estimation is performed on macroblocks. The video image is divided into 16x16 pixel units called macroblocks. For this reason, the size of a recent close match macroblock should be 16x16 pixels. An 18x18 pixel area is required to form all possible half pixels around the identified recently matched (16x16) macroblock. The form of half pixel used for motion estimation is mentioned later.

Memory fetch is involved in the motion estimation of the B picture. One picture (720x480 pixels) requires 346K bytes of luminance data, which is stored in the processor external memory that performs data compression.

One method used in the prior art is to fetch a reference picture from an external memory and perform motion estimation therefrom. Then, the second reference picture is fetched, and motion estimation is performed in the same manner as in the first reference picture. Subsequently, recent matching reference data (18x18 pixel blocks) from the first and second reference pictures are fetched again and motion estimation is performed on the interpolated picture.

In the method disclosed in the above-mentioned US patent application Ser. No. 08 / 411,100, each reference image is fetched only once. The latest match reference data from each reference picture is stored in an on-chip buffer. Subsequently, interpolated motion estimation is performed using the data stored in the buffer. This reduces the memory band requirements.

If a recent match is identified from these three phonetic boundary searches, half-pixel reference data is calculated and motion estimation is performed again to retrieve the latest match-reference data at the half-pixel boundary. Of the three phone search, only the best match search proceeds in the half pixel search. There are three half-pixel interpolation methods used to calculate half-pixel reference data: horizontal half, vertical half, and full half.

Let a, b, c, and d be four adjacent pixels in the reference image as follows.

a b

c d

Horizontal half-pixels are formed as follows:

(a + b) / 2

(c + d) / 2

However, the "/" operator here means rounding after division.

Vertical half-pixels are formed as follows.

(a + c) / 2

(b + d) / 2

The former half-pixel is formed as follows:

(a + b + c + d) / 4

However, the "/" operator above means rounding after division.

First, half pixels should be formed from the buffer data for each reference picture. Subsequently, an interpolation half-pixel for the B picture is formed from this result to complete the motion estimation. Because of the rounding process, the order of operations must be maintained to obtain accurate interpolation results. In the prior art, one is required for each reference picture, and two sets of 18x18x18 bit buffers are required for B pictures.

If only one 18x18x11 bit buffer is required, the required amount of buffer can be reduced. The buffer has a 7 bit partial sum for each pixel, which is formed by adding the six most significant bits of the corresponding I and P picture pixels. The remaining four bits of each word in the buffer contain the two least significant bits of the corresponding I and P pixels.

As mentioned above, half-pixel motion estimation of a B picture requires half-pixel interpolation for each reference picture and then interpolates these half-pixel values across two reference pictures.

The following picture is used for an I frame. The following shows the pixels in the reference I picture:

I ₀ (x) I ₁ (x)

I ₁₀ (x) I ₁₁ (x)

Here, x is a value representing the bit position of each pixel and is an integer value of 1-8.

Cooling for horizontal half-pixels

I _HH = (I ₀ (1) I ₀ (2)… I ₀ (7) I ₀ (8) +

I ₁ (1) I ₁ (2). I ₁ (7) I ₁ (8)) / 2

Considering only the least significant 2 bits of the pixel pair, we get:

I ₀ (7) I ₀ (8)

+ I ₁ (7) I ₁ (8)

-------------------

I _COx (6) I _SOx (7) I _SOx (8)

I _SOx (8) is the "rounding" term resulting from the division operation of two in the half-pixel operation. From this, the equation for obtaining the horizontal half-pixel for the I image can be modified as follows.

Similarly, the equation for obtaining the vertical half-pixel for an I picture is:

Here, I _COx (6), I _SOx (7), I _SOx (8) are obtained as follows:

I _SxO (8) is the "rounding" term resulting from the division operation of two in the half-pixel operation.

Similarly, the equation for obtaining the half-pixel is:

Where Ic (5), Ic (6) and Is (7) are obtained as follows:

Divide by 4 from the top and discard Is (8), so the rounding term is Is (7).

Half-pixel operations on P-pictures can be handled by this same method.

Assume that the pixels of the reference P picture are as follows:

Here, x is a value representing the bit position of each pixel and is an integer value of 1 to 8.

In the same way as in the I picture, the equation for horizontal half interpolation in the P picture is:

Here, the formula for obtaining P _COx (6), P _SOx (7), and P _SOx (8) is as follows:

The equation for vertical half interpolation for a P picture is:

Here, P _CxO (6), P _SxO (7), and P _SxO (8) are obtained as follows.

Finally, the equation for obtaining the half-pixel for a P picture is:

Where Pc (5), Pc (6), and Ps (7) are obtained as follows:

A horizontal half pixel interpolated bidirectionally in a B picture is formed by interpolating horizontal half pixels from an I and P reference picture. Equations for I _HH and P _HH can be combined with bidirectional interpolation equations. The resulting interpolated horizontal half-pixels are:

Here, IP _COx 4, IP _COx 5, IP _COx 6, and IP _SOx 7 are obtained as follows.

And I _COx (6), I _SOx (7), I _SOx (8), P _COx (6), P _SOx (7), and P _SOx (8) are previously described in the formula half-pixel equation for I and P images. Same as defined.

The HH equation may be modified as follows.

Where IP ₀ (n) represents the subtotal of I ₀ (n) and P ₀ (n), IP ₁ (n) represents the subtotal of I ₁ (n) and P ₁ (n), and 'n' A number that indicates the position of a bit in a pixel byte, 0 to 6.

Also, IP ₀ (0) is the carry output of I ₀ (1: 6) + P ₀ (1: 6), and IP ₁ (0) is the I ₁ (1: 6) + P ₁ (1: 6) Carry output.

Thus, the interpolated horizontal half-pixel result is computed from the partial sum (IP ₀ (n) and IP ₁ (n)) from the I and P reference pictures and the least significant two bits of each pixel from the I and P reference pictures.

The same modifications are possible for VH and FH. Thus, the formula for VH is:

Where IP ₀ (n) represents the subtotal of I ₀ (n) and P ₀ (n), IP ₁₀ (n) represents the subtotal of I ₁₀ (n) and P ₁₀ (n), and 'n' is A number representing the position of a bit in a pixel byte, 0 to 6.

Also, IP ₀ (0) is the carry output of I ₀ (1: 6) + P ₀ (1: 6) and IP ₁₀ (0) is the carry of I ₁₀ (1: 6) + P ₁₀ (1: 6) As an output, IP _CxO (4), IP _CxO (5), IP _CXO (6), and IP _SxO (7) are obtained as follows.

Here, I _CxO (6), I _SxO (7), I _SxO (8), P _CxO (6), P _SxO (7), and P _SxO (8) are the same as previously defined.

Similarly, the formula for FH is as follows:

Here, IP ₀ (n) represents a partial sum of I ₀ (n) and P ₀ (n), IP ₁ (n) represents a partial sum of ₁ (n) and P ₁ (n), and IP ₁₀ (n) Represents the subtotal of I ₁₀ (n) and P ₁₀ (7), IP ₁₁ (n) represents the subtotal of I ₁₁ (n) and P ₁₁ (n), and 'n' represents the position of the bit in the pixel byte. It is 0-6 as a number to show.

Also, IP ₀ (0) is the carry output of I ₀ (1: 6) + P ₀ (1: 6), and IP ₁ (0) is the I ₁ (1: 6) + P ₁ (1: 6) Carry output, IP ₁₀ (0) is the carry output of I ₁₀ (1: 6) + P ₁₀ (1: 6), IP ₁₁ (0) is I ₁₁ (1: 6) + P ₁₁ (1: 6) ) Is the carry output of

The preceding IPc (3), IPc (4), IPc (5), and IP (6) are obtained as follows:

Here, Ic (5), Ic (6), Is (7), Pc (5), Pc (6), Ps (7) are the same as previously defined.

In this case, the least significant bit of each subtotal (IP ₀ (6), IP ₁ (6), IP ₁₀ (6), IP ₁₁ (6)) must also be considered in the above equation.

The above equations clearly indicate that a half pixel required for B picture motion estimation can be formed by first interpolating a pixel at a corresponding byte position of two reference pictures and then calculating a half pixel value from the interpolated pixel.

In addition, the above equations clearly indicate that the half-pixels required for B-picture motion estimation can be formed from "reduced" data sets rather than the two 18x18x8 pixel blocks that are typically required. The reduced data set consists of an 18x18x11 array of 7-bit subtotals obtained from the upper 6 bits of the corresponding pixels in the I and P pictures and the least significant 2 bits of each pixel. Using this method, the on-chip buffer space was reduced from 5814 bits (2x18x18x8) to 3564 bits (18x18x11), resulting in a 31% improvement in buffer space.

The method and apparatus of the present invention relate in particular to a "B" or bidirectional predictive picture, and provides a half-pixel calculation processor having reference data symmetrical from both past and future pictures. In other words, the data in both reference pictures have the same size and the same direction with respect to the current macroblock position. In particular, when the two macroblocks do not have the same size, the method and apparatus of the present invention reduces the larger search window to the size of the smaller search window. The half-pixel calculation processor is simplified because it only refers to one macroblock for past, future, or bidirectional pictures.

Edge detection circuitry is used to detect when the telephone best match criteria macroblock is at the edge of the picture or search window. One edge indicator is provided for each side (top, bottom, left, right) of the macroblock. It is also possible to have an active configuration in which up to two corner indicators are active for each picture, but the best matched bidirectional macroblock can cause all four corner indicators to respond.

Each reference picture has an independent set of corner indicators, which are set when the best match reference macroblock is located for each picture. For example, when the best matching reference macroblock is found at the upper right corner of the reference search window, the upper corner indicator and the right corner indicator react. In this case, the telephone station search processing circuit does not have reference data available for the pixels on the top and right sides of the reference macroblock, and therefore the half pixel values on the top and right cannot be calculated.

When the telephone search of the past and future reference search window is completed, the best matching reference data from each picture is passed to the processor for calculating the half-pixel bidirectional value. At this time, corner indicators from both reference picture searches are used to form a symmetric search window.

If the corner indicator is 'on' for only one of the two reference pictures, then the picture with the corner indicator off is often the reference data associated with that edge and the other pictures are not. That is, one side of the macroblock has a total of 18x18 pixels, and the other side does not. In this case, the picture in which the indicator is off has data, but excess data is excluded and not transmitted. In other words, the redundant reference data associated with the corner indicators is conveyed only when the corner indicators are off for macroblocks of both pictures. The surplus pixel data existing only in one image is excluded, so that a symmetrical search window is always formed for both images.

6 shows the reference data needed to calculate the half-pixel value when there is any unavailable reference data. 7 shows the reference data needed to calculate half pixel values using bi-directional edge conditions with symmetrical past / future data.

Although the present invention has been described with respect to certain preferred embodiments, it is not intended to limit the scope of the present invention. Accordingly, the scope of the invention is limited only by the claims that follow.

According to the present invention, a search window symmetrical with respect to the past picture and the future picture is always formed.

Claims (3)

A method for forming a bidirectional coded picture from two reference pictures, the method comprising: performing a memory fetch from each reference picture, the best match macroblock at full pixel boundaries; Finding, interpolating the telephone boundary boundary macroblock to form a bidirectional macroblock, and calculating half pixel reference image data from the bidirectional macroblock, wherein the best match macro Reducing excess pixels from the block such that the best matched macroblock is symmetrical in terms of size, shape, and orientation. 2. The method of claim 1, further comprising setting edge detectors to represent the surplus pixels. The method of claim 1, wherein the surplus pixels correspond to pixels that make the best matching macroblock different in size, shape, or direction.