JPH08149479A - Extremely low bit rate video encoding method - Google Patents

Abstract

PURPOSE: To improve reproduced picture quality by reducing encoding noise at an extremely low bit rate. CONSTITUTION: The first frame of a GOP is encoded as an intraframe (I), the remaining odd-number frames of the GOP are encoded as predictive frames (P), and even-number frames are shifted from the preceding frame based on a motion vector for which a mobile object is processed. Then, a still object and/or the background is encoded by a shift interpolated frame encoding method for interpolating it from the preceding and following frames of the odd-number frames (SI), the mobile object is completed by resetting an error zero motion vector based on motion vector distribution in motion vector processing, a random motion vector is removed, and the motion vector is equalized concerning each of respective mobile objects.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an ultra low bit rate video coding method applicable to various applications having general scene contents.

[0002]

Ultra-low bit rate audiovisual coding with bit rates of 10-64 kbps has a very high application demand in various industries. ITU-T and ISO /
The IEC has embarked on a new round of standardization activities for this demand and has attracted many researchers and engineers to join in the development of international standards and recommendations for ultra-efficient audio and video coding. The intention is to apply some new coding method and compare it with the waveform-based coding method such as DCT-based coding algorithm adopted in JPEG, H.261, MPEG1, MPEG2 international standard and recommendation. Achieving high compression performance. Among new coding methods, model-based coding, object-oriented analysis and synthesis coding, fractal coding, etc. have attracted much interest, and many researchers are still conducting research.

From a practical point of view, it is expected that very low bit rate video coding will find application in a variety of different applications such as video telephony or mobile telephony such as personal communications, database retrieval, multimedia services and the like. For those applications, the coding scheme must be independent of the content of the video scene and be feasible with very high compression rates, high image quality, and low hardware complexity. So far, all approaches are still in their infancy and no new coding approaches meet the above coding requirements.

Compared to those new approaches, the waveform-based coding approach is still a potential coding method for very low bit rate video coding, and some preliminary research work has been done in this field. It is being appreciated. A technical problem with conventional waveform-based coding is that the image content is clearly distorted when the bit rate is very low.

[0005]

However, applying waveform-based coding to very low bit rate video coding maintains good intra-picture quality for random access of subsequent frames and better prediction. To
Intra-frame coding uses many bits. This leaves very limited bits for subsequent interframe coding. Coding between all frames of a picture group results in very few average bits being given between each frame. Video played from such methods is usually very frustrating to the human eye,
There is the problem of having poor quality with obvious distortion.

The present invention, taking into consideration the above problems of the conventional video encoding method, provides an ultra-low bit rate video encoding method capable of reducing encoding noise at an ultra-low bit rate and improving reproduced image quality. The purpose is to do.

[0007]

SUMMARY OF THE INVENTION The present invention divides an original video sequence into a set of picture groups, each picture group (hereinafter abbreviated as GOP) consisting of a predetermined number of picture frames, starting from 1. For the first time, they are numbered in ascending order, whereby odd-numbered frames are referred to as odd-numbered frames, even-numbered frames are referred to as even-numbered frames, and the first frame of each GOP is based on the waveform transform encoding technique and is It is encoded by the intra-frame encoding method that encodes the frame independently of the frame, and GOP 3
Forward predictive frame coding, where the remaining odd frames starting from the second frame are coded by predicting the motion of the object in the current frame from the previous frame and then coding the prediction error to achieve high image quality It is an ultra low bit rate video coding method in which even frames of each GOP are coded by the shift interpolation coding method.

That is, the high efficiency coding scheme disclosed herein reduces the coded frames per GOP so that the bits saved from the uncoded frames can be used to improve the coding quality of the coded frame. It is based on the basic intention of.

Therefore, the above idea is further developed to reduce the bits used for even frames of a GOP so as to achieve bit savings and maintain the frame rate of the sequence.

Bit saving from even frames is achieved with a special coding method called shift interpolation coding. The only information encoded for even frames is the moving object and / or background motion vector of the frame. Still objects and / or backgrounds are interpolated from previous and subsequent frames, which are coded purely as intra or predictive frames. Since motion vectors consume very few bits compared to the original video content, a significant bit savings can be achieved in that way.

However, if the motion vector is directly applied from the motion estimation process, the reproduction quality becomes very poor due to random noise and error motion vector detection. Such reconstructed images usually have very obvious geometric distortions.

Therefore, in order to improve the motion compensation quality, 1
Generate a set of special motion vector processing methods to reduce random noise and correct error motion vectors. This process presupposes a rigid body with a translational motion model. The reproduced image quality is clearly improved to be visually acceptable to the human eye.

[0013]

According to the invention, for example, an input video sequence which has already been sub-sampled for a low frame rate according to different application requirements is first converted into a picture group (GO).
P). Each GOP consists of a variable number of frames, labeled with ascending numbers. The first frame is frame 1. All odd-numbered frames are called odd frames, and all even-numbered frames are called even frames.

The first frame of a GOP is coded with an intra coding scheme consisting of waveform transform coding, scalar quantization and variable length coding steps.

The remaining odd frames starting from the frame 3 are coded by a predictive coding method including motion estimation and motion compensation, error signal conversion coding, scalar quantization and variable length coding.

Both the above-mentioned intra coding and predictive coding are similar to existing standard systems such as MPEG1 and MPEG2. However, it introduces special processing to fine tune the coding system based on the special features of ultra-low bit rate video coding.

Even frames are coded by a shift interpolation method consisting of motion estimation and motion compensation, motion vector processing, stationary object and / or background interpolation.

In the present invention, motion vectors are obtained from 16x16 block-based matching for both predictive and shift interpolation coding. It has been found that the motion vectors obtained from block matching are often not accurate enough. Image noise, brightness change between adjacent frames,
There are some error motion vectors due to rotation, zoom and fast motion. These drawbacks of conventional motion estimation processes prevent the motion compensation from reproducing good video quality.

In order to overcome the drawbacks of the conventional motion estimation process, the motion vector used in shift interpolation coding is preprocessed in three stages. The first step is to replace the zero motion vector belonging to the moving object with a representative motion vector surrounding the zero motion vector. The purpose of this step is to remove the zero motion vector from the moving rigid object.
The second step is to remove the random motion vector that only occasionally appears in the current frame. This is to reduce the number of error motion vectors detected from the motion estimation process. The third step is to simply replace those motion vectors that are too large or too small compared to the peripheral motion vectors with a 3x3 median filter. This is to make the motion vector of the rigid object uniform so that the geometric distortion is reduced.

The processed motion vectors are DP
It is CM-encoded and transmitted as bits dedicated to even frames for shift interpolation encoding. Significant bit savings can be made from this encoding method.

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings showing its embodiments.

FIG. 1 is a diagram showing an overall coding method from the viewpoint of a video frame sequence in an ultra low bit rate video coding method according to an embodiment of the present invention. In FIG. 1, I indicates intra-frame (intra-frame) encoding, P indicates predictive frame encoding, and S indicates
I indicates shift interpolation frame coding. MV_1 indicates the motion vector of the P frame obtained from the preceding I or P frame, and MV_2 indicates the motion vector of the SI obtained from the preceding I or P frame.

For I to P frames, the waveform transform is applied to the original signal or the prediction error signal, respectively.
The transform coefficient is quantized and variable-length coded. The generated bitstream contains both transform coefficients and motion vector information.

For SI frames, there is no error signal to encode and no waveform conversion is required. Only the processed motion vector information is encoded in this frame.
The SI frame shifts the moving object from the previous frame,
Reconstruct by interpolating stationary objects and / or backgrounds from previous and subsequent frames.

FIG. 2 is a block diagram showing an encoder for the encoding system in the above embodiment. The input video sequence 1 is first divided into the intra-frame coding 3 or the inter-frame coding 4 including the predictive coding and the shift interpolation coding by the intra decision unit 2.

For intraframe coding, the coding procedure is processed through a transformer 5, a quantizer 6, a run length and a VLC encoder 7. Its output 8 is passed through the adder 9 since there is no motion information in the intraframe coding. The generated bitstream is output through the link 10.

To facilitate intra-frame coding, the encoder produces locally decoded frames. The output 11 of the quantizer 6 is linked with the operation of the inverse quantizer 12 and the inverse transformer 13. The output 14 of the inverse transformer 13 is stored in the local decoded frame memory 15 as a local decoded frame. For intraframe coding, there is no signal from path 28.

For predictive frame coding, input frame 4 first passes through motion vector estimator 16. The generated motion vector is sent to the motion vector encoder 22 via the link 18. Coded motion vector is link 2
Through 5 to an adder 9 where they are mixed with encoded error data 8 to produce an encoded bitstream 10. The motion vector generated by the motion vector estimator 16 is further sent to the motion compensator 17 which forms a prediction frame based on the motion vector and the reference frame stored in the local decoding frame memory 15 from the link 30. To be The motion compensation frame is output to the subtractor 21 via the link 29 and further produces the error signal 24. One input to subtractor 21 is the original frame through link 23. The generated error signal 24 is further processed by the converter 5, the quantizer 6, the run length and the VLC encoder 7. The output 8 of those processes is mixed with the coded motion vector 25 by the adder 9.

To facilitate shift-interpolation frame coding, the reproduced error signal is obtained through an inverse quantizer 12 and an inverse transformer 13. It is then mixed in adder 26 with the motion compensated frame from link 28. The output 27 of the adder 26 is a completely reproduced frame and is stored in the locally decoded frame memory 15.

Regarding the shift interpolation frame coding, the motion vector 19 generated from the motion vector estimator 16 is processed by the motion vector processor 20 and coded by the motion vector encoder 22. Its output 25 is only the coding information for the SI frame. When the motion vector encoded output 25 is sent to the adder 9, there is no data from the link 8 so the output 10 of the adder 9 is a pure motion vector bitstream. This is a feature of SI coded frames that saves many bits and yields many bits that can be used for I and P frame coding.

FIG. 3 is a block diagram of a coding type decoder in this embodiment. The input coded bit stream 34 is VLC and run length decoder
LC and run length decoding. For intraframes, the data is sent over the link 36 to the inverse quantizer 37.
To the inverse converter 38. The output of the inverse transformer 38 is the replayed intraframe and goes through the link 43 and the frame rearranger 56 to the decoder output 57.

For the predicted frame, the motion vector information is extracted from the input data 39 by the motion vector extractor 40. The motion vector is then sent via link 44 to motion compensator 49 with reference to the previous decoded I-P frame stored in decoded frame memory 42.
At the same time, the error signal is processed by the inverse quantizer 37 and the inverse transformer 38. The decoded error signal 51 is combined with the motion compensation output 50 by the adder 52. Link reconstructed frame to link 5
3 to the decoded frame memory 42 and the link 54 to both the frame reorderer 56.

With respect to the shift interpolation frame, the moving object is shifted from the preceding frame using the extracted motion vector 46, and the stationary object and / or the background of the shift interpolator 45 are interpolated from the preceding and succeeding frames. The leading and trailing frames are stored in the decoded frame memory 42, and the link 47
Through shift interpolation operation. The output of the shift interpolation frame is further sent to the frame rearranger 56 through the link 55.

The function of frame reorderer 56 is to reorder the frame output array according to the frame time stamp.

FIG. 4 is a block diagram of the motion vector processing device 20 shown in FIG. For the input motion vector 19 of a frame, the motion vector histogram calculator 61 is first used to calculate the histogram of the motion vector, and the number of repetitions of each motion vector value in the frame is checked.

The function of the rigid object homogenizer 62 of FIG. 4 is to homogenize moving objects by resetting the values of their zero motion vectors belonging to the rigid moving objects.
The verb "completion" means to keep the entity of the rigid object as a single entity. This behavior is based on the assumption that any moving object in the scene can be modeled as a 2D rigid object by translational motion.

Regarding the motion of perfecting the object, the motion vector condition of 10 is taken into consideration for the design of the algorithm. FIG. 6 shows ten different cases with the following description.

First, the representative value of the zero motion vector, which will be shown as "c" below, is calculated as the average value of the non-zero motion vectors surrounding the current zero motion vector. This operation is shown in FIG.
(A). 6 (b) -6 (k) show ten different distributions of peripheral non-zero motion vectors. In FIG. 6, the character “#” indicates a non-zero motion vector, the character “0” indicates a zero motion vector, the character “c” is the representative motion vector as described above, and the character “x” is zero or non-zero. A motion vector having a zero value is shown. The arrow "->" indicates a change from the original zero motion vector to the representative motion vector. Based on these ten rules, we have completed the task of designing efficient algorithms to complete moving objects.

The next step is to remove the random motion vector from the block matching operation. This process is performed by the random motion vector remover 63 shown in FIG. Random noise in motion vectors is generated by many different factors. The block matching operation itself may generate an error motion vector because the matching is often purely based on an unsatisfactory or perfect condition mean square error measurement that explicitly matches the image content of the block.
In addition, the presence of image noise, brightness changes, fast motion, rotation and zoom motion during subsequent frames can also cause random motion vectors.

A preferred random motion vector removal algorithm is shown in FIG. The letter "0" indicates a motion vector that has a zero value. The letter "#" indicates a motion vector that has a non-zero value. The character "lr" indicates a motion vector having a low repetition number. The letter "m" indicates the average value of the peripheral motion vector. The letter "+" indicates a motion vector having a positive value, and the letter "-" indicates a motion vector having a negative value.
Here, consider the motion vector distribution in three cases. A) If the non-zero motion vector is surrounded by all zero motion vectors, reset the non-zero motion vector to zero. B) The motion vector appears less than twice in one frame,
If it has a much different value than its peripheral motion vector, reset the motion vector to the average motion vector of its peripheral motion vectors. C) If the motion vector has a different sign (positive or negative) from its peripheral motion vector, reset the motion vector to the average motion vector of its peripheral motion vectors.

The next step in motion vector processing is to apply a median filter to the non-zero motion vector at block 64 of FIG. The median filter used is a 3x3 window that can effectively equalize motion vectors belonging to different moving objects. The processed motion vector is then processed by the motion vector encoder 22 of FIG.
It is encoded by the M method.

FIG. 5 shows the basic concept of the shift interpolation coding method. As shown, while the background (and / or stationary object) is being bi-directionally and linearly interpolated from the previous and subsequent frames, the moving object is shifted from the previous frame based on the processed motion vector.

As described above, by applying the specially processed motion vector to the new shift interpolation frame coding method to achieve a large amount of bit saving, coding of all frames can be avoided. Those saved bits can be efficiently used to enhance the coding quality of intra and predictive coded frames. Improved in-frame image quality is highly desirable for many applications of random access, error resilience, still frame analysis. The bits saved could also effectively improve the overall coding quality of the image sequence, making it suitable for applications such as video telephony and multimedia terminals.

The motion vector processing step introduced in the present invention has a clear image quality improving effect, and it is possible to apply a pure shift and interpolation scheme to encode even frames without the need to send an error signal. it can. Subjective visual quality appears to be acceptable for a particular application.

In the above embodiment, the average value is used as the representative value of the peripheral motion vector, but the present invention is not limited to this, and other representative values such as the median value may be used.

[0046]

As is apparent from the above description, the present invention has an advantage that coding noise at an ultra-low bit rate can be reduced and reproduced image quality can be improved.

[Brief description of drawings]

FIG. 1 is a diagram showing an overall encoding method from a frame point of view in an ultra-low bit rate video encoding method according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an encoder in an encoding system according to the same embodiment.

FIG. 3 is a block diagram showing a decoder in an encoding system according to the same embodiment.

FIG. 4 is a block diagram of a motion vector processing portion of FIG.

FIG. 5 is an explanatory diagram of a shift interpolation coding method according to the embodiment.

FIGS. 6A to 6K are diagrams showing respective processing conditions for perfecting a rigid object in the embodiment.

FIG. 7 is a diagram showing a rule for removing a random motion vector in the embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Input video sequence 2 Intra decision unit 5 Transformer 6 Quantizer 7 Run length and VLC encoder 12, 37 Inverse quantizer 13, 38 Inverse transformer 15 Local decoding frame memory 16 Motion vector estimator 17, 49 Motion Compensator 20 Motion vector processor 22 Motion vector encoder 35 VLC and run length decoder 40 Motion vector extractor 42 Decoded frame memory 45 Shift interpolator 56 Frame rearranger

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display H04B 14/06 G

Claims (7)

[Claims] 1. An original video sequence is divided into a set of picture groups, and each picture group (hereinafter GOP) is divided.
Consists of a predetermined number of picture frames, numbered in ascending order starting from 1, whereby odd numbered frames are called odd frames, even numbered frames are called even frames, and the first of each GOP Frame is encoded by an intra-frame encoding method that encodes the frame independently of other frames of the GOP based on the waveform transform encoding method, and the remaining odd frames starting from the third frame of the GOP are encoded. , Predicting the motion of the object in the current frame from the previous frame, and coding the prediction error to achieve high image quality, and coding with the forward predictive frame coding method, which encodes the frame, and the even number of each GOP. An ultra-low bit rate video encoding method, characterized in that a frame is encoded by a shift interpolation encoding method. 2. The shift interpolation coding method shifts a moving object from a preceding frame to a current frame at a position indicated by a processed motion vector, and a still object and / or a background of the current frame are adjacent to the intra coding or the prediction coding. The ultra-low bit rate video encoding method according to claim 1, wherein the encoding is performed from an encoded frame. 3. The processed motion vector is for homogenizing a moving object by resetting an error zero motion vector, removing a random motion vector, and homogenizing a moving object by filtering the motion vector. The ultra-low bit rate video encoding method according to claim 2. 4. The process of homogenizing a moving object by resetting an erroneous zero motion vector comprises:
4. The ultra low bit rate video coding method according to claim 3, wherein if one of the conditions of 0 is satisfied, the zero motion vector is set to a representative value calculated from motion vectors surrounding the zero motion vector. 5. The process of removing random motion vectors checks motion vectors surrounding non-zero motion vectors,
If all the peripheral motion vectors are zero, set the motion vector to zero, check the motion vectors surrounding the motion vector that has a value if the occurrence frequency is less than a certain fixed number in the frame,
If all the differences between the motion vector and its peripheral motion vector are greater than a predetermined threshold, set the motion vector to the representative value of said peripheral motion vector and check the motion vectors surrounding the positive motion vector. , If all peripheral motion vectors are negative, set the motion vector to the representative value of the peripheral motion vectors, check the motion vectors surrounding the negative motion vector, and if all peripheral motion vectors are positive, 4. The ultra low bit rate video encoding method according to claim 3, wherein the representative value of the peripheral motion vector is set. 6. The step of homogenizing a moving object by filtering the motion vector comprises applying a 3 × 3 median filter to the processed motion vector after resetting the error zero motion vector to obtain all motion vectors of the rigid object. The ultra-low bit rate video coding method according to claim 3, wherein the random motion vector is removed so as to be uniform. 7. The operation of interpolating a stationary object and / or background comprises bi-directionally and linearly interpolating a current pixel value in the time domain from past and future pixel values at the same pixel location. Ultra low bit rate video encoding method described.