ES2780055T3 - Improved interpolation of video compression frames - Google Patents

Abstract

A method for compression of video images using direct mode prediction, including: providing a sequence of bidirectionally predicted and predicted frames, each comprising pixel values arranged in image areas, where at least one of the image areas in at least one of the bidirectionally predicted frames is predicted from image areas in one or more reference frames using direct mode prediction; and determining a predicted motion vector for a bidirectionally predicted frame with a direct mode prediction frame distance method, wherein the frame distance method comprises: given a predicted image area in a first reference frame, the frame bidirectionally predicted with another predicted image area, and a motion vector between the predicted image area in the first reference frame and an image area in a second reference frame, the predicted motion vector is the motion vector multiplied by a frame-scale fraction, where the numerator of the frame-scale fraction is equal to a first distance between the bidirectionally predicted frame of the other predicted image area and the first reference frame and the denominator of the scale fraction frame is equal to a second distance between the first reference frame and the second reference frame referenced by the first square reference river, wherein both the first and second reference frames are in a display order that is prior to a bidirectionally predicted frame display order of the predicted image area.

Description

DESCRIPTION

Improved interpolation of video compression frames

Technical field

This invention relates to a video compression method and, more particularly, to a method of improved interpolation of video compression frames in MPEG-type coding systems.

Background

MPEG video compression

MPEG-2 and MPEG-4 are international video compression standards that define respective video syntax that provide an efficient way of representing image sequences in the form of more compact encoded data. An introduction to MPEG-2 is provided by Tudor PN, “MPEG-2 video compression tutorial”, IEE Colloquium on MPEG-2 (Abstract No. 1995/012), London, UK, January 24, 1995, pages 2 / 1 to 2/8. The language of the encoded bits is "syntax." For example, a few tokens may represent an entire block of samples (eg 64 samples for MPEG-2). Both MPEG standards also describe a decoding (reconstruction) process where the encoded bits are mapped from the compact representation to an approximation of the original format of the image sequence. For example, a mark in the encoded bit stream may indicate whether the following bits are to be preceded by a prediction algorithm before being decoded with a discrete cosine transform (DCT) algorithm. The algorithms that comprise the decoding process are governed by the semantics defined by these MPEG standards. This syntax can be applied to exploit common video characteristics such as special redundancy, temporal redundancy, smooth motion, spatial masking, etc. An MPEG decoder must be able to parse and decode an incoming data stream, but as long as the data stream complies with the corresponding MPEG syntax, a wide variety of possible data structures and compression techniques can be used (although technically this it deviates from the standard since the semantics are not conforming). It is also possible to carry the necessary semantics within an alternative syntax.

These MPEG standards use a variety of compression methods, including intra-frame and inter-frame methods. In most video scenes, the background remains relatively stable while the action takes place in the foreground. The background can move, but much of the scene is often redundant. These MPEG standards begin compression by creating a reference frame called "intra" frame or "l" frame. The I frames are compressed without reference to other frames and thus contain an entire frame of video information. I-frames provide entry points into a random access data bit stream, but can only be moderately compressed. Typically, the data representing I frames is put into the bit stream every 12 to 15 frames (although it is also useful in some circumstances to use a much wider spacing between I frames). Thereafter, since only a small part of the frames falling between the reference I frames are different from the I frames they delimit, only the image differences are captured, compressed and stored. Two types of frames are used for such differences, predicted frames (P frames) and bidirectional predicted (or interpolated) frames (B frames).

P frames are generally coded with reference to a past frame (either an earlier I frame or a prior P frame), and are generally used as a reference for later P frames. The P frames receive a fairly high amount of compression. B frames provide the highest amount of compression, but require both a past and future reference frame in order to be encoded. The P and I tables are "referenced tables" because they can be referenced by P or B tables.

Macroblocks are regions of image pixels. For MPEG-2, a macroblock is a 16x16 pixel grouping of four 8x8 DCT blocks, along with a motion vector for P frames, and one or two motion vectors for B frames. The macroblocks within the P frames are they can encode individually using either intra-frame or inter-frame (predicted) coding. Macroblocks within B-frames can be individually encoded using intra-frame encoding, predicted forward encoding, predicted backward encoding, or both predicted forward and backward encoding (ie, bidirectionally interpolated). A slightly different but similar structure is used in MPEG-4 video encoding. After encoding, an MPEG data bit stream comprises a sequence of I, P, and B frames. A sequence can consist of almost any pattern of I, P, and B frames (there are a few minor semantic restrictions on their placement). However, it is common in industrial practice to have a fixed checker pattern (eg IBBPBBPBBPBBPBB).

Motion vector prediction

In MPEG-2 and MPEG-4 (and similar standards, such as H.263), the use of type B frames (bidirectionally predicted) has been found to benefit compression efficiency. The motion vectors for each macroblock in such frames can be predicted by any one of the following three methods: Mode 1: Predicted forward from the previous I or P frame (that is, a predicted frame not bidirectionally). Mode 2: Predicted backward from posterior I or P frame.

Mode 3: Bidirectionally predicted from both the posterior and anterior I or P frame

Mode 1 is identical to the forward prediction method used for P frames. Mode 2 is the same concept, except that it works backward from a later frame. Mode 3 is an interpolative mode that combines information from both before and after frames.

In addition to these three modes, MPEG-4 also supports a second interpolative motion vector prediction mode for B-frames: direct-mode prediction using the motion vector of the subsequent P-frame, plus a delta value (if the motion vector of the Co-located macroblock P is split in 8x8 mode, resulting in four motion vectors for the 16x16 macroblock, then the delta applies to all four independent motion vectors in frame B). The motion vector of the posterior P-frame points to the anterior P or I-frame. A ratio is used to weight the motion vector of the back P-frame. The ratio is the relative time position of the current B frame with respect to the previous P and P (or I) frames.

FIG. 1 is a prior art MPEG-4 forward mode motion vector and frame timeline. The concept of MPEG-4 direct mode (mode 4) is that the movement of a macroblock in each intermediate B frame is probably close to the movement that was used to encode the same location in the next P frame. A delta is used to make corrections less than a proportional motion vector derived from the corresponding motion vector (MV) 103 for the posterior P-frame. Shown in FIG. 1 is the proportional weight given to the motion vectors 101, 102 for each intermediate B frame 104a, 104b as a function of the "time distance" between the previous P or I frame 105 and the next P frame 106. The vector of motion 101, 102 assigned to a corresponding intermediate B frame 104a, 104b is equal to the assigned weight value (1/3 and 2/3, respectively) times the motion vector 103 for the next P frame, plus the delta value.

With MPEG-2, all prediction modes for B-frames are tested on encoding, and compared to find the best prediction for each macroblock. If neither prediction is good, then the macroblock is coded autonomously as a macroblock "I" (for "intra"). The encoding mode is selected as the best mode among forward (mode 1), backward (mode 2) and bidirectional (mode 3), or intracoding. With MPEG-4, the option of intracoding is not allowed. Instead, the direct mode becomes the fourth option. Again, the best encoding mode is chosen, based on some best match criteria. In the reference MPEG-2 and MPEG-4 software encoders, the best match is determined using a DC match (Absolute Difference Sum or "SAD").

The number of successive B-frames in an encoded data bit stream is determined by the value of the MPEG parameter "M". M minus one is the number of B frames between each P frame and the next P (or I). Thus, for M = 3, there are two B frames between each P (or I) frame, as illustrated in FIG. 1. The main limitation in restricting the value of M, and therefore the number of sequential B-frames, is that the amount of change of movement between P (or I) frames becomes large. High numbers of B frames means longer amounts of time between P (or I) frames. Thus, the efficiency and limitations of the coding range of the motion vectors create the ultimate limit on the number of intermediate B-frames.

It is also significant to note that the P-frames carry "energy of change" forward with the moving picture stream, since each decoded P-frame is used as a starting point to predict the next subsequent P-frame. The B frames, however, are discarded after use. Thus, any bit used to create B frames is used only for that frame, and does not provide corrections to aid decoding of later frames, unlike P frames.

A document by Flierl et al. "A locally optimal design algorithm for block-based multi-hypothesis motioncompensated prediction", Proceedings of the Data Compression Conference, Snowbird, Ut., USA, March 30-April 1, 1998, Los Alamitos, Ca ., USA, Soc. De Comput. IEEE, USA, March 30, 1998, pages 239 to 248 (ISBN-978-0-8186-8406-7) describes motion-compensated prediction for video coding in which a predicted frame can be encoded by reference to two or more referable frames earlier in the sequence, such as in bidirectional predicted frames (frames B). The paper generalizes this for a rate distortion frame, and proposes the use of multiple frames in the prediction operation.

The following additional documents were cited during the examination:

BJONTEGAARD G. (ED.): “H.26L TEST MODEL LONG TERM NUMBER 5 (TML-5) DRAFT0”, ITU-T TELECOMMUNICATIONS STANDARDIZATION SECTOR, GENEVA, CH, 11TH SESSION, PORTLAND, OR, EE .USA, 22-25 AUGUST 2000, 22 August 2000 (22-08-2000), pages 1 -31,

ITU STUDY GROUP 16 - VIDEO CODING EXPERT GROUP - ISO / IEC MPEG AND ITU-T VCEG (ISO / IEC JTC1 / SC29 / WG11 E ITU-T SG16 Q6), 11TH MEETING, PORTLAND, OR, EE .USA, 22-25 AUGUST 2000, N ° q15k44, 16 August 2000 (08-16-2000), pages 1-2,

HANNUKSELA: “Generalized B / MH-Picture Averaging”, ITU STUDY GROUP 16 - VIDEO ENCODING EXPERT GROUP -I SO / IEC MPEG E ITU-T VCEG (ISO / IEC JTC1 / SC29 / WG11 E ITU- T SG16 Q6), 3rd MEETING, FAIRFAX, VA, USA, MAY 6-10, 2002, N ° JVT-C077, May 6, 2002 (05-06-2002), pages 1-8,

GISLE BJOONTEGAARD ET AL: “H.26L Test Model Long Term Number 4 (TML-4)”, 10. VCEG MEETING; 16 05-2000 - 19-05-2000; OSAKA, JP; (ITU-T SG.16 VIDEO CODING EXPERTS GROUP), N ° q 15j72d0, June 16, 2000 (06-16-2000) ISSN: 0000-0464;

KIKUCHI Y .: “Improved multiframe motion compensation using frame interpolation”, ITU STUDY GROUP 16 - VIDEO ENCODING EXPERT GROUP - ISO / IEC MPEG E ITU-T VCEG (ISO / IEC JTC1 / SC29 / WG11 E ITU -T SG16 Q6), 2nd MEETING, GENEVA, CH, JANUARY 29-FEBRUARY 1, 2002, N ° JVT-B075, January 29, 2002 (01-29-2002), pages 1-8

Compendium

The invention is directed to methods and systems that are defined in claims 1 to 4.

Other aspects described include methods that include:

• A video image compression system having a sequence of referable frames comprising image regions, wherein at least one image region of at least one predicted frame is encoded by reference to one or more referable frames in order of display, where at least one such referable frame is not the closest previous referable frame in display order to the at least one predicted frame.

• A video image compression system having a sequence of referenced frames comprising macroblocks, wherein at least one macroblock within at least one predicted frame is encoded by interpolation of two or more referenceable frames.

• A video image compression system having a sequence of bidirectional predicted and referable frames comprising image regions, wherein at least one image region of at least one bidirectional predicted frame is encoded to include more than two vectors of motion, each motion vector such that it refers to a corresponding image region in at least one referable frame.

• A video image compression system having a sequence of referable frames comprising image regions, wherein at least one image region of at least one predicted frame is encoded to include at least two motion vectors, each of such motion vectors referencing a corresponding image region in a referable frame, where each such image region of such at least one predicted frame is encoded by interpolation of two or more referable frames.

• A video image compression system having a sequence of bidirectional predicted and referable frames comprising image regions, wherein at least one image region of at least one bidirectional predicted frame is encoded as an uneven weight of regions of image selected from two or more referenced frames.

• A video image compression system having a sequence of bi-directional predicted and referable frames comprising image regions, wherein at least one image region of at least one bidirectional predicted frame is encoded by interpolation of two or more frames referable, where at least one of the two or more referable frames is separated from the bidirectional predicted frame by at least one intermediate referable frame in display order, and where such at least one image region is encoded as an unequal weight of image regions selected from such at least two or more reference tables.

• A video image compression system having a sequence of bi-directional predicted and referenceable frames comprising image regions, wherein at least one image region of at least one bidirectional predicted frame is encoded by interpolation of two or more frames referable, where at least one of the two or more referential frames is separated from the bidirectional predicted frame by at least one intermediate back referable frame in display order.

• A video image compression system having a sequence of bi-directional predicted and referenceable frames comprising image regions, wherein at least one image region of at least one bidirectional predicted frame is encoded as unequal weighting of selected image regions from two or more referenced frames.

A video image compression system having a sequence of bidirectional predicted and predicted frames, each comprising pixel values arranged in macroblocks, wherein at least one macroblock within a bidirectional predicted frame is determined using direct mode prediction in based on motion vectors of two or more predicted frames.

A video image compression system having a sequence of bidirectional predicted and referable frames, each comprising pixel values arranged in macroblocks, wherein at least one macroblock within a bidirectional predicted frame is determined using direct mode prediction in based on motion vectors of one or more predicted frames in display order, wherein at least one of such one or more predicted frames is prior in display order to the bidirectional predicted frame.

A video image compression system having a sequence of bidirectional predicted and referable frames, each comprising pixel values arranged in macroblocks, wherein at least one macroblock within a bidirectional predicted frame is determined using direct mode prediction in based on motion vectors of one or more predicted frames, wherein at least one of such one or more predicted frames is subsequent in display order to the bidirectional predicted frame and separated from the bidirectional predicted frame by at least one intermediate referable frame.

A video image compression system having a frame sequence comprising a plurality of image regions having a DC value, each such image region comprising pixels each having a pixel value of AC, wherein at least one of the DC value and AC pixel values of at least one image region of at least one frame is determined as a weighted interpolation of the respective DC values and corresponding AC pixel values of at least one other frame .

A video image compression system having a sequence of referable frames comprising a plurality of image regions having a DC value, each such image region comprising pixels, each having a pixel value of AC, in where at least one of the DC value and AC pixel values of at least one image region of at least one predicted frame is interpolated from the respective DC values and corresponding AC pixel values of two or more frames referable.

Improve the image quality of a sequence of two or more bidirectional predicted intermediate frames in a video image compression system, each frame comprising a plurality of image regions having a DC value, each image region such comprising pixels each having an AC pixel value, including at least one of the following: determining the AC pixel values of each image region of a bidirectional predicted midframe as a first weighted ratio of corresponding AC pixel values in referenceable frames that delimit the sequence of bidirectionally predicted intermediate frames; and determining the DC value of each image region of such a bidirectional predicted midframe as a second weighted ratio of the corresponding DC values in referenceable frames delimiting the sequence of bidirectional predicted midframes.

A video image compression system having a sequence of frames comprising a plurality of pixels having an initial representation, wherein the pixels of at least one frame are interpolated from corresponding pixels of at least two other frames, wherein such corresponding pixels from the at least two other frames are interpolated while being transformed to a different representation, and the resulting interpolated pixels are transformed back to the initial representation.

In a video image compression system that has a sequence of bi-directional predicted and referable frames, dynamically determining a code pattern of such frames that have a variable number of bi-directional predicted frames, including: selecting an initial sequence beginning with a frame referable, having at least one immediately following two-way predicted frame, and ending in a referable frame; add a referable frame at the end of the initial sequence to create a test sequence; evaluating the test sequence against selected evaluation criteria; For each successful evaluation step in the test sequence, insert a bidirectional box before the added reference box and repeat the evaluation step; and if the evaluation of the test sequence is unsuccessful, then accept the previous test sequence as the current code pattern.

• A video image compression system having a sequence of referenced frames separated by at least one bidirectional predicted frame, wherein the number of such bidirectional predicted frames varies in such sequence, and wherein at least one image region of at least minus one such bidirectional predicted frame is determined using uneven weighting of pixel values corresponding to at least two referable frames.

• A video image compression system having a sequence of frames encoded by an encoder for decoding by a decoder, wherein at least one image region of at least one frame is based on weighted interpolations of two or more other frames , such weighted interpolations that are based on at least one set of weights available to the encoder and a decoder, wherein a designation of a selected one from such at least one set of weights is communicated to a decoder from the encoder to select one or more currently active weights.

• A video image compression system having a sequence of frames encoded by an encoder for decoding by a decoder, wherein at least one image region of at least one frame is based on weighted interpolations of two or more other frames , such weighted interpolations that are based on at least one set of weights, wherein at least one set of weights is downloaded to a decoder and, thereafter, a designation for a selected one of such at least one set of weights is communicated to a decoder from the encoder to select one or more currently active weights.

• A video image compression system having a sequence of referenceable frames encoded by an encoder for decoding by a decoder, wherein the predicted frames in the sequence of referenceable frames are transmitted by the encoder to the decoder in a delivery order that it differs from the display order of such predicted frames after decoding. • A video image compression system having a reference frame sequence comprising pixels arranged in image regions, in which at least one image region of at least one predicted frame is encoded by reference to two or more reference frames , wherein each such image region is determined using unequal weighting of pixel values corresponding to such two or more referenced frames.

• A video image compression system having a sequence of predicted, bidirectional predicted, and intraframe frames, each comprising image regions, where at least one filter selected from the set of sharpening and smoothing filters is applied to at least an image region of a bidirectional predicted or predicted frame during motion vector compensated prediction of such a bidirectional predicted or predicted frame.

Details of one or more embodiments of the invention are set forth in the accompanying drawings and description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.

Description of the drawings

FIG. 1 is a prior art MPEG-4 forward mode motion vector and frame timeline.

FIG. 2 is a timeline of frames and proportional pixel weight values.

FIG. 3 is a timeline of mixed, proportional, and equal pixel weights and frames.

FIG. 4 is a flow chart showing an illustrative example.

FIG. 5 is a diagram showing an example of multiple previous references by a current P frame to two previous P frames, and to a previous I frame.

FIG. 6A is a diagram of a typical prior art MPEG-2 coding pattern, showing a constant number of B frames between bounding I frames and / or P frames.

Figure 6B is a diagram of a theoretically possible MPEG-4 video coding pattern of the prior art, showing a variable number of B frames between I frames and / or delimiting P frames, as well as a variable distance between I frames .

FIG. 7 is a code pattern diagram.

FIG. 8 is a flow chart showing an embodiment of an interpolation method with DC interpolation that is other than AC interpolation.

FIG. 9 is a flow chart showing one embodiment of a method for luminance pixel interpolation using an alternative representation.

FIG. 10 is a flow chart showing an embodiment of a method for chroma pixel interpolation using an alternative representation.

FIG. 11 is a diagram showing unique motion vector region sizes for each of two P frames.

FIG. 12 is a diagram showing a sequence of P and B frames with interpolation weights for the B frames determined as a function of the distance from a 2-distance back P-frame referencing a 1-distance back P-frame.

FIG. 13 is a diagram showing a sequence of P and B frames with interpolation weights for the B frames determined as a function of the distance from a 1-distance back P-frame referring to a 2-distance leading P-frame.

FIG. 14 is a diagram showing a sequence of P and B frames in which a subsequent P frame has multiple motion vectors referring to earlier P frames.

FIG. 15 is a diagram showing a sequence of P and B frames in which a closer back P frame has a motion vector that refers to an earlier P frame, and a next closer P frame has multiple motion vectors that make reference to previous P tables.

FIG. 16 is a diagram showing a sequence of P and B frames in which a closer previous P frame has a motion vector that references a previous P frame.

FIG. 17 is a diagram showing a sequence of P and B frames in which a closer previous P frame has two motion vectors referencing previous P frames.

FIG. 18 is a diagram showing a sequence of P and B frames in which a closer previous P frame has a motion vector referring to a previous P frame.

FIG. 19 is a frame sequence showing the case of three P frames P1, P2 and P3, where P3 uses an interpolated reference with two motion vectors, one for each of P1 and P2.

FIG. 20 is a frame sequence showing the case of four P frames P1, P2, P3, and P4, where P4 uses an interpolated reference with three motion vectors, one for each of P1, P2, and P3.

FIG. 21 is a diagram showing a sequence of P and B frames in which various P frames have one or more motion vectors that refer to various previous P frames, and showing different weights assigned to respective forward and backward references by a box B particular.

FIG. 22 is a diagram showing a sequence of P and B frames in which the bit stream order of the P frames differs from the display order.

FIG. 23 is a diagram showing a sequence of P and B frames with assigned weights.

FIG. 24 is a graph of the position of an object within a box versus time.

Similar reference symbols throughout the various drawings indicate similar elements.

Detailed description

Overview

It is common practice to use a value for M of 3, which provides two B frames between each P (or I) frame. However, M = 2, and M = 4 or higher, are all useful. It is particularly important to note that the value of M (the number of B frames plus 1) also supports a natural relationship with the frame rate. At 24 frames per second (fps), the speed of movies, 1/24 of the second time distance between frames can result in substantial frame-to-frame image shifts. At 60fps, 72fps, or higher frame rates, however, the time distance between adjacent frames becomes correspondingly reduced. The result is that higher numbers of B frames (ie, higher values of M) become useful and beneficial in compression efficiency as the frame rate is increased.

Both MPEG-2 and MPEG-4 video compression use an oversimplified interpolation method.

For example, for mode 3, the bidirectional prediction for each macroblock in a frame is an equal average of the macroblocks in the preceding and preceding frames, as they move through the two corresponding motion vectors. This equal average is appropriate for M = 2 (that is, unique intermediate B frames), since frame B will be equidistant in time from the preceding and following P (or I) frames. However, for all higher values of M, only symmetrically centered B frames (ie, the middle frame if M = 4, 6, 8, etc.) will be optimally predicted using equal weighting. Similarly, in MPEG-4 direct mode 4, even though the motion vectors are proportionally weighted, the predicted pixel values for the middle B frame are an equal proportion of the corresponding pixels from the P (or I) frame above and the back P.

Thus, it is an improvement to apply an appropriate proportional weight, for M> 2, to the predicted pixel values for each B frame. The proportional weight for each pixel in a current B frame corresponds to the relative position of the current B frame with with respect to the previous and later P (or I) tables. Thus, if M = 3, the first frame B would use 2/3 of the corresponding pixel value (adjusted motion vector) from the previous frame, and 1/3 of the corresponding pixel value from the subsequent frame (adjusted motion vector) . FIG. 2 is a timeline of frames and proportional pixel weight values. The pixel values within each macroblock of each intermediate B frame 201a, 201b are weighted as a function of the “distance” between the previous P or I frame A and the next P or IB frame, with higher weighting according to frames Closest I or P. That is, each pixel value of a bidirectionally predicted B frame is a weighted combination of the corresponding pixel values of the bounding non-bidirectionally predicted frames A and B. In this example, for M = 3, the weight for the first frame B 201a is equal to 2 / 3A 1 / 3B; the weight for the second frame B 201b is equal to 1 / 3A 2 / 3B. The average weight is also shown as it would be assigned under conventional MPEG systems; MPEG-1, 2 and 4 weighting for each B frame 201a, 201b would be equal to (A B) / 2.

Extended dynamic range and contrast range application

If M is greater than 2, proportional weighting of the pixel values in intermediate B frames will improve the effectiveness of bidirectional (mode 3) and direct (MPEG-4 mode 4) encoding in many cases. Example cases include common movie and video editing effects such as fades and cross dissolves. These types of video effects are problematic coding cases for both MPEG-2 and MPEG-4 due to the use of a simple DC matching algorithm, and the common use of M = 3 (i.e. two intermediate B-frames) , resulting in equal proportions for B frames. The coding of such cases is improved using proportional B frame interpolation.

Proportional B-frame interpolation also has direct application to improving coding efficiency to extend dynamic range and contrast. A common case in image coding is a change in lighting. This occurs when an object gradually moves into (or out of) the shadow (soft shadow edges). If a logarithmic coding representation is used for the brightness (incorporated by logarithmic Y luminance, for example), then a change in the brightness of the illumination will be a DC offset change. If the brightness of the lighting drops by half, the pixel values will all be reduced by an equal amount. Thus, to encode this change, an AC match should be found, and an encoded DC difference applied to the region. Such a DC difference that is encoded in a P frame should also be applied proportionally in each intermediate B frame. (See copending US Patent Application No. 09 / 905,039, entitled "Method and System for Improving Compressed Image Chroma Information", published as US 2003/0112863 A1, for additional information on logarithmic encoding representations).

In addition to changes in lighting, changes in contrast also benefit from proportional B-frame interpolation. For example, as an airplane moves toward a viewer out of a cloud or haze, its contrast will gradually increase. This increase in contrast will be expressed as an increased amplitude in the AC coefficients of the DCT in the corresponding P-frame coded macroblocks. Again, the contrast changes in the intermediate B-frames will be more closely approximated by proportional interpolation, thereby improving coding efficiency.

Improvements in dynamic range and contrast coding efficiency using proportional B-frame interpolation become more and more significant as frame rates become higher and as the value of M.

Applying high M values to temporary layers

The use of embodiments of the invention allows an increase in the value of M and therefore in the number of B frames between delimiting P and / or I frames, while maintaining or obtaining encoding efficiency. Such use benefits a number of applications, including temporary layers. For example, in our US Patent No. 5,988,863, entitled "Temporal and Resolution Layering for Advanced Television", it was noted that B-frames are a suitable mechanism for layered temporal rates (frames). The flexibility of such rates is related to the number of consecutive B frames available. For example, single B-frames (M = 2) can support a 36 fps decoded temporal layer within a 72 fps stream or a 30 fps decoded temporal layer within a 60 fps stream. Triple B frames (M = 4) can support both 36 fps and 18 fps decoded time layers within a 72 fps stream, and 30 fps and 15 fps decoded time layers within a 60 fps stream. Using M = 10 within a 120fps stream can support 12fps, 24fps and 60fps decoded temporal layers. M = 4 can also be used with 144fps stream to provide temporal layers decoded at 72fps and 36fps.

As an enhancement to taking each N-order frame, multiple frames can be decoded and proportionally mixed at 120 fps or 72 fps, as described in copending US Patent Application No. 09 / 545,233, entitled "Enhancements to Temporal and Resolution Layering ”see WO 01/17787 A1, to improve the motion blur characteristics of 24 fps results.

Even higher frame rates can be synthesized using the methods described in copending US Patent Application No. 09 / 435,277, entitled "System and Method for Motion Compensation and Frame Rate Conversion" (see WO 01 / 35657 A1).

For example, a stock 72fps camera can be used with motion compensated frame rate conversion to create an effective frame rate of 288 frames per second. Using M = 12, both 48fps and 24fps frame rates can be derived, as well as other useful rates such as 144fps, 96fps, and 32fps (and of course the original 72fps). Frame rate conversions using this method do not need to be integral multiples. For example, an effective rate of 120 fps can be created from a 72 fps source, and then it can be used as the source for both 60 fps and 24 fps (using M = 10).

Thus, there are benefits of temporal layers to optimize the performance of B-frame interpolation. The proportional B-frame interpolation described above makes higher consecutive B-frame numbers work more efficiently, thereby enabling these benefits.

Mixed B-frame interpolation ratios

One reason that an equal average weight has been used in conventional systems as the motion compensation mode predictor for B-frame pixel values is that the P (or I) frame before or after a particular B-frame may be noisy and therefore represent an imperfect match. Equal mixing will optimize noise reduction in the interpolated motion compensation block. There is a residual difference that is encoded using the quantized DCT function. Of course, the better the motion compensated ratio match, the smaller the residual bit difference that will be required, and the higher the resulting image quality.

In cases where there are objects moving in and out of shadow or haze, a true ratio where M> 2 provides a better prediction. However, when lighting and contrast changes are not occurring, equal weighting may turn out to be a better predictor, since the errors of moving a macroblock forward along a motion vector will be averaged with the errors of the forward-shifted block. back, thereby reducing the errors in each by half.Even so, macroblocks of B frames closer to a P (or I) frame are more likely to correlate more with that frame than with a P (or I) frame. ) more distant.

Thus, it is desirable in some circumstances, such as regional contrast or brightness shift, to use a true ratio for the B frame macroblock pixel weighting (for both luminance and color), as described above. In other circumstances, it may be more optimal to use equal ratios, as in MPEG-2 and MPEG-4.

Thus, an exemplary look uses a mixture of these two ratio techniques (equal average ratio and frame distance) for interpolation of B-frame pixels. For example, in the case M = 3, 3/4 of the proportions 1/3 and 2/3 can be mixed with 1/4 of the equal average, resulting in the two ratios being 3/8 and 5/8. This technique can be generalized using a “mixing factor” F:

Weighting = F * (Square Distance Proportional Weighting) (1-F) * (Average Equal Weighting) The useful range of the mixing factor F is from 1, which indicates purely proportional interpolation, to 0, which indicates purely equal average ( reverse assignment of values can also be used).

FIG. 3 is a timeline of mixed, proportional, and equal pixel weights and frames. The pixel values of each macroblock in each intermediate B frame 301a, 301b are weighted as a function of the "time distance" between the previous P or I frame A and the next P or IB frame, and as a function of the equal average of A and B. In this example, for M = 3 and a blending factor F = 3/4, the mixed weight for the first frame B 301a equals 5 / 8A 3 / 8B (that is, 3/4 of the proportional weight of 2 / 3A 1 / 3B, plus 1/4 of the equal average weight of (AB) / 2). Similarly, the weight for the second frame B 301 b equals 3 / 8A 5 / 8B.

The value of the mixing factor F can be generally set for a complete encoding, or for each group of pictures (GOP), a range of B frames, each B frame, or each region within a B frame (including, for example, as finely as for each macroblock or, in the case of MPEG-4 direct mode using a P vector in 8x8 mode, even individual 8x8 motion blocks).

In the interest of bit economy, and reflecting the fact that the mix ratio is not normally large enough to be carried with each macroblock, the optimal use of the mix should be related to the type of images that are compressed. For example, for images that fade, dissolve, or where the lighting or overall contrast is gradually changing, a blending factor F close to or by 1 (that is, selecting proportional interpolation) is generally the most optimal. For moving images without such changes in lighting or contrast, then the lowest blending factor values, such as 2/3, 1/2, or 1/3, might form the best choice, thereby retaining some of the benefits of proportional interpolation, as well as some of the benefits of equal average interpolation. All blending factor values within the range of 0 to 1 will generally be useful, with one particular value within this range being optimal for any given B frame.

For wide dynamic range and wide contrast range images, the blending factor can be determined regionally, depending on the characteristics of the local region. In general, however, a wide range of light and contrast is recommended toward mixing factor values that favor purely proportional interpolation, rather than equal averaging.

An optimal mixing factor is generally determined empirically, although experience with particular types of scenes can be used to create a table of mixing factors by scene type. For example, a determination of image shift characteristics can be used to select the blend ratio for a frame or region. Alternatively, the B-frames can be encoded using a number of candidate mixing factors (either for the entire frame, or regionally), with each then being evaluated to optimize image quality (determined, for example, by the ratio signal-to-noise, or SNR, higher) and for the lowest bit count. These candidate evaluations can then be used to select the best value for the mixing ratio. A combination of both coded quality / efficiency and makeover features can also be used.

B frames near the middle of a sequence of B frames, or resulting from low values of M, are not greatly affected by proportional interpolation, since the calculated proportions are already close to the equal average. However, for higher values of M, the extreme positions of B frames can be significantly affected by the choice of the mixing factor. Note that the mixing factor may be different for these extreme positions, using more than the average, than the more central positions, which derive little or no benefit from the deviation from the average, since they already have high proportions of both P (or I ) neighbors. . For example, if M = 5, the first and fourth B frames could use a blending factor F that blends by more than the equal average, but the second and third average B frames may use the equal average proportions 2/5 and 3/5 strict. If the ratio-to-average mixing factor varies as a function of the position of a B-frame in a sequence, the variable value of the mixing factor can be carried in the compressed bit stream or as side information to the decoder.

If a static general mixing factor is required (due to the lack of a method to transport the value), then the value of 2/3 is normally near optimal, and can be selected as a static value for B-frame interpolation both in the encoder as well as the decoder. For example, using F = 2/3 for the blend factor, for M = 3 the ratios of successive frames will be 7/18 (7/18 = 2/3 * 1/3 1/3 * 1/2) and 11 / 18 (11/18 = 2/3 * 2/3 1/3 * 1/2).

Linear interpolation

Video frame pixel values are generally stored in a particular representation that correlates the original image information with numerical values. Such a correlation can result in a linear or non-linear representation. For example, the luminance values used in compression are not linear. The use of various forms of non-linear representation includes logarithmic, exponential (at various powers), and exponential with a black correction (commonly used for video signals).

At narrow dynamic ranges, or for interpolations from close regions, nonlinear representation is acceptable, since these close interpolations represent piecewise linear interpolations. Thus, small variations in brightness are reasonably approximated by linear interpolation. However, for wide variations in brightness, such as occurs over a wide dynamic range and wide contrast range images, the treatment of non-linear signals as linear will be inaccurate. Even for normal contrast range images, linear fades and cross-dissolves can be degraded by linear interpolation. Some fades and cross-dissolves use non-linear dissolve and fade rates, adding additional complexity.

Thus, a further improvement to using proportional blends, or even simple equal or proportional average interpolations, is to perform such interpolations on pixel values represented in linear space, or on other optimized non-linear spaces that differ from luminance representation. original nonlinear.

This can be achieved, for example, by first converting the two non-linear luminance signals (from the P (or I) frames before and after to a linear representation, or a different non-linear representation. Then, a proportional mixing is applied, then of which the inverse conversion is applied, producing the mixed result in the original non-linear luminance representation of the image, however, the ratio function will have been performed on a more optimal representation of the luminance signals.

It is also useful to beneficially apply this linear or non-linear conversion to color values (chroma), in addition to luminance, when colors fade or become more saturated, as occurs in contrast changes associated with variations in color. mist and cloudiness.

Example realization

FIG. 4 is a flow chart showing an illustrative example as a method that can be implemented by computer:

Step 400: In a video image compression system, for direct and interpolative mode to calculate B frames, determine an interpolation value to apply to each pixel of an input sequence of two or more bi-directionally predicted intermediate frames using one of (1) the frame distance ratio or (2) a mix of equal weighting and the frame distance ratio, derived from at least two non-bidirectionally predicted frames that delimit such sequence input from a source (e.g. a stream video image).

Step 401: Optimize the interpolation value with respect to an image unit (for example, a group of images (GOP), a sequence of frames, a scene, a frame, a region within a frame, a macroblock, a block DCT, or useful grouping or selection of similar pixels). The interpolation value can be set statically for the entire encoding session, or dynamically for each imaging unit.

Step 402: Further optimize the interpolation value with respect to scene type or coding simplicity. For example, you can set an interpolation value: statically (such as proportional 2/3 and average equal 1/3); proportionally for squares near the same average, but mixed with the same average near the adjacent P (or I) squares; dynamically based on general scene characteristics, such as fades and cross dissolves; dynamically (and locally) based on local image region characteristics, such as local contrast and local dynamic range; or dynamically (and locally) based on encoding performance (such as highest encoded SNR) and minimum encoded bits generated.

Step 403: Transport the appropriate ratio amounts to the decoder, if not determined statically. Step 404: Optionally, convert the luminance information (and optionally chroma) for each frame to an alternative linear or non-linear representation, and transport this alternative representation to the decoder, if not determined statically.

Step 405: Determine the proportional pixel values using the determined interpolation value.

Step 406: If necessary (due to Step 404), revert to the original representation.

Frame reference P extended

As noted above, in prior art MPEG-1, 2, and 4 compression methods, the P frames refer to the above P or I frame, and the B frames refer to the prior P and / or I frames and later closer. The same technique is used in the H.261 and H.263 motion compensated DCT compression standards, which encompass low bit rate compression techniques.

In the H.263 ++ and H.26L standard under development, the B frame reference was extended to point to P or I frames that were not directly bounding a current frame. That is, macroblocks within B frames could point to a P or I frame before the preceding P frame, or to a P or I frame after the subsequent P frame. With one or more bits per macroblock, the skipping of the leading or trailing P-frame can be simply pointed out. Conceptually, using the above P-frames for reference to B-frames only requires storage. For the use of H.263 ++ or H.26L low-bit rate encoding, this is a small amount of additional memory. For later P-frame reference, the P-frame encoding order must be modified relative to the B-frame encoding, so that future P-frames (or possibly I-frames) must be decoded before intermediate B-frames. Thus, the encoding order is also a problem for later P-frame references.

The main distinctions between P and B frame types are: (1) B frames can be bi-directionally referenced (up to two motion vectors per macroblock); (2) B-frames are discarded after use (which also means that they can be skipped during decoding to provide temporary layers); and (3) the P frames are used as "rungs", one to the next, since each P frame must be decoded for use as a reference for each subsequent P frame.

As an example P frames (as opposed to B frames) are decoded with reference to one or more previous P or I frames (excluding the case of each P frame referring only to the closest previous P or I frame). Thus, for example, two or more motion vectors per macroblock can be used for a current P-frame, all pointing backward in time (ie, to one or more previously decoded frames). Such P-frames still maintain a "rung" character. FIG. 5 is a diagram showing an example of multiple prior references by a current P frame 500 to two prior P frames 502, 504, and to a prior I frame 506.

Furthermore, it is possible to apply the concepts of macroblock interpolation, as described above, in such P-frame references. Thus, in addition to marking single references to more than one previous P or I frame, it is also possible to mix proportions of multiple P or I frames above, using a motion vector for each such frame reference. For example, the previously described technique of using a B-frame interpolation mode that has two frame references can be applied to allow any macroblock in a P-frame to reference either two prior P-frames or a prior P-frame and an I-frame. above, using two motion vectors. This technique interpolates between two motion vectors, but is not bidirectional in time (as is the case with B-frame interpolation), since both motion vectors point backward in time. Memory costs have decreased to the point where keeping multiple previous P or I frames in memory for such concurrent reference is quite practical. In applying such P-frame interpolation, it is constructive to select and signal to a decoder various useful ratios of the previous two or more P-frames (and optionally a previous I-frame). In particular, an equal mix of frames is one of the useful mix ratios. For example, with two previous P-frames as references, an equal amount of 1/2 of each P-frame can be mixed. For three previous P-frames, an equal 1/3 mixture could be used.

Another useful combination of two P-frames is 2/3 of the most recent previous frame, and 1/3 of the least recent previous frame. For three older P frames, another useful mix is 1/2 of the most recent prior frame, 1/3 of the next most recent prior frame, and 1/6 of the least recent prior frame.

In either case, a simple set of useful mixes of multiple previous P-frames (and optionally an I-frame) can be used and simply pointed from an encoder to a decoder. The specific mixing ratios used can be selected as often as is useful to optimize encoding efficiency for a imaging unit. A series of mix ratios can be selected using a small number of bits, which can be transported to the decoder as long as it is suitable for a desired imaging unit.

As an example, it may also be useful to toggle-select single P-frame references from the most recent previous P (or I) frame to a more "distant" previous P (or I) frame. Thus, the P-frames would use a single motion vector per macroblock (or optionally per 8x8 block in MPEG-4 style encoding), but would use one or more bits to indicate that the reference refers to a single frame. above specific. P-frame macroblocks in this mode would not be interpolative, but would instead refer to a selected previous frame, being selected from two, three or more possible previous P (or I) frame options for reference. For example, a 2-bit code could designate one of up to four previous frames as the single reference frame of choice. This 2-bit code could be changed to any convenient imaging unit.

Adaptive number of frames B

It is typical in MPEG encoding to use a fixed pattern of I, P and B frame types. The number of B frames between P frames is typically a constant. For example, it is typical in MPEG-2 encoding to use two B frames between P (or I) frames. FIG. 6A is a diagram of a typical prior art MPEG-2 coding pattern, showing a constant number of B frames (ie, two) between I frames 600 and / or P frames 602 they delimit.

The MPEG-4 video coding standard conceptually allows a variable number of B frames between the I frames and / or the delimiting P frames, and a variable amount of spacing between I frames. FIG. 6B is a diagram of a theoretically possible MPEG-4 video coding pattern of the prior art, showing a variable number of B frames between the I frames 600 and / or the bounding P frames 602, as well as a variable distance between tables I 600.

This flexible coding structure can be theoretically used to improve coding efficiency by matching the most effective B and P frame coding types with the picture frames in movement. Although this flexibility has been specifically allowed, very little has been explored and no mechanism is known to actually determine the placement of the B and P frames in such a flexible structure.

The concepts described herein can be applied to this flexible coding structure as well as to the simple fixed coding patterns in common use. The B frames can thus be interpolated using the methods described above, while the P frames can refer to more than one previous P or I frame and can be interpolated according to the present description.

In particular, macroblocks within B-frames can use appropriate proportional mixes for a flexible coding structure just as efficiently as with a fixed structure. Proportional blends can also be used when the B frames refer to P or I frames that are further away than the closest P or I frames they delimit.

Similarly, P frames can refer to more than one previous P or I frame in this flexible coding structure just as effectively as with a fixed pattern structure. Furthermore, mixing ratios can be applied to macroblocks in such P-frames when they refer to more than one previous P-frame (optionally plus one I-frame).

(A) Determination of placement in flexible coding patterns

The following method allows an encoder to optimize the efficiency of both the frame encoding pattern and the mixing ratios used. For a selected range of frames, a number of candidate coding patterns can be tried to determine an optimal or near optimal pattern (relative to specific criteria). FIG. 7 is a diagram of code patterns that you can examine. An initial sequence 700 is arbitrarily selected, ending in a P or I frame, and used as the basis for adding additional P and / or B frames, which are then evaluated (as described below). In one embodiment, a P frame is added to the initial sequence 700 to create a first test sequence 702 for evaluation. If the assessment is successful, an intermediate B-frame is inserted to create a second 704 test sequence. For each successful assessment, additional B-frames are inserted to create increasingly long test sequences 706-712, until the assessment criteria become unsatisfactory. At that point, the above coding sequence is accepted. This process is then repeated, using the final P frame for the previously accepted coding sequence as the starting point for adding a new P frame and then inserting new B frames.

An optimal or near-optimal coding pattern can be selected based on various evaluation criteria, often involving trade-offs of various coding characteristics, such as coded image quality versus the number of coding bits required. Common evaluation criteria include the least number of bits used (in a fixed quantization parameter test), or the best signal-to-noise ratio (in a fixed bit rate test), or a combination of both.

It is also common to minimize an absolute difference sum (SAD), which forms a DC match measure. As described in copending US Patent No. 09 / 904,192, entitled "Motion Estimation for Video Compression Systems", (published as US 2003/0112873 A1), an AC matching criterion is also a useful measure. of the quality of a particular candidate match (the patent application also describes other useful optimizations). In this way, the AC and DC match criteria, accumulated over the best matches of all macroblocks, can be examined to determine the overall match quality of each candidate encoding pattern. This AC / DC matching technique can augment or replace the signal-to-noise ratio (SNR) and low-bit tests used when used in conjunction with an estimate of the number of bits encoded for each type of frame pattern. It is typical to encode macroblocks within B frames with a higher quantization parameter (QP) value than for P frames, affecting both quality (often measured as signal-to-noise ratio) and the number of bits used within the frames. various candidate encoding patterns.

(B) Mix ratio optimization in flexible coding patterns

Optionally, for each candidate standard determined according to the above method, the mixing ratios can be tested for suitability (eg, optimal or near-optimal mixing ratios) against one or more criteria. This can be done, for example, by testing for best quality (lowest SNR) and / or efficiency (least bits used). The use of one or more previous references for each macroblock in P-frames can also be determined in the same way, by testing each candidate reference standard and mixing ratio, to determine a set of one or more suitable references.

Once the coding pattern for this next step (Step 700 in Figure 7) has been selected, then subsequent steps (Steps 702-712) can be tested for various candidate coding patterns. In this way, a more efficient encoding of a moving picture sequence can be determined. In this way, efficiency can be optimized / improved as described in subsection (A) above; Mix optimization can be applied in every tested coding step.

DC vs. AC interpolation

In many cases of image coding, such as when using a logarithmic representation of image frames, the interpolation of frame pixel values described above will optimally encode changes in lighting. However, in alternative video, linear, and other "gamma curve" representations, it will often be useful to apply different interpolation mixing factors to the DC values than to the AC values of the pixels. FIG. 8 is a flow chart showing an embodiment of an interpolation method with DC interpolation that is different from AC interpolation. For a selected image region (typically a DCT block or macroblock) of a first and second input frames 802, 802 ', the average pixel value for each such region is subtracted 804, 804', thereby separating the value of DC (that is, the average value of the entire selected region) 806, 806 'of the AC values (that is, the remaining flagged pixel values) 808, 808' in the selected regions. The respective DC values 806, 806 'can then be multiplied by interpolation weights 810, 810' different from the interpolation weights 814, 814 'used to multiply the AC (flagged) pixel values 808, 808'. The newly interpolated DC value 812 and the newly interpolated AC values 816 can then be combined 818, resulting in a new prediction 820 for the selected region.

As with the other interpolation values described, the appropriate weights can be signaled to one decoder per picture unit. A small number of bits can be selected from a series of interpolation values, as well as selecting independent interpolation of the AC vs. DC aspects of the pixel values.

Linear and nonlinear interpolation

The interpolation is a linear weighted average. Since the interpolation operation is linear, and since the pixel values in each image frame are often represented in a non-linear way (such as video gamma or logarithmic representations), further optimization of the process becomes possible. interpolation. For example, interpolation of pixels for a particular frame sequence, as well as interpolation of DC values separately from AC values, will sometimes be optimal or near optimal with a linear representation of pixels. However, for other frame sequences, such interpolation will be optimal or near optimal, if the pixels are represented as logarithmic values or in other pixel representations. Furthermore, the optimal or near-optimal representations for interpolating the U and V (chroma) signal components may differ from the optimal or near-optimal representations for the Y (luminance) signal component. Therefore, it is useful to convert a pixel representation to an alternate representation as part of the interpolation procedure.

FIG. 9 is a flow chart showing one embodiment of a method for luminance pixel interpolation using an alternative representation. Starting with a region or block of luminance (Y) pixels in an initial representation (for example, video gamma or logarithmic) (Step 900), the pixel data is transformed to an alternate representation (for example, linear, logarithmic, video gamma) different from the initial rendering (Step 902). The transformed region or block of pixels is then interpolated as previously described (Step 904), and transformed back to the initial representation (Step 906). The result is interpolated pixel luminance values (Step 908).

FIG. 10 is a flow chart showing an embodiment of a method for chroma pixel interpolation using an alternative representation. Starting with a chroma (U, V) region or block of pixels in an initial representation (for example, video gamma or logarithmic) (Step 1000), the pixel data is transformed to an alternate representation (for example, linear, logarithmic, video gamma) different from the initial representation (Step 1002). The transformed region or block of pixels is then interpolated as previously described (Step 1006), and transformed back to the initial representation (Step 1006). The result is interpolated pixel chroma values (Step 1008).

Transformations between representations can be performed according to the teachings of US Patent Application No. 09 / 905,039, entitled "Method and System for Improving Compressed Image Chroma Information", published as US 2003/0112863 A1. Note that the alternate representation transformation and its inverse can often be performed using a simple lookup table.

As a variation, the alternate representation space (linear or nonlinear) for the AC interpolation may differ from the alternate representation space for the DC interpolation.

As with interpolation weights, the selection of which alternate interpolation representation to use for each of the luminance (Y) and chroma (U and V) pixel representations can be signaled to the decoder using a small number of bits. for each selected imaging unit.

Number of motion vectors per macroblock

In MPEG-2, a 16x16 motion vector per macroblock is allowed in P-frames. In B-frames, MPEG-2 allows a maximum of 2 motion vectors per 16x16 macroblock, corresponding to interpolative mode bidirectional. In MPEG-4 video encoding, up to 4 motion vectors per 16x16 macroblock are allowed in P-frames, corresponding to an 8x8 DCT motion vector per block. In MPEG-4 B-frames, a maximum of two motion vectors are allowed for each 16x16 macroblock, when using interpolative mode. A single motion vector delta in MPEG-4 forward mode can result in four independent "implicit" motion vectors, if the corresponding subsequent P-frame macroblock was set to 8x8 mode having four vectors. This is achieved by adding the delta of the motion vector carried in a 16x16 B frame macroblock to each of the four corresponding independent motion vectors of the next P frame macroblock, after scaling the distance in time (frame B is more close in time as the previous P or I frame reference of the P frame).

There is the option of increasing the number of motion vectors per image region, such as a macroblock. For example, it will sometimes be beneficial to have more than two motion vectors per macroblock of B frames. These can be applied by referencing additional P or I frames and having three or more interpolation terms in the weighted sum. Additional motion vectors can also be applied to allow independent vectors for the 8x8 DCT blocks of the B-frame macroblock. Also, four independent deltas can be used to extend the concept directly by applying a separate delta to each of the four vectors. movement of 8x8 regions of the posterior P-frame.

Furthermore, P-frames can be adapted using B-frame implementation techniques to refer to more than one previous frame in an interpolative mode, using the two-term B-frame interpolation technique described above. This technique can easily be extended to more than two previous P or I frames, with a resulting interpolation that has three or more terms in the weighted sum.

As with other aspects of the method (eg, pixel representation and DC vs. AC interpolation methods), particular weighted sums can be communicated to a decoder using a small number of bits per picture unit.

Applying this aspect of the method, the correspondence between the 8x8 pixel DCT blocks and the motion vector field need not be as strict as with MPEG-2 and MPEG-4. For example, it may be useful to use alternative region sizes other than 16x16, 16x8 (used only with interlacing in MPEG-4) and 8x8 for motion vectors. Such alternatives could include any number of useful region sizes, such as 4x8, 8x12, 8x16, 6x12, 2x8, 4x8, 24x8, 32x32, 24x24, 24x16, 8x24, 32x8, 32x4, etc. Using a small number of such useful sizes, a few bits can signal to a decoder the correspondence between the motion vector region sizes and the DCT block sizes. In systems where a conventional 8x8 DCT block is used, a simple set of maps to the motion vector field is useful to simplify processing during motion compensation. In systems where the DCT block size is different from 8x8, then greater flexibility can be achieved by specifying the motion vector field, as described in co-pending US Patent Application No. 09 / 545,233, entitled "Enhanced Temporal and Resolution Layering in Advanced Television", see WO 01/177871 A1. Note that the boundaries of the motion vector region need not correspond to the boundaries of the DCT region. In fact, it is often useful to define motion vector regions such that one edge of the motion vector region falls within a DCT block (and not on its edge).

The concept of extending the flexibility of the motion vector field to interpolation also applies. As long as the correspondence between each pixel and one or more motion vectors to one or more reference frames is specified, the interpolation method described above can be applied to the full flexibility of useful motion vectors using the full generality of this method. Even the size of the regions corresponding to each motion vector may differ for each previous frame reference when using P frames, and each previous and future frame reference when using B frames. If the region sizes for the motion vectors differ when the improved interpolation method is applied, then the interpolation reflects the common region of overlap. The common region of overlap for motion vector references can be used as the region over which the DC term is determined when the DC and AC pixel values are interpolated separately.

FIG. 11 is a diagram showing unique motion vector region sizes 1100, 1102 for each of the two P frames 1104, 1106. Before calculating the interpolation values, the union 1108 of the vector region sizes of movement. Junction 1108 defines all regions that are considered to have an assigned motion vector.

Thus, for example, when interpolating 4x4 DCT regions of a B frame 1112 backward with the preceding P frame 1104, a 4x4 1110 region within junction 1108 would use the motion vector corresponding to the 8x16 region 1114 in Table P above. If predicted forward, region 1110 within junction 1108 would use the motion vector corresponding to 4x16 region 1115 in the next frame P. Similarly, interpolation of region 116 within junction 1108 backward would use the motion vector corresponding to the 8x16 1114 region, while the forward prediction of the same region would use the motion vector corresponding to the 12x161117 region.

In one method, two steps are used to achieve interpolation of generalized (that is, non-uniformly sized) motion vectors. The first step is to determine the common motion vector regions, as described with respect to FIG. 11. This maps pixels to motion vectors (that is, the number of motion vectors per specified pixel region size) for each previous or next frame reference. The second step is to use the appropriate interpolation method and active interpolation factors for each pixel region. It is the task of the encoder to ensure that optimal or near optimal motion vector regions and interpolation methods are specified, and that all pixels have their vectors and interpolation methods fully specified. This can be very simple in the case of a fixed pattern of motion vectors (such as a motion vector for each 32x8 block, specified for an entire frame), with a single specified interpolation method (such as a ratio mix sets to each referenced frame distance, specified for the entire frame). This method can also become quite complex if regional changes are made to the motion vector region sizes, and where the region sizes differ depending on which front or back frame is referenced (e.g. 8x8 blocks for closest previous frame and 32x8 blocks for next closest previous frame). Also, the interpolation method can be specified regionally within the box.

When encoding, it is the job of the encoder to determine the optimal or near optimal use of the bits to select between shapes and sizes of motion vector regions, and to select the optimal or near optimal interpolation method. A determination is also required to specify the number and distance of the referenced frames. These specifications can be determined by extensive testing of a series of candidate motion vector region sizes, candidate frames for reference, and interpolation methods for each such motion vector region, until an optimal or near optimal coding is found. Optimization (relative to a selected criterion) can be determined by finding the smallest SNR after encoding a block or the smallest number of bits for a fixed quantization parameter (QP) after encoding the block, or by applying another measure adequate.

Direct mode extension

Conventional direct mode, used in B-frame macroblocks in MPEG-4, can be efficient in motion vector coding, providing the benefits of 8x8 block mode with a single common delta. Direct mode weights each corresponding motion vector in the back P-frame, which references the preceding P-frame, at the location of the corresponding macroblock based on distance in time. For example, if M = 3 (that is, two intermediate B-frames), with simple linear interpolation, the first B-frame would use -2/3 times the motion vector of the subsequent P-frame to determine a pixel offset with respect to such P-frame, and 1/3 times the motion vector of the subsequent P-frame to determine a pixel offset relative to the preceding P-frame. Similarly, the second frame B would use -1/3 times the same P frame motion vector to determine a pixel offset relative to that P frame, and 2/3 times the subsequent P frame motion vector to determine a pixel offset from the previous P-frame. In direct mode, a small delta is added to each corresponding motion vector.

According to the invention, this concept can be extended to frame references B pointing to one or more frames P at n distances, which in turn refer to one or more frames P or frames I before or after, taking into account the distance of frame to determine a frame scale fraction.

FIG. 12 is a diagram showing a sequence of P and B frames with interpolation weights for the B frames determined as a function of the distance from a 2-distance back P-frame referencing a 1-distance back P-frame. In the illustrated example, M = 3, indicating two consecutive B frames 1200, 1202 between the P frames 1204, 1206 that they delimit. In this example, each co-located macroblock in the next closest subsequent P-frame 1208 (that is, n = 2) could point to the middle (that is, closest) P frame 1204, and the first two B-frames 1200, 1202 may refer to the next closest back P-frame 1208 instead of the closest back P-frame 1204, as in conventional MPEG. Thus, for the first frame B 1200, the frame scale fraction 5/3 times the motion vector mv of the next nearest subsequent P frame 1208 would be used as a pixel offset relative to frame P 1208, and the second frame B 1202 would use a displacement of 4/3 times that same motion vector.

If a nearest back P-frame referenced by a B-frame points to the next closest preceding P-frame, then again the single-frame distance can be used to obtain the appropriate frame-scale fraction to apply to the motion vectors . FIG. 13 is a diagram showing a sequence of P and B frames with interpolation weights for the B frames determined as a function of the distance from a 1-distance back P-frame referring to a 2-distance leading P-frame. In the illustrated example, M = 3, and frames B 1300, 1302 refer to the nearest back frame P 1304, which in turn refers to frame P at 2 distances 1306. Thus, for the first frame B 1300 , the pixel shift fraction is the 2/6 frame scale fraction multiplied by the motion vector mv of the nearest back P frame 1304, and the second B frame 1302 would have a pixel shift of the frame scale fraction 1/6 multiplied by the same motion vector, since the vector of Nearest posterior P-frame movement 1304 points to 2-distance anterior P-frame 1306, which is 6-frame distant.

In general, in the case of a B frame referencing a single P frame in direct mode, the frame distance method sets the numerator of a frame scale fraction equal to the frame distance from that B frame to its P-frame referenced, or “target”, and sets the denominator equal to the frame distance from the target P-frame to another P-frame referenced by the target P-frame. The sign of the frame scale fraction is negative for measurements made from a B frame to a later P frame, and positive for measurements made from a B frame to an earlier P frame. This simple method of applying a frame distance or the frame scale fraction to a frame motion vector P can achieve effective direct mode coding.

Furthermore, it is possible to allow direct mode to be applied to multiple interpolated motion vector references from a P-frame. For example, if a P-frame was interpolated from the nearest and next-closest previous P-frames, a direct mode reference allows interpolated mixing for each macroblock of B-frames in multi-reference direct mode. In general, the two or more motion vectors of a P-frame may have a suitable frame-scale fraction applied. The two or more frame distance modified motion vectors can then be used with the corresponding interpolation weights for each B frame that references or is the target of that P frame, as described below, to generate motion compensation frame macroblock B interpolated.

FIG. 14 is a diagram showing a sequence of P and B frames in which a subsequent P frame has multiple motion vectors referring to earlier P frames. In this example, a frame B 1400 refers to a frame P back P3. This frame P3 in turn has two motion vectors, mv1 and mv2, which refer to the previous P frames P2, P1, corresponding. In this example, each macroblock in frame B 1400 can be direct mode interpolated using either two weighting terms or a combination of such weighting terms.

Each macroblock for frame B 1400 would be constructed as a mixture of:

• corresponding pixels from frame P2 shifted by frame scale fraction 1/3 of mv1 (where pixels can then be multiplied by some proportional weight i) plus corresponding pixels from frame P3 offset by frame scale fraction -2 / 3 of mv1 (where pixels can then be multiplied by some proportional weight j); and

• corresponding pixels from frame P1 displaced by the 2/3 (4/6) frame scale fraction of mv2 (where the pixels can then be multiplied by some proportional weight k) plus corresponding pixels from frame P3 offset by the scale fraction frame -1/3 (-2/6) of mv2 (where pixels can then be multiplied by some proportional weight l).

As with all direct modes, a motion vector delta can be used with each of mv1 and mv2. Macroblocks predicted in direct mode in B frames can also reference multiple subsequent P frames, using the same motion vector frame scaling and interpolation methodology as with multiple preceding P frames. FIG. 15 is a diagram showing a sequence of P and B frames in which a closer back P frame has a motion vector referencing a previous P frame, and next a closer P frame has multiple motion vectors referencing to previous P frames. In this example, a B frame 1500 refers to two subsequent P frames P2, P3. Frame P3 has two motion vectors, mv1 and mv2, which refer to previous P frames P2, P1. Frame P2 has a motion vector, mv3, which refers to frame P above P1. In this example, each macroblock in frame B 1500 is interpolated in direct mode using three weighting terms. In this case, the frame-scale fractions of the motion vector can be greater than 1 or less than -1.

The weights for this form of direct mode B-frame macroblock interpolation can utilize the full interpolation generality as described herein. In particular, each weight, or combinations of the weights, can be tested for best performance (eg, quality versus number of bits) for various imaging units. The interpolation fraction set for this enhanced direct mode can be specified to a decoder with a small number of bits per picture unit.

Each macroblock for frame B 1500 would be constructed as a mixture of:

• corresponding pixels from frame P3 displaced by the frame scale fraction -5/3 of mv1 (where the pixels can then be multiplied by some proportional weighting i), plus corresponding pixels from frame P2 offset by the frame scale fraction - 2/3 of mv1 (where pixels can then be multiplied by some proportional weight j);

• corresponding pixels from frame P3 shifted by the -5/6 frame scale fraction of mv2 (where pixels can then be multiplied by some proportional weight k) plus pixels Corresponds from frame P1 shifted by the 1/6 frame scale fraction of mv2 (where pixels can then be multiplied by some proportional weight l); and

• corresponding pixels from frame P2 displaced by the -2/3 frame scale fraction of mv3 (where the pixels can then be multiplied by some proportional weight m) plus corresponding pixels from frame P1 displaced by the scale fraction of frame 1 / 3 of mv3 (where the pixels can then be multiplied by some proportional weight n).

As with all direct modes, a motion vector delta can be used with each of mv1, mv2, and mv3.

Note that a particularly beneficial direct coding mode often occurs when the next closest subsequent P-frame references the closest P-frames that delimit a candidate B-frame. B-frame direct mode encoding in MPEG-4 always uses the back P-frame motion vectors as a reference. It is also possible that a B frame refers to the motion vectors of the co-located macroblocks of the previous P frame, which will sometimes be a beneficial choice of direct mode coding reference. In this case, the frame scale fractions of the motion vector will be greater than one, when the next closest previous P frame is referenced by the motion vector of the closest previous P frame. FIG. 16 is a diagram showing a sequence of P and B frames in which a closer previous P frame has a motion vector referencing a previous P frame. In this example, a B frame 1600 refers to the previous P frame at 1 distance P2. The motion vector mv in frame P2 refers to the next previous frame P P1 (at 2 distances relative to frame B 1600). Appropriate frame scale fractions are shown.

If the closest previous P-frame is interpolated from multiple vectors and frames, then methods similar to those described in conjunction with FIG. 14 are applied to obtain the frame-scale fractions of the motion vector and the interpolation weights. FIG. 17 is a diagram showing a sequence of P and B frames in which a closer previous P frame has two motion vectors referring to previous P frames. In this example, a frame B 1700 refers to frame P above P3. A motion vector mv1 from the previous frame P3 refers to the next previous frame P P2, while the second motion vector mv2 refers to the previous frame P at 2 distances P1. Appropriate frame scale fractions are shown.

Each macroblock for frame B 1700 would be constructed as a mixture of:

• corresponding pixels from frame P3 displaced by the 1/3 frame scale fraction of mv1 (where the pixels can then be multiplied by some proportional weighting i) plus corresponding pixels from frame P2 offset by the 4/3 frame scale fraction from mv1 (where the pixels can then be multiplied by some proportional weight j); and

• corresponding pixels from frame P3 shifted by the 1/6 frame scale fraction of mv2 (where the pixels can then be multiplied by some proportional weight k) plus corresponding pixels from frame P1 offset by the frame scale fraction 7/6 from mv2 (where pixels can then be multiplied by some proportional weight l).

When the motion vector of a previous P frame (relative to a B frame) points to the next closest previous P frame, it is not necessary to use only the next closest previous frame as the interpolation reference, as in FIG. 16. The closest anterior P-frame may prove to be a better choice for motion compensation. In this case, the motion vector of the closest previous frame P is shortened by the fraction of the frame distance from frame B to that frame P. FIG. 18 is a diagram showing a sequence of P and B frames in which a closer previous P frame has a motion vector referring to a previous P frame. In this example, for M = 3, a first B 1800 frame would use 1/3 frame distance fractions and -2/3 times the motion vector mv of the closest previous P frame P2. The second frame B 1802 would use frame distance fractions 2/3 and -1/3 (not shown). Such a selection would be signaled to the decoder to distinguish this case from the case shown in FIG. 16.

As with all other encoding modes, use of the direct mode preferably involves testing the candidate mode against other available modes of interpolation and encoding of a single vector and reference frames. For testing in direct mode, the closest back P-frame (and optionally the next closest back P-frame or even the most distant back P-frames, and / or one or more earlier P-frames) can be tested as candidates, and a small number of bits (generally one or two) can be used to specify the reference frame distance (s) P in direct mode to be used by a decoder. Extended interpolation values

It is specified in MPEG-1, 2 and 4, as well as in the H.261 and H.263 standards, that B-frames use equal weighting of pixel values of forward-referenced and back-referenced frames, as displaced by the motion vectors. This invention includes the application of various useful unequal weights that can significantly improve B-frame coding efficiency, as well as the extension of such unequal weights to more than two references, including two or more references backward or forward in time.

The invention also includes methods for more than one frame to be referenced and interpolated for P frames. Also, when two or more references point forward in time, or when two or more references point backward in time, it will sometimes be useful use negative weights as well as weights greater than 1.0.

For example, FIG. 19 is a frame sequence showing the case of three P frames P1, P2 and P3, where P3 uses an interpolated reference with two motion vectors, one for each of P1 and P2. If, for example, a continuous change is occurring over the frame span between P1 and P3, then P2-P1 (that is, the pixel values of frame P2, shifted by the motion vector for P2, minus the pixel values from square P1, displaced by the motion vector for P1) will equal P3-P2. Similarly, P3-P1 will be twice the magnitude of P2-P1 and P3-P2. In such a case, the pixel values for frame P3 can be predicted differentially from P1 and P2 through the formula:

P3 = P1 2 x (P2 - P1) = (2 x P2) - P1

In this case, the interpolative weights for P3 are 2.0 for P2 and -1.0 for P1.

As another example, FIG. 20 is a frame sequence showing the case of four P frames P1, P2, P3, and P4, where P4 uses an interpolated reference with three motion vectors, one for each of P1, P2, and P3. Thus, since P4 is predicted from P3, P2, and P1, three motion vectors and interpolative weights would be applied. If, in this case, a continuous change was occurring over this frame span, then P2- P1 would be equal to both P3-P2 and P4-P3, and P4-P1 would be equal to both 3 x (P2- P1) and 3 x (P3-P2).

Thus, in this example case, a prediction of P4 based on P2 and P1 would be:

P4 = P1 + 3 x (P2 - P1) = (3 x P2) - (2 x P1) (weights 3.0 and - 2.0)

The prediction of P4 based on P3 and P1 would be:

P4 = P1 3/2 x (P3 - P1) = (3/2 x P3) - (1/2 x P1) (weights 1.5 and -0.5)

The prediction of P4 based on P3 and P2 would be:

P4 = P2 2 x (P3 - P2) = (2 x P3) - P2 (weights 2.0 and -1.0)

However, it could also be likely that the change closer to P4, involving P3 and P2, is a more reliable predictor of P4 than predictions involving P1. In this way, giving 1/4 weight to each of the two previous terms that involve P1, and 1/2 weight to the term that only involves P3 and P2, would result in:

1/2 (2 P3 - P2) 1/4 (3/2 P3 - 1/2 P1) 1/4 (3 P2 - 2 P1) =

13/8 P3 1/4 P2 - 5/8 P1 (weights 1.375, 0.25 and -0.625)

Therefore, it will sometimes be useful to use weights both above 1.0 and below zero. At other times, if there is a noise-like variation from one frame to the next, a positive weighted average having mild coefficients between 0.0 and 1.0 might produce the best predictor of the macroblock of P4 (or other pixel region). For example, a weight equal to 1/3 of each of P1, P2, and P3 in the FlG. 20 could form the best predictor of P4 in some cases.

Note that the best match motion vector is applied to determine the region of P1, P2, P3, etc., which is used by the calculations in this example. This match might be better an AC match in some cases, allowing a variable DC term to be predicted through the AC coefficients. Alternatively, if a DC match is used (such as Absolute Difference Sum), then changes in AC coefficients can often be predicted. In other cases, various forms of motion vector matching will form a better prediction with various weighting mixes. In general, the best predictor for a particular case is determined empirically using the methods described herein.

These techniques are also applicable to B-frames that have two or more motion vectors pointing either backward or forward in time. When pointing forward in time, the coefficient pattern described above for P-frames is reversed to accurately predict backward to the current P-frame. It is possible to have two or more vectors of motion in both the forward and backward directions, thereby predicting in both directions concurrently. A suitable weighted mix can be optimized of these various predictions by selecting the blend weight that best predicts the macroblock (or other region of pixels) of a current B-frame.

FIG. 21 is a diagram showing a sequence of P and B frames in which various P frames have one or more motion vectors referring to various previous P frames, and showing different weights ae assigned to the respective forward and backward references. by a particular B frame. In this example, a 2100 B-frame refers to three earlier P-frames and two later P-frames.

In the example illustrated in FIG. 21, frame P5 must be decoded for this example to work. Sometimes it is useful to arrange frames in a bitstream in the order required for decoding ("delivery order"), which is not necessarily the display order ("display order"). For example, in a sequence of frames showing cyclical movement (eg, rotation of an object), a particular P-frame may be more similar to a distant P-frame than the nearest back P-frame. FIG. 22 is a diagram showing a sequence of P and B frames in which the delivery order of the bit stream of the P frames differs from the display order. In this example, frame P3 is more similar to frame P5 than frame P4. Therefore it is useful to deliver and decode P5 before P4, but show P4 before P5. Preferably, each P frame should signal to the decoder when such P frame can be discarded (eg, an expiration of n frames in bitstream order or after X frame in display order).

If the weights are selected from a small set of options, then a small number of bits can signal to the decoder which weight to use. As with all the other weights described herein, this can be signaled to a decoder once per picture unit, or at any other point in the decoding process where a change in weights is useful.

It is also possible to download new weight sets. In this way, a small number of weight sets can be active at any one time. This allows a small number of bits to signal to a decoder which of the active weight sets to use at any given point in the decoding process.To determine suitable weight sets, a large number of weights can be tested during encoding. . If a small subset is found to provide high efficiency, then that subset can be signaled to a decoder for use. A particular element of the subset can thus be signaled to the decoder with only a few bits. For example, 10 bits can select 1 of 1024 subset elements. Furthermore, when a particular small subset should be changed to maintain efficiency, a new subset may be signaled to the decoder. In this way, an encoder can dynamically optimize the number of bits required to select among the elements of the weight set versus the number of bits required to update the weight sets. Also, a small number of shortcodes can be used to point out common useful weights, such as 1/2, 1/3, 1/4, etc. In this way, a small number of bits can be used to designate the set of weights, such as for a forward vector prediction K in a P frame (where K = 1, 2, 3, ...), or a prediction of forward vector K and backward vector L in frame B (where K and L are selected from 0, 1,2, 3, ...), or a prediction of forward vector K and backward vector L in a P frame (where K and L are selected from 0, 1, 2, 3, ...), as a function of the current M value (that is, the relative position of the B frame with respect to the P frames (or I) neighbors).

FIG. 23 is a diagram showing a sequence of P and B frames with assigned weights. A table B 2300 has weights ae, the values of which are assigned from a table of sets of weights from table B 2302. A table P 2304 has weights m, n, the values of which are assigned from a P frame weight set table 2306. Some weights may be static (ie, permanently downloaded to the decoder), and signaled by an encoder. Other weights can be dynamically downloaded and then flagged.

This same technique can be used to dynamically update weight sets to select DC interpolation versus AC interpolation. In addition, code values that select normal (linear) interpolation (of pixel values normally represented in a non-linear representation) can be flagged versus linear interpolation of converted values (in an alternative linear or non-linear representation). Similarly, such code values can indicate which interpolation to apply to the AC or DC values or whether to divide the AC and DC parts of the prediction.

The active subset can also be used to minimize the number of bits required to select from the sets of weighting coefficients currently in use. For example, if 1024 downloaded weight sets were kept on a decoder, perhaps 16 might need to be active during a particular part of a frame. Thus, in selecting which subset of 16 (out of 1024) weight sets are to be active, only 4 bits need to be used to select which weight set of these 16 is active. Subsets can also be flagged using short codes for the most common subsets, thus allowing a small number of bits to be selected from commonly used subsets.

Smoothness and sharpness

As with the simple separation of a DC component from AC signals via subtraction from the average value, other filtering operations are also possible during motion vector compensated prediction. For example, various high-pass, band-pass, and low-pass filters can be applied to a region of pixels (such as a macroblock) to extract various frequency bands. These frequency bands can then be modified when performing motion compensation. For example, it could often be useful in a noisy moving picture to filter out the higher frequencies in order to soften (make the picture less sharp or slightly blur) the picture. Smoother image pixels, combined with a steeper skew matrix for quantization (a steeper skew matrix ignores the higher frequency noise in the current block), will typically form a more efficient encoding method. It is already possible to signal a change in the quantization tilt matrix for each imaging unit. Custom tilt matrices for luminance and chroma can also be downloaded. Note that the effectiveness of motion compensation can be improved if the tilt matrix is changed or not. However, it will often be more effective to change both the tilt matrix and the filter parameters that are applied during motion compensation.

It is common practice to use a reduced resolution for chroma encoding in conjunction with a chroma-specific skew matrix. However, the chroma encoding resolution is static in this case (such as the 4: 2: 0 encoding half resolution vertically and horizontally, or the 4: 2: 2 encoding half resolution only horizontally). Coding effectiveness can be increased by applying a dynamic filter process during motion compensation for both chroma and luminance (independently or in tandem), selected per unit of image.

US Patent Application No. 09 / 545,233, entitled "Enhanced Temporal and Resolution Layering in Advanced Television" (referenced above, see WO 01/17787 A1), describes the use of improved displacement filters having negative lobes (a truncated sync function). These filters have the advantage that they retain sharpness when the fractional pixel portion of the motion vector shift is performed. At both the whole pixel shift point and fractional points, some macroblocks (or other useful image regions) are shifted more optimally using filters that reduce or increase their sharpness. For example, for a “rack focus” (where some objects in the frame blur over time, and other parts of the frame become in focus), the transition is one of change in both sharpness and smoothness. Thus, a motion compensation filter that can both sharpen certain regions in an image while reducing sharpening in other regions can improve coding efficiency. In particular, if a region of an image becomes out of focus, it can be beneficial to decrease the sharpness, which will soften the image (thereby potentially creating a better match) and decrease grain and / or noise (thereby possibly improving the efficiency of coding). If a region of the image is in focus, it can be beneficial to retain maximum sharpness, or even increase sharpness by using larger negative lobe filter values.

Chroma filtering can also benefit from increasing and decreasing sharpness during encoding. For example, many of the encoding efficiency benefits of 4: 2: 0 encoding (half-resolution chroma horizontally and vertically) can be achieved using smoother motion compensation filters for chroma while preserving full resolution on the U and / or V channels. Only when the color detail in the U and V channels is high, it will be necessary to select the sharpest displacement filters; softer filters will be most beneficial where there is high color noise or grain.

In addition to changes in focus, it is also common to have a change in the direction and amount of motion blur from one frame to the next. At the moving movie frame rate of 24 fps, even a simple dialog scene can have significant changes in motion blur from one frame to the next. For example, an upper lip could go out of focus in one frame and become sharp in the next, entirely due to lip movement during the camera's shutter release time. For such motion blur, it will be beneficial to not only have sharpening and smoothing (blur) filters during motion compensation, but also having a directional look for sharpening and smoothing. For example, if a direction of movement can be determined, a smoothing or sharpening along that direction can be used to correspond to the movement or stopping of an image feature. The motion vectors used for motion compensation can provide themselves some useful information about the amount of motion and the change in the amount of motion (i.e. motion blur), for a particular frame (or region within a box) relative to any of the surrounding boxes (or corresponding regions). In particular, a motion vector is the best inter-frame motion match P, while motion blur results from motion during the shutter release time within a frame.

FIG. 24 is a graph of the position of an object within a box versus time. The shutter of a camera is only open for part of a frame time. Any movement of the subject while the shutter is open results in blur. The amount of motion blur is indicated by the amount of position change during the shutter release time. Thus, the slope of the 2400 position curve while the shutter is open is a measure of motion blur.

The amount of motion blur and the direction of motion can also be determined from a combination of sharpness metrics, surrounding motion vectors (where image regions coincide), feature blur detection, and human-assisted designation of regions of picture. A filter can be selected based on the specified amount of motion blur and motion direction. For example, a correlation of various filters can be determined empirically against the determined motion blur and direction.

When combined with the other features of the method, such intelligently applied filters can significantly improve compression coding efficiency. A small number of such filters can be selected with a small number of bits signaled to the decoder. Again, this can be done once per picture unit or at other useful points in the decoding process. As with weight sets, a dynamically loaded filter set, as well as an active subsets mechanism, can be used to minimize the number of bits required to select from the most beneficial set of filter parameters.

Implementation

The video compression method can be implemented in hardware or software, or a combination of both (for example, programmable logic matrices). Unless otherwise specified, the algorithms included as part of the invention are not inherently related to any particular computer or other apparatus. In particular, various general purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to build more specialized apparatus (eg, integrated circuits) to perform particular functions. Thus, the method can be implemented in one or more computer programs running on one or more programmable computer systems, each comprising at least one processor, at least one data storage system (including volatile memory and non-volatile and / or storage elements), in at least one device or input port, and at least one device or output port. The program code is applied to the input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in a known way.

Each such program can be implemented in any desired computer language (including machine, assembly, or high-level procedural, logical, or object-oriented programming languages) to communicate with a computer system. In either case, the language can be a compiled or interpreted language.

Each such computer program is preferably stored or downloaded to a storage medium or device (for example, memory or solid state media, or magnetic or optical media) readable by a programmable general or special purpose computer, to configure and operate the computer when the storage medium or device is read by the computer system to perform the procedures described herein. It can also be considered that the system is implemented as a computer-readable storage medium, configured with a computer program, where the storage medium thus configured causes a computer system to operate in a specific and predefined way to perform the functions described. In the present memory.

A number of embodiments of the invention have been described. However, it will be understood that various modifications can be made without departing from the scope of the invention as defined by the claims. For example, some of the steps described above may be independent of order, and thus can be performed in a different order than described.

Claims (4)

1. A method for compression of video images using direct mode prediction, including: providing a sequence of bidirectionally predicted and predicted frames, each comprising pixel values arranged in image areas, where at least one of the areas image in at least one of the bidirectionally predicted frames is predicted from image areas in one or more reference frames using direct mode prediction; and determining a predicted motion vector for a bidirectionally predicted frame with a direct mode prediction frame distance method, wherein the frame distance method comprises: given a predicted image area in a first reference frame, the predicted frame bidirectionally with another predicted image area, and a motion vector between the predicted image area in the first reference frame and an image area in a second reference frame, the predicted motion vector is the motion vector multiplied by a frame-scale fraction, wherein the numerator of the frame-scale fraction is equal to a first distance between the bidirectionally predicted frame of the other predicted image area and the first reference frame and the denominator of the scale fraction of frame is equal to a second distance between the first reference frame and the second reference frame referenced by the first frame reference, wherein both the first and second reference frames are in a display order that is earlier than a bidirectionally predicted frame display order of the predicted image area. 2. A video compression system adapted to providing a sequence of bidirectionally predicted and predicted frames, each comprising pixel values arranged in image areas, wherein at least one of the image areas in at least one of the bidirectionally predicted frames is predicted from image areas in one or more reference frames using direct mode prediction; and determining a predicted motion vector for a bidirectionally predicted frame with a direct mode prediction frame distance method, wherein the frame distance method comprises: given a predicted image area in a first reference frame, the predicted frame bidirectionally with another predicted image area, and a motion vector between the predicted image area in the first reference frame and an image area in a second reference frame, the predicted motion vector is the motion vector multiplied by a frame-scale fraction, wherein the numerator of the frame-scale fraction is equal to a first distance between the bidirectionally predicted frame of the other predicted image area and the first reference frame and the denominator of the scale fraction of frame is equal to a second distance between the first reference frame and the second reference frame referenced by the first frame reference, wherein both the first and second reference frames are in a display order that is earlier than a bidirectionally predicted frame display order of the predicted image area. 3. A method for decompressing video images using direct mode prediction, including: receiving a sequence of bidirectionally predicted and predicted frames, each comprising pixel values arranged in image areas, where at least one of the areas image in at least one of the bidirectionally predicted frames is predicted from image areas in one or more reference frames using direct mode prediction; and determining a predicted motion vector for a bidirectionally predicted frame with a direct mode prediction frame distance method, wherein the frame distance method comprises: given a predicted image area in a first reference frame, the predicted frame bidirectionally with another predicted image area, and a motion vector between the predicted image area in the first reference frame and an image area in a second reference frame, the predicted motion vector is the motion vector multiplied by a frame-scale fraction, wherein the numerator of the frame-scale fraction is equal to a first distance between the bidirectionally predicted frame of the other predicted image area and the first reference frame and the denominator of the scale fraction of frame is equal to a second distance between the first reference frame and the second reference frame referenced by the first frame reference, wherein both the first and second reference frames are in a display order that is earlier than a bidirectionally predicted frame display order of the predicted image area. 4. A video decompression system adapted for receiving a sequence of bidirectionally predicted and predicted frames, each comprising pixel values arranged in image areas, wherein at least one of the image areas in at least one of the bidirectionally predicted frames is predicted from image areas in one or more reference frames using direct mode prediction; and determining a predicted motion vector for a bidirectionally predicted frame with a direct mode prediction frame distance method, wherein the frame distance method comprises: given a predicted image area in a first reference frame, the predicted frame bidirectionally with another predicted image area, and a motion vector between the predicted image area in the first reference frame and an image area in a second reference frame, the predicted motion vector is the motion vector multiplied by a frame-scale fraction, wherein the numerator of the frame-scale fraction is equal to a first distance between the bidirectionally predicted frame of the other predicted image area and the first reference frame and the denominator of the scale fraction of frame is equal to a second distance between the first reference frame and the second reference frame referenced by the first frame reference, wherein both the first and second reference frames are in a display order that is earlier than the display order of the bidirectionally predicted frame of the predicted image area.