BR122017003549B1 - DIGITAL VIDEO INFORMATION SPECIFICATION METHOD AND CODING SYSTEM FOR CODING A DIGITAL VIDEO INFORMATION - Google Patents

Abstract

The present invention relates to a method and apparatus for a variable accuracy interimage synchronization specification for digital video encoding which are shown. Specifically, the present invention shows a system that allows relative synchronization of nearby video images to be encoded in a very efficient manner. In one embodiment, the difference in display time between a current video image (105) and a nearby video image is determined. The display time difference is then encoded (180) in a digital representation of the video image. In a preferred embodiment, the next video image is the most recently transmitted stored image. For coding efficiency, the display time difference can be coded using a variable length coding system or an arithmetic coding. In an alternative embodiment, the display time difference is coded as a power of two to reduce the number of bits transmitted.

Description

[001] The present invention relates to the field of multimedia compression systems. In particular, the present invention describes methods and systems for specifying an interimage timing of varying accuracy.

Background of the Invention [002] Digitally based electronic media formats are finally at the apex of largely replacing analog electronic media formats. Digital compact discs (CDs) have long since replaced analog vinyl records. Analog magnetic cassette tapes are becoming increasingly rare. Second- and third-generation digital audio systems, such as mini-discs and MP3 (MPEG Audio - layer 3) are now taking over the first-generation digital audio format market.

[003] Video media was slower to move to digital storage and transmission formats than audio. This was largely due to the massive amounts of digital information required to accurately represent video in digital form. The massive amounts of digital information needed to accurately represent a video require very high-capacity digital storage systems and high-bandwidth transmission systems.

[004] However, the video is now moving rapidly towards digital storage and transmission formats. Processed Petition 870170011651, of 02/21/2017, p. 15/36

2/15 faster computer res, high density storage systems and new efficient compression and encoding algorithms have finally made digital video practical at consumer price points. The DVD (Digital Versatile Disc), a digital video system, was one of the fastest selling consumer electronics products in years. DVDs are quickly supplanting VCR recorders (VCRs) as the pre-recorded video playback system of choice due to their high video quality, very high audio quality, convenience, and extra features. The outdated analog NTSC (National Television Standards Committee) video transmission system is currently in the process of being replaced by the digital ATSC (Advanced Television Standards Committee) video transmission system.

[005] Computer systems have been using several different digital video encoding formats for several years. Among the best digital video compression and encoding systems used by computer systems were the digital video systems supported by the Group of Film Specialists commonly known by the acronym MPEG. The three most well-known and highly used digital video formats in MPEG are known simply as MPEG-1, MPEG-2 and MPEG-4. Video CDs (VCDs) and legacy consumer-grade digital video editing systems use the old MPEG-1 digital video encoding format. Digital Network Versatile Discs (DVDs) and the Direct Network Satellite Diffusion (DBS) television broadcast system by Dish Network use the highest quality MPEG-2 digital video compression and encoding system. The MPEG-4 encoding system is rapidly being adapted by the latest computer-based digital video encoders and associated digital video devices. [006] The MPEG-2 and MPEG-4 standards compress a series of

Petition 870170011651, of 02/21/2017, p. 16/36

3/15 video frames or video fields and then encode the compressed frames or fields into a digital bit stream. When encoding a frame or video field with MPEG-2 and MPEG-4 systems, the frame or video field is divided into a rectangular grid of macroblocks. Each macroblock is independently compressed and coded.

[007] When compressing a frame or video field, the MPEG-4 standard can compress the frame or field into one of three types of frames or compressed fields: intra-frames (frames I), Unidirectional Predicted frames (frames P), or Bidirectional Predicted frames (frames B). Intra-frames encode an independent video frame completely without reference to other video frames. P frames define a video frame with reference to a single previously displayed video frame. B frames define a video frame with reference to a video frame displayed before the current frame and a video frame to be displayed after the current frame. Due to their efficient use of redundant video information, P frames and B frames generally provide the best compression.

Summary of the Invention [008] A method and apparatus for specifying variable accuracy interimaging timing for digital video encoding are shown. Specifically, the present invention shows a system that allows the relative timing of nearby video images to be encoded in a very efficient manner. In one embodiment, the difference in display time between a current video image and a nearby video image is determined. The display time difference is then encoded in a digital representation of the video image. In a preferred embodiment, the next video image is the most recently stored image transPetition 870170011651, of 02/21/2017, p. 17/36

Mitigated.

[009] For coding efficiency, the display time difference can be coded using a variable length coding system or arithmetic coding. In an alternative mode, the difference in display time is coded as a power of two, to reduce the number of bits transmitted.

[010] Other objectives, resources and advantages of the present invention will be evident from the attached drawings and the detailed description below.

Brief Description of the Drawings [011] The objectives, resources and advantages of the present invention will be evident to someone skilled in the art, in view of the detailed description below, in which:

[012] Figure 1 illustrates a high-level block diagram of a possible digital video encoder system.

[013] Figure 2 illustrates a series of video images in the order in which the images are to be displayed, where the arrows connecting different images indicate an inter-image dependency created using motion compensation.

[014] Figure 3 illustrates the video images in Figure 2 listed in a preferred order of transmission of images, where the arrows connecting different images indicate an inter-image dependency created using motion compensation.

[015] Figure 4 graphically illustrates a series of video images in which the distances between video images that reference each other are chosen to be powers of two. Detailed Description of the Preferred Mode [016] A method and system for specifying an Interimage Synchronism of Variable Accuracy in a competition system 870170011651, of 02/21/2017, p. 18/36

5/15 pressure and multimedia coding are described. In the following description, for the purpose of explanation, a specific nomenclature is established to provide a complete understanding of the present invention. However, it will be apparent to one skilled in the art that these specific details are not required in order to practice the present invention. For example, the present invention has been described with reference to the MPEG-4 multimedia encoding and compression system. However, the same techniques can be easily applied to other types of compression and encoding systems.

Multimedia Compression and Encoding Overview [017] Figure 1 illustrates a high-level block diagram of a typical 100 digital video encoder, as is well known in the art. The digital video encoder 100 receives an input video stream of video frames 105 to the left of the block diagram. Each video frame is processed by a Discrete Cosine Transformation (DCT) unit 110. The frame can be processed independently (an intraframe) or with reference to information from other frames received from the motion compensation unit (an interframe) ). Then, a Quantifier unit (Q) 120 quantifies the information from the Discrete Cosine Transformation unit 110. Finally, the quantized video frame is then encoded with an entropy encoder unit (H) 180 to produce a flow of encoded bit. The entropy encoder unit (H) 180 can use a variable length encoding system (VLC).

[018] Once an interframe encoded video frame is defined with reference to other nearby video frames, digital video encoder 100 needs to create a copy of how each decoded frame will appear in a digital video decoder, so that interframe can be coded. Thus, the portion

Petition 870170011651, of 02/21/2017, p. 19/36

The bottom 6/15 of the digital video encoder 100 is actually a digital video decoder system. Specifically, an inverse quantizer unit (Q ^-1 ) 130 reverses the quantification of the video frame information and an inverse Discrete Cosine Transformation unit (DCT ^-1 ) 140 reverses the Discrete Cosine Transformation of the video frame information. After all DCT coefficients are reconstructed from the iDCT, the motion compensation unit will use the information, along with the motion vectors, for the reconstruction of the coded frame, which is then used as the frame of reference for the estimate. motion of the next frame.

[019] The decoded video frame can then be used for encoding interframes (P frames and B frames) that are defined in relation to information in the decoded video frame. Specifically, a motion compensation unit (MC) 150 and a motion estimate unit (ME) 160 are used for the determination of motion vectors and for the generation of differential values used for the encoding of frames.

[020] A rate controller 190 receives information from many different components in a digital video encoder 100 and uses the information to allocate a bit stock to each video frame. The rate controller 190 must allocate the bit stock in a manner that generates the highest quality digital video bit stream that conforms to a specified set of restrictions. Specifically, the rate controller 190 attempts to generate the highest quality compressed video stream without positive overflow plugs (exceeding the amount of memory available in a decoder by sending and more information than can be stored) or overflow plugs. negative (not sending video frames fast enough that a decoder runs out of video frames)

Petition 870170011651, of 02/21/2017, p. 20/36

7/15 video to display).

Multimedia Compression and Encoding Overview [021] On some video signals, the time between successive video images (frames or fields) may not be constant. (Note: this document will use the term video images to refer generically to video frames or video fields). For example, some video images may be dropped because of transmission bandwidth restrictions. Furthermore, the video timing can also vary due to a camera irregularity or special effects, such as slow motion or faster than normal display. In some video streams, the original video source may simply have non-uniform interimage times per project. For example, synthesized video, such as graphic computer animations, can have non-uniform timing, since no arbitrary video timing is created by a uniform video capture system, such as a video camera system. A flexible digital video encoding system must be able to handle non-uniform timing.

[022] Many digital video encoding systems divide video images into a rectangular grid of macroblocks. Each individual macroblock of the video image is independently compressed and encoded. In some embodiments, sub-blocks of macroblocks known as pixel blocks are used. Such pixel blocks can have their own motion vectors that can be interpolated. This document refers to macroblocks, although the teachings of the present invention can be applied equally to both macroblocks and pixel blocks.

[023] Some video encoding standards, for example, the ISO MPEG standards or the ITU H.264 standard, use different types of macroblocks predicted for encoding video images. On a

Petition 870170011651, of 02/21/2017, p. 21/36

8/15 scenario, a macroblock can be of one of three types:

[024] 1. Macroblock I - an intra-macroblock (I) does not use any information from any other video images in its encoding (it is completely self-defined);

[025] 2. Macroblock P - a unidirectional predicted macroblock (P) refers to image information from a previous video image; or [026] 3. Macrobloco B - a bidirectional predicted macroblock (B) uses information from a previous image and a future video image.

[027] If all macroblocks in a video image are intramacroblocks, then the video image is an intraframe. If a video image only includes unidirectional or intramacroblocks predicted macroblocks, then the video image is known as a P frame. If the video image contains any bidirectionally predicted macroblocks, then the video image is known as a Table B. For simplicity, this document will consider the case where all macroblocks in a given image are of the same type.

[028] An example sequence of video images to be encoded could be represented as

Il B2 B3 B4 P5 B6 B7 B8 B9 P10 B11 ^P 12 ^B 13 ^I 14 ...

where the letter (I, P or B) represents whether the video image is an I frame, a P frame or a B frame, and the number represents the camera order of the video image in the video image sequence. The camera order is the order in which a camera recorded the video images, and thus is also the order in which the video images are to be displayed (the display order).

[029] The previous series of example video images is graphically illustrated in Figure 2. With reference to Figure 2, the indiPetition arrows 870170011651, of 02/21/2017, p. 22/36

9/15 show that macroblocks of a stored image (an I frame or a P frame in this case) are used in the prediction of compensated movement of other images.

[030] In the scenario of Figure 2, no information from other images is used when encoding the intraframe I ₁ video image. The P5 video image is a P frame that uses video information from a previous I ₁ video image in its encoding, so that an arrow is drawn from the I ₁ video image to the P ₅ video image. The video image B ₂ , the video image B ₃ , the video image B ₄ all use information from a video image I ₁ and a video image P ₅ in their encoding, so that the arrows are drawn from the video image I ₁ and video image P5 for video image B2, video image B3 and video image B ₄ . As stated above, interimage terms are generally not the same.

[031] Since B images use information from future images (images that will be displayed later), the order of transmission is usually different from the display order. Specifically, the video images that are necessary for the construction of other video images must be transmitted first. For the above sequence, the order of transmission could be:

I1 P5 B2 B3 B4 P10 B6 B7 B8 B9 P12 B11

I14 B13 ...

[032] Figure 3 graphically illustrates the transmission order above the video images in Figure 2. Again, the arrows in the figure indicate which macroblocks of a stored video image (I or P in this case) are used in the prediction of compensated motion other video images.

[033] With reference to Figure 3, the system first transmits a frame II ₁ , which does not depend on any other frame. The system then transmits a PP ₅ frame video image,

Petition 870170011651, of 02/21/2017, p. 23/36

10/15 which depends on the video image I ₁ . The system then transmits the BB ₂ frame video image after the P ₅ video image, although the B ₂ video image is displayed before the P ₅ video image. The reason for this is that when it comes time to decode B ₂ , the decoder will have received and stored the information in the video images I ₁ and P ₅ necessary for decoding a B ₂ video image. Similarly, video images I ₁ and P ₅ are ready to be used for decoding a subsequent B ₃ video image and B ₄ video image. The receiver / decoder reorders the video image sequence for proper display. In these operations, images I and P are often referred to as stored images.

[034] The encoding of P frame images typically uses Motion Compensation, where a Motion Vector is computed for each macroblock in the image. Using the computed motion vector, a prediction macroblock (macroblock P) can be formed by translating pixels into the previous image mentioned above. The difference between the actual macroblock in the P frame image and the prediction macroblock is then encoded for transmission.

[035] Each motion vector can also be transmitted using predictive encoding. For example, a motion vector prediction can be formed using nearby motion vectors. In this case, then, the difference between the actual motion vector and the motion vector prediction is coded for transmission.

[036] Each macroblock B uses two motion vectors: a first motion vector referencing the previous video image mentioned above and a second motion vector referencing the future video image. From these two motion vectors 870170011651, of 02/21/2017, p. 24/36

11/15 development, two prediction macroblocks are computed. The two predicted macroblocks are then combined together, using some function, to form a final predicted macroblock. As above, the difference between the actual macroblock in the B-frame image and the final predicted macroblock is then encoded for transmission.

[037] As with macroblocks P, each motion vector (MV) of a macroblock B can be transmitted through predictive coding. Specifically, a predicted motion vector is formed using nearby motion vectors. Then, the difference between the real and the predicted motion vector is coded for transmission.

[038] However, with macroblocks B, there is an opportunity for interpolation of motion vectors from motion vectors in the nearest stored image macroblock. Such an interpolation is performed on the digital video encoder and digital video decoder.

[039] This motion vector interpolation works particularly well on video images from a video sequence in which a camera is slowly panning across a stationary background. In fact, such a motion vector interpolation may be good enough to be used alone. Specifically, this means that no differential information needs to be calculated or transmitted for these macroblock B motion vectors encoded using interpolation.

[040] For further illustration, in the scenario above, the interimage display time between i and i images is represented as D _ij , that is, if the display times of the images are T _i and Tj, respectively, then

D _i , _j = T _i + Tj from the following that:

Di, _k = D |, j + Dj, _k ^D i, k = ^-D k, i

Petition 870170011651, of 02/21/2017, p. 25/36

12/15

Note that Djj can be negative in some cases.

[041] Thus, if MV ₅ , ₁ is a motion vector for a macroblock P5 as referenced to I1, then, for the corresponding macroblocks in B ₂ , B ₃ and B ₄ , the motion vectors as referenced to I ₁ and P ₅ , respectively, would be interpolated by:

MV ₂ , ₁ = MV ₅ , ₁ * D ₂ , ₁ / D ₅ , ₁ MV5,2 = MV5,1 * D5,2 / D5,1 MV3,1 = MV5,1 * D3,1 / D5,1 MV5 , 3 = MV5,1 * D5,3 / D5,1 ^MV 4,1 ^{= MV} 5,1 ^{* D} 4,1 ^{/ D} 5,1 MV5,4 = ^MV 5,1 ^{* D} 5,4 ^{/ D} 5,1

Note that since display time ratios are used for motion vector prediction, absolute display times are not required. Thus, relative display times can be used for Dij's display time values.

[042] This scenario can be generalized, as, for example, in the H.264 standard. In generalization, a P or B image can use any previously transmitted image for its motion vector prediction. Thus, in the case above, image B3 can use image I ₁ and image B ₂ in its prediction. Furthermore, motion vectors can be extrapolated, not just interpolated. So, in this case, it would have:

MV ₃ , ₁ = MV ₂ , ₁ * D ₃ , ₁ / D ₂ , ₁ [043] Such motion vector extrapolation (or interpolation) can also be used in the prediction process for predictive motion vector coding.

[044] In any event, the problem in the case of non-uniform interimage times is to transmit the relative display time values from Dij to the receiver, and this is the subject of the present invention. In one embodiment of the present invention, for each image

Petition 870170011651, of 02/21/2017, p. 26/36

13/15 after the first image, the display time difference between the current image and the most recently transmitted stored image would be transmitted. For error resilience, the transmission could be repeated several times in the image, for example, in the so-called slice headers of the standard MPEG or H.264 standards. If all slice headers are lost, then, presumably other images that are based on the lost image for decoding information cannot be decoded as well.

[045] Thus, in the scenario above, the following would be transmitted:

D5,1 D2,5 D3,5 D4,5 D10,5 D6,10 D7,10 D8,10 D9,10 D12,10 D11,12 D14,12 D13,14 ··· [046] For the purpose of estimating motion vector, accuracy requirements for D _ij may vary from image to image. For example, if there is only a single BB ₆ frame image halfway between two PP ₅ and P ₇ frame images, then it is sufficient to send only:

D7.5 = 2 and D6.7 = -1 where D _ij display time values are relative time values. If, instead, video image B ₆ is a quarter of the distance between video image P ₅ and video image P ₇ , then the appropriate display time values D _ij to send would be: D7, 5 = 4 and D6.7 = -1

Note that in both of the preceding examples, the display time between the B6 video image and the P7 video image is being used as the display time unit and the difference in display time between the P5 video image and the image P7 image video is four display time units.

[047] In general, a motion vector estimate is less complex if the divisors are powers of two. This is easily achieved in our mode if D _ij (the interimage time) between two stored images is chosen to be a power of two,

Petition 870170011651, of 02/21/2017, p. 27/36

14/15 as illustrated graphically in Figure 4. Alternatively, the estimation procedure could be defined to truncate or round all dividers to a power of two.

[048] In the case where an interimage time is to be a power of two, the number of data bits can be reduced if only the entire power (of two) is transmitted instead of the entire value of the interimage time. Figure 4 graphically illustrates a case in which the distances between the images are chosen to be powers of two. In this case, the display time value D ₃ , ₁ of 2 between the video image P ₁ and the video image P ₃ is transmitted as 1 (since 2 ¹ = 2) and the display time value D ₇ , ₃ of 4 between the video image P7 and the video image of image P3 can be transmitted as two (since 2 ² = 4).

[049] In some cases, a motion vector interpolation may not be used. However, it is still necessary to transmit the display order of the video images to the receiver / executor system, so that the receiver / executor system displays the video images in the appropriate order. In this case, single signed integer values for D _ij are sufficient, regardless of actual display times. In some applications, only the signal may be necessary.

[050] Dij interimage times can simply be transmitted as simple signed integer values. However, many methods can be used to encode Dij values to obtain additional compression. For example, a signal bit followed by a variable length coded magnitude is relatively easy to implement and provides coding efficiency. [051] Such a variable length coding system, which can be used, is known as UVLC (Universal Variable Length Code). The length coding system 870170011651, of 02/21/2017, p. 28/36

15/15 the UVLC variable is given by the code words:

=

2 =

3 =

4 =

5 =

6 =

7 =

8 = 0 [052]

1 0 1

0 ...

Another method of encoding interimage times may be to use arithmetic encoding. Typically, arithmetic coding uses arithmetic probabilities to effect very high compression of data bits.

[053] Thus, the present invention introduces a simple yet powerful method of coding and transmitting interimage display times. The coding of interimage display times can be made very efficient by using variable length coding or arithmetic coding. Furthermore, a desired accuracy can be chosen to suit the needs of the video decoder, but no more.

[054] The precedent described a system for specifying an interimage timing of variable accuracy in a multimedia compression and encoding system. It is contemplated that changes and modifications can be made by someone versed in the technique, materials and arrangements of elements of the present invention, without deviating from the scope of the invention.

Claims (14)

1. Digital video information specification method, characterized by the fact that it comprises: determining a first difference in display time between a first video image (P10) and a nearby video image (I1); determining a second display time difference between a second video image (B2) and said nearby video image (I1), in which the second video image comprises at least one predicted bidirectional macroblock, in which the first video image comprises no predicted bidirectional macroblock and at least one predicted unidirectional macroblock that references the second video image; calculate a particular value that is directly proportional to the first watch time difference and indirectly proportional to the second watch time difference; and calculating a first motion vector of the first video image by multiplying the particular value by a second motion vector of the second video image. 2. Method, according to claim 1, characterized by the fact that it comprises encoding the first video image using the first motion vector. 3. Method, according to claim 1, characterized by the fact that it still comprises transmitting the first video image, the second video image, the next video image, the first difference in display time, and the second difference in display time. 4. Method according to claim 1, characterized by the fact that the next video image is a video image I that does not comprise a macroblock that refers to another image Petition 870170060460, of 08/21/2017, p. 9/14 2/3 of video. 5. Method according to claim 1, characterized by the fact that the second video image is a video image B. 6. Method. according to claim 1, characterized by the fact that the first video image is a video image P. 7. Method, according to claim 1, characterized by the fact that the first video image, the second video image, the next video image, the first difference in viewing time, and the second difference in viewing time are stored in a bit stream. 8. Encoding system (100) for encoding digital video information, characterized by the fact that it comprises: a first difference in display time, said difference in display time specifying a difference between a display time for a first video image (P10) and a display time for a nearby video image (I1); a second difference in display time between a second video image (B2) and said nearby video image (I1), wherein the second video image comprises at least one predicted bidirectional macroblock, wherein the first video image comprises no predicted bidirectional macroblock and at least one predicted unidirectional macroblock that references the second video image; and a first motion vector of the first video image calculated based on a particular value and a second motion vector of the second video image, the particular value directly proportional to the first display time difference and indirectly proportional to the second time difference display. Petition 870170060460, of 08/21/2017, p. 10/14 3/3 9. Encoding system according to claim 8, characterized by the fact that it still comprises a first digital video image which comprises an encoding of the first video image using the first motion vector. 10. Encoding system according to claim 8, characterized by the fact that it still comprises transmitting the first video image, the second video image, the next video image, the first display time difference, and the second difference in display time. 11. Encoding system according to claim 8, characterized by the fact that the next video image is a video image I that does not comprise a macroblock that refers to another video image. 12. Encoding system according to claim 8, characterized by the fact that the second video image is a B video image. 13. Encoding system according to claim 8, characterized by the fact that the first video image is a video image P. 14. Encoding system according to claim 8, characterized by the fact that the first video image, the second video image, the next video image, the first display time difference, and the second time difference displays are stored in a bit stream.