JP2005524355A - H.264 for improved packet loss recovery. 26L flexible data partitioning and packetization - Google Patents

Abstract

H. A method and system for partitioning and packetizing video data in a 26L environment. H. A 26L encoding system (10) is disclosed. The system has a first partition mode and a second partition mode for partitioning video data, wherein the second partition mode is a video coding layer (12) that partitions the low and high frequency DCT coefficients separately. (VCL) and data is packetized into a first packet and a second packet, and when the second partition mode is implemented by the VCL, the first packet is configured to include all low frequency DCT coefficients And the second packet has a network adaptation layer (16) (NAL) configured to include all high frequency DCT coefficients.

Description

  The present invention relates generally to data packetization, and more particularly to partitioning and packetization schemes for transmission of encoded video, improvements in data partitioning syntax, and H.264. It relates to a corresponding network adaptation layer (NAL) packetization process that enables flexible data partitioning in a 26L protocol environment.

  The advent of WLANs with high bandwidth capabilities from several megabits per second to tens of megabits per second has enabled high quality video streaming over such networks. Recently, 802.11b has become a popular standard wireless Ethernet networking technology for both business and home. This has a realistic payload throughput of 6 Mbps and is fast enough for many network applications including encoded video broadcasts.

  However, there are many challenges associated with transmitting high quality wireless video signals, primarily due to limitations associated with bandwidth constraints and high error rates. Since wireless networks are very susceptible to interference from other devices operating in the same frequency band, packet errors or losses can often occur. This is especially true for 802.11b wireless LAN environments that use 2.4 GHz ISM shared by microwave ovens, cordless phones, and / or other 802.11b networks. Another challenge with transmitting video over 802.11b networks is that the 802.11b medium access (MAC) layer was received with bit errors to limit the possibility of error correction at the receiver. The packet needs to be discarded.

  However, the MAC layer and the application layer can provide non-uniform error protection (UEP) for a particular packet to ensure on-time arrival of the particular packet. For example, at least 50% of all packets can be delivered substantially lossless even under joint channel interference that reduces channel throughput by 50%. Through a combination of scalable or layered coding and transmission using UEP, the absolutely essential part of the video can be reached even under channel disturbances, while the non-essential part can only be reached if the channel has sufficient throughput. Can be ensured. Because of the maximum benefits afforded by data partitioning and UEP, an enhancement partition ratio of about 50% base to 50% may be preferred for optimal video quality if enhancement layer packets are lost. . An excessively low partition ratio will not fully utilize the UEP capability of the underlying network.

  Recently, it has been proposed to achieve enhanced compression performance while providing “network friendly” video representations for “conversational” (video telephony) and “non-conversational” (store, broadcast, or streaming) applications. The 26L standard was introduced. H. The 26L standard has a video coding layer (VCL) that provides a high-compression representation of the core of the video image content and a network adaptation layer (NAL) that packages the representation for delivery over a particular type of network.

  Unfortunately, H.M. The current data partitioning syntax in the 26L video coding layer (VCL) provides little flexibility in selecting the partitioning ratio. Fixed partitioning does not fit well with the various non-uniform error protection capabilities provided by various networks such as 802.11a and 802.11b. Fixed partitioning further does not allow rate-distortion optimization of base layer video quality.

  Current H.C. 26L (or joint video team, ie JVT, or MPEG-4 video part 10) has three fixed partition types, namely partition A containing the header symbol of the encoded macroblock and encoding for the intra block. A data partitioning syntax that enables partition B containing block patterns and DCT data and partition C containing coded block patterns and DCT data for inter blocks is specified in a byte stream (video elementary stream). H. 26L further specifies in the network adaptation layer (NAL) a packetization process that packetizes the three partitions into three packets. The three packets have different transport (such as RTP, ie real-time transport protocol) payload types, which can be used to differentiate services or non-uniform errors at the application layer or below the network transport layer. Instruct them to give protection. On the receiver side, packets containing various partitions are de-packetized and merged into one bitstream (with multiple partitions) for decoding. Although this system provides some level of resiliency to packet loss, it has several drawbacks.

  First, the fixed partitioning and NAL packetization process does not allow any rate-distortion optimization of the partitioning operation. This is compared to the flexible adaptive data partitioning where the partitioning points and corresponding base-to-enhancement layer packet boundaries can be adaptively changed based on image statistics (enhancement layer Results in low video quality (when data is lost).

  Second, fixed partitioning and packetization limits the base and enhancement layer ratio to a small range determined by the amount of header and motion vector information to DCT data. H. Since the 26L standard is designed for many applications such as home cinema and video streaming, the application or the network below it will have different capabilities for non-uniform error protection. Fixed ratios for base and enhancement layer partitions will not allow for optimal overall system performance in systems where non-uniform error protection is available.

  Therefore, it is absolutely essential to support video communication applications over various networks with high packet loss. Improvements are needed to provide greater data partitioning flexibility in the 26L environment.

  The present invention addresses the above and other problems by modifying the VCL data partitioning syntax and the corresponding NAL packetization process to allow flexible data partitioning. In a first aspect, the present invention relates to H.264. A 26L encoding system is provided. This H. The 26L coding system has a first partition mode and a second partition mode for partitioning video data, the second partition mode being a video coding layer that partitions the low and high frequency DCT coefficients separately ( VCL), when data is packetized into a first packet and a second packet, and when the second partition mode is implemented by the VCL, the first packet is configured to include a low frequency DCT coefficient, The two packets have a network adaptation layer (NAL) configured to include high frequency DCT coefficients.

  In a second aspect, the present invention relates to H.264. A method for partitioning and packetizing video data in a 26L environment is provided. The method includes providing a video coding layer (VCL) having a first partition mode and a second partition mode for partitioning video data, and when the first partition mode is selected. Partitioning the data into three partitions (A, B, and C) and, when the second partition mode is selected, the video data into the first partition (including header information and low frequency DCT data); Partitioning in a second partition (high frequency DCT data).

  In a third aspect, the present invention relates to H.264. A program product stored on a recordable medium for packetizing and partitioning video data in a 26L environment is provided. The program product includes a video coding layer (VCL) having a first partition mode and a second partition mode for partitioning video data, and video data when the first partition mode is selected. Means for partitioning into three partitions (A, B, and C), and when the second partition mode is selected, the video data is divided into the first partition (including header information and low frequency DCT data) and the first Means for partitioning into two partitions (high-frequency DCT data);

  In a fourth aspect, the present invention relates to H.264. A decoding system for decoding video data in a 26L environment is provided. In this decoding system, the video data is packetized in the first packet type, the header data is packetized in the second packet type, and the coding block pattern and DCT data for the intra block are packetized. The coded block pattern and DCT data are packetized in a third packet type, the header data and low frequency DCT coefficients are packetized in the first packet type, and the high frequency DCT coefficients are Packetized in one of the second schemes packetized into two packet types, the decoding system determines the scheme used between the first scheme and the second scheme; Video and non-packetized systems that depacketize video data from packets Having a decoder for decoding the over data.

  The foregoing and other features of the present invention will be readily understood from the following detailed description of various aspects of the invention, taken together with the accompanying drawings.

  Referring to FIG. 26L encoding system 10 and H.264. 2 shows a video transmission network for transmitting a byte stream to and from a 26L decoding system 18. H. The 26L encoding system 10 includes a video encoding layer (VCL) 12 and a network adaptation layer (NAL) 16. As is well known in the art, the VCL 12 has a unique syntax that effectively represents the content of the video data, and the NAL 16 formats the data and makes the header information suitable for transport by high level systems. Is defined to supply Data is organized into data packets, each data packet having an integer number of bytes. These data packets are then transmitted in a manner defined by NAL 16.

  In data partitioning, all symbols of one data type (eg, DC coefficient, microblock header, motion vector) belonging to a single slice are collected into one VLC encoded bitstream starting with byte alignment. Reconstruct symbols in such a way. Decoding system 18 can process the partitioned data stream by fetching symbols from the correct partition.

In the present invention, the VCL 12 has a data partitioning system 14 that can be partitioned based on one of two modes: Mode One (M1) and Mode Two (M2). Mode One with header vs. DCT data is the current H.264 standard. It is defined syntactically by the 26L specification. Current H.C. The 26L specification defines that there are eight syntax element types in VCL as shown below.

0-TYPE_HEADER
1-TYPE_MBHEADER
2-TYPE_MVD
3-TYPE_CBP
4-TYPE_2 × 2DC
5-TYPE_COEFF_Y
6-TYPE_COEFF_C
7-TYPE_EOS

Partition mode one is the current H.264 standard. Since it does not change from the 26L specification, backward compatibility is maintained. Mode to provide DCT partitioning is added to give increased partitioning flexibility. As described below, the NAL packetization scheme varies depending on the partition mode used in the byte stream.

In partition mode-to-that is, DCT partitioning, TYPE_COEFF_Y and TYPE-COEFF_C are further divided into two new partitions representing high and low frequency DCT coefficients, respectively. That is, TYPE_COEFF_Y is divided into TYPE_COEFF_Y_L and TYPE_COEFF_Y_H, and TYPE-COEFF_C is divided into TYPE-COEFF_C_L and TYPE-COEFF_C_H. It should be understood that the choice of naming method for these new types can be changed without departing from the scope of the present invention. Thus, for mode-to, the data partitioning system provides 10 syntax element types as follows:

0-TYPE_HEADER
1-TYPE_MBHEADER
2-TYPE_MVD
3-TYPE_CBP
4-TYPE_2 × 2DC
5-TYPE_COEFF_Y_L
6-TYPE_COEFF_C_L
7-TYPE_COEFF_Y_H
8-TYPE_COEFF_C_H
9-TYPE_EOS

In addition, a new field, Packetization Boundary Indication (PBI) is added at the end of TYPE_HEADER. An exemplary field structure for PBI is shown in FIG. As shown, the PBI field is further divided into three subfields. The first subfield is the packetization division point (PBP) (2 bits), which indicates which partition packetization is to be divided. That is, by changing the PBI, the user can select which packets should contain TYPE_CBP and TYPE_2 × 2DC. The second subfield is the partition type (PT) (1 bit), which is set to 0 and reserved by 1. The third subfield is the DCT split point (DBP) (5 bits), which is the first DCT run-length VLC pair in TYPE_COEFF_X_H (where X is either Y or C). Indicates the beginning of the index. Thus, the pre-selected PBI identifies the boundary between the high frequency DCT coefficient TYPE_COEFF_X_H and the low frequency DCT coefficient TYPE_COEFF_X_L. Of course, other PBI structures other than those described in FIG. 2 are contemplated to achieve the same functionality. Such other structures are within the scope of the present invention.

  As described above, the NAL packetization process creates two packets, each packet containing several different types of partitioned data. In the present invention, the packetization scheme is selected based on the mode implemented by the VCL 12. Thus, for example, partition mode one results in packetization scheme one being implemented, while partition mode two results in packetization scheme two being implemented. The NAL 16 can determine the scheme to use based on whether the TYPE_HEADER partition ends with a PBI field.

  If the PBI field is not included, the packetization scheme (P1) is used, which is the current H.264 standard. Reflects the scheme used for 26L use. As expected, all header and motion vector information is included in packet one, all intra-frame encoded block patterns and DCT data information are included in packet two, and all inter-frame encoded blocks The pattern (CBP) and DCT information are included in the packet three.

However, if the TYPE_HEADER partition ends with a PBI field, packetization operates under Scheme Two (P2). In this case, the content of the resulting packet depends on the value in the PBI field. The following is an example having a PBP value of 2 and a PT value of 0.

Packet one

TYPE_HEADER (with PBI field)
TYPE_MBHEADER
TYPE_MVD
TYPE_CBP
TYPE_2 × 2DC
TYPE_COEFF_Y_L
TYPE_COEFF_C_L

Packet toe

TYPE_COEFF_Y_H
TYPE_COEFF_C_H
TYPE_EOS

In this scheme-two example, the low frequency DCT coefficients are packetized in packet one, while the high frequency DCT coefficients are packetized in packet two. Thus, flexible packetization is achieved. As will be apparent, the contents of the packet under the scheme-to-lower are changed by changing the PBP value in the PBI field. Accordingly, it should be understood that various variations are achievable under Scheme Two.

  In FIG. A 26L decoding system 18 is shown, which has a non-packetized system 20 and a decoder 22. Prior to decoding, the packet is depacketized into 8 (packetization scheme one) or 10 (packetization scheme to) partitions for decoding. The decoder 22 fetches data from the correct partition depending on the scheme being implemented, such as Scheme One (S1) or Scheme Two (S2).

  It is to be understood that the systems, functions, mechanisms, methods, algorithms, and modules described in this application can be implemented by hardware, software, or a combination of hardware and software. These may be implemented by any type of computer system or other device adapted to perform the methods described herein. A common hardware and software combination would be a general purpose computer system having a computer program that, when loaded and executed, controls the computer system to perform the methods described herein. Alternatively, a special purpose computer having special hardware to perform one or more functional tasks of the present invention can be used. The present invention has all the features that enable the implementation of the methods and functions described herein and is embedded in a computer program product capable of performing these methods and functions when loaded into a computer system. Is also possible. In the current context, a computer program, software program, program, program product, or software can convert a specific function directly or into (a) another language, code, or scale system: And / or (b) any language, code, or scale of a set of instructions intended to cause a system with information processing functions to be performed after either or both of the reproduction in various material forms. Means notation.

  The foregoing description of the preferred embodiment of the present invention has been presented for purposes of illustration and description. These are not exhaustive or limit the invention to the precise forms disclosed, and of course many modifications and variations are possible in view of the above teachings. Such modifications and variations that may be apparent to a person skilled in the art are intended to be within the scope of the following claims.

In accordance with one embodiment of the present invention, H.264. It is a figure which shows a 26L video transmission system. FIG. 4 is a diagram illustrating a packetization boundary indication (PBI) field according to one embodiment of the present invention.

Claims (15)

H. 26L encoding system,
A video coding layer (VCL) for partitioning the low and high frequency DCT coefficients separately, having a first partition mode and a second partition mode for partitioning video data;
Data is packetized into a first packet and a second packet, and when the second partition mode is implemented by the VCL, the first packet is configured to include a low frequency DCT coefficient; The two packets are configured to include a high frequency DCT coefficient, a network adaptation layer (NAL);
Including system.   The first partition mode includes a first partition including header symbols of encoded macroblocks, a second partition including encoded block patterns and DCT data for intra blocks, and encoded block patterns for inter blocks. And a third partition containing DCT data. 26L encoding system.   The second partition mode includes TYPE_HEADER, TYPE_MBHEADER, TYPE_MVD, TYPE_CBP, TYPE_2 × 2DC, and TYPE_EOS, and divides each of TYPE_COEFF_Y and TYPE_COEFF_C into a high frequency type and a low frequency type. 26L encoding system.   The second partition mode includes: TYPE_HEADER, TYPE_MBHEADER, TYPE_MVD, TYPE_CBP, TYPE_2 × 2DC, TYPE_EOS, TYPE_COEFF_Y_L, TYPE_COEFF_C_L, TYPE_COEFF_Y_H, and 26L encoding system.   The NAL packetizes TYPE_HEADER, TYPE_MBHEADER, and TYPE_MVD into the first packet when the first partition mode is used, and TYPE_CBP, TYPE_2 × 2DC, TYPE_COEFF_Y, TYPE_COEFF_C, and TYPE2_OS The H.264 of claim 4 packetized into packets. 26L encoding system.   The NAL packetizes TYPE_COEFF_Y_L and TYPE_COEFF_C_L in the first packet and packetizes TYPE_COEFF_Y_H and TYPE_COEFF_C_H in the second packet when the second partition mode is used. 4 described above. 26L encoding system.   TYPE_HEADER has a field having a packetization boundary instruction for determining a division point between a high frequency DCT coefficient and a low frequency DCT coefficient, and instructs the NAL to partition the high frequency DCT coefficient and the low frequency DCT coefficient. The H.I. 26L encoding system.   8. The H.264 packet of claim 7, wherein the packetization boundary indication further determines a packet that should include TYPE_CBP, TYPE_2 × 2DC. 26L encoding system. H. A method of partitioning and packetizing video data in a 26L environment, comprising:
When the first partition mode is selected, the header data is in the first packet, the encoded block pattern and DCT data for the intra block are in the second packet, and the encoded block pattern and DCT data are in the inter block. Packetizing into a third packet;
Packetizing header data and low frequency DCT coefficients into a first packet and high frequency DCT coefficients into a second packet when the second partition mode is selected;
Having a method. Providing a video coding layer (VCL) for partitioning the video data;
When the first mode is used, TYPE_HEADER, TYPE_MBHEADER, and TYPE_MVD are packetized in the first packet, and TYPE_CBP, TYPE_2 × 2DC, TYPE_COEFF_Y, TYPE_COEFF_C, and TYPE_EOS in the second packet Packetizing, and
Packetizing TYPE_COEFF_Y_L and TYPE_COEFF_C_L into the first packet and packetizing TYPE_COEFF_Y_H and TYPE_COEFF_C_H into the second packet when the second mode is used;
10. The method of claim 9, further comprising: Setting a dividing point between the high frequency DCT coefficient and the low frequency DCT coefficient;
Storing the division point in a boundary indication field in TYPE_HEADER;
The method of claim 10 further comprising:   When the boundary indication field is included in TYPE_HEADER, TYPE_COEFF_Y_L and TYPE_COEFF_C_L are packetized in the first packet, and TYPE_COEFF_Y_H and TYPE_COEFF_C_H are packetized in the second packet. Item 12. The method according to Item 11.   The method of claim 11, wherein the boundary indication field further determines a packet to include TYPE_CBP and TYPE_2 × 2DC. H. A program product stored on a recordable medium for packetizing and partitioning video data in a 26L environment,
When the first partition mode is selected, the header data is in the first packet, the encoded block pattern and DCT data for the intra block are in the second packet, and the encoded block pattern and DCT data are in the inter block. Means for packetizing into a third packet;
Means for packetizing header data and low frequency DCT coefficients into a first packet and high frequency DCT coefficients into a second packet when the second partition mode is selected;
Program product with H. A decoding system for decoding video data in a 26L environment,
In the video data, header data is packetized in a first packet type, a coded block pattern for an intra block and DCT data is packetized in a second packet type, and a coded block pattern for an inter block. And DCT data is packetized in a third packet type, header data and low frequency DCT coefficients are packetized in the first packet, and high frequency DCT coefficients are in the second packet. Packetized in one of the second schemes to be packetized,
The decryption system comprises:
A non-packetizing system that determines which one of the first scheme and the second scheme is used and depacketizes video data from the packet;
A decoder for decoding the video data;
A decoding system.

Images