JP6309463B2 - System and method for providing error resilience, random access, and rate control in scalable video communication - Google Patents

Description

Cross-reference to related applicationsThis application is a U.S. provisional application No. 60 / 778,760 filed on Mar. 3, 2006, U.S. Provisional Application No. 60 / 787,031 filed on Mar. 29, 2006, and Oct. 23, 2006. Claims the benefit of US Provisional Application No. 60 / 862,510, filed on the same day. Further, the present application is related to International Patent Application No. PCT / US06 / 28365, International Patent Application No.PCT / US06 / 028366, International Patent Application No.PCT / US06 / 028367, International Patent Application No.PCT / US06 / 028368. International Patent Application No. PCT / US06 / 061815, International Patent Application No.PCT / US06 / 62569, and International Patent Application No.PCT / US07 / 62357, and US Provisional Application No. 60 / 884,148, US Provisional Application No. 60. / 786,997, and US Provisional Application No. 60 / 829,609. All of the aforementioned priority and related applications that are assigned to the assignee of the present application are hereby incorporated by reference in their entirety.

  The present invention relates to a video data communication system. The present invention specifically relates to simultaneously providing error resilience, random access, and rate control functions in a video communication system that utilizes scalable video coding techniques.

  Transmission of digital video in packet-based networks, such as networks based on the Internet Protocol (IP), is extremely difficult due to at least the fact that data transport is usually done on a best effort basis. In modern packet-based communication systems, errors usually appear as packet loss rather than bit errors. Furthermore, such packet loss is usually the result of congestion at intermediate routers and not the result of physical layer errors (one exception to this is wireless networks and cellular networks). When an error occurs in transmitting or receiving a video signal, it is important to ensure that the receiver can quickly recover from the error and return to an error-free display of the incoming video signal. However, in typical digital video communication systems, receiver robustness is reduced by the fact that incoming data is strongly compressed to save bandwidth. In addition, video compression techniques used in communication systems (for example, the latest codecs ITU-T H.264 and H.263 or ISO MPEG-2 codec and MPEG-4 codec) are used for sequential video packets or sequential. It can create a very strong temporal dependency between video frames. Specifically, the use of motion compensated prediction (e.g., with the use of P or B frames) codecs is a chain of frame dependencies where the displayed frame depends on the past frame (s). Produce. The dependency chain can extend to the beginning of the video sequence. As a result of the dependency chain, the loss of a particular packet can affect the decoding of multiple subsequent packets at the receiver. Error propagation due to the loss of a particular packet ends only in frames that do not use "intra" (I) refresh points or temporal prediction at all.

  Error resilience in digital video communication systems requires having at least some level of redundancy in the transmitted signal. However, this requirement is incompatible with the purpose of video compression techniques that attempt to eliminate or minimize redundancy in the transmitted signal.

  On networks that provide differentiated services (for example, DiffServ IP-based networks, private networks over leased lines, etc.), video data communication applications use network functions to receive some or all of the video signal data. Can be delivered lossless or almost lossless. However, in any best effort network (such as the Internet) that does not provide differentiated services, data communication applications must rely on their own capabilities to achieve error resilience. Well-known techniques useful in general data communication (eg, Transmission Control Protocol-TCP) are not suitable for video or audio communication, which adds the low end-to-end delay constraints that arise from human interface requirements. For example, TCP techniques can be used for error resilience in data transport using a file transfer protocol. TCP continues to retransmit data until confirmation that all data has been received, even with a delay of a few seconds. However, TCP is unsuitable for video data transport in live or interactive video conferencing applications, as endless end-to-end delays are unacceptable for participants.

  A related problem is that of random access. Assume that the receiver participates in an existing transmission of the video signal. A typical example is a user attending a video conference or a user tuned to a broadcast. Such a user must find a point in the incoming bitstream where the user can start decoding and where the user synchronizes with the encoder. However, providing such a random access point has a significant impact on compression efficiency. Note that any error propagation ends at a random access point, so by definition that point is an error resilience function (ie, that point is an error recovery point). Therefore, the better the random access support provided by a particular coding scheme, the faster the error recovery achieved by that particular coding scheme. The converse is not always true and depends on assumptions made about the duration and extent of errors that the error resilience technique is designed to address. For error resilience, it can be assumed that the status information on the receiving side is available when an error occurs.

  As an example, in an MPEG-2 video codec for a digital television system (digital cable TV or satellite TV), I pictures are used at periodic intervals (typically 0.5 seconds) to allow quick switching to a stream Is done. However, an I picture is significantly larger (usually 3-6 times) than its P or B counterpart, and therefore must be avoided, especially in low bandwidth and / or low delay applications.

  In interactive applications such as video conferencing, the concept of requesting an intra update is often used for error resilience. In operation, the update includes a request from the receiving side to the transmitting side for intra-picture transmissions that allow the decoder to synchronize. The bandwidth overhead of this operation is considerable. Furthermore, this overhead is also incurred when a packet error occurs. If packet loss is caused by congestion, the use of intra pictures only exacerbates the congestion problem.

  Another conventional technique for error resilience, used in the past to mitigate drift caused by inconsistencies in IDCT implementations (e.g. H.261 standard), is to periodically each macroblock in intra mode. Is to code. The H.261 standard requires forced intracoding every time a macroblock is transmitted 132 times.

  Coding efficiency decreases as the percentage of macroblocks that are forced to be coded as intra in a given frame increases. Conversely, when this percentage is low, the time to recover from packet loss increases. The forced intracoding process requires extra care to avoid motion-related drift, which further limits the performance of the encoder. This is because even if a certain motion vector value is most effective, it must be avoided.

  In addition to conventional single layer codecs, layered coding or scalable coding is a well-known technique in multimedia data coding. Scalable coding is used to generate two or more “scaled” bitstreams that collectively represent a given medium in a bandwidth efficient manner. Scalability can be provided in several different dimensions: time, space, and quality (also referred to as SNR “signal to noise ratio” scalability or fidelity scalability). For example, video signals can be scalable coded as different layers at frame rates 7.5, 15, and 30 frames per second (fps) with CIF and QCIF resolution. Depending on the structure of the codec, any combination of spatial resolution and frame rate can be obtained from the codec bitstream. Bits corresponding to different layers can be transmitted as separate bitstreams (ie, one stream per layer) or they can be multiplexed together as one or more bitstreams. For the purposes of this description, the coded bits corresponding to a particular layer may be referred to as that layer's bitstream, even if the various layers are multiplexed and transmitted as a single bitstream. is there. Codecs that are specifically designed to provide scalability features include, for example, MPEG-2 (ISO / IEC13818-2, also known as ITU-T H.262), currently developed SVC (ITU-T H.264 Annex G or MPEG-4 Part 10 SVC). A scalable coding technique specifically designed for video communication is described in International Patent Application PCT / US06 / 028365, "SYSTEM AND METHOD FOR SCALABLE AND LOW-DELAY VIDEOCONFERENCING USING SCALABLE VIDEO CODING", assigned to the assignee of the present application. . Note that even codecs that are not specifically designed to be scalable can exhibit scalability characteristics in the time dimension. For example, consider the MPEG-2 Main Profile codec, which is a non-scalable codec used in DVD and digital TV environments. Furthermore, it is assumed that the codec operates at 30 fps and uses the GOP structure (period N = 15 frames) of IBBPBBPBBPBBPBB. Sequential removal of B pictures followed by removal of P pictures leads to a total of three temporal resolutions of 30 fps (including all picture types), 10 (I and P only), and 2 fps (I only) It is possible. Since the MPEG-2 Main Profile codec is designed so that the coding of P pictures does not depend on B pictures, and similarly the coding of I pictures does not depend on other P pictures or B pictures, decoding as a result of the sequential removal process A possible bitstream is obtained. In the following, a single layer codec with temporal scalability capability is considered to be a special case of scalable video coding and is therefore included in the term scalable video coding unless there is an explicit indication.

  A scalable codec typically has a pyramidal bitstream structure that is essential in recovering the original medium with a basic quality of one of the constituent bitstreams (called the “base layer”) . Using one or more remaining bitstreams (referred to as “enhancement layer”) with the base layer improves the quality of the recovered media. While loss of data at the enhancement layer is acceptable, loss of data at the base layer can cause significant distortion, or the recovered media can be completely lost.

  Scalable codecs pose challenges similar to those raised by single-layer codecs for error resilience and random access. However, the coding structure of a scalable codec has unique characteristics that do not exist in a single layer video codec. Further, unlike single layer coding, scalable coding can include switching from one scalability layer to another (eg, switching back and forth between CIF resolution and QCIF resolution). Instantaneous layer switching with very little bit rate overhead when switching between different resolutions is a scalable coding system where multiple signal resolutions (space / time / quality) may be available from the encoder, random access Desirable for.

  A problem associated with error tolerance and random access problems is the problem of rate control. The output of a typical video encoder will be a variable bit rate due to extensive use of prediction, transformation, and entropy coding techniques. Buffer limited rate control is typically used in video communication systems to construct a constant bit rate stream. In such a system, an encoder output buffer is assumed, which is emptied at a constant rate (channel rate), and the encoder monitors buffer occupancy and prevents buffer overflow or buffer underflow. Perform parameter selection (eg, quantizer step size). However, such a rate control mechanism can only be applied at the encoder and further assumes that the desired output rate is known. In some video communication applications, including video conferencing, such a rate control decision is an intermediate gateway (e.g., Multipoint Control Unit or MCU) that is placed between the sender and receiver It is desirable to be performed at. Bitstream level manipulation or transcoding can be used at the gateway, but with significant processing and complexity costs. Therefore, it is desirable to use a technique that achieves rate control without requiring any additional processing at the intermediate gateway.

  We will now consider improved error resilience and random access to the coded bitstream as well as rate control in video communication systems. Attention is directed to the development of error resilience techniques, rate control techniques, and random access techniques with minimal impact on end-to-end delay and bandwidth used by the system.

  The present invention provides systems and methods that increase error resilience and provide random access and rate control functions in video communication systems that use scalable video coding. This system and method also allows the derivation of output signals at a resolution different from the coded resolution with excellent rate-distortion performance.

  In one embodiment, the present invention provides a mechanism to recover from the loss of a high resolution spatial scalable layer packet by using information from the low resolution spatial layer. In another embodiment, the present invention provides a mechanism for switching from low spatial resolution or low SNR resolution to high spatial resolution or high SNR resolution with little or no delay. In another embodiment, the present invention contemplates that the encoder or intermediate gateway (e.g., MCU) will use a suitable error recovery mechanism that minimizes the impact of lost packets on the quality of the received signal at the receiver. Provide a mechanism to perform rate control, selectively removing high resolution spatial layer packets. In another embodiment, the encoder or intermediate gateway re-approximates the packet from the high-resolution spatial layer to the high-resolution data that is replaced by using information from the base layer and past frames in the enhancement layer. It selectively replaces with information that efficiently instructs the encoder to configure. In another embodiment, the present invention describes a mechanism for deriving an output video signal with a resolution different from the coded resolution, in particular an intermediate resolution between the resolutions used for spatial scalable coding. These embodiments, alone or in combination, allow the construction of video communication systems with significant rate control and resolution flexibility as well as error resilience and random access.

  The inventive system and method is based on an “error concealment” technique used in conjunction with a scalable coding technique. These techniques simultaneously achieve error resilience and rate control for a particular sequence of video encoders called scalable video encoders. The rate-distortion performance of the error concealment technique matches or exceeds the rate-distortion performance of coding at the effective transfer rate (the total transmission rate minus the rate of lost packets). By appropriate selection of picture coding structure and transport mode, these techniques allow near instantaneous layer switching with very little bit rate overhead.

  Further, these techniques can be used to derive a decoded version of the received signal that has a resolution that differs from the coded resolution (s). This allows, for example, the creation of a 1/2 CIF (HCIF) signal from a spatially scalable coded signal of QCIF resolution and CIF resolution. Unlike normal scalable coding, the receiver must use the QCIF signal and upsample it (with low quality), or use the CIF signal and downsample it (good quality). With high bit rate utilization). The same problem exists when QCIF and CIF are broadcast simultaneously as a single layer stream.

  These techniques also provide rate control with minimal processing of the encoded video bitstream and without adversely affecting image quality.

  Further features of the invention, its nature and various advantages will be more apparent from the following detailed description of the preferred embodiments and the accompanying drawings.

  Throughout the drawings, identical reference numerals and characters are used to indicate the same features, elements, components, or parts of the illustrated embodiments unless otherwise indicated. Further, the present invention will now be described in detail with reference to the drawings, which are done in connection with the exemplary embodiments.

1 is a block diagram illustrating the overall architecture of a video conferencing system in accordance with the principles of the present invention. Examples expressly is a block diagram showing an end-user terminal. FIG. 2 is a block diagram illustrating an exemplary end user terminal in accordance with the principles of the present invention. FIG. 3 illustrates an exemplary picture coding structure in accordance with the principles of the present invention. FIG. 6 illustrates an example of an alternative picture coding structure according to the principles of the present invention. FIG. 2 is a block diagram illustrating an exemplary architecture of a video encoder for a spatial enhancement layer according to the principles of the present invention. FIG. 3 illustrates an exemplary picture coding structure when spatial scalability is used in accordance with the principles of the present invention. FIG. 6 illustrates an exemplary decoding process with enhancement layer picture concealment according to the principles of the present invention. FIG. 5 shows an exemplary RD curve of concealment processing when applied to a “foreman” sequence according to the principles of the present invention. FIG. 6 illustrates an exemplary picture coding structure when spatial scalability using SR pictures is used in accordance with the principles of the present invention.

  Systems and methods for error tolerant transmission, random access, and rate control in video communication systems are provided. The system and method takes advantage of error concealment techniques based on features of scalable video coding that can be used in video communication systems.

  In a preferred embodiment, the exemplary video communication system may be a multipoint videoconferencing system 10 operated over a packet-based network (see, eg, FIG. 1). The multipoint videoconferencing system can include optional bridges 120a and 120b (e.g. multipoint control) to mediate scalable multilayer or single layer video communication between endpoints (e.g. users 1-k and 1-m) over the network. A unit (MCU) or a scalable video communication server (SVCS)) can be included. The operation of the exemplary video communication system is the same and advantageous for point-to-point connections with or without the use of optional bridges 120a and 120b. The techniques described in the present invention are directly applicable to all other video communication applications, including point-to-point streaming, broadcasting, multicasting, etc.

  Detailed descriptions of scalable video coding techniques and video conferencing systems based on scalable video coding are provided, for example, in International Patent Applications Nos. PCT / US06 / 28365 and PCT / US06 / 28366 assigned to the assignee of the present application. The In addition, descriptions of scalable video coding techniques and video conferencing systems based on scalable video coding are provided in International Patent Applications Nos. PCT / US06 / 62569 and PCT / US06 / 061815, assigned to the assignee of the present application.

  FIG. 1 shows a general structure of the video conference system 10. The video conference system 10 includes a plurality of end user terminals (for example, a user 1-k and a user 1-m) linked via the network 100 through the LANs 1 and 2 and the servers 120a and 120b. The server may be a conventional MCU or may be a scalable video coding server (SVCS) or a synthetic scalable video coding server (CSVCS). The latter server has the same purpose as a conventional MCU, but with significantly reduced complexity and improved functionality (see, eg, International Patent Applications PCT / US06 / 28366 and PCT / US06 / 62569). In the description herein, the term “server” may be used generically to refer to SVCS or CSVCS.

FIG. 3 illustrates the architecture of an end user terminal 140 designed for use with a video conferencing system (eg, system 100) based on multi-layer coding. Terminal 140 includes human interface input / output devices (e.g., camera 210A, microphone 210B, video display 250C, speaker 250D) and input and output signal multiplexer and signal demultiplexer units (e.g., packet MUX 220A and packet DMUX 220B). One or more network interface controller cards (NICs) 230 coupled to each other. The NIC 230 can be a standard hardware component such as an Ethernet LAN adapter, any other suitable network interface device, or a combination thereof.

  Camera 210A and microphone 210B are designed to capture participant video signals and participant audio signals, respectively, for transmission to other conference participants. Conversely, video display 250C and speaker 250D are designed to display and play video and audio signals received from other participants, respectively. Video display 250C may also be configured to optionally display the video of participant / terminal 140 itself. Camera 210A output and microphone 210B output are coupled to video encoder 210G and audio encoder 210H via analog-to-digital converters 210E and 210F, respectively. Video encoder 210G and audio encoder 210H are designed to compress the input video digital signal and the input audio digital signal to reduce the bandwidth required for transmission of the signal over the electronic communication network. The input video signal may be a live video signal or a pre-recorded and stored video signal. The encoder compresses the local digital signal to minimize the bandwidth required for signal transmission.

  In an exemplary embodiment of the invention, the audio signal is encoded using any suitable technique known in the art (e.g., G.711, G.729, G.729EV, MPEG-1, etc.). Can do. In the preferred embodiment of the present invention, the scalable audio codec G.729EV is used in the audio encoder 210G to encode the audio signal. The output of the audio encoder 210G is sent to the multiplexer MUX220A, and transmitted via the network 100 through the NIC230.

  The packet MUX 220A can perform conventional multiplexing using the RTP protocol. The packet MUX 220A may implement any associated quality of service (QoS) processing that can be provided in the network 100 or directly in a video communication application (see, eg, International Patent Application PCT / US06 / 061815). Each stream of data from the terminal 140 is transmitted on its own virtual channel, or “port number” in IP terminology.

  Video encoder 210G is a scalable video encoder having multiple outputs corresponding to various layers (labeled “base” and “extension” in this figure). Note that simultaneous broadcasting is a special case of scalable coding where no inter-layer prediction is performed. In the following, when the term scalable coding is used, this term scalable coding includes simultaneous broadcast cases. The operation of the video encoder and the nature of the outputs are described in more detail herein below.

The H.264 standard specification allows multiple participant views to be combined into a single coded picture by using a flexible macroblock ordering (FMO) scheme. In this scheme, each participant occupies a portion of the coded image corresponding to one of its slices. Conceptually, a single decoder can be used to decode all participant signals. However, from a practical point of view, the receiver / terminal must decode multiple smaller independently coded slices. Therefore, the terminal 140 shown in FIG. 3 having the decoder 230A can be used for an application of the H.264 specification. Note that the server that transfers the slice is CSVCS.

  At terminal 140, demultiplexer DMUX 220B receives packets from NIC 320 and redirects those packets to the appropriate decoder unit 230A.

  The SERVER CONTROL block in terminal 140 allows interaction between the server (SVCS / CSVCS) and the end user terminal as described in International Patent Applications Nos. PCT / US06 / 028366 and PCT / US06 / 62569. adjust. In a point-to-point communication system without an intermediate server, the SERVER CONTROL block is not necessary. Similarly, in non-conferencing applications, in point-to-point conferencing applications, or when CSVCS is used, only a single decoder may be required at the receiving end-user terminal. For applications involving stored video (e.g. pre-recorded and pre-coded material broadcasts), the transmitting end-user terminal will have the full functionality of the audio and video coding blocks as well as It may not require all the preceding blocks (camera, microphone, etc.). Specifically, it is only necessary to provide a portion related to the selective transmission of video packets described below.

  Although the word “terminal” is used in this context, the various components of the terminal may be separate devices interconnected with each other and may be integrated as software or hardware within a personal computer, or It can be a combination thereof.

A base layer video encoder 300 illustrating an exemplary architecture of a video encoder (base layer and time enhancement layer) will be described . Encoder 300 includes FRAME BUFFERS block 310 and encoder reference control (in addition to conventional `` text-book '' variety video coding processing block 330 for motion estimation (ME), motion compensation (MC), and other coding functions. ENC REF CONTROL) block 320 is included. Video encoder 300 is, for example, H.264 / MPEG-4 AVC (ITU-T and ISO / IEC JTC 1, `` Advanced video coding for generic audiovisual services '', ITU-T Recommendation H.264 and ISO / IEC 14496-10 (MPEG4-AVC)) or SVC (J.Reichel, H.Schwarz, and M.Wien, `` Joint Scalable Video Model JSVM 4 '', JVT-Q202, Document of Joint Video Team (JVT) of ITU T SG16 / Q .6 and ISO / IEC JTC 1 / SC 29 / WG 11, October 2005). It should be understood that any other suitable codec or design can be used for the video encoder, including, for example, the designs disclosed in International Patent Applications PCT / US06 / 28365 and PCT / US06 / 62569. If spatial scalability is used, DOWNSAMPLER is optionally used on the input to reduce the input resolution (eg, from CIF to QCIF).

The ENC REF CONTROL block 300 is used to create a “threaded” coding structure (see, eg, International Patent Application No. PCT / US06 / 28365). A standard block-based motion compensation codec has a regular structure of I-frames, P-frames, and B-frames. For example, in a picture sequence like IBBPBBP (in display order), the “P” frame is predicted from the previous P or I frame in the sequence, but the B picture is next to the previous P or I frame. Predicted using both P and I frames. The number of B pictures between consecutive I or P pictures can vary, as can the rate at which I pictures appear, but for example, P pictures can be different in time before the most recent P picture. It is impossible to use a P picture as a reference for prediction. The H.264 coding standard advantageously provides the exception that two reference picture lists are maintained by the encoder and decoder, respectively, with appropriate signaling information for reordering and selective use of pictures from within this list. To do. This exception can be exploited to select the picture used as a reference and also the reference used for the particular picture to be coded. In the exemplary base layer video encoder 300 , the FRAME BUFFERS block 310 represents a memory that stores the reference picture list (s). The ENC REF CONTROL block 320 is designed to determine which reference picture should be used for the current picture at the encoder side.

  The operation of the ENC REF CONTROL block 320 is placed in further context with respect to the exemplary layered picture coding “threading” or “prediction chain” structure 400 shown in FIG. 4, where the letter “L” is optional Is used to represent the scalability layer, followed by a number representing the time layer (0 being the least or coarsest). Arrows indicate prediction direction, source, and target. L0 is simply a series of regular P pictures separated by 4 pictures. L1 has the same frame rate, but prediction is only allowed from the previous L0 frame. The L2 frame is predicted from the most recent L0 frame or L1 frame. L0 provides 1/4 (1: 4) of full time resolution, L1 doubles L0 frame rate (1: 2), L2 doubles L0 + L1 frame rate (1 : 1).

  Depending on the requirements of certain embodiments of the invention, additional or fewer layers may be similarly configured to address different bit rate / scalability requirements. A simple example is shown in FIG. 5, where the traditional prediction series of IPPP ... frames has been converted into two layers.

  The codec 300 utilized in embodiments of the present invention is a separate picture “thread” to allow multiple levels of temporal scalability resolution (eg, L0-L2) and other extended resolutions (eg, S0-S2). Can be configured to generate a set (eg, a set of three threads 410-430). A thread or prediction chain is defined as a sequence of pictures that are motion compensated using either pictures from the same thread or pictures from lower level threads. The arrows in FIG. 4 indicate the prediction direction, source, and target of the three threads 410-430. The threads 410-420 have a common source L0 but have different targets and paths (eg, targets L2, L2, and L0, respectively). The use of threads makes it possible to implement temporal scalability. This is because an arbitrary number of top level threads can be removed without affecting the decoding process of the remaining threads.

  Note that within the encoder 300, the ENC REF CONTROL block can only use a P picture as a reference picture. The use of a B picture with both forward and backward prediction increases the coding delay by the time required to capture and encode the reference picture used for the B picture. In traditional interactive communication, the use of B pictures with predictions from future pictures increases coding delay and is therefore avoided. However, B pictures can also be used because they increase the overall compression efficiency. Even using a single B picture within a set of threads (eg, by having L2 coded as a B picture) can improve compression efficiency. For applications that are not delay sensitive, some or all pictures (with possible exceptions for L0) can be B pictures with bidirectional prediction. Especially for the H.264 standard, this standard allows the use of two motion vectors that use past reference pictures in the display order, allowing the use of B pictures without incurring extra delay. Please note that. In this case, such B pictures can be used without increasing coding delay compared to P picture coding. Similarly, the L0 picture can be an I picture that forms a conventional group of pictures (GOP).

Referring to the newly exemplary base layer video encoder 300 , the base layer encoder 300 has been augmented, as described, for example, in the draft H.264 SVC standard and international patent application No. PCT / US06 / 28365. A spatial enhancement layer and / or a quality enhancement layer can be created. FIG. 6 shows the structure of an exemplary encoder 600 that creates a spatial enhancement layer. The structure of the encoder 600 is similar to the base layer codec 300, but has the additional feature that base layer information is also made available from the encoder 600. This information can include motion vector data, macroblock mode data, coded prediction error data, and reconstructed pixel data. The encoder 600 may reuse some or all of this information to make enhancement layer coding decisions. For this purpose, the base layer data needs to be scaled to the target resolution of the enhancement layer (eg, twice if the base layer is QCIF and the enhancement layer is CIF). Spatial scalability usually requires that two coding loops be maintained, but the base layer data used for enhancement layer coding is only a value that can be calculated from the information encoded in the base layer of the current picture. By limiting, it is possible to perform single loop decoding (eg, under the H.264 SVC draft standard). For example, if a base layer macroblock is intercoded, the enhancement layer cannot use the reconstructed pixels of that macroblock as a basis for prediction. However, the enhancement layer can use its motion vector and prediction error value. This is because they can only be obtained by decoding the information contained in the current base layer picture. Single loop decoding is desirable because the complexity of the decoder is greatly reduced.

  The threading structure can be utilized for the enhancement layer frame in the same manner as the base layer frame. FIG. 7 shows an exemplary threading structure 700 of the enhancement layer frame according to the design shown in FIG. In FIG. 7, the enhancement layer block in structure 700 is indicated by the letter “S”. Note that the threading structure of the enhancement layer frame and base layer can be different as described in International Patent Application No. PCT / US06 / 28365.

  Furthermore, a similar enhancement layer codec for quality scalability can be constructed, for example as described in the SVC draft standard and as described in International Patent Application No. PCT / US06 / 28365. In such a quality-scalability codec, rather than creating an enhancement layer based on a higher resolution version of the input, the enhancement layer is created by coding the residual prediction error with the same spatial resolution as the input. . Similar to spatial scalability, all base layer macroblock data can be reused in the enhancement layer for quality scalability in either a single loop coding configuration or a double loop coding configuration.

  For the sake of brevity, the following description is limited to spatial scalability, but it should be understood that the described techniques can also be applied to quality scalability or fidelity scalability.

  Due to the inherent temporal dependence resulting from motion compensated prediction in modern video codecs, packet loss of a particular picture not only affects the quality of that particular picture, but directly or indirectly, Note that it also affects all future pictures for which that particular picture serves as a reference. This is because the reference frame that the decoder can construct for future prediction is not the same as the reference frame used in the encoder. Subsequent differences or drift can have a profound effect on the visual quality of the decoded video signal. However, as described in International Patent Applications Nos. PCT / US06 / 28365 and PCT / US06 / 061815, structure 400 (FIG. 4) has an obvious advantage in terms of robustness in the presence of transmission errors. Have.

  As shown in FIG. 4, the threading structure 400 creates three self-contained dependency chains. Packet loss that occurs in the L2 picture affects only the L2 picture, and the L0 and L1 pictures can still be decoded and displayed. Similarly, packet loss that occurs in the L1 picture affects only the L1 and L2 pictures, and the L0 picture can still be decoded and displayed.

  The same error containment property of the thread extends to S packets. For example, using structure 700 (FIG. 7), the erasure that occurs in an S2 picture affects only that particular picture, but the erasure in an S1 picture also affects subsequent S2 pictures. In either case, the drift ends when the next S0 picture is decoded.

  If the threaded structure causes the base layer picture and some enhancement layer pictures to be transmitted in a way that guarantees their delivery, the remaining layers will not have catastrophic consequences in case of packet loss. Can be transmitted on a best-effort basis. The required guaranteed transmission can be performed using DiffServ, FEC techniques, or other suitable techniques known in the art. In the description herein, guaranteed transmission and best effort transmission are two real or virtual channels that provide such differentiated quality of service (e.g., reliable channels (e.g., (E.g., International Patent Applications PCT / US06 / 028366 and PCT / US06 / 061815 are assumed to occur on High Reliability Channel (HRC)) and Low Reliability Channel (LRC). See)

  For example, consider that layers L0-L2 and S0 are transmitted on HRC, and S1 and S2 are transmitted on LRC. The loss of S1 or S2 packets causes limited drift, but it should still be desirable to be able to hide the loss of information as much as possible. The concealment of the lost S1 picture or S2 picture can use only the information available to the decoder, ie, the past S2 picture and the base layer coded information of the current picture.

  An exemplary concealment technique according to the present invention utilizes the base layer information of the lost enhancement layer frame and uses it in the enhancement layer decoding loop. Available base layer information includes motion vector data (scaled appropriately for target layer resolution), coded prediction error difference (upsampled for enhancement layer resolution, if necessary), and intra Contains data (upsampled for enhancement layer resolution if necessary). The prediction criteria from the previous picture is taken from the enhancement layer resolution picture instead of the corresponding base layer picture when necessary. This data allows the decoder to reconstruct a very good approximation of the missing frame, thus minimizing the actual and perceived distortion for the missing frame. In addition, a good approximation of the missing frame is now available, so that any dependent frame can be decoded.

  FIG. 8 illustrates exemplary steps 810-840 of a concealment decoding process 800 using an example of a two-layer spatial scalability encoded signal with resolution QCIF and CIF and two prediction threads (L0 / S0 and L1 / S1). Indicates. It should be understood that process 800 is applicable to other resolutions and a different number of threads than shown. In this example, it is assumed that the coded data arrival step 810, the L0, S0, and L1 coded data arrives intact at the receiving terminal, but the S1 coded data is lost. Further, it is assumed that all the coded data of the picture preceding the picture corresponding to the time t0 has been received by the receiving terminal. Therefore, the decoder can correctly decode both the QCIF picture and the CIF picture at time t0. The decoder can further use the information contained in L0 and L1 to reconstruct a correctly decoded L1 picture corresponding to time t1.

  FIG. 8 shows a specific example, where a block LB1 of the L1 picture at time t1 uses motion compensated prediction using the motion vector LMV1 and the residual LRES1 that must be added to the motion compensated prediction Thus, encoding is performed in the base layer decoding step 820. The data of LMV1 and LRES1 is included in the L1 data received by the receiving terminal. The decoding process requires a block LB0 from the previous base layer picture (L0 picture), which can be used at the decoder as a result of the normal decoding process. Since S1 data is assumed to be lost in this example, the decoder cannot use the corresponding information to decode the enhancement layer picture.

  The concealment decoding process 800 constitutes an approximation of the enhancement layer block SB1. At concealment data generation step 830, process 800 generates concealment data by obtaining the coded data of the corresponding base layer block LB1, in this example LMV1 and LRES1. Next, the process 800 scales the motion vector to the enhancement layer resolution to construct the enhancement layer motion vector SMV1. In the example of a two-layer video signal under consideration, SMV1 is equal to twice LMV1 because the ratio of the resolution of the scalable signal is 2. Furthermore, the concealment decoding process 800 upsamples the base layer residual signal to the enhancement layer resolution by a factor of two in each dimension, and then optionally uses a filter LPF according to the well-known principles of the sample rate conversion process. Filter the results. A further result of the concealment data generation step 830 is the residual signal SRES1. The next step 840 (enhancement layer decoding with concealment) approximates the block SB1 using the concealed data SMV1 and SRES1. Note that this approximation requires block SB0 from the previous enhancement layer picture, and this previous enhancement layer picture is assumed to be usable at the decoder as a result of the normal decoding process of the enhancement layer. Different encoding modes can operate in the same or similar manner.

A further exemplary application of the inventive concealment technique relates to the example of a high resolution image. In high resolution images (eg, higher than CIF), multiple MTUs (maximum transmission units) are often needed to transmit enhancement layer frames. If the probability of successful transmission of a single MTU size packet is p, the probability of successful transmission of a frame consisting of n MTUs is ^pn . As before, in order to display such a frame, all n packets must be successfully delivered.

  In an application of the inventive concealment technique, S-layer frames are broken down into MTU-sized slices for transmission at the encoder. On the decoder side, any slice that can be used from the received S picture is used. Missing slices are compensated using a concealment method (eg, process 800), thus reducing overall distortion.

  In laboratory experiments, this concealment technique yielded similar or better performance when compared to direct coding at the effective communication rate (total rate minus erasure rate). For this experiment, it was assumed that layers L0-L2 were transmitted reliably on HRC, while layers S1 and S2 were transmitted on LRC. The actual quality loss for Y-PSNR is in the range of 0.2-0.3 dB per 5% packet loss, clearly surpassing the performance of other known concealment techniques such as frame copy or motion compensated frame copy ( For example, see S. Bandyopadhyay, Z. Wu, P. Pandit, and J. Boyce, “Frame Loss Error Concealment for H.264 / AVC”, Doc. JVT-P072, Poznam, Poland, July 2005. They have reported several dB of loss even at an erasure rate of 5% in the evaluation of single-layer AVC coding using an IPP ... PI structure and an I period of 1 second). The laboratory experimental results demonstrate that this technique is effective in providing error resilience with scalable codecs.

  FIG. 9 shows the rate-distortion curve obtained using the standard “Foreman” video test sequence with different QPs. For each QP, rate-distortion values were obtained by discarding different amounts of S1 and S2 frames while applying the inventive error concealment technique described above. As can be seen from Figure 9, the rightmost point of each QP curve corresponds to no disappearance, then (in the direction from right to left), then S2 50% dropout, S2 100% dropout, S2 100 Corresponds to 50% dropout of% and S1, and 100% dropout of S1 and S2. The codec R-D curve obtained by connecting the 0 vanishing points of different QPs is overlaid. From FIG. 9, it can be seen that the various curves, especially for QPs less than 30, are close to the RD curve but are higher in some cases. This difference is expected to be removed by further optimization of the basic codec used.

  Laboratory experimental results show that Y-PSNR is the same as Y-PSNR for the same encoder operating at an effective transmission rate. This implies that the concealment technique can be advantageously used for rate control. The effective transmission rate is defined as the transmission rate minus the erasure rate, that is, the rate calculated based on the packets that actually arrive at the destination. The bit rate corresponding to S1 and S2 frames is typically 30% of the total for a particular coding structure, which allows any bit rate between 70% and 100% for rate control. It implies that it can be achieved by removing a selected number of S1 and S2 frames. A bit rate between 70% and 100% can be achieved by selecting the number of S2 frames or S1 and S2 frames that are discarded within a given time period.

  An even wider range for rate control can be obtained for picture coding structures using LR / SR pictures as described, for example, in International Patent Application No. PCT / US06 / 061815. With such a picture structure, it is possible to include only SR with lower temporal resolution in HRC, rather than transmitting S0 in HRC. This feature allows a wider range for rate control.

Table 1 summarizes the rate percentages for different frame types of normal video sequences (eg, spatial scalability, QCIF-CIF resolution, 3 layer threading, 380 Kbps).

  By combining different frame types, the concealment technique can achieve virtually any desired rate. For example, a rate of about 72 + 1.8 = 73.8% of the total can be achieved when all of the L0-L2 and S0 pictures are included and only one of the 10 S1 pictures is discarded. Alternative techniques known in the art, such as Fine Granularity Scalability (FGS), attempt to achieve similar rate flexibility but with very poor rate-distortion performance and significant computational overhead . The concealment technique of the present invention provides rate scalability associated with FGS, but without the coding efficiency penalty associated with such techniques.

  Intentional removal of S1 and S2 frames from video transmission can be performed either at the encoder or at an available intermediate gateway (eg, SVCS / CSVCS).

  Further, it should be understood that an example application of the concealment technique of the present invention to achieve rate control has been described herein with reference to erasure of S1 frames in a two-layer structure for illustrative purposes only. In practice, the technique is not limited to a particular threading structure, but any spatial scalable that uses pyramidal temporal structures (e.g., structures with multiple quality levels or spatial levels, different temporal structures, etc.) It can be applied to codecs.

  A further use of the inventive concealment technique is to display a video signal at a resolution that is between the two coded resolutions. For example, assume that a video signal is coded with QCIF resolution and CIF resolution using a spatial scalable codec. If the user wants to display its output at 1/2 CIF resolution (HCIF), the conventional decoder has two approaches: 1) decode the QCIF signal and upsample to HCIF, or 2) It should follow either decoding the CIF signal and downsampling to HCIF. In the first case, the HCIF image quality is not good, but the bit rate used is low. In the second case, the quality can be very good, but the bit rate used is almost twice that required by the first approach. These disadvantages of conventional decoders are overcome by the inventive error concealment technique.

  For example, deliberately discarding all S1 and S2 frames can result in significant bandwidth reduction with very little quality degradation by applying the S1 / S2 error concealment techniques described herein. Can bring. By downsampling the resulting decoded CIF signal, a very good rendition of the HCIF signal is obtained. Traditional simultaneous broadcast techniques, where separate single layer streams are transmitted at QCIF and CIF resolution, do not allow such derivation of intermediate resolution signals at a bit rate suitable for use unless the frame rate is also reduced. Please note that. The inventive concealment technique exploits spatial scalable coding to derive an intermediate resolution signal at a bit rate suitable for use.

  In practice, the application of the inventive concealment technique for deriving the intermediate resolution requires the operation of the enhancement layer decoding loop for S0 at full resolution. This decoding involves both the generation of decoded prediction errors and the application of motion compensation at full resolution. To reduce computational requirements, only the decoded prediction error can be generated at full resolution and then downsampled to the target resolution (eg, HCIF). The reduced resolution signal can then be motion compensated using appropriately scaled motion vectors and residual information. This technique can be used for any portion of the “S” layer that is retained for transmission to the receiver. Since there is a drift introduced in the enhancement layer decoding loop, a mechanism to periodically remove the drift may be required. In addition to standard techniques such as I-frame, the periodic use of the spatial scalability INTRA_BL mode can be adopted per enhancement layer macroblock, in which only information from the base layer is used for prediction (e.g. See PCT / US06 / 28365). Since no time information is used, drift for that particular macroblock is removed. If SR pictures are used, drift can also be eliminated by decoding all SR pictures at full resolution. Since SR pictures are far away, there can still be a significant benefit in computational complexity. In some cases, the technique for deriving the intermediate resolution signal can be modified by operating the enhancement layer decoder loop at a reduced resolution. If CPU resources are not the rate limiting factor and switching faster than SR separation is required or desired, the same technique (i.e., operating the decoder loop at full resolution) can be applied to higher time levels as needed (e.g. , S0).

  Another exemplary application of the inventive concealment technique is in video conferencing systems where the spatial level or quality level is achieved via simultaneous broadcasting. In this case, concealment is performed using the base layer information as described above. Enhancement layer drift can be removed via any one of a) threading, b) standard SVC time scalability, c) periodic I-frames, and d) periodic intra macroblocks.

  SVCS using simultaneous broadcasts to provide spatial scalability and sending only higher resolution information for a particular destination for a particular stream (e.g. assuming no or few errors) / CSVCS anticipates such an error concealment mechanism in the decoder and replaces high resolution missing frames with lower resolution frames by relying on temporal scalability to eliminate drift as described above be able to. It should be understood that the concealment process described can be easily adapted to provide effective rate control in such systems.

  If the SVCS, CSVCS, or encoder responsible for discarding higher resolution frames or detecting their loss cannot assume that the decoder receiving such frames has the concealment method described herein. In turn, such entities can create replacement high-resolution frames that achieve similar functionality in one of the following ways:

a) For error resilience in spatial scalability coding, lower resolution including only proper signaling to use upsampled base layer information without any additional residual or motion vector refinement Create composite frames based on frame parsing,
b) For rate control in systems using spatial scalability, the method described in (a) and that several macroblocks (MB) containing important information from the original high resolution frame are maintained. In combination with additional,
c) Create a permutation high-resolution frame containing a composite MB containing upsampled motion vectors and residual information for an error tolerant system that uses simultaneous broadcast for spatial scalability;
d) Added that some MB containing important information from the original high-resolution frame is maintained for rate control in systems that use simultaneous broadcasting for spatial scalability, to (c) The method described.

  In cases a) and b) above, the signaling for using only the upsampled version of the base layer picture is received in-band via the coded video bitstream or from the encoder or SVCS / CSVCS. This can be done either via out-of-band information transmitted to. In the case of in-band signaling, there must be a specific syntax element in the coded video bitstream to instruct the decoder to use only base layer information for some or all of the enhancement layer MBs. I must. Based on the JD7 version of the SVC specification (edited by T. Wiegand, G. Sullivan, J. Reichel, H. Schwarz, M. Wien, which is incorporated herein by reference in its entirety, "Joint Draft 7, Rev. 2: Scalable. `` Video Coding '', Joint Video Team, Doc. JVT-T201, Klagenfurt, July 2006), in the exemplary codec of the present invention described in US Provisional Patent Application No. 60 / 862,510, macroblocks A set of flags can be introduced in the slice header to indicate that a unique prediction mode that utilizes base layer data must be used when not coded. By skipping all enhancement layer macroblocks, the encoder or SVCS / CSVCS actually removes the S1 or S2 frames, but these indicate the default prediction mode, and the fact that all macroblocks are skipped Replace with a very small data packet containing only the few bytes needed to Similarly, to perform rate control, the encoder or SVCS / SVCS can selectively remove some information from the enhancement layer MB. For example, an encoder or SVCS / SVCS selectively maintains motion vector refinement, but can remove residual prediction, or can retain residual prediction but remove motion vector refinement.

  Continuing to refer to the SVC JD7 specification, the MB layer (in the scalable extension) has multiple flags that are used to predict information from the base layer, if the base layer exists. These are base_mode_flag, motion_prediction_flag, and residual_prediction_flag. Similarly, a flag adaptive_prediction_flag used to indicate the presence of base_mode_flag in the MB layer already exists in the slice header. In order to trigger the concealment operation, it is necessary to set base_mode_flag to 1 for all MBs, which can be done using the existing adaptive_prediction_flag. By setting the slice header flag adaptive_prediction_flag to 0 and taking into account that the default value of the inter MB's residue_prediction_flag is 1, we indicate that all MBs in a slice are skipped (mb_skip_run or mb_skip_flag (Using signaling) and thus can instruct the decoder to perform the concealment operations disclosed herein indispensably.

  The potential disadvantage of the concealment technique is that the S0 frame is usually very large (e.g. 45% of the total bandwidth), so the bit rate of the coded stream without S1 and S2 frames is very uneven or `` It is recognized that it can be “burst”. To alleviate this behavior, a modified form (hereinafter “gradual concealment”) splits the S0 packets into smaller packets and / or slices and divides the transmission into time intervals between consecutive S0 pictures. Can be transmitted by spreading across. The entire S0 picture will not be usable for the first S2 picture, but the information received by the first S2 picture (ie, parts of S0 and the whole of L0 and L2) can be used for concealment purposes. In this way, the decoder can also recover the appropriate reference frame in time to display the L1 / S1 picture, which is decoded by both the L1 / S1 picture and the second L2 / S2 This is even more useful when creating new versions. Otherwise, they may be more distant from the L0 frame and thus show more concealment artifacts due to motion.

  Another alternative solution to mitigate the impact of bursty S0 transmission is to smooth variable bit rate (VBR) traffic with additional buffering at the expense of increased end-to-end delay. Note that in multipoint conferencing applications, there is statistical multiplexing inherent on the server side. Therefore, the VBR behavior of traffic originating from the server is naturally smoothed.

  International Patent Application No. PCT / US06 / 061815 describes the problem of error resilience and random access and provides a suitable solution for different application scenarios.

  The progressive concealment technique provides a further solution for performing video switching. The progressive concealment technique described above can also be used for video switching. An exemplary switching application is an application to a single-loop spatial scalable signal coded at QCIF and CIF resolution with a three-layer threading structure, which is illustrated in FIG. As described in International Patent Application No. PCT / US06 / 061815, enhanced error tolerance can be achieved by ensuring reliable transmission of some of the L0 pictures. An L0 picture that is reliably transmitted is referred to as an LR picture. The same threading pattern can be extended to S pictures as shown in FIG. The temporal prediction path of the S picture is the same as the temporal prediction path of the L picture. FIG. 10 shows an example SR period of 1/3 (one of all three S0 pictures is SR) for illustration. In practice, different time periods and different threading patterns can be used in accordance with the principles of the present invention. Furthermore, although different paths in the S picture and L picture can be used, it involves a reduction in the coding efficiency of the S picture. Like LR pictures, SR pictures are assumed to be transmitted reliably. This is accomplished using multiple techniques such as DiffServ coding (LR and SR are in HRC), FEC, or ARQ, as described in International Patent Application No. PCT / US06 / 061815. Can do.

  In an exemplary switching application of the progressive concealment technique, an end user of a terminal receiving a QCIF signal may desire to switch to a CIF signal. In order to be able to start decoding the enhancement layer CIF signal, the terminal must obtain at least one correct CIF reference picture. The technique disclosed in International Patent Application No. PCT / US06 / 061815 involves the use of periodic intra macroblocks, so that all macroblocks of a CIF picture are intracoded within a certain time period. . The disadvantage is that if the percentage of intra macroblocks is kept low (to minimize the impact on total bandwidth), it takes a considerable amount of time to do so. In contrast, the gradual concealment technique switching application takes advantage of the reliable transmission of SR pictures in order to be able to start decoding the enhancement layer CIF signal.

  The SR picture can be transmitted to the receiver and decoded even when the receiver operates at the QCIF level. Since SR pictures are infrequent, the overall impact of SR pictures on bit rate can be minimized. When the user switches to CIF resolution, the decoder can use the most recent SR frame and proceed as if the intermediate S picture up to the first received S picture has been lost. If additional bit rates are available, the sender or server can cache all intermediate S0 pictures to further help the receiver build a reference picture as close as possible to the starting frame of CIF playback. You can also transfer the copied copy. The rate-distortion performance of the S1 / S2 concealment technique ensures that the impact on quality is minimized.

  The techniques of the present invention can also be advantageously used when an end user desires to decode at an intermediate output resolution, eg, HCIF, and then switch to CIF. The HCIF signal can be effectively derived from the L0-L2 and part of the S0-S2 picture (eg, only S0) combined with the concealment of the discarded S frame. In this case, the decoder receives at least part of the S0 picture, but can immediately switch to CIF resolution with a very small PSNR penalty. Furthermore, this penalty is removed as soon as the next S0 / SR picture arrives. Therefore, in this case, there is practically no overhead and almost instantaneous switching can be achieved.

  Note that normal spatial coding structures use a 1: 4 picture area ratio, but some users feel more comfortable with a 1: 2 resolution change. Thus, in practice, for example, in desktop communication applications, a switching transition from HCIF to CIF is much more likely than a switching transition from QCIF to CIF. A common scenario in video conferencing is that the screen real estate is split into a larger picture of the active speaker surrounded by smaller pictures of other participants, and the active speaker image is a larger image. Automatically occupy. In the case of smaller images created using the rate control methods described herein, such switching can be done frequently without any overhead. Switching between participant images can be done frequently in an “active” layout without any overhead. This feature is desirable to address both conference participants who prefer to see such an active layout and other conference participants who prefer a static view. Since the concealment switching method does not require any additional information transmission by the encoder, the selection of the layout by one receiver does not affect the bandwidth received by other receivers.

  The foregoing description refers to the creation of efficient rendering for intermediate resolutions and bit rates that span the range between resolution / bit rates provided directly by the encoder. Other known methods for reducing bit rate (e.g., by introducing drift), such as data partitioning or requantization, are used by SVCS / CSVCS in conjunction with the inventive method described herein, It should be understood that more detailed manipulation of the bitstream can be achieved. For example, assume that 1/3 CIF resolution is desired when only QCIF and CIF are available, and SR, S0-S2 coding structures are used. Removal of S1 and S2 may result in a bit rate that is too high to be effectively used as a 1/3 CIF. Further, removal of S0 may result in a bit rate that is too low and / or may be visually unacceptable due to motion related artifacts. In that case, reducing the amount of bits in the S0 frame using known methods such as data partitioning or requantization will result in SR transmission (in VBR mode or progressively to yield a more optimized result). May be useful in conjunction with any of these). It should be understood that these methods can be applied to the S1 and S2 levels to achieve more fine-tuned rate control.

  The preferred embodiment described herein uses the H.264 SVC draft standard, but as will be apparent to those skilled in the art, this technique can be applied to any spatial / quality level and temporal level that allows multiple levels. It can be applied directly to the coding structure.

  It should also be understood that the scalable codec and concealment techniques described herein can be implemented in accordance with the present invention using any suitable combination of hardware and software. Software (ie, instructions) for implementing and operating the scalable codec described above can be provided on a computer readable medium that includes firmware, memory, storage devices, microcontrollers, microprocessors, integrated circuits. , ASICs, online downloadable media, and other usable media.

Claims (54)

A communication network;
A conference server located in the network and linked to at least one receiving and at least one transmitting endpoint by at least one communication channel each on the communication network;
At least one endpoint for transmitting digital video coded using the scalable video coding format;
At least one receiving endpoint capable of decoding digital video signals coded in a scalable video coding format that supports temporal scalability and at least one of spatial scalability and quality scalability, wherein the scalable video coding format includes spatial scalability. Including a base spatial layer and at least one spatial enhancement layer, including a base quality layer and at least one quality enhancement layer for quality scalability, and including a base time layer and at least one time enhancement layer for temporal scalability, the base time layer And the extended temporal layer is based on a threaded picture prediction structure for at least one of the spatial scalability layer or the quality scalability layer. Are linked to each other,
The conference server is signaled or explicitly used in an output video signal where the use of lower spatial layer data or quality layer data is used in decoding pictures at a higher resolution than the base spatial layer or base quality layer. Higher than the base spatial layer or quality layer of the input video signal received from the transmission endpoint before creating the output video signal to be transferred to the at least one reception endpoint. A video communication system configured to selectively remove or change portions corresponding to layers of the. The scalable video coding format is based on hybrid coding such as H.264 standard, VC-1 standard, or AVS standard, for use within an output video signal transferred to the at least one receiving endpoint. The lower spatial layer data or quality layer data that is signaled or explicitly coded is
Motion vector data,
The difference between the coded prediction errors and
Intra data and
Including at least one of a reference picture indicator and
The lower spatial layer data or quality layer data is further appropriately scaled to a desired target resolution when explicitly coded in an output video signal transmitted to one or more receiving endpoints. The system described in. The conference server further includes:
A transcoding Multipoint Control Unit using cascaded decoding and encoding; and
A switching multipoint communication device by selecting an input to be transmitted as output;
A scalable video communication server using selective multiplexing;
2. The output of the output video signal forwarded to the at least one receiving endpoint as one of a combined scalable video communication server using selective multiplexing and bitstream level combining. The system described.   The at least one transmitting endpoint encoder is configured to encode the transmitted medium as frames in a threaded coding structure having a plurality of different time levels, wherein a subset of frames (“R”) Frame selected specifically for secure transport, including at least the lowest time layer frame in the threaded coding structure, wherein the at least one receiving endpoint decoder has received type R reliably after a packet loss or error Can decode at least a portion of the received medium and then be synchronized with the encoder, and the conference server creates an output video signal that is forwarded to the at least one receiving endpoint Before starting, only the non-R frame base sky of the input video signal received from the transmission endpoint Selectively removing the portion corresponding to the upper layer than the layer or base quality layer system according to claim 1.   The conference server prevents the retained portion corresponding to the layer higher than the base spatial layer or the base quality layer in the input video signal received from the transmission end point from adversely affecting the smoothness of the output bit rate. The system of claim 1, further configured to control a transmission rate of an output video signal transferred to at least one receiving endpoint.   The system of claim 1, wherein the selectively removing or modifying by the conference server is performed according to a desired output bit rate requirement.   The at least one receiving endpoint receives an output picture decoded with a desired spatial resolution contained directly between the lower spatial layer and directly the upper spatial layer provided by the received coded video signal. The system of claim 1, configured to display. The at least one receiving endpoint is further configured to scale all the coded data of the directly upper spatial layer to the desired spatial resolution to thereby directly add the upper spatial layer at the desired spatial resolution. Configured to operate the decoding loop, the resulting drift is
Periodic intra pictures,
Periodic use of intra-base layer mode,
The system of claim 7, wherein the system is removed by using at least one of the full spatial resolution of at least the lowest temporal layer of the upper spatial layer directly. In order to avoid drift when the scalable video coding format is changed or removed, when the coded information of the spatial layer or quality layer above the base corresponds to the base time layer,
Periodic intra pictures,
Periodic intra macroblocks;
The system of claim 1, configured with at least one of threaded picture prediction.   The receiving endpoint is further configured to allow at least the base time to be displayed immediately at a new target layer resolution when the at least one receiving endpoint switches a target layer so that the at least one receiving endpoint can immediately display a decoded picture. The system of claim 1 configured to operate at least one decoding loop for a spatial layer or quality layer above a target spatial layer or quality layer for a layer. A communication network;
A digital video signal coded in a scalable video coding format that supports one transmission endpoint that transmits digital video coded using the scalable video coding format and temporal scalability and at least one of spatial scalability and quality scalability And the scalable video coding format includes a base spatial layer and at least one spatial enhancement layer for spatial scalability, and includes a base quality layer and at least one quality enhancement layer for quality scalability. A base time layer and at least one time extension layer with respect to time scalability, said base time layer and extended time layer Are linked to each other by threaded picture prediction structure for at least one of the spatial scalability layers or quality scalability layer,
The transmitting endpoint is signaled in an output video signal used in decoding a picture at a higher resolution than the base spatial layer or base quality layer, or explicitly used by the lower spatial layer data or quality layer data In order to be coded, before creating the output video signal to be transferred to the at least one receiving endpoint, the coded video signal of the transmitting endpoint is higher than the base space layer or quality layer. A video communication system configured to selectively remove or change a portion corresponding to a layer. The scalable video coding format is based on hybrid coding such as H.264 standard, VC-1 standard, or AVS standard, for use within an output video signal transferred to the at least one receiving endpoint. The lower spatial layer data or quality layer data that is signaled or explicitly coded is
Motion vector data,
The difference between the coded prediction errors and
Intra data and
Including at least one of a reference picture indicator and
12. The lower spatial layer data or quality layer data is further appropriately scaled to a desired target resolution when explicitly coded in an output video signal transmitted to one or more receiving endpoints. The system described in.   The transmitting endpoint is configured to encode the medium to be transmitted as a frame in a threaded coding structure having a plurality of different time levels, and a subset of frames (“R”) is for reliable transport And including at least the lowest time layer frame in the threaded coding structure, wherein the at least one receiving endpoint decoder is received based on a type R positively received frame after a packet loss or error. At least a portion of the transmitted medium, and then synchronized with the encoder at the transmitting endpoint, before the transmitting endpoint creates an output video signal to be transmitted to the at least one receiving endpoint. In addition, a base space layer or base product of only non-R frames among the input video signals received from the transmission end point. Selectively removing the portion corresponding to a higher layer than the layer system of claim 11.   In order to prevent the retained portion corresponding to the layer higher than the base spatial layer or the base quality layer in the input video signal of the transmission end point from adversely affecting the smoothness of the output bit rate. The system of claim 11, further configured to control a transmission rate of an output video signal transmitted to at least one receiving endpoint.   The system of claim 11, wherein the selectively removing or changing by the transmitting endpoint is performed according to a desired output bit rate requirement.   The at least one receiving endpoint is an output picture decoded with a desired spatial resolution included between the directly lower spatial layer and directly the upper spatial layer provided by the received coded video signal The system of claim 11, wherein the system is configured to display The at least one receiving endpoint is further configured to directly scale all coded data of the upper spatial layer to the desired spatial resolution, thereby directly decoding the upper spatial layer decoding loop with the desired spatial resolution. And the resulting drift is
Periodic intra pictures,
Periodic use of intra-base layer mode,
The system of claim 11, wherein the system is removed by using at least one of the full spatial decoding of at least the lowest temporal layer of the upper spatial layer directly. In order to avoid drift when the scalable video coding format is changed or removed, when the coded information of the spatial layer or quality layer above the base corresponds to the base time layer,
Periodic intra pictures,
Periodic intra macroblocks;
The system of claim 11, configured with at least one of threaded picture prediction.   The receiving endpoint is further configured to allow at least the base time to be displayed immediately at a new target layer resolution when the at least one receiving endpoint switches a target layer so that the at least one receiving endpoint can immediately display a decoded picture. 12. The system of claim 11, configured to operate at least one decoding loop for a spatial layer or quality layer above a target spatial layer or quality layer for a layer. A method of decoding a digital video signal, wherein the digital video signal is coded in a scalable video coding format that supports temporal scalability and at least one of spatial scalability and quality scalability,
The scalable video coding format includes a base spatial layer and at least one spatial enhancement layer for spatial scalability, includes a base quality layer and at least one quality enhancement layer for quality scalability, and a base time layer and at least one time for temporal scalability. Including an enhancement layer, wherein the base time layer and the enhancement time layer are linked together by a threaded picture prediction structure for at least one of a spatial scalability layer or a quality scalability layer,
The method
Receiving the digital video signal at a decoder;
When a part of the coded information of the target spatial layer or target quality layer is lost or not available for decoding of a picture in the target spatial layer or target quality layer above the corresponding base layer, the thread Using coded information from a spatial layer or quality layer below the target spatial layer or target quality layer in an expression prediction structure. The decoder is arranged in a receiving end point in a communication network to be linked;
A conference server is linked to the receiving endpoint and the at least one transmitting endpoint by at least one communication channel each on the communication network;
The at least one transmission endpoint transmits the coded digital video coded in the scalable video coding format;
The method corresponds to a layer higher than the base space layer or the quality layer in the input video signal received from the transmission end point before the output of the output video signal to be transferred to the reception end point in the conference server. 21. The method of claim 20, further comprising selectively removing portions to be removed. The conference server linked to the receiving endpoint and at least one transmitting endpoint;
A transcoding Multipoint Control Unit using cascaded decoding and encoding; and
A switching multipoint communication device by selecting an input to be transmitted as output;
A scalable video communication server using selective multiplexing;
23. The method of claim 21, wherein the method is one of a combining scalable video communication server using selective multiplexing and bitstream level combining.   Said at least one transmitting endpoint encoder further comprising encoding the transmitted medium as a frame in a threaded coding structure having a plurality of different time levels, wherein the subset of frames (“R”) Specially selected for secure transport, the decoder can decode at least a part of the received medium based on a type R positively received frame after packet loss or error and then synchronize with the encoder And including at least the lowest time layer frame in the threaded coding structure, and the conference server is received from the transmitting endpoint before creating the output video signal to be transferred to the receiving endpoint. Of the input video signal, a layer higher than the base spatial layer or the base quality layer of only non-R frames Selectively removing corresponding portions A method according to claim 21.   In the conference server, in order to prevent the retained portion of the input video signal received from the transmission endpoint corresponding to the layer higher than the base space layer or the base quality layer from adversely affecting the smoothness of the output bit rate The method of claim 21, further comprising controlling a transmission rate of an output video signal transferred to at least one receiving endpoint.   The method of claim 21, wherein the selectively removing by the conference server is performed according to a desired output bit rate requirement. The transmitting endpoint transmits digital video coded using the scalable video coding format,
The communication network links the transmitting endpoint to the receiving endpoint;
The method includes: generating the output video signal to be transmitted to at least one receiving endpoint to achieve a desired output bit rate at the transmitting endpoint before the base of the input video signals at the transmitting endpoint. The method of claim 21, further comprising selectively not transmitting portions corresponding to layers above the spatial layer or base quality layer.   Further comprising encoding the medium to be transmitted as a frame in a threaded coding structure having a plurality of different time levels at the transmitting endpoint, wherein a subset of frames (“R”) is for reliable transport At least a portion of the medium that is selected in particular and includes at least the lowest time layer frame in the threaded coding structure, the decoder receiving on the basis of a type R positively received frame after packet loss or error Can be decoded and then synchronized with the encoder, and the encoder at the transmission end point is more than the base space layer or the base quality layer of the non-R frame only in the input video signal at the transmission end point. 27. The portion corresponding to the upper layer is not selectively transmitted to at least one receiving endpoint. Law.   In order to prevent a retained portion corresponding to a layer higher than the base space layer or the base quality layer in the input video signal of the transmission end point from adversely affecting the smoothness of the output bit rate at the transmission end point. 27. The method of claim 26, further comprising controlling a transmission rate of an output video signal that is transferred to at least one receiving endpoint.   27. The method of claim 26, wherein the selective transmission decision by the transmission endpoint is performed according to a desired output bit rate requirement.   The decoder further comprises displaying an output picture decoded with a desired spatial resolution included between the directly lower spatial layer and directly the upper spatial layer provided by the coded video signal. The method of claim 20. Further operating the decoding layer of the upper spatial layer directly at the desired spatial resolution by directly scaling all coded data of the upper spatial layer to the desired spatial resolution at the decoder. The resulting drift is
Periodic intra pictures,
Periodic use of intra-base layer mode,
21. The method of claim 20, wherein the method is removed by using at least one of at least one of the directly upper spatial layers and full resolution decoding of the least temporal layer. The scalable video coding format is further adapted to avoid drift when coded information of the target space layer or target quality layer that is lost or unavailable corresponds to the base time layer,
Periodic intra pictures,
Periodic intra macroblocks;
21. The method of claim 20, comprising at least one of threaded picture prediction. The scalable video coding format is based on hybrid coding such as H.264 standard, VC-1 standard, or AVS standard, and some or all of the coded information in the target spatial layer or target quality layer is lost. The coded information from a spatial layer or quality layer below the target spatial layer or target quality layer used by the decoder when it is or is not usable is:
Motion vector data appropriately scaled to the resolution of the target space layer or target quality layer;
The difference in coded prediction error upsampled to the resolution of the target spatial layer or target quality layer;
And at least one of intra data up-sampled to the resolution of the target space layer or the target quality layer,
In the method, the decoder uses the decoded picture of the target space layer or the target quality layer as a reference in a decoding process to construct a decoded output picture instead of a decoded reference picture of a lower layer. Further comprising using,
The method of claim 20.   In the decoder, at least the base time layer to enable the decoder to immediately display the decoded picture at a new target spatial layer or target quality layer resolution when the decoder switches between a target spatial layer or a target quality layer 21. The method of claim 20, further comprising operating at least one decoding loop for a spatial layer or quality layer above the target spatial layer or quality layer. A method of video communication on a communication network comprising: a conference server arranged in the network, each linked to at least one reception and at least one transmission endpoint by at least one communication channel on the communication network The at least one endpoint transmits digital video coded using a scalable video coding format, and the at least one receiving endpoint is temporal scalability as well as at least one of spatial scalability and quality scalability. A digital video signal coded in a scalable video coding format that supports the same, and the scalable video coding format is based on spatial scalability. Includes a spatial layer and at least one spatial enhancement layer, comprises a base quality layer and at least one quality enhancement layer with respect to quality scalability, includes a base time layer and at least one temporal enhancement layer with respect to temporal scalability, the base time layer and extended The temporal layers are linked together by a threaded picture prediction structure for at least one of the spatial scalability layer or the quality scalability layer,
The method
At the conference server, the use of lower spatial layer data or quality layer data is signaled in the output video signal used in decoding a picture at a higher resolution than the base spatial layer or base quality layer, or explicitly Higher than the base spatial layer or quality layer of the input video signal received from the transmission endpoint before creating the output video signal to be transferred to the at least one reception endpoint. Selectively removing or altering a portion corresponding to a layer of the layer. The scalable video coding format is based on hybrid coding such as H.264 standard, VC-1 standard, or AVS standard, for use within an output video signal transferred to the at least one receiving endpoint. The lower spatial layer data or quality layer data that is signaled or explicitly coded is
Motion vector data,
The difference between the coded prediction errors and
Intra data and
Including at least one of a reference picture indicator and
36. The lower spatial layer data or quality layer data is further appropriately scaled to a desired target resolution when explicitly coded in an output video signal transmitted to one or more receiving endpoints. The method described in 1. The conference server further includes:
A transcoding Multipoint Control Unit using cascaded decoding and encoding; and
A switching multipoint communication device by selecting an input to be transmitted as output;
A scalable video communication server using selective multiplexing;
36. The method of claim 35, configured to create an output video signal that is forwarded to the at least one receiving endpoint as one of a combined scalable video communication server using selective multiplexing and bitstream level combining. The method described.   Said at least one transmitting endpoint encoder further comprising encoding the transmitted medium as a frame in a threaded coding structure having a plurality of different time levels, wherein the subset of frames (“R”) Frame selected specifically for secure transport, including at least the lowest time layer frame in the threaded coding structure, wherein the at least one receiving endpoint decoder has received type R reliably after a packet loss or error Can decode at least a portion of the received medium and then be synchronized with the encoder, and the conference server creates an output video signal that is forwarded to the at least one receiving endpoint Before the base point of only non-R frames of the input video signal received from the transmission endpoint. Between layers or portions corresponding to a higher layer than the base quality layer or change selectively removed, The method of claim 35.   In the conference server, in order to prevent the retained portion of the input video signal received from the transmission endpoint corresponding to the layer higher than the base space layer or the base quality layer from adversely affecting the smoothness of the output bit rate 36. The method of claim 35, further comprising controlling a transmission rate of an output video signal transferred to at least one receiving endpoint.   36. The method of claim 35, further comprising performing at the conferencing server the selectively removing or modifying according to a desired output bit rate requirement.   At said at least one receiving end point, an output picture decoded with a desired spatial resolution contained between the directly lower spatial layer and directly the upper spatial layer provided by the received coded video signal 36. The method of claim 35, further comprising displaying. The directly upper spatial layer decoding loop at the desired spatial resolution by scaling all directly coded data of the upper spatial layer to the desired spatial resolution at the at least one receiving endpoint. And the resulting drift is
Periodic intra pictures,
Periodic use of intra-base layer mode,
42. The method of claim 41, wherein the method is removed by using at least one of at least one of the directly upper spatial layers and full resolution decoding of the least temporal layer. In order to avoid drift when the scalable video coding format is changed or removed, when the coded information of the spatial layer or quality layer above the base corresponds to the base time layer,
Periodic intra pictures,
Periodic intra macroblocks;
36. The method of claim 35, configured using at least one of threaded picture prediction.   At least the base to enable the at least one receiving endpoint to immediately display the decoded picture at the new target layer resolution when the at least one receiving endpoint switches the target layer. 36. The method of claim 35, further comprising operating at least one decoding loop for a spatial layer or quality layer above a target spatial layer or quality layer for the temporal layer. A communication network;
A digital video signal coded in a scalable video coding format that supports one transmission endpoint that transmits digital video coded using the scalable video coding format and temporal scalability and at least one of spatial scalability and quality scalability A video communication method including at least one receiving endpoint capable of decoding
The scalable video coding format includes a base spatial layer and at least one spatial enhancement layer for spatial scalability, includes a base quality layer and at least one quality enhancement layer for quality scalability, and a base time layer and at least one time for temporal scalability. Including an enhancement layer, wherein the base time layer and the enhancement time layer are linked together by a threaded picture prediction structure for at least one of a spatial scalability layer or a quality scalability layer,
The transmitting endpoint is signaled in the output video signal used in decoding a picture with a higher resolution than the base spatial layer or base quality layer, or the explicit use of lower spatial layer data or quality layer data, or explicitly Before generating the output video signal to be transferred to the at least one receiving endpoint, so that the coded video signal of the transmitting endpoint is higher than the base space layer or the quality layer. A method of video communication configured to selectively remove or change a portion corresponding to a layer of a video. The scalable video coding format is based on hybrid coding such as H.264 standard, VC-1 standard, or AVS standard, for use within an output video signal transferred to the at least one receiving endpoint. The lower spatial layer data or quality layer data that is signaled or explicitly coded is
Motion vector data,
The difference between the coded prediction errors and
Intra data and
Including at least one of a reference picture indicator and
46. The lower spatial layer data or quality layer data is further appropriately scaled to a desired target resolution when explicitly coded in an output video signal transmitted to one or more receiving endpoints. The method described in 1.   Further comprising encoding the medium to be transmitted as a frame in a threaded coding structure having a plurality of different time levels at the transmitting endpoint, wherein a subset of frames (“R”) is for reliable transport And including at least the lowest time layer frame in the threaded coding structure, wherein the at least one receiving endpoint decoder is received based on a type R positively received frame after a packet loss or error. At least a portion of the transmitted medium, and then synchronized with the encoder at the transmitting endpoint, before the transmitting endpoint creates an output video signal to be transmitted to the at least one receiving endpoint. In addition, the base spatial layer or the base quality of only non-R frames among the input video signals at the transmission end point Or change to selectively remove portions corresponding to the higher layers, the method according to claim 45.   In order to prevent a retained portion corresponding to a layer higher than the base space layer or the base quality layer in the input video signal of the transmission end point from adversely affecting the smoothness of the output bit rate at the transmission end point. 46. The method of claim 45, further comprising controlling a transmission rate of an output video signal transmitted to at least one receiving endpoint.   46. The method of claim 45, further comprising performing the selective removal or modification at the transmitting endpoint according to a desired output bit rate requirement.   Output picture decoded at the at least one receiving end point with a desired spatial resolution included between the directly lower spatial layer and directly the upper spatial layer provided by the received coded video signal 46. The method of claim 45, further comprising displaying. The directly upper spatial layer decoding loop at the desired spatial resolution by scaling all directly coded data of the upper spatial layer to the desired spatial resolution at the at least one receiving endpoint. And the resulting drift is
Periodic intra pictures,
Periodic use of intra-base layer mode,
51. The method of claim 50, wherein the method is removed by using at least one of the full spatial decoding of at least the lowest temporal layer of the upper spatial layer directly. In order to avoid drift when the scalable video coding format is changed or removed, when the coded information of the spatial layer or quality layer above the base corresponds to the base time layer,
Periodic intra pictures,
Periodic intra macroblocks;
46. The method of claim 45, configured using at least one of threaded picture prediction.   At least for the base time layer to allow the at least one receiving end point to immediately display the decoded picture at the receiving end point at a new target layer resolution when the at least one receiving end point switches the target layer. 46. The method of claim 45, further comprising operating at least one decoding loop for a spatial layer or quality layer above the target spatial layer or quality layer.   54. A computer readable medium comprising a set of instructions for performing the steps of at least one of the methods of claims 21-53.

Images