JP2009508454A - Scalable low-latency video conferencing system and method using scalable video coding - Google Patents

Abstract

  A scalable video codec is provided for use in videoconferencing systems and applications hosted on heterogeneous endpoint / recipient and network environments. A scalable video codec provides an encoded representation of a source video signal in multiple temporal, quality, and spatial resolutions.

Description

  The present invention relates to multimedia and telecommunications technology. In particular, the present invention relates to a system and method for conducting a video conference between user endpoints having various access devices or terminals and over non-uniform network links.

(Cross-reference of related applications)
No. 60 / 701,108, filed Jul. 20, 2005, No. 60 / 714,741 filed Sep. 7, 2005, and No. 60 / 723,392 filed Oct. 4, 2005. Insist on the interests of priority. In addition, this application is related to co-pending US patent applications [SVCSystem], [Trunk], and [Jitter]. All of the foregoing priority and related applications are incorporated herein by reference in their entirety.

  Video conferencing systems allow two or more remote participants / endpoints to communicate video and audio to each other in real time using both audio and video. If only two remote participants are involved, direct transmission of communications can be used between the two endpoints via an appropriate electronic network. If more than two participants / endpoints are involved, a multipoint conference unit (MCU) or bridge is commonly used to connect to all of the participants / endpoints. The MCU arbitrates communication between multiple participants / endpoints that can be connected to, for example, a star configuration.

  For video conferencing, participants / endpoints or terminals are equipped with appropriate encoding and decoding devices. The encoder formats the local audio and video output at the sending endpoint into a suitable encoding format for signaling over the electronic communication network. In contrast, the decoder processes the received signal with encoded audio and video information into a decoding format suitable for playing the audio at the receiving endpoint and displaying the image.

  Traditionally, the terminal user's own image is also displayed on his screen to provide feedback (eg, to ensure proper placement of the person in the video window).

  In an actual videoconferencing system implementation over a communications network, the quality of an interactive videoconference between remote participants is determined by the end-to-end signal delay. End-to-end delays of over 200 ms prevent realistic live or natural interactions between conference participants. Such long end-to-end delays cause video conference participants to actively participate or respond so that the video and audio data in transit from other participants can reach that endpoint. Will be suppressed unnaturally.

  The end-to-end signal delay is the acquisition delay (e.g., the time taken to fill the buffer in the A / D converter), encoding delay, transmission delay (the endpoint network interface controller sends the packet filled with data) Time it takes to submit), and transport delay (the time it takes for a packet to travel through the communication network from endpoint to endpoint). Furthermore, the signal processing time through the arbitrating MCU contributes to the total end-to-end delay in a given system.

  The main task of the MCU is to mix incoming audio signals so that a single audio stream is transmitted to all participants, and video frames sent by individual participants / endpoints or Mixing the pictures into a common composite video frame stream containing each participant's picture. The terms frame and picture are used interchangeably herein, and further refer to encoding of frames interlaced as individual fields or combined frames (field-based or frame-based picture encoding) It should be noted that it can be incorporated as obvious to those skilled in the art. MCUs deployed in traditional communication network systems have a single common resolution for all of the individual pictures that are mixed into a common composite video frame that is distributed to all participants in the video conference session. It only provides (eg CIF or QCIF resolution). Therefore, conventional communication network systems are not easy to provide customized video conferencing functions that allow participants to view other participants at different resolutions. Such a desirable feature allows participants to view, for example, other specific participants (e.g., speaking participants) at CIF resolution and other non-speaking participants at QCIF resolution. it can. The MCU may be configured to provide this desired functionality by repeating the video mixing operation as many times as the number of participants in the video conference. However, in such a configuration, the MCU operation causes significant end-to-end delay. In addition, the MCU decodes and mixes multiple audio streams, re-encodes them, and also decodes multiple video streams (with appropriate scaling as needed) into a single frame. It is necessary to have sufficient digital signal processing capability to combine them and re-encode the multiple video streams back into a single stream. Video conferencing solutions (such as a commercial system by Polycom Inc., Willow Road 4750, Polycom Inc., Pleasanton, Calif. 94588, and Tandberg, Inc., Park Avenue 200, New York 10166, USA) are acceptable. Dedicated hardware components need to be used to provide quality and performance levels.

  The performance level of the video conferencing solution, and the quality delivered thereby, is also strongly correlated with its underlying communication network through which the video conference operates. Video conferencing solutions that use ITU H.261, H.263, and H.264 video codecs require a lossless or lossless robust communication channel to deliver acceptable quality To do. The required communication channel transmission rate or bit rate can range from 64 Kbps up to several Mbps. Early video conferencing solutions used on dedicated ISDN lines, and newer systems, often use high speed Internet connections (eg, fractional T1, T2, T3, etc.) for high speed transmission. In addition, some video conferencing solutions utilize Internet Protocol ("IP") communications, which are implemented in a dedicated network environment to ensure bandwidth availability. In either case, traditional video conferencing solutions will have significant costs associated with implementing and maintaining the dedicated high speed networking infrastructure required for quality transmission.

  The cost of implementing and maintaining a dedicated video conferencing network is avoided by modern “desktop video conferencing” systems that utilize high bandwidth corporate data network connections (eg, 100 Mbit, Ethernet). These desktop video conferencing solutions include a USB-based digital video camera and a common personal computer (PC) with the appropriate software applications to perform encoding / decoding and network transmission, with participants / endpoint terminals Used as.

  Recent advances in multimedia and telecommunications technologies include integrating video communications and conferencing functions using Internet Protocol (“IP”) communication systems such as IP PBX, instant messaging, web conferencing. In order to effectively integrate videoconferencing into such systems, both point-to-point and multipoint communications must be supported. However, the available network bandwidth in IP communication systems varies widely (e.g., depending on time of day and overall network load), and these systems can be used for high bandwidth communications required for video communications. May be unreliable. In addition, video conferencing solutions implemented on IP communication systems must adapt to both the heterogeneity of network channels associated with the Internet system and the diversity of endpoint devices. For example, participants may use a variety of different personal computing devices to access video conferencing services over IP channels with very different bandwidths (e.g., DSL vs. Ethernet). possible.

  Communication networks in which videophone solutions are implemented can be categorized as providing two basic communication channel architectures. In one basic architecture, a guaranteed quality of service (QoS) channel is provided through a dedicated direct or switched connection between two points (eg, ISDN connection, T1 line, etc.). Conversely, in the second basic architecture, the communication channel does not guarantee QoS, but `` best effort '', such as that used in Internet Protocol (IP) -based networks (e.g. Ethernet LAN) It's just a packet delivery channel.

  Implementing a video conferencing solution on an IP-based network may be desirable, at least due to low cost, high total bandwidth, and wide availability of access to the Internet. As previously mentioned, IP-based networks typically operate on a best effort basis, i.e., it is not guaranteed that a packet will reach its destination or arrive in the order it was sent. However, techniques have been developed to provide different levels of quality of service (QoS) over channels that are estimated to be best effort. The technique can include protocols such as DiffServe to specify and control network traffic by class in order for some traffic types to obtain rank and RSVP. These protocols can guarantee some bandwidth and / or guarantee a delay for the portion of available bandwidth. Techniques such as Forward Error Correction (FEC) and Automatic Repeat Request (ARQ) mechanisms can also be used to improve recovery mechanisms for lost packet transmissions and to mediate the effects of packet loss. it can.

  To implement a video conferencing solution on an IP-based network, it is necessary to consider the video codec used. The main profiles (Main Profile) of the specified standards MPEG-1 and MPEG-2 for standard video codecs, such as the standard H.261 and H.263 codecs specified for video conferencing, and Video CD and DVD, respectively. The codec is designed to provide a single bit stream (“single layer”) at a constant bit rate. Some of these codecs can be deployed without rate control to provide various bit rate streams (eg, MPEG-2 used in DVD). However, in practice, even without rate control, depending on the specific infrastructure, a target operating bit rate is established. These video codecs are based on the assumption that the network can provide a constant bit rate and provide a practically error-free channel between the sender and the receiver. In particular, H-series standard codecs designed for human-to-human communication applications provide some additional functionality to increase robustness in the presence of channel errors, but a very small percentage of packets It is only resistant to loss (usually only up to 2-3%).

  Furthermore, standard video codecs are based on “single layer” coding techniques, which essentially cannot take advantage of the differentiated QoS features provided by current communication networks. A further limitation of single layer coding techniques for video communication is that the receiving endpoint or MCU receives and decodes and downscales the highest resolution signal, even if a low spatial resolution display is needed or desirable in the application. It is necessary to carry out. This consumes bandwidth and computational resources.

  In contrast to the single-layer video codec described above, a “scalable” video codec based on a “multi-layer” coding technique produces two or more bitstreams for a given source video signal. The That is, a base layer and one or more enhancement layers. The base layer can be a basic representation of the source signal at a minimum quality level. The minimum quality representation can be reduced in the SNR (quality), spatial or temporal resolution aspects of a given source video signal, or in a combination of these aspects. One or a plurality of enhancement layers corresponds to information for enhancing the SNR (quality), space, or temporal resolution aspect of the base layer. Scalable video codecs have been developed considering heterogeneous network environments and / or heterogeneous recipients. The base layer can transmit using a reliable channel, i.e. a channel with guaranteed quality of service (QoS). The enhancement layer may be transmitted with reduced QoS or no QoS. The effect is that the receiver is guaranteed to receive a signal (base layer signal) having at least a minimum quality level. Similarly, disparate recipients that may have different screen sizes can send a small picture size signal to a portable device, for example, and a full size picture to a system with a large display. Can be sent.

  Standards such as MPEG-2 specify several techniques for implementing scalable coding. However, the actual use of “scalable” video codecs is hampered by the increased cost and complexity associated with scalable coding and by the lack of extensive availability of high-bandwidth IP-based communication channels suitable for video. Has been.

  Consideration is currently being given to the development of improved scalable codec solutions for video conferencing and other applications. A desirable scalable codec solution will provide improved bandwidth, temporal resolution, spatial quality, spatial resolution, and computational power scalability. In particular, there is interest in developing scalable video codecs that are consistent with a simplified MCU architecture for multipurpose video conferencing applications. A desirable scalable codec solution enables a zero delay MCU architecture that allows MCUs to be cascaded in an electronic network with minimal or no end-to-end delay penalty.

  The present invention provides scalable video coding (SVC) systems and methods (collectively “solutions”) for point-to-point and multipoint conferencing applications. The SVC solution provides an encoded “layered” representation of the source video signal in multiple temporal, quality, and spatial resolutions. These representations are represented by separate layer / bitstream components created by the endpoint / terminal encoder.

  SVC solutions are designed to accommodate the diversity in endpoint / receiver devices and the diversity in heterogeneous network characteristics including, for example, network best effort, such as those based on the Internet protocol. Due to the scalable aspects of the video coding techniques used, conferencing applications can adapt to different network conditions, and different terminal user requirements (e.g., users can connect other users with high or low spatial resolution). Can choose to see).

  Scalable video codec design enables error-tolerant video transmission in point-to-point and multipoint scenarios, and stream error resilience without decoding or re-encoding the video stream in transit Without reducing anything, the conference bridge will be able to provide residency, rate matching, error localization, random entry, and personal layout conferencing capabilities.

  An endpoint terminal designed for video communication with other endpoints is a video encoder / decoder that can encode a video signal into one or more layers of a multi-layer scalable video format for transmission Including a bowl. The video encoder / decoder can therefore decode simultaneously or sequentially video signal layers received in as many video streams as there are video conference participants. A terminal may be implemented with hardware, software, or a combination thereof on a general purpose PC or other network access device. A scalable video codec embedded in a terminal may be consistent with or based on an encoding method and technique based on or based on an industry standard encoding method such as H.264.

  In the H.264-based SVC solution, the scalable video codec creates a base layer based on the standard H.264 AVC encoding. A scalable video codec can further encode the difference between the original signal and the one encoded at the previous layer with the appropriate offset, again using H.264 AVC, and Create an SNR enhancement layer. In this scalable video codec version, the DC value of the Discrete Cosine Transform (DCT) coefficient is not encoded in the enhancement layer, and no conventional deblocking filter is used.

  In an SVC solution designed to use SNR scalability as a means to implement spatial scalability, different quantization parameters (QP) are selected for the base layer and enhancement layer. Base layers encoded with higher QP are optionally low-pass filtered at the receiving endpoint / terminal and down-sampled for display.

  In other SVC solutions, the scalable video codec is a spatially scalable encoder where the reconstructed base layer H.264 low resolution signal is upsampled by the encoder and subtracted from the original signal. Designed. The difference is offset by a set value and then supplied to a standard encoder operating at high resolution. In other versions, the upsampled H.264 low resolution signal is used as a possible additional reference frame in the motion estimation process of a standards-based high resolution encoder.

  The SVC solution can include adjusting or changing the threading mode or spatial scalability mode to dynamically respond to network conditions and participant display preferences.

  Further features of the invention, its nature and various advantages will be more apparent from the following detailed description of the preferred embodiments and the accompanying drawings.

  Unless otherwise stated, the same numbers and characters throughout the figures are used to indicate similar functions, elements, components, or portions of the illustrative embodiments. Further, the present invention will now be described in detail with reference to the figures, which will be described in connection with exemplary embodiments.

  The present invention provides systems and techniques for scalable video coding (SVC) of video data signals for multipoint and point-to-point videoconferencing applications. SVC systems and techniques (generally "solutions") adapt the delivered video data, depending on the different user participants / endpoints in the video conference, network transmission capabilities, environment, or other requirements, or Designed to allow customization. Inventive SVC solutions provide video data compressed into a multi-layer format that can be easily switched between conference participants layer by layer using a convenient zero or low algorithm delay switching mechanism. An exemplary zero or low algorithm delay switching mechanism, namely a scalable video coding server (SVCS), is described in co-pending US patent application [SVCS].

  1A and 1B show an exemplary video conferencing system 100 configuration based on the inventive SVC solution. Video conferencing system 100 may be implemented in a heterogeneous electronic or computer network environment for multipoint and point-to-point client conferencing applications. System 100 uses one or more networked servers (e.g., SVCS or MCU 110) to coordinate delivery of customized data to conference participants or clients 120, 130, and 140 . As described in co-pending US Patent Application No., the MCU 110 can coordinate the delivery of the video stream 150 generated by the endpoint 140 for transmission to other conference participants. In system 100, a video stream is first appropriately encoded or downscaled using inventive SVC techniques into multiple data components or layers. Multiple data layers can have different characteristics or functions (eg, spatial resolution, frame rate, picture quality, signal-to-noise ratio quality (SNR), etc.). Different characteristics or functions of the data layer are appropriate, for example, considering changing individual user requirements and infrastructure specifications (e.g. CPU capacity, display dimensions, user preferences, and bandwidth) in an electronic network environment. You can choose. MCU 110 also selects the appropriate amount of information (i.e., SVC layer) for each particular participant / recipient of the conference from the received data stream (e.g., SVC video stream 150) and each participant Appropriately configured to transfer only a selected or requested amount of information / layers to subscriber / recipients 120-130. The MCU 110 may be configured to make an appropriate selection in response to the request of the receiving endpoint (eg, picture quality required by individual conference participants) and considering network conditions and policies.

  This customized data selection and transfer scheme takes advantage of the internal structure of the SVC video stream, thereby clearly dividing the video stream into multiple layers with different resolutions, frame rates, and / or bandwidths, etc. It becomes possible to do. FIG. 1B, which is a reproduction from the referenced patent application [SVCS], shows an exemplary internal structure of the SVC video stream 150 showing the media input of the endpoint 140 to the conference. An exemplary internal structure of the SVC video stream 150 includes a “base” layer 150b and one or more separate “enhancement” layers 150a.

  FIG. 2 shows an exemplary participant / endpoint terminal 140 designed for use in an SVC-based videoconferencing system (eg, system 100). Terminal 140 couples to human interface input / output devices (e.g., camera 210A, microphone 210B, video display 250C, speaker 250D), and input and output signal multiplexer and demultiplexer units (e.g., packet MUX 220A and packet DMUX 220B). Network interface controller card (NIC) 230. The NIC 230 may be a standard hardware component such as an Ethernet LAN adapter or any other suitable network interface device.

  Camera 210A and microphone 210B are designed to capture the participant's video and audio signals, respectively, for transmission to other conference participants. In contrast, video display 250C and speaker 250D are designed to display and play video and audio signals received from other participants, respectively. Video display 250C may also optionally be configured to display the participant / terminal 140's own video. The outputs of camera 210A and microphone 210B are coupled to video and audio encoders 210G and 210H via analog-to-digital converters 210c and 210D, respectively. Video and audio encoders 210G and 210H are designed to compress the input video and audio digital signals to reduce the bandwidth required to transmit the signals over the electronic communication network. The input video signal can be a raw signal or a pre-recorded and stored video signal.

  Video encoder 210G has multiple outputs that are directly connected to packet MUX 220A. The audio encoder 210H output is also directly connected to the packet MUX 220A. The compressed, layered video and audio digital signals from encoders 210G and 210H are multiplexed by packet MUX 220A for transmission over the communication network via NIC 230. Conversely, the compressed video and audio digital signals received by NIC 230 over the communication network are forwarded to packet DMUX 220B for demultiplexing and displayed and played back via video display 250C and speaker 250D. Therefore, the terminal 140 further processes.

  The captured audio signal can be encoded by the audio encoder 210H using any suitable encoding technique including known techniques, eg, G.711 and MPEG-1. In the implementation of the video conference system 100 and the terminal 140, G.711 encoding is preferable for audio encoding. The captured video signal is encoded in an encoding format layered by video encoder 210G using the SVC technique described herein. Packet MUX 220A may be configured to multiplex input video and audio signals using, for example, the RTP protocol or other suitable protocol. Packet MUX 220A may also be configured to perform any required QoS related protocol processing.

  In system 100, each stream of data from terminal 140 is transmitted over its electronic communication network on its own virtual channel (or port number in IP terminology). In an exemplary network configuration, QoS may be provided via DiffServ (Differentiated Services) for a particular virtual channel or by any other similar technique that allows QoS. The necessary QoS setup is performed before using the system described herein. DiffServ (or a technique that enables similar QoS used) creates two different categories of channels that are implemented at or through a network router (not shown). For purposes of explanation, two different categories of channels are referred to herein as “high reliability” (HRC) and “low reliability” (LRC) channels, respectively. If there is no explicit way to establish an HRC, or if the HRC itself is not reliable enough, the endpoint (or MCU 110 for the endpoint) will (i) prevent the HRC from The information is transmitted repeatedly (the actual number of repeated transmissions can depend on the error condition of the channel), or (ii) the receiving endpoint, for example, when an information loss in the transmission is detected and reported immediately Or if there is a request for SVCS, the information can be cached and retransmitted. These methods of establishing an HRC can be applied individually or in any combination in a client-to-MCU, MCU-to-client, or MCU-to-MCU connection, depending on available channel types and conditions .

  For use in a multi-participant video conferencing system, terminal 140 may be a video and audio decoder (e.g., designed to decode signals received from conference participants that are viewed or heard at terminal 140. , One or more pairs of decoders 230A and 230B). The pair of decoders 230A and 230B can be designed to process the signal of each participant individually or to process several participant signals sequentially. The configuration or combination of video and audio decoders 230A and 230B included in terminal 140 is all of the participant signals received at terminal 140, taking into account the design features of the encoders that process in parallel and / or sequentially. Can be appropriately selected to handle. In addition, packet DMUX 220B is suitable for video and audio encoders 230A and 230B to receive packetized signals from conference participants via NIC 230 and to process in parallel and / or sequentially. It can be configured to transfer signals to a pair.

  Further, in terminal 140, the output of audio decoder 230B is connected to audio mixer 240 and digital-to-analog converter (DA / C) 250B, which drives speaker 250D to reproduce the received audio signal. Audio mixer 240 is designed to combine individual audio signals into a single signal for playback. Similarly, the video decoder 230A output is synthesized by the compositor 260 in the frame buffer 250A. The synthesized or composite video picture from frame buffer 250A is displayed on monitor 250C.

  The compositor 260 can be suitably designed to place each decoded video picture at a corresponding designated location in a composite frame or displayed picture. For example, the display on the monitor 250C can be divided into four small areas. The compositor 260 can obtain pixel data from each video decoder 230A of the terminal 140 and place the pixel data at the appropriate frame buffer 250A position (eg, filling the lower right picture). In order to avoid double buffering (for example, once at the output of the decoder 230B and once at the frame buffer 250A), the compositor 260 is, for example, an address generator that drives the arrangement of the output pixels of the decoder 230B. Can be configured. Alternative techniques for optimizing the placement of individual video decoder 230A outputs on display 210C may also be used for similar effects.

  It should be understood that the various terminal 140 components shown in FIG. 2 can be implemented with any suitable combination of hardware and / or software components appropriately interfaced with each other. The component can be a separate stand-alone unit, or it can be integrated with a personal computer or other device having network access capabilities.

  Referring to video encoders used at terminal 140 for scalable video encoding, FIGS. 3-9 illustrate various scalable video encoders or codecs 300-900 that may be deployed in terminal 140, respectively. .

  FIG. 3 illustrates an exemplary encoder architecture for compressing an input video signal into a layered coding format (e.g., in SVC terminology, layers L0, L1, and L2, where L0 has the lowest frame rate) 300 is shown. Encoder architecture 300 represents a motion compensated, block-based transform codec, eg, based on a standard H.264 / MPEG-4 AVC design or other suitable codec design. The encoder architecture 300 is in addition to the traditional “text-book” various video encoding process blocks 330 for motion estimation (ME), motion compensation (MC), and other encoding functions. Frame buffer block 310, ENC REF control block 320, and deblocking filter block 360. The motion compensated, block-based codec used in system 100 / terminal 140 may be a single layer temporal prediction codec, which has a regular structure of I, P, and B pictures. The sequence of pictures (display order) can be, for example, “IBBPBBP”. In a picture sequence, a “P” picture is predicted from the previous P or I picture, while a B picture is predicted using both the previous and next P or I pictures. The number of B pictures between consecutive I or P pictures can change because the appearance rate of I pictures can change, but for example, other P pictures that are earlier in time than the recent ones. It is not possible to use a P picture as a reference for prediction. Standard H.264 encoding is advantageous because it provides an exception in which two reference picture lists are maintained at the encoder and decoder, respectively. This exception is exploited by the present invention to select which picture to use as a reference and which reference to use for a particular picture to be encoded. In FIG. 3, the frame buffer block 310 shows a memory for storing the reference picture list. The ENC REF control block 310 is designed on the encoder side to determine which reference picture to use for the current picture.

  The operation of the ENC REF control block 310 will be described with further reference to the exemplary layered picture coding “threading” or “prediction chain” structure shown in FIG. (FIGS. 8-9 show alternative threading structures). The codec 300 used in the implementation of the present invention is a set of separate to allow multiple levels of temporal scalability resolution (e.g., L0-L2), and other enhancement resolutions (e.g., S0-S2). Picture “threads” (eg, a set of three threads 410-430) may be generated. A thread or prediction chain is defined as a sequence of motion compensated with pictures from the same thread or pictures from lower level threads. The arrows in FIG. 4 indicate the direction, source, and prediction target for the three threads 410-430. Threads 410-430 have a common source L0 but have different targets and paths (eg, targets L2, L2, L0, respectively). By using threads, any number of top-level threads can be deleted without affecting the decoding process of the remaining threads, thus allowing temporal scalability to be implemented.

  Note that in encoder 300, according to H.264, the ENC REF control block can only use P pictures as reference pictures. However, B pictures can be used, and the overall compression efficiency is increased. The compression efficiency can also be improved using a single B picture (eg, by encoding L2 as a B picture) in a set of threads. In conventional interactive communication, the use of B pictures with predictions from future pictures increases coding delay and is avoided. However, the present invention allows the design of MCUs with practically zero processing delay. (See co-filed US Patent Application No. SVCS). If such an MCU is used, it is possible to operate with a lower end-to-end delay than the conventional system of the current technology even if a B picture is used.

  In operation, the output L0 of the encoder 300 is simply a set of P pictures with four picture intervals. The output L1 has the same frame rate as L0, but only prediction based on the previous L0 picture is possible. The output L2 picture is predicted from the most recent L0 or L1 picture. The output L0 provides a quarter of the highest temporal resolution (1: 4), L1 doubles the frame rate of L0 (1: 2), and L2 doubles the frame rate of L0 + L1 ( 1: 1). A smaller number (e.g., less than 3 from L0 to L2) or an additional number of layers, as well, by encoder 300 to accommodate different bandwidth / scalability requirements, or different specifications of implementations of the invention Can be configured.

  In accordance with the present invention, for further scalability, each compressed temporal video layer (e.g., L0-L1) includes or associates with one or more additional components related to SNR quality scalability and / or spatial scalability. be able to. FIG. 4 shows one additional enhancement layer (SNR or space). Note that this additional enhancement layer has three different components (S0-S2), each corresponding to three different temporal layers (L0-L2).

  5 and 6 show SNR scalability encoders 500 and 600, respectively. 7-9 illustrate spatial scalability encoders 700-900, respectively. SNR scalability encoders 500 and 600, and spatial scalability encoders 700-900 are based on and can use the same processing blocks as encoder 300 (Figure 3) (e.g., blocks 330, 310, and 320) Will be understood.

  It is understood that for the base layer of the SNR scalable codec, the input to the base layer codec is the highest resolution signal (FIGS. 5-6). In contrast, for the spatial scalability codec base layer, the input to the base layer codec is a downsampled version of the input signal FIGS. 7-9. It should also be noted that the SNR / spatial quality enhancement layers S0-S2 can be encoded according to the upcoming ITU-T H.264 Annex F standard, or other suitable technique.

  FIG. 5 shows the structure of an exemplary SNR enhancement encoder 500, which is similar to the structure of the layered encoder 300 based on H.264 shown in FIG. However, the input to the SNR enhancement layer coder 500 is the difference between the original picture (input in FIG. 3) and the reconstructed coded picture (REF in FIG. 3) recreated by the encoder. Please keep in mind.

  FIG. 5 also illustrates the use of an H.264-based encoder 500 to encode previous layer encoding errors. Such encoding requires a non-negative input. To ensure this, the input to encoder 500 (input-REF) is offset by a positive bias (eg, by offset 340). The positive bias is removed after decoding and before the enhancement layer is added to the base layer. Deblocking filters that are typically used in H.264 codec implementations (eg, deblocking filter 360 in FIG. 3) are not used in encoder 500. Further, to improve the subject code efficiency, DC direct cosine transform (DCT) coefficients in the enhancement layer may optionally be ignored or removed by encoder 500. Experimental results show that deletion of DC values in the SNR enhancement layer (S0-S2) does not adversely affect picture quality, probably due to the already fine quantization done in the base layer. The benefit of this design is that the exact same encoding / decoding hardware or software can be used for both the base layer and the SNR enhancement layer. In a similar manner, by applying H.264 layer coding to the downsampled image and by upsampling the reconstructed image before calculating the rest, spatial scalability (at any ratio) Can be introduced. In addition, standards other than H.264 can be used to compress both layers.

  In the codec of the present invention, in order to decouple SNR and temporal scalability, all motion prediction within and across the temporal layer can be performed using only the base layer stream. This function is illustrated in FIG. 4 by arrow 415 and shows temporal prediction in the base layer block (L) rather than a combination of L and S blocks. Because of this function, all layers can be encoded at CIF resolution. The QCIF resolution is then decoded by decoding the base layer stream with a temporal resolution and by downsampling in each spatial dimension with a dyadic factor (2) using appropriate low-pass filtering. Can be derived. In this way, SNR scalability can also be used to provide spatial scalability. It will be understood that the CIF / QCIF resolution is referenced for illustrative purposes only. Other resolutions (eg, VGA / QVGA) may be supported by the inventive codec without changing any codec design. The codec can also include conventional spatial scalability features in the same or similar manner as described above to include SNR scalability features. Techniques provided by MPEG-2 or H.264 Annex F can be used to include traditional spatial scalability features.

  The ratio of 1: 4 (only L0), 1: 2 (L0 and L1), or 1: 1 (all three layers), depending on the codec architecture designed to separate SNR and temporal scalability as described above Allows for a frame rate of. To double the frame rate, a 100% increase in bit rate is assumed (base is 50% in total), and to add an S layer, an increase of 150% is assumed in terms of scalability (base Is a total of 40%). In a preferred implementation, the total stream can operate at, for example, 500 Kbps and the base layer can operate at 200 Kbps. For the base layer, a rate load of 200/4 = 50 Kbps per frame and (500-200) / 4 = 75 Kbps for each frame can be assumed. The above target bit rate and layer bit rate ratio values are exemplary and are only specified to illustrate the features of the present invention, and the inventive codec may have other target bits. It will be appreciated that the rate or layer bit rate ratio can be easily adapted.

Theoretically, up to 1:10 scalability (total vs. base) is available when the total stream and base layer operate at 500 Kbps and 200 Kbps, respectively. Table I shows examples of the different scalability options available when SNR scalability is used to provide spatial scalability.

  FIG. 6 shows an alternative SNR scalable encoder 600 based on a single coding loop scheme. The structure and operation of SNR scalable encoder 600 is based on the structure and operation of encoder 300 (FIG. 3). Further, in encoder 600, the DCT coefficient quantized by Q0 is inversely quantized and subtracted from the original unquantized coefficient to obtain a residual quantization error (QDIFF 610) of the DCT coefficient. The remaining quantization error information (QDIFF 610) is further quantized by a finer quantizer Q1 (block 620), entropy-coded (VLC / BAC), and output as an SNR enhancement layer S. Note that it is a single coding loop that operates, ie, a loop that operates at the base layer.

  The terminal 140 / video 230 encoder may be configured to provide a spatial scalability enhancement layer in addition to or instead of the SNR quality enhancement layer. In order to encode the spatial scalability enhancement layer, the input to the encoder is the difference between the original high resolution picture and the upsampled and reconstructed encoded picture created by the encoder. The encoder operates on a downsampled version of the input signal. FIG. 7 shows an example encoder 700 for encoding the base layer for spatial scalability. The encoder 700 includes a downsampler 710 at the input of the low resolution base layer encoder 720. For the highest resolution input signal at CIF resolution, the base layer encoder 720 can operate at QCIF, HCIF (half CIF), or any other resolution lower than CIF with appropriate downsampling. In an exemplary mode, the base layer encoder 720 can operate with HCIF. Operation in HCIF mode requires that each dimension of the CIF resolution input signal be downsampled by approximately √1 / 2, thereby reducing the total number of pixels in the picture to approximately one half of the original input. Reduce. Note that in video conferencing applications, if QCIF resolution is desired for display purposes, the decoded base layer needs to be further downsampled from HCIF to QCIF.

  It will be appreciated that an essential difficulty in optimizing a scalable video encoding process for video conferencing applications is that there are more than one resolution in the transmitted video signal. Improving the quality of one resolution may result in a corresponding decrease in the quality of the other resolution. This difficulty is particularly pronounced for spatially scalable coding and in state-of-the-art videoconferencing systems where the encoded resolution and display resolution are the same. Inventive techniques that separate the encoded signal resolution from the intended display resolution are still another means for codec designers to better balance between the quality and bit rate associated with each resolution. Provide tools. According to the present invention, the selection of the encoded resolution for a particular codec should be provided by the total available bandwidth, the desired bandwidth partition across different resolutions, and each additional layer of desired quality difference. It can be obtained by considering the differential and considering the rate-distortion (RD) performance of the codec over different spatial resolutions.

  Under such a scheme, the signal can be encoded in CIF and at 1/3 CIF (1/3 CIF) resolution. Both CIF and HCIF resolution signals can be derived from the CIF encoded signal for display. Furthermore, both 1 / 3CIF and QCIF resolution signals can be similarly derived from the 1 / 3CIF encoded signal for display. CIF and 1/3 CIF resolution signals can be used directly from the decoded signal, while the latter HCIF and QCIF resolution signals can be obtained if the decoded signal is appropriately downsampled. Similar schemes can also be applied for other target resolutions (eg, VGA and 1/3 VGA from which 1/2 VGA and 1/4 VGA can be derived).

  In accordance with the present invention, a scheme for separating the encoded signal resolution from the intended display resolution, along with a scheme for threading the video signal layer (FIGS. 4 and 15, 16), targets with different bit rates. Provides further possibilities for obtaining spatial resolution. For example, in a video signal encoding scheme, spatial scalability can be used to encode a source signal with CIF and 1/3 CIF resolution. SNR and temporal scalability can be applied to video signals as shown in FIG. Further, the SNR encoding used may be a single loop or double loop encoder (e.g., encoder 600 of FIG. 6 or encoder 500 of FIG. 5), or alternatively, data partition (DP) It can be obtained by. Double loop or DP encoding schemes are likely to cause drift if data is lost or removed. However, the use of a layered structure limits its drift error propagation to the next L0 picture as long as the lost or removed data belongs to the L1, L2, S1, or S2 layer . In addition, considering that the perception of error is reduced when the spatial resolution of the displayed video signal is reduced, by deleting or removing data from the L1, L2, S1, and S2 layers, 1/3 By decoding the CIF resolution and displaying it by down-sampling it with the QCIF resolution, a low bandwidth signal can be obtained. Data loss due to downsampling causes errors in the corresponding L1 / S1 and L2 / S2 pictures and propagates the error to future pictures (until the next L0 picture), but the display resolution is reduced This makes it difficult for a human observer to see a decrease in quality. A similar scheme can be applied to the CIF signal for display at HCIF, 2/3 CIF, or any other desired resolution. These schemes advantageously allow for spatial scalability at different resolutions and at different bit rates by using quality scalability.

  FIG. 8 shows the structure of an exemplary spatially scalable enhancement layer encoder 800, which, like encoder 500, is the same H.264 for encoding previous layer encoding errors. An encoder structure is used, but includes an upsampler block 810 for the reference (REF) signal. In such an encoder, a non-negative input is assumed, so the input value is offset before encoding (eg, by offset 340). Values that still remain negative are clipped to zero. The offset is removed after decoding and before adding the enhancement layer to the upsampled base layer.

  In the case of spatial enhancement layer coding, it may be advantageous to use frequency weighting in the DCT coefficient quantizer (Q), as in SNR layer coding (FIG. 6). Specifically, coarser quantization can be used for DC and its surrounding AC coefficients. For example, doubling the quantizer step size for DC coefficients can be very effective.

  FIG. 9 shows an exemplary structure of another spatially scalable video encoder 900. In encoder 900, unlike encoder 800, the upsampled and reconstructed base layer picture (REF) is not subtracted from the input, but as an additional possible reference picture in motion estimation and enhancement. Acts as a mode selection block 330 of the layer encoder. The encoder 900 can therefore either from the previous encoded highest resolution picture (or a future picture for a B picture) or an upsampled version (interlayer) of the same picture encoded at low spatial resolution. Prediction) can be configured to predict the current highest resolution picture. Encoder 800 can be implemented using the same codec for the base and enhancement layers, with only the addition of downsampler 710, upsampler 810, and offset 340 blocks. Note that the encoder motion estimation (ME) block 330 * needs to be modified. It is further noted that enhancement layer encoder 900 operates in the regular pixel domain rather than the difference domain.

  It is also possible to combine prediction from the previous high resolution picture and the upsampled base layer picture by using B picture prediction logic of a standard single layer encoder such as an H.264 encoder. This is a B picture prediction for a high resolution signal so that the first picture is a regular or standard previous high resolution picture and the second picture is an upsampled version of the base layer picture This can be achieved by changing the reference. The encoder then performs predictions as if the second picture is a regular B picture, thereby ensuring that all high-efficiency motion vector predictions and the encoding mode of the encoder (e.g., spatial or temporal) Direct mode) is used. In H.264, “B picture” coding means that two reference pictures can be past or future pictures of the pictures that are coded together, instead of “bidirectional”. On the other hand, in conventional “bidirectional” B-picture coding (eg MPEG-2), one of the two reference pictures is a past picture and the other is a future picture. Note that there are. This embodiment can use a standard encoder design with minimal changes limited to picture reference control logic and upsampling modules.

  In implementations of the invention, SNR and spatial scalability coding modes can be combined in one encoder. In such an implementation, the video threading structure (eg, shown in two dimensions in FIG. 4) can be extended to three dimensions, corresponding to an additional third scalability layer (SNR or space). Implementations in which SNR scalability is added to the highest resolution signal of a spatially scalable codec can be attractive in terms of available quality and bit rate range.

  10-14 each have a base layer decoder 1000, an SNR enhancement layer decoder 1100, a single loop SNR enhancement layer decoder 1200, a spatially scalable enhancement layer decoder 1300, and inter-layer motion prediction FIG. 9 shows an example architecture for a spatially scalable enhancement layer decoder 1400. FIG. These decoders supplement the encoders 300, 500, 600, 700, 800, and 900. Decoders 1000, 1100, 1200, 1300, and 1400 may be included in decoder 230A of terminal 140 as appropriate or required.

  The scalable video encoding / decoding configuration of terminal 140 presents several options for transmitting the resulting layer via the HRC and LRC of system 100. For example, the (L0 and S0) layer or the (L0, S0 and L1) layer can be transmitted via the HRC. After careful consideration of network conditions and the bandwidth of reliable and unreliable channels, alternative combinations can also be used as desired. For example, depending on network conditions, it may be desirable to encode the S0 intra mode, but it may be desirable not to transmit S0 over a protected HRC. In such a case, the frequency of intra-mode coding without prediction may depend on network conditions or may be determined according to the loss reported by the receiving endpoint. The S0 prediction chain can be refreshed in this way (ie, if there is an error at the S0 level, any drift is removed).

  FIGS. 15 and 16 illustrate alternative threading or prediction chain architectures 1500 and 1600, which can be used in video communication or conferencing applications according to the present invention. Implementations of threading structures or prediction chains 1500 and 1600 do not require any substantial changes to the codec design described above with reference to FIGS.

  In architecture 1580, an exemplary combination of layers (S0, L0, and L1) is transmitted over the reliable channel 170. Note that L1 is part of the L0 prediction chain 430 and not for S1, as shown. Architecture 1600 shows a further example of a threading configuration that can also achieve non-dyadic frame rate resolution.

  The codec design of the system 100 and terminal 140 described above is flexible and can be easily extended to incorporate alternative SVC schemes. For example, S-layer coding can be achieved by the forthcoming ITU-T H.264 SVC FGS specification. When FGS is used, S layer encoding can use any part of the “S” packet due to the characteristics built into the generated bitstream. It may be possible to use portions of the FGS component to create reference pictures for higher layers. Loss of FGS component information in transmission over the communication network can cause drift in the decoder. However, the threading architecture used in the present invention is advantageous because it minimizes the effects of such losses. Error propagation can be limited to a small number of frames so that the observer is not aware. The amount of FGS included to create the reference picture can change dynamically.

A proposed feature of the H.264 SVC FGS specification is a leak prediction technique in the FGS layer. Y. Bao et al. , Joint Video Team (JVT) of ISO / IEC MPEG & ITU-T VCEG, 15 th meeting, Busan, Korea, see the April 18 to 22, 2005. The leak prediction technique consists of using a normalized weighted average of the previous FGS enhancement layer picture and the current base layer picture. The weighted average is controlled by the alpha of the weighting parameter, if alpha is 1, only the current base layer picture is used, while if it is 0, only the previous FGS enhancement layer picture is used . The alpha of 0 is the same as the use of motion estimation (ME 330 in FIG. 5) for the SNR enhancement layer of the present invention, which is a limited case of using only zero motion vectors. The leak prediction technique can be used with the regular ME described in this invention. In addition, the alpha value can be periodically switched to 0 to interrupt the FGS layer prediction loop and eliminate error drift.

  FIG. 17 shows an exemplary MCU / SVCS 110 switch structure used in the video conference system 100 (FIG. 1). MCU / SVCS 110 determines which packets are sent from each possible source (for example, endpoints 120-140) to which destination via which channel (high reliability vs. low reliability) And switch the signal accordingly. The design and switching function of the MCU / SVCS 110 is described in co-pending US patent application [SVCS], which is incorporated herein by reference. For simplicity, only the limited details of the switching structure and switching function of the MCU / SVCS 110 will be further described here.

  FIG. 18 illustrates the operation of an exemplary embodiment of the MCU / SVCS switch 110. The MCU / SVCS switch 110 maintains two data structures in its memory. That is, SVCS switch layer configuration matrix 110A and SVCS network configuration matrix 110B, examples of which are shown in FIGS. 19 and 20, respectively. SVCS switch layer configuration matrix 110A (FIG. 19) provides information on how a particular data packet should be processed for each layer and for each pair of source and destination endpoints 120-140. . For example, an element value of zero in matrix 110A indicates that the packet should not be transmitted, a negative matrix element value indicates that the entire packet should be transmitted, and a positive matrix element value indicates that the packet data Indicates that only the specified percentage of should be sent. A specified percentage transmission of the data in the packet may be appropriate only if FGS type techniques are used for the scalable encoded signal.

  FIG. 18 also shows an algorithm 1800 in MCU / SVCS 110 for sending data packets using switch layer configuration matrix 110A information. In step 1802, the MCU / SVCS 110 can examine the received packet header (assuming H.264 is used, for example, a NAL header). In step 1804, the MCU / SVCS 110 evaluates the value of the association matrix 110A element for the source, destination, and layer combination to establish the processing instruction and the specified destination of the received packet. For applications that use FGS encoding, a positive matrix element value indicates that the payload size of the packet needs to be reduced. Accordingly, at step 1806, the associated length entry of the packet is changed and the data is not replicated. At step 1808, the associated layer or combination of layers is switched to its designated destination.

  Referring to FIGS. 18 and 20, the SVCS network configuration matrix 110B tracks the port number for each participating endpoint. The MCU / SVCS 110 can use the matrix 110B information to send and receive data to each layer.

  The operation of the MCU / SVCS 110 based on processing the matrices 110A and 110B allows signal switching to occur with zero or minimal internal algorithm delay as opposed to conventional MCU operation. Conventional MCUs need to configure incoming video into new frames for transmission to various participants. This configuration requires complete decoding of the incoming stream and re-encoding of the output stream. The decoding / recoding processing delay in such an MCU is quite large, and the required computing power is the same. By using a scalable bitstream architecture and by providing multiple instances of the decoder 230A in each endpoint terminal 140 receiver, the MCU / SVCS 110 provides the appropriate layer for each recipient destination. It is only necessary to filter incoming packets to select. By requiring no or minimal DSP processing, the MCU / SVCS 110 is advantageously implemented at a very low cost and is superior (in terms of the number of sessions that can be hosted simultaneously on a given device). Can be implemented with an end-to-end delay that can be slightly larger than the delay in a direct end-to-end point connection.

Terminal 140 and MCU / SVCS 110 may be deployed in different network scenarios with different bit rates and stream combinations. Table II shows possible bit rate and stream combinations in various exemplary network scenarios. Note that base bandwidth / total bandwidth> = 50% is the limit of the effectiveness of DiffServe layering, and further, time resolutions below 15 fps are not useful.

  Terminal 140 and similar configurations of the present invention make it possible to utilize scalable coding techniques for point-to-point and multipoint video conferencing systems deployed over channels that can provide different QoS guarantees. Scalable codec selection as described herein, threading model selection, selection of layers to transmit over a reliable or low-reliability channel, and appropriate bit rates (or quantizers) for the various layers The step size) is an important design parameter that can vary in a particular implementation of the invention. Typically, such design choices need only be made once, and the parameters remain constant during the video conference system deployment, or at least during a particular video conference session. However, it should be understood that the SVC configuration of the present invention provides the flexibility to dynamically adjust these parameters within a single video conference session. Participant / endpoint requirements (e.g. which other participants should receive, at what resolution) and network conditions (e.g. loss rate, jitter, bandwidth availability for each participant, high reliability) Dynamic adjustment of the parameters in consideration of the bandwidth that divides between reliable and unreliable channels) may be desirable. Under an appropriate dynamic adjustment scheme, individual participants / endpoints can interactively switch between different threading patterns (e.g., between the threading patterns shown in Figures 4, 8, and 9) to increase layers. One can choose to change the method assigned to the reliable and unreliable channels, choose to delete one or more layers, or change the bit rate of individual layers. Similarly, the MCU / SVCS 110 changes the way in which layers are assigned to reliable and unreliable channels linked to various participants, deletes one or more layers, and FGS / SNR enhancement layer. Can be configured to scale to some participants.

  In an exemplary scenario, a video conference can have three participants, A, B, and C. Participants A and B can have access to a high speed 500 Kbps channel that can guarantee a continuous rate of 200 Kbps. Participant C may have access to a 200 Kbps channel that can guarantee 100 Kbps. Participant A can use an encoding scheme with the following layers: That is, increasing the spatial resolution at one of the three temporal frame rates, the base layer (“Base”), the temporal scalability layer (“Temporal”) providing 7.5fps, 15fps, 30fps video at CIF resolution A possible SNR enhancement layer ("FPS"). Base and Temporal components each require 100 Kbps and FGS requires 300 Kbps, for a total bandwidth of 500 Kbps. Participant A can send all three Base, Temporal, and FPS components to MCU 110. Similarly, Participant B can receive all three components. However, in the scenario, since Participant B is only guaranteed 200 Kbps, the FGS is transmitted over a non-guaranteed 300 Kbps channel segment. Participant C can only receive Base and Temporal components, which are guaranteed at 100 Kbps. If the available (guaranteed or total) bandwidth changes, Participant A's encoder (eg, terminal 140) dynamically changes the target bit rate for any of the components accordingly. can do. For example, if the guaranteed bandwidth exceeds 200 Kbps, more bits can be allocated to the Base and Temporal components. Since the encoding is performed in real time (ie, the video is not pre-encoded), such changes can be performed dynamically with a real-time response.

  If Participants B and C are both linked by limited capacity, for example, a 100 Kbps channel, Participant A can choose to send only the Base component. Similarly, if Participants B and C choose to view the received video only at QCIF resolution, the additional quality enhancement provided by the FGS component by downsampling the received CIF video to QCIF resolution is Participant A can respond by not sending the FGS component because it will be lost.

  Note that in some scenarios it may be appropriate to transmit a single layer video stream (base layer or total video) and avoid the use of scalability layers at all.

  In transmitting scalable video layers over HRC and LRC, whenever information on LRC is lost, only the information transmitted on HRC can be used for video reconstruction and display. it can. In practice, some parts of the displayed video picture will contain data generated by decoding the base layer and the specified enhancement layer, while other parts only decode the base layer. The data generated by doing is included. If the quality levels associated with different base layer and enhancement layer combinations are significantly different, the quality difference between the displayed video picture with lost LRC data and the video picture without it can be significant. The visual impact can be more clearly seen in the temporal dimension, and repeatedly changing the displayed picture from the base layer to “base layer plus enhancement layer” can be perceived as flicker. To mitigate this effect, the quality difference between the base layer picture and the “base layer plus enhancement layer” picture (for example, in terms of PSNR), especially in pictures where flicker is visually more apparent. It may be desirable to ensure that the target part is kept low. The quality difference between the base layer picture and the “base layer plus enhancement layer” picture can be intentionally kept low by using appropriate rate control techniques to enhance the quality of the base layer itself. it can. One such rate control technique may be to encode all or some of the L0 pictures with lower QP values (ie, finer quantization values). For example, any L0 picture can be encoded with a QP reduced to one third. Such finer quantization can increase the quality of the base layer, thus minimizing any flicker effects or equivalent spatial artifacts caused by loss of enhancement layer information. Lower QP values can also be applied to every other L0 picture, or every four L0 pictures, with similar effectiveness in mitigating flicker and similar artifacts. Static objects with proper rate control applied to the base layer, with a unique usage of a combination of SNR and spatial scalability (e.g. using HCIF coding to represent the base layer responsible for QCIF quality) Can be brought closer to the HCIF resolution, thus reducing flicker artifacts that occur when the enhancement layer is lost.

  While what has been considered as preferred embodiments of the invention has been described, other changes, further changes and modifications can be made to the embodiments, and thus, without departing from the spirit of the invention. Those skilled in the art will appreciate that all such changes and modifications are intended to be claimed as falling within the true scope of the present invention.

  It will also be appreciated that the scalable codec described herein in accordance with the present invention can be implemented using any suitable combination of hardware and software. Software (i.e., instructions) for implementing and operating the scalable codec described above can be provided on a computer readable medium, including but not limited to firmware, memory, storage, microcontroller, Microprocessors, integrated circuits, ASICS, online downloadable media, and other available media can be included.

1 is a schematic diagram illustrating an exemplary architecture of a video conference system in accordance with the principles of the present invention. FIG. 1 is a schematic diagram illustrating an exemplary architecture of a video conference system in accordance with the principles of the present invention. FIG. FIG. 2 is a block diagram illustrating an exemplary terminal user terminal in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary architecture of an encoder for the base and time enhancement layers (ie, 0 to 2) in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary layered picture coding structure for base, temporal enhancement, and SNR or spatial enhancement layers in accordance with the principles of the present invention. FIG. 2 is a block diagram illustrating the structure of an exemplary SNR enhancement layer encoder in accordance with the principles of the present invention. FIG. 2 is a block diagram illustrating the structure of an exemplary single loop SNR video encoder in accordance with the principles of the present invention. FIG. 2 is a block diagram illustrating an exemplary structure of a base layer for a spatial scalability video encoder in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary structure of a spatial scalability enhancement layer video encoder in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary structure of a spatial scalability enhancement layer video encoder with inter-layer motion prediction in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary base layer video decoder in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary SNR enhancement layer video decoder in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary SNR enhancement layer, single loop video decoder in accordance with the principles of the present invention. FIG. 2 is a block diagram illustrating an exemplary spatial scalability enhancement layer video decoder in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary video decoder for a spatial scalability enhancement layer with inter-layer motion prediction in accordance with the principles of the present invention. FIG. 4 is a block diagram illustrating an exemplary alternative layered picture coding structure in accordance with the principles of the present invention. 1 is a block diagram illustrating an exemplary threading architecture in accordance with the principles of the present invention. FIG. 1 is a block diagram illustrating an exemplary scalable video coding server (SVCS) in accordance with the principles of the present invention. FIG. FIG. 3 is a schematic diagram illustrating the operation of an SVCS switch according to the principles of the present invention. FIG. 3 is an exemplary SVCS switch layer configuration matrix, in accordance with the principles of the present invention. FIG. 3 is an exemplary SVCS network layer configuration matrix, in accordance with the principles of the present invention.

Claims (71)

A system for video communication between a plurality of endpoints via an electronic communication network and one or more servers, wherein the network links with the plurality of endpoints and servers different quality of service and bandwidth A plurality of channels, the channels including a designated high reliability channel (HRC) and a low reliability channel (LRC);
Comprising a transmitting and receiving terminal located at the endpoint;
At least one transmitting terminal creates at least one scalable encoded video signal for transmission to other terminals in base layer and enhancement layer formats, and at least through the designated HRC Configured to transmit at the base layer,
At least one receiving terminal decodes the scalable encoded video signal layer received via a network channel including a designated HRC, and combines the decoded video signal layer; The scalable transmitted by the transmitting terminal towards the receiving terminal via an electronic communication network channel leading to the receiving terminal, further configured to reconstruct the video for local use A system configured to arbitrate the transfer of a video signal layer encoded in the.   At least one terminal has a raw video signal for encoding and transmission, a stored video signal for encoding and transmission, a synthesized video signal for encoding and transmission, and a prior for transmission The system of claim 1, wherein the system is configured to access at least one of the video signals encoded in the. An endpoint terminal for video communication with other endpoints via one or more servers located in an electronic communication network, wherein the network links to the multiple endpoints And a bandwidth channel, wherein the channel contains a designated HRC;
At least one scalable video encoder configured to scalable encode at least one video signal in a base layer and enhancement layer format;
A packet multiplexer configured to multiplex layers of the video signal encoded in the base layer and enhancement layer format for transmission over the electronic communication network;
An endpoint terminal configured to designate at least the base layer from the base layer and enhancement layer of the video signal for transmission via the network interface controller and the designated HRC.   4. The endpoint terminal according to claim 3, further comprising an audio signal encoder whose output is connected to the packet multiplexer. The scalable video encoder is motion compensated, and a block-based codec
A frame memory in which one or more decoded frames are stored for future reference;
A picture type (I, P, or B), as well as a reference controller configured to select a picture in the frame memory to be used as a predictive reference for the currently encoded picture,
The terminal according to claim 3, further configured to perform picture prediction using threads as means for implementing a temporal scalability layer.   6. The terminal of claim 5, wherein the scalable video encoder is configured to create one continuous prediction chain path for the base layer. The thread
A base layer thread further comprised of several distant pictures, wherein the temporal prediction is performed using one or more previous pictures of the same thread;
Temporal enhancement layer thread composed of remaining pictures, wherein the prediction is performed from one or more preceding base layer pictures and / or from one or more preceding temporal enhancement layer pictures 6. The terminal according to claim 5, wherein the terminal is a picture thread comprising a layer thread. The picture thread is
A base layer thread further composed of pictures, which are a fixed number of distant pictures, wherein the temporal prediction is performed using the previous frame of the same thread;
A first temporal enhancement layer thread composed of frames intermediate between frames of the base layer thread, wherein prediction is performed from the previous base layer picture or from the previous first temporal enhancement layer picture A first time enhancement layer thread,
A second temporal enhancement layer thread consisting of the remaining pictures, where the prediction is either the immediately preceding second temporal enhancement layer picture, the immediately preceding first temporal enhancement layer picture, or the immediately preceding base layer thread picture 6. The terminal according to claim 5, comprising a second time enhancement layer thread implemented from the above.   The scalable video encoder is configured to encode the base time layer frame with quantization finer than the quantization used for other time layers, whereby the base layer is 6. The terminal according to claim 5, wherein the terminal is encoded more accurately than other layers.   6. The terminal of claim 5, wherein the scalable video encoder is configured to create at least one prediction chain that terminates at an enhancement layer.   6. The terminal of claim 5, wherein the temporal prediction codec further comprises an SNR quality scalability layer encoder.   The SNR quality scalability enhancement layer encoder consists of an input of residual coding error of the base layer obtained by subtracting the decoded base layer frame from the original frame and applying a positive offset; 12. The terminal according to claim 11, which encodes the difference in the same manner as the base layer encoder.   12. The terminal of claim 11, wherein the SNR enhancement layer encoder is further configured to use the predicted path that is different from the predicted path for the base layer or low enhancement layer.   The method of claim 12, wherein the SNR enhancement layer encoder is further configured to omit a DC component of a direct cosine transform (DCT) coefficient when predicting a video frame to be encoded for an SNR quality enhancement layer. The listed terminal.   The SNR enhancement layer encoder is further configured to quantize the DC and surrounding AC DCT coefficients at a coarser level than the remaining DCT coefficients when encoding a video frame for the SNR quality enhancement layer; The terminal according to claim 12.   The SNR quality scalability layer decoder at the receiving endpoint displays the decoded base layer frame at the desired reduced resolution by applying low pass filtering after decoding and by downsampling The terminal according to claim 11, configured as follows.   The terminal according to claim 11, wherein the SNR quality scalability enhancement layer codec comprises an H.264 SVC FGS codec with threading.   The spatial scalability layer codec includes an H.264 SVC FGS codec configured to use a weighted average of a previous enhancement layer picture and a current base layer picture in motion compensated prediction, wherein the weighting comprises the prediction chain 18. The terminal of claim 17, wherein the terminal dynamically changes to include a zero value that is a value that terminates, thereby eliminating drift.   The SNR quality scalability layer encoder encodes the difference between the DCT coefficients before and after quantization by requantizing the difference and applying entropy coding to the requantized difference. The terminal according to claim 11, wherein the terminal is configured.   The temporal prediction codec further comprises a spatial scalability layer encoder configured to low-pass filter and downsample the original input signal, wherein the low resolution is different from an intended display resolution 6. The terminal of claim 5, wherein the terminal can be used as an input to the base layer encoder.   21. The terminal of claim 20, wherein the spatial scalability layer encoder is configured such that a predicted path for an enhancement layer is different from the predicted path for the base layer or a low enhancement layer. The spatial scalability layer encoder is
Up-sampling the decoded low resolution signal to the original input signal resolution;
Subtracting the original input signal from the upsampled and decoded low resolution signal to obtain a differential signal;
21. The terminal of claim 20, configured to apply an offset to the difference signal and further encode the offset difference signal.   23. The terminal of claim 22, wherein the spatial scalability layer encoder is configured to quantize DC and surrounding DCT AC coefficients more coarsely than the remaining DCT AC coefficients.   The spatial scalability layer encoder is configured to use two predictive encodings when predicting a high-resolution video frame, the first reference picture is a decoded past highest resolution picture, and the second 21. The terminal of claim 20, wherein a reference picture is obtained by first encoding and decoding the downsampled base layer signal and then upsampling it to the original resolution.   The spatial scalability layer encoder is composed of an H.264 AVC encoder using two prediction encodings, the upsampled and decoded base layer frame is inserted as an additional reference frame, and motion vector prediction 25. A terminal according to claim 24, wherein a time and space direct mode is used to increase the compression efficiency.   6. The terminal of claim 5, comprising a base layer encoder and further comprising at least one of an SNR quality layer encoder, a spatial scalability layer encoder, and a temporal enhancement layer encoder. An endpoint terminal for video communication with other endpoints via one or more servers located in an electronic communication network, wherein the network links different endpoints of quality of service and Providing a bandwidth channel, the channel containing a designated HRC;
A scalable video decoder configured to scalable decode one or more video signals in base layer and enhancement layer formats;
A packet demultiplexer configured to demultiplex the layer of the video signal encoded in the base layer and enhancement layer format after being received via the electronic communication network via a network interface controller An endpoint terminal comprising:   28. The terminal of claim 27, wherein the decoder is comprised of an SNR quality scalability decoder.   The SNR quality scalability decoder is configured to display the decoded base layer frame at a desired reduced resolution by applying low pass filtering after decoding and by downsampling; The terminal according to claim 28.   The SNR quality scalability enhancement layer decoder is configured to add a decoded residual error carried by the enhancement layer data to the decoded base layer frame after subtracting a positive offset. 28. The terminal according to 28.   28. The terminal of claim 27, wherein the decoder further comprises a spatial scalability decoder. The spatial scalability layer decoder comprises:
Up-samples the decoded base resolution signal to the enhancement layer resolution;
Decoding the offset differential signal carried by the enhancement layer;
32. The terminal of claim 31, configured to subtract an offset from the decoded enhancement layer signal and add the result to the upsampled and decoded base resolution signal.   The spatial scalability layer decoder comprises an H.264 AVC decoder with two predictive coding support, and the upsampled and decoded base layer frame is inserted as an additional reference frame. The terminal described in.   6. The terminal of claim 5, wherein the scalable video encoder is configured to encode an input signal with two or more spatial and / or quality resolutions that can be transmitted simultaneously.   The terminal according to claim 3, wherein the scalable coding structure can be dynamically changed in any scalability dimension according to a preference of a network condition or a receiving endpoint. A method for communicating between multiple endpoints via an electronic communication network and one or more servers, the network providing different quality of service and bandwidth channels that link to the multiple endpoints And the channel contains the designated HRC;
Encoding the video signal in a base layer and enhancement layer format in a scalable manner;
Multiplexing the layers of the video signal for transmission over the electronic communication network;
Transmitting at least the base layer from the base layer and enhancement layer of the video signal for transmission via the designated HRC.   The step of multiplexing the layers of the video signal for transmission over the electronic communication network further comprises the step of multiplexing an audio signal for transmission over the electronic communication network. 36. The method according to 36. A method for communicating a video signal picture scalable in base and enhancement layer formats between multiple endpoints over an electronic communication network, comprising:
Selecting a picture type (I, P, or B) for the picture that is currently encoded and a predicted reference picture from the decoded pictures stored in the frame memory;
Creating a temporal scalability layer by performing picture prediction using a thread.   40. The method of claim 38, further comprising creating a continuous predicted chain path for the base layer. The picture thread includes a base layer thread that includes pictures that are a number of distant pictures, and the method further uses each one or more preceding pictures of the base layer picture to Including performing a time prediction within,
The temporal enhancement layer thread includes the remaining pictures, and the method further predicts in each enhancement layer picture using one or more preceding base layer pictures, or one or more preceding temporal enhancement layer pictures 40. The method of claim 38, comprising the step of: The picture thread includes a base layer thread that includes a picture that is a number of distant pictures, and the method further comprises performing temporal prediction using a frame immediately preceding the picture;
The first temporal enhancement layer thread includes a frame that is intermediate between the frames of the base layer thread, and the method further predicts from the immediately preceding base layer frame or the immediately preceding first temporal enhancement layer thread picture. And the second temporal enhancement layer thread includes the remaining frames, and the method further includes the immediately preceding second temporal enhancement layer thread picture, the immediately preceding first temporal enhancement layer thread. 40. The method of claim 38, comprising performing temporal prediction using a picture or a frame from a previous base layer thread picture.   Further comprising encoding the base time layer frame with quantization finer than that used for other time layers, whereby the base layer is even more than other layers. 40. The method of claim 38, wherein the method is encoded correctly.   39. The method of claim 38, further comprising creating at least one prediction chain that terminates at the enhancement layer.   40. The method of claim 38, wherein the step of scalable encoding a temporal scalability layer by performing picture prediction using a thread further comprises encoding an SNR quality scalability enhancement layer.   The step of encoding an SNR quality scalability enhancement layer applies a positive offset to the residual encoding error of the base layer obtained by subtracting the decoded base layer frame from the original frame; 45. The method of claim 44, comprising encoding the difference similar to the step of encoding the base layer.   45. The method of claim 44, wherein the step of encoding an SNR quality scalability enhancement layer comprises using the predicted path that is different from the predicted path for the base layer or low enhancement layer.   The method of claim 45, wherein the step of encoding an SNR quality scalability enhancement layer includes omitting a DC component of a discrete cosine transform (DCT) coefficient when encoding a picture for the SNR quality scalability enhancement layer. Method.   The step of encoding an SNR quality scalability enhancement layer, when encoding a video frame for the SNR quality scalability enhancement layer, quantizes the DC and surrounding AC DCT coefficients at a coarser level than the remaining DCT coefficients 46. The method of claim 45, comprising the step of:   The step of encoding an SNR quality scalability enhancement layer further includes applying low pass filtering after decoding and down-sampling at the receiving endpoint to reduce the decoded base layer frame at a desired reduced resolution. 45. The method of claim 44, comprising the step of displaying.   45. The method of claim 44, wherein the step of encoding an SNR quality scalability enhancement layer further comprises an H.264 SVC FGS codec with threading.   Further comprising using an H.264 SVC FGS codec configured to use a weighted average of the previous enhancement layer picture and the current base layer picture in motion compensated prediction, wherein the weighting is a zero value at which the prediction chain ends 51. The method of claim 50, wherein the method dynamically changes to include, thereby eliminating drift.   The step of encoding the SNR quality scalability enhancement layer includes encoding the difference between the DCT coefficients before and after quantization by requantizing the difference and applying entropy coding to the requantized difference. 45. The method of claim 44, comprising the step of:   The step of encoding the temporal quality layer is further intended to apply a low pass filtering to the original input signal and to encode a spatial scalability layer by downsampling, and the low resolution. 39. The method of claim 38, further comprising encoding the downsampled signal different from a display resolution and similar to the base layer.   54. The method of claim 53, wherein the step of encoding a spatial scalability layer comprises using the predicted path for an enhancement layer different from the predicted path for the base layer or low enhancement layer. Said step of encoding a spatial scalability layer comprises:
Up-sampling the decoded low resolution signal to the original input signal resolution;
Subtracting the original input signal from the upsampled and decoded low resolution signal to obtain a differential signal;
Applying an offset to the difference signal;
54. The method of claim 53, comprising encoding the offset difference signal. Said step of encoding a spatial scalability layer comprises:
56. The method of claim 55, comprising quantizing the DC and surrounding DCT AC coefficients more coarsely than the remaining DCT AC coefficients. Said step of encoding a spatial scalability layer comprises:
When predicting a high-resolution video frame, the method includes using two predictive encodings, wherein a first reference picture is a decoded past highest resolution picture, and a second reference picture is the downsampled 54. The method of claim 53, wherein the obtained base layer signal is obtained by first encoding and decoding and then upsampling it to its original resolution. Said step of encoding a spatial scalability layer comprises:
Using an H.264 AVC encoder with two predictive encodings, wherein the upsampled and decoded base layer frame is inserted as an additional reference frame, and temporal and spatial direct modes of motion vector prediction 58. The method of claim 57, wherein: is used to increase compression efficiency.   39. The method of claim 38, comprising using a base layer encoder, further comprising at least one of an SNR quality layer encoder, a spatial scalability layer encoder, and a temporal enhancement layer encoder. A method for communicating a video signal picture scalable encoded in base and enhancement layer formats between multiple endpoints and one or more servers via an electronic communication network, comprising:
A scalable video decoder configured to scalable decode one or more video signals in base layer and enhancement layer formats;
A packet demultiplexer configured to demultiplex the layer of the video signal encoded in the base layer and enhancement layer format after being received via the electronic communication network via a network interface controller A method comprising the step of using.   64. The method of claim 60, wherein the decoder is comprised of a SNR quality scalability decoder.   Using the SNR quality scalability decoder to display the decoded base layer frame at a desired reduced resolution by applying low pass filtering after decoding and by downsampling; 64. The method of claim 61, further comprising:   The method further comprises using the SNR quality scalability decoder to add a decoded residual error carried by the enhancement layer data to the decoded base layer frame after subtracting a positive offset. Item 62. The method according to Item 61.   64. The method of claim 60, wherein the decoder further comprises a spatial scalability decoder. Up-samples the decoded base resolution signal to the enhancement layer resolution;
Decoding the offset difference signal carried by the enhancement layer;
The method further comprising: subtracting an offset from the decoded enhancement layer signal and using the spatial scalability decoder to add the result to the upsampled and decoded base resolution signal. the method of.   The spatial scalability layer decoder comprises an H.264 AVC decoder with two predictive coding support, further comprising inserting an upsampled and decoded base layer frame as an additional reference frame. The method according to 64.   39. The method of claim 38, wherein the step of scalable encoding a video signal comprises encoding the signal with two or more spatial and / or quality resolutions that can be transmitted simultaneously.   40. The method of claim 38, wherein the scalable coding structure can be dynamically changed in any scalability dimension depending on network conditions or preference indications indicated by a receiving endpoint. A method for video communication between a plurality of endpoints via an electronic communication network and one or more servers, wherein the networks link different endpoints and servers with different quality of service and bandwidth The channel includes a designated high reliability channel (HRC) and a low reliability channel (LRC);
Placing transmitting and receiving terminals at the endpoint;
Creating at least one scalable encoded video signal for transmitting at least one transmitting terminal to other terminals in base layer and enhancement layer formats, and at least said via a designated HRC; Configuring to transmit at the base layer;
At least one receiving terminal to decode the scalable encoded video signal layer received via a network channel including a designated HRC and by combining the decoded video signal layer Configuring the video to be reconfigured for local use;
Configuring the server to arbitrate transfer of the scalable encoded video signal layer transmitted by the transmitting terminal towards the receiving terminal via an electronic communication network channel leading to the receiving terminal; And a method comprising.   Said steps comprising at least one transmitting terminal comprising: a raw video signal for encoding and transmission; a stored video signal for encoding and transmission; a combined video signal for encoding and transmission; 70. The method of claim 69, further comprising: configuring the terminal to access at least one of a pre-encoded video signal for transmission.   71. A computer readable medium comprising a set of instructions for performing the steps set forth in at least one of claims 36 to 70.

Images