JP4921488B2 - System and method for conducting videoconference using scalable video coding and combining scalable videoconference server - Google Patents

Description

(Cross-reference of related applications)
This application claims the benefit of US Provisional Patent Application No. 60 / 753,343, filed December 22, 2005. In addition, this application includes International Patent Application Nos. PCT / US06 / 28365, PCT / US06 / 028366, PCT / US06 / 028367, PCT / US06 / 027368, and PCT / US06 / 061815, And US Provisional Patent Applications Nos. 60 / 778,760, 60 / 787,031, 60 / 774,094, and 60 / 827,469. All of the foregoing priority and related applications are incorporated herein by reference in their entirety.

  The present invention relates to multimedia technology and telecommunications. In particular, the present invention relates to communicating or delivering audio and video data for person-to-person and multi-party conferencing applications. More specifically, the present invention supports the reception of a video bitstream corresponding to a single picture encoded using a video encoding technique that is scalable by a person-to-person conference application or a number of participants using scalable video coding techniques. Intended for implementation of multiparty conferencing applications that may only be possible. The present invention also provides such a system via a communication network connection that can provide different levels of quality of service (QoS), and in an environment where end users can access conferencing applications using different capabilities of devices and communication channels. It is intended to implement.

  The video conferencing system allows two or more remote participants / endpoints to communicate video and audio with each other in real time using both audio and video. If only two remote participants are involved, direct transmission of communications over the appropriate electronic network can be used between the two endpoints. If more than two participants / endpoints are involved, a multipoint conference unit (MCU), or bridge, is commonly used to connect all participants / endpoints. The MCU mediates communication between multiple participants / endpoints that can be connected to, for example, a star configuration. Note that even if only two participants are involved, it may still be advantageous to utilize an MCU between the two participants.

  For video conferencing, participants / endpoints or terminals are equipped with appropriate encoding and decoding devices. The encoder formats the local audio and video output at the transmission endpoint into an encoding format suitable for signaling over the electronic network. In contrast, the decoder processes the received signal with encoded audio and video information into a decoding format suitable for audio playback or image display at the receiving endpoint.

  Traditionally, the end user's own image is also displayed on his screen to provide feedback (eg, to ensure proper placement of the person in the video window).

  In an actual videoconferencing system implementation over a communications network, the quality of interactive videoconferencing between remote participants is determined by end-to-end signal delay. End-to-end delays of over 200 ms prevent realistic live or natural interactions between conference participants. This long end-to-end delay is proactive for video conference participants to allow video and audio data in transit from other participants to reach that endpoint. To unnaturally participate or respond.

  End-to-end signal delay includes acquisition delay (e.g., the time taken to fill the A / D converter buffer), encoding delay, transmission delay (e.g., data packet to endpoint network interface) Delay that corresponds to the time it takes to transmit to the controller) and transport delay (delay that corresponds to the time it takes for the packet to travel from endpoint to endpoint over the network). In addition, signal processing time through the intervening MCU also contributes to the total end-to-end delay in a given system.

  The main task of the MCU is to mix the incoming audio signals so that a single audio stream is sent to all participants, and the video frames or pictures sent by individual participants / endpoints Mixing them into a common composite video frame stream containing each participant's picture. The terms frame and picture are used interchangeably herein and, furthermore, as will be apparent to those skilled in the art, encoding interlaced frames as individual fields or as combined frames. Note that (field-based or frame-based picture coding) can also be incorporated. MCUs deployed in traditional communication network systems have a single common resolution for all of the individual pictures mixed in a common composite video frame delivered to all participants in the video conference session. It only provides (eg CIF or QCIF resolution). Thus, conventional communication network systems cannot easily provide customized video conferencing functions that allow one participant to view other participants at different resolutions. Customized features, for example, allow other specific participants (e.g. speaking participants) to see at CIF resolution and allow other silent participants to see at QCIF resolution be able to. MCUs in the network may be configured to provide such customized functionality by repeating the video mixing operation as many times as the number of participants in the video conference. However, in such a configuration, the operation of the MCU generates significant end-to-end delay. In addition, the MCU decodes, mixes, re-encodes multiple audio streams, decodes multiple video streams, combines them into a single frame (with appropriate scaling if necessary), and combines them Again, it is necessary to have sufficient digital signal processing capability to re-encode into a single stream. Video conferencing solutions (such as systems available from Willow Road, Pleasanton, California, U.S. 94588, Polycom Inc., 4750, and Tandberg, Park Avenue 200, New York, U.S. 10166) In order to provide acceptable quality and performance levels, dedicated hardware components must be used.

  ITU-T Recommendation H.261, ITU-T Recommendation H.262 | ISO / IEC 13818-2 (MPEG-2 Video) main profile, ITU-T Recommendation H.263 baseline profile, ISO / IEC 11172-2 (MPEG-1 Video), ISO / IEC 14496-2 simple profile or advanced simple profile, ITU-T recommendation H.264 | ISO / IEC 14496-10 (MPEG-4 AVC) baseline profile or Conventional video codecs whose bitstreams and decoding operations are standardized in the main profile or high profile are specified to provide a single bitstream at a given spatial resolution and bitrate. Thus, for a coded video signal, if a lower spatial resolution or lower bit rate is required compared to the original encoded spatial resolution or bit rate, a full resolution signal is received. And must be decoded and possibly downscaled and re-encoded with the desired spatial resolution and bit rate. The decoding, possibly down-sampling, and re-encoding process requires significant computational resources, and usually adds significant subjective distortion to the video signal and delays the video transmission. Append.

  Furthermore, standard video codecs for video communications are based on “single layer” coding techniques that are inherently unable to take advantage of the differentiated QoS features provided by modern communications networks. A further limitation of single layer coding techniques for video communication is that the application requires a lower spatial resolution display, or even when desired, a full resolution signal is received and implemented at the receiving endpoint or MCU. It needs to be decoded using downscaling. This consumes bandwidth and computational resources.

  Contrary to the single layer video codec described above, a “scalable” video codec based on a “multilayer” coding technique generates two or more bitstreams for a given source video signal. A base layer and one or more reinforcing layers. The base layer can be a basic representation of the source signal at a minimum quality level. The minimum quality representation may be a reduced quality (ie, signal to noise ratio (“SNR”)), spatial or temporal resolution aspect, or a combination of these aspects for a given source video signal. One or a plurality of enhancement layers correspond to information for enhancing the quality of the SNR, spatial or temporal resolution aspect of the base layer. Scalable video codecs have been developed considering heterogeneous network environments and / or heterogeneous receivers.

  Scalable coding has been part of the ITU-T Recommendation H.262 | ISO / IEC 13818-2 (MPEG-2 Video) SNR scalable, spatial scalable, and high profile standards. However, the actual use of such “scalable” video codecs in video conferencing applications is due to the increased cost and complexity associated with scalable coding, as well as high bandwidth IP-based communications appropriate for video. It has been hampered by the lack of widespread availability of channels.

  International Patent Application No. PCT / US06 / 02836, co-pending and assigned to the present applicant, incorporated herein by reference, describes practical scalable video coding techniques specifically for video conferencing applications. ing. In addition, the co-pending and assigned international patent application PCT / US06 / 02835, incorporated herein by reference, takes advantage of the features of scalable video coding techniques for videoconferencing applications. Describes a conference server architecture designed to benefit. International Patent Application No. PCT / US06 / 061815, co-pending and assigned to the present applicant, incorporated herein by reference, takes advantage of the features of scalable video coding techniques for video conferencing applications. Describes a technique for providing error-tolerance, layer switching, and random access capabilities to a conference server architecture designed to achieve

  Currently, an extension to the ITU-T Recommendation H.264 | ISO / IEC 14496-10 standard is being considered that provides a more efficient tradeoff than the previously standardized scalable video codec (Annex G, Scalable Video Coding-SVC). Further developments in video coding research and standardization include error resilience and video mixing in MCUs, ie the concept of multiple slice groups to combine multiple input videos into one output video. (`` Scattered Slices: A New Error Resilience Tool for H.26L '' by S. Wenger and M. Horowitz, JVT-B027, ITU-T SG16 / Q.6 and ISO / IEC JTC 1 / SC29 / WG 11 and ITU (See -T Recommendation H.264 | ISO / IEC 14496-10 Joint Video Team (JVT) document). If all input video signals are encoded using ITU-T Recommendation H.264 | ISO / IEC 14496-10, the various input signals can be placed in the MCU's output picture as separate slice groups. As such, decoding and re-encoding may not be required in the MCU. (See "Coding of Parameter Sets" by MM Hannuksela and YK Wang, JVT-C078, ITU-T SG16 / Q.6 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Team (JVT) documents. )

  Improvements are currently underway to improve the conference server or MCU architecture for video conferencing applications. In particular, a server architecture for combining one or more input video signals into a single output video signal, together with possible server generated data, using coding domain combining techniques such as multiple slice groups. There is interest in developing. The preferred conference server architecture supports desirable video conferencing features such as continuous presence, personal view or layout, rate matching, error resilience, and random entry, and the complexity and delay overhead of traditional MCUs Will be avoided.

  Systems and methods for conducting a video conference are provided. Each video conference participant sends an encoded data bitstream to the conference bridge MCU or server. The encoded data bitstream can be a single layer, or scalable video coding (SVC) data, and / or a scalable audio coding (SAC) data bitstream, from which multiple qualities are derived. Can do. The MCU or server (e.g., “Complex Scalable Video Coding Server” (CSVCS) below) combines the input video signal from the sending conference participant into a single composite output video signal that is forwarded to the receiving participant. Configured to do. CSVCS is specifically configured to synthesize the output video signal picture without decoding, rescaling, and re-encoding the input signal, thereby minimizing end-to-end delay Or nothing at all. This “zero delay” architecture of CSVCS advantageously allows its use in a cascaded configuration. The CSVCS composite output bitstream allows it to be decoded with a single decoder.

  For video conferencing applications, each participant receives a scalable data bitstream having multiple layers (e.g., a base layer and one or more augmented layers encoded using SVC) with a corresponding number of physical Send to CSVCS over channel or virtual channel. Some participants can also send single layer bitstreams. CSVCS can select a portion of the scalable bitstream from each participant according to requirements based on the characteristics and / or settings of the particular receiving participant. The selection can be based, for example, on the bandwidth of the particular receiving participant and the desired video resolution.

  CSVCS combines the selected input scalable bitstream portions into one (or more) output video bitstreams that can be decoded by one (or more) decoders. If SVCs are used for the output video bitstream, the decoding will be performed for each input video along with the generation of possible supplemental layer data so that the output stream is a valid SVC bitstream. This is accomplished by assigning signals to slices in different slice groups of the output video signal. CSVCS is configured to generate a composite output video signal without signal processing or with minimal signal processing. CSVCS reads the packet header of the incoming data so that the appropriate packet can be selectively multiplexed into the access unit of the output bitstream, for example, to synthesize the output signal, and then join For each of the parties, it may be configured to transmit the combined output signal along with any generated layer data.

  In a videoconference scene, the input video signal content may or may not be sufficient to cover all areas of the picture in the output bitstream in time at a given moment. The insufficiency may be due to, for example, different temporal resolutions of the input video signal, shifts between temporal samplings of the input video signal, and incomplete filling of the output video signal. CSVCS minimizes end-to-end delays or generates higher temporal resolution of the output video signal in order to minimize other problems caused by the incoming video signal coming late. It can be configured to ameliorate the problem of sufficient picture area coverage. For example, CSVCS inserts pre-encoded slices obtained from accessible storage media for those portions of the output video signal where input video signal content is not present or not available Can be configured. A pre-coded slice may be composed of header and coded slice data that may be computed by CSVCS or pre-computed according to the specific layout of the output picture. Alternatively, CSVCS processes the input video signal with higher temporal resolution by inserting encoded picture data that instructs the receiving endpoint to simply duplicate the previously encoded picture. can do. Note that such coded picture data has a very short length on the order of a few bytes.

  Exemplary embodiments of video conferencing in accordance with the present invention provide differentiated quality of service (QoS) (i.e., provide a reliable communication channel for some portion of the total bandwidth required). Communication network connections, video codecs, CSVCS, and end user terminals can be included. Video codecs for transmitting participants are single layer video or scalable video that provides scalability in terms of time, quality, or spatial resolution at different transmission bandwidth levels. A video codec for at least one of the receiving participants supports scalable video decoding. The end-user terminals used by the sending and receiving participants are dedicated hardware systems or general purpose PCs capable of operating multiple instances of the video decoder and at least one instance of the video encoder be able to. An exemplary system implementation combines the functionality of a conventional MCU and / or other conference server (such as the SVCS described in PCT / US06 / 28366) with the functionality of the CSVCS described herein. it can. In such a combined system, MCU, SVCS, and CSVCS functions may be used selectively, individually, or in combination to serve different parts or entities in a video conference session.

  The CSVCS function can complement the SVCS function. CSVCS can be configured to have some or all of the features and benefits of SVCS. However, instead of sending multiple SVC streams to each endpoint as SVCS does, CSVCS encapsulates each stream in a single output SVC stream where each stream is assigned to a different slice group, Or at least different from SVCS in terms of synthesis. CSVCS can then be considered as an SVCS for all purposes, and the output stage can further generate additional layer data that may be needed to ensure that the output bitstream is compliant. In addition, it includes the additional process of slice group based assignment. All SVCS functions (e.g., rate matching, personalized layout, error resilience, random access, and layer switching, rate control) can therefore be supported by CSVCS, and the number of packets transmitted from CSVCS is Note that it is approximately the same as the number that should be sent from the SVCS in the same conference setup.

  Further features of the invention, its nature and various advantages will be more apparent from the following detailed description of the preferred embodiments and the accompanying drawings.

  The present invention provides a system and method for implementing a video conferencing system that uses scalable video coding with a server that provides picture decoding in the coding domain. The systems and methods deliver video and audio data that is encoded by a transmitting video conference participant using single layer encoding or scalable encoding techniques. Scalable video coding techniques encode source data into a number of different bitstreams (e.g., base layer and enhancement layer bitstreams) that can vary in time resolution, quality resolution (i.e., in terms of SNR). ), And in the case of video, the spatial resolution provides a representation of the original signal. A receiving participant can decode a bitstream that is encoded using scalable video coding techniques and that includes multiple slice group functions for various input signals.

  There can be multiple servers in the communication path between the sending participant or endpoint and the receiving participant or endpoint. In such a case, at least the last server in the path combines the video picture input from the sending participant into a single composite output picture encoded using scalable video coding techniques, and The composite output picture will be sent to the receiving participant. Importantly, the compositing process at or by the server does not require decoding and re-encoding of the picture data received from the sending participant, but the output bitstream complies with the requirements of a scalable video decoder It may be necessary to generate additional layer data to ensure that

  For reference purposes and to assist in understanding the present invention, for the embodiments of the present invention described herein (hereinafter referred to as “SVC embodiments”), the base layer bitstream is ITU-T and ITU-T Recommendation H.264 | ISO / IEC 14496-10 (MPEG4-AVC) specified in ISO / IEC JTC 1, “Advanced video coding for generic audiovisual services” ) ", ITU-T Recommendation H.264, and ISO / IEC 14496-10 (MPEG4-AVC). Furthermore, it is assumed that the enhancement layer bitstream conforms to the scalable extension of ITU-T Recommendation H.264 | ISO / IEC 14496-10 (MPEG4-AVC) (Annex G, Scalable Video Coding, hereafter “SVC”) To do. The use of the SVC codec may be beneficial, for example, when a variable picture size of the input video signal is required to be present in the output video picture of the MCU. Note that the H.264 AVC and SVC standards are different. SVC is another Annex of H.264 that appears in the 2007 version of H.264. For the embodiment described in the present invention, H.264 AVC is used for the scalable codec base layer, while H.264 SVC is used for the scalable codec enhancement layer. However, for the sake of convenience, the scalable video codecs used for the base layer (H.264 AVC) and for the enhancement layer (H.264 SVC) are collectively referred to herein as “ It can be called the “SVC” codec. Note further that H.264 AVC is considered a single layer codec, but provides scalability in the time dimension. The use of the H.264 AVC and H.264 SVC codecs in the embodiments described in the present invention is exemplary only, and other codecs suitable for combining pictures are in accordance with the principles of the present invention. It will also be appreciated that it can be used instead.

  FIG. 1 illustrates an example system 100 for compositing pictures that can be implemented in an electronic or computer network environment in multipoint and point-to-point conferencing applications. The system 100 includes one or more network servers (e.g., a complex scalable videoconferencing server (CSVCS) server 110) to coordinate delivery of customized data to conference participants or clients 120, 130, and 140. use. The CSVCS server 110 can coordinate the delivery of the video stream generated by the endpoint 140 for transmission to other conference participants, for example. In system 100, using SVC techniques, video stream 150 is first appropriately encoded or scaled down into various data components or layers. The multiple data layers can have different characteristics or functions (eg, spatial resolution, frame rate, picture quality, signal-to-noise ratio (SNR), etc.). Different characteristics or functions of the data layer take into account, for example, changing individual user requirements and infrastructure specifications in the electronic network environment (e.g. CPU capacity, display size, user preferences, and bit rate) You can choose appropriately.

  The CSVCS 110 can have a scalable video signal processing function similar to the scalable video conference server (SVCS) and the scalable audio conference server (SACS) described in International Patent Application No. PCT / US06 / 028366. However, CSVCS 110 is further configured to use the H.264 AVC and H.264 SVC codecs, especially to combine multiple input video signals into one output video signal using multiple slice groups. The

  In system 100, clients 120, 130, and 140 can each use a terminal that is appropriate for an interactive conference. Terminals may include human interface input / output devices (e.g., cameras, microphones, video displays, and speakers) and other signal processing components such as encoders, decoders, multiplexers (MUX), demultiplexers (DEMUX), etc. it can.

  Further, as described in co-pending International Patent Application No. PCT / US06 / 028366, in an exemplary terminal, the camera and microphone can transmit the participant's video and video for transmission to other conference participants. Each audio signal is designed to be captured. Conversely, video displays and speakers are designed to display and play video and audio signals received from other participants, respectively. The video display can also be configured to optionally display the participant's / terminal's own video. The camera and microphone at the terminal can be coupled to an analog-to-digital converter (AD / C), which is then coupled to its respective encoder. The encoder compresses the local digital signal in order to minimize the bit rate required for signal transmission. The encoder output data can be “packetized” into RTP packets (eg, via packet MUX) for transmission over an IP-based network. The packet MUX can perform conventional multiplexing using the RTP protocol and can also perform any necessary QoS related protocol processing. For example, as described in co-pending International Patent Application No. PCT / US06 / 061815, QoS support is provided for packets required to decode at least the lowest time level of the base layer for reliable delivery. It can be provided with an acknowledgment and / or a negative response along with the marking. Each of the terminal data streams may be transmitted on its own virtual channel, i.e. port number in IP terminology.

  In the implementation of the SVC embodiment of the present invention, the system 100 uses an AVC or SVC codec for the input bitstream to the CSVCS and uses an SVC for the output video bitstream from the CSVCS 110. Thus, the characteristics of a plurality of slice groups are used in the composite of output pictures. However, the audio signal in the system 100 is independent of the composite of the output picture, such as the technique described in ITU-T recommendation G.711 or ISO / IEC 11172-3 (MPEG-1 Audio). It can be encoded using any suitable technique known in the art.

  FIG. 2 shows an exemplary output video picture 200 provided by CSVCS 110, which is a composite of multiple slice groups (eg, slice groups 1, 2, 3, 4). The partitions or boundaries between slice groups are indicated in FIG. 2 by dashed lines. Slice groups 1, 2, 3, and 4 can have a syntax structure in ITU-T recommendation H.264 | ISO / IEC 14496-10. The assignment of a specific slice group to a picture shall be specified in the bitstream, one per picture, in the picture parameter set (PPS) of ITU-T Recommendation H.264 | ISO / IEC 14496-10 bitstream. Can do. The PPS can be carried in-band or out-of-band as part of the bitstream. Carrying the PPS in-band requires that the PPS be multiplexed into the access unit of the bitstream. Conversely, carrying PPS out-of-band means that a separate transmission channel is used for PPS transmission, or the PPS is implemented in the decoder before the decoder is used in a transmission scenario. It may be necessary to It is possible to use up to 256 different PPS. The signaling of which PPS needs to be used for a picture may be indicated in the slice header by number reference.

  FIG. 3 shows an exemplary assignment of an input video signal or picture to a slice group of output video picture 200 (FIG. 2) generated by CSVCS 110. Input video signal assignment can be achieved in the compression domain by changing the slice header and assigning it to a slice group of the output video. For example, in the assignment shown in FIG. 3, input video signal 0 is assigned to slice group 0, input video signal 1 is assigned to slice group 1, input video signal 2 is assigned to slice group 2, and input video signal 3 And 4 are both assigned to slice group 3. The assignment can be performed by mapping the input video signal to a slice of a slice group in the output picture. This mapping method results in both portions and regions assigned in a particular slice group (FIG. 3) and unassigned portions and regions 310.

  According to ITU-T recommendation H.264 | ISO / IEC 14496-10, the entire decoded picture (for example, output video picture 200) must be described by encoded slice data included in the bitstream. Since the input video signal allocation for slices in a slice group can be both allocated and unallocated regions, the CSVCS 110 creates encoded slice data for the unallocated regions while compositing pictures. Configured to do.

  In the implementation of the SVC embodiment of the present invention, the coded slice data may include skip macroblock data or intra-coded macroblock data. The latter data may be necessary to create content for the unallocated area of the output picture. Intra-encoded data can have any suitable content. The content can describe a picture signal that can be transmitted at a low bit rate, eg, a uniform gray or black texture. Alternatively or additionally, the content can describe the addition of MCU control functions such as user information, graphical annotations, and conference control functions.

  In system 100, conference control functions are activated in response to simple signaling or requests by the client / participant (e.g., signaling by the client / participant pointing to a specific coordinate or region on the video display image screen). obtain. For this purpose, the CSVCS 110 is an action represented by specific coordinates or regions on a video display image screen (e.g. shown as a button to initiate some action and having a working image region). And is configured to convert the signal. The signaling by the client can be implemented, for example, using HTTP techniques in which CSVCS provides an HTTP interface for receiving such signals, similar to a web server.

  Furthermore, the CSVCS 110 has multiple versions of encoded slice data bits stored in a storage medium accessible to it and / or on the fly with minimal complexity, depending on the conference context in which it is operating. It is configured to generate such coded slice data bits.

  The system 100 can be advantageously configured to minimize end-to-end delay performance parameters in video conferencing applications. For example, in operation of the system 100, the input video signal to the CSVCS 110 can have different temporal resolutions, or can have a shift between temporal samplings of pictures. Accordingly, the arrival time of the input video signal forming the output video signal at the CSVCS 110 may vary. The CSVCS 110 can be configured to handle changing arrival times by being triggered by the arrival time of the input video signal and generating an output picture. By doing this, a higher temporal resolution of the output video signal can be obtained and the end-to-end delay and other problems caused by the incoming video signal arriving late can be minimized. In addition, the CSVCS 110 can be configured to insert a pre-encoded slice from an accessible storage medium for the portion of the video signal that has no content.

  In one videoconferencing implementation of the present invention, skip pictures (ie, duplicates of all picture content from previous frames), or low bit rate encoded slices are used to represent unchanged output picture content. be able to. In such a videoconference implementation, a receiving videoconference participant uses his / her ref_pic_list_reordering (reordering of reference picture list) syntax structure of ITU-T Recommendation H.264 | ISO / IEC 14496-10. Operating the terminal decoder allows access to the correct reference picture (ie, the picture originally intended to be used as the reference picture by the transmitting participant's encoder). Further, the CSVCS 110 may be suitably configured to change the reordering of the reference picture list. Similar processing or procedures can be used for any other temporal layering structure used.

  In other videoconferencing implementations of the present invention, the input video signal can be encoded with increased temporal resolution. Increasing temporal resolution can be achieved by sending additional pictures that are duplicates of previously encoded pictures (ie, skipped pictures). Independent of picture resolution, the number of bytes for a skipped CIF picture is 2-3 bytes for the picture / slice header and 2-3 bytes for skip signaling for the macroblock. Note that this bit rate is negligible. The encoded representations of the additional pictures can be stored on a storage medium accessible to the sending participant, or can be generated on the fly with minimal complexity and inserted into the bitstream . In the implementation of the SVC embodiment of the present invention, this increase in the number of macroblocks transmitted per second adversely affects the processing power at the receiving endpoint since special measures can be implemented to efficiently handle skip slices. There is no need to give Further, the H.264 MaxStaticMBPS processing rate parameter (referred to as MaxStaticMBPS in ITU-T recommendation H.241) may be used to adjust ITU-T recommendation H.264 | ISO / IEC 14496-10 level signaling. If the input video signal is at a higher temporal resolution, the CSVCS 110 can operate at that higher temporal resolution. CSVCS 110 is further configured to include incoming pictures from the input video signal according to a given schedule and to use non-reference pictures inserted as skip pictures to compensate for incoming jitter. Can do. This compensation can be achieved by replacing the skipped picture with a late-arrived coded picture. In such an implementation, the transmitting participant operates the coder using the ref_pic_list_reordering syntax structure of ITU-T Recommendation H.264 | ISO / IEC 14496-10 to ensure that the correct reference picture (i.e. A reference picture originally intended to be used by the transmitting participant's encoder can be used.

In a further multipoint video conferencing implementation of the present invention, when various participants request different bit rates and different spatial and temporal resolutions in the system 100, the transmitting participant creates video signals at multiple temporal resolutions. can do. FIG. 4 shows an exemplary layered threading time prediction structure 400 for a video signal having multiple time resolution pictures L0, L1, L2. Note that the picture labeled L2 in FIG. 4 is not used as a reference picture for inter prediction. However, the pictures labeled L0 and L0, L1 form a prediction chain. If one of these pictures (L0, L1) is not available for reference at the receiving participant's decoder, the propagation of space-time errors may lead to subjective visual distortion. In the SVC embodiment of the present invention, the picture labeled L2 sent to the CSVCS 110 as an input signal can be marked as “not-used-for-reference”. When sent as a component of a composite output picture by CSVCS, the same L2 picture is marked as “used-for-reference” if the other components of the composite picture are marked as used-for-reference. Need to be marked. This is because of its usefulness in SVC-based video conferencing systems described in International Patent Applications Nos. PCT / US06 / 28365 and PCT / US06 / 28366 where L2 pictures do not need to be marked as used-for-reference Is in contrast. The difference in the use of L2 pictures is that ITU-T Recommendation H.264 | ISO / IEC 14496-10 does not allow a picture to be a composite of reference and non-reference slices, only one or the other. To occur. In accordance with ITU-T Recommendation H.264 | ISO / IEC 14496-10, if multiple input video signals to CSVCS 110 at the same time contain reference and non-reference slices, they are mixed in the same output picture I can't do it. Accordingly, in operation of the system 100, the CSVCS 110 labels and uses the picture L2 as a reference picture to mix non-reference L2 pictures into the output stream. Picture L2 is encoded as a normal coded picture that requires a similar amount of bits as picture L0 or L1, and in the output picture sent to the receiving participant requesting a specific (L2) resolution Can be inserted. For output pictures sent to other receiving participants that do not require a picture labeled as L2, the CSVCS 110 will replace the bits received for the L2 picture from the corresponding input video signal with a skip picture. May be configured to replace with a bit corresponding to. In the aforementioned multipoint videoconferencing scenario, the sending participant operates picture coder L0 and ITU-T Recommendation H.264 | ISO / IEC 14496-10 by operating his encoder using the ref_pic_list_reordering syntax structure. The correct reference picture for L2 (ie, the picture originally intended to be used as a reference by the transmitting participant's encoder) can be utilized. This process can be further extended to L1 pictures and can be used for rate matching and statistical multiplexing purposes, similar to SVCS.

  FIG. 5 shows an exemplary layered structure 500 suitable for spatial scalable prediction, alternating SNR scalable prediction, or a mixture of these predictions that may be used in the operation of system 100. In structure 500, the base layer for prediction is labeled L0. The two reinforcement layers are labeled S0 and Q0. S0 does not depend on Q0 and vice versa. However, there may be other layers that depend on S0 or Q0 throughout the prediction. In the implementation of the SVC embodiment of the present invention, L0 can be a QCIF picture and Q0 can be a 3/2 QCIF picture or a CIF picture. In an exemplary multi-party video conference scenario, only one receiving participant may request a 3/2 QCIF picture and all other participants may request a CIF or QCIF picture. In this scenario in the operation of the system 100, in addition to generating QCIF and CIF pictures, the transmission participant can also generate 3/2 QCIF pictures for overall system efficiency in transmission. Further, in this scenario, the CSVCS 110 can be appropriately configured to transfer the bits necessary to decode the signal at the resolution of each receiving participant. In addition, for improved CSVCS 110 operation, the sending participant can label portions of the bitstream that are not designated or used for prediction with a discardable flag, which is, for example, an international patent application. It is described in PCT / US06 / 28365.

  FIG. 6 shows a further layer picture coding structure 600 that combines a temporal layering structure (FIG. 4) and a spatial scalable layering structure (FIG. 5). The combined structure can be used in the operation of the system 100. In such a case, the system 100 may be configured such that the conferencing entities (i.e., each is a scalable video encoder, a transmitting participant operating the CSVCS 110, and each receiving participant operating a scalable video decoder) It is configured to maintain a bidirectional control channel between each other. The control channel from the sending participant to the CSVCS 110 and from the CSVCS 110 to the receiving participant may be referred to herein as a forward control channel. Conversely, the control channel from the receiving participant to the CSVCS 110 and from the CSVCS 110 to the sending participant may be referred to herein as a reverse control channel. In operation of the system, capability exchange can occur via its control channel prior to actual communication in the conference entity. Capability exchange can include a range of spatial and temporal video resolution ranges supported by each sending participant. The range of capabilities of the sending participant is sent to each receiving participant who can then select or limit his request for video functionality from the sender accordingly.

  Through the reverse control channel, the receiving participant can request a different spatial video resolution than that currently being sent. Similarly, a receiving participant entering a video conference session can request a video with a particular spatial video resolution. In the implementation of the SVC embodiment of the present invention, the CSVCS 110 is configured to meet the receiving participant's request by changing the slice group boundary for the output picture sent to the receiving participant. Depending on the spatial resolution supported by the sending participant's scalable video encoder, the CSVCS 110 supports other spatial resolutions to satisfy the receiving participant's request via its reverse control channel or The scalable video encoder can be notified if it needs to be generated.

  International Patent Application No. PCT / US06 / 28366 is a scalable video conferencing server (SVCS) designed to process coding structures, as described, for example, in International Patent Application No. PCT / US06 / 028365. Note that what is stated here. The SVSC described in the former application has various functions designed for multipoint conferencing based on its ability to manipulate video quality, resolution, and bit rate using scalable video coding. The aforementioned SVSC assumes that the conference participant's endpoint has deployed several decoders to provide the end user with multiple participant views ("screen split display"). However, in some conferencing situations it may be advantageous or necessary to operate only a single decoder in the endpoint. In such a meeting situation, the aforementioned SVSC may further have the complex functions of CSVCS described herein and can be configured or modified to apply. In operation, a modified SVCS can utilize some or all of the unmodified SVCS functions and then utilize the CSVCS 110 functions.

  International Patent Applications Nos. PCT / US06 / 28365, PCT / US06 / 028366, PCT / US06 / 028367, incorporated herein by reference to aid understanding of CSVCS or modified SVCS functionality, With reference to PCT / US06 / 027368 and its association with PCT / US06 / 061815, it is useful to consider here examples of how SVCS functionality can be provided by CSVCS .

  First, referring to International Patent Application No. PCT / US06 / 028366, the similar principle of protecting at least base layer data applied to SVCS operations, described in the referenced application, is similar to the transmission endpoint and CSVCS. Note that it can be applied directly to CSVCS operations in the network connection between the CSVCS and the receiving endpoint, and also between the cascaded CSVCS. Such quality of service support may be provided by CSVCS using means and techniques similar to or the same as those used by SVCS, such as FEC, ARQ (Positive / Negative Acknowledgment), proactive retransmission. When CSVCS creates artificial layers, they are highly reliable in a manner similar to regular layer data (i.e., encoded pictures received from one or more sending endpoints). Can be transmitted over low or unreliable channels. In a manner similar to that of SVCS, CSVCS can adapt to changing network conditions (eg, congestion) by selectively removing enhancement layer data from the composite output video stream. The statistical multiplexing technique used by SVCS can also be used by CSVCS, so the temporal alignment of pictures in the composite output video stream is only a subset of the component pictures received from the sending endpoint Is implemented in such a way that it is allowed to greatly exceed its long-term average size. CSVCS can also characterize audio capabilities using scalable encoded audio streams in a manner similar to SVCS. In the case of audio, there is no equivalent to the concept of a slice group existing in video corresponding to “spatial multiplexing”. Parallel processing for the SVCS audio function is a traditional mix of audio streams. However, this audio mixing can be considered as a further output stage of SVCS audio operation, and thus, for example, algorithms associated with reducing or eliminating the audio clipping effect can still be used by CSVCS as well. Finally, CSVCS can also perform network-related functions such as network address translation and proxy operations in the same way as SVCS.

  It should be noted that SVCS can be deployed with CSVCS in a cascade configuration that links one or more sending and receiving endpoints. If a composite output picture is needed by the receiving endpoint, it may be advantageous to place the CSVCS as the last server in the cascade and place the SVCS at another higher position in the cascade. It should be further noted that the trunking design described in International Patent Application No. PCT / US06 / 028367 can be applied to the CSVCS / SVCS cascade configuration in a similar manner to the SVCS cascade configuration.

  In addition, the jitter reduction technique for SVCS systems described in International Patent Application No. PCT / US06 / 027368 can be applied directly to CSVCS, and any augmented layer data that is not transmitted is a suitable artificial layer according to the principles of the present invention. Can be replaced by data.

  To further assist in understanding CSVCS or modified SVCS functionality, refer to International Patent Application No. PCT / US06 / 061815, here for more examples of how CSVS functionality is provided by CSVCS It is also useful to consider in

  In the context of SVCS, the error resilience, random access, and layer switching techniques described in International Patent Application No. PCT / US06 / 061815 also have direct application in a CSVCS system. When applying these techniques, the connection between the sending endpoint and the CSVCS is transmitted because the characteristic difference between the SVCS and the CSVCS is in the format of its output video signal and not in the nature of the connection. Note that it can be handled in the same way as the connection between the endpoint and the SVCS. For connections between CSVCS and receiving endpoints, consider each slice group data in the CSVCS context to be equivalent to the sending participant's picture data in the SVCS context, and firstly in both cases the packet Similar error resilience and random access protection techniques are applied to CSVCS output packets by noting that only the header data can be different and, second, that additional artificial layer data can be generated by CSVCS. It can be seen that For example, the marking of picture data for reliable transmission in the CSVCS environment can be performed in the same way as in the SVCS environment (for example, by RTP header extension, RNACK with RTCP feedback, etc.). The concept of R picture in the SVCS environment is converted to the concept of R slice group in the CSVCS environment. R-picture caching, the use of periodic intra macroblocks at the encoder of the transmitting endpoint, and fast-forward decoding at the receiving endpoint are also within the context of individual slice groups in the CSVCS environment. Applied. Layer switching techniques useful in SVCS environments can also be used as well. For example, the concept of error recovery or server-based intraframes to support new participants can be applied to slice groups in a CSVCS environment. Like SVCS, CSVCS needs to decode part of the video data coming from the sending participant, especially at least the lowest time level of the base layer, and replay the decoded picture data as intra. It is necessary to prepare for encoding. If the multi-loop coding function is available at the receiving endpoint, layer switching is simplified as much as SVCS because the server does not need to supply intra data.

  Finally, rate control techniques as described in U.S. Provisional Patent Applications 60 / 778,760 and 60 / 787,031, stream thinning techniques as described in U.S. Provisional Patent Application 60 / 774,094, and U.S. Provisional Patent Applications. The multicast SVCS technique described in 60 / 827,469 is also directly applicable to CSVCS. For example, the technique described in Provisional Patent Application No. 60 / 787,031, in which S2 pictures are hidden in the decoder by using appropriately scaled base layer encoded information (modes, motion vectors, etc.) It can be applied to data in a specific slice group in a CSVCS environment. Importantly, a similar concealment effect is to replace the encoded data that instructs the decoder to use base layer information with the S2 picture in CSVCS and insert it at that location in the composite output picture. It can be realized. The advantage of this approach is that the receiving endpoint does not require any special support, so any SVC compliant decoder will work correctly.

  The above examples are illustrative only and are not intended to be exhaustive or limiting. It will be appreciated that any SVCS operation can be implemented in a CSVCS if the composite output video signal generation process is properly handled in accordance with the principles of the present invention.

  Referring again to FIG. 1, in the operation of system 100 and CSVCS 110, it can be seen that individual bitstreams associated with individual participants present in the composite bitstream can be easily extracted from the composite bitstream. Note further. CSVCS 110 may be configured to easily extract these individual bitstreams from the combined bitstream and reinsert them into a different combined bitstream. This configuration of CSVCS 110 allows cascaded CSVCS 110 to provide full remultiplexing of the streams it configures according to participant or downstream server preferences. Thus, such a CSVCS 110 with re-multiplexing capability can fully support extended video conferencing system cascading and distributed operation capabilities, which can be found, for example, in International Patent Application No. PCT / It is described in US06 / 28366.

  System 100 may be further configured to carry signal source identification information or other useful information (e.g., directory information, on-screen help, etc.) to individual participants and / or slice groups in accordance with the present invention. Thus, source identification or other information can be displayed on the participant's display screen. This configuration of the system 100 allows the participant to identify the source of the stream contained in the composite picture. The identification information may include identifying a text string or pre-synthesized slice data displayed beside the slice group corresponding to the individual participant's video signal. For example, the identification information can include a text string that identifies the participant by name (eg, “John Smith”) or by location (Dallas, Room A). In a composite picture, identification information or other conveyed information can be superimposed on individual pixels of each participant, or an unassigned image area that surrounds an image area assigned to each participant (Eg, unallocated area 310, FIG. 3). The identification information may be transmitted as out-of-band or in-band as dedicated data.

  The description of the SVC embodiment of the present invention will be described below with respect to the specific mechanism of synthesis using slice groups, as well as when it is necessary to ensure that the output bitstream is compliant with a scalable video decoder. It relates to generation of layer data.

  In order to assign an input bitstream to a slice group of a composite picture, CSVCS uses a map that describes the layout of the slice group in the composite picture. Specifically, this map, hereinafter referred to as MapOfMbsToSliceGroups, provides an association between a macroblock that includes a composite picture of the output bitstream and a slice group that identifies the input bitstream.

  Referring to FIG. 7, the server has three input streams with resolutions QCIF, CIF, and CIF, respectively, and creates a composite video signal with a picture size of 4CIF from the three input streams. Assume that is desirable. A possible map MapOfMbsToSliceGroups (map 700) is shown in FIG. In the map 700, a slice group 705 assigned 0 and an index corresponds to a QCIF stream, and slice groups 1 and 2 (710 and 720, respectively) correspond to a CIF stream. The unallocated area 730 in the picture also has a slice group index (eg, 3 in this case).

  Note that the map MapOfMbsToSliceGroups (eg, map 700) is not unique, and there can be multiple ways to lay out different slice groups in a composite picture. The specific layout can be obtained by a specific request by the user, and can be calculated automatically by CSVCS or by any other suitable technique. Similarly, specific numbering of slice groups can be obtained using any suitable technique, for example, in one technique by indexing the incoming bitstream and then responding according to that index. Slice groups to be arranged are arranged from the minimum to the maximum, and in raster scanning, from left to right and from top to bottom in the composite picture.

  It may be necessary to send the map MapOfMbsToSliceGroups to participants receiving composite video signals so that they can be properly decoded. Such transmission may be accomplished by incorporating MapOfMbsToSliceGroups into the picture parameter set for the composite signal, with the slice group identification syntax specified in sections 7.3.2.2 and 7.4.2.2 of H.264.

  Specifically, MapOfMbsToSliceGroups can be incorporated into the picture parameter set of the composite video signal by setting as follows.

num_slice_groups_minus1 = NumAssignedAreas;
/

slice_group_map_type = 6;
// (indicates explicit allocation of MB to slice group)

pic_size_in_map_units_minus1 = NumMbs-1;

for (i = 0; i <= pic_size_in_map_units_minus1; i ++)
slice_group_id [i] = MapOfMbsToSliceGroups [i];
However, in the example assignment of FIG. 7, NumAssignedAreas is 3, and NumMbs is 4 times 396 (4 times CIF), ie 1583. Note that instead of type 6 (arbitrary allocation), slice group map type 2 (a set of rectangles and background) can also be used here.

  In order to achieve proper allocation of slices from the input bitstream to the corresponding slice groups in the output bitstream, the CSVCS is an additional map if the slice header syntax is given as specified by the SVC standard. Need to create. This additional map is a correspondence map between the individual block macroblock (MB) index and the composite signal MB index. For example, macroblock index 0 of stream 1 (710 in FIG. 7) corresponds to MB index 22 in the composite picture. For the example given above, if this 2D map is specified as MapMbIndex, MapMbIndex [1] [0] = 22.

  The procedure for assigning slices to slice groups is as follows.

Considering a slice from stream n (for example, n = 0, 1, 2 in the example of FIG. 7), the following steps are performed.

  (a) Parsing the bit stream of the slice header to find the index of the first MB (first_mb_in_slice) in the slice. Let that number be k.

  (b) Use MapMbIndex to determine the corresponding index / position of that MB in the composite picture. That is MapMbIndex [n] [k].

  (c) Remove the emulation_prevention_three_byte syntax element from the NAL unit for the slice according to section 7.3.1 / H.264.

  (d) Replace / replace the existing first_mb_in_slice syntax element with the number MapMbIndex [n] [k].

  (e) Insert the emulation_prevention_three_byte syntax element into the NAL unit again according to section 7.3.1 / H.264.

Steps (a) to (e) described above are repeated for all slices of all input streams that will be included in the composite output picture.

  Still referring to FIG. 7, for an area 730 in a composite picture that is unassigned (ie, not assigned to any of the incoming streams), the CSVCS procedure is as follows.

  For the first or first composite picture, the following steps are performed.

  (a) Create a slice that should contain compressed MB bits in this area. For a given limited set of picture sizes and CSSVC configuration options, this slice may be pre-stored or otherwise calculated online.

  (b) Set the slice type (in the slice header) to 2 (I slice).

  (c) The index of the first MB (set in the slice header) of this slice corresponds to the position of the first unallocated MB in the composite picture (this is 11 in the example above) Should.

  (d) Fill the unallocated area with pixel values that are preferably all equal for efficient encoding. This value is preferably a gray value, i.e. the sample value should be equal to 128 for efficient use of the Intra_16x16_DC prediction mode in the upper left corner MB.

  (e) Here, all MBs are compressed as Intra16x16, and the mb_type parameter in the corresponding MB header is set to this mode. In particular, depending on the specific position of the macroblock, its mode (mb_type) should be selected from:

(i) I_16x16_0_0_0 (vertical prediction from MB above)
(ii) I_16x16_1_0_0 (horizontal prediction from the left MB)
(iii) I_16x16_2_0_0 (DC prediction when there is nothing available in the vicinity)
If CAVLC is used, the preference is given to the mb_type value of I_16x16_0_0_0 or I_16x16_1_0_0. If CABAC is used, the preference is given to I_16x16_2_0_0 and this mb_type value is equal for all macroblocks in the slice, so CABAC can encode it efficiently.

  With continued reference to FIG. 7, the following steps are performed for subsequent pictures in region 730 in unassigned composite pictures.

  (a) Create a slice that should contain compressed MB bits in this area. For a given limited set of picture sizes and CSSVC configuration options, this slice may be pre-stored or otherwise calculated online.

  (b) Set the slice type (in the slice header) to 0 (P slice).

  (c) The index of the first MB (first_mb_in_slice) in this slice should correspond to the position of the first unallocated MB in the composite picture (this is 11 in the example of FIG. 7) .

  (d) Set all macroblock types mb_type equal to P_Skip by setting mb_skip_run (for CAVLC) or by setting mb_skip_flag equal to 1 (for CABAC).

  Note that the composite output picture must have the same value in the temporal_id and dependency_id parameters of the NAL unit header across all slices and slice groups.

  The temporal_id assignment is obtained as follows:

  (a) If the input bitstream is temporally synchronized with respect to its temporal structure, the output picture is assigned the same temporal_id value as that assigned to the corresponding input picture. This is the preferred mode of operation. The output video is manipulated as the input video when it relates to temporal layering and error resilience processing.

  (b) Otherwise (when the input bitstream is not synchronized in time), the assignment of temporal_id to the output picture is implemented to allow all inter prediction structures used in the various input bitstreams It must be. In general (and actually) this results in assigning the same layer number (temporal_id = 0) to all pictures in the output bitstream.

  However, CSVCS can track the temporal dependency structure of various input bitstreams. Since slices (and consequently slice groups) are transmitted in separate packets, an error resilience mechanism, including packet-based retransmission, forward error correction, and generally any technique designed for SVCS, It can be applied to slices in a CSVCS system and thus to slice groups.

  In the CSVCS system, the procedure for assigning dependency_id is as follows.

  (a) If the input bitstream is synchronized so that the same value of dependency_id exists in the input picture for all output pictures in all layers, this value of dependency_id or the shifted value is used.

  (b) Otherwise (if the dependency_id is different), the dependency_id value of the input bitstream is adjusted for each layer of the composite output picture so that they are equal across slice groups. This may require increasing the dependency_id value of some input signals and adding an extra base layer.

  The procedure can be understood with continued reference to the example of FIG. In that example, two CIF signals (slice group 1 710 and 2 720) and one QCIF input signal (slice group 0 705) are combined into a 4CIF output picture. Assume that each of the CIF signals is encoded using spatial scalability and that a base layer with QCIF resolution is provided for each signal. The base layer of the output picture is a CIF picture that includes (in the example) two QCIF base layers (dependency_id = 0) of two CIF enhancement layer input signals (slice groups 1 710 and 2 720, dependency_id = 1) . Further assume that the QCIF input signal (slice group 0 705) does not have a base layer. In that case, its dependency_id value is equal to 0, and if this same signal is used in the same layer as two CIF input signals (slice groups 1 710 and 2 720) in the composite output picture, its value is increased to 1 It is necessary to let Therefore, for the base layer of the composite output picture, for example, a further QQCIF (quarter QCIF) base layer needs to be created by CSVCS. The pictures included in this generated layer can be completely empty, i.e. only contain P_Skip macroblocks and are not used for inter-layer prediction. It is created and added to the composite output picture only for the purpose of allowing an SVC compliant decoder to properly decode the composite output picture.

  If spatial scalability is used, the same ratio of spatial resolution must be used for the slice group corresponding to the input signal. Depending on the spatial resolution ratio, the following steps are performed.

  (a) If one ratio of resolution is present in the input signal (for example, input A: QCIF, CIF, 4CIF, and input B: QQVGA, QVGA, VGA, etc., but the ratio is 2), spatial resolution The ratio between is always matched. These resolutions can then be mixed in all spatial layers of the composite output picture.

  (b) If not (if multiple spatial resolution ratios are present in the input signal), to ensure that the spatial resolution ratio is the same for all layers of the composite output picture Specific layers can be inserted.

  For example, assume that spatial ratios 1.5 and 2 are both present in the input signal intended to be composited. More precisely, referring to FIG. 7, CIF slice group 1 710 input signal has a base layer with 2/3 CIF resolution, CIF slice group 2 720 has a QCIF base layer, and QCIF slice group 0 Assume that has a QQCIF base layer. CSVCS must be configured to work with three spatial layers and their corresponding dependency_id values 0, 1, and 2. For these input signals inserted into the composite output picture by CSVCS, an intermediate artificial (“dummy”) layer needs to be generated. This is shown in FIG. 8, and the same composite picture layout as in FIG. 7 is used, but the lower layer picture with the corresponding layer data of the components of the incoming video signal is also shown. It is necessary to create an artificial intermediate layer 822 having 2/3 CIF resolution for slice group 2 CIF input signal 832, while having resolution 2/3 QCIF for slice group 0 QCIF input signal 830 An artificial intermediate layer 820 needs to be created. Finally, it is necessary to create an artificial base layer 811 with the QCIF resolution for the CIF input signal 831 of slice group 1. An efficient way to encode these artificial layers is to encode all macroblocks using P_Skip mode, and as previously mentioned, an intra that can be represented very efficiently. It is not to use it for inter-layer prediction except from the first picture macroblock that can contain coded gray values.

  The further description herein relates to the synchronization of incoming pictures received from a transmitting endpoint with respect to a composite output signal transmitted to one or more receiving endpoints.

  Since at least one of the incoming frames that are part of the composite output picture is very likely to be used as a reference picture for each of its own streams, CSVCS is used as a reference picture in the output bitstream. Note that it is necessary to flag all of the output composite pictures. In addition, picture data coming from one or more sending endpoints arrives asynchronously on the CSVCS, so it has different frame numbers for the same picture in the incoming bitstream and in the composite output bitstream there is a possibility. This can cause inconsistencies when the composite picture is decoded at the receiving participant, because the proper reference to the previous picture in each slice group may not be properly established.

  Therefore, CSVCS needs to address two issues. First, create composite pictures when frames of different incoming streams arrive at the CSVCS asynchronously in time. Second, make sure that the picture containing the slice group maintains a proper reference for prediction (for the composite signal sent).

  Picture synchronization may be performed by one of the following two techniques.

  1. Buffer the incoming picture using a window corresponding to the picture arrival time for a given sampling frequency in CSVCS that is equal to or higher than the sampling frequency of the input stream with the maximum sampling frequency

  2. Buffer the incoming picture with a window corresponding to the sampling time in CSVCS, which has a period of ΔT, where ΔT is the inverse of the frame rate (FPS) of the composite signal. In order to create a new composite picture that needs to be sent every time sample, the new content that has reached the CSVCS within the last W time unit is examined. The window width W can be selected to be, for example, 1 / FPS.

  The following algorithm illustrates an exemplary CSVCS operation for picture synchronization.

frame_num = 0
for t = ΔT, 2ΔT, ...,
n For each input video stream,
if (new slice data coming for stream n is in (t, tW))
Assign this slice data to the corresponding slice group Apply ref_pic_list_reordering () to each slice in the group Update the map MapOrigInd and MapCompInd for this stream
else
Skip this slice data in the corresponding slice group (using general data) Set frame_num in the slice header for all slices in the group Send this composite picture Update the frame counter: frame_num ++
However, the statement:
Apply ref_pic_list_reordering () to each slice in the group Updating the map MapOrigInd and MapCompInd for this stream is related to the problem of maintaining the correct reference picture data in the composite output picture, here I will explain it.

  The ref_pic_list_reordering () syntax provided in the slice header and the maps MapOrigInd and MapCompInd are used to create an appropriate reference picture list whenever new content arrives at the server. In particular, the CSVCS must always keep track of how the original reference picture index for a slice group (input video stream) is mapped to the output composite picture index. Specifically, whenever new slice data for a stream arrives at CSVCS, the server places its original index at the beginning of a map called MapOrigIndex, and its composite picture index at the beginning of a map called MapCompIndex. Place and simultaneously shift the position of one original entry to the right. In addition, if the length of these maps exceeds a certain length at some point, then the server will simply change the last entry in these two maps whenever a new entry is prepended. Should be discarded. Therefore, these maps operate as a finite capacity stack.

  CSVCS maintains a pair of such maps for each incoming stream. These maps can then be represented as a two-dimensional array, with the first index of the map pointing to the stream index (n = 0, 1, or 2 in the example of FIG. 7), and The size of the index of 2 ranges between zero and some predetermined number (MaxNumRefFrame), which specifies how much past frames you want to track for the incoming stream.

  Assume that new picture slice data for stream n arrives and is placed in the composite picture of the appropriate slice group. For each slice in the group, CSVCS performs the following operations on the slice header data:

// check if reordering has already been done
if (ref_pic_list_reordering_flag_l0 = = 1) do
// This flag can be read from the slice header
index = 0; CurrPic = frame_num;
read first reordering_of_pic_nums_idc from the header
while (reordering_of_pic_nums_idc! = 3) do
if (reordering_of_pic_nums_idc = = 0 || reordering_of_pic_nums_idc = = 1) do
// Short-term reference picture
read abs_diff_pic_num_minus1 from the slice header
if (reordering_of_pic_nums_idc = = 0)
PredOrigPic = MapOrigInd [n] [index]-(abs_diff_pic_num_minus1 + 1)
else
PredOrigPic = MapOrigInd [n] [index] + (abs_diff_pic_num_minus1 + 1)
compIndex = find index (MapOrigInd [n] [:] = = PredOrigPic)
PredCompPic = MapComInd [n] [compIndex];
if (CurrPic> PredCompPic)
abs_diff_pic_num_minus1 = CurrPic-PredCompPic-1;
write reordering_of_pic_nums_idc = 0 in the slice header;
// replace the existing reordering_of_pic_nums_idc value
else
abs_diff_pic_num_minus1 = PredCompPic-CurrPic-1;
write reordering_of_pic_nums_idc = 1 in the slice header;
// replace the existing reordering_of_pic_nums_idc value
write abs_diff_pic_num_minus1 in the slice header;
index ++; // move to next entry
CurrPic = PredCompPic;
else if (reordering_of_pic_nums_idc = = 2) do
read long_term_pic_num from the slice header
index_long_term = find (MapOrigInd [n] [:] = = long_term_pic_num)
write MapCompInd [n] [index_long_term] in the slice header
read next reordering_of_pic_nums_idc from the slice header
end (while (reordering_of_pic_nums_idc! = 3))
else
// (ref_pic_list_reordering_flag_l0 = = 0) // no previous reordering requested
set ref_pic_list_reordering_flag_l0 (= 1) in the slice header
CurrPic = frame_num;
for index = 0, ..., MaxNumRefFrame-1
if (CurrPic> MapCompInd [n] [index])
abs_diff_pic_num_minus1 = CurrPic-MapCompInd [n] [index]-1;
write reordering_of_pic_nums_idc = 0 in the slice header;
else
abs_diff_pic_num_minus1 = MapOCompInd [n] [index]-CurrPic-1;
write reordering_of_pic_nums_idc = 1 in the slice header;
write abs_diff_pic_num_minus1 in the slice header;
CurrPic = MapCompInd [n] [index];
write reordering_of_pic_nums_idc = 3;
end (of the if-else-check on existing ref_pic_list_reordering_flag_l0 flag)
Note that the operations described here assume that there are only P slices. For B slices, a similar procedure is applied in the slice header (setting ref_pic_list_reordering_flag_11 in the slice header) with the ref_pic_list_reordering () syntax. Furthermore, it should be noted that the index of the reference picture is stored from the latest one that arrived at the server (index = 0) to the farthest one that arrived in the past (index = MaxNumRefFrame-1).

  After new picture data arrives from the sending participant's video stream, the CSVCS registers its index (if it is a reference picture) in the maps MapOrigInd and MapCompInd so that the picture can be used in subsequent operations. There is a need. Specifically, the following operations are performed. First, the CSVCS extracts an original frame number (“orig_frame_num”) from an arbitrary slice header of new picture data for the stream n. MapOrigInd and MapCompInd are then updated as follows (stack insertion):

for index = MaxNumRefFrame-1, ..., 1
MapOrigInd [n] [index]) = MapOrigInd [n] [index-1])
MapCompInd [n] [index]) = MapCompInd [n] [index-1])
MapOrigInd [n] [index-1]) = orig_frame_num;
MapCompInd [n] [index-1]) = frame_num;
If the input video signal received from the transmitting endpoint is compatible in time coding dependency structure, CSVCS can perfectly align them even if the frame rates are different. For example, the threaded picture coding structure of International Patent Application No. PCT / US06 / 028365 is used, and pictures from two input participants, one of which is a total of 30 frames per second L0, Assume that L1 and L2 are combined in three layers, and the second is combined in two layers of L0 and L1 in total 15 frames per second. CSVCS proceeds to create an artificial temporal layer L2 ′ for the second participant to build a composite output picture, so the L0, L1, and L2 pictures of the first participant are Each can be combined into the same output picture as the L0, L1, and L2 ′ pictures of the second participant. This makes it possible to save the threading pattern in the composite output video picture.

  CSVCS can also perform spatial resolution switching, upsampling, and input signal shifting in the composite output video signal.

  Upsizing (by one layer) is realized by sending intra macroblocks in the I slice to all layers, ie to the corresponding slice group. All intras are required because the value of dependency_id needs to be adjusted as described above, and motion compensation across different dependency_id values is not allowed in SVC compliant decoders. In that case, the corresponding slice group covers a wider area of the composite output picture. Other slice groups in the composite output picture may need to be shifted for that purpose. Intra data can be calculated by the CSVCS itself, in which case at least the lowest time level of the base layer needs to be decoded, or intra data can be created by endpoints upon request from the CSVCS it can. Downsizing is performed in the same manner as upsizing.

  Upsampling of a specific video signal received from the sending endpoint can be performed by inserting an additional enhancement layer generated by CSVCS, in which case all macroblocks are macros with lower layer content. It is simply duplicated from the block and encoded. Including additional layers in the participant's video signal may require reorganization of the overall scalability structure of the composite output picture using the techniques described herein.

  The shifting of the input signal is preferably performed by a plurality of macroblocks. The recipient can shift the picture using a user interface request (eg, dragging the mouse). CSVCS adjusts the shift by adjusting the motion vector accordingly (add / subtract an integer multiple of 16-sample position). Note that motion vectors are usually encoded differentially, and in this case it is most likely that only the value of the first motion vector needs to be changed.

  While what has been considered as a preferred embodiment of the present invention has been described, those skilled in the art will be able to make further changes and modifications thereto without departing from the spirit of the present invention and are included in the spirit of the present invention. It will be understood that all such changes and modifications are intended to be claimed.

  It will also be appreciated that the systems and methods of the present invention can be implemented using any suitable combination of hardware and software. Software (ie, instructions) for implementing and operating the aforementioned systems and methods includes firmware, memory, storage devices, microcontrollers, microprocessors, integrated circuits, ASICS, online downloadable media, and other Available media may be provided on computer readable media that may include, without limitation ,.

1 is a schematic diagram of a video conference system in which a complex scalable video conference server (CSVCS) is configured to deliver scalable video and audio data from an endpoint transmitter to a client receiver in accordance with the principles of the present invention. FIG. FIG. 3 is a block diagram illustrating exemplary partitioning of output video pictures into slice groups in accordance with the principles of the present invention. FIG. 4 is a block diagram illustrating exemplary assignment of input video to various slice groups in an output video picture in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary layered picture coding structure for the temporal layer, in accordance with the principles of the present invention. FIG. 3 is a block diagram illustrating an exemplary layered picture coding structure for an SNR or spatial enhancement layer in accordance with the principles of the present invention. FIG. 4 is a block diagram illustrating an exemplary layered picture coding structure for a base layer, temporal enhancement layer, and SNR or spatial enhancement layer, having different prediction paths for the base layer and enhancement layer, in accordance with the principles of the present invention. FIG. FIG. 3 is a block diagram illustrating exemplary partitioning of output video pictures into slice groups in a slice group based compositing process in accordance with the principles of the present invention. FIG. 4 is a block diagram illustrating an exemplary structure for the construction of an artificial layer in the synthesis of an output video signal transmitted from a CSVCS, combined with different spatial scalability ratios, in accordance with the principles of the present invention.

Claims (35)

A multi-endpoint video signal conferencing system for conducting a videoconference between a plurality of endpoints via a communication network,
For at least one communication channel comprises a conference bridge, each being linked to at least one receiving endpoint Contact and at least one transmitting endpoint,
Wherein said at least one transmitting endpoint, a single layer encoding formats, and using a coding format selected from the group consisting of scalable video coding format, and transmits the digital video signal encoded,
Wherein the at least one receiving endpoint can in decoding at least one digital video stream coded in a scalable video coding format,
The conference bridge is configured to combine an input video signal received from the at least one transmission endpoint into a composite signal that is a single composite encoded digital video output signal , the combination comprising (i) Allocating a portion of the region of the composite signal to the at least one transmission endpoint to be included in the composite signal; and (ii) one or more higher than that to be used for the picture to be synthesized The transmission endpoint corresponding to resolution and / or data that does not need to be decoded at the resolution to be used for the picture to be synthesized and / or at least one transmission endpoint not included in the composite signal Discarding any input video signal data received from (iii) the at least one transmission entity not discarded. Changing the header information of the video signals received from the point made by,
The conference bridge is further configured to forward the single composite encoded digital video output signal to the at least one receiving endpoint. The at least one receiving endpoint is capable of decoding video encoded in an H.264 SVC scalable video encoding format, and the composite for a transmission endpoint intended to be included in the composite signal; the allocation of part of the area of the output picture, the is performed by defining a slice group map in the picture parameter set No. double Goshin, each transmitting endpoint corresponds to one slice group,
The allocation of a portion of a region of the composite signal to the transmission endpoints, by transmitting the picture parameter set to said at least one receiving endpoint, the transmitted to the at least one receiving endpoint to claim 1 The described system. The system of claim 2 , wherein the picture parameter set is configured to be carried in-band or out-of-band to the one or more receiving endpoints. The composite signal is
When at least one of the input pictures received from the transmission endpoint included in the composite signal is flagged as used-for-reference (used as a reference), as used-for-reference, Also, if all of the input pictures received from the sending endpoint included in the composite signal are flagged as not-used-for-reference (not used as a reference), not-used- Further configured to be flagged as a for-reference,
When the composite signal is flagged as used-for-reference, a reference frame reordering command is used to ensure proper operation of a reference picture buffer at the one or more receiving endpoints. The system of claim 2 , wherein the system is inserted into the slice of a picture subsequently received from the transmitting endpoint prior to its transmission to one receiving endpoint. The NAL extension header for the SVC of the NAL unit of the composite signal is
The same dependency_id value corresponding to the highest scalable coding layer present in the composite signal is used for the NAL unit of the composite output picture and is the same for the next lower layer NAL unit. Is set to use the next lower dependency_id value, and
temporal_level (time level) is
If the incoming pictures from the at least one transmission endpoint are combined such that the time levels are synchronized, the same temporal_level value is for the NAL unit corresponding to the highest scalable coding layer For the next lower layer, the next lower temporal_level value is used, and the incoming picture from the at least one transmission endpoint is combined so that the time level is synchronized The system of claim 2 , wherein a value of 0 is used for all NAL units of the composite output picture if not. The system of claim 1 , wherein the assignment of a particular portion of a region of the composite output video picture to a video signal of a particular transmission endpoint by the conference bridge is predefined. The assignment of a particular portion of the region of the composite signal to a video signal of a particular transmission endpoint;
A request from the receiving endpoint for a specific spatial resolution;
A request from the receiving endpoint to determine a specific spatial position in the composite signal ;
Based on the combinations thereof, it is dynamically performed by the conference bridge system according to claim 1. For a specific transmitting endpoint video signal, the conference bridge assignment of specific portions of the area of the composite signal, the taking into account at least one receiving endpoint decoding capability or resolution preferences, by the conference bridge The system of claim 1 , wherein the system is implemented. The conference bridge, is in La,
Sending pre-encoded slice data instructing the at least one receiving endpoint to repeat data from a previous picture;
In order to ensure that proper reference picture selection is performed for subsequent pictures, a reference picture list reorder command is sent to the input video signal before being sent to the at least one receiving endpoint. The system of claim 1 , wherein the system is configured to accommodate when a new picture of the input video signal does not arrive in time for transmission by inserting into the picture header of a subsequent picture. The conference bridge is further configured to decode at least the lowest time level of the lowest spatial and quality resolution of the video signal received from the at least one transmission endpoint, and the conference bridge further If the composite picture configuration for an existing receiving endpoint needs to be changed, so as to generate an intra coding for the video signal of the affected sending endpoint, and the intra coding the place of the corresponding coded picture data received from the transmitting endpoint, system according constituted, in claim 1 to be sent to the receiving endpoint. A plurality of conference bridges in a cascade configuration, wherein at least one conference bridge that is not the last in the cascade configuration ,
Forward the composite coded picture received from the previous conference bridge in the cascade configuration to another conference bridge , or the composite coded picture received from the previous conference bridge in the cascade configuration The system of claim 1, configured to disassemble and re-synthesize using a different layout before transferring them to other conference bridges . The conference bridge is configured to receive a request from the receiving endpoint to determine the desired spatial resolution of the output video signal, according to claim 1, wherein the system. The system of claim 1, wherein the conference bridge is configured to include source identification information or other information for display via one of an in-band bitstream and an out-of-band bitstream. 30
The conference bridge, the information conveyed Source Identification information or other, (1) pixel of the partial regions of the composite signals allocated to each participant in signal, (2) the transmission participants of the video The system of claim 1, wherein the system is configured to overwrite one of pixels of a portion of the region of the composite signal that is not assigned to any of the signals. The conference bridge further comprises:
Be statistically multiplexed video signals from a plurality of transmitting endpoints, and the order double Goshin shift the average larger video picture in No. (stagger), the composite video signal received from the transmitting endpoint The conference system of claim 1 , wherein the conference system is configured to comply with a bandwidth condition by at least one of synchronizing transmissions. The conference bridge further comprises:
The encoded picture data received from the at least one transmission endpoint is replaced with the encoded data indicated by the at least one reception endpoint so as to duplicate corresponding pixel data from a previous picture. that change the bit rate of the composite output signal, and the output bit rate, so that it can match the desired properties, the replacement the encoded data is configured to transmit a conference system according to claim 1. The conference of claim 1, wherein the conference bridge is further configured to generate an input video signal by generating artificial layer data for at least one of the transmitted endpoint video signals. system. A method for conducting a video conference between a plurality of endpoints via a communication network,
At least one communication channel, comprising the steps of using the conference bridge, each being linked to at least one receiving endpoint Contact and at least one transmitting endpoint,
Transmitting encoded digital video from at least one transmission endpoint in an encoding format selected from the group consisting of a single layer encoding format and a scalable video encoding format;
In the conference bridge, the step of combining the input video signals received from the transmitting endpoint, encoded in a scalable video coding format, into a composite signal which is a single double Gode digital video output signal,
Transferring the single composite encoded digital video output signal to at least one receiving endpoint ;
Synthesizing into the composite signal comprises:
Assigning a portion of the region of the composite signal to each transmitting endpoint intended to be included in the composite signal;
A resolution higher than that intended for the synthesized picture, data that does not need to be decoded at the resolution intended for the synthesized picture, and a transmission endpoint that is not included in the composite signal Discarding input video signal data received from the transmitting endpoint corresponding to one;
Changing the remaining data of the input encoded video signal by changing header information to form proper data of the composite output video signal;
Including methods. The at least one receiving endpoint is capable of decoding video encoded in an H.264SVC scalable video encoding format, and a portion of the composite output picture region is included in the composite signal Said step of assigning to each sending endpoint intended to
Defining a slice group map in a picture parameter set of the composite output signal, wherein each transmission endpoint corresponds to one slice group;
The allocation of parts of the regions of the composite signal for a particular transmitting endpoint, in order to transmit to the at least one receiving endpoint, by transmitting the picture parameter set to said at least one receiving endpoint you run, the method according to claim 18. The method of claim 18 , further comprising conveying the picture parameter set in-band or out-of-band to at least one receiving endpoint. When at least one of the input pictures received from the transmission endpoint included in the composite signal is flagged as used-for-reference, it is used as a used-for-reference, and the composite output picture If all of the input pictures received from the sending endpoint are flagged as not-used-for-reference, flag the composite signal as not-used-for-reference Further comprising:
When the composite signal is flagged as used-for-reference, a reference frame reordering command is used to guarantee proper operation of the reference picture buffer at the at least one receiving endpoint. The method of claim 18 , wherein the method is inserted into the slice of a picture subsequently received from the transmitting endpoint prior to its transmission to the receiving endpoint. The same dependency_id value corresponding to the highest scalable coding layer present in the composite signal, the being used for the NAL units of the composite signal, and with respect to the NAL unit of a lower layer to the next, the same Setting a NAL extension header for the SVC of the NAL unit of the composite output picture so that the next lowest dependency_id value is used,
If the incoming pictures from the at least one transmission endpoint are combined so that the time levels are synchronized, the same temporal_level value is given to the NAL unit corresponding to the highest scalable coding layer. For the next lower layer, the next lower temporal_level value is used, and the incoming picture from the at least one transmission endpoint is synchronized in the time level. 19. The method of claim 18 , further comprising setting the temporal_level such that a value 0 is used for all NAL units of the composite signal if not synthesized. The method of claim 18 , wherein the step of assigning a particular portion of the region of the composite signal to a video signal of a particular transmission endpoint by the conference bridge is predefined. The parts of the regions of the composite signal, said step of assigning a video signal of a specific transmitting endpoint,
A request from the receiving endpoint for a specific spatial resolution;
A request from the receiving endpoint to determine a specific spatial position in the composite output picture;
19. The method of claim 18 , wherein the method is dynamically performed by the conference bridge based on a combination thereof. Wherein the parts of the regions of the composite signal during said step of assigning to the video signal of each transmitting endpoint, further comprising said at least one receiving endpoint decoding capability or contemplates step resolution preferences, claim 18 The method described in 1. If the conference bridge, a new picture of the entering force the video signal does not arrive in time for the time for transmission,
Sending pre-encoded slice data instructing the at least one receiving endpoint to repeat data from a previous picture;
In order to ensure that proper reference picture selection is performed for subsequent pictures, a reference picture list reorder command is sent to the input video signal before being sent to the at least one receiving endpoint. Inserting into the picture header of subsequent pictures;
By the execution, configured as response method according to claim 18. The conference bridge is further configured to decode at least the lowest time level of the lowest spatial and quality resolution of the video signal received from the at least one transmission endpoint, the method further comprising:
Generating, in the conference bridge , intra coding for the video signal of the affected sending endpoint if the composite picture configuration for an existing receiving endpoint needs to be changed;
19. The method of claim 18 , comprising transmitting the intra coding to the receiving endpoint instead of corresponding encoded picture data received from the transmitting endpoint. When the communication network comprises a plurality of conference bridges in a cascade configuration,
At least one conference bridge that is not the last in the cascade configuration, optionally to other conference bridges without processing composite coded pictures received from a previous conference bridge in the cascade configuration Transferring, or decomposing the composite coded pictures received from a previous conference bridge in the cascade configuration and recombining with different layouts before transferring them to other conference bridges. The method of claim 18 further comprising: 19. The method of claim 18 , further comprising receiving a request from a receiving endpoint for a desired spatial resolution of an output video signal at the conference bridge . The method of claim 18 , further comprising including source identification information and other information via one of an in-band bit stream and an out-of-band bit stream sent by the conference bridge . In the conference bridge, the information carried in the Source Identification information or other, (1) pixel of the partial regions of the composite signals allocated to each participant in the signals, (2) the transmission participant's video The method of claim 18 , further comprising overwriting one of the pixels of the portion of the composite signal region that is not assigned to any of the signals. Using the conference bridge ,
Statistically multiplexing video signals from multiple transmission endpoints;
In order to shift larger than average video pictures in the composite output video signal, the method further comprising the step of complying with a bandwidth condition by at least one of synthesizing and transmitting the video signal received from the transmission endpoint. The method of claim 18 comprising. The step of responding to bandwidth conditions using the conference bridge further comprises:
The encoded picture data received from the at least one transmission endpoint is replaced with the encoded data indicated by the at least one reception endpoint so as to duplicate corresponding pixel data from a previous picture. Changing the bit rate of the combined composite output signal;
19. The method of claim 18 , comprising transmitting the replacement encoded data so that the output bit rate can match a desired characteristic. 19. The conference bridge of claim 18, further configured to generate an input video signal by generating artificial layer data for at least one of the video signals of the transmitting endpoint. Method.   A computer-readable medium having a program for causing a computer to execute the method according to any one of claims 18 to 34.

Images