KR20070021284A - Method, apparatus, and system for enhancing robustness of predictive video codecs using a side-channel based on distributed source coding techniques - Google Patents

Abstract

A method, apparatus, and system for providing distributed source coding techniques that improve data coding performance, such as video data coding, when channel errors or losses occur. Errors in the reconstruction of the data are eliminated or reduced by sending extra information. The correlation between the predicted sequence and the original sequence can be used to design the codebook and find the corset required to represent the original image. This information may be transmitted on another channel, namely the second channel.

Data encoding, data decoding, encoder, decoder.

Description

METHOD, APPARATUS, AND SYSTEM FOR ENHANCING ROBUSTNESS OF PREDICTIVE VIDEO CODECS USING A SIDE-CHANNEL BASED ON DISTRIBUTED SOURCE CODING TECHNIQUES}

35 Claims of priority claims under U.S.C §119

This application is entitled “ Method, Apparatus, and System for Improving the Robustness of Predictive Video Codec Using Side-Channel Based on Distributed Source Coding Techniques”, filed June 1, 2004, Claims priority to US provisional patent application 60 / 576,193, assigned to the applicant and incorporated herein in its entirety.

Reference to Joint Application for this Application

This application is entitled "Method, Apparatus and System for Encoding and Decoding for Side Information for Multimedia Transmission", filed March 24, 2003, assigned to the applicant of this specification, US Patent Application No. 10 / 396,982, which is incorporated by reference.

Background

1. Technology Field

The present application is generally related to multimedia by a network, which may be in error, and more particularly to video coding and decoding techniques used for multimedia by a wireless network.

2. Background Technology

Current video compression techniques, such as ITU-T H.263, H.263 +, H263 ++, ISO MPEG-4 and JVT H.264, are very efficient in the compressed video domain. The ability to achieve good visual quality at relatively low bit rates has led to an increase in popularity for video applications in emerging multimedia applications for bandwidth limited channels, such as wireless channels. However, the prediction characteristics of the compression techniques used in current video codecs make the compressed video bit streams very vulnerable and easily degraded due to packet loss in the channel.

Channel errors or packet loss can result in a loss of synchronization between the video encoder and the decoder. More specifically, the locally decoded copy of the reconstructed video frame at the encoder may not match the corresponding reconstructed video frame at the decoder. Because there is a drift between the receiver's decoder and the sender's decoder, this loss of synchronization between the video encoder and the decoder is also sometimes called "drift". Drift can cause large losses in the decoded video quality.

The reduction in decoded quality due to drift is a direct result of the prediction based coding underlying current video codecs. In other words, to decode the current video frame now, all previous video frames need to be correctly reconstructed. This problem is exacerbated in wireless channels, where packet losses are much more frequent than wire-line networks and tend to burst.

Re-transmission schemes or Forward Error Correction (FEC) schemes, such as Automatic Repeat Request (ARQ), handle drift problems between the encoder and decoder and thereby ensure It was used to mitigate the loss. However, ARQ or FEC schemes (or a combination of both) have not proven effective because the latency constraints of video applications prevent the use of such schemes. In addition, FEC basic schemes cannot guarantee that data will be received, and if not, the drift will continue until the next intra-frame ¹ is received. (Intra-frame ¹ is used by video encoders to break the prediction loop to stop drift.)

Wyner-Ziv theory of AD Wyner and J.Ziv "rate distortion function for source coding with side information at the decoder (The rate distortion function for source coding with side information at the decoder)" IEEE Trans . Inf . Theory , vol. 22, pp. 1-10, January 1976 issue. This theory solves the problem of source coordination with side information. Consider two correlated sources X and Y. Only when the decoder has access to source Y the encoder needs to compress the source X. The mean squared error is the distortion magnitude and X = Y + N, and the ratio-distortion performance for X coding is the same whether or not the encoder has access to Y.

Recently, based on the principle of source coding with side-information between the Wyner-Ziv frameworks, junction source-channel coding techniques have been proposed to solve the problem of drift. For example, Allerton of R. Puri and K. Ramchandran Conference on Communication , Control and "PRISM: A New Robust Video Coding Architecture based on Distributed Compression Principles" in Computing , 2002; IEEE of A. Sehgal, A. Jagmohan, and N. Ahuja Trans . on Multimedia , vol. 6, pp. 249-258, 2004, "Wyner-ziv Coding of Video: An Error-Resilient Compression Framework"; Proc . In A. Aaron, S. Rane, R. Zhang, and B. Girod . IEEE Data Compression Conf . , "Wyner-Ziv Coding of Video: Applications to compression and error resilience" in 2003; And Proc . By A. Aaron, S. Rane, D. Rebollo-Monedero, and B. Girod . IEEE Int . Conf . Image Proc . See, "Systematic Lossy Forward Error Protection for Video Waveforms," in 2003.

Allerton by R. Puri and K. Ramchandran Conference on Communication , Control and "PRISM: A New Robust Video Coding Architecture based on Distributed Compression Principle" in Computing , 2002; And Proc . In A. Aaron, S. Rane, R. Zhang, and B. Girod . IEEE DATA Compression Conf . Codecs described in "Wyner-Ziv Coding of Video: Applications to compression and error resilience" in 2003, are full -fledged) Video codec.

Meanwhile, IEEE of A. Sehgal, A. Jagmohan, and N. Ahuja Trans . on Multimedia , vol. 6, pp. 249-258, 2004, "Wyner-ziv Coding of Video: An Error-Resilient Compression Framework"; And Proc . By A. Aaron, S. Rane, D. Rebollo-Monedero, and B. Girod . IEEE Int . Conf . Image Proc . The codecs described in "Systematic Lossy Forward Error Protection for Video Waveforms of Video Waveforms" in 2003 maintain a predictive coding framework but transmit extra information to mitigate the effects of drift.

Despite these developments, in addition to enabling robust video delivery over wireless channels by helping to mitigate the effects of packet loss on the compressed video bit stream transmitted for such wireless channels, You need skills that satisfy your needs.

Can work with existing compression techniques.

Low processing latency.

• Quickly remove / reduce drift between video encoder and decoder.

Require a lower bit rate compared to that used by the video codec.

Summary of Embodiments

Embodiments disclosed herein satisfy this need by providing distributed source coding techniques that improve video coding performance when channel errors or packet losses occur.

The following terms are used.

Main stream: video data compressed using predictive codecs

● Main channel: Logical channel through which the main stream is sent

● WyZER stream: Wy ner Z iv E rror R esilience stream composed using the distributed coding principles

● Secondary channel: Logical channel through which the WyZER stream is sent

● Main Channel Encoder: the encoder used to encode the main stream

● main channel decoder: decoder used to decode the main stream

● WyZER encoder: Encoder used to encode the WyZER stream

● WyZER Decoder: Decoder used to decode WyZER streams

Side information: Decoded main stream data used by the WyZER decoder to reconstruct a frame at the receiver.

DSC: distributed source coding

The correlation between the reconstructed video frame at the decoder with the channel error and the video frame at the encoder is evaluated. In one embodiment, the correlation is calculated in the discrete cosine transform (DCT) domain. The interrelationship was filed on March 24, 2003, entitled "Method, Apparatus and System for Encoding and Decoding Side Information for Multimedia Transmission". "Can be used to find the coset described in US Patent Application No. 10 / 396,982.

Embodiments of the present invention may help to mitigate the drift problem in conventional video codecs by the WyZER stream as an error recovery stream. Embodiments can help eliminate or reduce drift by sending extra information. For example, in one embodiment, an auxiliary channel technique can be used in which the WyZER stream can be used to improve video quality and stop drift. The interrelationship between the predicted sequence and the original sequence can be used to find the corsets required to design the codebook and represent the original image. This information may be transmitted for another channel, namely the second channel. In another embodiment, the number of required corsets may be communicated to the WyZER decoder for the second channel in the form of header information.

In one embodiment, using DSC, another technique of the video sequence is at a lower rate relative to the second channel that can be used to correct errors caused by channel losses in the reconstructed video frame at the receiver. Can be sent. The bit stream transmitted for the second channel can be decoded with the help of the video frame reconstructed by the prediction decoder. As a result, the influence of error propagation can be mitigated.

In one embodiment, if the decoder can only decode main stream data, the decoder will discard the additional information transmitted on the auxiliary channel. If the decoder decodes all of the streams, the decoder first decodes the main stream data and then uses this information to decode the WyZER stream data and give the user the final reconstructed image.

Brief description of the drawings

1 is a block diagram of a wireless communication system.

FIG. 2 (a) is a diagram illustrating a peak signal with noise ratio (PSNR) to frame number for a video sequence encoded with an H.263 + encoder and DSC techniques.

2 (b) is a diagram of PSNR vs. frame number for a video sequence encoded with H.263 + encoder, FEC, and DSC techniques.

FIG. 2 (c) is a diagram of PSNR versus frame number for another video sequence encoded with an H.263 + encoder, FEC, and DSC techniques.

FIG. 2 (d) is a diagram of PSNR versus error probability for a video sequence named “Stefan” encoded with H.263 + encoder, FEC and DSC techniques.

2 (e) is a diagram of PSNR vs. error probability for a video sequence named “Football” encoded with H.263 + encoder, FEC and DSC techniques.

2 (f) is a diagram of the average burst length to PSNR ratio for a “Football” video sequence encoded with FEC and DSC techniques.

3 is a block diagram illustrating a portion of a wireless communication system that can implement DSC techniques.

4 is a flow diagram illustrating an embodiment for determining intra-frame bit allocation for a side-channel encoder.

5 is a flow diagram illustrating an embodiment for determining the maximum number of divisions of a source codebook that can be used by the side-channel encoder.

6 is a block diagram of a WyZER encoder.

7 is a block diagram of a WyZER decoder.

details

Current video coding standards use a prediction framework to achieve good compression performance for video signals. Examples of such compression schemes include MPEG-4, H.263 + and H.264. However, when such compressed bit streams are transmitted for channels that may be in error, video quality is severely compromised, as demonstrated by using an objective PSNR metric in FIGS. 2 (a)-(c). do. Examples of channels that may be in error include cdma2000®, WCDMA, GSM and other emerging wireless networks. In the present invention, video data compressed using prediction codecs is called a "main stream" and a logical channel through which the main stream is transmitted is called a "main channel".

Examples of mitigating the negative effects of channel errors include retransmission of transmission units with errors (RLP retransmission, hybrid ARQ, etc.), error recovery packetization (data where the more significant bits in the transmission units are fed with greater error protection). Distribution, data interleaving, etc.), and bit stream syntax that limits the range of lost source data (resynchronization markers, reversible variable length coding, etc.).

In the present invention, individual bit streams are created for use with the main stream to improve error recovery. This stream is constructed using Wyner-Ziv coding of video data based on distributed source coding principles. We call this stream Wy ner Z ev er R e silience stream (WyZER stream). The WyZER stream is transmitted on a logical channel, " second channel, " different from the primary channel used for video codecs such as MPEG-4, H.263 + and H.264. The video frame reconstructed by the predictive video coder serves as side-information to the WyZER decoder for the purpose of decoding the WyZER stream.

According to another aspect of the invention, the techniques are described in R. Zhang, SL Regunathan, and K. Rose Proc . 33 rd Ann . Asilomar Conf . on Sig . Syst . Comp . Discrete Cosine Transformation (DCT) using some of the concepts disclosed in 1999, "Optimal intra / inter mode switching for robust video communication over Internet." It is provided to evaluate the interrelationship between the frame itself and the misconstructed frame in the domain.

According to another aspect of the invention, the WyZER encoder operates in the DCT domain to take advantage of the energy density characteristics of the DCT. Depending on the rate available for side-channel and correlation evaluation, as many DCT coefficients as possible, starting from DC and low-frequency coefficients, are encoded.

If the customer receives in both the main stream and the WyZER stream but can only decode the main stream (i.e. has only a predictive decoder, such as an MPEG-4 or H.263 + or H.264 decoder), then the customer only Will decode and ignore the WyZER stream. A customer with both a predictive decoder and a WyZER can decode all of the bit streams and thus receive a higher quality decoded video than a customer with only a predictive decoder.

By A. Sehgal, A. Jagmohan, and N. AhujaIEEE Trans . on Multimedia, vol. 6, pp. 249-258, 2004, "Wyner-ziv Coding of Video: An Error-Resilient Compression Framework"; And by A. Aaron, S. Rane, D. Rebollo-Monedero, and B. GirodProc . IEEE Int . Conf . Image Proc .Algorithm in “Systematic Lossy Forward Error Protection for Video Waveforms of Video Waveforms,” in 2003, also uses wrong frames reconstructed by the predictive coder as side information in the auxiliary channel decoder, There are many important differences between the disclosed techniques and the present invention. For example, A. Sehgal, A. Jagmohan, and N. AhujaIEEE Trans. on Multimedia, vol. 6, pp. 249-258, 2004, in "Wyner-ziv Coding of Video: An Error-Resilient Compression Framework," authors write "peg" "Display some frames as frames. These techniques allow error propagation to occur from one peg frame to the next. In each peg frame, some extra information is sent from the encoder to the decoder, which allows the decoder to correct the errors in the peg frame.

In contrast, aspects of the present invention stop or reduce drift, ie errors, as soon as possible. In this way, it becomes possible to maintain a rather stable quality, which is important in visual terms.

Meanwhile, A. Aaron, S. Rane, D. Rebollo-Monedero, and B. Girod's Proc . IEEE Int. Conf . Image Proc . In 2003, "Systematic Lossy Forward Error Protection for Video Waveforms," an independent description of the video sequence is transmitted in each frame, but with coarser quantization. do. This coarser technique can be coded using the Wyner-Ziv framework and decoded with the help of error frames in a decoder that acts as side-information. Thus, the method of the document does not attempt to stop the drift because the method may at best recover poor technology. The techniques described in this document also work in the pixel domain and do not work in the DCT domain, thus giving up taking advantage of spatial interrelationships.

For source coding with side information, it is necessary for the decoder to be able to decode an encoded codeword susceptible to correlation noise N between the source and side-information, while the encoder encodes the source under distortion constraints. It is necessary to do. Although the results demonstrated by Wyner and Ziv are inherently asymptotic and non-constructive, many interpretative methods to solve this problem have since been proposed, where the source codebook is interrelated. It is distributed to the corsets of the channel code that match the noise N. The number of distributions or corsets depends on the statistics of N.

1 is a system block diagram illustrating aspects of encoded video for a wireless channel according to one embodiment. The system includes a predictive encoder 10, comprising a frame buffer, which communicates a bit stream to the predictive decoder 20 over a wireless primary channel 25, and an auxiliary channel (WyZER) decoder over a wireless auxiliary channel 45. And an auxiliary channel (WyZER) encoder 30 that communicates the bit stream to 40.

를 재구성한다. 예측 디코더 (20) 재구성된 비트 스트림,

는 무선 채널을 통해서 비트 스트림의 전송에서의 오차들에 의해서 영향을 받을 수도 있다. 재구성된 신호

는 최종 재구성으로써

를 출력하는 WyZER 디코더 (40) 에 대한 사이드 정보로써 역할을 한다.In FIG. 1, predictive decoder 20 receives a bit stream from predictive encoder 10 over a wireless channel 25 and is a representation of a video signal X input to encoder 10.

Reconstruct Predictive decoder 20 reconstructed bit stream,

May be affected by errors in the transmission of the bit stream over the wireless channel. Reconstructed signal

Is the final reconstruction

It serves as side information for the WyZER decoder 40 that outputs.

The input video signal X is, in one embodiment, the current video signal encoded by the predictive encoder 10, the predictive encoder 10 compresses the input video signal and performs motion estimation (ME) and displaced frame difference (ME). A compressed bit stream may be transmitted based on techniques, such as conversion of Displaced Frame Difference (DFD). WyZER encoder 30 sends the independent description of X to WyZER decoder 40 using the Wyner-Ziv framework.

를 사용해서,

로써 비디오 신호 X의 최종 재구성

을 출력한다. 최종 재구성된 신호

는, 전술한 바와 같이, WyZER 디코더 (40) 가 무선 채널을 통한 비트 스트림의 전송 동안 발생하는 일부 오차들을 수정할 수 있기 때문에, 재구성된 신호

보다 통상적으로 더 좋은 원 비디오 신호 X의 재구성이 될 것이다. 최종 재구성 신호

는 예측 디코더 (20) 의 프레임 버퍼에 다시 기록될 수도 있다.WyZER decoder 40 then decodes the bit stream transmitted on the second channel, and reconstructs the signal.

Using

Final reconstruction of video signal X

Outputs Final reconstructed signal

As described above, since the WyZER decoder 40 can correct some errors that occur during transmission of the bit stream over the wireless channel, the reconstructed signal

More typically, there will be a better reconstruction of the original video signal X. Final reconstruction signal

May be written back to the frame buffer of prediction decoder 20.

+ Z가 되며, 여기에서, X 는 원 입력 신호를 나타내며,

는 예측 디코더에 의한 원 신호 X의 재구성을 나타내며, Z는 상호 관계 노이즈 벡터를 나타내도록 가정된다.In this embodiment, for the purpose of designing the WyZER encoder 30, the ratio assigned to the WyZER encoder is fixed and the correlation structure is X =

+ Z, where X represents the original input signal,

Denotes the reconstruction of the original signal X by the predictive decoder, and Z is assumed to represent the correlation noise vector.

Also, in this embodiment, since the operation is performed in the discrete cosine transform (DCT) domain, the components of the correlation noise vector Z are assumed to be independent. The components can be modeled as distributed Gaussian, with Gaussian having maximum entropy and thereby providing worst case analysis, with variance variance with equal differences. WyZER encoder 30 needs to find the number of distributions of the source codebook, i.e., the number of corsets, for each component in X, so that WyZER encoder 30 recognizes the variation of each component of Z. It is necessary to do To find these variations, correlation evaluation techniques can be used as described below.

For example, in one embodiment, the correlation evaluation algorithm is described in R. Zhang, SL Regunathan, and K. Rose's Proc . 33 rd Ann . Asilomar Conf . on Sig . Syst . Comp ., 1999, employs some aspects of the algorithm presented in "Optimal intra / inter mode switching for robust video communication over Internet." However, unlike the above documents, embodiments of the present invention are implemented in the DCT domain. For both primary and secondary channels, packet extinction on both channels is independent and it is assumed that packets can be decoded independently. The probability of packet loss is assumed to be equal to the probability of loss of information for the block. When a block is lost, the prediction decoder replaces the block with a block in frame memory at the same location in the previous frame. The probability of packet loss on the primary and secondary channels is denoted by p and q, respectively.

As described in co-pending US Application No. 10 / 396,982, referred to as "Method, Apparatus and System for Encoding and Decoding Side Information for Multimedia Transmission". Likewise, the WyZER encoder can communicate the corset index to the WyZER decoder. The WyZER decoder then decodes the index to identify a corset containing a codeword that together represent side information. In one embodiment, SS Pradhan and K. Ramchandran, Proc . IEEE DATA Compression The concepts disclosed in Conf ., 1999, "Distributed source coding using syndromes (DISCUS): Design and construction" can be used. Typically, the source codebook may be distributed into corsets of any channel code.

A detailed block diagram of the WyZER encoder is shown in FIG. The WyZER encoder consists of three blocks, a distortion statistics calculator, a corset encoder and a frame statistics calculator. The WyZER encoder uses the predicted reconstruction of the video frame and the current video frame data at the decoder to calculate statistics of correlation noise. These statistics are used by the corset encoder and generate a WyZER bit stream. Frame statistics are updated after WyZER encoding so frame statistics can be used in the next frame.

7 shows a block diagram of a WyZER decoder. WyZER bits consisting of corset bits and corset index information are used with side information (from the main decoder) to generate a WyZER decoded output.

는 n^th 프레임의 k^th 블록에서의 지그-재그 스캔 순서로 i^th DCT 계수의 원 값이 된다.

는 인코더 재구성값을 나타내며, 즉,

는

의 양자화 표현이다. 예측 디코더에 의해서

로써 이 계수들이 재구성되게 한다. 재구성

는 인코더에 대한 임의의 변수이다.R. Zhang, SL Regunathan, and K. Rose Proc. 33rd Ann. Asilomar Conf. on Sig. Syst. In accordance with the notation of "Optimal intra / inter mode switching for robust video communication over Internet" in Comp., 1999,

Is the original value of the i ^th DCT coefficient in the zig-zag scan order in the k ^th block of the n ^th frame.

Denotes the encoder reconstruction value, i.e.

Is

Is a quantized representation of. By predictive decoder

This allows these coefficients to be reconstructed. Reconstruction

Is any variable for the encoder.

Consider the situation where there is no second channel. In both cases, there are cases where a block is coded intra (I block) or inter (prediction, ie P block). I blocks are self-contained, that is, they contain all the information needed to be or represent one complete block of data. The P block is not self-inclusive and will typically include differential information for the previous frame, such as motion vectors and differential texture information. p is considered the possibility that an information unit is lost due to channel errors. It is also assumed that this area corresponding to the lost information unit is filled by the previously reconstructed, co-located video. Then, if the block is intra-coded,

식 (1)

Formula (1)

If the block is inter-coded,

식 (2)

Formula (2)

는

의 예측이며, 즉, n^th 프레임에서 k^th 블록에 대한 가장 좋은 예측은 (n-1)^th 프레임에서의 j^th 블록이다. 블록이 손실될 때, 블록은 이전 프레임에서 동일한 위치에서의 블록으로 대체된다. 특정 블록에 대한 예측 비트 스트림이 오차없이 수신되더라도,

는

와 동일하지 않으며, 즉, 블록은 그럼에도 불구하고 이전 오차들이 존재하기 때문에 오차가 있을 수도 있다. 이러한 경우에 대한 예상된 왜곡은 다음과 같다.Where,

Is

And the prediction, that is, the best prediction for the k ^th block in the n ^th frame (n-1) is the j ^th block in the ^th frame. When a block is lost, the block is replaced with the block at the same position in the previous frame. Even if the prediction bit stream for a particular block is received without error,

Is

Is not the same, i.e., the block may nevertheless exist because previous errors exist. The expected distortion for this case is as follows.

식 (3)

Formula (3)

는 블록에서 모든 DCT 계수들에 대해서 계산될 수 있으며 WyZER 인코더는 이 수정을 위해서 인코딩해야한다. 목적은 잘못 재구성된 DCT 계수들을 오차없는 양자화된 표현으로 개량해서 드리프트를 중지시키는 것이다. 하지만, 제 2 채널의 비율 제약이 주어진 모든 계수들을 인코딩하는 것은 가능하지 않을 수도 있다. 여기에서 올바른 접근 방법은 예상된 왜곡을 최소화시키는 DCT 계수들의 서브세트를 개량하는 가능한 비율을 사용하는 것이다. 하지만, 모든 블록에 대한 이러한 최적화를 수행하는 것은 매우 복잡하다. 오프라인 테 스트에 기초할 때 가장 좋은 전략은 대부분의 시간에서 더 높은 주파수 계수들을 개량하기 전에 더 낮은 주파수 계수들을 개량하는 것이다. 이것은 또한 비주얼 관점에서 의미가 있다. 따라서, 온라인 알고리즘에 대해서, 제 2 채널에 대한 주어진 비율하에서 수용될 수 있는 오직 m개의 가장 낮은 주파수 DCT 계수들이 전송되었다. 이러한 m개의 DCT 계수들에 대해서, WyZER 디코더는, 주 스트림 및 WyZER 스트림 모두가 도달할 때 디코더에서 계수들의 인코더 재구성을 회복할 것이다. 이 m개의 DCT 계수들에 대해서, 디코더에서의 재구성은 다음과 같을 것이다.Expected distortion,

Can be calculated for all DCT coefficients in the block and the WyZER encoder must encode for this modification. The goal is to stop drift by improving misreconstructed DCT coefficients into error-free quantized representations. However, it may not be possible to encode all coefficients given the ratio constraint of the second channel. The correct approach here is to use a possible ratio that refines the subset of DCT coefficients that minimizes the expected distortion. However, performing this optimization for every block is very complicated. The best strategy when based on offline testing is to improve the lower frequency coefficients before improving the higher frequency coefficients most of the time. This is also meaningful from a visual point of view. Thus, for the on-line algorithm, only the m lowest frequency DCT coefficients that were acceptable under the given rate for the second channel were transmitted. For these m DCT coefficients, the WyZER decoder will recover the encoder reconstruction of the coefficients at the decoder when both the main stream and the WyZER stream arrive. For these m DCT coefficients, the reconstruction at the decoder would be as follows.

식 (4)

Formula (4)

는 상기 식 (2) 에 의해서 주어진다.For remaining DCT coefficients that are not encoded by the WyZER encoder,

Is given by the above formula (2).

을 계산하기 위해서, (

및

는 이미 인코더에 인지되었기 때문에)

및

을 계산하는 것이 필요하다. 식 (1), (2) 및 (4) 는

및

를

및

와 관계되도록 사용될 수 있다. 이렇게 WyZER 인코더는 (R. Zhang, S. L. Regunathan, 및 K. Rose의 Proc. 33rd Ann. Asilomar Conf. on Sig. Syst. Comp., 1999에서의 "인터넷을 통한 견고성 비디오 통신을 위한 최적의 인트라/인터 모드 스위칭 (Optimal intra/inter mode switching for robust video communication over Internet)"에 의해서 사용된 알고리즘과 유사한) 단순한 귀납 알고리즘을 사용해서

및

를 계산하고 이렇게 식 (3) 을 사용해서

을 계산할 수 있다. 인코딩의 목적으로, 예측 인코더는 프레임을 비-중첩 공간 블록들의 그리드 (grid) 로 나누는 것을 주의한다. 예측 블록 (상기 분석에서 (n-1)^th 프레임에서의 j^th 블록) 이 비-중첩 블록들의 이러한 그리드로 정렬되지 않는다면, (j 는 (n-1)^th 프레임에서 명확한 블록이 아니기 때문에) WyZER 인코더는 (n-1)^th 프레임을 인코딩하는 동안

및

의 값들을 저장하지 않을 것이다. 이 경우에서, 명확한 블록 (valid block) 인 (n-1)^th 프레임에서 j 블록에 대한 최대 중첩을 가진 블록이 계산에서 사용된다. 대신에, j^th 블록을 중첩하는 (n-1)^th 프레임에서의 대응하는 블록들의 가중치 부여된 평균이 또한 사용될 수 있다.Expected Distortion Using Equation (3)

To calculate,

And

Is already known to the encoder)

And

It is necessary to calculate. Formulas (1), (2) and (4) are

And

To

And

Can be used to relate to This WyZER encoder is described in (R. Zhang, SL Regunathan, and K. Rose, Proc. 33rd Ann. Asilomar Conf. On Sig. Syst. Comp., 1999, "Optimal Intra / Inter for Robust Video Communication over the Internet. Using a simple induction algorithm (similar to the algorithm used by "Optimal intra / inter mode switching for robust video communication over Internet")

And

Calculate and use equation (3)

Can be calculated. For the purpose of encoding, the prediction encoder is careful to divide the frame into a grid of non-overlapping spatial blocks. If the prediction block (j ^th block in (n-1) ^th frame in the analysis) is not aligned with this grid of non-overlapping blocks, then WyZER (because j is not an explicit block in (n-1) ^th frame) While the encoder encodes (n-1) ^th frames

And

It will not store the values of. In this case, the block with the maximum overlap for the j block in the (n-1) ^th frame, which is a valid block, is used in the calculation. Instead, a weighted average of the corresponding blocks in the (n-1) ^th frame overlapping j ^th blocks can also be used.

) 는 코세트의 개수 또는 소스 코드북의 분배의 개수를 결정한다. 이 정보, 코세트의 개수는 헤더 정보의 형태로 제 2 채널을 통해서 WyZER 디코더에 통신할 수 있다. 헤더의 크기를 제한하기 위해서, 코세 트의 개수가 취할 수 있는 가능한 값은, 각각의 계수에 대해서, 제한될 수도 있다. 가능한 값들에 대한 제한은 광범위한 테스트에 기초할 수도 있다.Correlation Noise Evaluation (

) Determines the number of corsets or the number of distributions of the source codebook. This information, the number of corsets, can communicate to the WyZER decoder via the second channel in the form of header information. In order to limit the size of the header, the possible values that the number of corsets can take may be limited for each coefficient. The limitation on possible values may be based on extensive testing.

2 (a)-(f) are graphs of simulation results for various performance comparisons demonstrating the effectiveness of the WyZER codec described above. For the simulation, standard video sequences, known to those skilled in the art, were encoded with an H.263 + coder used as the predictive video codec. The second channel is allocated at a rate of about 30% on the primary channel.

FIG. 2 (a) is a diagram of a peak signal of noise ratio (PSNR) to frame number for a video sequence encoded with the DSC techniques 204 and H.263 + encoder 202 described above. In Figure 2 (a), the third frame of the standard "Flower Garden" video sequence (352x240, 15 fps, one intra frame followed by 29 intra frames) is dropped. For a fair comparison, H.263 + is supplemented with an extra proportion of the same size. Figure 2 (a) shows that with DSC techniques, the PSNR increases steadily following the dropped frame because the drift is corrected, about 7-8 dB better than H.263 + until the end of the 30th frame sequence. .

The aforementioned DSC techniques have also been tested using a wireless channel simulator that adds packet errors to multimedia data streams transmitted over a wireless network conforming to the CDMA 2000 1X Standard 5. For this test, DSC technology has been modified so that the proportion of the portion on the auxiliary channel is assigned to the forward modification codec (FEC) and the remaining portion is assigned to the WyZER stream. The reason for this is that, unlike the case where FEC is used, the WyZER stream cannot be recovered immediately from loss (see Figure 2 (a)) and the number of extinctions is less than the correction performance of the FEC. This is important from both visual and PSNR perspectives. However, as mentioned above, since it does not guarantee error recovery, the overall ratio for the side-channel is not assigned to FECs. This is especially true for wireless channels that often have large error bursts.

FIG. 2 (b) is a diagram of PSNR versus frame number for video sequences encoded with the H.263 + encoder 212, FEC 214, and DSC technique 216 described above. The standard video sequence used when generating FIG. 2 (b) is a “Stefan” video sequence (352 × 240, 15 fps, 1 GOP). In FIG. 2 (b), the three different encoding techniques compared are as follows.

1) H.263 + (curve 212) encoded in equal proportions in total assigned to primary and secondary channels

2) the full auxiliary channel (curve 214) assigned to the FEC (using Reed-Solomon code), and

3) The proposed algorithm (curve 216), which is modified by assigning a portion of the ratio to the FEC (RS codes are used again) on the auxiliary channel.

For both FEC and DSC techniques, for (2) and (3) above, the latency constraint is one frame. Tests were performed in the case of (3) to find the proper ratio allocation between the WyZER stream and the FEC. In most cases, it was best to allocate approximately equal proportions to the FEC and WyZER streams, which is the setup for the results presented here. The main and WyZER streams were transmitted on the same (simulated) radio channel.

In the case of the Stefan sequence shown in FIG. 2 (b), the WyZER 216 method presented is 2.9 dB higher than 214 for FEC only and achieves an average PSNR of 5.2 dB higher than baseline H.263 +, Better than others (curve 212).

FIG. 2 (c) is a diagram of PSNR vs. frame number for another video sequence encoded with H.263 + encoder 222, FEC 224, and WyZER technology 226 described above. The standard video sequence used when making Figure 2 (c) is the "Football" video sequence (352x240, 15 fps, 1 GOP). 2 (c) shows the typical performance of the Stefan and Football sequences (352x240, 15 fps, 1 GOP) for a packet loss rate of 8.5%.

In Figure 2 (c), the PSNR of the proposed DSC technique 226 has two large drops, in frames 23 and 26, but recovers quickly in both cases. The average PSNR for the WyZER technique 226 for this test is 3.1 dB higher than 224 for FEC only and 5.6 dB higher than baseline H.263 + (curve 222).

FIG. 2 (d) is a plot of PSNR versus error probability for video sequence “Stefan” encoded with H.263 + encoder 232, FEC 234, and DSC technology 236. 2 (e) is a plot of PSNR vs. error probability for video sequence “Football” encoded with H.263 + encoder 242, FEC 244, and DSC technology 246. As shown in Figures 2 (d) and 2 (e), in the conventional performance of the three methods of encoding, the methods of the present invention are consistently superior to the other two methods. Results are given only for the SIF (352x240) sequence, but a similar aspect is observed for other spatial size video sequences, such as QCIF, CIF.

Experiments have been performed to study the effect of burst length on performance. FIG. 2 (f) is a plot of PSNR versus average burst length for a “Football” video sequence (176 × 144, 15 fps, 30 frames) encoded with FEC 252 and DSC 254. As shown in FIG. 2 (f), if the average burst length is greater than 0.6, the techniques 254 according to the present invention are superior to the case of FEC 252 alone, but if the burst length is less than 0.6, only FEC Worse than the case.

Therefore, simulation results using the above-described techniques show significant progress in performance through conventional error protection schemes such as forward error correction codecs under fair latency constraints.

The DSC techniques described may operate with many wireless devices and wireless standards. Examples of wireless devices that can implement the described DSC techniques include cellular telephones, wireless communicable PCs, personal digital assistants (PDAs), and other wireless devices. Examples of wireless standards in which the described DSC technologies can operate include Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), and Enhanced Data GSM Environment (EDGE). Standards include Enhanced Data GSM Environment (TIA / EIA-95-B (IS-95), TIA / EIA-98-C (IS-98), CDMA2000, Wideband CDMA (WCDMA), and the like.

3 is a block diagram illustrating a portion of a wireless communication system that can implement DSC techniques. As shown in FIG. 1, a wireless communication system includes a first wireless communication device (WCD) 302, also called a mobile base station (MS). The wireless communication system includes a second wireless device 304, which may be part of a wireless structure, and a landline device connected to other wireless devices, ie, the wireless structure.

The first wireless device 302 includes a WyZER encoder 306, a WyZER decoder 308, a transmitter 310 and a receiver 312. The second wireless device 304 includes a WyZER encoder 320, a WyZER decoder 322, a transmitter 324 and a receiver 324. The second wireless device 304 may receive an input signal X ₁ , for example a video signal. The input signal X ₁ is connected to the DSC encoder 320. DSC encoder 320 includes prediction encoder 330 and WyZER encoder 332. The input signal is encoded according to the DSC techniques described above, and the main stream and the WyZER stream are output from the predictive encoder 320 and the WyZER encoder 332 to the transmitter 324.

은 또한 WyZER 디코더 (336) 의 입력이다. 상술한 DSC 기술 들에 따라 예측 디코더 및 WyZER 디코더는 서로 협력하여 입력 신호 X₁의 재구성 신호

를 생성하고 출력한다.The primary and WyZER streams are sent to the receiver 312 of the first wireless device 302 via a wireless communication channel (multiple logical channels can be used to send two streams). The main stream is output from the receiver 312 to the prediction decoder 334 in the DSC decoder 308 at the first wireless device 302. The WyZER stream is output from the receiver 312 to the WyZER decoder 336. Output of prediction decoder 334

Is also the input of the WyZER decoder 336. According to the DSC techniques described above, the prediction decoder and the WyZER decoder cooperate with each other to reconstruct the input signal X ₁ .

Create and print

은 또한 WyZER 디코더 (346) 에 대한 입력이다. 전술한 DSC 기술들에 따라서 예측 및 WyZER 디코더들은 서로 협력하여 입력 신호 X₂의 재구성 신호

을 생성하고 출력한다.In a manner similar to the method described above, the first wireless device receives an input signal X ₂ encoded by the prediction encoder 340 and the WyZER encoder 342 included in the DSC encoder 306 of the first wireless device 302. You may. The main and WyZER streams are output from the DSC encoder 306 to the transmitter 310. The receiver 326 of the second wireless device 304 receives the bit streams and outputs the predicted bit stream from the DSC decoder 322 of the second wireless device 304 to the prediction decoder 344 and outputs the WyZER stream to the DSC decoder ( And output to the WyZER decoder 346 in 322. Output of predictive decoder 344

Is also the input to the WyZER decoder 346. According to the DSC techniques described above, the prediction and WyZER decoders cooperate with each other to reconstruct the input signal X ₂ .

Create and print

In another embodiment, the wireless device may include only a DSC encoder or only a DSC decoder. For example, if the device is configured to broadcast, i.e. transmit, but not receive encoded data, the device may have a DSC encoder that encodes the broadcast data but may not have a DSC decoder because the device does not receive the encoded data. have. Similarly, an apparatus configured to receive and decode a signal but not to transmit encoded data may include a decoder but not an encoder. An example of this type of system would be a broadcast system, such as a video broadcast system, where a broadcaster transmits encoded data to one or more users, typically many users, but the users do not transmit the encoded data back to the broadcaster. will be. In this example, the broadcast station wireless device may include a DSC encoder for encoding broadcast data, and the user wireless device may include a DSC decoder to decode the received encoded broadcast.

4 is a flowchart illustrating an embodiment of determining intra frame bit allocation for a WyZER encoder. The flowchart begins at block 402 where the entire data frame is examined, distributed to the data blocks. The flow chart continues at block 404, which determines which data blocks are to be protected. For example, data blocks are "ranked" based on the detrimental effect that the loss of an individual data block is lost or corrupted if a particular block is lost or reconstructed. Lossy blocks will be more detrimental to higher rank reconstructions.

The flowchart continues at block 406 where the amount of data allocated to the WyZER encoder is determined. Typically, it is desirable to allocate the minimum amount of data allocated to the second channel since the amount of data allocated must be transmitted on the second channel and consumes additional system resources. Thus, more data will be sent and allocated on the second channel, depending on how desirable it is to protect a particular block of data. For example, if a data block is " ranked " high, then more data will be sent and allocated on the second channel for that block than a lower " ranked " block.

5 is a flowchart illustrating an embodiment for determining the maximum number of distributions of source codebooks used by the WyZER encoder. The flowchart begins at block 502 where a correlation noise assessment is determined. For example, the correlation noise of the transform coefficients may be determined for a case where a predictive bit stream, representing a data block, is successfully transmitted from a predictive encoder to a predictive decoder. Although the block is successfully transmitted, the decoded data may have errors due to errors in the previous blocks. The flowchart continues at block 504 where the number of source codebooks is used to represent a data block. The distribution number of the source codebook is also called a corset.

In one embodiment, the number of corsets may be communicated to the WyZER decoder with wireless auxiliary channel communication in the header of the WyZER stream.

Those skilled in the art will appreciate that signals and information may be represented using a variety of other technologies and technologies. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the specification may appear as precursors, currents, electromagnetic waves, magnetic or magnetic particles, photons or photons, or combinations thereof. Can be.

In addition, those skilled in the art may appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination thereof. . To clearly illustrate this alternative possibility of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above primarily in terms of their functionality. Whether such functionality is implemented in hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present invention.

The various illustrative logical blocks, modules, circuits described in connection with the embodiments disclosed herein may be general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate array signals (FPGAs), or Other programmable logic devices, separate gate or transistor logic, separate hardware components, or any combination thereof, designed to perform the functions described herein, may be implemented or performed. A general purpose processor may be a microprocessor, but in other ways, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, software module, or a combination of the two executed by a processor. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, which can read information from and write information to the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside within an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Those skilled in the art will clearly appreciate various modifications to these embodiments, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (63)

A prediction encoder configured to receive an input signal, encode the signal according to a prediction encoding standard, and output a prediction encoded bit stream, and An auxiliary channel encoder configured to receive the input signal, encode the signal according to an auxiliary channel encoding standard, and output an auxiliary channel encoded bit stream. The method of claim 1, And the predictive encoded bit stream is transmitted over a primary communication channel. The method of claim 1, The auxiliary channel encoded bit stream is transmitted over an auxiliary channel communication channel. The method of claim 1, And the predictive encoding standard is a standard compatible with the MPEG-4 standard. The method of claim 1, And the predictive encoding standard is a standard compatible with the H.26x standard. The method of claim 1, The auxiliary channel encoding standard is a standard compatible with Wyner-Ziv technologies. The method of claim 1, And the input signal is a video signal. The method of claim 1, The input signal is an intra-coded data block and an inter-coded data block. The method of claim 8, Wyner-Ziv techniques are applied to an inter-coded data block. The method of claim 1, And the auxiliary channel encoder encodes the input signal using a predetermined number of corsets. The method of claim 10, And the predetermined number of corsets is included in a header of the auxiliary channel encoded bit stream. A prediction decoder configured to receive a bit stream representation of an original signal, decode the bit stream according to a prediction decoding standard, and output a reconstructed signal, and Receive the decoded auxiliary channel bit stream and the bit stream to receive a bit stream representation of the original signal and the reconstructed signal, decode the bit stream according to an auxiliary channel decoding standard, and generate and output a final reconstruction of the original signal. A secondary channel decoder configured to combine the reconstructed signal. The method of claim 12, And the bit stream decoded by the predictive decoder is received over a primary communication channel. The method of claim 12, And the bit stream decoded by the predictive decoder is received over an auxiliary channel communication channel. The method of claim 12, And the predictive decoding standard is a standard compatible with the MPEG-4 standard. The method of claim 12, And the predictive decoding standard is a standard compatible with the H.26x standard. The method of claim 12, The secondary channel decoding standard is a standard compatible with Wyner-Ziv technologies. The method of claim 12, The original signal is a video signal. The method of claim 12, The original signal comprises an intra-coded data block and an inter-coded data block. The method of claim 19, Wyner-Ziv techniques are applied to the inter-coded data blocks. The method of claim 12, And the auxiliary channel decoder decodes the auxiliary channel bit stream using a predetermined number of corsets. The method of claim 21, And the predetermined number of corsets is included in a header of the auxiliary channel encoded bit stream. As an encoder, A prediction encoder configured to receive an input signal, encode the signal according to a prediction encoding standard, and output a prediction encoded bit stream, and An auxiliary channel encoder configured to receive the input signal, encode the signal according to an auxiliary channel encoding standard, and output an auxiliary channel encoded bit stream; And As a decoder, A predictive decoder configured to receive a bit stream, decode the bit stream according to the predictive decoding standard, and output a reconstructed signal, and Receive the bit stream and the reconstructed signal, decode the bit stream according to an auxiliary channel decoding standard, and combine the decoded bit stream and the reconstructed signal to produce and output a final reconstruction of the original signal. And said decoder comprising an auxiliary channel decoder. The method of claim 23, And the predictive decoding standard is a standard compatible with the MPEG-4 standard. The method of claim 23, And the predictive decoding standard is a standard compatible with the H.26x standard. The method of claim 23, The secondary channel decoding standard is a standard compatible with Wyner-Ziv technologies. The method of claim 23, And the input signal is a video signal. The method of claim 23, And the wireless communication device is a mobile phone. The method of claim 23, The wireless device is part of a wireless infrastructure. As a first wireless device, A predictive encoder configured to receive an input signal, encode the signal according to a predictive encoding standard, and output a predictive encoded bit stream representation of the input signal; A secondary channel encoder configured to receive the input signal, encode the signal according to an auxiliary channel encoding standard, and output an auxiliary channel encoded bit stream representation of the input signal; And As a second wireless device, A predictive decoder configured to receive the bit stream by the predictive encoder, decode the bit stream according to the predictive decoding standard, and output a reconstruction of the input signal, and Receive the decoded auxiliary channel bit stream and the reconstruction to receive the bit stream output by the auxiliary channel encoder, decode the bit stream according to an auxiliary channel decoding standard, and generate and output a final reconstruction of the input signal. And the second wireless device comprising an auxiliary channel decoder configured to combine the received signals. The method of claim 30, And the predictive decoding standard is a standard compatible with the MPEG-4 standard. The method of claim 30, And the predictive decoding standard is a standard compatible with the H.26x standard. The method of claim 30, The secondary channel decoding standard is a standard compatible with Wyner-Ziv technologies. The method of claim 30, And the first wireless communication device is a mobile phone. The method of claim 30, And the first wireless device is part of a wireless communication rescue device. The method of claim 30, And the second wireless communication device is a mobile phone. The method of claim 30, And the second wireless device is part of a wireless communication rescue device. Receiving an input signal, Encoding the signal according to a prediction encoding standard, outputting a prediction encoded bit stream, and Encoding the signal according to an auxiliary channel encoding standard and outputting an auxiliary channel encoded bit stream. The method of claim 38, And the predictive encoding standard is a standard compatible with the MPEG-4 standard. The method of claim 38, And the predictive encoding standard is a standard compatible with the H.26x standard. The method of claim 38, The auxiliary channel encoding standard is a standard compatible with Wyner-Ziv techniques. The method of claim 38, And the input signal is a video signal. The method of claim 38, Wherein the input signal comprises an intra-coded data block and an inter-coded data block. The method of claim 43, Wyner-Ziv techniques are applied to the inter-coded data blocks. The method of claim 38, And the auxiliary channel encoder encodes the input signal using a predetermined number of corsets. The method of claim 45, And the predetermined number of corsets is included in a header of the auxiliary channel encoded bit stream. Receiving a predictive encoded bit stream representation of an original signal, Decoding the prediction encoded bit stream according to a prediction decoding standard and outputting a reconstructed signal, Receiving an auxiliary channel encoded bit stream of the original signal and the reconstructed signal, and Decoding the auxiliary channel encoded bit stream according to an auxiliary channel encoding standard and combining the decoded auxiliary channel bit stream and the reconstructed signal to generate and output a final reconstruction of the original signal. Decoding method. The method of claim 47, And the predictive decoding standard is a standard compatible with the MPEG-4 standard. The method of claim 47, And the predictive decoding standard is a standard compatible with H.26x techniques. The method of claim 47, The auxiliary channel decoding standard is a standard compatible with Wyner-Ziv techniques. The method of claim 47, And the original signal is a video signal. The method of claim 47, The original signal is an intra-coded data block and an inter-coded data block. The method of claim 52, wherein Wyner-Ziv techniques are applied to the inter-coded data block. The method of claim 47, And the auxiliary channel decoder decodes the auxiliary channel bit stream using a predetermined number of corsets. The method of claim 54, wherein And the predetermined number of corsets is included in a header of the auxiliary channel encoded bit stream. A method of determining intra frame bit allocation for an auxiliary channel encoder, Examining the entire frame of data, Determining which data blocks in the data frame need protection, and Determining the amount of data in the frame that is allocated to the second channel based on the block of data that needs protection and is transmitted. As a method of determining the distribution of source codebooks, Determining a correlation noise assessment, and And determining the number of distributions of the source codebook based on the correlation noise evaluation. A computer readable medium implementing the data encoding method, The method, Receiving an input signal, Encoding the signal according to a prediction encoding standard, outputting a prediction encoded bitstream, and Encoding the signal according to an auxiliary channel encoding standard and outputting an auxiliary channel encoded bit stream. A computer readable medium implementing the data decoding method, The method, Receiving a first bit stream representation of an original signal, Decoding the first bit stream according to a prediction decoding standard and outputting a reconstructed signal, Receiving a second bit stream representation of the original signal and the reconstructed signal, and Combining the auxiliary channel decoded bit stream and the reconstructed signal to decode the second bit stream in accordance with an auxiliary channel decoding standard and to generate and output a final reconstruction of the original signal. . The method of claim 59, And the first bit stream is encoded according to a prediction encoding standard. The method of claim 59, Wherein the second bit stream is encoded according to an auxiliary channel encoding standard. Means for receiving an input signal, Means for encoding the signal according to a predictive encoding standard and outputting a predictive encoded bit stream; and Means for encoding the signal according to an auxiliary channel encoding standard and outputting an auxiliary channel encoding bit stream. Means for receiving a first bit stream representation of the original signal, Means for decoding the bit stream and output a reconstructed signal in accordance with a prediction decoding standard, Means for receiving a second bit stream representation of the original signal and the reconstructed signal, and Means for decoding the second bit stream in accordance with an auxiliary channel encoding standard and combining the auxiliary channel decoded bit stream and the reconstructed signal to generate and output a final reconstruction of the original signal. .

Images