WO2001031928A1 - Method for looping compressed bitstreams - Google Patents

Method for looping compressed bitstreams Download PDF

Info

Publication number
WO2001031928A1
WO2001031928A1 PCT/GB2000/004158 GB0004158W WO0131928A1 WO 2001031928 A1 WO2001031928 A1 WO 2001031928A1 GB 0004158 W GB0004158 W GB 0004158W WO 0131928 A1 WO0131928 A1 WO 0131928A1
Authority
WO
WIPO (PCT)
Prior art keywords
stream
audio
delay
streams
elementary
Prior art date
Application number
PCT/GB2000/004158
Other languages
French (fr)
Inventor
Bruce Fairbairn Devlin
Violet Snell
Original Assignee
Snell & Wilcox Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Snell & Wilcox Limited filed Critical Snell & Wilcox Limited
Priority to AU10440/01A priority Critical patent/AU1044001A/en
Publication of WO2001031928A1 publication Critical patent/WO2001031928A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/04—Synchronising
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H04N21/4392—Processing of audio elementary streams involving audio buffer management
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
    • H04N21/44004—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving video buffer management, e.g. video decoder buffer or video display buffer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/426—Internal components of the client ; Characteristics thereof

Definitions

  • This invention relates to compressed bit-streams and in an important example to MPEG-2 bit-streams.
  • the present invention consists, in one aspect, in a method of producing a multiplexed stream comprising at least two elementary streams which have been compression coded in respective encoders which each have an encoder buffer, the elementary streams requiring to be synchronised when decoded, comprising the steps of estimating the differential buffer delay between the two streams; removing from one end of one elementary stream a stream section approximately equal to said differential delay; and inserting said stream section at the other end of said one elementary stream.
  • Figure 1 is a block diagram illustrating the different buffer sizes of typical video and audio encoder/decoder pairs (codecs);
  • Figure 2 is a block diagram illustrating the use of a compensating delay at an audio encoder
  • Figure 3 is a diagram illustrating the desired timing relationship between video and audio after decoding of the continuous stream which results from looping of a transport stream segment;
  • Figure 4 is a diagram similar to Figure 3, illustrating the desired timing relationship between video and audio before decoding
  • Figure 5 is a diagram illustrating a process according to the present invention.
  • the example will be taken of a single programme MPEG-2 transport stream, with a requirement for precise synchronism between video and audio.
  • video and audio signals are typically delayed by different amounts as they pass through their respective encoders. This is illustrated in Figure 1 , which also shows the corresponding delay in the respective decoders.
  • the video encoder 10 has a relatively large buffer 12 corresponding in size to the buffer 14 of the video decoder 16.
  • the buffer 18 of the audio encoder 20 (corresponding in size with the buffer 22 of the audio decoder 24) is relatively small.
  • FIG 3 shows the desired relationship of the audio and video after decoding.
  • the loops start co-timed; end co-timed and remain co-timed every time the loops are decoded. Looking now at the timing relationships in the MPEG stream just prior to decoding, the need can be seen to shift the streams by the same amount as the delay in the respective decoder.
  • Figure 4 now shows the desired relationship of the audio and video prior to decoding.
  • the decoder delays are applied to the loops above, the desired synchronisation will be achieved. Notice that there is a differential delay ⁇ between the components, which under normal circumstances is greater than the delay through the audio decoder buffer and cannot be corrected by simple numerical manipulation of the compressed audio parameters.
  • the decoder buffer should be able to accommodate the residual offset between video and audio streams.
  • the differential audio and video coder delay can be estimated in various ways. Audio is generally delivered in fixed size frames. This means that the value of ⁇ must be rounded so that the number of bytes chopped is an integer number of audio frames. Other content may have qualities which mean that large segments may have to be chopped. Subtitles for example may need to be split at whole page description boundaries, depending on the presentation model being used.
  • ⁇ is the vbv_delay of the first I (or non-predicted) frame of the video image.
  • integral number of audio frames which approximates most closely to this vbv_delay.
  • Other, similar values could be used.
  • ⁇ values ranging from decoder buffer management terms from: "difficult to avoid underflow” through "easy” to "difficult to avoid overflow”.
  • An alternative to the preparation of a multiplexed transport stream that can be looped is the processing of an existing transport stream (if it conforms with certain constraints) to make it capable of being looped.
  • the processes required to make each elementary stream "loopable" are discussed in the earlier mentioned reference.
  • the section of - say - the audio stream which is to be cut from the end and pasted at the beginning of the transport stream will effectively overwrite the beginning of the audio stream. Similar considerations will apply to the concatenating, rather than looping of streams. If, for example, three existing transport streams are to be concatenated and then looped, the end section of the audio stream of the third transport stream will be pasted at the start of the audio stream of the first transport stream.
  • the "breaks" in the video and the audio between the first and second transport streams (and between the second and third) will be offset from each other by a corresponding amount.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Television Systems (AREA)
  • Time-Division Multiplex Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

A multiplexed MPEG-2 stream having video and audio elementary streams is prepared for continuous looping by: estimating the differential buffer delay between the two streams; removing from the end of the audio stream a stream section approximately equal to said differential delay; and inserting that stream section at the beginning of the audio stream.

Description

METHOD FOR LOOPING COMPRESSED BITSTREAMS
This invention relates to compressed bit-streams and in an important example to MPEG-2 bit-streams.
It is an important requirement for test and other purposes to produce MPEG-2 bit-streams which can be played in a loop, without discontinuities that would create difficulties for a downstream decoder. This problem is addressed by the invention of EP-A-0 893 030, to which reference is directed. In the production of a looped transport stream, a further problem arises from the need to ensure synchronism between decoded video and audio, or indeed between other elementary streams having a time critical relationship. It is important to realise that the video and audio signals are delayed as they pass through their respective encoders. There is also a corresponding delay in their decoders. The delay is caused by the compressed data bytes waiting in the encode/decode buffers so that the overall end-to-end delay is constant. This will be true regardless of the nature of the encoding. Unfortunately, in a typical system, the delays in the video chain are different from the delays in the audio chain, because the buffers are of very different sizes.
Under normal circumstances, extra delay needs to be added to the audio so that the final decoded audio-video synchronisation is maintained. This is not feasible in a looped bitstream. The alternative of instructing the audio decoder to "hold" the compressed data bytes for a longer time than strictly required for decoding and hence obtain synchronisation, is also not feasible because audio decoder buffers are generally not large enough. This situation becomes especially difficult when trying to create a seamlessly looped MPEG transport stream with perfectly synchronised audio and video components.
It is therefore an object of one form of the present invention to provide method and apparatus for looping a transport stream, which ensures synchronism between decoded elementary streams. Accordingly, the present invention consists, in one aspect, in a method of producing a multiplexed stream comprising at least two elementary streams which have been compression coded in respective encoders which each have an encoder buffer, the elementary streams requiring to be synchronised when decoded, comprising the steps of estimating the differential buffer delay between the two streams; removing from one end of one elementary stream a stream section approximately equal to said differential delay; and inserting said stream section at the other end of said one elementary stream.
The invention will now be described by way of example with reference to the accompanying drawings, in which:
Figure 1 is a block diagram illustrating the different buffer sizes of typical video and audio encoder/decoder pairs (codecs);
Figure 2 is a block diagram illustrating the use of a compensating delay at an audio encoder; Figure 3 is a diagram illustrating the desired timing relationship between video and audio after decoding of the continuous stream which results from looping of a transport stream segment;
Figure 4 is a diagram similar to Figure 3, illustrating the desired timing relationship between video and audio before decoding; Figure 5 is a diagram illustrating a process according to the present invention.
The example will be taken of a single programme MPEG-2 transport stream, with a requirement for precise synchronism between video and audio. As mentioned above, video and audio signals are typically delayed by different amounts as they pass through their respective encoders. This is illustrated in Figure 1 , which also shows the corresponding delay in the respective decoders. Thus the video encoder 10 has a relatively large buffer 12 corresponding in size to the buffer 14 of the video decoder 16. The buffer 18 of the audio encoder 20 (corresponding in size with the buffer 22 of the audio decoder 24) is relatively small.
Under normal circumstances, extra delay needs to be added to the audio at the encoder, so that the final decoded audio-video synchronisation is maintained. This is shown in Figure 2 with compensating delay 26. If the audio decoder buffer were big enough, it would be a simple task to instruct the decoder to "hold on" to the compressed data bytes for a longer time than usual and hence obtain synchronisation. Unfortunately, the audio buffer is not big enough to allow this to happen.
Figure 3 shows the desired relationship of the audio and video after decoding. The loops start co-timed; end co-timed and remain co-timed every time the loops are decoded. Looking now at the timing relationships in the MPEG stream just prior to decoding, the need can be seen to shift the streams by the same amount as the delay in the respective decoder.
Figure 4 now shows the desired relationship of the audio and video prior to decoding. When the decoder delays are applied to the loops above, the desired synchronisation will be achieved. Notice that there is a differential delay δ between the components, which under normal circumstances is greater than the delay through the audio decoder buffer and cannot be corrected by simple numerical manipulation of the compressed audio parameters.
In essence it is not possible to make a simple loop where each component starts at the same time T. When the components are multiplexed together for synchronisation at the output of the decoder, there will always be too great a delay in the audio (or other time critical component).
The solution offered by the present invention is illustrated in Figure 5. With an appropriate method for defining a value of the delay δ, it is possible to create the following simple multiplex: the video starts at time T and has duration D the audio (or other co-timed stream) starts at time T + δ and has duration D
In effect, there is a pre-compensation for the differential delays in the decoders by cutting a portion of the audio from the end of the stream and inserting it at the front of the stream. Provided that the value that is chosen for δ is sufficiently close to the differential audio and video coder delays, the decoder buffer should be able to accommodate the residual offset between video and audio streams.
The differential audio and video coder delay can be estimated in various ways. Audio is generally delivered in fixed size frames. This means that the value of δ must be rounded so that the number of bytes chopped is an integer number of audio frames. Other content may have qualities which mean that large segments may have to be chopped. Subtitles for example may need to be split at whole page description boundaries, depending on the presentation model being used.
One starting point for δ is the vbv_delay of the first I (or non-predicted) frame of the video image. In the case of video and audio streams, there will then be determined that integral number of audio frames which approximates most closely to this vbv_delay. Other, similar values could be used. In a particular form of the invention, where the intent is to produce a stream which is looped continuously to exercise and test decoders, there might deliberately be chosen a number of audio frames which is not the closest approximation to the vbv_delay. This would exercise more strenuously the buffer management of the decoder. Thus, calculations could be performed to define a range of δ values ranging from decoder buffer management terms from: "difficult to avoid underflow" through "easy" to "difficult to avoid overflow".
An alternative to the preparation of a multiplexed transport stream that can be looped, is the processing of an existing transport stream (if it conforms with certain constraints) to make it capable of being looped. The processes required to make each elementary stream "loopable" are discussed in the earlier mentioned reference. In this case, the section of - say - the audio stream which is to be cut from the end and pasted at the beginning of the transport stream will effectively overwrite the beginning of the audio stream. Similar considerations will apply to the concatenating, rather than looping of streams. If, for example, three existing transport streams are to be concatenated and then looped, the end section of the audio stream of the third transport stream will be pasted at the start of the audio stream of the first transport stream. The "breaks" in the video and the audio between the first and second transport streams (and between the second and third) will be offset from each other by a corresponding amount.
It should be understood that the invention has been described by way of example only and a wide variety of modifications are possible without departing from the scope of the invention. Whilst video and audio streams are important examples of multiplexed elementary streams which may require to be synchronised on decoding, the invention will apply to other such examples such as video and subtitles. Also, whilst MPEG-2 has been taken as an example of a compression scheme, the invention is not limited in this respect.

Claims

1. A method of producing a multiplexed stream comprising at least two elementary streams which have been compression coded in respective encoders which each have an encoder buffer, the elementary streams requiring to be synchronised when decoded, comprising the steps of estimating the differential buffer delay between the two streams; removing from one end of one elementary stream a stream section approximately equal to said differential delay; and inserting said stream section at the other end of said one elementary stream.
2. A method according to Claim 1 , wherein the two elementary streams are respectively video and audio.
3. A method according to Claim 2, wherein the step of estimating the differential buffer delay between the video and audio streams, comprises the step of determining the vbv-delay of the first I frame in the video stream.
4. A method according to Claim 3, wherein said stream section comprises a section of the audio stream of length given by the number of whole audio frames corresponding closest to said vbv-delay.
5. A multiplexed stream loop comprising at least two elementary streams which have been compression coded in respective encoders which each have an encoder buffer, the average delay in the respective buffers being substantially different as between the two elementary streams, the elementary streams requiring to be synchronised when decoded, the start point of one elementary stream in the multiplexed stream loop being offset from the start point of the other elementary stream by an amount corresponding to the differential buffer delay between the two streams.
6. A method according to Claim 5, wherein the two elementary streams are respectively video and audio.
7. A method according to Claim 6, wherein the amount of the offset is given by the number of whole audio frames corresponding closest to the vbv- delay of a non-predicted video frame.
8. A method of preparing a length of multiplexed stream for continuous looping, the multiplexed stream comprising at least two elementary streams which have been compression coded in respective encoders which each have an encoder buffer, the elementary streams requiring to be synchronised when decoded, the method comprising the steps of demultiplexing the elementary streams, estimating the differential buffer delay between the two streams; cutting from one end of one elementary stream, a stream section approximately equal to said differential delay; inserting said stream section at the other end of said one elementary stream and remultiplexing the elementary streams .
9. A method according to Claim 8, wherein the two elementary streams are respectively video and audio.
PCT/GB2000/004158 1999-10-26 2000-10-26 Method for looping compressed bitstreams WO2001031928A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU10440/01A AU1044001A (en) 1999-10-26 2000-10-26 Method for looping compressed bitstreams

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB9925331.2 1999-10-26
GB9925331A GB2357674B (en) 1999-10-26 1999-10-26 Compressed bitstreams

Publications (1)

Publication Number Publication Date
WO2001031928A1 true WO2001031928A1 (en) 2001-05-03

Family

ID=10863414

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2000/004158 WO2001031928A1 (en) 1999-10-26 2000-10-26 Method for looping compressed bitstreams

Country Status (3)

Country Link
AU (1) AU1044001A (en)
GB (1) GB2357674B (en)
WO (1) WO2001031928A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1997039587A1 (en) * 1996-04-12 1997-10-23 Snell & Wilcox Limited Method and apparatus for looping of compressed video bitstreams
US5793431A (en) * 1994-12-02 1998-08-11 U.S. Philips Corporation Audio/video discrepancy management

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB9001295D0 (en) * 1990-01-19 1990-03-21 British Broadcasting Corp Buffer resynchronisation
JPH08223131A (en) * 1995-02-14 1996-08-30 Oki Electric Ind Co Ltd Data multiplexer/demultiplexer
US5751694A (en) * 1995-05-22 1998-05-12 Sony Corporation Methods and apparatus for synchronizing temporally related data streams

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5793431A (en) * 1994-12-02 1998-08-11 U.S. Philips Corporation Audio/video discrepancy management
WO1997039587A1 (en) * 1996-04-12 1997-10-23 Snell & Wilcox Limited Method and apparatus for looping of compressed video bitstreams

Also Published As

Publication number Publication date
AU1044001A (en) 2001-05-08
GB2357674B (en) 2003-12-10
GB2357674A (en) 2001-06-27
GB9925331D0 (en) 1999-12-29

Similar Documents

Publication Publication Date Title
EP1397918B1 (en) Splicing of digital video transport streams
EP0846396B1 (en) Switching between bit-rate reduced signals
US6252873B1 (en) Method of ensuring a smooth transition between MPEG-2 transport streams
EP1002424B1 (en) Processing coded video
US20010019585A1 (en) Method of switching between video sequences and corresponding device
AU751335B2 (en) Video and audio signal processing
WO1999030503A3 (en) Method and apparatus for encoding and slicing digital video bit streams
JP2002510947A (en) Burst data transmission of compressed video data
WO1999052067A3 (en) Apparatus and method of splicing digital video streams
US8170401B2 (en) Optimizing ad insertion by removing low information frames
EP0995315A1 (en) Switching compressed video bitstreams
CA2303149C (en) Seamless splicing of compressed video programs
US8904426B2 (en) Preconditioning ad content for digital program insertion
AU766738B2 (en) Data compression unit control for alignment of output signal
GB2372657A (en) Preserving compression parameters, including buffer occupancy values, when decoding a signal
WO2001031928A1 (en) Method for looping compressed bitstreams
EP0871337A2 (en) Method and apparatus for modifying a digital data stream
KR20010090514A (en) Demultiplexing a statistically multiplexed mpeg transport stream into cbr single program transport streams
JP3573171B2 (en) Transmission method and transmission system for multiple variable rate signals
KR100978506B1 (en) Digital video player and the method for controlling buffer the player
Helms et al. Switching and Splicing of MPEG-2 Transport Streams
AU3798199A (en) Video compression

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AU JP US

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP