EP2752023A1 - Method to match input and output timestamps in a video encoder and advertisement inserter - Google Patents

Method to match input and output timestamps in a video encoder and advertisement inserter

Info

Publication number
EP2752023A1
EP2752023A1 EP12766810.1A EP12766810A EP2752023A1 EP 2752023 A1 EP2752023 A1 EP 2752023A1 EP 12766810 A EP12766810 A EP 12766810A EP 2752023 A1 EP2752023 A1 EP 2752023A1
Authority
EP
European Patent Office
Prior art keywords
video
timestamp
data frame
transcoder
video data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP12766810.1A
Other languages
German (de)
French (fr)
Other versions
EP2752023B1 (en
Inventor
Robert S. Nemiroff
Jing Yang Chen
Yendo Hu
Shanhua Xue
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google Technology Holdings LLC
Original Assignee
Motorola Mobility LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Motorola Mobility LLC filed Critical Motorola Mobility LLC
Publication of EP2752023A1 publication Critical patent/EP2752023A1/en
Application granted granted Critical
Publication of EP2752023B1 publication Critical patent/EP2752023B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4305—Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4

Definitions

  • the present invention relates to a method and system for decoding and encoding video data frames.
  • the present invention further relates to calculating a timestamp offset.
  • a video transcoder may receive a compressed video data frame to be converted into a format usable by a media device.
  • the video transcoder may decode the compressed video data frame into an uncompressed intermediate format.
  • the video transcoder may then re-encode the uncompressed intermediate video frame into a new format usable by the media device.
  • Data channels passed through the transcoder may have Presentation timestamps (PTSs) embedded in the data to trigger time critical events, for example, advertisement insertion and data display.
  • PTSs Presentation timestamps
  • the PTS of the stream output by the transcoder may lose synchronization with the PTS input to the decoder and with the frame. Therefore, a need exists to synchronize the PTSs of a frame input to the encoder with the PTSs of the same frame when output by the encoder, and to maintain input stream timing in the instance of audio splicing.
  • a method, a video processing system, and an electronic device are disclosed.
  • a video transcoder may decode a compressed video data frame creating a decoded video data frame.
  • the video transcoder may embed a network presentation timestamp in the decoded video data frame.
  • the video transcoder may re-encode the decoded video data frame creating a transcoded video data frame.
  • a field programmable gate array may compare the network presentation timestamp with a transcoder presentation timestamp to determine a timestamp offset.
  • Figure 1 illustrates, in a block diagram, one embodiment of a mobile system or electronic device to create a radio connection.
  • Figure 2 illustrates, in a block diagram, one embodiment of a video transcoder.
  • Figure 3 illustrates, in a flowchart, one embodiment of a method for processing a compressed video data frame.
  • Figure 4 illustrates, in a flowchart, one embodiment of a method for determining a timestamp offset using a network presentation timestamp.
  • Figure 5 illustrates, in a flowchart, one embodiment of a method for determining a program clock reference offset using multiple counters.
  • the present invention comprises a variety of embodiments, such as a method, an electronic device, and a video processing system, and other embodiments that relate to the basic concepts of the invention.
  • the video processing system or electronic device may be any manner of computer, mobile device, or wired or wireless communication device.
  • a method, a video processing system, and an electronic device are disclosed.
  • a video transcoder may decode a compressed video data frame creating a decoded video data frame.
  • the video transcoder may embed a network presentation timestamp in the decoded video data frame.
  • the video transcoder may re-encode the decoded video data frame creating a transcoded video data frame.
  • the video transcoder may embed a network presentation timestamp in the transcoded video frame.
  • a field programmable gate array may compare the network presentation timestamp with a transcoder presentation timestamp to determine a timestamp offset.
  • a video processing system may mark the time using a program clock reference.
  • the video processing system may receive a compressed video frame having a decode timestamp and a presentation timestamp.
  • the decode timestamp indicates when a compressed video frame is to be decoded.
  • the presentation timestamp indicates when a compressed video frame is to be presented. Compressed video frames may be decoded or presented in a different order than the order in which the compressed video frames are received.
  • the video processing system may decode the compressed video frame when the decode timestamp matches a program clock reference.
  • the video processing system may present the decoded video frame when the presentation timestamp matches the program clock reference.
  • a video processing system may have an audio transport bypass for a video transcoder for pass-thru.
  • the video processing system may adjust the presentation timestamp and program clock reference for audio bypass using a field programmable gate array.
  • the video processing system may synchronize the video and audio data so that the output presentation timestamp matches the input presentation timestamp for each
  • the transcoder decoder may embed a presentation timestamp of each decoded video frame into the vertical blanking interval output of that frame.
  • a vertical blanking interval is the interval between the end of one frame of video data and the beginning of the next frame of video data. Traditionally, the interval is blanked to allow a raster to move from the bottom of a screen to the top of the screen without leaving a line.
  • the transcoder encoder may put that network presentation timestamp in the encoded bitstream of that transcoded video frame.
  • a new output program clock reference may be based on the program clock reference plus a timestamp offset.
  • the timestamp offset may equal the transcoder presentation timestamp minus the network presentation timestamp.
  • the network presentation timestamp is a presentation timestamp received from the network transport stream.
  • the transcoder presentation timestamp is a presentation timestamp received with the transcoded compressed video data frame.
  • the timestamp offset references difference between the transcoder presentation timestamp and the network presentation timestamp.
  • the encoding program clock reference is the program clock reference in the transport output from the video encoder.
  • the network presentation timestamp may be placed in adaptation stuffing in packetized elementary stream header packet. Once extracted, the transcoder will replace the network presentation timestamp with adaptation stuffing bytes (OxFF).
  • the least significant bit of the 720p presentation timestamp and decode timestamp toggles since each 720p-5994 frame is 1501.5 90Khz (kilohertz) clock cycles. So sequential presentation timestamp values in display order may be 1501 and 1502. In order to preserve the least significant bit for both the network presentation timestamp and the transcoder presentation timestamp, the presentation timestamp offset may not change unless the absolute difference between the new computed values is more than one (1) with a negligible offset error.
  • the audio packets may be delayed since they have not gone through the transcoder.
  • Delay may be up to 4 seconds for decoding and encoding delay.
  • the output time of each audio frame may be presumed to be an input audio packet presentation timestamp minus 32ms.
  • the output program clock reference may be used as reference.
  • a margin that is rate dependent to keep the audio buffer as full as possible may be added in to improve accuracy.
  • Each packetized elementary stream may have several audio frames. Each audio frame may be output based on the audio rate. If the frames are sent out too fast, the decoder audio buffer may overflow. If the frames are sent out too slow, the decoder audio buffer may underflow. The audio rate may be extracted from the header. If the audio presentation timestamp with respect to the program clock reference is late or greater than 5 seconds, the audio packets in the memory may be dropped or flushed out to the transport until a presentation timestamp in range is found.
  • the field programmable gate array may have a local counter at a frequency of at least lKhz, producing an error no greater than 1ms.
  • the field programmable gate array may have two counters to track the input program clock reference and the output program clock reference. For each program clock reference that arrives, the counter may be loaded with that program clock reference value.
  • a second counter For each program clock reference that is output after the presentation timestamp offset adjustment, a second counter may be loaded with the output program clock reference value. Both the input program clock reference counter and output program clock reference counter may then be clocked at 27Mhz (Megahertz). In a steady-state error free case, each counter may be within 1 tick of the next received program clock reference for each corresponding counter. The value may be one (1) tick off due to program clock reference jitter of the input and small differences in clock frequencies between the source program clock reference and local field programmable gate array 27Mhz clock. The difference between the input program clock reference counter and output program clock reference counter may be the delay from transport input to transport output caused by the decode and encode process time.
  • the difference between the input and output program clock reference counter may be very large and may not represent the decode and encode process time. Therefore, if the difference is negative or greater than the worst case decode and encode process time, the video processing system may ignore the difference and use the previous valid difference for the delay.
  • the video processing system may compute the timestamp offset each frame and use that value, based on the transcoder presentation timestamp and the network presentation timestamp.
  • the program clock reference discontinuity bit may be set in the first program clock reference packet where the new timestamp offset is applied.
  • the field programmable gate array may compute the program clock reference offset between the original program clock reference and new program clock reference.
  • the new timestamp offset may be the current timestamp offset minus the program clock reference offset between the original program clock reference and new program clock reference.
  • the difference between the transcoder presentation timestamp and encode presentation timestamp may be equal to the new timestamp offset within a 90Khz tick.
  • a local data stream may be stored on an on-board drive. Five to ten seconds prior to a switch time, the video processing system may read the transport stream from the drive. The video processing system may send the data to a field programmable gate array. The field programmable gate array may send the local data stream to the local decoder. The field programmable gate array may monitor the transcoder transport buffer level to provide flow control to the video processing system and avoid local decoder buffer overflow.
  • the local decoder may wait in an idle state, but output black or repeat frames from the video port.
  • the clock time may be input from the network.
  • the local timestamp stream may be sent to the local decoder several frames ahead of the switch time.
  • the local decoder may start to decode the transport and have the first output frame ready to transmit. At that time, the local decoder may stop decoding until that first frame is transmitted.
  • the local data input may signal the local decoder and related hardware at least 10ms early to begin outputting the local video.
  • the local data input may send the splice timestamp value for that frame.
  • the transcoder may embed the network presentation timestamp and a switching flag in the vertical blanking interval.
  • the transcoder may embed the local presentation timestamp in the vertical blanking interval.
  • a field programmable gate array may perform the video switch.
  • the field programmable gate array may execute the switch based on a switching flag embedded in the network video vertical blanking interval.
  • the field programmable gate array may embed the network presentation timestamp, the local presentation timestamp and the switching flag in the vertical blanking interval.
  • the transcoder may place the network presentation timestamp and local presentation timestamp in a packetized elementary stream stuffing in the packetized elementary stream header.
  • Output hardware may not detect any changes in the timestamp offset since the network presentation timestamp originated from the network feed and the transcoder presentation timestamp continues uninterrupted.
  • the switch may occur on the audio frame in which the presentation timestamp switch time is greater than the network audio presentation timestamp value.
  • the field programmable gate array may monitor the switching flag in the video to determine the presentation timestamp switch time.
  • the local presentation timestamp values may be replaced with the network presentation timestamp value, incorporating the timestamp offset. Note that the audio presentation timestamp values may not be the same values as the video presentation timestamp since audio and video have different frame rates.
  • the audio timestamp rate may occur in bursts.
  • the hardware may read the audio rate in the audio header and throttle the output of the audio packets at that audio rate plus a percentage to accommodate for the timestamp, the packetized elementary stream headers, the packetized elementary stream alignment, and others.
  • An audio switch may be executed as a "dirty switch.”
  • the last network audio frame before the switch may be cut off and the first audio packet of the switch may contain an audio frame fragment.
  • the "dirty switch” may result in an audible noise being heard.
  • An audio switch may be executed as a "seamless switch.”
  • the "seamless switch” may be a switch like ad-insert.
  • the video processing system may complete last network audio frame.
  • the video processing system may insert mute frames to keep the audio buffer from becoming empty.
  • the video processing system may keep the presentation timestamp continuous.
  • the first audio packet of the switch may contain a packetized elementary stream header.
  • An audio switch may be executed as a "restart switch.”
  • the video processing system may complete the last network audio frame before the switch.
  • the video processing system may replace any fragment of the next frame with adaptation stuffing with the last packet containing that frame.
  • the first audio packet of the switch may contain a packetized elementary stream header.
  • the video processing system may empty, but receive intact audio frames.
  • the last audio frame prior to the splice time may be when the presentation timestamp of that frame is less than 32ms before the splice presentation timestamp time.
  • the last packet of that frame may be modified.
  • the field programmable gate array may compute the presentation timestamp of the local audio stream.
  • the first packet may contain a data fragment from the previous audio frame.
  • the video processing system may replace the partial frame with adaptation stuffing.
  • each audio frame may not contain a packetized elementary stream header.
  • the field programmable gate array may wait until the packetized elementary stream arrives before performing the audio switch.
  • the field programmable gate array may leave a gap in audio of up to 700ms, the maximum packetized elementary stream period. In practice, the maximum packetized elementary stream period may be about 200ms.
  • the field programmable gate array may track the presentation timestamp for the audio sync (synchronization) frames so that the switch may occur at the correct audio frame.
  • the field programmable gate array may insert a packetized elementary stream header.
  • the field programmable gate array may re- packetize subsequent audio frames.
  • the transcoder may signal the field programmable gate array at least 10 ms early to switch back to the primary video or audio.
  • the field programmable gate array may monitor the switching flag in the network vertical blanking interval, perform the video switch, and place the network presentation timestamp in the output video vertical blanking interval.
  • the network and local video output may be clock and frame synchronized.
  • the output hardware may not detect any changes in the timestamp offset since the local video output embedded the network presentation timestamp in the vertical blanking interval and the network presentation timestamp continues uninterrupted.
  • the video delay through the field programmable gate array may change by a frame.
  • the audio delay may be changed accordingly.
  • the field programmable gate array may measure this variable delay.
  • the transcoder may add the variable delay value to each audio presentation timestamp as the audio data set arrives.
  • the field programmable gate array may force the audio to increase the delay time by the variable delay.
  • FIG 1 a block diagram is provided illustrating an electronic device 100, capable of acting as a video processing system, in accordance with an embodiment of the present invention.
  • the electronic device 100 may also support one or more applications for performing various communications with a network.
  • Device software may be written in any programming language, such as C, C++, Java or Visual Basic, for example.
  • Device software may run on an application framework, such as, for example, a Java® server or .NET ® framework
  • the electronic device 100 may include a transceiver 102, which is capable of sending and receiving data over the network.
  • the electronic device 100 may include a processor, or controller, 104 that executes stored programs.
  • the controller/processor 104 may be any programmed processor known to one of skill in the art.
  • the disclosed method may also be implemented on a general-purpose or a special purpose computer, a programmed microprocessor or microcontroller, peripheral integrated circuit elements, an application- specific integrated circuit or other integrated circuits, hardware/electronic logic circuits, such as a discrete element circuit, a programmable logic device, such as a programmable logic array, field programmable gate-array, or the like.
  • any device or devices capable of implementing the disclosed method as described herein may be used to implement the disclosed system functions of this invention.
  • the electronic device 100 may also include a volatile memory 106 and a nonvolatile memory 108 to act as data storage for the processor 104.
  • the volatile memory 106 or the non- volatile memory 108 may include one or more electrical, magnetic or optical memories such as a random access memory (RAM), cache, hard drive, or other memory device.
  • the volatile memory 106 may have a cache to speed access to specific data.
  • the non- volatile memory 108 may also be connected to a compact disc - read only memory (CD-ROM), digital video disc - read only memory (DVD-ROM), DVD read write input, tape drive, or other removable memory device that allows media content to be directly uploaded into the system.
  • the electronic device 100 may include a user input interface 110 that may comprise elements such as a keypad, display, touch screen, and the like.
  • the electronic device 100 may also include a user output device that may comprise a display screen and an audio interface 112 that may comprise elements such as a microphone, earphone, and speaker.
  • the electronic device 100 also may include a component interface 114 to which additional elements may be attached, for example, a universal serial bus (USB) interface or a geographical positioning system (GPS).
  • the electronic device 100 may include a power supply 116.
  • a video transcoder 202 may receive a compressed video data frame from a network video channel input 204, optionally via an input counter 214.
  • the video transcoder 202 may decode the compressed video data frame and re-encode the compressed video data frame into a target video data format.
  • a video transcoder 202 may have an audio data bypass 206 that receives an audio data set associated with the compressed video data frame via a network audio channel input 208.
  • a field programmable gate array 210 may mix the transcoded video data frame with the associated audio data set for presentation to a user.
  • the video transcoder 202 may receive a network presentation timestamp from the network video channel input 204 along with the compressed video data frame.
  • the video transcoder 202 may decode the compressed video data frame and determine the network presentation timestamp.
  • the video transcoder 202 may embed the network presentation timestamp in the compressed video data frame.
  • the video transcoder 202 may embed the network presentation timestamp in a vertical blanking interval.
  • the video transcoder 202 may re-encode the compressed video data frame.
  • the video transcoder 202 may embed the network presentation timestamp in the video frame header.
  • the field programmable gate array 210 may receive the transcoded compressed video data frame and a transcoder timestamp.
  • the field programmable gate array 210 may compare the network presentation timestamp to the transcoder presentation timestamp to determine the timestamp offset.
  • the video processing system may determine a program clock reference offset.
  • An input counter 216 may track an input program clock reference and an output counter 218 may track an output program clock reference.
  • the audio data bypass 212 may delay the associated audio data set based on the input program clock reference and the output program clock reference.
  • the field programmable gate array 210 may subtract the input program clock reference from the output program clock reference to determine a program clock reference offset.
  • An audio delay 212 may use the program clock reference offset to delay the associated audio data set to match the compressed video data frame.
  • a local video channel input 218 may receive a local video stream for insertion after the compressed video data frame using a local offset.
  • the local video stream may be sent to the video transcoder 202.
  • a local audio channel input 220 may receive an associated local audio stream for insertion after the compressed video data frame using a network presentation timestamp.
  • the audio data stream may go to the audio data bypass 206.
  • a local data input channel 222 may feed a splice timestamp to the video transcoder, indicating the time that the local video stream and local audio stream is to be inserted.
  • the local video channel input 218 and the local audio channel input 220 may be attached to a network or a local hard drive.
  • the field programmable gate array 210 may make the insertion upon determining a switch frame using the program clock reference and the splice timestamp.
  • the field programmable gate array 210 may switch to a compressed audio data set using the network presentation timestamp.
  • the field programmable gate array 210 may switch to a pulse code modulated audio data set using the network presentation timestamp.
  • the field programmable gate array 210 may determine a local offset by subtracting a local video presentation timestamp from the network presentation timestamp. The local offset may be added to the local audio presentation timestamps to synchronize the local audio presentation timestamps to the network presentation timestamp.
  • the video processing system 200 may receive a compressed video data frame (Block 302).
  • the video processing system 200 may decode the compressed video data frame in a video transcoder 202 creating a decoded video data frame (Block 304).
  • the video processing system 200 may re-encode the decoded video data frame in the video transcoder 202 creating a transcoded video data frame (Block 306).
  • the video processing system 200 may determine a timestamp offset (Block 308).
  • the video processing system 200 may apply the timestamp offset to a program clock reference, program presentation timestamp, and program decode timestamp to match the output program clock reference and the timestamps to the input program clock reference and timestamps (Block 310).
  • the video processing system 200 may bypass an associated audio data set around the video transcoder 202 (Block 312).
  • the video processing system 200 may delay the associated audio dataset based on a program clock reference offset, determined from an input program clock reference and an output program clock reference (Block 314).
  • the video processing system 200 may delay the associated audio dataset to match the transcoded video data frame (Block 316).
  • the video processing system 200 may insert a local video stream using the splice timestamp (Block 320).
  • the video processing system 200 may switch to a compressed audio data set using the splice timestamp (Block 322).
  • the compressed audio data set may be a pulse code modulated audio data set.
  • the video processing system 200 may decode a compressed video data frame in a video transcoder 202 creating a decoded video data frame (Block 402).
  • the video processing system 200 may embed a network presentation timestamp from a video frame header in the compressed video data frame in a vertical blanking interval of the decoded video data frame (Block 404).
  • the video processing system 200 may re-encode the compressed video data frame in the video transcoder 202 creating a transcoded video data frame (Block 406).
  • the video processing system 200 may embed the network presentation timestamp in a video frame header of the transcoded video data frame (Block 408).
  • the video processing system 200 may compare the network presentation timestamp with a transcoder presentation timestamp (Block 410).
  • the video processing system 200 may calculate a timestamp offset (Block 412).
  • FIG. 5 a flowchart is provided that illustrates a method 500 performed by the video processing system 200 for determining an audio pass through delay using multiple counters in accordance with an embodiment of the present invention.
  • the video processing system 200 may use an input counter 216 to track an input program clock reference (PCR) (Block 502).
  • the video processing system 200 may use an output counter 218 to track an output program clock reference (Block 504).
  • the video processing system 200 may calculate the difference between the output program clock reference and the input program clock reference to determine the audio pass through delay (Block 506).
  • Embodiments within the scope of the present invention may also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon.
  • Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer.
  • Such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures.
  • a network or another communications connection either hardwired, wireless, or combination thereof
  • any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media.
  • Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network.
  • Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions.
  • Computer- executable instructions also include program modules that are executed by computers in stand-alone or network environments.
  • program modules include routines, programs, objects, components, and data structures, etc. that perform particular tasks or implement particular abstract data types.
  • Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Description

METHOD TO MATCH INPUT AND OUTPUT TIMESTAMPS IN A VIDEO ENCODER AND ADVERTISEMENT INSERTER
1. Field of the Invention
[0001] The present invention relates to a method and system for decoding and encoding video data frames. The present invention further relates to calculating a timestamp offset.
2. Background
[0002] A video transcoder may receive a compressed video data frame to be converted into a format usable by a media device. The video transcoder may decode the compressed video data frame into an uncompressed intermediate format. The video transcoder may then re-encode the uncompressed intermediate video frame into a new format usable by the media device.
[0003] Data channels passed through the transcoder may have Presentation timestamps (PTSs) embedded in the data to trigger time critical events, for example, advertisement insertion and data display. However, due to the decoding and re-encoding and to audio splicing, which involves audio delay, in order to insert advertisements, the PTS of the stream output by the transcoder may lose synchronization with the PTS input to the decoder and with the frame. Therefore, a need exists to synchronize the PTSs of a frame input to the encoder with the PTSs of the same frame when output by the encoder, and to maintain input stream timing in the instance of audio splicing.
SUMMARY OF THE INVENTION
[0004] A method, a video processing system, and an electronic device are disclosed. A video transcoder may decode a compressed video data frame creating a decoded video data frame. The video transcoder may embed a network presentation timestamp in the decoded video data frame. The video transcoder may re-encode the decoded video data frame creating a transcoded video data frame. A field programmable gate array may compare the network presentation timestamp with a transcoder presentation timestamp to determine a timestamp offset.
BRIEF DESCRIPTION OF THE DRAWINGS
[0005] Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
[0006] Figure 1 illustrates, in a block diagram, one embodiment of a mobile system or electronic device to create a radio connection.
[0007] Figure 2 illustrates, in a block diagram, one embodiment of a video transcoder.
[0008] Figure 3 illustrates, in a flowchart, one embodiment of a method for processing a compressed video data frame.
[0009] Figure 4 illustrates, in a flowchart, one embodiment of a method for determining a timestamp offset using a network presentation timestamp.
[0010] Figure 5 illustrates, in a flowchart, one embodiment of a method for determining a program clock reference offset using multiple counters.
DETAILED DESCRIPTION OF THE INVENTION
[0011] Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth herein. [0012] Various embodiments of the invention are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the invention.
[0013] The present invention comprises a variety of embodiments, such as a method, an electronic device, and a video processing system, and other embodiments that relate to the basic concepts of the invention. The video processing system or electronic device may be any manner of computer, mobile device, or wired or wireless communication device.
[0014] A method, a video processing system, and an electronic device are disclosed. A video transcoder may decode a compressed video data frame creating a decoded video data frame. The video transcoder may embed a network presentation timestamp in the decoded video data frame. The video transcoder may re-encode the decoded video data frame creating a transcoded video data frame. The video transcoder may embed a network presentation timestamp in the transcoded video frame. A field programmable gate array may compare the network presentation timestamp with a transcoder presentation timestamp to determine a timestamp offset.
[0015] A video processing system may mark the time using a program clock reference. The video processing system may receive a compressed video frame having a decode timestamp and a presentation timestamp. The decode timestamp indicates when a compressed video frame is to be decoded. The presentation timestamp indicates when a compressed video frame is to be presented. Compressed video frames may be decoded or presented in a different order than the order in which the compressed video frames are received. The video processing system may decode the compressed video frame when the decode timestamp matches a program clock reference. The video processing system may present the decoded video frame when the presentation timestamp matches the program clock reference. [0016] A video processing system may have an audio transport bypass for a video transcoder for pass-thru. The video processing system may adjust the presentation timestamp and program clock reference for audio bypass using a field programmable gate array. The video processing system may synchronize the video and audio data so that the output presentation timestamp matches the input presentation timestamp for each video frame.
[0017] First, the transcoder decoder may embed a presentation timestamp of each decoded video frame into the vertical blanking interval output of that frame. A vertical blanking interval is the interval between the end of one frame of video data and the beginning of the next frame of video data. Traditionally, the interval is blanked to allow a raster to move from the bottom of a screen to the top of the screen without leaving a line. The transcoder encoder may put that network presentation timestamp in the encoded bitstream of that transcoded video frame.
[0018] A new output program clock reference may be based on the program clock reference plus a timestamp offset. The timestamp offset may equal the transcoder presentation timestamp minus the network presentation timestamp. The network presentation timestamp is a presentation timestamp received from the network transport stream. The transcoder presentation timestamp is a presentation timestamp received with the transcoded compressed video data frame. The timestamp offset references difference between the transcoder presentation timestamp and the network presentation timestamp. The encoding program clock reference is the program clock reference in the transport output from the video encoder.
[0019] The network presentation timestamp may be placed in adaptation stuffing in packetized elementary stream header packet. Once extracted, the transcoder will replace the network presentation timestamp with adaptation stuffing bytes (OxFF).
[0020] The least significant bit of the 720p presentation timestamp and decode timestamp toggles since each 720p-5994 frame is 1501.5 90Khz (kilohertz) clock cycles. So sequential presentation timestamp values in display order may be 1501 and 1502. In order to preserve the least significant bit for both the network presentation timestamp and the transcoder presentation timestamp, the presentation timestamp offset may not change unless the absolute difference between the new computed values is more than one (1) with a negligible offset error.
[0021] The audio packets may be delayed since they have not gone through the transcoder. Delay may be up to 4 seconds for decoding and encoding delay.
[0022] Assuming an audio frame of 32ms (milliseconds), the output time of each audio frame may be presumed to be an input audio packet presentation timestamp minus 32ms. The output program clock reference may be used as reference. A margin that is rate dependent to keep the audio buffer as full as possible may be added in to improve accuracy.
[0023] Each packetized elementary stream may have several audio frames. Each audio frame may be output based on the audio rate. If the frames are sent out too fast, the decoder audio buffer may overflow. If the frames are sent out too slow, the decoder audio buffer may underflow. The audio rate may be extracted from the header. If the audio presentation timestamp with respect to the program clock reference is late or greater than 5 seconds, the audio packets in the memory may be dropped or flushed out to the transport until a presentation timestamp in range is found.
[0024] Alternately, the field programmable gate array may have a local counter at a frequency of at least lKhz, producing an error no greater than 1ms. The field programmable gate array may have two counters to track the input program clock reference and the output program clock reference. For each program clock reference that arrives, the counter may be loaded with that program clock reference value.
[0025] For each program clock reference that is output after the presentation timestamp offset adjustment, a second counter may be loaded with the output program clock reference value. Both the input program clock reference counter and output program clock reference counter may then be clocked at 27Mhz (Megahertz). In a steady-state error free case, each counter may be within 1 tick of the next received program clock reference for each corresponding counter. The value may be one (1) tick off due to program clock reference jitter of the input and small differences in clock frequencies between the source program clock reference and local field programmable gate array 27Mhz clock. The difference between the input program clock reference counter and output program clock reference counter may be the delay from transport input to transport output caused by the decode and encode process time.
[0026] If a program clock reference discontinuity occurs, the difference between the input and output program clock reference counter may be very large and may not represent the decode and encode process time. Therefore, if the difference is negative or greater than the worst case decode and encode process time, the video processing system may ignore the difference and use the previous valid difference for the delay.
[0027] When the input program clock reference changes or a discontinuity occurs, the video processing system may compute the timestamp offset each frame and use that value, based on the transcoder presentation timestamp and the network presentation timestamp. When the timestamp offset changes, the program clock reference discontinuity bit may be set in the first program clock reference packet where the new timestamp offset is applied.
[0028] If a program clock reference discontinuity bit is set at the source, the field programmable gate array may compute the program clock reference offset between the original program clock reference and new program clock reference. The new timestamp offset may be the current timestamp offset minus the program clock reference offset between the original program clock reference and new program clock reference. As a check, when the next frame arrives, the difference between the transcoder presentation timestamp and encode presentation timestamp may be equal to the new timestamp offset within a 90Khz tick.
[0029] A local data stream may be stored on an on-board drive. Five to ten seconds prior to a switch time, the video processing system may read the transport stream from the drive. The video processing system may send the data to a field programmable gate array. The field programmable gate array may send the local data stream to the local decoder. The field programmable gate array may monitor the transcoder transport buffer level to provide flow control to the video processing system and avoid local decoder buffer overflow.
[0030] The local decoder may wait in an idle state, but output black or repeat frames from the video port. The clock time may be input from the network. The local timestamp stream may be sent to the local decoder several frames ahead of the switch time. The local decoder may start to decode the transport and have the first output frame ready to transmit. At that time, the local decoder may stop decoding until that first frame is transmitted. The local data input may signal the local decoder and related hardware at least 10ms early to begin outputting the local video. The local data input may send the splice timestamp value for that frame. The transcoder may embed the network presentation timestamp and a switching flag in the vertical blanking interval. The transcoder may embed the local presentation timestamp in the vertical blanking interval.
[0031] A field programmable gate array may perform the video switch. The field programmable gate array may execute the switch based on a switching flag embedded in the network video vertical blanking interval. The field programmable gate array may embed the network presentation timestamp, the local presentation timestamp and the switching flag in the vertical blanking interval. The transcoder may place the network presentation timestamp and local presentation timestamp in a packetized elementary stream stuffing in the packetized elementary stream header.
[0032] Output hardware may not detect any changes in the timestamp offset since the network presentation timestamp originated from the network feed and the transcoder presentation timestamp continues uninterrupted.
[0033] For audio, the switch may occur on the audio frame in which the presentation timestamp switch time is greater than the network audio presentation timestamp value. The field programmable gate array may monitor the switching flag in the video to determine the presentation timestamp switch time. The local presentation timestamp values may be replaced with the network presentation timestamp value, incorporating the timestamp offset. Note that the audio presentation timestamp values may not be the same values as the video presentation timestamp since audio and video have different frame rates.
[0034] Since the local timestamp originated from a disk, the audio timestamp rate may occur in bursts. The hardware may read the audio rate in the audio header and throttle the output of the audio packets at that audio rate plus a percentage to accommodate for the timestamp, the packetized elementary stream headers, the packetized elementary stream alignment, and others.
[0035] An audio switch may be executed as a "dirty switch." In the "dirty switch," the last network audio frame before the switch may be cut off and the first audio packet of the switch may contain an audio frame fragment. The "dirty switch" may result in an audible noise being heard.
[0036] An audio switch may be executed as a "seamless switch." The "seamless switch" may be a switch like ad-insert. The video processing system may complete last network audio frame. The video processing system may insert mute frames to keep the audio buffer from becoming empty. The video processing system may keep the presentation timestamp continuous. The first audio packet of the switch may contain a packetized elementary stream header.
[0037] An audio switch may be executed as a "restart switch." In the "restart switch," the video processing system may complete the last network audio frame before the switch. The video processing system may replace any fragment of the next frame with adaptation stuffing with the last packet containing that frame. The first audio packet of the switch may contain a packetized elementary stream header. The video processing system may empty, but receive intact audio frames.
[0038] The last audio frame prior to the splice time may be when the presentation timestamp of that frame is less than 32ms before the splice presentation timestamp time. When the last audio frame is detected, the last packet of that frame may be modified.
[0039] The field programmable gate array may compute the presentation timestamp of the local audio stream. The first packet may contain a data fragment from the previous audio frame. The video processing system may replace the partial frame with adaptation stuffing.
[0040] In some instances, each audio frame may not contain a packetized elementary stream header. The field programmable gate array may wait until the packetized elementary stream arrives before performing the audio switch. The field programmable gate array may leave a gap in audio of up to 700ms, the maximum packetized elementary stream period. In practice, the maximum packetized elementary stream period may be about 200ms.
[0041] If this is not acceptable, the field programmable gate array may track the presentation timestamp for the audio sync (synchronization) frames so that the switch may occur at the correct audio frame. The field programmable gate array may insert a packetized elementary stream header. The field programmable gate array may re- packetize subsequent audio frames.
[0042] The transcoder may signal the field programmable gate array at least 10 ms early to switch back to the primary video or audio. The field programmable gate array may monitor the switching flag in the network vertical blanking interval, perform the video switch, and place the network presentation timestamp in the output video vertical blanking interval. The network and local video output may be clock and frame synchronized. The output hardware may not detect any changes in the timestamp offset since the local video output embedded the network presentation timestamp in the vertical blanking interval and the network presentation timestamp continues uninterrupted.
[0043] Upon each power-cycle, the video delay through the field programmable gate array may change by a frame. The audio delay may be changed accordingly. The field programmable gate array may measure this variable delay. The transcoder may add the variable delay value to each audio presentation timestamp as the audio data set arrives. The field programmable gate array may force the audio to increase the delay time by the variable delay. [0044] Referring now to Figure 1, a block diagram is provided illustrating an electronic device 100, capable of acting as a video processing system, in accordance with an embodiment of the present invention. For some embodiments of the present invention, the electronic device 100 may also support one or more applications for performing various communications with a network. Device software may be written in any programming language, such as C, C++, Java or Visual Basic, for example. Device software may run on an application framework, such as, for example, a Java® server or .NET ® framework
[0045] The electronic device 100 may include a transceiver 102, which is capable of sending and receiving data over the network.
[0046] The electronic device 100 may include a processor, or controller, 104 that executes stored programs. The controller/processor 104 may be any programmed processor known to one of skill in the art. However, the disclosed method may also be implemented on a general-purpose or a special purpose computer, a programmed microprocessor or microcontroller, peripheral integrated circuit elements, an application- specific integrated circuit or other integrated circuits, hardware/electronic logic circuits, such as a discrete element circuit, a programmable logic device, such as a programmable logic array, field programmable gate-array, or the like. In general, any device or devices capable of implementing the disclosed method as described herein may be used to implement the disclosed system functions of this invention.
[0047] The electronic device 100 may also include a volatile memory 106 and a nonvolatile memory 108 to act as data storage for the processor 104. The volatile memory 106 or the non- volatile memory 108 may include one or more electrical, magnetic or optical memories such as a random access memory (RAM), cache, hard drive, or other memory device. The volatile memory 106 may have a cache to speed access to specific data. The non- volatile memory 108 may also be connected to a compact disc - read only memory (CD-ROM), digital video disc - read only memory (DVD-ROM), DVD read write input, tape drive, or other removable memory device that allows media content to be directly uploaded into the system. [0048] The electronic device 100 may include a user input interface 110 that may comprise elements such as a keypad, display, touch screen, and the like. The electronic device 100 may also include a user output device that may comprise a display screen and an audio interface 112 that may comprise elements such as a microphone, earphone, and speaker. The electronic device 100 also may include a component interface 114 to which additional elements may be attached, for example, a universal serial bus (USB) interface or a geographical positioning system (GPS). Finally, the electronic device 100 may include a power supply 116.
[0049] Referring now to Figure 2, a block diagram is provided illustrating a video processing system 200 in accordance with an embodiment of the present invention. A video transcoder 202 may receive a compressed video data frame from a network video channel input 204, optionally via an input counter 214. The video transcoder 202 may decode the compressed video data frame and re-encode the compressed video data frame into a target video data format. A video transcoder 202 may have an audio data bypass 206 that receives an audio data set associated with the compressed video data frame via a network audio channel input 208. A field programmable gate array 210 may mix the transcoded video data frame with the associated audio data set for presentation to a user.
[0050] The video transcoder 202 may receive a network presentation timestamp from the network video channel input 204 along with the compressed video data frame. The video transcoder 202 may decode the compressed video data frame and determine the network presentation timestamp. The video transcoder 202 may embed the network presentation timestamp in the compressed video data frame. The video transcoder 202 may embed the network presentation timestamp in a vertical blanking interval. The video transcoder 202 may re-encode the compressed video data frame. The video transcoder 202 may embed the network presentation timestamp in the video frame header. The field programmable gate array 210 may receive the transcoded compressed video data frame and a transcoder timestamp. The field programmable gate array 210 may compare the network presentation timestamp to the transcoder presentation timestamp to determine the timestamp offset. [0051] To properly mix the transcoded video data frame with the associated audio data set, the video processing system may determine a program clock reference offset. An input counter 216 may track an input program clock reference and an output counter 218 may track an output program clock reference. The audio data bypass 212 may delay the associated audio data set based on the input program clock reference and the output program clock reference. The field programmable gate array 210 may subtract the input program clock reference from the output program clock reference to determine a program clock reference offset. An audio delay 212 may use the program clock reference offset to delay the associated audio data set to match the compressed video data frame.
[0052] A local video channel input 218 may receive a local video stream for insertion after the compressed video data frame using a local offset. The local video stream may be sent to the video transcoder 202. A local audio channel input 220 may receive an associated local audio stream for insertion after the compressed video data frame using a network presentation timestamp. The audio data stream may go to the audio data bypass 206. A local data input channel 222 may feed a splice timestamp to the video transcoder, indicating the time that the local video stream and local audio stream is to be inserted. The local video channel input 218 and the local audio channel input 220 may be attached to a network or a local hard drive. The field programmable gate array 210 may make the insertion upon determining a switch frame using the program clock reference and the splice timestamp. The field programmable gate array 210 may switch to a compressed audio data set using the network presentation timestamp. The field programmable gate array 210 may switch to a pulse code modulated audio data set using the network presentation timestamp. The field programmable gate array 210 may determine a local offset by subtracting a local video presentation timestamp from the network presentation timestamp. The local offset may be added to the local audio presentation timestamps to synchronize the local audio presentation timestamps to the network presentation timestamp.
[0053] Referring now to Figure 3, a flowchart is provided that illustrates a method 300 performed by the video processing system 200 for processing a compressed video data frame in accordance with an embodiment of the present invention. The video processing system 200 may receive a compressed video data frame (Block 302). The video processing system 200 may decode the compressed video data frame in a video transcoder 202 creating a decoded video data frame (Block 304). The video processing system 200 may re-encode the decoded video data frame in the video transcoder 202 creating a transcoded video data frame (Block 306). The video processing system 200 may determine a timestamp offset (Block 308). The video processing system 200 may apply the timestamp offset to a program clock reference, program presentation timestamp, and program decode timestamp to match the output program clock reference and the timestamps to the input program clock reference and timestamps (Block 310). The video processing system 200 may bypass an associated audio data set around the video transcoder 202 (Block 312). The video processing system 200 may delay the associated audio dataset based on a program clock reference offset, determined from an input program clock reference and an output program clock reference (Block 314). The video processing system 200 may delay the associated audio dataset to match the transcoded video data frame (Block 316).
[0054] If the video processing system identifies a switch frame using the splice timestamp (Block 318), the video processing system 200 may insert a local video stream using the splice timestamp (Block 320). The video processing system 200 may switch to a compressed audio data set using the splice timestamp (Block 322). The compressed audio data set may be a pulse code modulated audio data set.
[0055] Referring now to Figure 4, a flowchart is provided that illustrates a method 400 performed by the video processing system 200 for determining a timestamp offset using a network presentation timestamp in accordance with an embodiment of the present invention. The video processing system 200 may decode a compressed video data frame in a video transcoder 202 creating a decoded video data frame (Block 402). The video processing system 200 may embed a network presentation timestamp from a video frame header in the compressed video data frame in a vertical blanking interval of the decoded video data frame (Block 404). The video processing system 200 may re-encode the compressed video data frame in the video transcoder 202 creating a transcoded video data frame (Block 406). The video processing system 200 may embed the network presentation timestamp in a video frame header of the transcoded video data frame (Block 408). The video processing system 200 may compare the network presentation timestamp with a transcoder presentation timestamp (Block 410). The video processing system 200 may calculate a timestamp offset (Block 412).
[0056] Referring now to Figure 5, a flowchart is provided that illustrates a method 500 performed by the video processing system 200 for determining an audio pass through delay using multiple counters in accordance with an embodiment of the present invention. The video processing system 200 may use an input counter 216 to track an input program clock reference (PCR) (Block 502). The video processing system 200 may use an output counter 218 to track an output program clock reference (Block 504). The video processing system 200 may calculate the difference between the output program clock reference and the input program clock reference to determine the audio pass through delay (Block 506).
[0057] Embodiments within the scope of the present invention may also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media.
[0058] Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network.
[0059] Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer- executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, objects, components, and data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
[0060] Although the above description may contain specific details, they should not be construed as limiting the claims in any way. Other configurations of the described embodiments of the invention are part of the scope of this invention. For example, the principles of the invention may be applied to each individual user where each user may individually deploy such a system. This enables each user to utilize the benefits of the invention even if any one of the large number of possible applications do not need the functionality described herein. In other words, there may be multiple instances of the electronic devices each processing the content in various possible ways. It does not necessarily need to be one system used by all end users. Accordingly, the appended claims and their legal equivalents should only define the invention, rather than any specific examples given.

Claims

CLAIMS We claim:
1. A machine-implemented method, comprising:
decoding a compressed video data frame in a video transcoder creating a decoded video data frame;
embedding a network presentation timestamp in the decoded video data frame; re-encoding the decoded video data frame in the video transcoder creating a transcoded video data frame; and
comparing the network presentation timestamp with a transcoder presentation timestamp to determine a timestamp offset.
2. The method of claim 1, further comprising:
embedding the network presentation timestamp in a vertical blanking interval.
3. The method of claim 1, further comprising:
embedding the network presentation timestamp in a video frame header.
4. The method of claim 1, further comprising:
bypassing an associated audio data set around the video transcoder.
5. The method of claim 4, further comprising:
delaying the associated audio data set to match the transcoded video data frame.
6. The method of claim 1, further comprising:
tracking an input program clock reference with an input counter; and
tracking an output program clock reference with an output counter.
7. The method of claim 6, further comprising:
delaying an associated audio data set based on the input program clock reference and the output program clock reference.
8. The method of claim 1, further comprising:
inserting a local video stream using a splice timestamp.
9. The method of claim 1, further comprising:
identifying a switch frame using a splice timestamp.
10. The method of claim 1, further comprising:
switching to a compressed audio data set using a splice timestamp.
11. The method of claim 1 , further comprising:
switching to a pulse code modulated audio data set using a splice timestamp.
12. A video processing system comprising:
a network video channel input that receives a compressed video data frame with a network presentation timestamp;
a video transcoder that decodes the compressed video data frame creating a decoded video data frame, embeds the network presentation timestamp in the decoded video data frame, and re-encodes the decoded video data frame creating a transcoded video data frame.
13. The video processing system of claim 12, wherein the video transcoder embeds the network presentation timestamp in a vertical blanking interval.
14. The video processing system of claim 12, wherein the video transcoder embeds the network presentation timestamp in a header stuffing.
15. The video processing system of claim 12, further comprising:
an audio data bypass that receives an associated audio data set around the video transcoder.
16. The video processing system of claim 15, wherein the audio data bypass delays the associated audio data set to match the compressed video data frame.
17. The video processing system of claim 12, further comprises:
a field programmable gate array that compares the network presentation timestamp with a transcoder presentation timestamp to determine a timestamp offset.
18. The video processing system of claim 12, further comprising:
a local video channel input that receives a local video stream for insertion after the compression video data frame.
19. A electronic device, comprising:
a network video channel input that receives a compressed video data frame;
a video transcoder that decodes the compressed video data frame and re-encodes the compressed video data frame.
an input counter that tracks an input program clock reference; and
an output counter that tracks an output program clock reference.
20. The electronic device of claim 19, further comprising:
an audio data bypass that delays an associated audio data set based on the input program clock reference and the output program clock reference.
EP12766810.1A 2011-10-04 2012-09-17 Method to match input and output timestamps in a video encoder and advertisement inserter Active EP2752023B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/252,753 US20130083859A1 (en) 2011-10-04 2011-10-04 Method to match input and output timestamps in a video encoder and advertisement inserter
PCT/US2012/055753 WO2013052265A1 (en) 2011-10-04 2012-09-17 Method to match input and output timestamps in a video encoder and advertisement inserter

Publications (2)

Publication Number Publication Date
EP2752023A1 true EP2752023A1 (en) 2014-07-09
EP2752023B1 EP2752023B1 (en) 2019-12-25

Family

ID=46964075

Family Applications (1)

Application Number Title Priority Date Filing Date
EP12766810.1A Active EP2752023B1 (en) 2011-10-04 2012-09-17 Method to match input and output timestamps in a video encoder and advertisement inserter

Country Status (5)

Country Link
US (1) US20130083859A1 (en)
EP (1) EP2752023B1 (en)
CN (1) CN103959807B (en)
BR (1) BR112014008248B1 (en)
WO (1) WO2013052265A1 (en)

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9838450B2 (en) 2010-06-30 2017-12-05 Brightcove, Inc. Dynamic chunking for delivery instances
US9762639B2 (en) * 2010-06-30 2017-09-12 Brightcove Inc. Dynamic manifest generation based on client identity
US9148707B2 (en) * 2012-07-13 2015-09-29 Lodgenet Interactive Corporation System and method to provide out-of-band broadcast trigger synchronization and communication to insertion devices
US9635334B2 (en) * 2012-12-03 2017-04-25 Avago Technologies General Ip (Singapore) Pte. Ltd. Audio and video management for parallel transcoding
US20140198839A1 (en) * 2013-01-17 2014-07-17 Nvidia Corporation Low latency sub-frame level video decoding
US8922713B1 (en) * 2013-04-25 2014-12-30 Amazon Technologies, Inc. Audio and video synchronization
US9332047B2 (en) 2013-09-30 2016-05-03 Brightcove Inc. Dynamic chunk manipulation for streaming mixed live and on-demand media: dynamic permutation layer
CN103686203A (en) * 2013-12-04 2014-03-26 深圳市融创天下科技股份有限公司 Video transcoding method and device
CN104954630B (en) * 2014-03-28 2018-01-12 深圳市海派尔科技开发有限公司 Video time stamp acquisition methods, video processing equipment and video system
CN105338281B (en) * 2014-06-27 2018-07-31 阿里巴巴集团控股有限公司 A kind of image display method and device
US10148989B2 (en) * 2016-06-15 2018-12-04 Divx, Llc Systems and methods for encoding video content
CN106572360A (en) * 2016-11-11 2017-04-19 协创数据技术股份有限公司 Advertisement intercutting device applied to video live streaming
EP3396964B1 (en) 2017-04-25 2020-07-22 Accenture Global Solutions Ltd Dynamic content placement in a still image or a video
EP3528196A1 (en) * 2018-02-16 2019-08-21 Accenture Global Solutions Limited Dynamic content generation
US10742955B2 (en) * 2018-05-21 2020-08-11 Microsoft Technology Licensing, Llc Correlation of video stream frame timestamps based on a system clock
US11051057B2 (en) 2019-06-24 2021-06-29 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to establish a time offset, to facilitate taking content-related action
US11234049B2 (en) * 2019-06-24 2022-01-25 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to control implementation of dynamic content modification
CN110349564B (en) * 2019-07-22 2021-09-24 思必驰科技股份有限公司 A method and device for cross-language speech recognition
MX2022013497A (en) * 2020-04-28 2023-01-16 Arris Entpr Llc System for jitter recovery from a transcoder.
WO2021221945A1 (en) 2020-04-28 2021-11-04 Arris Enterprises Llc System for presentation time stamp recovery from a transcoder
CN113949898B (en) * 2021-10-13 2024-03-08 北京奇艺世纪科技有限公司 Multimedia processing methods, devices, equipment and storage media

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001251616A (en) * 2000-03-02 2001-09-14 Media Glue Corp Method and device for converting multiplexed sound/ moving picture compressing-coded signal, and medium recorded with conversion program
US7548565B2 (en) * 2000-07-24 2009-06-16 Vmark, Inc. Method and apparatus for fast metadata generation, delivery and access for live broadcast program
US6724825B1 (en) * 2000-09-22 2004-04-20 General Instrument Corporation Regeneration of program clock reference data for MPEG transport streams
US7068719B2 (en) * 2001-06-01 2006-06-27 General Instrument Corporation Splicing of digital video transport streams
JP2003230092A (en) * 2002-02-04 2003-08-15 Sony Corp Information processing apparatus and method, program storage medium, and program
US7499469B2 (en) * 2003-01-13 2009-03-03 Broadcom Corporation Method and system for generating digital video broadcast (DVB) transport stream from direct satellite system (DSS) transport stream
KR20050052717A (en) * 2003-12-01 2005-06-07 엘지전자 주식회사 System and method for transcording stream data except audio data
US7454019B2 (en) * 2003-12-05 2008-11-18 Motion Picture Association Of America System and method for controlling display of copy-never content
US8879635B2 (en) * 2005-09-27 2014-11-04 Qualcomm Incorporated Methods and device for data alignment with time domain boundary
US8111971B2 (en) * 2007-12-05 2012-02-07 Cisco Technology, Inc. Systems and methods of reducing media stream delay through independent decoder clocks

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2013052265A1 *

Also Published As

Publication number Publication date
BR112014008248B1 (en) 2022-09-20
US20130083859A1 (en) 2013-04-04
BR112014008248A2 (en) 2017-04-11
WO2013052265A1 (en) 2013-04-11
EP2752023B1 (en) 2019-12-25
CN103959807A (en) 2014-07-30
CN103959807B (en) 2019-06-04

Similar Documents

Publication Publication Date Title
EP2752023B1 (en) Method to match input and output timestamps in a video encoder and advertisement inserter
EP2038886B1 (en) Clock drift compensation techniques for audio decoding
KR101535827B1 (en) Apparatus and method for synchronizing audio and video in a wireless communication network
US9832507B2 (en) System and method for synchronizing media output devices connected on a network
US9489980B2 (en) Video/audio synchronization apparatus and video/audio synchronization method
US9549027B2 (en) Network-synchronized media playback
CN100588261C (en) Method and system for synchronizing video data and audio data
US8931024B2 (en) Receiving apparatus and subtitle processing method
CN102057687B (en) Synchronization of media stream components
US11178451B2 (en) Dynamic playout of transition frames while transitioning between playout of media streams
US7760770B2 (en) Stream data processing apparatus and stream data processing method
WO2009076180A3 (en) Reducing media stream delay tgrough independent decoder clocks
US20040264577A1 (en) Apparatus and method for controlling the synchronization of a video transport stream
CN100502473C (en) Apparatus and method for coordinating synchronization of video and captions
US20170055045A1 (en) Recovering from discontinuities in time synchronization in audio/video decoder
KR101600891B1 (en) Synchronization method and system for audio and video of a plurality terminal
CN103491430A (en) Streaming media data processing method and electronic device
JP2010521089A (en) Transcoder media time conversion
CN102075803A (en) Method for synchronously playing video and audio
WO2014128360A1 (en) Synchronization of audio and video content
CN103581730A (en) Method for achieving synchronization of audio and video on digital set top box
KR100864009B1 (en) Audio / Video Synchronization Method
US7961792B2 (en) Robust system for maintaining audio/video synchronization during playback of multimedia streams with no embedded time stamps
TW200629918A (en) Transcoding closed captioning data from broadcast DTV onto DVD
JP2004248104A (en) Information processing apparatus and information processing method

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20140331

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: GOOGLE TECHNOLOGY HOLDINGS LLC

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20170110

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190809

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602012066725

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1218464

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200115

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200326

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200520

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200425

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602012066725

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1218464

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

26N No opposition filed

Effective date: 20200928

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20200917

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200917

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200930

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200930

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200917

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200930

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200917

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230510

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250929

Year of fee payment: 14