US20230370691A1 - Systems and methods for determining delay of a plurality of media streams - Google Patents
Systems and methods for determining delay of a plurality of media streams Download PDFInfo
- Publication number
- US20230370691A1 US20230370691A1 US18/319,704 US202318319704A US2023370691A1 US 20230370691 A1 US20230370691 A1 US 20230370691A1 US 202318319704 A US202318319704 A US 202318319704A US 2023370691 A1 US2023370691 A1 US 2023370691A1
- Authority
- US
- United States
- Prior art keywords
- source
- destination
- time
- processor
- stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000005540 biological transmission Effects 0.000 claims abstract description 43
- 230000008569 process Effects 0.000 claims description 7
- 241000023320 Luma <angiosperm> Species 0.000 claims description 4
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 description 29
- 238000004891 communication Methods 0.000 description 11
- 230000001360 synchronised effect Effects 0.000 description 11
- 230000001934 delay Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000004519 manufacturing process Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64723—Monitoring of network processes or resources, e.g. monitoring of network load
- H04N21/64738—Monitoring network characteristics, e.g. bandwidth, congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
- H04L43/0858—One way delays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64784—Data processing by the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Definitions
- the described embodiments relate to determining delay of a plurality of media streams, and in particular to determining transmission times and relative synchronization errors.
- VANC vertical ancillary data
- HANC horizontal ancillary data
- each of the streams is typically a separate stream, in the sense that the information for one stream is not embedded in another stream. This is in contrast to SDI transmission, in which audio and ancillary data is embedded in non-visible portions of a video signal.
- Media streams can originate from different sources and may, as a result, be out of sync with one another. In some cases, media streams can originate from the same source but may still be out of sync with each other. For example, a video stream may be “running ahead” or “running behind” a corresponding audio stream, resulting in lip-sync errors. Furthermore, when media streams are transmitted over a network, the media streams can travel via different network paths, or be processed by different intermediate devices. As a result, the media streams may arrive at a downstream device at different times, resulting in further desynchronization. Accordingly, it may be desirable to determine transmission times and relative synchronization errors.
- some embodiments provide a system for determining delay of a plurality of media streams.
- the system includes a source processor and a destination processor.
- the source processor is configured to generate a series of source time packets; and transmit, through a network, the series of source time packets as a source packet stream.
- Each source time packet includes source time data and source signature data.
- the source time data corresponds to a first time when the source time packet is generated.
- the source signature data corresponds to characteristic features of each of the plurality of media streams.
- the destination processor is configured to generate a series of destination time packets; receive, through the network, the source packet stream; determine a transmission time for the source packet stream based on the source time data and the destination time data; and determine a relative synchronization error based on the source signature data and the destination signature data.
- Each destination time packet includes destination time data and destination signature data.
- the destination time data corresponds to a second time when the destination time packet is generated.
- the destination signature data corresponds to characteristic features of each of the plurality of media streams.
- the source packet stream is transmitted in-band with the plurality of media streams.
- the source packet stream is transmitted out-of-band from the plurality of media streams.
- the source time data and the destination time data is generated using PTP (Precision Time Protocol).
- the source processor is further configured to transmit, through the network, the plurality of media streams.
- the network includes at least one processing device configured to process at least one media stream of the plurality of media streams.
- the destination processor is further configured to receive the plurality of media streams.
- the source time data and the destination time data further include a clock signal.
- the source packet stream is transmitted synchronously.
- the source packet stream is transmitted asynchronously.
- the characteristic features include at least one of: an average luma value, an average color value, an average motion distance, and a contrast level.
- the characteristic features include at least one of: an envelope of signal amplitude, an average loudness level, a peak formant, and an average zero crossing rate.
- the plurality of media streams include at least one of: a video stream, an audio stream, and a metadata stream.
- some embodiments provide a system for determining delay of a plurality of media streams.
- the system includes a source processor, a destination processor, and an analysis processor.
- the source processor configured to generate a series of source time packets; and transmit, through a network, the series of source time packets as a source packet stream.
- Each source time packet includes source time data and source signature data.
- the source time data corresponds to a first time when the source time packet is generated.
- the source signature data corresponds to characteristic features of each of the plurality of media streams.
- the destination processor is configured to generate a series of destination time packets; and transmit, through the network, the series of destination time packets as a destination packet stream.
- Each destination time packet includes destination time data and destination signature data.
- the destination time data corresponds to a second time when the destination time packet is generated.
- the destination signature data corresponds to characteristic features of each of the plurality of media streams.
- the analysis processor is configured to receive, through the network, the source packet stream and the destination packet stream; determine a transmission time for at least one of the source packet stream and the destination packet stream based on at least one of the source time data and the destination time data; and determine a relative synchronization error based on the source signature data and the destination signature data.
- some embodiments provide a method for determining delay of a plurality of media streams.
- the method involves generating, at a source processor, a series of source time packets; transmitting, at the source processor, through a network, the series of source time packets as a source packet stream; generating, at a destination processor, a series of destination time packets; receiving, at the destination processor, through the network, the source packet stream; determining, at the destination processor, a transmission time for the source packet stream based on the source time data and the destination time data; and determining, at the destination processor, a relative synchronization error based on the source signature data and the destination signature data.
- Each source time packet includes source time data and source signature data.
- the source time data corresponds to a first time when the source time packet is generated.
- the source signature data corresponds to characteristic features of each of the plurality of media streams.
- Each destination time packet includes destination time data and destination signature data.
- the destination time data corresponds to a second time when the destination time packet is generated.
- the destination signature data corresponds to characteristic features of each of the plurality of media streams.
- the source packet stream is transmitted in-band with the plurality of media streams.
- the source packet stream is transmitted out-of-band from the plurality of media streams.
- the source time data and the destination time data is generated using PTP (Precision Time Protocol).
- the source time data and the destination time data further include a clock signal.
- the method further involves transmitting, at the source processor, through the network, the plurality of media streams; processing, at at least one processing device the network, at least one media stream of the plurality of media streams; and receiving, at the destination processor, the plurality of media streams.
- the source packet stream is transmitted synchronously.
- the source packet stream is transmitted asynchronously.
- the characteristic features include at least one of: an average luma value, an average color value, an average motion distance, and a contrast level.
- the characteristic features include at least one of: an envelope of signal amplitude, an average loudness level, a peak formant, and an average zero crossing rate.
- the plurality of media streams include at least one of: a video stream, an audio stream, and a metadata stream.
- some embodiments provide a method for determining delay of a plurality of media streams.
- the method involves generating, at a source processor, a series of source time packets; transmitting, at the source processor, through a network, the series of source time packets as a source packet stream; generating, at a destination processor, a series of destination time packets; transmitting, at the destination processor, through a network, the series of destination time packets as a destination packet stream; receiving, at an analysis processor, through the network, the source packet stream and the destination packet stream; determining, at the analysis processor, a transmission time for at least one of the source packet stream and the destination packet stream based on at least one of the source time data and the destination time data; and determining, at the analysis processor, a relative synchronization error based on the source signature data and the destination signature data.
- Each source time packet includes source time data and source signature data.
- the source time data corresponds to a first time when the source time packet is generated.
- the source signature data corresponds to characteristic features of each of the plurality of media streams.
- Each destination time packet includes destination time data and destination signature data.
- the destination time data corresponds to a second time when the destination time packet is generated.
- the destination signature data corresponds to characteristic features of each of the plurality of media streams.
- FIG. 1 is a block diagram of a system for determining delay of a plurality of media streams, in accordance with at least one embodiment
- FIG. 2 is a block diagram of a system for determining delay of a plurality of media streams, in accordance with at least one embodiment
- FIGS. 3 A and 3 B are illustrations of a plurality of media streams, source time packets, and destination time packets, in accordance with at least one embodiment
- FIG. 4 is a block diagram of a processor, in accordance with at least one embodiment
- FIG. 5 is a block diagram of a packet, in accordance with at least one embodiment
- FIG. 6 is a flowchart of a method for determining delay of a plurality of media streams, in accordance with at least one embodiment.
- FIG. 7 is a flowchart of a method for determining delay of a plurality of media streams, in accordance with at least one embodiment.
- the wording “and/or” is intended to represent an inclusive-or. That is, “X and/or Y” is intended to mean X or Y or both, for example. As a further example, “X, Y, and/or Z” is intended to mean X or Y or Z or any combination thereof.
- the term “coupled” used herein indicates that two elements can be directly coupled to one another or coupled to one another through one or more intermediate elements.
- body typically refers to the body of a patient, a subject or an individual who receives the ingestible device.
- the patient or subject is generally a human or other animal.
- the embodiments of the systems and methods described herein may be implemented in hardware or software, or a combination of both. These embodiments may be implemented in computer programs executing on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface.
- the programmable computers may be a server, network appliance, embedded device, computer expansion module, a personal computer, laptop, personal data assistant, cellular telephone, smart-phone device, tablet computer, a wireless device or any other computing device capable of being configured to carry out the methods described herein.
- the communication interface may be a network communication interface.
- the communication interface may be a software communication interface, such as those for inter-process communication (IPC).
- IPC inter-process communication
- there may be a combination of communication interfaces implemented as hardware, software, and combination thereof.
- Program code may be applied to input data to perform the functions described herein and to generate output information.
- the output information is applied to one or more output devices, in known fashion.
- Each program may be implemented in a high level procedural or object oriented programming and/or scripting language, or both, to communicate with a computer system.
- the programs may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language.
- Each such computer program may be stored on a storage media or a device (e.g. ROM, magnetic disk, optical disc) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.
- Embodiments of the system may also be considered to be implemented as a non-transitory computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
- system, processes and methods of the described embodiments are capable of being distributed in a computer program product comprising a computer readable medium that bears computer usable instructions for one or more processors.
- the medium may be provided in various forms, including one or more diskettes, compact disks, tapes, chips, wireline transmissions, satellite transmissions, internet transmission or downloadings, magnetic and electronic storage media, digital and analog signals, and the like.
- the computer useable instructions may also be in various forms, including compiled and non-compiled code.
- FIG. 1 illustrates a block diagram of system 100 for determining delay of a plurality of media streams 110 , in accordance with at least one embodiment.
- System 100 includes source processor 102 , destination processor 104 , and network 108 .
- Source processor 102 is connected to destination processor 104 via network 108 .
- Various data can be transmitted from source processor 102 to destination processor 104 across network 108 .
- Source processor 102 and destination processor 104 can be any suitable processors, controllers, digital signal processors, graphics processing units, application specific integrated circuits (ASICs), and/or field programmable gate arrays (FPGAs) that can provide sufficient processing power depending on the configuration, purposes and requirements of the system 100 .
- source processor 102 and destination processor 104 can include more than one processor with each processor being configured to perform different dedicated tasks.
- Source processor 102 can be connected to one or more source devices (not shown) that generate media content.
- the source devices may be cameras, microphones, or other devices for generating video, audio, or metadata content.
- Source processor 102 can receive media content from the source devices and generate media streams 110 .
- source processor 102 can receive media streams 110 from the source devices and does not generate media streams 110 .
- source processor 102 is a source device.
- source processor 102 can be connected to one or more other processing devices (not shown) that transmit media streams 110 to source processor 102 .
- Each stream of media streams 110 can include video, audio, or metadata content. In some embodiments, each stream includes only one type of content. In other embodiments, each stream can include more than one type of content.
- a media stream that includes video, audio, or metadata may be referred to as a video stream, audio stream, or metadata stream, respectively.
- each stream of media streams 110 is packetized. That is, the data within each stream is formatted as a plurality of packets. Accordingly, each media stream can include a plurality of media packets, each video stream can include a plurality of video packets, each audio stream can include a plurality of audio packets, and each metadata stream can include a plurality of metadata packets. It will be appreciated that although only three media streams 110 are shown, there can be any number of media streams 110 .
- Source processor 102 can transmit media streams 110 through network 108 to destination processor 104 .
- media streams 110 are transmitted by source processor 102 using a synchronous communication standard, such as SDI (Serial Digital Interface).
- media streams 110 are transmitted using an asynchronous communication standard, such as IP (Internet Protocol).
- IP Internet Protocol
- media streams 110 are transmitted in a steady stream. In some cases, media streams 110 are transmitted intermittently.
- Network 108 can include various network paths (not shown) through which data, such as media streams 110 , can be routed.
- the network paths can include various switches and intermediate processing devices.
- the switches can selectively reconfigure one or more network paths to change the routing of media streams 110 .
- the switches can route a stream from the source processor 102 , to one or more intermediate processing devices, to the destination processor 104 .
- the intermediate processing devices can process one or more of media streams 110 .
- an intermediate processing device can process a video stream to adjust various characteristics of the video content, such as resolution, color, contrast, brightness, orientation, level of compression, etc.
- an intermediate processing device may process an audio stream to adjust characteristics of the audio content, such as equalization, level of compression, etc.
- An intermediate processing device may also process a metadata stream to add new metadata, or remove or modify existing metadata.
- Destination processor 104 can receive media streams 110 from source processor 102 , through network 108 .
- destination processor 104 can buffer one or more of media streams 110 . That is, destination processor 104 can temporarily store data from one or more streams in a memory (not shown). For example, media streams 110 received at different times or at different rates can be buffered by destination processor 104 for later processing.
- Destination processor 104 can be connected to one or more downstream devices (not shown). For example, destination processor 104 can be connected to a video production system. Destination processor 104 can transmit media streams 110 to the video production system, which can output media streams 110 . For example, a video production system can display video streams on one or more monitors or play audio streams on one or more speakers. In some cases, the video production system can be used to facilitate production of a television broadcast.
- media streams 110 may be out of sync or temporally misaligned with respect to each other when they are received by destination processor 104 .
- a video stream may be “running ahead” or “running behind” a corresponding audio stream, resulting in lip-sync errors. This may be caused by the fact the media content was generated by different source devices.
- the desynchronization may be caused by media streams 110 traveling on different network paths or having different intermediate processing.
- system 100 can determine transmission times and relative synchronization errors.
- Source processor 102 can generate a series of source time packets 112 . It will be appreciated that although only three source time packets 114 are shown, source processor 102 can generate any number of source time packets 114 . In some embodiments, source processor 102 can generate a source time packet 114 for each segment of a media stream. For example, for a video stream, the segment may correspond to a video frame. Accordingly, source processor 102 can generate a series of source time packets 112 at the same frequency as a video frame rate of a video stream. In some embodiments, source processor 102 can generate a source time packet 114 for each set of contemporaneous media segments. For example, a set of media segments may include a video frame, an audio segment cotemporaneous with the video frame, and metadata cotemporaneous with the video frame.
- Each source time packet includes source time data.
- Source time data corresponds to the time when the source time packet 114 is generated.
- source time data can include a timestamp identifying when the source time packet 114 was generated. This time may be referred to as a first time.
- each source time packet 114 is generated approximately contemporaneous with the transmission of the source time packet 114 .
- each source time packet 114 is generated approximately contemporaneous with the transmission of a segment of a media stream or with the transmission of a set of cotemporaneous media segments.
- the source time data can correspond to the time when the source time packet 114 or a particular segment of a media stream is transmitted.
- source time data can be generated using a clock which is synchronized throughout system 100 .
- source time data can be generated using PTP (Precision Time Protocol).
- PTP Precision Time Protocol
- PTP can ensure that time values determined at the same time by different devices, in possibly different locations, share a common time.
- Each source time packet 114 further includes source signature data.
- Source signature data corresponds to characteristic features of each of media streams 110 .
- the characteristic features can be used to identify a particular segment of a particular stream.
- the characteristic features may correspond to a particular video frame.
- the characteristic features may include an average luma value, an average color value, an average motion distance, or a contrast level.
- the characteristic feature may include an envelope of signal amplitude, an average loudness level, a peak formant, and an average zero crossing rate.
- the characteristic feature may include a hash value of some or all of the metadata.
- the characteristic features can correspond to a set of cotemporaneous segments of media streams 110 .
- the characteristic features can identify a video frame, an audio segment cotemporaneous with the video frame, and metadata cotemporaneous with the video frame.
- each source time packet 114 can include additional time data, such as a clock signal, to facilitate video network communication.
- additional time data such as a clock signal
- some video transmission standards such as some SDI standards, require a 90 kHz clock to be embedded with video data, on a frame-by-frame basis.
- the 90 kHz clock can be embedded in each source time packet 114 to allow each source time packet 114 to be synchronized with each specific video frame.
- source time packets 114 can be any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is, a source time packet 114 may, in some embodiments, never be assembled as a packet or transmitted.
- Source processor 102 can transmit the series of source time packets 112 as source packet stream 116 through network 108 .
- Source packet stream 116 can be a packetized stream. That is, source packet stream 116 can include data that is formatted in a plurality of packets.
- Source packet stream 116 can be transmitted using a synchronous communication standard or an asynchronous communication standard.
- Source packet stream 116 can be transmitted out-of-band from media streams 110 . That is, source packet stream 116 and media streams 110 are transmitted in separate streams. However, in some embodiments, source packet stream 116 is transmitted in-band with media streams 110 . That is, source packet stream 116 and media streams 110 are transmitted in the same stream. In such embodiments, source packet stream 116 travels along the same network path as one or more media streams. For example, source time packets 114 can be transmitted in the same stream as other video, audio, or metadata packets. In another example, source time packets 114 can be embedded in a metadata packet (such as in VANC) in a metadata stream or media stream. In some embodiments, source packet stream 116 can be transmitted to the same IP address as media streams 110 , but to a different UDP port number.
- Destination processor 104 can receive, through network 108 , source packet stream 116 and media streams 110 . Destination processor 104 can generate a series of destination time packets 118 . In some embodiments, destination processor 104 generates each destination time packet 120 for each segment of a media stream. For example, for a video stream, a segment may correspond to a video frame. In some embodiments, destination processor 104 can generate a source time packet 120 for each set of contemporaneous media segments.
- Each destination time packet 120 includes destination time data, similar to source time packets 114 and source time data. Destination time data corresponds to the time when the destination time packet 120 is generated. This time may be referred to as a second time. In some embodiments, each destination time packet 120 is generated approximately contemporaneous with the reception of each source time packet 114 . In some embodiments, each destination time packet 120 is generated approximately contemporaneous with the reception of each segment of a media stream or each set of cotemporaneous media segments. In some embodiments, the destination time data is generated using PTP. In some embodiments, the destination time data can include a clock signal.
- Each destination time packet 120 also includes destination signature data, similar to source time packets 114 and source signature data.
- Destination signature data corresponds to characteristic features of each of the media streams 110 .
- the characteristic features can be similar as those described source time packets 114 .
- the characteristic features can correspond to a set of cotemporaneous segments of media streams 110 .
- destination time packets 120 may refer to any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is, a destination time packet 120 may, in some embodiments, never be assembled as a packet or transmitted.
- Destination processor 104 can determine a transmission time for the source packet stream 116 based on the source time data and the destination time data. For example, destination processor 104 can determine a difference between a first time when a source time packet is generated and a second time when a destination time packet is generated. The source time packet can be generated contemporaneously with the transmission of source packet stream 116 and the destination time packet can be generated contemporaneously with the reception of source packet stream 116 . Accordingly, the difference between the first time and the second time can indicate a transmission time of the source packet stream 116 through network 108 . In some cases, the transmission time of the source packet stream 116 can be substantially equal to the transmission time of one or more of media streams 110 . For example, this may be the case where source packet stream 116 travels along the same network path as one or more of media streams 110 , or where the source time packet is generated approximately cotemporaneous with the transmission of the one or more media stream.
- Destination processor 104 can also determine a relative synchronization error for media streams 110 .
- Relative synchronization error can refer to a difference between the delays of two or more media streams. For example, for an audio stream that was delayed 100 ms and a video stream that was delayed 25 ms, the relative synchronization error is 75 ms. That is, the audio stream is running 75 ms behind the video stream.
- the relative synchronization can be based on relative delays of media streams 110 . That is, the delays are relative to another time, rather than absolute.
- the delay of the streams can be relative to the transmission time for the source packet stream 116 . That is, the delays are relative to the time when the source packet stream 116 is received by destination processor 104 .
- Destination processor 104 can determine the relative synchronization error based on the source signature data and the destination signature data. As discussed above, the source signature data and destination signature data can include characteristic features of the media streams that correspond to particular segments of the media stream. Destination processor 104 can compare the source signature data of each source time packet 114 and destination signature data of each destination time packet 120 . The comparison can be used by destination processor 104 to locate temporal misalignments or relative synchronization errors between media streams. The comparison of source and destination signature data will be described in further detail below with respect to FIGS. 3 A and 3 B . In some embodiments, destination processor 104 can then realign media streams 110 to correct for the synchronization error. In some embodiments, destination processor 104 can determine the transmission time for each of media streams 110 based on the transmission time and the relative synchronization error.
- system 200 for determining delay of a plurality of media streams, in accordance with at least one embodiment. Similar to system 100 of FIG. 1 , system 200 includes source processor 102 , destination processor 104 , and network 108 . However, in contrast to system 100 , system 200 further includes analysis processor 106 .
- source processor 102 can transmit media streams 110 through network 108 .
- Source processor 102 can also generate a series of source time packets 112 , where each source time packet 114 includes source time data and source signature data.
- source processor 102 transmits the series of source time packets 112 as source packet stream 116 to analysis processor 106 (rather than to destination processor 104 ).
- source processor 102 transmits source packet stream 116 to analysis processor 106 through network 108 .
- source processor 102 can transmit source packet stream 116 to destination processor 104 and destination processor 104 can transmit source packet stream 116 to analysis processor 106 .
- destination processor 104 can receive media streams 110 through network 108 .
- Destination processor 104 can also generate a series of destination time packets 118 , where each destination time packet 120 include source time data and source signature data.
- destination processor 104 transmits the series of destination time packets 118 as destination packet stream 122 to analysis processor 106 .
- destination processor 104 transmits destination packet stream 122 to analysis processor 106 through network 108 .
- Analysis processor 106 can be any suitable processors, controllers, digital signal processors, graphics processing units, application specific integrated circuits (ASICs), and/or field programmable gate arrays (FPGAs) that can provide sufficient processing power depending on the configuration, purposes and requirements of the system 200 .
- analysis processor 106 can include more than one processor with each processor being configured to perform different dedicated tasks.
- Analysis processor 106 can receive source packet stream 116 from source processor 102 and receive destination packet stream 122 from destination processor 104 . In some embodiments, analysis processor 106 can receive source packet stream 116 and destination packet stream 122 through network 108 .
- Analysis processor 106 can determine a transmission time for source packet stream 116 based on source time data. For example, analysis processor 106 can compare a first time from the source time data indicating when a source time packet was generated with the time at which the source time packet was received at analysis processor 106 . Since the source time packet can be generated approximately contemporaneously with the transmission of source packet stream 116 , the difference can correspond to the transmission time of the source time packet. Similarly, analysis processor 106 can also determine a transmission time for destination packet stream 122 based on destination time data.
- Analysis processor 106 can also determine a relative synchronization error, based on the source signature data and the destination signature data, in a similar manner as described above with respect to destination processor 106 .
- the relative synchronization error is a difference between relative delays of media streams 110 , where the delays are relative to the transmission time for source packet stream 116 or destination packet stream 122 .
- analysis processor 106 can realign media streams 110 to correct for the relative synchronization error.
- analysis processor 106 can determine a transmission time for media streams 110 based on the transmission time for source packet stream 116 or destination packet stream 122 and relative synchronization error.
- Media streams 110 include video stream 110 v , audio stream 110 a , and metadata stream 110 m .
- Video stream 110 v includes video segments V 1 , V 2 , . . . V n .
- audio stream 110 a includes audio segments A 1 , A 2 , . . . A n
- metadata stream 110 m includes metadata segments M 1 , M 2 , . . . M n . It will be appreciated that although only three media streams are shown, there may be any number of media streams 110 .
- video stream 110 v is aligned temporally with audio stream 110 a and metadata stream 110 m .
- V 1 can correspond to a video frame
- a 1 can correspond to audio cotemporaneous to that video frame
- M 1 can correspond to metadata cotemporaneous to the video frame.
- V 1 is synchronized with A 1 and M 1
- V 2 is synchronized with A 2 and M 2
- V n is synchronized with A n and M n .
- there is no difference in delays between video stream 110 v , audio stream 110 a , and metadata stream 110 m This may be the case, for example, for media streams 110 at source processor 102 of FIGS. 1 and 2 .
- video stream 110 v , audio stream 110 a , and metadata stream 110 m can become misaligned or desynchronized with respect to each other. That is, a difference in delays can develop between one or more of video stream 110 v , audio stream 110 a , and metadata stream 110 m . This may be the case, for example, for the media streams 110 at destination processor 104 of FIGS. 1 and 2 .
- the desynchronization can be caused, for example, when media streams 110 are transmitted through network 108 of FIGS. 1 and 2 .
- video stream 110 v is now no longer synchronized with audio stream 110 a and metadata stream 110 m . That is, V 1 is now cotemporaneous with A 2 and M 0 , instead of A 1 and M 1 , V 2 is now cotemporaneous with A 3 and M 1 , instead of A 2 and M 2 ; and V n is cotemporaneous with A n+1 and M n ⁇ 1 , instead of A n and M n .
- Audio stream 110 a is “running ahead” of video stream 110 v
- metadata stream 110 m is “running behind” video stream 110 v.
- Source time packets 112 include packets ST 1 , ST 2 , . . . ST n .
- Source time packets 112 can be generated, for example, by source processor 102 of FIGS. 1 and 2 .
- a source time packet 114 is generated for each set of cotemporaneous segments of media stream 110 v , 110 a , 110 m .
- source time packet ST 1 is generated for segments A 1 , and M 1 ;
- source time packet ST 2 is generated for segments V 2 , A 2 , and M 2 ;
- source time packet ST n is generated for segments V n , A n , and M n .
- destination time packets 118 include packets DT 1 , DT 2 , . . . DT n and can be generated, for example, by destination processor 104 of FIGS. 1 and 2 .
- a destination time packet 120 is generated for each set of cotemporaneous segments of media stream 110 v , 110 a , 110 m .
- source time packet DT 1 is generated for segments V 1 , A 2 , and M 0
- source time packet DT 2 is generated for segments V 2 , A 3 , and M 1
- source time packet DT n is generated for segments V n , A n+1 , and M n ⁇ 1 .
- Each source time packet ST 1 , ST 2 , . . . ST n includes signature data that corresponds to characteristic features of media streams 110 v , 110 a , 110 m .
- the characteristic features correspond to the respective segments of media streams 110 v , 110 a , 110 m .
- source time packet ST 1 includes signature data corresponding to a characteristic feature of video segment V 1 , audio segment A 1 , and metadata segment A 1 .
- source time packet ST 2 includes signature data corresponding to V 2 , A 2 , and M 2
- source time packet ST n includes signature data corresponding to V n , A n , and M n .
- Each destination time packet DT 1 , DT 2 , . . . DT n also includes signature data that corresponds to characteristic features of media streams 110 v , 110 a , 110 m .
- the characteristic features correspond to the respective segments of media streams 110 v , 110 a , 110 m .
- destination time packet DT 1 includes signature data corresponding to characteristic feature of video segment V 1 , audio segment A 2 , and metadata segment M 0 .
- source time packet DT 2 includes signature data corresponding to V 2 , A 3 , and M 1
- source time packet DT n includes signature data corresponding to media segments V n , A n+1 , and M n ⁇ 1 .
- the signature data of source time packets 112 and destination time packets 118 can be compared to determine a relative synchronization error of media streams 110 v , 110 a , 110 m .
- the signature data of source time packet ST 1 indicates that segment V 1 should be aligned temporally with segments A 1 and M 1 .
- the signature data of destination time packet DT 1 indicates that segment V 1 is aligned temporally with segments A 2 and M 0 . Accordingly, a difference in relative delays between video stream 110 v and audio stream 110 a can be determined based on A 1 and A 2 .
- a relative synchronization error between video stream 110 v and metadata stream 110 m can be determined based on M 0 and M 1 .
- media streams 110 v , 110 a , and 110 m can be realigned or resynchronized, so that V 1 is synchronized with A 1 and M 1 , V 2 is synchronized with A 2 and M 2 , and V n is synchronized with A n and M n .
- the signature data of source time packets 112 and destination time packets 118 may be compared using cross-correlation.
- the signature data of a source time packet may not be identical with the signature data of a destination time packet. This may be the case when intermediate processing is performed on the media streams 110 v , 110 a , 110 m .
- the signature data of source time packets 112 and destination time packets 118 may be cross-correlated to determine relative synchronization errors.
- Processor 402 may be source processor 102 or destination processor 104 of system 100 or system 200 .
- Processor 402 includes signature data generator 404 , time data generator 406 , and packet generator 408 .
- Processor 402 can be any suitable processors, controllers, digital signal processors, graphics processing units, application specific integrated circuits (ASICs), and/or field programmable gate arrays (FPGAs) that can provide sufficient processing power depending on the configuration, purposes and requirements of the system.
- processor 402 can include more than one processor with each processor being configured to perform different dedicated tasks.
- Signature data generator 402 can receive media streams 110 .
- processor 402 receives source signals (not shown) and generates media streams 110 that are received by signature data generator 402 .
- Signature generator 402 can generate signature data based on media streams 110 . As discussed above, signature data corresponds to characteristic features of each of media streams 110 .
- Time data generator 406 can generate time data.
- the time data corresponds to a time when a packet of packets 410 is generated.
- packets 410 are generated approximately contemporaneous with the transmission of packets 410 or media streams 110 .
- the time data can also include a clock signal.
- Packet generator 408 can generate packets 410 that include the signature data and time data. For example, packet generator 408 can generate source time packets 112 and destination time packets 118 of systems 100 and 200 . It will be appreciated that packets 410 can any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is, packets 410 may, in some embodiments, never be assembled as a packet or transmitted.
- Processor 402 can then transmit the generated packets 410 as a packet stream (not shown). For example, processor can transmit the source packet stream 116 or the destination packet stream 122 of FIGS. 1 and 2 . Processor 402 can also transmit the media streams 110 .
- Packet 502 includes time data 504 and signature data 506 .
- packet 502 may be a source time packet 114 or a destination time packet 118 of system 100 or system 200 .
- packet 502 can any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is, packet 502 may, in some embodiments, never be assembled as a packet or transmitted.
- Time data 504 includes time stamp data 508 and clock signal data 510 .
- Time stamp data 508 can include data indicating a time when packet 502 was generated. In some cases, packet 502 is generated approximately contemporaneous with its transmission.
- Clock signal data 510 can include data required by certain video transmissions standards, such as a 90 kHz clock.
- Signature data 506 includes video signature data 512 , audio signature data 514 , and metadata signature data 516 .
- Signature data can include characteristic features of particular segments of one or more media streams.
- Method 600 can be implemented using source processor 102 , destination processor 104 , and network 108 of system 100 .
- Method 600 begins with generating, at a source processor, a series of source time packets at 602 .
- source processor 104 can generate a series of source time packets 112 .
- Each source time packet includes source time data and source signature data.
- the source time data corresponds to a first time when the source time packet is generated.
- the source signature data corresponds to characteristic features of each of the plurality of media streams.
- the series of source time packets is transmitted, at the source processor, as a source packet stream through a network.
- the series of source time packets 112 can be transmitted as source packet stream 116 by source processor 102 through network 108 .
- a series of destination time packets is generated at a destination processor.
- destination processor 104 can generate a series of destination time packets 112 .
- Each destination time packet includes destination time data and destination signature data.
- the destination time data corresponds to a second time when the destination time packet is generated.
- the destination signature data corresponds to characteristic features of each of the plurality of media streams.
- the source packet stream is received, at the destination processor, through the network.
- destination processor 104 can receive source packet stream 116 .
- a transmission time for the source packet stream is determined, at the destination processor, based on the source time data and the destination time data. For example, destination processor 104 can determine a transmission time for source packet stream 116 based on source time data and destination time data.
- a relative synchronization error is determined, at the destination processor, based on the source signature data and the destination signature data.
- destination processor 104 can determine a synchronization error based on the source signature data and the destination signature data.
- Method 700 begins with generating, at a source processor, a series of source time packets at 702 .
- source processor 102 can generate a series of source time packets 112 .
- Each source time packet includes source time data and source signature data.
- the source time data corresponds to a first time when the source time packet is generated.
- the source signature data corresponds to characteristic features of each of the plurality of media streams.
- the series of source time packets is transmitted, at the source processor, as a source packet stream through a network.
- source processor 102 can transmit source time packets 112 as source packet stream 116 .
- a series of destination time packets is generated at a destination processor.
- destination processor 104 can generate a series of destination time packets 118 .
- Each destination time packet includes destination time data and destination signature data.
- the destination time data corresponds to a second time when the destination time packet is generated.
- the destination signature data corresponds to characteristic features of each of the plurality of media streams.
- the series of destination time packets is transmitted, at the destination processor, as a destination packet stream through the network.
- destination processor 104 can transmit destination time packets 118 as destination packet stream 122 .
- the analysis processor receives the source packet stream and the destination packet stream.
- analysis processor 106 can receive source packet stream 116 and destination packet stream 122 .
- the analysis processor determines a transmission time for at least one of the source packet stream and the destination packet stream based on at least one of the source time data and the destination time data. For example, analysis processor 106 can determine the transmission time for at least one of source packet stream 116 and destination packet stream 122 .
- the analysis processor determines a relative synchronization error based on the source signature data and the destination signature data. For example, analysis processor 106 can determine a relative synchronization error.
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Security & Cryptography (AREA)
- Multimedia (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
A system and method are provided for determining delay of a plurality of media streams. The system and method involve generating, at a source processor, a series of source time packets; transmitting, at the source processor, through a network, the series of source time packets as a source packet stream; generating, at a destination processor, a series of destination time packets; receiving, at the destination processor, through the network, the source packet stream; determining, at the destination processor, a transmission time for the source packet stream based on the source time data and the destination time data; and determining, at the destination processor, a relative synchronization error based on the source signature data and the destination signature data. Each source time packet includes source time data and source signature data. Each destination time packet includes destination time data and destination signature data.
Description
- This application is a continuation of U.S. application Ser. No. 17/708,185 filed on Mar. 30, 2022, which is a continuation of U.S. application Ser. No. 16/834,181 filed on Mar. 30, 2020 (now U.S. Pat. No. 11,323,780), which claims the benefit of U.S. Provisional Application No. 62/829,319 filed on Apr. 4, 2019, the complete disclosures of which are incorporated herein by reference.
- The described embodiments relate to determining delay of a plurality of media streams, and in particular to determining transmission times and relative synchronization errors.
- Media transmission systems can route media streams from various source devices to various downstream devices. Media streams can contain video, audio, or metadata content. The metadata is often referred to as vertical ancillary data (VANC) or horizontal ancillary data (HANC). In separate elementary essence transmission systems, each of the streams is typically a separate stream, in the sense that the information for one stream is not embedded in another stream. This is in contrast to SDI transmission, in which audio and ancillary data is embedded in non-visible portions of a video signal.
- Media streams can originate from different sources and may, as a result, be out of sync with one another. In some cases, media streams can originate from the same source but may still be out of sync with each other. For example, a video stream may be “running ahead” or “running behind” a corresponding audio stream, resulting in lip-sync errors. Furthermore, when media streams are transmitted over a network, the media streams can travel via different network paths, or be processed by different intermediate devices. As a result, the media streams may arrive at a downstream device at different times, resulting in further desynchronization. Accordingly, it may be desirable to determine transmission times and relative synchronization errors.
- In one aspect, some embodiments provide a system for determining delay of a plurality of media streams. The system includes a source processor and a destination processor. The source processor is configured to generate a series of source time packets; and transmit, through a network, the series of source time packets as a source packet stream. Each source time packet includes source time data and source signature data. The source time data corresponds to a first time when the source time packet is generated. The source signature data corresponds to characteristic features of each of the plurality of media streams. The destination processor is configured to generate a series of destination time packets; receive, through the network, the source packet stream; determine a transmission time for the source packet stream based on the source time data and the destination time data; and determine a relative synchronization error based on the source signature data and the destination signature data. Each destination time packet includes destination time data and destination signature data. The destination time data corresponds to a second time when the destination time packet is generated. The destination signature data corresponds to characteristic features of each of the plurality of media streams.
- In some embodiments, the source packet stream is transmitted in-band with the plurality of media streams.
- In some embodiments, the source packet stream is transmitted out-of-band from the plurality of media streams.
- In some embodiments, the source time data and the destination time data is generated using PTP (Precision Time Protocol).
- In some embodiments, the source processor is further configured to transmit, through the network, the plurality of media streams. The network includes at least one processing device configured to process at least one media stream of the plurality of media streams. The destination processor is further configured to receive the plurality of media streams.
- In some embodiments, the source time data and the destination time data further include a clock signal.
- In some embodiments, the source packet stream is transmitted synchronously.
- In some embodiments, the source packet stream is transmitted asynchronously.
- In some embodiments, the characteristic features include at least one of: an average luma value, an average color value, an average motion distance, and a contrast level.
- In some embodiments, the characteristic features include at least one of: an envelope of signal amplitude, an average loudness level, a peak formant, and an average zero crossing rate.
- In some embodiments, the plurality of media streams include at least one of: a video stream, an audio stream, and a metadata stream.
- In one aspect, some embodiments provide a system for determining delay of a plurality of media streams. The system includes a source processor, a destination processor, and an analysis processor. The source processor configured to generate a series of source time packets; and transmit, through a network, the series of source time packets as a source packet stream. Each source time packet includes source time data and source signature data. The source time data corresponds to a first time when the source time packet is generated. The source signature data corresponds to characteristic features of each of the plurality of media streams. The destination processor is configured to generate a series of destination time packets; and transmit, through the network, the series of destination time packets as a destination packet stream. Each destination time packet includes destination time data and destination signature data. The destination time data corresponds to a second time when the destination time packet is generated. The destination signature data corresponds to characteristic features of each of the plurality of media streams. The analysis processor is configured to receive, through the network, the source packet stream and the destination packet stream; determine a transmission time for at least one of the source packet stream and the destination packet stream based on at least one of the source time data and the destination time data; and determine a relative synchronization error based on the source signature data and the destination signature data.
- In one aspect, some embodiments provide a method for determining delay of a plurality of media streams. The method involves generating, at a source processor, a series of source time packets; transmitting, at the source processor, through a network, the series of source time packets as a source packet stream; generating, at a destination processor, a series of destination time packets; receiving, at the destination processor, through the network, the source packet stream; determining, at the destination processor, a transmission time for the source packet stream based on the source time data and the destination time data; and determining, at the destination processor, a relative synchronization error based on the source signature data and the destination signature data. Each source time packet includes source time data and source signature data. The source time data corresponds to a first time when the source time packet is generated. The source signature data corresponds to characteristic features of each of the plurality of media streams. Each destination time packet includes destination time data and destination signature data. The destination time data corresponds to a second time when the destination time packet is generated. The destination signature data corresponds to characteristic features of each of the plurality of media streams.
- In some embodiments, the source packet stream is transmitted in-band with the plurality of media streams.
- In some embodiments, the source packet stream is transmitted out-of-band from the plurality of media streams.
- In some embodiments, the source time data and the destination time data is generated using PTP (Precision Time Protocol).
- In some embodiments, the source time data and the destination time data further include a clock signal.
- In some embodiments, the method further involves transmitting, at the source processor, through the network, the plurality of media streams; processing, at at least one processing device the network, at least one media stream of the plurality of media streams; and receiving, at the destination processor, the plurality of media streams.
- In some embodiments, the source packet stream is transmitted synchronously.
- In some embodiments, the source packet stream is transmitted asynchronously.
- In some embodiments, the characteristic features include at least one of: an average luma value, an average color value, an average motion distance, and a contrast level.
- In some embodiments, the characteristic features include at least one of: an envelope of signal amplitude, an average loudness level, a peak formant, and an average zero crossing rate.
- In some embodiments, the plurality of media streams include at least one of: a video stream, an audio stream, and a metadata stream.
- In one aspect, some embodiments provide a method for determining delay of a plurality of media streams. The method involves generating, at a source processor, a series of source time packets; transmitting, at the source processor, through a network, the series of source time packets as a source packet stream; generating, at a destination processor, a series of destination time packets; transmitting, at the destination processor, through a network, the series of destination time packets as a destination packet stream; receiving, at an analysis processor, through the network, the source packet stream and the destination packet stream; determining, at the analysis processor, a transmission time for at least one of the source packet stream and the destination packet stream based on at least one of the source time data and the destination time data; and determining, at the analysis processor, a relative synchronization error based on the source signature data and the destination signature data. Each source time packet includes source time data and source signature data. The source time data corresponds to a first time when the source time packet is generated. The source signature data corresponds to characteristic features of each of the plurality of media streams. Each destination time packet includes destination time data and destination signature data. The destination time data corresponds to a second time when the destination time packet is generated. The destination signature data corresponds to characteristic features of each of the plurality of media streams.
- Embodiments of the present invention will now be described in detail with reference to the drawings, in which:
-
FIG. 1 is a block diagram of a system for determining delay of a plurality of media streams, in accordance with at least one embodiment; -
FIG. 2 is a block diagram of a system for determining delay of a plurality of media streams, in accordance with at least one embodiment; -
FIGS. 3A and 3B are illustrations of a plurality of media streams, source time packets, and destination time packets, in accordance with at least one embodiment; -
FIG. 4 is a block diagram of a processor, in accordance with at least one embodiment; -
FIG. 5 is a block diagram of a packet, in accordance with at least one embodiment; -
FIG. 6 is a flowchart of a method for determining delay of a plurality of media streams, in accordance with at least one embodiment; and -
FIG. 7 is a flowchart of a method for determining delay of a plurality of media streams, in accordance with at least one embodiment. - The drawings, described below, are provided for purposes of illustration, and not of limitation, of the aspects and features of various examples of embodiments described herein. For simplicity and clarity of illustration, elements shown in the drawings have not necessarily been drawn to scale. The dimensions of some of the elements may be exaggerated relative to other elements for clarity. It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements or steps.
- It will be appreciated that numerous specific details are set forth in order to provide a thorough understanding of the example embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this description and the drawings are not to be considered as limiting the scope of the embodiments described herein in any way, but rather as merely describing the implementation of the various embodiments described herein.
- It should be noted that terms of degree such as “substantially”, “about” and “approximately” when used herein mean a reasonable amount of deviation of the modified term such that the end result is not significantly changed. These terms of degree should be construed as including a deviation of the modified term if this deviation would not negate the meaning of the term it modifies.
- In addition, as used herein, the wording “and/or” is intended to represent an inclusive-or. That is, “X and/or Y” is intended to mean X or Y or both, for example. As a further example, “X, Y, and/or Z” is intended to mean X or Y or Z or any combination thereof.
- It should be noted that the term “coupled” used herein indicates that two elements can be directly coupled to one another or coupled to one another through one or more intermediate elements. Furthermore, the term “body” typically refers to the body of a patient, a subject or an individual who receives the ingestible device. The patient or subject is generally a human or other animal.
- The embodiments of the systems and methods described herein may be implemented in hardware or software, or a combination of both. These embodiments may be implemented in computer programs executing on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface. For example and without limitation, the programmable computers may be a server, network appliance, embedded device, computer expansion module, a personal computer, laptop, personal data assistant, cellular telephone, smart-phone device, tablet computer, a wireless device or any other computing device capable of being configured to carry out the methods described herein.
- In some embodiments, the communication interface may be a network communication interface. In embodiments in which elements are combined, the communication interface may be a software communication interface, such as those for inter-process communication (IPC). In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combination thereof.
- Program code may be applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in known fashion.
- Each program may be implemented in a high level procedural or object oriented programming and/or scripting language, or both, to communicate with a computer system. However, the programs may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program may be stored on a storage media or a device (e.g. ROM, magnetic disk, optical disc) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. Embodiments of the system may also be considered to be implemented as a non-transitory computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
- Furthermore, the system, processes and methods of the described embodiments are capable of being distributed in a computer program product comprising a computer readable medium that bears computer usable instructions for one or more processors. The medium may be provided in various forms, including one or more diskettes, compact disks, tapes, chips, wireline transmissions, satellite transmissions, internet transmission or downloadings, magnetic and electronic storage media, digital and analog signals, and the like. The computer useable instructions may also be in various forms, including compiled and non-compiled code.
- Reference is first made to
FIG. 1 , which illustrates a block diagram ofsystem 100 for determining delay of a plurality ofmedia streams 110, in accordance with at least one embodiment.System 100 includessource processor 102,destination processor 104, andnetwork 108.Source processor 102 is connected todestination processor 104 vianetwork 108. Various data can be transmitted fromsource processor 102 todestination processor 104 acrossnetwork 108. -
Source processor 102 anddestination processor 104 can be any suitable processors, controllers, digital signal processors, graphics processing units, application specific integrated circuits (ASICs), and/or field programmable gate arrays (FPGAs) that can provide sufficient processing power depending on the configuration, purposes and requirements of thesystem 100. In some embodiments,source processor 102 anddestination processor 104 can include more than one processor with each processor being configured to perform different dedicated tasks. -
Source processor 102 can be connected to one or more source devices (not shown) that generate media content. For example, the source devices may be cameras, microphones, or other devices for generating video, audio, or metadata content.Source processor 102 can receive media content from the source devices and generatemedia streams 110. In some embodiments,source processor 102 can receivemedia streams 110 from the source devices and does not generatemedia streams 110. In some embodiments,source processor 102 is a source device. In some embodiments,source processor 102 can be connected to one or more other processing devices (not shown) that transmitmedia streams 110 to sourceprocessor 102. - Each stream of
media streams 110 can include video, audio, or metadata content. In some embodiments, each stream includes only one type of content. In other embodiments, each stream can include more than one type of content. A media stream that includes video, audio, or metadata may be referred to as a video stream, audio stream, or metadata stream, respectively. In some embodiments, each stream ofmedia streams 110 is packetized. That is, the data within each stream is formatted as a plurality of packets. Accordingly, each media stream can include a plurality of media packets, each video stream can include a plurality of video packets, each audio stream can include a plurality of audio packets, and each metadata stream can include a plurality of metadata packets. It will be appreciated that although only threemedia streams 110 are shown, there can be any number of media streams 110. -
Source processor 102 can transmitmedia streams 110 throughnetwork 108 todestination processor 104. In some embodiments,media streams 110 are transmitted bysource processor 102 using a synchronous communication standard, such as SDI (Serial Digital Interface). In other embodiments,media streams 110 are transmitted using an asynchronous communication standard, such as IP (Internet Protocol). In some cases,media streams 110 are transmitted in a steady stream. In some cases,media streams 110 are transmitted intermittently. -
Network 108 can include various network paths (not shown) through which data, such asmedia streams 110, can be routed. In some embodiments, the network paths can include various switches and intermediate processing devices. The switches can selectively reconfigure one or more network paths to change the routing of media streams 110. For example, the switches can route a stream from thesource processor 102, to one or more intermediate processing devices, to thedestination processor 104. The intermediate processing devices can process one or more of media streams 110. For example, an intermediate processing device can process a video stream to adjust various characteristics of the video content, such as resolution, color, contrast, brightness, orientation, level of compression, etc. Similarly, an intermediate processing device may process an audio stream to adjust characteristics of the audio content, such as equalization, level of compression, etc. An intermediate processing device may also process a metadata stream to add new metadata, or remove or modify existing metadata. -
Destination processor 104 can receivemedia streams 110 fromsource processor 102, throughnetwork 108. In some embodiments,destination processor 104 can buffer one or more of media streams 110. That is,destination processor 104 can temporarily store data from one or more streams in a memory (not shown). For example,media streams 110 received at different times or at different rates can be buffered bydestination processor 104 for later processing. -
Destination processor 104 can be connected to one or more downstream devices (not shown). For example,destination processor 104 can be connected to a video production system.Destination processor 104 can transmitmedia streams 110 to the video production system, which can output media streams 110. For example, a video production system can display video streams on one or more monitors or play audio streams on one or more speakers. In some cases, the video production system can be used to facilitate production of a television broadcast. - In some cases,
media streams 110 may be out of sync or temporally misaligned with respect to each other when they are received bydestination processor 104. For example, a video stream may be “running ahead” or “running behind” a corresponding audio stream, resulting in lip-sync errors. This may be caused by the fact the media content was generated by different source devices. In some cases, there may be desynchronization even where media content was generated by the same source device. The desynchronization may be caused bymedia streams 110 traveling on different network paths or having different intermediate processing. As will be discussed in further detail below,system 100 can determine transmission times and relative synchronization errors. -
Source processor 102 can generate a series ofsource time packets 112. It will be appreciated that although only threesource time packets 114 are shown,source processor 102 can generate any number ofsource time packets 114. In some embodiments,source processor 102 can generate asource time packet 114 for each segment of a media stream. For example, for a video stream, the segment may correspond to a video frame. Accordingly,source processor 102 can generate a series ofsource time packets 112 at the same frequency as a video frame rate of a video stream. In some embodiments,source processor 102 can generate asource time packet 114 for each set of contemporaneous media segments. For example, a set of media segments may include a video frame, an audio segment cotemporaneous with the video frame, and metadata cotemporaneous with the video frame. - Each source time packet includes source time data. Source time data corresponds to the time when the
source time packet 114 is generated. For example, source time data can include a timestamp identifying when thesource time packet 114 was generated. This time may be referred to as a first time. In some embodiments, eachsource time packet 114 is generated approximately contemporaneous with the transmission of thesource time packet 114. In some embodiments, eachsource time packet 114 is generated approximately contemporaneous with the transmission of a segment of a media stream or with the transmission of a set of cotemporaneous media segments. Accordingly, in some embodiments, the source time data can correspond to the time when thesource time packet 114 or a particular segment of a media stream is transmitted. - In some embodiments, source time data can be generated using a clock which is synchronized throughout
system 100. For example, source time data can be generated using PTP (Precision Time Protocol). PTP can ensure that time values determined at the same time by different devices, in possibly different locations, share a common time. - Each
source time packet 114 further includes source signature data. Source signature data corresponds to characteristic features of each ofmedia streams 110. The characteristic features can be used to identify a particular segment of a particular stream. For example, for a video stream, the characteristic features may correspond to a particular video frame. For a video stream, the characteristic features may include an average luma value, an average color value, an average motion distance, or a contrast level. Similarly, for an audio stream, the characteristic feature may include an envelope of signal amplitude, an average loudness level, a peak formant, and an average zero crossing rate. For a metadata stream, the characteristic feature may include a hash value of some or all of the metadata. In some embodiments, the characteristic features can correspond to a set of cotemporaneous segments of media streams 110. For example the characteristic features can identify a video frame, an audio segment cotemporaneous with the video frame, and metadata cotemporaneous with the video frame. - In some embodiments, each
source time packet 114 can include additional time data, such as a clock signal, to facilitate video network communication. For example, some video transmission standards, such as some SDI standards, require a 90 kHz clock to be embedded with video data, on a frame-by-frame basis. The 90 kHz clock can be embedded in eachsource time packet 114 to allow eachsource time packet 114 to be synchronized with each specific video frame. - It will be appreciated that
source time packets 114 can be any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is, asource time packet 114 may, in some embodiments, never be assembled as a packet or transmitted. -
Source processor 102 can transmit the series ofsource time packets 112 assource packet stream 116 throughnetwork 108.Source packet stream 116 can be a packetized stream. That is,source packet stream 116 can include data that is formatted in a plurality of packets.Source packet stream 116 can be transmitted using a synchronous communication standard or an asynchronous communication standard. -
Source packet stream 116 can be transmitted out-of-band frommedia streams 110. That is,source packet stream 116 andmedia streams 110 are transmitted in separate streams. However, in some embodiments,source packet stream 116 is transmitted in-band withmedia streams 110. That is,source packet stream 116 andmedia streams 110 are transmitted in the same stream. In such embodiments,source packet stream 116 travels along the same network path as one or more media streams. For example,source time packets 114 can be transmitted in the same stream as other video, audio, or metadata packets. In another example,source time packets 114 can be embedded in a metadata packet (such as in VANC) in a metadata stream or media stream. In some embodiments,source packet stream 116 can be transmitted to the same IP address asmedia streams 110, but to a different UDP port number. -
Destination processor 104 can receive, throughnetwork 108,source packet stream 116 andmedia streams 110.Destination processor 104 can generate a series ofdestination time packets 118. In some embodiments,destination processor 104 generates eachdestination time packet 120 for each segment of a media stream. For example, for a video stream, a segment may correspond to a video frame. In some embodiments,destination processor 104 can generate asource time packet 120 for each set of contemporaneous media segments. - Each
destination time packet 120 includes destination time data, similar tosource time packets 114 and source time data. Destination time data corresponds to the time when thedestination time packet 120 is generated. This time may be referred to as a second time. In some embodiments, eachdestination time packet 120 is generated approximately contemporaneous with the reception of eachsource time packet 114. In some embodiments, eachdestination time packet 120 is generated approximately contemporaneous with the reception of each segment of a media stream or each set of cotemporaneous media segments. In some embodiments, the destination time data is generated using PTP. In some embodiments, the destination time data can include a clock signal. - Each
destination time packet 120 also includes destination signature data, similar tosource time packets 114 and source signature data. Destination signature data corresponds to characteristic features of each of the media streams 110. The characteristic features can be similar as those describedsource time packets 114. In some embodiments, the characteristic features can correspond to a set of cotemporaneous segments of media streams 110. - It will be appreciated that
destination time packets 120 may refer to any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is, adestination time packet 120 may, in some embodiments, never be assembled as a packet or transmitted. -
Destination processor 104 can determine a transmission time for thesource packet stream 116 based on the source time data and the destination time data. For example,destination processor 104 can determine a difference between a first time when a source time packet is generated and a second time when a destination time packet is generated. The source time packet can be generated contemporaneously with the transmission ofsource packet stream 116 and the destination time packet can be generated contemporaneously with the reception ofsource packet stream 116. Accordingly, the difference between the first time and the second time can indicate a transmission time of thesource packet stream 116 throughnetwork 108. In some cases, the transmission time of thesource packet stream 116 can be substantially equal to the transmission time of one or more of media streams 110. For example, this may be the case wheresource packet stream 116 travels along the same network path as one or more ofmedia streams 110, or where the source time packet is generated approximately cotemporaneous with the transmission of the one or more media stream. -
Destination processor 104 can also determine a relative synchronization error formedia streams 110. Relative synchronization error can refer to a difference between the delays of two or more media streams. For example, for an audio stream that was delayed 100 ms and a video stream that was delayed 25 ms, the relative synchronization error is 75 ms. That is, the audio stream is running 75 ms behind the video stream. In some cases, the relative synchronization can be based on relative delays of media streams 110. That is, the delays are relative to another time, rather than absolute. For example, the delay of the streams can be relative to the transmission time for thesource packet stream 116. That is, the delays are relative to the time when thesource packet stream 116 is received bydestination processor 104. -
Destination processor 104 can determine the relative synchronization error based on the source signature data and the destination signature data. As discussed above, the source signature data and destination signature data can include characteristic features of the media streams that correspond to particular segments of the media stream.Destination processor 104 can compare the source signature data of eachsource time packet 114 and destination signature data of eachdestination time packet 120. The comparison can be used bydestination processor 104 to locate temporal misalignments or relative synchronization errors between media streams. The comparison of source and destination signature data will be described in further detail below with respect toFIGS. 3A and 3B . In some embodiments,destination processor 104 can then realignmedia streams 110 to correct for the synchronization error. In some embodiments,destination processor 104 can determine the transmission time for each ofmedia streams 110 based on the transmission time and the relative synchronization error. - Referring now to
FIG. 2 , shown therein is a block diagram ofsystem 200 for determining delay of a plurality of media streams, in accordance with at least one embodiment. Similar tosystem 100 ofFIG. 1 ,system 200 includessource processor 102,destination processor 104, andnetwork 108. However, in contrast tosystem 100,system 200 further includesanalysis processor 106. - Similar to
system 100,source processor 102 can transmitmedia streams 110 throughnetwork 108.Source processor 102 can also generate a series ofsource time packets 112, where eachsource time packet 114 includes source time data and source signature data. However, in contrast withsystem 100,source processor 102 transmits the series ofsource time packets 112 assource packet stream 116 to analysis processor 106 (rather than to destination processor 104). In some embodiments,source processor 102 transmitssource packet stream 116 toanalysis processor 106 throughnetwork 108. In some embodiments,source processor 102 can transmitsource packet stream 116 todestination processor 104 anddestination processor 104 can transmitsource packet stream 116 toanalysis processor 106. - Similarly,
destination processor 104 can receivemedia streams 110 throughnetwork 108.Destination processor 104 can also generate a series ofdestination time packets 118, where eachdestination time packet 120 include source time data and source signature data. However, in contrast tosystem 100,destination processor 104 transmits the series ofdestination time packets 118 asdestination packet stream 122 toanalysis processor 106. In some embodiments,destination processor 104 transmitsdestination packet stream 122 toanalysis processor 106 throughnetwork 108. -
Analysis processor 106 can be any suitable processors, controllers, digital signal processors, graphics processing units, application specific integrated circuits (ASICs), and/or field programmable gate arrays (FPGAs) that can provide sufficient processing power depending on the configuration, purposes and requirements of thesystem 200. In some embodiments,analysis processor 106 can include more than one processor with each processor being configured to perform different dedicated tasks. -
Analysis processor 106 can receivesource packet stream 116 fromsource processor 102 and receivedestination packet stream 122 fromdestination processor 104. In some embodiments,analysis processor 106 can receivesource packet stream 116 anddestination packet stream 122 throughnetwork 108. -
Analysis processor 106 can determine a transmission time forsource packet stream 116 based on source time data. For example,analysis processor 106 can compare a first time from the source time data indicating when a source time packet was generated with the time at which the source time packet was received atanalysis processor 106. Since the source time packet can be generated approximately contemporaneously with the transmission ofsource packet stream 116, the difference can correspond to the transmission time of the source time packet. Similarly,analysis processor 106 can also determine a transmission time fordestination packet stream 122 based on destination time data. -
Analysis processor 106 can also determine a relative synchronization error, based on the source signature data and the destination signature data, in a similar manner as described above with respect todestination processor 106. In some cases, the relative synchronization error is a difference between relative delays ofmedia streams 110, where the delays are relative to the transmission time forsource packet stream 116 ordestination packet stream 122. In some embodiments,analysis processor 106 can realignmedia streams 110 to correct for the relative synchronization error. In some embodiments,analysis processor 106 can determine a transmission time formedia streams 110 based on the transmission time forsource packet stream 116 ordestination packet stream 122 and relative synchronization error. - Referring now to
FIGS. 3A and 3B , shown therein is an illustration ofmedia streams 110,source time packets 112, anddestination time packets 118. Media streams 110 includevideo stream 110 v,audio stream 110 a, andmetadata stream 110 m.Video stream 110 v includes video segments V1, V2, . . . Vn. Similarly,audio stream 110 a includes audio segments A1, A2, . . . An andmetadata stream 110 m includes metadata segments M1, M2, . . . Mn. It will be appreciated that although only three media streams are shown, there may be any number of media streams 110. - In
FIG. 3A ,video stream 110 v is aligned temporally withaudio stream 110 a andmetadata stream 110 m. For example, V1 can correspond to a video frame, A1 can correspond to audio cotemporaneous to that video frame, and M1 can correspond to metadata cotemporaneous to the video frame. V1 is synchronized with A1 and M1, V2 is synchronized with A2 and M2, and Vn is synchronized with An and Mn. In this case, there is no difference in delays betweenvideo stream 110 v,audio stream 110 a, andmetadata stream 110 m. This may be the case, for example, formedia streams 110 atsource processor 102 ofFIGS. 1 and 2 . - In some cases,
video stream 110 v,audio stream 110 a, andmetadata stream 110 m can become misaligned or desynchronized with respect to each other. That is, a difference in delays can develop between one or more ofvideo stream 110 v,audio stream 110 a, andmetadata stream 110 m. This may be the case, for example, for themedia streams 110 atdestination processor 104 ofFIGS. 1 and 2 . The desynchronization can be caused, for example, when media streams 110 are transmitted throughnetwork 108 ofFIGS. 1 and 2 . - In
FIG. 3B ,video stream 110 v is now no longer synchronized withaudio stream 110 a andmetadata stream 110 m. That is, V1 is now cotemporaneous with A2 and M0, instead of A1 and M1, V2 is now cotemporaneous with A3 and M1, instead of A2 and M2; and Vn is cotemporaneous with An+1 and Mn−1, instead of An and Mn. There is a difference in delays betweenvideo stream 110 v,audio stream 110 a andmetadata stream 110 m.Audio stream 110 a is “running ahead” ofvideo stream 110 v, andmetadata stream 110 m is “running behind”video stream 110 v. - In order to realign or resynchronize
media streams source time packets 112 anddestination time packets 118 can be used.Source time packets 112 include packets ST1, ST2, . . . STn.Source time packets 112 can be generated, for example, bysource processor 102 ofFIGS. 1 and 2 . Asource time packet 114 is generated for each set of cotemporaneous segments ofmedia stream - Similarly,
destination time packets 118 include packets DT1, DT2, . . . DTn and can be generated, for example, bydestination processor 104 ofFIGS. 1 and 2 . Adestination time packet 120 is generated for each set of cotemporaneous segments ofmedia stream - Each source time packet ST1, ST2, . . . STn includes signature data that corresponds to characteristic features of
media streams media streams - Each destination time packet DT1, DT2, . . . DTn also includes signature data that corresponds to characteristic features of
media streams media streams - The signature data of
source time packets 112 anddestination time packets 118 can be compared to determine a relative synchronization error ofmedia streams video stream 110 v andaudio stream 110 a can be determined based on A1 and A2. Similarly, a relative synchronization error betweenvideo stream 110 v andmetadata stream 110 m can be determined based on M0 and M1. Based on the relative synchronization error,media streams - In some cases, the signature data of
source time packets 112 anddestination time packets 118 may be compared using cross-correlation. For example, in some cases, the signature data of a source time packet may not be identical with the signature data of a destination time packet. This may be the case when intermediate processing is performed on themedia streams source time packets 112 anddestination time packets 118 may be cross-correlated to determine relative synchronization errors. - Referring now to
FIG. 4 , shown therein is a block diagram of aprocessor 402 for determining delay of a plurality of media streams, in accordance with at least one embodiment. For example,Processor 402 may besource processor 102 ordestination processor 104 ofsystem 100 orsystem 200.Processor 402 includessignature data generator 404,time data generator 406, andpacket generator 408. -
Processor 402 can be any suitable processors, controllers, digital signal processors, graphics processing units, application specific integrated circuits (ASICs), and/or field programmable gate arrays (FPGAs) that can provide sufficient processing power depending on the configuration, purposes and requirements of the system. In some embodiments,processor 402 can include more than one processor with each processor being configured to perform different dedicated tasks. -
Signature data generator 402 can receivemedia streams 110. In some embodiments,processor 402 receives source signals (not shown) and generatesmedia streams 110 that are received bysignature data generator 402.Signature generator 402 can generate signature data based onmedia streams 110. As discussed above, signature data corresponds to characteristic features of each ofmedia streams 110. -
Time data generator 406 can generate time data. The time data corresponds to a time when a packet ofpackets 410 is generated. In some embodiments,packets 410 are generated approximately contemporaneous with the transmission ofpackets 410 or media streams 110. In some cases, the time data can also include a clock signal. -
Packet generator 408 can generatepackets 410 that include the signature data and time data. For example,packet generator 408 can generatesource time packets 112 anddestination time packets 118 ofsystems packets 410 can any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is,packets 410 may, in some embodiments, never be assembled as a packet or transmitted. -
Processor 402 can then transmit the generatedpackets 410 as a packet stream (not shown). For example, processor can transmit thesource packet stream 116 or thedestination packet stream 122 ofFIGS. 1 and 2 .Processor 402 can also transmit the media streams 110. - Referring now to
FIG. 5 , shown therein is a block diagram of apacket 502, in accordance with at least one embodiment.Packet 502 includestime data 504 andsignature data 506. For example,packet 502 may be asource time packet 114 or adestination time packet 118 ofsystem 100 orsystem 200. It will be appreciated thatpacket 502 can any data structure or collection of the various data items, regardless of whether they are assembled or transmitted in any particular structure. That is,packet 502 may, in some embodiments, never be assembled as a packet or transmitted. -
Time data 504 includestime stamp data 508 andclock signal data 510.Time stamp data 508 can include data indicating a time whenpacket 502 was generated. In some cases,packet 502 is generated approximately contemporaneous with its transmission.Clock signal data 510 can include data required by certain video transmissions standards, such as a 90 kHz clock. -
Signature data 506 includesvideo signature data 512,audio signature data 514, andmetadata signature data 516. Signature data can include characteristic features of particular segments of one or more media streams. - Referring now to
FIG. 6 , shown therein is a flowchart of amethod 600 for determining delay of a plurality of media streams, in accordance with at least one embodiment. For example,method 600 can be implemented usingsource processor 102,destination processor 104, andnetwork 108 ofsystem 100.Method 600 begins with generating, at a source processor, a series of source time packets at 602. For example,source processor 104 can generate a series ofsource time packets 112. Each source time packet includes source time data and source signature data. The source time data corresponds to a first time when the source time packet is generated. The source signature data corresponds to characteristic features of each of the plurality of media streams. - At 604, the series of source time packets is transmitted, at the source processor, as a source packet stream through a network. For example, the series of
source time packets 112 can be transmitted assource packet stream 116 bysource processor 102 throughnetwork 108. - At 606, a series of destination time packets is generated at a destination processor. For example,
destination processor 104 can generate a series ofdestination time packets 112. Each destination time packet includes destination time data and destination signature data. The destination time data corresponds to a second time when the destination time packet is generated. The destination signature data corresponds to characteristic features of each of the plurality of media streams. - At 608, the source packet stream is received, at the destination processor, through the network. For example,
destination processor 104 can receivesource packet stream 116. - At 610, a transmission time for the source packet stream is determined, at the destination processor, based on the source time data and the destination time data. For example,
destination processor 104 can determine a transmission time forsource packet stream 116 based on source time data and destination time data. - At 612, a relative synchronization error is determined, at the destination processor, based on the source signature data and the destination signature data. For example,
destination processor 104 can determine a synchronization error based on the source signature data and the destination signature data. - Referring now to
FIG. 6 , shown therein is a flowchart of amethod 700 for determining delay of a plurality of media streams, in accordance with at least one embodiment. For example,method 700 can be implemented usingsource processor 102,destination processor 104,analysis processor 106, andnetwork 108 ofsystem 200.Method 700 begins with generating, at a source processor, a series of source time packets at 702. For example,source processor 102 can generate a series ofsource time packets 112. Each source time packet includes source time data and source signature data. The source time data corresponds to a first time when the source time packet is generated. The source signature data corresponds to characteristic features of each of the plurality of media streams. - At 704, the series of source time packets is transmitted, at the source processor, as a source packet stream through a network. For example,
source processor 102 can transmitsource time packets 112 assource packet stream 116. - At 706, a series of destination time packets is generated at a destination processor. For example,
destination processor 104 can generate a series ofdestination time packets 118. Each destination time packet includes destination time data and destination signature data. The destination time data corresponds to a second time when the destination time packet is generated. The destination signature data corresponds to characteristic features of each of the plurality of media streams. - At 708, the series of destination time packets is transmitted, at the destination processor, as a destination packet stream through the network. For example,
destination processor 104 can transmitdestination time packets 118 asdestination packet stream 122. - At 710, the analysis processor receives the source packet stream and the destination packet stream. For example,
analysis processor 106 can receivesource packet stream 116 anddestination packet stream 122. - At 712, the analysis processor determines a transmission time for at least one of the source packet stream and the destination packet stream based on at least one of the source time data and the destination time data. For example,
analysis processor 106 can determine the transmission time for at least one ofsource packet stream 116 anddestination packet stream 122. - At 714, the analysis processor determines a relative synchronization error based on the source signature data and the destination signature data. For example,
analysis processor 106 can determine a relative synchronization error. - The present invention has been described here by way of example only. Various modification and variations may be made to these exemplary embodiments without departing from the spirit and scope of the invention, which is limited only by the appended claims.
Claims (20)
1. A system for aligning a plurality of media streams comprising:
a source processor configured to:
generate a series of source time signals corresponding to the plurality of media streams, each source time signal including source time data and source signature data, wherein:
the source time data corresponds to a first time when the source time signal is generated;
the source signature data corresponds to characteristic features of the corresponding media stream;
transmit, through a network, the series of source time signals;
a destination processor configured to:
generate a series of destination time signals, each destination time signal including destination time data and destination signature data, wherein:
the destination time data corresponds to a second time when the destination time signal is generated;
the destination signature data corresponds to characteristic features of the corresponding media stream;
receive, through the network, the series of source time signals;
for at least one source time signal, determine a relative synchronization error based on the corresponding source signature data and the destination signature data; and
realign the corresponding at least one media stream to correct the relative synchronization error.
2. The system of claim 1 , wherein the destination processor is further configured to determine a transmission time for one or more source time signals based on the corresponding source time data and the destination time data.
3. The system of claim 2 , wherein each source time signal is packetized, and the series of source time signals are transmitted as a source packet stream.
4. The system of claim 2 , wherein each destination time signal is packetized.
5. The system of claim 3 , wherein the source packet stream is transmitted in-band with the plurality of media streams.
6. The system of claim 1 , wherein the source packet stream is transmitted out-of-band with the plurality of media streams.
7. The system of claim 1 , wherein the source time data and the destination time data is generated using PTP (Precision Time Protocol).
8. The system of claim 1 , wherein:
the source processor is further configured to transmit, through the network, the plurality of media streams;
the network comprises at least one processing device configured to process at least one media stream of the plurality of media streams; and
the destination processor is further configured to receive the plurality of media streams.
9. The system of claim 1 , wherein the source time data and the destination time data further include a clock signal.
10. The system of claim 3 , wherein the source packet stream is transmitted synchronously.
11. The system of claim 3 , wherein the source packet stream is transmitted asynchronously.
12. The system of claim 1 , wherein the characteristic features include at least one of: an average luma value, an average color value, an average motion distance, and a contrast level.
13. The system of claim 1 , wherein the characteristic features include at least one of: an envelope of signal amplitude, an average loudness level, a peak formant, and an average zero crossing rate.
14. The system of claim 1 , wherein the plurality of media streams include at least one of: a video stream, an audio stream, and a metadata stream.
15. A method for aligning a plurality of media streams comprising:
generating, at a source processor, a series of source time signals, each source time signal including source time data and source signature data, wherein:
the source time data corresponds to a first time when the source time signal is generated;
the source signature data corresponds to characteristic features of the corresponding media stream;
transmitting, through a network, the series of source time signals;
generating, at a destination processor, a series of destination time signals, each destination time signal including destination time data and destination signature data, wherein:
the destination time data corresponds to a second time when the destination time signal is generated;
the destination signature data corresponds to characteristic features of the corresponding media stream;
receiving, at the destination processor, through the network, the series of source time signals;
for at least one source time signal, determining a relative synchronization error based on the corresponding source signature data and the destination signature data; and
realigning the corresponding at least one media stream to correct the relative synchronization error.
16. The method of claim 15 , further comprising, determining, at the destination processor, a transmission time for one or more source time signals based on the corresponding source time data and the destination time data.
17. The method of claim 16 , wherein each source time signal is packetized, and the series of source time signals are transmitted as a source packet stream, and wherein each destination time signal is packetized.
18. The method of claim 17 , wherein the source packet stream is transmitted in-band with the plurality of media streams.
19. The method of claim 17 , wherein the source packet stream is transmitted out-of-band with the plurality of media streams.
20. The method of claim 15 , further comprising:
transmitting, by the source processor, through the network, the plurality of media streams;
processing, by at least one processing device comprised in the network, at least one media stream of the plurality of media streams; and
receiving, at the destination processor, the plurality of media streams.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/319,704 US20230370691A1 (en) | 2019-04-04 | 2023-05-18 | Systems and methods for determining delay of a plurality of media streams |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962829319P | 2019-04-04 | 2019-04-04 | |
US16/834,181 US11323780B2 (en) | 2019-04-04 | 2020-03-30 | Systems and methods for determining delay of a plurality of media streams |
US17/708,185 US11695999B2 (en) | 2019-04-04 | 2022-03-30 | Systems and methods for determining delay of a plurality of media streams |
US18/319,704 US20230370691A1 (en) | 2019-04-04 | 2023-05-18 | Systems and methods for determining delay of a plurality of media streams |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/708,185 Continuation US11695999B2 (en) | 2019-04-04 | 2022-03-30 | Systems and methods for determining delay of a plurality of media streams |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230370691A1 true US20230370691A1 (en) | 2023-11-16 |
Family
ID=72662558
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/834,181 Active 2040-10-13 US11323780B2 (en) | 2019-04-04 | 2020-03-30 | Systems and methods for determining delay of a plurality of media streams |
US17/708,185 Active US11695999B2 (en) | 2019-04-04 | 2022-03-30 | Systems and methods for determining delay of a plurality of media streams |
US18/319,704 Pending US20230370691A1 (en) | 2019-04-04 | 2023-05-18 | Systems and methods for determining delay of a plurality of media streams |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/834,181 Active 2040-10-13 US11323780B2 (en) | 2019-04-04 | 2020-03-30 | Systems and methods for determining delay of a plurality of media streams |
US17/708,185 Active US11695999B2 (en) | 2019-04-04 | 2022-03-30 | Systems and methods for determining delay of a plurality of media streams |
Country Status (2)
Country | Link |
---|---|
US (3) | US11323780B2 (en) |
CA (1) | CA3077449A1 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11323780B2 (en) | 2019-04-04 | 2022-05-03 | Evertz Microsystems Ltd. | Systems and methods for determining delay of a plurality of media streams |
CA3077469A1 (en) | 2019-04-04 | 2020-10-04 | Evertz Microsystems Ltd. | Systems and methods for operating a media transmission network |
Family Cites Families (39)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5602992A (en) | 1993-11-29 | 1997-02-11 | Intel Corporation | System for synchronizing data stream transferred from server to client by initializing clock when first packet is received and comparing packet time information with clock |
US6380968B1 (en) | 1998-01-06 | 2002-04-30 | Intel Corporation | Method and apparatus for controlling a remote video camera in a video conferencing system |
US8341662B1 (en) | 1999-09-30 | 2012-12-25 | International Business Machine Corporation | User-controlled selective overlay in a streaming media |
WO2004047357A1 (en) * | 2002-11-18 | 2004-06-03 | Telefonaktiebolaget Lm Ericsson (Publ) | Data unit sender and method of controlling the same |
US7545794B2 (en) | 2003-08-14 | 2009-06-09 | Intel Corporation | Timestamping network controller for streaming media applications |
US20050094562A1 (en) | 2003-10-30 | 2005-05-05 | Sumit Roy | Methods and devices for reducing total bandwidth when streaming multiple media streams |
GB2443867A (en) * | 2006-03-21 | 2008-05-21 | Zarlink Semiconductor Ltd | Timing source with packet size controller providing a distribution of packet sizes |
EP1876828B1 (en) | 2006-07-03 | 2016-10-26 | Axis AB | Method and apparatus for configuring parameter values for cameras |
US8929360B2 (en) | 2006-12-07 | 2015-01-06 | Cisco Technology, Inc. | Systems, methods, media, and means for hiding network topology |
US20110069179A1 (en) | 2009-09-24 | 2011-03-24 | Microsoft Corporation | Network coordinated event capture and image storage |
US10264029B2 (en) | 2009-10-30 | 2019-04-16 | Time Warner Cable Enterprises Llc | Methods and apparatus for packetized content delivery over a content delivery network |
US9237331B2 (en) | 2011-01-18 | 2016-01-12 | Disney Enterprises, Inc. | Computational stereoscopic camera system |
GB2492749B (en) | 2011-07-04 | 2013-11-20 | Canon Kk | Receiving video data |
CA2864003C (en) | 2012-02-23 | 2021-06-15 | Charles D. Huston | System and method for creating an environment and for sharing a location based experience in an environment |
US20140244854A1 (en) | 2013-02-27 | 2014-08-28 | Google Inc. | Content Streaming Between Devices |
US20140354768A1 (en) | 2013-05-30 | 2014-12-04 | Microsoft Corporation | Socialized Mobile Photography |
AU2014289922B2 (en) | 2013-07-11 | 2017-08-31 | Dejero Labs Inc. | Systems and methods for transmission of data streams |
US10218524B2 (en) | 2013-09-17 | 2019-02-26 | Cisco Technology, Inc. | Bit indexed explicit replication for layer 2 networking |
US9571897B2 (en) | 2013-09-17 | 2017-02-14 | Cisco Technology, Inc. | Bit indexed explicit replication for professional media networks |
US9438432B2 (en) | 2013-09-17 | 2016-09-06 | Cisco Technology, Inc. | Bit indexed explicit replication packet encapsulation |
US9942053B2 (en) | 2013-09-17 | 2018-04-10 | Cisco Technology, Inc. | Bit indexed explicit replication using internet protocol version 6 |
CN103561212A (en) | 2013-10-30 | 2014-02-05 | 上海广盾信息系统有限公司 | Camera system |
NL2012399B1 (en) | 2014-03-11 | 2015-11-26 | De Vroome Poort B V | Autonomous camera system for capturing sporting events. |
US9811748B2 (en) | 2014-06-09 | 2017-11-07 | Verizon Patent And Licensing Inc. | Adaptive camera setting modification based on analytics data |
JP6354442B2 (en) | 2014-08-12 | 2018-07-11 | カシオ計算機株式会社 | Imaging apparatus, control method, and program |
US9826138B1 (en) | 2016-02-12 | 2017-11-21 | Surround.IO | Integration of transducer outputs in a conferencing system |
WO2016202885A1 (en) | 2015-06-15 | 2016-12-22 | Piksel, Inc | Processing content streaming |
EP3366004B1 (en) | 2015-10-23 | 2020-02-19 | Telefonaktiebolaget LM Ericsson (PUBL) | Providing camera settings from at least one image/video hosting service |
GB2545181A (en) * | 2015-12-07 | 2017-06-14 | Fujitsu Ltd | Synchronisation device, method, program and system |
EP3485646B1 (en) | 2016-07-15 | 2022-09-07 | Koninklijke KPN N.V. | Streaming virtual reality video |
US10321051B2 (en) | 2016-08-02 | 2019-06-11 | International Business Machines Corporation | Intelligent image enhancement |
US10341672B2 (en) | 2017-02-03 | 2019-07-02 | Korea Advanced Institute Of Science And Technology | Method and system for media synchronization |
US10911694B2 (en) | 2017-03-15 | 2021-02-02 | Gvbb Holdings S.A.R.L. | System and method for creating metadata model to improve multi-camera production |
US10667017B2 (en) | 2017-07-20 | 2020-05-26 | International Business Machines Corporation | Adaptive packaging and distribution of channels |
US10270962B1 (en) | 2017-12-13 | 2019-04-23 | North Of You Llc | Automatic camera settings configuration for image capture |
EP3750301B1 (en) | 2018-02-06 | 2023-06-07 | Phenix Real Time Solutions, Inc. | Simulating a local experience by live streaming sharable viewpoints of a live event |
US11134185B2 (en) | 2019-04-01 | 2021-09-28 | Grass Valley Canada | System and method of partial matching of control settings across cameras |
CA3077469A1 (en) | 2019-04-04 | 2020-10-04 | Evertz Microsystems Ltd. | Systems and methods for operating a media transmission network |
US11323780B2 (en) | 2019-04-04 | 2022-05-03 | Evertz Microsystems Ltd. | Systems and methods for determining delay of a plurality of media streams |
-
2020
- 2020-03-30 US US16/834,181 patent/US11323780B2/en active Active
- 2020-03-30 CA CA3077449A patent/CA3077449A1/en active Pending
-
2022
- 2022-03-30 US US17/708,185 patent/US11695999B2/en active Active
-
2023
- 2023-05-18 US US18/319,704 patent/US20230370691A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US20220224991A1 (en) | 2022-07-14 |
US11323780B2 (en) | 2022-05-03 |
US20200322693A1 (en) | 2020-10-08 |
CA3077449A1 (en) | 2020-10-04 |
US11695999B2 (en) | 2023-07-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230370691A1 (en) | Systems and methods for determining delay of a plurality of media streams | |
US7400653B2 (en) | Maintaining synchronization of streaming audio and video using internet protocol | |
EP1398931B1 (en) | Synchronous play-out of media data packets | |
US8923342B2 (en) | Method of providing timing information for synchronizing MMT packet stream in MMT hybrid delivery service and method of synchronizing MMT packet stream in MMT hybrid delivery service | |
US10856018B2 (en) | Clock synchronization techniques including modification of sample rate conversion | |
US9837093B2 (en) | Packet based delivery of multi-channel audio over wireless links | |
US9088818B2 (en) | Adaptive media delay matching | |
US10587954B2 (en) | Packet based delivery of multi-channel audio over wireless links | |
EP3041244B1 (en) | Methods and devices for distributed audio/video synchronization and playback using an a/v orchestrator | |
US10778361B1 (en) | Stream synchronization | |
US10673609B2 (en) | Synchronization device, method, program and system | |
US9396762B2 (en) | Systems and methods for networked media synchronization | |
US10116977B2 (en) | Dynamic delay equalization for media transport | |
KR20210078985A (en) | Method of display playback synchronization of digital contents in multiple connected devices and apparatus using the same | |
WO2014128360A1 (en) | Synchronization of audio and video content | |
US20170374243A1 (en) | Method of reducing latency in a screen mirroring application and a circuit of the same | |
KR101480653B1 (en) | Method for controlling output of sound in mirrorlink | |
KR101810883B1 (en) | Live streaming system and streaming client thereof | |
JP6684433B2 (en) | Transmission device, transmission method, and program | |
CN117176279A (en) | System and method for synchronized data playback and aircraft |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |