JP5025722B2 - Audio / video synchronization delay measuring method and apparatus - Google Patents

Description

Detailed Description of the Invention

  In the field of audio-visual content, it is common to provide the content as digital data. A common condition is to pass the digital data to one or more encoding processes prior to the broadcast transmission of digital audio-visual data, for example a television program broadcast. The encoding process routinely requires data compression and the use of digital audio filters to process the audio signal. The encoding process also typically requires that multiple separate data streams be multiplexed together. This is generally the case when audio data is processed differently than video data, since each of the different stages in the encoding process potentially introduces a time delay in the digital data signal, The encoding process potentially introduces a loss of synchronization between audio and video data, which is most noticeable as a loss of lip sync on spoken video screens. The human brain can perceive even the very slight time delay between video and audio data, most notably the situation where the audio signal is more advanced than the video signal. For this reason, applicable coding and transmission standards specify a maximum time delay between audio and video data. For example, according to some standards, an audio signal should not travel over a time delay longer than 40 ms than the corresponding video signal.

  Thus, it is advantageous that any given coding system can pre-measure the exact amount of time delay that may occur between audio and video signals of an audiovisual program.

According to a first concept of the present invention, the present invention is a method for measuring a delay between an audio and visual signal comprising:
Providing a video signal having a plurality of visually encoded time stamps and an audio signal having a corresponding plurality of time encoded audio signals, wherein the audio and video signals are Are synchronized,
Encoding the audio and video signals to generate a digitally encoded audio and video data stream;
Analyzing the encoded video and audio data streams to extract each of the audio and visually encoded time stamps;
Measuring the delay between the reception of corresponding video and audio time stamps.

  In the preferred embodiment, audio and video time stamps are encoded as binary codes, eg, gray codes.

  Each visually encoded timestamp preferably comprises a plurality of display segments, and the color or shadow of each segment represents a binary state. Preferably, the display segment comprises a macroblock portion.

  Each aurally encoded time stamp preferably comprises an audio tone having a plurality of predetermined frequency components, with the presence of frequency components representing a binary state.

  Preferably, each encoded timestamp comprises a frame count.

According to a second concept of the present invention, the present invention is an apparatus for measuring the delay between digitally encoded audio and video signals, wherein the video signal is visually encoded. A plurality of consecutive time stamps, the audio signal has a corresponding plurality of time-stamps that are aurally encoded, and the audio and video signals are synchronized with each other;
Arranged to detect each of the time stamps encoded in the encoded video signal, decode the time stamp and provide a first time signal representative of the actual time of receipt of the video time stamp. A video time stamp detector,
Arranged to detect each of the time stamps encoded in the encoded audio signal, decode the time stamp, and provide a second time signal representative of the actual time of the audio time stamp. Audio time stamp detector,
A time stamp comparator arranged to receive the first and second time signals and measure a delay between these reception points.

  Embodiments of the invention will now be described with reference to the accompanying drawings, which are shown by way of example only.

  According to the time analysis scheme detailed in the same-patent application filed by the applicant, the audio and video data following the encoding process performed on the originally available audio and video data. Any time delay between is measured using a predetermined video sequence with existing temporal characteristics. The video / audio data sequence is provided in either an uncompressed data format or a standard encoded data format such as MPEG-2 video or audio. A given audio / video sequence consists of a series of visible “flashes” having a given duration and a time interval between each flash. The sequence also has a corresponding number of audible tones, the duration of which and the time interval between the tones correspond exactly to the occurrence of a visible flash. An example of a suitable timing diagram for visible and audible signals is shown graphically in FIG.

  In FIG. 1, the upper signal trace 2 shows the binary level of the visible signal, which is either black as a whole or white as a whole in the visible state. The lower signal trace 4 represents an audible signal, with the upper signal level indicating the occurrence of an audible tone and the lower signal level indicating the absence of a tone. As can be seen from FIG. 1, one unit of initial time period, ie, one second after one unit of visible flash and audible tone, followed by no visible flash and no audible tone. Is done. This is followed by a further time period, which is a period in which there is no visible flash or audible tone, and this second time period has a duration of 2 units. This is followed by 2 units of duration of visible flash and audible tone generation in the sequence shown in FIG. 1, followed by 3 units of invisible flash or audible tone. The overall sequence has five periods of time during which visible flash and audible tones occur, each period lasting for an hour unit longer than the previous period, and correspondingly during periods of no visible flash or audible tone The time period increases. Thus, in the illustrated example, the entire sequence lasts for a total of 30 hours, which is typically 30 seconds. The entire sequence is preferably repeated continuously.

  In a preferred arrangement of this analysis scheme, a visible flash occurs at least within a macroblock or at least an integer multiple thereof, which is shown in the upper left corner of the display screen. Preferably, the visible flash is encoded using a 4 × 4 block array, ie 32 × 32 pixels. As will be appreciated by those skilled in the art, the scanning method that produces the displayed image results in digital data representing this portion of the display screen very quickly in the associated data stream, so that it is always always accurate. This position is carefully selected. The choice of visible flash as a 32 × 32 pixel area also tends to ensure accurate encoding of this video data. Similarly, using only a visible flash clock and white shadow maximizes the likelihood that video data will be correctly encoded. This is because these are “basic” digital values that are undesirably degraded in the encoding process. In a similar manner, audio tones are provided as single frequency component only tones, eg, 10 KHz or some other single frequency. Since only a single frequency component is used for audio tones, it can be faithfully encoded by any audio encoder included in the coding system under test.

  Additional visual data, such as a large amount of visual representation of the visible flash, may be provided to the user, for example, as a series of rotating circular segments, each segment representing a single unit of time, so a complete sequence A complete “rotation” through multiple segments is required. Of course, it is clear that such visual coordination is merely for the convenience of a human operator and is not a necessary part of the present invention.

According to this analysis scheme, a predetermined audio-visual sequence passes through the coding system under test and the encoded digital data stream is subsequently analyzed. This analysis process is performed by detecting temporal points in the encoded data stream where the complete 32 × 32 pixel macroblock changes from “black” to “white” or vice versa. Detecting one or both of the start and end of a visible flash. Since the display is only refreshed every frame, the point in time when this occurs is accurate within the duration of one frame of visual data. A typical frame rate is 25 frames per second. Thus, the encoded audio signal is analyzed to determine one or both of the start and end of the audio tone. A preferred method of detecting the beginning or end of an audio tone is to detect the sudden rising or falling amplitude of the tone to the extent that each transition from “tone” to “no tone” or vice versa occurs. Thus, the analysis process can determine any time delay between video and audio “events” (events that are rising or falling audio or video signal edges). In the preferred embodiment, an alert is automatically generated with any predetermined delay outside a predetermined set of parameters, such as set by one or more transmission standards.

A system in accordance with Applicant's pending analysis scheme seeking that video / audio synchronization in the encoded data stream has been arbitrarily corrupted is shown schematically in FIG. The predetermined video stream 10 in the uncoded state as described above is stored in a data storage medium such as a hard disk 12 and supplied to the encoding system 14 under test as an input. The encoding system typically outputs an encoded data stream that can be broken down into separate video 16 and audio 18 streams. Each of the video and audio streams is provided as an input to the analysis engine 20 and as an input to a separate video and audio event detection unit 22,24. These video and audio streams are shown as separate video and audio streams in FIG. 2 as separate inputs to the analysis engine, but the decomposition of the encoded data stream provided by the encoding system 14 is within the analysis engine: For example, it can be seen that it is achieved equally by a wrapper demultiplexer. Detecting an associated video or audio “event” in the encoded test data stream that is the start or end or both of a visible “flash” and an audio tone as described above in connection with FIG. Each event detection unit is arranged to provide an output signal indicating when each event occurred. Output signals from the audio and video event detection units 22, 24 are supplied to a time comparison unit. This time comparison unit measures any time interval that exists between the output signals from the event detection unit and any time interval that is delayed or advanced between the occurrence of audio and video "events" To be arranged. This time interval data is supplied from the time comparison unit 26 to the output interface unit 28, which is arranged so that the time interval data is supplied to the appropriate user interface. Preferably, the output interface unit 28 is also arranged to compare any time delay between the audio and video signals with a predefined maximum allowable delay. This predefined maximum allowable delay may be stored in a further data storage area 30 or may be stored internally in the output interface unit. The output interface unit may be arranged to supply a warning signal when the detected time interval exceeds a predetermined value.

  As described above with reference to FIG. 1, the sequence of visible flash and audible tones comprises “events” with an extended time interval between each “event”. This makes the time delay between the video and audio signals long enough for one of the video events to ensure that it matches the audio event and does not report or alert the analysis engine. Is not maintained at the next assembly of video and audio events. This is shown graphically in FIG. 3, with the upper trace 32 representing the video event signal and the lower trace 34 representing the audio event signal. As can be seen from FIG. 3, the encoding process delays the audio event signal with respect to the video signal by a unit of 3 hours, for example, a time period of 3 seconds, as indicated by arrow A. Thus, the start of the second video event 36 occurs at the same time as the start of the first audio event 38. If these are first video and audio events detected by the analysis engine, false reports about synchronization between the audio and video streams may be made. However, since the events are not evenly spaced and not at a fixed time interval, it can be seen that at the start of the next video event 40 the audio stream is asynchronous. Thus, the analysis engine can determine whether the video and audio streams are actually out of sync. As soon as the analysis engine detects both the start and end of video and audio events, it is detected that the synchronization is lost. This is because the end of the first audio event 38 occurs before the end of the second video event 36 even if the start of both events coincide. In this case, the analysis engine detects that the synchronization between the audio and video streams has been lost in an hourly unit, for example 1 second.

However, by using the above scheme, one can simply determine that synchronization has been arbitrarily compromised, and that the overall synchronization has been compromised, resulting in a “mismatch” between video and audio events. Occurs when the audio or video event occurs best and, as described above in connection with FIG. 3, the time required to determine if the delay is measured and the synchronization is lost You can understand what you can do. This is a waste of system resources because each second of audio-visual data typically comprises 25 frames of data. That is, the same data is processed only 25 times per second. In accordance with embodiments of the present invention, an analysis scheme can be provided to improve the determination of time delay between audio and video data streams. This can be accomplished by providing a predetermined audio-visual test sequence to be decoded at the coding system under test that includes audio and visual data that allows identification of each frame.

  In the preferred embodiment, visual coding is achieved using a pattern of black and white sequences representing binary codes. Since the known property of gray codes is that only one bit of a binary word changes at a time, the preferred binary code is a gray code. An example of a possible sequence of black and white rectangles is shown in FIG. Here, the sequence is shown in a rectangle for three consecutive frames of audio-visual data. The first five rectangular sequences represent the decimal 7 which is the gray code 00101. Therefore, the frame as frame number 7 is identified using this. The second and third sequences represent 00100 and 01100 which are decimal 8 and 9, respectively. In FIG. 4, a 5-bit word is shown for purposes of clarity only, and it will be apparent that any length of word can be selected depending on the number of frames in the test sequence. Similar to the scheme described above, in an embodiment of the present invention, individual sequences are encoded as complete portions of macroblocks or separate blocks, such as 32 × 32 blocks of individual rectangles of 32 × 32 pixels. Facilitates encoding of rectangular sequences without reliability errors. Therefore, the encoded frame identification code is maintained with reliability.

The audio signal is also encoded with a timing sequence for identifying which frame of the video signal and is synchronized with a specific portion of the audio data. In a particular embodiment of the invention, this is accomplished by including audio tones that make up many separate individual frequencies. One bit of the data word represents each frequency in an analog manner for each sequential video code representing a single bit of the gray code. This allows the encoded tone to be analyzed using Fourier analysis techniques to determine whether the individual frequency components and the binary code represented by the tone are present or not. An example of such a coded tone frequency analysis is shown graphically in FIG. The horizontal axis represents the frequency of the detected frequency component in KHz, while the vertical axis represents the power of that component. In the example shown in FIG. 5, the two frequency components 40 are shown at 9 KHz and 15 KHz. If the selected code is a 5-bit code and the frequency component centered at 3 KHz represents the most significant bit and 15 KHz is the least significant bit, then the frequency spectrum shown in FIG. That is, it represents 7 in decimal notation. Care must be taken in selecting the frequency components selected to represent the individual bits of the encoded timing word. This is because it is common for audio encoders to discard certain frequency components of a signal based on an analysis of what frequencies are audible and inaudible to the human ear. Therefore, the frequency component selected for the timing word must be such that it cannot be discarded by such an encoding technique.

  In other embodiments of the present invention, the audio code may be encoded as a series of short audio tones in a predetermined time interval, where each tone in the series represents a bit in the timing word and there is a tone. Or not represents the binary state of the bit. Also by way of example, an 8-bit binary word is represented using a series of eight audio tones. The frequency of the audio tone is preselected to facilitate these detections.

  FIG. 6 graphically illustrates an analysis engine that analyzes a test audio-visual data stream in the format described above according to an embodiment of the present invention after encoding by the encoding system under test. The basic components are the same as for the system shown in FIG. Separate video and audio data streams 616, 618 are provided as inputs to each time code detection unit 622, 624. Each time code detection unit is arranged to identify and decode embedded video and audio time codes. As a result, in the preferred embodiment, the video time code detection unit 622 looks for a coded black and white rectangular sequence, determines the binary code that the particular sequence represents, and identifies the individual frame number. Be placed. The time point at which each frame is received is also determined. Similarly, the audio time code detection unit 624 is preferably arranged to perform the necessary frequency analysis of the embedded audio time code to determine the current frequency component and the binary code representing it. An associated output signal from the time code detection unit is provided to the time comparison unit 626. This time comparison unit is arranged to determine an arbitrary time delay between the audio and video data streams based on the time of reception of the corresponding part of the data stream, as identified by the associated embedded time code. . Any time delay is provided as an input to the reporting and / or warning unit 628. This unit 628 is arranged, for example, to determine whether the time delay has exceeded certain parameters that may be stored as a look-up table in the local data storage 630.

Each frame of encoded audio-visual data storage is individually identified by its respective embedded time code, so that the analysis engine remains as is according to the scheme described in Applicant's ongoing application. The relative portion of the audio and video data streams separated within a single frame space can be determined, and the audio for each frame as opposed to the delay information for each video / audio "event" only / Video time delay information can be provided. As a result, the apparatus and method of the present invention can provide delay information and improved resolution of this delay information at an improved rate.

FIG. 1 graphically illustrates the timing and duration of audio and video events contained within possible test signals. FIG. 2 graphically illustrates a time delay analysis system that measures the time delay between the audio and video signals shown in FIG. FIG. 3 graphically illustrates the relative time of the audio and video signals as shown in FIG. 1, with a delay between the audio and video signals. FIG. 4 graphically illustrates a method for visually encoding a time stamp according to an embodiment of the present invention. FIG. 5 schematically illustrates a method for audio encoding a time stamp according to an embodiment of the present invention. FIG. 6 schematically illustrates a time delay analysis system according to an embodiment of the present invention that measures the time delay between audio and video signals having encoded time stamps of the type shown in FIGS.

Claims (4)

A method for measuring a delay between an audio signal and a video signal, comprising:
Providing a video signal having a plurality of visually encoded video time stamps ;
A step of supplying the audio signal having the video signal and synchronized, aurally corresponding plurality of audio time stamp encoded,
And generating the audio signal and the video signal by coding, digitally encoded audio data stream and the video data stream,
By analyzing the O Dio data stream and the video data stream, extracting each of the audio time stamps and the video time stamp,
Extracted corresponding comprising the steps of measuring the delay between the reception time of the video time stamp and said audio time stamp,
The video time stamp comprises a plurality of display segments, wherein the color or shadow of each segment is represented in a binary state, the video time stamp provides a visual representation of the video binary code, and the video 2 The hexadecimal code represents the frame number of the video signal,
Said audio time stamp, comprises an audio tone having a plurality of different predetermined frequency components, the presence or absence of each of said plurality of different frequency components represent a binary state of each bit of the audio binary code, the audio time stamp to provide an audio representation of the audio binary code, audio / video synchronization delay measuring method the audio binary code is equal to or representing the frame number of the video signal. The video each time stamp and said audio time stamp, the video time stamp and said audio according to claim 1, characterized in that it comprises a sequence different duration increases through the sequence of time stamps the audio / video synchronization Delay measurement method. An apparatus for measuring a delay between a digitally encoded audio signal and a video signal, the video signal having a plurality of visually encoded video time stamps, audio signal audibly has a plurality of audio time stamp corresponding encoded signal of the audio signal and the video are synchronized with each other,
Detecting each of said video time stamp from said video signal, said video time stamp and decrypts the video time stamp detector that provides a first time signal representing the actual time of reception of the video time stamp And
Detecting each of said audio time stamp from the audio signal, and decoding the above-mentioned audio time stamp, the actual audio time stamp detector that provides second time signal indicating a time of said audio time stamp ,
Receiving the first hour signal and the second time signal, comprises a time stamp comparator you measure the delay between the reception time of the video time stamp and said audio time stamp,
The video time stamp comprises a plurality of display segments, wherein the color or shadow of each segment is represented in a binary state, the video time stamp provides a visual representation of the video binary code, and the video 2 The hexadecimal code represents the frame number of the video signal,
Said audio time stamp, comprises an audio tone having a plurality of different predetermined frequency components, the presence or absence of each of said plurality of different frequency components represent a binary state of each bit of the audio binary code, the audio time stamp to provide an audio representation of the audio binary code, audio / video synchronization delay measurement device where the audio binary code is equal to or representing the frame number of the video signal. 4. The audio / video synchronization of claim 3, wherein each of the video timestamp and the audio timestamp has a different duration that sequentially increases through the sequence of the video timestamp and the audio timestamp. Delay measurement device.

Images