JP2007306110A - Video audio synchronization detector, and video audio reproduction system having verification function - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To evaluate a reproduction apparatus by a video audio synchronization detector using video data and audio data having no time information at the time of reproduction. <P>SOLUTION: An output timing section 122 detects a video data output timing at which a video data decoder 112 outputs predetermined decoded video data, and an audio data output timing at which an audio data decoder 113 outputs predetermined decoded audio data. A reproduction time acquiring section 125 acquires a video reproduction reference time when the decoded video data of predetermined one image should be outputted, and an audio reproduction reference time when the predetermined continuous audio data should be outputted, on the basis of the reproduction time information included in the encoded video data or the encoded audio data. A calculator 126 calculates, as a synchronization shift time, a difference between a video audio output interval as an interval between the video data output timing and the audio data output timing, and a reference interval between the video reproduction reference time and the audio reproduction reference time. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention decodes encoded video data and audio data, and synchronizes the output of the decoded video data and audio data using reproduction time information, and outputs the synchronized video data and audio data. The present invention relates to a technique for verifying by measuring the synchronization.

  2. Description of the Related Art Conventionally, there has been known a technique for applying video (Experiment Picture Group) (MPEG) standards, etc., separately compressing and multiplexing to transmit video data and audio data or record them on a storage medium. Multiplexed compressed data transmitted or recorded using such a technique is input to a playback device and demultiplexed. Then, the separated video data and audio data are decompressed and reproduced. In such a so-called codec system, when multiplexing, reproduction time information called a time stamp indicating the reproduction time of each predetermined unit of data is embedded in video data and audio data. Then, the playback device plays back video data and audio data at a timing based on the time stamp.

  Here, as a method for verifying whether or not the playback device is normally playing back video data and audio data at a timing based on the time stamp, the video and audio to be played back are synchronized, for example, the lip when a human speaks There is a method of verifying by observing whether the movement and voice are synchronized.

  However, there is a problem that such a method of human observation and verification cannot be quantitatively evaluated. Therefore, Patent Document 1 proposes an AV synchronization detection device that can quantitatively evaluate how faithfully the playback device is playing back video data and audio data with respect to a time stamp.

In the AV synchronization detection apparatus shown in FIG. 1 of the same document, the video signal generating means 1 generates video data of video in which the time corresponding to the frame number is superimposed on the lower right of the screen. On the other hand, the time code generating means 3 generates a time code using an LTC (Longitudinal Time Code) signal. Then, the time superimposed on the lower right of the screen of the video displayed on the display device 10 is compared with the time code displayed on the time code reading means 12. By this comparison, the time difference between the reproduction timing indicated by the time stamp and the actual reproduction timing can be calculated. That is, it is possible to quantitatively evaluate how faithfully the playback device plays back video data and audio data with respect to the time stamp.
JP-A-8-172597

  However, the verification method using the conventional AV sync detection apparatus can be evaluated only by using video data and audio data reproduced with time information attached. Therefore, a special device for generating video data and audio data to be reproduced with time information attached thereto, such as the video signal generating means 1 and the time code generating means 3, has been required. In addition, a device for reading time information from reproduced audio data, such as the time code reading means 12, has been required.

  In view of the above points, the present invention enables a video / audio synchronization detection device to evaluate a playback device using video data and audio data without time information during playback.

In order to solve the above problems, the invention of claim 1
A video / audio synchronization detection device for verifying a synchronization operation of a video / audio reproduction device that decodes encoded video data and encoded audio data and outputs the decoded video data and decoded audio data in synchronization with each other,
Video data output timing at which the video / audio reproduction device outputs decoded video data of one predetermined image, and audio data output timing at which the video / audio reproduction device outputs predetermined continuous decoded audio data continuous for a predetermined unit time are detected. An output timing detector;
Based on the playback time information included in the encoded video data or the encoded audio data, the video playback reference time at which the decoded video data of the predetermined one image should be output and the predetermined continuous audio data should be output A reproduction reference time acquisition unit for acquiring the audio reproduction reference time;
A calculation unit that calculates a difference between a video / audio output interval, which is an interval between the video data output timing and the audio data output timing, and a reference interval between the video reproduction reference time and the audio reproduction reference time as a synchronization shift time; ,
It is provided with.

  As a result, the difference between the video / audio output interval, which is the difference between the video data output timing and the audio data output timing, and the reference interval, which is the difference between the video playback reference time and the audio playback reference time, is calculated as the synchronization deviation time. The synchronization between video data and audio data can be quantitatively evaluated.

The invention of claim 2
The video / audio synchronization detection apparatus according to claim 1,
The output timing detector is
Video for collating target video data corresponding to decoded video data of one predetermined image with each decoded video data sequentially output by the video / audio reproduction device, and detecting decoded video data matching the target video data A data verification unit;
The collation target audio data corresponding to the predetermined continuous decoded audio data continuous for the predetermined unit time is collated with each decoded audio data sequentially output by the video / audio reproduction device, and the decoded audio data matching the collation target audio data is obtained. A voice data matching unit to be detected;
With
The timing at which the decoded video data that matches the verification target video data is output is detected as the video data output timing,
The timing at which decoded audio data that matches the verification target audio data is output is detected as the audio data output timing.

  As a result, the decoded video data and the decoded audio data to be evaluated for synchronization deviation can be identified and detected by collation.

The invention of claim 3
The video / audio synchronization detection device according to claim 2,
The video data collating unit sequentially detects decoded video data that matches each of video data to be collated of a plurality of images, and the calculating unit detects the synchronization deviation time for each decoded video data in which the match is detected. Is calculated.

  As a result, the synchronization deviation with the audio data is evaluated for the video data of the plurality of images, so that the synchronization deviation occurring during the reproduction of the video data of the plurality of images can be evaluated, and the image in which the synchronization deviation has occurred is identified. You can also.

The invention of claim 4
The video / audio synchronization detection apparatus according to claim 3,
The video data verification unit
An image number counter that counts the number of matching times until the matching decoded video data is detected for the next matching target video data after the matching decoded video data is detected for each matching target video data;
The calculation unit is
Each time the decoded video data that matches each verification target video data is output, the value of the image number counter, the output time of one image, and the output time of the video / audio output when the decoded video data that matches the previous time is output. Is obtained as a video / audio output interval for decoded video data that matches each of the verification target video data.

  As a result, the difference between the decoded video data matching the respective verification target video data and the predetermined decoded audio data is sequentially obtained as the video / audio output interval.

The invention of claim 5
The video / audio synchronization detection device according to claim 2,
The voice data collating unit sequentially detects decoded voice data that matches each of a plurality of sets of decoding target decoded voice data, and the calculation unit includes the synchronization deviation for each decoded voice data in which the match is detected. It is characterized in that time is calculated.

  Thereby, since the synchronization deviation with the video data is evaluated for a plurality of sets of audio data continuous for a predetermined unit time, it is also possible to evaluate the synchronization deviation generated during the reproduction of a plurality of sets of audio data continuous for a predetermined unit time, It is also possible to specify audio data in which synchronization loss has occurred.

The invention of claim 6
The voice data verification unit
A set number counter that counts the number of collations until the matching decoded voice data is detected for the next matching target decoded voice data after the matching decoded voice data is detected for each matching target decoded voice data;
The calculation unit is
Each time decoded audio data that matches each verification target audio data is output, the value of the set counter and one set of predetermined unit time are set at the video and audio output interval when the decoded audio data that matched previously is output. A value obtained by adding a product of output times of continuous continuous decoded audio data is obtained as a video / audio output interval for decoded audio data that matches each of the verification target audio data.

  As a result, the difference between the decoded audio data that matches each audio data to be verified and the predetermined decoded video data is sequentially obtained as the video / audio output interval.

The invention of claim 7
The video / audio synchronization detection apparatus according to claim 1,
The calculation unit is
A counter that starts counting the value according to the passage of time at the earlier timing of the video data output timing and the audio data output timing and outputs the count value at the other timing as the video / audio output interval; It is characterized by that.

The invention of claim 8
The video / audio synchronization detection apparatus according to claim 1,
The calculation unit is
A video data output timing counter that starts counting a value according to the passage of time at a predetermined timing and outputs a count value at the video data output timing;
An audio data output timing counter that starts counting a value according to the passage of time at the predetermined timing and outputs a count value at the audio data output timing;
A video / audio output interval calculation unit that outputs a difference between a count value output by the video data output timing counter and a count value output by the audio data output timing counter as the video / audio output interval;
It is provided with.

  Accordingly, the video / audio output interval can be easily calculated by using the counter.

The invention of claim 9
The video / audio synchronization detection apparatus according to claim 1,
In addition, a determination unit that compares the synchronization deviation time with a predetermined threshold value is provided.

  Thereby, the synchronization between the video data and the audio data can be evaluated by comparing the calculated synchronization deviation time with a threshold value set freely in advance.

The invention of claim 10
A video / audio synchronization detection apparatus according to claim 1;
A video / audio playback device;
A video / audio reproduction system with a verification function having
The video / audio reproduction device is
A demultiplexing unit that separates the encoded multiplexed data into encoded video data and encoded audio data;
A video data decoding unit for decoding the encoded video data;
An audio data decoding unit for decoding the encoded audio data;
With
While outputting decoded video data and decoded audio data in synchronization,
The video / audio synchronization detecting device is:
A synchronization shift time between the decoded video data and the decoded audio data is detected.

  Thus, the synchronization operation of the video / audio reproduction device is verified by the video / audio synchronization detection device.

The invention of claim 11
The video / audio reproduction system with verification function according to claim 10,
A determination unit that compares the synchronization deviation time with a predetermined threshold;
The video data decoding unit has a function of performing a plurality of types of decoding processing with different processing times, and the case where the synchronization shift time is larger than a predetermined threshold due to the video data output timing being late is smaller It is configured to perform a decoding process with a shorter processing time.

The invention of claim 12
The video / audio reproduction system with a verification function according to claim 11,
The video data decoding unit performs a plurality of types of decoding processes with different processing times by increasing / decreasing the number of pixels per image.

The invention of claim 13
The video / audio reproduction system with a verification function according to claim 11,
The video data decoding unit performs a plurality of types of decoding processing with different processing times by increasing / decreasing the bit rate of the decoded video data.

The invention of claim 14
The video / audio reproduction system with verification function according to claim 10,
A determination unit that compares the synchronization deviation time with a predetermined threshold;
The video data decoding unit delays and outputs the decoded video data when the synchronization shift time is greater than a predetermined threshold due to the video data output timing being early, and the audio data output timing is early. When the time is larger than a predetermined threshold, the decoded audio data is output after being delayed.

  As a result, the synchronization deviation time between the decoded video data and the decoded audio data output by the video / audio reproduction device is evaluated by the video / audio synchronization detection device, and if the synchronization deviation time exceeds a predetermined threshold, the synchronization deviation is detected. The video / audio reproduction device is controlled by the video / audio synchronization detection device so that the video / audio synchronization device is reduced.

  With the video / audio synchronization detection device, the playback device can be evaluated using video data and audio data without time information at the time of playback. Therefore, without providing a special device for generating video data and audio data for evaluation that is reproduced with time information attached, the reproduction device can reproduce the video data and audio data at a timing faithful to the time stamp. You can evaluate whether you are playing.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each of the following embodiments, components having functions similar to those of the other embodiments are denoted by the same reference numerals and description thereof is omitted.

Embodiment 1 of the Invention
FIG. 1 is a block diagram showing a configuration of a codec system according to Embodiment 1 of the present invention.

  In the codec system according to the first embodiment, the encoded multiplexed data output from the video / audio encoding / multiplexing device 100 is stored in the storage medium 130 and reproduced by the video / audio reproducing device 110 with a verification function. The video / audio reproduction device with verification function 110 (video / audio reproduction system with verification function) includes a video / audio synchronization detection device 120 that detects the synchronization between the decoded video data to be reproduced and the decoded audio data.

  The video / audio encoding and multiplexing apparatus 100 includes a video data generation unit 101, a video data encoding unit 102, an audio data generation unit 103, an audio data encoding unit 104, and a multiplexing unit 105.

  The video data generating unit 101 generates video data. As the video data generator 101, for example, a video camera is used.

  The video data encoder 102 encodes the video data generated by the video data generator 101. The encoding performed here is encoding that compresses video data.

  The audio data generation unit 103 is configured to generate audio data. As the audio data generation unit 103, for example, a microphone is used.

  The audio data encoding unit 104 encodes the audio data generated by the audio data generation unit 103. The encoding performed here is encoding that compresses audio data.

  The multiplexing unit 105 adds video reproduction time information such as a time stamp to the encoded video data encoded by the video data encoding unit 102, and adds the encoded video data encoded by the audio data encoding unit 104 to the encoded audio data. Audio reproduction time information such as a time stamp is added. Then, the encoded video data and the encoded audio data to which the reproduction time information is added are multiplexed (synthesized) into data that can be transmitted through one communication line or transmission channel.

  The video / audio reproduction device 110 with a verification function includes a demultiplexing unit 111, a video data decoding unit 112, an audio data decoding unit 113, and a video / audio synchronization detection device 120.

  The demultiplexer 111 separates the encoded multiplexed data multiplexed by the multiplexer 105 into encoded video data and encoded audio data, and outputs the separated data.

  The video data decoding unit 112 decodes the encoded video data output by the demultiplexing unit 111 and outputs the decoded video data at a timing based on video reproduction time information attached to the encoded video data. Decoding performed here is decoding that decompresses compressed encoded video data. A determination signal is input to the video data decoding unit 112 from a determination unit 127 described later. When the determination signal value is “1”, the determination signal value is “1” than when the determination signal value is “0”. Decoding processing with a short processing time is performed by reducing the number of pixels per image or reducing the bit rate.

  The audio data decoding unit 113 decodes the encoded audio data output by the demultiplexing unit 111 and outputs the decoded audio data at a timing based on the audio reproduction time information attached to the encoded audio data. Decoding performed here is decoding that decompresses compressed encoded audio data.

  The video / audio synchronization detection device 120 includes a recording unit 121, an output timing detection unit 122, a reproduction reference time acquisition unit 125, a calculation unit 126, and a determination unit 127.

  The recording unit 121 records the decoded video data and the decoded audio data output from the video data decoding unit 112 and the audio data decoding unit 113, respectively. Then, the recorded decoded video data and decoded audio data are output at the same speed and order as output from the video data decoding unit 112 and the audio data decoding unit 113, respectively. The recording unit 121 is composed of, for example, a hard disk, memory, or video tape for AV (Audio Visual) data.

  As shown in FIG. 2, the output timing detection unit 122 includes a video data verification unit 122a and an audio data verification unit 122b. The video data matching unit 122 a compares the matching target video data 123 with the decoded video data sequentially output from the recording unit 121 for each image, and detects the decoded video data that matches the matching target video data 123. It has become. The timing at which the decoded video data that matches the verification target video data 123 is detected is output as the video data output timing.

  The audio data collating unit 122b compares the collation target audio data 124 with the decoded audio data sequentially output from the recording unit 121 for each decoded audio data that continues for a predetermined unit time. It is designed to detect data. And the timing which detected the decoding audio | voice data which correspond with collation object audio | voice data 124 is output as audio | voice data output timing.

  The verification target video data 123 is stored for a plurality of images, and when matching decoded video data is detected, the verification target video data 123 to be compared and verified is switched to the next new verification target video data 123. It has become. Similarly, the collation target audio data 124 is stored for a plurality of sets of decoded audio data continuous for a predetermined unit time, and when matching decoded audio data is detected, the collation target audio data 124 to be compared and collated is It is possible to switch to new collation target audio data 124.

  The reproduction reference time acquisition unit 125 reads the video reproduction time information included in each encoded video data for each encoded video data of one image, and counts with the passage of time by a counter 126a of the calculation unit 126 described later. The video reproduction reference time converted so as to align the value and the unit is obtained and stored. Further, for each piece of encoded audio data in a predetermined unit, audio reproduction time information included in each encoded audio data is read, and each encoded audio data is output from the audio data decoding unit 113 as decoded audio data. The reproduction reference time is acquired and stored. For example, in the case of an MPEG2 system in which a value counted by a 27 MHz clock is used as reproduction time information, an interval between the video data output timing and the audio data output timing is 9 MHz in a counter 126a of the calculation unit 126 described later. When counted by the clock, the video playback reference time and the audio playback reference time are obtained by multiplying the video playback time information and the audio playback time information by 1/3.

  The calculation unit 126 includes a video / audio output interval that is an interval between video data output timing of video data of a predetermined one image and predetermined audio data output timing that continues for a predetermined unit time, and video of the video data of the predetermined one image. The difference between the reproduction reference time and the reference interval between the predetermined audio data and the audio reproduction reference time is calculated as the synchronization shift time. As a specific configuration, for example, as shown in FIG. 3, a counter 126a, a video / audio reproduction reference interval calculation unit 126b, and a synchronization shift time calculation unit 126c are provided.

  The counter 126a resets the count value at the earlier timing of the video data output timing and the audio data output timing, and at the same time starts counting the value according to the passage of time, and calculates the count value at the other timing as the video / audio. Output as an output interval.

  The video / audio reproduction reference interval calculation unit 126b calculates a difference between the video reproduction reference time and the audio reproduction reference time, and outputs the difference as a reference interval between the video reproduction reference time and the audio reproduction reference time.

  The synchronization shift time calculation unit 126c calculates the difference between the video / audio output interval and the reference interval as the synchronization shift time.

  The determination unit 127 compares the synchronization deviation time output by the calculation unit 126 with a predetermined threshold value, and outputs “0” as a determination signal when the synchronization deviation time is smaller than the predetermined threshold value. When the value is larger than the threshold, “1” is output as the determination signal.

  The storage medium 130 stores the encoded multiplexed data output by the multiplexing unit 105 of the video / audio encoding / multiplexing apparatus 100.

-Operation-
Next, the operation of each unit in the codec system configured as described above will be described.

  First, when the video data generation unit 101 generates video data, the generated video data is encoded by the video data encoding unit 102. On the other hand, when the audio data generation unit 103 generates audio data, the generated audio data is encoded by the audio data encoding unit 104. The encoded encoded video data and encoded audio data are multiplexed with the reproduction time information added thereto by the multiplexing unit 105 and stored in the storage medium 130 as encoded multiplexed data.

  The encoded multiplexed data stored in the storage medium 130 is separated into encoded video data and encoded audio data by the demultiplexing unit 111. The separated encoded video data is decoded by the video data decoding unit 112, and the separated encoded audio data is decoded by the audio data decoding unit 113. The decoded decoded video data is output to the recording unit 121 at a timing based on the video reproduction time information attached to the encoded video data before decoding. Similarly, the decoded decoded audio data is output to the recording unit 121 at a timing based on the audio reproduction time information attached to the encoded audio data before decoding.

  The recording unit 121 records the decoded video data output from the video data decoding unit 112 and the decoded audio data output from the audio data decoding unit 113, and each output from the video data decoding unit 112 and the audio data decoding unit 113. Then, the data is output to the output timing detection unit 122 at the same speed and order.

  On the other hand, the reproduction reference time acquisition unit 125 reads the video reproduction time information included in the encoded video data for each encoded video data of one image, and from the video data decoding unit 112 after each encoded video data is decoded. A video playback reference time to be output as decoded video data is acquired and stored. Further, for each encoded audio data that continues for a predetermined unit time, the audio reproduction time information included in each encoded audio data is read, and after each encoded audio data is decoded, output as decoded audio data from the audio data decoding unit 113 Obtain and store the audio playback reference time to be played.

  Hereinafter, regarding the operations of the output timing detection unit 122 and the calculation unit 126, first, a case will be described in which the output timing is detected for the first decoded video data and the first decoded audio data and compared with the reproduction reference time.

  The decoded video data output from the recording unit 121 is input to the output timing detection unit 122. The decoded video data input to the output timing detection unit 122 is compared and verified for each image with the verification target video data 123 indicating the decoded video data of the first image by the video data verification unit 122a. The video data collating unit 122a detects the decoded video data that matches the collation target video data 123. The timing at which the decoded video data that matches the verification target video data 123 is detected is detected as the video data output timing. Then, after the decoded video data matching the verification target video data 123 is detected, the verification target video data 123 to be compared and verified becomes the verification target video data 123 indicating the decoded video data of the second image.

  Similarly, the decoded audio data output from the recording unit 121 is input to the output timing detection unit 122. The decoded speech data input to the output timing detection unit 122 is, for each decoded speech data that is continuous for a predetermined unit time, the speech data collating unit 122b, and the verification target speech data 124 that indicates the continuous decoded speech data that is continuous for the first predetermined unit time. And compared. The voice data collating unit 122b detects decoded voice data that matches the collation target voice data 124. And the timing which detected the decoding audio | voice data which correspond with collation object audio | voice data 124 is detected as audio | voice data output timing. After the decoded speech data that matches the verification target speech data 124 is detected, the verification target speech data 124 that is compared and verified is the verification target speech data 124 that indicates the second set of continuous decoded speech data that continues for a predetermined unit time. become.

  At the video data output timing of the decoded video data of the first one image detected by the video data collating unit 122a and the audio data output timing of the decoded audio data continuously detected by the audio data collating unit 122b for a predetermined unit time, The detection signal is input to the counter 126a of the calculation unit 126. The counter 126a is reset at the earlier of the video data output timing and the audio data output timing and starts counting a value according to the passage of time, and the count value at the other timing is obtained. Output as the video / audio output interval of the first video and audio.

  On the other hand, among the video playback reference time and the audio playback reference time stored in the playback reference time acquisition unit 125, the video playback reference time included in the encoded video data of the first image and the head for a predetermined unit time are continuous. The audio reproduction reference time included in the encoded audio data to be input is input to the video / audio reproduction reference interval calculation unit 126b of the calculation unit 126. Then, the video / audio reproduction reference interval calculation unit 126b calculates the difference between the video reproduction reference time and the audio reproduction reference time, and outputs the difference as the reference interval between the first video reproduction reference time and the first audio reproduction reference time. .

  The video / audio output interval and the reference interval of the first video and audio are input to the synchronization shift time calculation unit 126c. Then, the difference between the video / audio output interval and the reference interval is calculated as the synchronization deviation time by the synchronization deviation time calculation unit 126c. In this way, it is quantitatively evaluated whether or not the playback apparatus normally synchronizes the video data and the audio data.

  The operations described above of the output timing detection unit 122 and the calculation unit 126 are switched between the comparison target video data 123 and the verification target audio data 124 to be compared, and then the decoded video data of the second image is next. And the second set of continuous decoded audio data that continues for a predetermined unit time, and then the third set of decoded video data and the third set of continuous decoded audio data that continue for a predetermined unit time. Repeated. In this way, whether or not the video data and the audio data are synchronized is evaluated for each of the decoded video data of a plurality of images and a plurality of sets of continuous decoded audio data continuous for a predetermined unit time.

  The determination unit 127 compares the synchronization deviation time output by the calculation unit 126 with a predetermined threshold value, and outputs “0” as a determination signal when the synchronization deviation time is smaller than the predetermined threshold value. If it is larger than the threshold value, “1” is output as the determination signal.

  When the value of the determination signal is “1”, the video data decoding unit 112 reduces the processing time by reducing the number of pixels per image or reducing the bit rate compared to when the value of the determination signal is “0”. A short decoding process is performed. In this manner, the video / audio synchronization detection device 120 eliminates or reduces the synchronization shift between the video data and the audio data that occurs in the video / audio reproduction device 110 with a verification function.

  In this embodiment, the recording unit 121 outputs the decoded video data at the same speed as that output by the video data decoding unit 112. Therefore, the timing at which the decoded video data matching the verification target video data 123 is detected The decoded video data that matches the target video data 123 is handled as the timing when it is output from the video data decoding unit 112. The same applies to audio data. However, the recording unit 121 does not necessarily have to be configured so that the data recording speed is equal to the data reading speed. If the counter 126a of the calculation unit 126 can obtain a video / audio output interval in the same time unit as the reference interval calculated by the reproduction reference time acquisition unit 125 with a clock according to the ratio of the recording speed to the reading speed in the recording unit 121. Good.

  Further, the timing at which the decoded video data is output from the video data decoding unit 112 and the timing at which the decoded audio data is output from the audio data decoding unit 113, that is, the video data output timing and the audio data output timing are not necessarily from the recording unit 121. It may not be detected by detecting the timing at which the decoded video data or decoded audio data is output. For example, in the case where the recording unit 121 is configured to record each video data output timing or audio data output timing as a time stamp on the decoded video data or decoded audio data, the time stamp is read out. You may detect by.

  Further, in the present embodiment, when the reproduction reference time acquisition unit 125 acquires each video reproduction reference time and each audio reproduction reference time, the video / audio output obtained by the calculation unit 126 as the time unit of the reproduction reference time is obtained. Although the interval is adjusted, the unit of time may not necessarily be the reproduction reference time acquisition unit 125. When the calculation unit 126 calculates the difference between the reference interval of the reproduction reference time and the video / audio output interval, the time unit of the reference interval and the video / audio output interval may be the same. Therefore, for example, the playback reference time acquisition unit 125 stores the video playback time information and the audio playback time information as they are, and the video / audio playback reference interval calculation unit 126b of the calculation unit 126 calculates and calculates the difference between them. The difference may be converted so as to be the same time unit as the video / audio output interval.

<< Modification of Embodiment 1 of the Invention >>
The video / audio synchronization detection apparatus 120 according to the first embodiment may include a calculation unit 226 illustrated in FIG. 4 instead of the calculation unit 126.

  The calculation unit 226 includes a video data output timing counter 226a, an audio data output timing counter 226b, and a video / audio output interval calculation unit 226c instead of the counter 126a of the calculation unit 126.

  The video data output timing counter 226a resets the count value at a predetermined timing, starts counting the value according to the passage of time, and outputs the count value at the video data output timing.

  The audio data output timing counter 226b resets the count value at the same predetermined timing as the video data output timing counter 226a starts counting, starts counting a value corresponding to the passage of time, and counts at the audio data output timing. A value is output.

  The video / audio output interval calculation unit 226c outputs the difference between the count value output by the video data output timing counter 226a and the count value output by the audio data output timing counter 226b as the video / audio output interval. .

-Operation-
The video data collating unit 122a and the audio data collating unit 122b start collating operations at the same time, and the calculating unit 226 when the video data output timing counter 226a and the audio data output timing counter 226b start counting simultaneously with the start of the collating operation. The operation will be described as an example.

  When the video data collating unit 122a and the audio data collating unit 122b start the collating operation, the video data output timing counter 226a and the audio data output timing counter 226b are reset at the same time, and counting of values corresponding to the passage of time is started. . The video data output timing counter 226a outputs a count value at the video data output timing. The audio data output timing counter 226b outputs a count value at the audio data output timing. Then, the video / audio output interval calculation unit 226c uses the difference between the count value output by the video data output timing counter 226a and the count value output by the audio data output timing counter 226b as the video / audio output interval, and the synchronization deviation time calculation unit. To 126c.

<< Embodiment 2 of the Invention >>
The codec system according to the second embodiment includes an output timing detection unit 122 and a calculation unit 326 as illustrated in FIG. 5 instead of the output timing detection unit 122 and the calculation unit 126 of the codec system according to the first embodiment.

  In addition to the function of the video data matching unit 122a of the first embodiment, the video data matching unit 122a of this embodiment is sequentially output from the matching target video data 123 and the recording unit 121 to the mismatched image number counter 326a of the calculation unit 326. A signal indicating that the decoded video data of each image matches, and a signal indicating that there is a mismatch, are output to the mismatched image number counter 326a.

  In the present embodiment, the audio data collating unit 122b only performs comparison and collation on the collation target audio data 124 indicating the continuous decoded audio data continuous for the first predetermined unit time, and the audio of the decoded audio data continuous for the first predetermined unit time. After detecting the data output timing, no other detection is performed.

  The calculation unit 326 includes a counter 126a, a mismatched image number counter 326a, an output time information calculation unit 326b, a video / audio reproduction reference interval calculation unit 326c, and a synchronization shift time calculation unit 326d.

  The mismatched image number counter 326a is configured to add 1 to the held value when a signal indicating mismatch is input from the video data matching unit 122a. The value of the mismatched image number counter 326a is output to the output time information calculation unit 326b and reset to 0 when a signal indicating a match is input from the video data matching unit 122a. For example, the mismatch does not occur in time when the supply of encoded video data to the video data decoding unit 112 is delayed due to a problem on the transmission path, and the decoded video data of a predetermined image is output at the timing indicated by the reproduction time information. This occurs when the decoded video data of the previous image is output as it is at the timing indicated by the reproduction time information.

  First, the output time information calculation unit 326b first outputs the video data output timing of the decoded video data of the first image output from the counter 126a and the audio data output timing of the decoded audio data that continues for the first predetermined unit time (hereinafter referred to as the head audio). (Data output timing) and the audio / video output interval are stored. Each time the decoded video data that matches the verification target video data 123 for the second and subsequent images is detected, 1 is added to the value of the mismatched image number counter 326a before the reset to the stored video / audio output interval. The product of the calculated value and the output time per image is added and stored as a new video / audio output interval.

  That is, the video / audio output interval between the video data output timing of the decoded video data of the nth image and the head audio data output timing is calculated by the following equation.

{(Video data output timing of decoded video data of nth image) − (Start audio data output timing)}
= {(Video data output timing of decoded video data of (n-1) th image)-(Start audio data output timing)}
+1 output time per image × {(value of mismatched image number counter 326a) +1}
The video / audio reproduction reference interval calculation unit 326c includes encoded audio data that is continuous with the video reproduction reference time included in the encoded video data of the first image output from the reproduction reference time acquisition unit 125 and the first predetermined unit time. The reference interval of the audio reproduction reference time (hereinafter referred to as “first audio reproduction reference time”) included in the is calculated. For the encoded video data of each of the second and subsequent images, the reference interval between the video playback reference time included in each image and the audio playback reference time of the head audio is calculated.

  The synchronization deviation time calculation unit 326d calculates and outputs the difference between the stored video / audio output interval and the corresponding reference interval as the synchronization deviation time. The value calculated as the synchronization shift time is shown in the following equation.

  Synchronous shift time = {(video data output timing of decoded video data of nth image) − (start audio data output timing)} − {(video playback reference time of nth image) − (start audio playback reference time) }

-Operation-
Next, operations of the output timing detection unit 122 and the calculation unit 326 of the present embodiment configured as described above will be described.

  First, the video data collating unit 122a detects the video data output timing of the decoded video data of the first one image, and the audio data collating unit 122b detects the audio data output timing of the decoded audio data continuing for the predetermined unit time at the head, The operation for calculating the synchronization shift time is the same as that in the first embodiment.

  The operation of each unit after the video data matching unit 122a detects the decoded video data that matches the matching target video data 123 of the first image will be described.

  When the video data matching unit 122a detects decoded video data that matches the matching target video data 123 of the first image, the value of the mismatched image number counter 326a is reset to zero. The collation target video data 123 to be collated becomes the collation target video data 123 indicating the decoded video data of the second image. Then, the video data collating unit 122a compares the collation target video data 123 indicating the decoded video data of the second image with the input decoded video data for each image. In the case of mismatch, 1 is added to the value held by the mismatched image number counter 326a. If they match, the value held by the mismatched image number counter 326a is output to the output time information calculation unit 326b and reset to zero.

  When the matching between the collation target video data 123 indicating the decoded video data of the second image and the decoded video data is detected, the output time information calculation unit 326b displays the video / audio output interval of the first video and audio stored at that time. Further, a value obtained by adding the product of the value obtained by adding 1 to the value of the mismatched image number counter 326a before resetting and the output time per image is stored as a new video / audio output interval.

  On the other hand, the video / audio reproduction reference interval calculation unit 326c calculates the difference between the video reproduction reference time of the second image and the audio reproduction reference time of the head audio as a reference interval.

  Then, the synchronization deviation time calculation unit 326d has a video / audio output interval that is the difference between the video data output timing of the decoded video data of the second image stored in the output time information calculation unit 326b and the head audio data output timing, and 2 The difference between the reference interval, which is the difference between the video playback reference time of the image and the audio playback reference time of the head audio, is calculated as the synchronization shift time.

  The above operation is repeated for the decoded image data after the third image.

  In addition, instead of the mismatched image number counter 326a of the present embodiment, after matching decoded video data is detected for each matching target video data 123, the matching decoded video data is detected for the next matching target video data 123. Counter, which counts the number of collations for each collation target video data 123, and the output time information calculation unit 326b outputs the counter value and the output per image at the stored video / audio output interval. A value obtained by adding the product with time may be stored as a new video / audio output interval.

<< Other Embodiments >>
The present invention is not limited to the case where the encoding and decoding of data is performed for storing data in the storage medium 130 as in the above embodiments, but the case where the data is transmitted for transmitting on a predetermined transmission path. Etc.

  In the above embodiment, the verification target video data 123 is stored for a plurality of images, and the verification target audio data 124 is stored for a plurality of sets of decoded audio data continuous for a predetermined unit time. At the start of the synchronization detection operation for audio data, it is not necessary to store all of the series of verification target video data 123 and verification target audio data 124. For example, the video data decoding unit 112 and the audio data decoding unit 113 may be sequentially input.

  In the second embodiment, the difference between the audio / video output interval between the audio data output timing of the first decoded audio data and the video data output timing of each decoded video data and the corresponding reference interval is sequentially calculated as the synchronization shift time. Conversely, the difference between the video / audio output interval between the video data output timing of the first decoded video data and the audio data output timing of each decoded audio data and the corresponding reference interval is the synchronization shift time. May be calculated sequentially.

  In the above embodiment, when the value of the determination signal is “1”, the processing time is reduced by reducing the number of pixels per image or reducing the bit rate compared to when the value of the determination signal is “0”. However, the method of reducing the synchronization shift is not limited to this, but the synchronization shift between the video data and the audio data is eliminated or reduced. For example, when the video data output timing is early and the synchronization deviation time is long, the video data decoding unit 112 outputs the decoded video data with a delay, and the audio data output timing is early and the synchronization deviation time is large. If so, the audio data decoding unit 113 may output the decoded audio data with a delay.

  In the above embodiment, the video playback time information and the audio playback time information are read from the encoded video data and the encoded audio data output from the demultiplexing unit 111. The method by which the unit 125 obtains the video playback time information and the audio playback time information is not limited to this. For example, the multiplexing unit 105 may output the video reproduction time information and the audio reproduction time information to be added to the reproduction reference time acquisition unit 125 for recording.

  In the above-described embodiment, the calculation unit performs the calculation of the order of | video data output timing−audio data output timing |-| video playback reference time−audio playback reference time | However, the synchronization deviation time is calculated by performing the following calculation: | video data output timing-video playback reference time |-| audio data output timing-audio playback reference time | Also good.

  The video / audio synchronization detection apparatus and the video / audio reproduction system with a verification function according to the present invention have an effect that the reproduction apparatus can be evaluated using video data and audio data without time information during reproduction. Decoding the video data and audio data, and synchronizing the output of the decoded video data and audio data using the reproduction time information and measuring the synchronization of the output video data and audio data This is useful as a technology to verify by

It is a block diagram which shows the structure of the codec system which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the output timing detection part 122 which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the calculation part 126 which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the calculation part 126 which concerns on the modification of Embodiment 1 of this invention. It is a block diagram which shows the structure of the output timing detection part 122 and the calculation part 326 which concern on Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Video / audio encoding / multiplexing device 101 Video data generating unit 102 Video data encoding unit 103 Audio data generating unit 104 Audio data encoding unit 105 Multiplexing unit 110 Video / audio reproduction device with verification function 111 Demultiplexing unit 112 Video data decoding unit 113 audio data decoding unit 120 video / audio synchronization detection device 121 recording unit 122 output timing detection unit 122a video data collation unit 122b audio data collation unit 123 collation target video data 124 collation target audio data 125 reproduction reference time acquisition unit 126 calculation unit 126a counter 126b Video / audio reproduction reference interval calculation unit 126c Synchronization deviation time calculation unit 127 Determination unit 130 Storage medium 226 Calculation unit 226a Video data output timing counter 226b Audio data output timing counter 226c Video / audio output interval calculation unit 326 calculation unit 326a mismatched image number counter 326b output time information calculation unit 326c video / audio reproduction reference interval calculation unit 326d synchronization deviation time calculation unit

Claims (14)

A video / audio synchronization detection device for verifying a synchronization operation of a video / audio reproduction device that decodes encoded video data and encoded audio data and outputs the decoded video data and decoded audio data in synchronization with each other,
Video data output timing at which the video / audio reproduction device outputs decoded video data of one predetermined image, and audio data output timing at which the video / audio reproduction device outputs predetermined continuous decoded audio data continuous for a predetermined unit time are detected. An output timing detector;
Based on the playback time information included in the encoded video data or the encoded audio data, the video playback reference time at which the decoded video data of the predetermined one image should be output and the predetermined continuous audio data should be output A reproduction reference time acquisition unit for acquiring the audio reproduction reference time;
A calculation unit that calculates a difference between a video / audio output interval, which is an interval between the video data output timing and the audio data output timing, and a reference interval between the video reproduction reference time and the audio reproduction reference time as a synchronization shift time; ,
An audio / video synchronization detection apparatus comprising: The video / audio synchronization detection apparatus according to claim 1,
The output timing detector is
Video for collating target video data corresponding to decoded video data of one predetermined image with each decoded video data sequentially output by the video / audio reproduction device, and detecting decoded video data matching the target video data A data verification unit;
The collation target audio data corresponding to the predetermined continuous decoded audio data continuous for the predetermined unit time is collated with each decoded audio data sequentially output by the video / audio reproduction device, and the decoded audio data matching the collation target audio data is obtained. A voice data matching unit to be detected;
With
The timing at which the decoded video data that matches the verification target video data is output is detected as the video data output timing,
A video / audio synchronization detecting apparatus, wherein a timing at which decoded audio data matching the verification target audio data is output is detected as the audio data output timing. The video / audio synchronization detection device according to claim 2,
The video data collating unit sequentially detects decoded video data that matches each of video data to be collated of a plurality of images, and the calculating unit detects the synchronization deviation time for each decoded video data in which the match is detected. An audio / video synchronization detection apparatus characterized by calculating The video / audio synchronization detection apparatus according to claim 3,
The video data verification unit
An image number counter that counts the number of matchings until the matching decoded video data is detected for the next matching target video data after the matching decoded video data is detected for each matching target video data;
The calculation unit is
Each time the decoded video data that matches each verification target video data is output, the value of the image number counter, the output time of one image, and the output time of the video / audio output when the decoded video data that matches the previous time is output. A video / audio synchronization detecting apparatus that obtains a value obtained by adding the products of the two as a video / audio output interval for decoded video data that matches each of the verification target video data. The video / audio synchronization detection device according to claim 2,
The voice data collating unit sequentially detects decoded voice data that matches each of a plurality of sets of decoding target decoded voice data, and the calculation unit includes the synchronization deviation for each decoded voice data in which the match is detected. A video / audio synchronization detecting apparatus characterized in that time is calculated. The video / audio synchronization detection apparatus according to claim 5,
The voice data verification unit
A set number counter that counts the number of collations until the matching decoded voice data is detected for the next matching target decoded voice data after the matching decoded voice data is detected for each matching target decoded voice data;
The calculation unit is
Each time decoded audio data that matches each verification target audio data is output, the value of the set counter and one set of predetermined unit time are set at the video and audio output interval when the decoded audio data that matched previously is output. A video / audio synchronization detecting apparatus, wherein a value obtained by adding a product of output times of continuous continuous decoded audio data is obtained as a video / audio output interval for decoded audio data that matches each of the verification target audio data. The video / audio synchronization detection apparatus according to claim 1,
The calculation unit is
A counter that starts counting the value according to the passage of time at the earlier timing of the video data output timing and the audio data output timing and outputs the count value at the other timing as the video / audio output interval; A video / audio synchronization detecting apparatus characterized by the above. The video / audio synchronization detection apparatus according to claim 1,
The calculation unit is
A video data output timing counter that starts counting a value according to the passage of time at a predetermined timing and outputs a count value at the video data output timing;
An audio data output timing counter that starts counting a value according to the passage of time at the predetermined timing and outputs a count value at the audio data output timing;
A video / audio output interval calculation unit that outputs a difference between a count value output by the video data output timing counter and a count value output by the audio data output timing counter as the video / audio output interval;
An audio / video synchronization detection apparatus comprising: The video / audio synchronization detection apparatus according to claim 1,
The video / audio synchronization detecting apparatus further comprising a determination unit that compares the synchronization deviation time with a predetermined threshold value. A video / audio synchronization detection apparatus according to claim 1;
A video / audio playback device;
A video / audio reproduction system with a verification function having
The video / audio reproduction device is
A demultiplexing unit that separates the encoded multiplexed data into encoded video data and encoded audio data;
A video data decoding unit for decoding the encoded video data;
An audio data decoding unit for decoding the encoded audio data;
With
While outputting decoded video data and decoded audio data in synchronization,
The video / audio synchronization detecting device is:
A video / audio reproduction system with a verification function, wherein a synchronization shift time between the decoded video data and the decoded audio data is detected. The video / audio reproduction system with verification function according to claim 10,
A determination unit that compares the synchronization deviation time with a predetermined threshold;
The video data decoding unit has a function of performing a plurality of types of decoding processes with different processing times, and the case where the synchronization deviation time is larger than a predetermined threshold due to the late video data output timing is smaller A video / audio reproduction system with a verification function, which is configured to perform a decoding process with a shorter processing time. The video / audio reproduction system with a verification function according to claim 11,
The video / audio reproduction system with a verification function, wherein the video data decoding unit performs a plurality of types of decoding processing with different processing times by increasing / decreasing the number of pixels per image. The video / audio reproduction system with a verification function according to claim 11,
The video / audio reproduction system with a verification function, wherein the video data decoding unit performs a plurality of types of decoding processing with different processing times by increasing / decreasing a bit rate of the decoded video data. The video / audio reproduction system with verification function according to claim 10,
A determination unit that compares the synchronization deviation time with a predetermined threshold;
The video data decoding unit delays and outputs the decoded video data when the synchronization shift time is greater than a predetermined threshold due to the video data output timing being early, and the audio data output timing is early. A video / audio reproduction system with a verification function, characterized in that when the time is larger than a predetermined threshold, the decoded audio data is delayed and output.

Images