JP2998786B2 - Method and apparatus for synchronous reproduction of video and audio - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for synchronously reproducing image and audio, and more particularly to a method and apparatus for synchronously reproducing image and audio for synchronizing and reproducing compression-encoded image data and audio data in real time. .

[0002]

2. Description of the Related Art In a conventional synchronous reproduction method of an image and a sound, when an image and a sound are respectively encoded and recorded or transmitted, a synchronization code is inserted into each encoded data and decoded. When performing the reproduction, the synchronous reproduction of the image data and the audio data is performed with reference to these synchronization codes.

For example, Japanese Patent Application Laid-Open No. 63-205849 discloses that an optical disk for recording image data and a magnetic tape for recording audio data respectively record data and a synchronization code on the optical disk. A visual and auditory information reproducing apparatus that synchronously reproduces an image and a sound by comparing both codes is disclosed.

[0004] Further, in a compression coding system which has been discussed and standardized by the International Standard Committee on Moving Picture Coding Systems (ISO-IEC / JTC1 / SC29 / WG11),
At the time of compressing and encoding image data and audio data, a time code is inserted at the end of each of the encoded data, and at the time of decoding and reproducing, the data is compared with a reference clock of the decoding device to perform synchronous reproduction.

[0005]

In the conventional synchronous reproduction method of video and audio, synchronous reproduction of an image and audio is performed by referring to a synchronization code inserted in encoded data in advance at the time of reproduction. Here, the synchronization code is finely inserted into the code data, and when the synchronization control unit for performing the synchronization control is configured by software, the processing becomes complicated and the calculation load on the processing unit is large. is there.

In order to solve the above problem, if a dedicated control circuit for performing synchronization control with reference to a synchronization code is provided, there is a problem that the circuit becomes complicated and large in scale.

SUMMARY OF THE INVENTION The present invention provides a synchronous reproduction of an image and a sound which can reproduce the image and the sound in a synchronized manner without performing a synchronous control using a synchronization code requiring a complicated arithmetic processing or a large-scale control circuit. The aim is to provide a scheme.

[0008]

According to the present invention, the image data and the audio data are separated and extracted from encoded data in which digitally encoded image data and audio data are multiplexed. In a method for synchronously reproducing an image and a sound in which data and the sound data are decoded and reproduced in synchronization with each other, at least the sound data of the image data and the sound data is stored in a storage means, and the data is stored in the storage means. While reading and decoding data sequentially, the storage amount of the storage means is monitored, and skipping of data to be decoded and repetition of decoding processing are performed so that the storage amount falls within a predetermined range. Thus, a method for synchronously reproducing an image and a sound is obtained.

Here, the skipping of the data and the repetition of the decoding process can be performed with one unit of data at the time of writing and reading to the storage means as one unit. Further, the predetermined range can be determined by a setting value included in the encoded data on the encoding side.

According to the present invention, the image data and the audio data are separated and extracted from the encoded data in which the digitally encoded image data and audio data are multiplexed, and the image data and the audio data are separated. In an image / synchronous reproduction apparatus for decoding and reproducing data in synchronization with data, a storage unit for storing the image data and the audio data respectively, and reading out the image data and the audio data from the storage unit, respectively, and performing decoding processing Decoding means for performing the following, and monitoring means for respectively monitoring the data storage amount of the storage means,
Storage amount control means for controlling each of the decoding means based on the monitoring result from the monitoring means so that the storage amount of the data stored in the storage means falls within a predetermined range. As a result, a synchronous playback apparatus for video and audio can be obtained.

More specifically, the storage amount control means skips data reading when the data storage amount of the storage means exceeds the predetermined range, and when the data storage amount falls below the predetermined range. The decoding means is controlled so as to repeat the decoding process of the data read into the memory.

The monitoring means includes subtraction means for calculating a difference between a write address and a read address indicated by a write pointer and a read pointer of the storage means, and storage for storing a set value indicating the predetermined range. Means for comparing the subtraction result of the subtraction means with the contents stored in the storage means, and generating an interrupt to the accumulation amount control means.

Here, the skipping of the data and the repetition of the decoding process can be performed for one unit of data at the time of writing and reading to the storage means.

Also, of the monitoring means and the storage amount control means, instead of the monitoring means and the storage amount control means corresponding to the image data, decoding clock information is detected from the code data to decode the decoding clock. Clock recovery means for generating and supplying the decoding clock to the decoding means corresponding to the image data among the decoding means may be provided.

[0015]

The coded data storage unit receives and stores coded data from the multiplexed data separation unit at a constant input rate. The temporarily stored coded data is read out by the decoding unit and subjected to decoding processing. If there is no precedence or delay in the decoding process, the output rate in the encoded data storage unit becomes equal to the input rate, and the data storage amount does not change.
Therefore, if the decoding unit is controlled so that the data storage amount is constant, synchronous reproduction can be performed without referring to the synchronization code.

[0016]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows a first embodiment of the video / audio synchronous reproduction system of the present invention. The video / audio synchronous reproduction method shown in FIG. 1 includes a multiplexed data separation unit 11, an encoded data accumulation unit 12, an encoded data accumulation amount monitoring unit 13,
It has an encoded data storage amount control unit 14 and a decoding unit 15. Note that FIG. 1 shows only one coded data storage unit 12, one coded data storage amount monitoring unit 13, one coded data storage amount control unit 14, and one decoding unit 15. , Two for each image and sound.

The multiplexed data obtained by coding the image signal and the audio signal and multiplexing the obtained coded data is supplied to an input terminal I.
The data is input to the multiplexed data separation unit 11 at a constant rate via N. The multiplexed data separation unit 11 separates the input multiplexed data into image encoded data and audio encoded data. The separated encoded image data and encoded audio data are stored in the encoded data storage unit 12, respectively. The stored encoded data is read by the decoding unit 5, subjected to decoding processing, and output to the output terminal OUT.

Here, consider a phenomenon in which the synchronization between the image and the sound gradually shifts with time. The out-of-synchronization between the image and the audio occurs because one decoding process is ahead of or delayed from the other decoding process. There is practically no problem that this phenomenon occurs because a difference occurs between the input rate and the output rate of the encoded data to the encoded data storage unit 12 for each of the image and the audio. That is, if the input rate of the encoded data to the encoded data storage unit 12 and the output rate are completely equal for each of the image and the audio, no loss of synchronization occurs.

Therefore, attention is paid to the amount of data stored in the encoded data storage unit 12. If the input rate of the encoded data to the encoded data storage unit 12 and the output rate are equal, the encoded data accumulation amount does not change. If the output rate is higher than the input rate, the storage amount decreases, and if the output rate is lower than the input rate, the storage amount increases. Therefore, if the input rate to the encoded data storage unit 12 is constant, by detecting a change in the amount of data stored in the encoded data storage unit 12, a change in the output rate, that is, the preceding / previous state of the processing of the decoding unit 15, You can know the delay.

In the video / audio synchronous reproduction method, the coding side multiplexes the input data of the coded data of the image and the voice input to the coded data storage unit 12 so that the input rates are constant. At this time, the multiplexed signal also includes a signal indicating a permissible value of advance and delay of the processing in the decoding unit 15, that is, a signal indicating a permissible value of the storage amount of the encoded data storage unit 12.

The coded data storage amount monitoring unit 13 receives a signal representing the allowable storage amount separated by the multiplexed data separation unit 11. Then, the coded data storage amount monitoring unit 13 monitors the storage amount of the coded data stored in the coded data storage unit 12, and when the storage amount exceeds the storage amount allowable value or becomes zero. In this case, the coded data storage amount control unit 14 is notified of the fact.

Hereinafter, the operation of the encoded data storage amount monitoring unit 13 will be described in detail. First, the encoded data storage unit 12 is configured as shown in FIG.
It is assumed that the memory capacity (point a = 0) shown in FIG. 2 is provided, and that the accumulation amount allowable value is determined to be point b by a signal from the multiplexed data separation unit 11. Then, it is assumed that the decoding process is started when the amount of encoded data stored in the encoded data storage unit 12 reaches an intermediate point between the points a and b. At this time, the input rate and output rate of the encoded data to the encoded data storage unit 12 are completely equal, that is, if there is no advance or delay in the decoding process, the encoded data storage amount of the encoded data storage unit 12 becomes It does not change. Actually, the input rate and the output rate hardly coincide completely, and the encoded data storage amount (indicated by point c) of the encoded data storage unit 12 transitions between points a and b. .

However, when the output rate is slower than the input rate of the encoded data to the encoded data storage unit 12, that is, when the decoding process in the decoding unit 15 is slow,
The coded data storage amount of the coded data storage unit 12 gradually increases, and eventually exceeds the point b. Conversely, when the output rate is higher than the input rate of the encoded data to the encoded data storage unit 12, that is, when the decoding process in the decoding unit 15 is fast, the encoded data storage unit 12 The amount gradually decreases and eventually reaches the point a (the amount of encoded data storage becomes 0 (empty)).

As shown in FIG. 3, the coded data storage amount monitoring section 13 gradually increases the coded data storage amount represented by the point c and, when the coded data storage amount exceeds the point b, reports that fact to the coded data storage section. Notify the storage amount control unit 14. The coded data storage amount control unit 14 that has received the notification controls the decoding unit 15 to skip the coded data for one access unit (AU) and read the next coded data. By this control, the encoded data storage amount (point c) of the encoded data storage unit 12 is changed as shown in FIG.
As shown in (4), the process returns to the accumulation amount allowable range (between point a and point b).

Further, the encoded data storage amount monitoring unit 13
As shown in FIG. 5, when the coded data storage amount represented by the point c gradually decreases and reaches the point a, the fact is notified to the coded data storage amount control unit 14. As shown in FIG. 6, the encoded data storage amount control unit 14 that has received the notification controls the decoding unit 15 to reread the previously read encoded data for one access unit (AU).

As described above, in the video / synchronous reproduction system of the present embodiment, the speed of the decoding process is detected based on the input rate of the input coded signal, By decoding the same coded data again and discarding the coded data without decoding if it is delayed, the decoding process can be performed at a substantially constant speed. The image and the sound can be synchronized without using the synchronization code.

Next, a second embodiment of the present invention will be described with reference to FIG. The video / audio synchronous reproduction method shown in FIG. 7 includes a multiplexed data separation unit 71, an encoded audio data storage unit 72, an encoded audio data storage amount reference circuit 73, an arithmetic processing unit (CPU) 74, an audio decoding unit 75, It has an encoded data storage unit 76, a clock recovery unit 77, and an image decoding unit 78.

The multiplexed data obtained by multiplexing the coded image data and the coded audio data is input to the multiplexed data separation unit 71 at a constant rate via the input terminal IN. The multiplexed data separation unit 71 separates the input multiplexed data into image encoded data and audio encoded data. The separated coded data is input to and stored in the voice coded data storage unit 72 and the video coded data storage unit 76, respectively. The encoded audio data stored in the encoded audio data storage unit 72 is read out to the audio decoding unit 75,
The decoded data is output to the audio decoded data output terminal AOUT. The encoded image data stored in the encoded image data storage unit 76 is read by the image decoding unit 78, subjected to decoding processing, and subjected to image encoded data output terminal VOUT.
Output to

In the present embodiment, similarly to the first embodiment, the audio decoding unit 75 is controlled so that the amount of encoded voice data stored in the encoded voice data storage unit 72 falls within a predetermined range. I do. This control will be described later. On the other hand, such control is not performed on the encoded image data. This is because the video and audio synchronous reproduction method uses the international standard committee of the moving image coding method (ISO-IEC / JTC1 / SC2) as input multiplexed data.
This is because multiplexed data using the compression coding method standardized in 9 / WG11) is assumed. That is,
In such multiplexed data, when multiplexing audio encoded data and image encoded data on the encoding side, correction data of an image decoding clock is inserted. This is because the same clock (image decoding clock) as that on the side can be generated. In the video / audio synchronous reproduction method shown in FIG.
Generates an image decoding clock based on the image decoding clock correction data separated from the multiplexed data. This image decoding clock completely matches the clock on the encoding side. The image decoding unit 78 performs a decoding process on the encoded image data in synchronization with the image decoding clock. Thus, the input rate and the output rate of the image encoded data storage unit 76 match.

Next, the decoding processing of the encoded audio data will be described. As shown in FIG. 8, the audio encoded data accumulation amount monitoring unit 73 includes a subtracter 81, an accumulation amount allowable value storage unit 8
2 and a comparison unit 83. And the subtractor 81
, The line pointer (Write Pointer) and the read pointer (Read Pointer) of the audio encoded data storage unit 72.
And the difference between the write address and the read address indicated by This difference indicates the amount of data stored in the encoded voice data storage unit 72. In addition, the storage amount allowable value storage unit 82
Stores an allowable storage amount value separated from the multiplexed data. The comparing unit 83 compares the result of the subtraction from the subtractor 81 with the allowable storage amount stored in the allowable storage amount storage unit 82, and determines that the data storage amount of the audio encoded data storage unit 72 is within the allowable storage amount range. Is determined. Then, when the data storage amount of the audio coded data storage unit 72 is not within the storage amount allowable range, an interrupt is issued to the CPU 74. That is, the comparing unit 83 generates an interrupt when the data storage amount of the audio coded data storage unit 72 exceeds the storage amount allowable value and when the data storage amount becomes zero.

When receiving the interrupt, the CPU 74 instructs the audio decoding unit 75 to skip or repeat reading of one AU of the audio encoded data, and the data storage amount of the audio encoded data storage unit 72 is reduced. , So that the accumulated amount falls within the allowable range.

In the audio compression coding system standardized by the committee described above, one AU is defined as 1152 samples obtained by sampling in layer II. For example, if the sampling rate is 48
This is about 24 ms for kHz, about 26 ms for 44.1 kHz, and about 36 ms for 32 kHz.

In the video / audio synchronous reproduction system according to the present embodiment, as described above, there is no difference between the input rate and the output rate of the encoded image data in the encoded image data storage section. . Therefore, the difference between the image and the sound is caused only by the leading / delaying on the sound side.

The delay of the audio with respect to the image becomes the maximum when the data storage amount of the audio coded data storage unit 72 becomes equal to the allowable storage amount (point c in FIG. 2 corresponds to point b).
). In addition, the leading of the audio with respect to the image is the maximum when the data storage amount of the audio coded data storage unit 73 becomes 0 (point c in FIG.
).

For example, when the allowable storage amount of the voice coded data storage unit 72 set on the code side is 4 kbytes and the input rate and the output rate of the voice coded data storage unit 72 are equal, voice coding is performed. Assuming that the storage amount of the data storage unit 72 is 2 kbytes, the compression rate is 192 kbytes.
The leading / delaying times L ₁₉₂ , L ₂₅₆ and L ₃₈₂ at bps, 256 kbps and 382 kbps are respectively L
₁₉₂ = (2 × 1024 × 8) / (192 × 1000) =
85 [ms], L ₂₅₆ = (2 × 1024 × 8) / (256
× 1000) = 64 [ms] and L ₃₈₂ = (2 × 102)
4 × 8) / (382 × 1000) = 43 [ms], which is within about two to three image frames.
In other words, the range can be set so as not to make the viewer recognize that there is a shift between the image and the sound.

[0036]

According to the present invention, the data storage amount of the encoded data storage unit is monitored, and the decoding unit skips data and repeats the decoding process so that the data storage amount falls within a predetermined range. By doing so, complicated processing by software for performing synchronization control with reference to the synchronization code and large-scale synchronization control circuit are not required,
Synchronous reproduction in which the synchronization shift is suppressed within a maximum of two to three frames can be performed.

Further, the use of the interrupt processing from the coded data storage amount monitoring unit to keep the data storage amount within the predetermined range can simplify the synchronization control processing in the arithmetic processing unit.

Further, the coded data storage amount monitoring section obtains the coded data storage amount from the addresses indicated by the write pointer and the read pointer of the coded data storage section, so that a simple operation such as a subtracter can be performed. It can be composed of a circuit.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of a video / audio synchronous reproduction apparatus of the present invention.

FIG. 2 is a diagram for explaining a relationship between a storage capacity and a data storage amount of an encoded data storage unit in FIG.

FIG. 3 is a diagram illustrating a relationship between a storage capacity of an encoded data storage unit and a data storage amount when a delay occurs in a decoding unit.

FIG. 4 is a diagram for explaining operations of an encoded data storage amount control unit and a decoding unit in the state of FIG. 3;

FIG. 5 is a diagram illustrating a relationship between a storage capacity of an encoded data storage unit and a data storage amount when a preceding occurs in a decoding unit;

FIG. 6 is a diagram for explaining operations of an encoded data storage amount control unit and a decoding unit in the state of FIG. 5;

FIG. 7 is a block diagram showing a second embodiment of the video / audio synchronous playback apparatus of the present invention.

FIG. 8 is a block diagram showing a configuration of an audio encoded data storage amount monitoring unit of FIG. 7;

[Explanation of symbols]

 Reference Signs List 11 multiplexed data separation unit 12 coded data storage unit 13 coded data storage amount monitoring unit 14 coded data storage amount control unit 15 decoding unit 71 multiplexed data separation unit 72 voice coded data storage unit 73 voice coded data storage Quantity reference circuit 74 Operation processing unit (CPU) 75 Audio decoding unit 76 Image encoded data storage unit 77 Clock recovery unit 78 Image decoding unit

──────────────────────────────────────────────────の Continuation of the front page (51) Int.Cl. ⁷ identification code FI H04N 7/24 (58) Fields investigated (Int.Cl. ⁷ , DB name) H04J 3/00 H04L 7/00 H04N 7/08 H04N 7/24

Claims (8)

(57) [Claims] An image data and audio data are separated and extracted from encoded data in which digitally encoded image data and audio data are multiplexed, and the image data and the audio data are synchronized with each other. In the synchronous audio / video reproduction method for decoding and reproducing, at least the audio data of the image data and the audio data, while storing the data in the storage means, and sequentially reading and decoding the data from the storage means, A video / audio synchronous reproduction method, wherein the amount of data stored in the storage means is monitored, and skipping of data to be decoded and repetition of the decoding process are performed so that the amount of storage falls within a predetermined range. . 2. The image / audio data according to claim 1, wherein the skipping of the data and the repetition of the decoding process are performed in units of one unit of data at the time of writing and reading to and from the storage unit. Synchronous playback method. 3. The method of claim 1, wherein the predetermined range is determined by a setting value included in the encoded data on the encoding side. 4. Separating and extracting the image data and the audio data from encoded data in which digitally encoded image data and audio data are multiplexed, and synchronizing the image data with the audio data. In the apparatus for synchronously reproducing an image and a sound to be decoded and reproduced, storage means for respectively storing the image data and the sound data, decoding means for reading the image data and the sound data from the storage means and performing decoding processing, Monitoring means for respectively monitoring the data storage amount of the storage means, and controlling the decoding means based on the monitoring results from the monitoring means, respectively, so that the storage amount of data stored in the storage means falls within a predetermined range. And a storage control means for controlling the amount of reproduction. 5. The storage amount control means skips data when the data storage amount of the storage means exceeds the predetermined range, and reads the data when the data storage amount falls below the predetermined range. 5. An apparatus according to claim 4, wherein said decoding means is controlled so as to repeat a data decoding process. 6. The image / audio data according to claim 5, wherein the skipping of the data and the repetition of the decoding process are performed in units of one unit of data at the time of writing and reading to and from the storage unit. Synchronous playback device. 7. A subtraction means for calculating a difference between a write address and a read address indicated by a write-side pointer and a read-side pointer of the storage means, and a storage for storing a set value indicating the predetermined range. And a comparing means for comparing the subtraction result of said subtracting means with the contents stored in said storage means and generating an interrupt to said storage amount control means. Video and audio synchronous playback device. 8. A decoding clock is detected from the code data and a decoding clock is detected from the code data instead of the monitoring unit and the storage amount controlling unit corresponding to the image data among the monitoring unit and the storage amount controlling unit. 8. The video / audio synchronization according to claim 4, further comprising clock recovery means for generating and supplying the decoding clock to the decoding means corresponding to the image data among the decoding means. Playback device.