JPH02246432A - Video audio multiplex system - Google Patents

Abstract

PURPOSE:To make the reproduced video output coincident with a reproduced audio output at the receiver side by obtaining a mean value of a processing delay time of a video signal in advance in comparison with a processing time of an audio signal so as to apply fixed delay control. CONSTITUTION:A mean value of a processing delay time being the result of subtracting a coding and decoding processing delay time for an audio signal from a delay time requiring the coding and decoding processing of a video signal is measured in advance. When a coded audio frame by an audio coding section 2 in the multiplex of the video and audio signal in a multiplex section 7 is subject to multi-frame processing, the coded audio signal is retarded by a mean delay time of the video reproducing output with respect to the audio reproducing output and the coded video information is multiplexed and sent to the transmission line. Thus, the synchronization between the video signal and the audio signal ensured without provision of a special delay circuit and natural reproduction is attained.

Description

[Detailed Description of the Invention] [Summary] Regarding a method of encoding and multiplexing audio information and video information and transmitting the same, we have developed a video/audio multiplexing method that can match the reproduced video output and the reproduced audio output on the receiving side. an A/D converter for converting audio input into digital audio;
An audio encoding section that encodes the digital audio, an A/D converter that converts the video input into digital video, a video encoding section that encodes the digital video, and a multiprocessor that encodes the encoded audio information. - It is composed of an audio delay/multiplexing unit that frames and delays the encoded audio by the average delay time of the video reproduction output with respect to the audio reproduction output, and then multiplexes and transmits the encoded video information.

[Industrial application field]

The present invention relates to a video/audio multiplexing system, and particularly to a system for encoding, multiplexing, and transmitting audio information and video information.

In recent years, communications that transmit both audio and video information, such as video conferences and video calls, have become popular.
In such a multiplex communication system, it is important to encode and decode both with high efficiency and to maintain a balance between audio information and video information.

[Conventional technology and issues]

The video encoding unit used in such video/audio multiplexing systems uses encoding techniques such as interframe predictive encoding, intraframe predictive encoding, motion compensated predictive encoding, and variable length encoding to encode video information. Although redundancy compression is attempted, since the amount of information in the video signal is much larger than that in the audio signal, a delay occurs in the encoding/decoding processing time of the audio signal.

Therefore, synchronization of video and audio (rip
sync) could not be secured, resulting in unnatural playback where video and audio did not match.

Accordingly, an object of the present invention is to provide a video/audio multiplexing system that allows reproduction video output and reproduction audio output to match on the receiving side.

[Means to solve the problem]

The present invention attempts to perform fixed delay control by determining in advance the average value of the processing delay time of a video signal when compared to the processing time of an audio signal.

That is, in the video/audio multiplexing system according to the present invention, as shown in principle in FIG. an A/D converter 3 that converts video input into digital video, a video encoder 4 that encodes the digital video, and a multi-digital converter 2 that converts the encoded audio information into digital video.
an audio delay/multiplexing unit 7 that frames and delays the encoded audio by an average delay time of the video reproduction output relative to the audio reproduction output, and then multiplexes and transmits the encoded video information. There is.

[For production]

Delay time Ta required for encoding/decoding processing of the audio signal in the encoding/decoding section of the audio signal is subtracted from delay time Tv required for encoding/decoding processing of the video signal in the video encoding section and decoding section. The processing delay time Td-Tv-Ta brings about an asynchronous state between the video playback output and the audio playback output. Therefore, by measuring the average value of this delay time Td in advance, it is possible to obtain an average synchronization state between the two.

For this reason, in the present invention, as shown in FIG. 1, the audio frames encoded by the audio encoding unit 2 are multi-framed when the multiplexing unit 7 multiplexes the video and audio.

In this case, the multi-frame format is as shown in FIG.
For example, by accumulating ・ as frames ■～■, ■～[phase], ■～■, the audio information can be delayed, so after the delay time T due to the accumulation of audio frames, the video information By multiplexing the signals and sending them to the transmission path, it is possible to synchronize the video playback output and the audio playback output.

〔Example〕

FIG. 3 shows an embodiment of the transmitting side and receiving side of the video/audio multiplexing system of the present invention shown in FIG. A plurality of adaptive encoding units each having a different number of encoding bits (
For example, ADPCM)2, ~2n, and each adaptive encoding unit 2I
-2n playback output and the digital audio to determine the encoding and bit rate based on the noise evaluation method and generate encoding bit rate information, and a corresponding adaptive encoding unit based on the encoding bit rate information. It is comprised of a selection section 6 that selects output audio information.

The receiving side also includes a separating unit 11 that separates the multiplexed signal from the transmission path 8 into video information and other information, and a separating unit 11 that separates the multiplexed signal from the transmission path 8 into video information and other information, and further includes the above-mentioned encoded bit rate information and audio information. a separation unit 12 that separates the audio information into audio information; an audio decoding unit 13 that decodes the audio information according to the encoded bit rate information; and a D/A converter 14 that converts the decoded digital audio into an audio signal. It is comprised of a video decoding section 15 that decodes video information, and a D/A converter 16 that converts the decoded digital video output into video output.

Adaptive encoding unit 2. ~2. The frame (packet) format of the audio information output from is as shown in Figure 4, and consists of a frame ladder a and a series of audio data following this frame ladder a, and has received the optimal noise evaluation. The signal has a number of bits equal to the number of bits coded by the adaptive encoder multiplied by the sampling rate n (n=8kHz), and is sent to the audio delay/multiplexer 7 via the selector 6.

FIG. 5 shows an embodiment of the audio delay/multiplexing section 7. In this embodiment, the CPU 21, the CPU bus 22, and the frame format shown in FIG. 6 are used to send data to the transmission path. ROM23 that stores the procedure program of
and an interface (
f/F) 24 to 26 and these interfaces 24 to 2
R having an address space for temporarily writing video information, audio information, and encoding bit rate information from 6.
It consists of an AM 27 and an interface 28 for sending out each piece of information to a transmission path.

The transmission frame format shown in FIG. 6 is CCITT
The bit allocation is based on Recommendation Y.221, and each multiframe consists of 16 transmission frames (1 multiframe - 16 transmission frames), 8 horizontal
One transmission frame consists of bits x 80 bits vertically (640 pits in total), and one transmission frame contains audio information, video information, and control information (A C (Application)).
FA S (Fr n Channel) information) as
ame Alignment Signal) information and BAS (Bit Alignment Signal) information
) consists of information.

Note that one multiframe is composed of 16 transmission frames as described above, but the data is divided into two transmission frames in the direction of ■ → [phase] as shown in FIGS. 6(a) and (b), e.g. F l-1~F +-16 and F, -1~F!-14,
are constructed and sent out in l pairs. However, FAS
The BAS information is different for transmission frames F and F8.

FAS information is frame information, (1) Y, 22
It is used to establish synchronization through one-frame synchronization and (2) multi-frame synchronization synchronization procedures.

That is, according to (1) above, it is possible to distinguish on a frame-by-frame basis, and (
2) makes it possible to identify each frame. Note that the necessity of identifying all frames is determined by R A S (Bit
A11ocation Signal) This is to recognize a response unit to a change in information.

The BAS information is a set of encoding information for the amount of audio information and the amount of video information (for example, the ratio of the two) on the transmitting side, and this encoded bit is determined by receiving the encoding bit rate information from the evaluation unit 5 in FIG. This BA is used to incorporate the rate into RAS information and transmit it, and to separate each data on the receiving side after establishing frame synchronization (see Figure 8).
The S information is determined every l sub-multiframe (1 multiframe - 2 sub-multiframes), and the previous B
The next encoding bit rate is detected from the AS information by majority logic (more than 5 frames out of 8 frames) (see Figure 9); If it is constant, it may be stored in the multiplexing unit 7 in advance.

Next, the audio multi-frame delay operation according to this embodiment will be explained.

In the multiplexing unit 7 when using the transmission frame configuration (FIG. 6) as described above, the bit allocation is determined according to the encoding bit rate stored as BAS information by the CPU 21, and as described above, each frame is - If the header is excluded, the frame has an 8Xn bit configuration, so frame allocation can be performed optimally as shown in FIG.

That is, for example, in units of 8 bits in the vertical direction (for example, in Fig. 7 (1
) to (8)) 1 to 7 bits to the right (horizontal axis)
When the audio data is filled up to 8 bits, the data moves to the lower 8-bit set, and audio data is sequentially filled in in a form corresponding to the selected encoding bit rate.

Here, in order to include the audio frame header in the BAS information,
It is necessary to use the same encoding rate for a certain number of frames.
This unit becomes the above-mentioned "1 multiframe (-16 frames)". Therefore, when performing adaptive coding, the audio coding rate becomes variable for each multi-transmission frame.

Then, audio information and video information are stored in the RAM 27,
When the transmission frame is completed, data is sent to the transmission path 8 at a rate that is slightly lower than the transmission speed.

Here, the transmission frame shown in FIG. 7 has 8 bits in the horizontal direction and 80 bits in the vertical direction as described above, and one transmission frame has 640 bits, but the transmission speed is now set to 64 kb/s.
Then, 100 transmission frames can be transmitted per unit time, and the time per unit transmission frame is
It will be 10m5ec. Therefore, one multi-frame
Due to the relationship defined as 6 transmission frames, the transmission time for one multi-transmission frame is 160 m5ec.

This is very close to the measured average value of the audio-video delay time Td mentioned above.

Therefore, after preparing 16 transmission frames each consisting of 10 multi-audio frames and storing audio information for 160 multi-audio frames in the RAM 27, video information is written from the buffer 43, and this completes the transmission frame. By sending it to the transmission path 8, audio information can be delayed as desired with respect to video information, and video/audio playback synchronization (lip sync) can be ensured without providing an independent delay section for audio delay. can do.

Further, by arranging the leading data position from a certain frame number, the operation in the separation section is simplified and the processing time is shortened.

On the other hand, the demultiplexer 11 on the receiving side has the same configuration as the multiplexer 7 shown in FIG. 5, with the arrows pointing in opposite directions.

That is, by analyzing this FAS information as described above for the input data from the transmission path, (1) Y,
H.221 frame synchronization and (2) multi-frame synchronization are established. Then, each data is separated by a separation unit 11 based on the RAS information, and the separated signal data is sent to the separation unit 12 via each interface for information other than video information, and the video information is sent to a video decoding unit. Send to 15. Then, the separation unit 12 further separates the audio information into audio information (a frame header is added to each frame based on the BAS information) and encoding bit rate information in the BAS information.

Then, the audio decoding section 13 receives the encoding bit rate information from the separating section 12 and decodes the audio information according to the bit rate.

As a result, the audio reproduction output of the audio decoding section 13 and the video reproduction output of the video decoding section 15 are generated at substantially the same time, so that synchronization can be ensured.

In the above embodiment, a plurality of adaptive encoding units are used in the audio encoding unit 2, but the same result can be obtained even if audio frames from a single encoding unit whose bit rate is fixed in advance are used. Needless to say, the action is taken.

〔Effect of the invention〕

As described above, according to the video/audio multiplexing method according to the present invention, the delay time for converting encoded audio frames into multi-frames and causing the audio playback output and the video playback output to fail once has elapsed. Since the configuration was configured to multiplex the video information later, it is possible to ensure synchronization (lip sync) between video and audio without installing a special delay circuit, allowing natural playback and maximizing the efficiency of the transmission path. It can be used to a limited extent.

[Brief explanation of drawings]

FIG. 1 is a block configuration diagram showing the principle of the video/audio multiplexing system according to the present invention. FIG. 2 is a time chart diagram for explaining the audio multi-frame delay operation according to the present invention. FIG. 3 is a block diagram showing an embodiment of the method of the present invention. FIG. 4 is a diagram showing audio frames used in the present invention. FIG. 5 is a configuration of a multiplexing/separating section used in the present invention. A block diagram showing an example; FIG. 6 is a configuration diagram of a transmission frame used in the present invention; FIG. 7 is a configuration diagram of a transmission frame showing multi-packetization of audio according to an embodiment of the present invention; FIG. 9 is a diagram showing an example of a video/audio allocation code of BAS information used in the present invention. FIG. 9 is a diagram illustrating video/audio separation using BAS information. In FIG. 1, 1.3...A/D converter, 2...audio encoder, 4...video encoder, 7...audio delay/multiplexer. In the figures, the same reference numerals indicate the same or corresponding parts.

Claims (4)

[Claims] (1) An A/D converter (1) that converts audio input into digital audio, an audio encoder (2) that encodes the digital audio, and an A/D converter that converts video input into digital video. (3
), a video encoding unit (4) that encodes the digital video, and converts the encoded audio information into a multi-frame and encodes the encoded audio by an average delay time of the video playback output with respect to the audio playback output. A video/audio multiplexing system comprising: an audio delay/multiplexer (7) that multiplexes and transmits the encoded video information after delaying the encoded video information. (2) The video/audio multiplexing system according to claim 1, wherein the period of multi-frame conversion of the audio information corresponds to the period of one multi-transmission frame as a unit of information processing. (3) The audio encoding unit (2) includes a plurality of adaptive encoding units (2_1-2n) each having a different number of encoding bits for the digital audio, and each adaptive encoding unit (2_1-2n).
an evaluation unit (5) that determines a coding bit rate based on a noise evaluation method from the reproduced output of n) and the digital audio and generates coding bit rate information, and adaptive coding corresponding to the coding bit rate information. 2. The video/audio multiplexing system according to claim 1, further comprising a selection section (6) for selecting output audio information of the section. (4) A separation unit (11) that separates the multiplexed signal from the transmission path into the video information and other information, and further separates the information other than the video information into the encoded bit rate information and the audio information. an audio decoding unit (13) that decodes the audio information according to the encoded bit rate information; and an audio decoding unit (13) that converts the decoded digital audio into an audio signal.
/A converter (14), a video decoding section (15) that decodes the video information, and a D converter that converts the decoded digital video output into video output.
3. The video/audio multiplexing system according to claim 2, further comprising: a /A converter (16).