JP2000078531A - Method and system for editing audio data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To adjust the reproducing time length of audio data and to cancel a synchronous deviation between video data and audio data by generating dummy audio data consisting of dummy data which is neglected at the time of decoding. SOLUTION: The data part 34 of dummy audio data is constituted of Error Check 31, AudioData 32 and Ancillary Data 33 and their sizes are different in layer and sampling frequency. Besides, Audio Data 32 is variable length data and data '0' is stored in Audio Data 32 when the size of audio data is less than the size of AAU. Thus, the configuration is made to be the dummy audio data one where '0' is stored in the header of AAU and its data part so that audio data which are reproduced as the soundless one in result are generated without a compression processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an audio data editing process and editing apparatus.

[0002]

2. Description of the Related Art In recent years, Moving Picture Explorer, which is an international standard as a video compression technique, has been developed.
The improvement of the compression ratio by the ts Group (MPEG) and the reduction in the price of the secondary storage device have made it possible to handle video on a home computer.

[0003] MPEG is an international standard for moving picture compression established by the ISO. MPE, the first standard
After G1 was announced, a broadcasting compression standard called MPEG2 was established. MPEG1 uses an image transferred at a transfer rate of about 1.5 Mbps at a resolution of about 352 × 240 pixels at about 30 frames per second (in the case of NTSC) or 2 frames.
Playback is performed in about 4 frames (PAL). Decrypted MPEG
It is widely known that one data has an image quality equivalent to VHS. In contrast, MPEG2 is 4.0-8.0 Mb.
An image of about 720 × 480 pixels is reproduced at a transfer rate of about ps. The image quality of MPEG2 is lower than that of MPEG1.
It is said to be on par with D.

Normally, MPEG data is generated by compressing (encoding) an analog moving image input from a camera or a capture board into an MPEG format. The captured MPEG data can be reproduced by a PC in which an MPEG decoder (software or hardware) is installed.

[0005] MPEG data is formed by multiplexing an MPEG video stream which is data obtained by compressing video data and an MPEG audio stream which is data obtained by compressing audio data to form an MPEG system stream. MPEG data is usually called MPEG data.
Although it is a system stream, only an MPEG video stream and an MPEG audio stream can be reproduced as MPEG data by a software decoder or the like.

In MPEG, the picture rate (frames used per second) is 30, and in this case, 900
The playback time length of the frame video is 30 seconds. Therefore, when 30 frames are used per second, the reproduction time length per frame is about 33 ms. On the other hand, there are three types of MPEG audio data, Layer 1, Layer 2, and Layer 3, and their sampling frequencies are 32 KHz, 44.1 KHz, and 48 KHz. In addition, Audio Access Unit (AAU), which is a compression unit of MPEG audio data, is used.
Is 384 samples for layer 1, layer 2 and layer 3
In this example, 1152 samples are the compression unit.

As with MPEG data, similarly to normal uncompressed data, captured data may not be used as it is, but may be partially deleted or effectively pasted. When pasting video data and audio data, it is necessary to synchronize them. However, the length of each data is often not the same, and there is a problem that a gap occurs between the video data and the audio data. Was.

This problem will be described with reference to FIG.
First, BGM (audio) B for 20 seconds is attached to video A of a frame. When a video C of 900 frames (30 seconds) and an audio D of 30 seconds are pasted together, a shift occurs at the beginning of the video C and the audio D. Further, when the video E and the audio F having the same playback time length are pasted after the video C and the audio D, the synchronization shift continues.

Therefore, as a means for solving this synchronization deviation, there is a technique described in Japanese Patent Application Laid-Open No. 9-37204. This technique separates compressed moving image data and audio data, and compares their playback times. If the playback time of the audio data is shorter than the playback time of the moving image data, the P
The CM data is compressed to generate silence compressed audio data for a required time, connected to the audio data, and the moving image data and the audio data are synthesized again.

[0010]

However, in the above-mentioned technology, since processing for compressing silent PCM data is necessary, when a silent section that needs to be added extends for a long time, the PCM data is not compressed. There was a problem that it took time.

[0011] In addition, while the MPEG system stream is being created using the encoder, moving image data is continuously input, but audio may be interrupted or audio data may be interrupted due to use of a mute function. In such a case, the encoder generates the moving image data by compressing the silent audio data. However, if the silent period is long, there is a problem that the compression process takes a long time.

An object of the present invention is to generate dummy audio data that does not require compression processing.

A second object of the present invention is to adjust the reproduction time length of audio data by using dummy audio data that does not require compression processing, and to eliminate the synchronization deviation between video data and audio data.

[0014]

According to the present invention, there is provided an editing method for generating dummy audio data including a header including at least information indicating the start of a speech decoding unit and dummy data ignored during decoding without performing a compression operation. And an apparatus are disclosed. Since the search for the next header starts without decoding the dummy audio data at the time of decoding, as a result, a silence section continues for the time for searching for the next data.

In one embodiment, in the operation of pasting video data and audio data, if the playback time length of the audio data is shorter than the playback time length of the video data, the audio data may be shorter than the MPEG audio stream. The header portion from which the header information of the data is extracted and the data portion storing the dummy data to be ignored by the playback device are combined as audio data and added.

Further, in the process of creating one moving picture audio data while taking in the video data and the audio data, when the audio data is silent, dummy audio data including a header of the compressed audio data and dummy data is generated. And video data to generate moving image audio data.

The present invention provides a number of effects, but a particularly significant effect is that audio data compression processing is not required when generating dummy audio data. In addition, when the audio and the video are combined, if the audio is shorter, the reproduction time length of the audio data can be adjusted in a short time. Further, it is possible to reduce the processing time for creating moving image audio data in which the audio portion is silent.

[0018]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration according to an embodiment of the present invention.

In FIG. 1, a processing device 10 for controlling each device, a main memory 11 for loading a program for realizing this embodiment, and a frame memory for temporarily storing image data for display 12, a display device 13 for displaying image data, image data,
A decoder 14 for expanding audio data, an encoder 15 for compressing data, and an A for digitally converting analog image data and audio data.
/ D converter 16, image input device 17 for inputting analog video data, audio input device 18 for inputting analog audio data, secondary storage device 1 for storing decoded data and programs
9. It comprises a speaker 101 which is an audio output device.

The analog signals input from the image input device 17 and the audio input device 18 are converted into digital signals for video and audio separately by 16 A / D converters and input to the encoder 15. The encoder 15 compresses these digital signals and outputs them as data in MPEG format. MP generated by the encoder 15
The EG data is stored in the secondary storage device 19 or the main memory 1
1 is stored. The data stored in the main memory 11 or the secondary storage device 19 is decompressed by the decoder 14 when a user requests data reproduction. The expanded video data is written into the frame memory 12 and displayed on the display 13. The audio data expanded by the decoder 14 is reproduced through the speaker 101.

The program of this embodiment is started by an editing engine capable of performing many editing operations. These operations include cuts, cuts, pastes, fades, blends, morphs, tilts, and tilts from input files and streams for use in other files.
An operation of bonding audio data and moving image data can be given. In general, an editing engine manages many different editing tasks depending on the type of operator provided by an application requesting the editing task. Editing engine and application, and the program of this embodiment,
It is stored in the secondary storage device 19, loaded into the main memory 11 by a start command, and the control device 10 executes each process as an editing processing device according to each command of this program.

FIG. 3 shows the data structure of the dummy audio data of the present invention. The header part 30 is syncwo
rd, ID, layer, protection bi
t, bitrate index, sampling
frequency, padding bit, pr
irate bit, mode, mode ext
enhancement, copyright, originala
It contains information such as 1 / home and emphasis (for details of these data, refer to ISO11172-3), and is 4 bytes in size.

The data portion 34 is an ErrorCheck
31, Audio Data32, Ancillary
Data (external data) 33, the size of which differs depending on the layer and sampling frequency. Audio Data 32 is variable-length data. When audio data is smaller than the AAU size, the remaining portion is Ancillary Data 3.
3 and any data other than MPEG audio can be inserted into this part. In the present invention, data "0" is stored in the audio data 32.
If such data is present in the audio data, MP
The EG decoder performs the next AA without decoding the data.
Search for syncword in the header that is the beginning of U. Thus, the AAU header and its data part
With the configuration of the dummy audio data storing “0”, it is possible to create audio data that is consequently reproduced as silence without performing compression processing.

Next, when pasting video data and audio data from an input file using FIG.
The outline of the processing steps when the dummy audio data shown in FIG. 3 is generated will be described. Generally, the bonding operation is started when the application requests to perform the bonding operation in response to an instruction from the command input device 102.

First, when the video data and audio data to be synchronized are specified by the command input device 102, in a process 40, the editing device accesses the specified video data and audio data to obtain the video and audio data to be synchronized. Calculate the playback time length of.

The video playback time length Lv can be calculated by the following equation (1).

Lv = number of picture headers / picture rate (1) Further, the reproduction length La of the audio data is expressed by the following equation (2).
Can be calculated by

La = number of AAUs × X (2) Here, X is the playback time length per AAU, and is obtained by the following equation (3) according to the number of samples in each layer.

In the case of Layer 1, X = 384 / sampling rate In the case of Layer 2, X = 1152 / sampling rate (3) Therefore, in the process 40, the picture rate in the sequence header of the video data and the command input device 10
The video playback time length is calculated based on the number of pictures in the editing range designated by 2. Also, the audio playback time length is calculated from the layer information in the audio header, the sampling rate, and the number of AAUs included in the editing range.

Although the number of pictures and the number of AAUs can be calculated by counting the picture header and the audio sequence header, respectively, the PTS (Presenta)
Tion Time Stamp) and TR (Tempo)
ral Re Reference).

Next, in steps 41 and 46, the video playback time length and the audio playback time length are compared. If Yes in the process 41, that is, if the video playback time length is longer than the audio playback time length, the process proceeds to the process 42 because it is necessary to create dummy audio data. When the determination in step 41 is No, the process proceeds to step 46. Y in processing 46
In the case of es, that is, when the audio reproduction time length is longer than the video reproduction time length, it is necessary to generate blank video data, so that the process proceeds to processing 47. In the case of No in step 46, both are equal, so in step 45, the video and audio are joined together.

In the process 42, the required number of AAUs N is obtained.
Assuming that the difference between the video and audio playback time lengths is Y,
The number N of AAUs in the dummy audio data portion is obtained by the following equation (4).

N = Y / X (4) where Y = Lv-La If a fraction occurs, the value of N is obtained by rounding off.

Next, in process 43, the header information of the dummy audio data is extracted from the header information of the audio data to be pasted. Here, the header information of the dummy audio data must be the same as the data before it. This information is acquired, and the dummy audio data shown in FIG. Here, the number of bytes S per 1 AAU is obtained by the following equation (5).

In the case of layer 1: S = audio bit rate / sampling rate × 12 In the case of layers 2 and 3: S = audio bit rate / sampling rate × 144 (5) Further, the header information of one AAU is 4 bytes. Since the error check is 16 bytes, the number B of bytes storing 0 is as follows: when there is no error check: B = S−4, when there is an error check: B = S−20. One AAU dummy audio data is created by adding 0 of the number of bytes after the header portion.

Finally, in a process 45, a video and audio joining process is performed.

As described above, the dummy audio data is
By generating the pieces and pasting them together, it is possible to create data in a short time without deviation between audio and video, as shown by data 22 in FIG.

The processing for creating a blank video shown in block 47 of FIG. 4 is performed by the same applicant as a patent application (Japanese Patent Application No. 9-339842, filed on Dec. 1997).
On the 10th, the invention will be described in detail in the title of the invention, "Method and apparatus for controlling code amount of image data".

Next, another embodiment will be described. In this embodiment, for example, in FIG.
A method for creating dummy audio data having the data configuration shown in FIG. 3 when creating analog video data and audio data from the audio input device 18 and creating moving image audio compressed data, respectively.

FIG. 5 shows an encoder 15 in this embodiment.
It is a block diagram of. The encoder in this embodiment is
Moving image compression means 51 for compressing video data, audio compression means 52 for compressing audio data, dummy audio data generation means 53 for generating dummy audio data of the present invention, switching means 54, audio compression means 52, a control unit 55 for controlling the dummy audio data generating unit 53 and the switching unit 55.

The video data and audio data converted into digital signals by the A / D conversion circuit 16 are input to the encoder 15, and the video data is supplied to the moving picture compression means 51.
Is compressed. The audio data is input to the audio compression unit 52 and the control unit 55. The control means 55
When the output of the audio data is less than the predetermined value, the dummy audio data generating means 53 generates dummy audio data. This dummy audio data generating means generates a header portion of normal compressed audio data, and further generates dummy audio data including a data portion shown in FIG. At this time, the control means 55 controls the sound compression means 5
The encoding process in step 2 is interrupted. When the output of the audio data becomes equal to or more than the predetermined value, the processing restart of the audio compression unit 52 is supported. The control unit 55 further controls the switching unit 54. If the output of the audio data is equal to or more than a predetermined value, the control unit 55 outputs the compressed audio data compressed by the normal audio compression unit 52, and if the output is less than the predetermined value. Is
The switching unit 54 is controlled so as to output dummy audio data. The compressed video data and the compressed audio data or dummy audio data are synchronized and stored in a storage means such as a secondary storage device or a main memory.

As described above, when the output of the fetched audio data is equal to or less than a predetermined value, it is determined that there is no sound, and by generating dummy audio data, the compression processing of the audio data can be omitted. Processing time of the compression processing can be shortened.

In this embodiment, the encoder including the control means for controlling the dummy audio data of the present invention when compressing the data fetched by the voice input means is described. It is also conceivable that a certain processor instructs.

Further, in this embodiment, a local type architecture is described. However, in order to transmit only a video using a mute function to another client connected via a network, a moving image is used. Needless to say, it is used in various cases where compression of silence data is necessary, such as when data and silence data need to be compressed.

[0046]

As described above, according to the present invention, when creating silence data, an AAU header and dummy data that match the format of the audio data to be encoded are used. MPEG audio data can be freely generated, and the processing time for generating and editing video and audio having different reproduction time lengths can be reduced.

[Brief description of the drawings]

FIG. 1 is a diagram of a system configuration for realizing an embodiment of the present invention.

FIG. 2 is an example of edit data for explaining a problem and a known example of the present invention.

FIG. 3 is a data configuration of dummy audio data used in the present invention.

FIG. 4 is a flowchart illustrating a method of combining video and audio according to the present invention.

FIG. 5 is a block diagram schematically showing an encoder for implementing an embodiment of the present invention.

[Explanation of symbols]

30: header part in one AAU of MPEG audio stream 31: error check part in one AAU of MPEG audio stream 32: data part in one AAU of MPEG audio stream 33: ancillary data part in one AAU of MPEG audio stream 33, a data portion in one AAU of the MPEG audio stream, 51, moving image compression means, 52, audio compression means, 53, dummy audio data generation means, 54, switching means, 55, control means

Claims (9)

[Claims] A header for storing at least information indicating the start of a speech decoding unit, to which dummy data is added; and a header creating step for creating a header portion having the same value for each component of the header; An audio data editing method including a dummy data creating step of creating audio data composed of dummy data that is sometimes ignored. 2. The audio data editing method according to claim 1, further comprising: calculating a reproduction time of the compression-encoded audio data and a reproduction time of the compression-encoded video data. Determining the number of minimum audio decoding units from the playback time difference from the video data, wherein the header creating step creates a header portion by extracting header information of the audio data, and the dummy data creating step includes: An audio data editing method, wherein the dummy data is created in an amount corresponding to the number of the minimum audio decoding units. 3. The audio data editing method according to claim 1, wherein the audio decoding unit is the same minimum unit of audio data as the original audio data that can be decoded, and the dummy data is a bit corresponding to the reproduction time difference. An audio data editing method, characterized in that the audio data editing value is a value of Numerical 0. 4. A step of capturing audio data by audio input, a step of detecting an output of the audio data, and information indicating a start of an audio decoding unit when the output of the audio data is less than a predetermined value. An audio data editing method, comprising the step of creating dummy audio data at least including a header portion that stores and has a value of each component equal to the audio data and dummy data that is ignored during decoding. 5. The audio data editing method according to claim 4, wherein: a step of capturing video data from a video input device; a step of compressing the video data; a step of instructing a start of creation of the dummy audio data; Issuing an instruction to end the creation of the audio data. 6. The audio data editing method according to claim 4, wherein the audio decoding unit is the same minimum unit of audio data as the input audio data that can be decoded, and the dummy data is zero. Audio data editing method. 7. A storage device for storing compression-encoded audio data, and header information obtained by accessing the compression-encoded audio data, and a header having the same value as the acquired header information and each component element. An audio data editing system comprising: an editing processing device that generates dummy audio data composed of dummy data ignored during decoding. 8. The audio data editing system according to claim 7, wherein the storage device further stores compression-encoded video data, and the editing processing device stores the compression-encoded video data and the compression-encoded video data. Audio data, wherein the number of minimum audio decoding units corresponding to the reproduction time difference of the encoded audio data is calculated, and dummy audio data corresponding to the minimum audio decoding unit number of the reproduction time difference is created. Editing system. 9. A step of calculating a reproduction time of the compression-encoded audio data, a reproduction time of the compression-encoded video data, and a minimum corresponding to a reproduction time difference between the audio data and the video data. Determining a number of audio decoding units and storing at least information indicating the start of the audio decoding unit, and creating a header portion in which the value of each component element is equal to the audio data; and A computer-readable recording medium storing a program including a dummy data creating step of creating a minimum unit of dummy audio data corresponding to a time difference.