JP4155990B2 - Synchronous reconstruction method and apparatus for acoustic data and moving image data - Google Patents

Description

  The present invention relates to a technique for compressing and decompressing and reproducing data in which audio and moving images are integrated while synchronizing audio and moving images.

  With the recent rapid development of computer processing technology related to moving images and sounds, multimedia devices that handle sound and moving images such as digital video are rapidly spreading. In particular, in the field of animation with a story that handles audio and video at the same time, processing using a computer is actively conducted for editing and the like, and advanced technology is also required for audio and video editing functions. It is being done. For example, when video data is created by linking video data containing the movement of a character and audio data edited separately, so that the natural data can be obtained while synchronizing the two data. Whether to do so is an important issue. Here, the term “video” is a general term for multimedia information including audio and moving images, and includes video and the like having a story by animation or live-action.

  In addition, when it is desired to divert part of video data completed as one scenario to other works, it is often necessary to compress and decompress the video data of the part to be diverted. In this case, when the video data is compressed and expanded, it is important to compress and expand so that the synchronization between the audio data and the moving image data is not broken.

  However, in many conventional video data, audio data and video data are generated independently in time series for each character, and the recording time length of each data may be different. There is also a need to synchronize such data or both data by compressing / decompressing them.

  Various methods have been considered in order to maintain synchronization between audio data and moving image data even when video data is compressed and expanded. For example, in Japanese Patent Laid-Open No. 5-3561, in editing / playing a story composed of audio and moving images, the audio data and moving image data are edited separately, and then synchronized as a single story. A method of configuring is disclosed.

  However, in this method, since the audio data and the video data are compressed and expanded separately and synchronized, the content and audio of the video can be accurately synchronized except for the start point and end point of the synchronization point. There is no guarantee that In recent years, particularly when there is a tendency to demand precise images, fine adjustments such as making sounds in accordance with mouth movements are often required. For example, audio that does not match the character's mouth movements is output. However, there is a problem that there is a possibility that a defect such as sound is generated even though the mouth of the character is not moving.

  The present invention eliminates the above-mentioned adverse effects and realizes fine synchronization processing by a simple method, and even when video data synchronized between audio data and video data is compressed and expanded, It is an object of the present invention to provide a method and apparatus for synchronous reconstruction of sound data / moving image data that can be reproduced.

  In order to achieve the above object, an audio data / video data synchronization reconstruction apparatus according to the present invention compresses and decompresses a pair of data composed of audio data and video data synchronized with each other. A data section dividing unit that divides a series of sound data and a series of moving image data into a plurality of sections, and compression / decompression of acoustic data and moving picture data for each section divided by the data section dividing unit A parameter time-series indicating unit for instructing a parameter time-series for determining the degree of sound, an input frame dividing unit for further dividing the acoustic data divided by the data section dividing unit into input frames at equal intervals, and a parameter time-series indicating unit The output frame length determination unit that determines the output frame length of the acoustic data based on the parameter time series determined in step 1 and the parameter time series instruction unit An audio video synchronization data output unit for outputting a correspondence table of synchronization points between the audio data before compression and expansion and the audio data after compression and expansion for the audio data to be compressed and expanded based on the parameter time series, and output frame length And a moving image data reconstruction unit that compresses and expands moving image data based on the correspondence table.

  With such a configuration, by dividing the data section for each point that should be synchronized in advance, as long as only compression and decompression is performed on the acoustic data, the video data can be compressed and decompressed while synchronizing with it. It becomes possible to easily and reliably perform compression / decompression of video data.

  In the acoustic data / moving image data synchronous reconstruction apparatus according to the present invention, the parameter time-series instruction unit preferably extracts the parameter time-series based only on the acoustic data. This is because the place where the sound is generated is often an important scene for synchronizing on the video such as when the character is talking.

  In the acoustic data / moving image data synchronous reconstruction apparatus according to the present invention, the parameter time series indicating unit preferably extracts the parameter time series based only on the moving image data. This is because the synchronization point can be easily extracted even when the character's movement changes.

  In the acoustic data / moving image data synchronous reconstruction apparatus according to the present invention, the parameter time series indicating unit preferably extracts the parameter time series based on both the acoustic data and the moving image data. This is because it is possible to set a finer synchronization point by combining both and weighting.

  In the synchronous reconstruction apparatus for audio data / moving image data according to the present invention, it is preferable that the parameter time series instruction unit inputs the parameter time series by the user. This is so that fine adjustments can be made in order to minimize the feeling of discomfort by the human eye.

  Next, in order to achieve the above object, the acoustic data / moving image data synchronous reconstruction method according to the present invention compresses / decompresses a pair of data composed of sound data and moving image data synchronized with each other. A method for synchronous reconstruction of moving image data, the step of dividing the sound data and moving image data into a plurality of sections, and the parameter time series for determining the degree of compression and expansion of the sound data and moving image data for each divided section In accordance with the determined parameter time series, the step of dividing the divided acoustic data into input frames of equal intervals, the step of determining the output frame length based on the determined parameter time series, and the For the audio data to be compressed / expanded, a step of outputting a correspondence table of synchronization points between the audio data before compression / expansion and the audio data after compression / expansion, and correspondence Characterized in that it comprises a step of compressing and expanding video data based the steps of compressing and expanding the sound data, the correspondence table based on.

  With such a configuration, by dividing the data section for each point that should be synchronized in advance, as long as only compression and decompression is performed on the acoustic data, the video data can be compressed and decompressed while synchronizing with it. It becomes possible to easily and reliably perform compression / decompression of video data.

  In the synchronous reconstruction method of acoustic data / moving image data according to the present invention, it is preferable to extract a parameter time series based only on acoustic data in the parameter instruction step. This is because the place where the sound is generated is often an important scene for synchronizing on the video such as when the character is talking.

  In the method for synchronously reconstructing acoustic data / moving image data according to the present invention, it is preferable to extract a parameter time series based only on moving image data in the step of instructing parameters. This is because the synchronization point can be easily extracted even when the character's movement changes.

  In the acoustic data / moving image data synchronous reconstruction method according to the present invention, it is preferable that the parameter time series is extracted based on both the acoustic data and the moving image data in the parameter instruction step. This is because it is possible to set a finer synchronization point by combining both and weighting.

  In the method for synchronously reconstructing audio data / moving image data according to the present invention, it is preferable that the parameter time series is input by a user's hand in the step of instructing the parameter. This is so that fine adjustments can be made in order to minimize the feeling of discomfort by the human eye.

  Next, a computer-readable recording medium recording a program to be executed by a computer according to the present invention is a program to be executed by a computer that compresses and decompresses a pair of data composed of acoustic data and moving image data that are synchronized with each other. Is a computer-readable recording medium in which sound data and moving image data are divided into a plurality of sections, and parameters for determining the degree of compression / expansion of the sound data and moving image data for each of the divided sections. A step of instructing, a step of further dividing the divided acoustic data into equally spaced input frames, a step of determining an output frame length based on the determined parameter time series, and a compression based on the determined parameter time series Regarding the sound data to be expanded, the sound data before compression and expansion and the sound data after compression and expansion And outputting the correspondence table of synchronization points between the steps of compressing and expanding audio data based on the correspondence table, characterized in that it comprises a step of compressing and expanding video data based on the correspondence table.

  With such a configuration, by loading and executing the program on a computer and dividing the data section for each point that should be synchronized in advance, if only compression / decompression is performed only on the acoustic data, a moving image is accompanied accordingly. Data can also be compressed and decompressed in synchronization, and an audio data / video data synchronous reconstruction apparatus that can easily and reliably compress and decompress video data can be realized.

  As described above, according to the synchronous reconstruction apparatus for acoustic data / moving image data according to the present invention, it is only necessary to compress and decompress only acoustic data by dividing a data section for each point to be synchronized in advance. Accordingly, the video data can also be compressed and decompressed while synchronizing, and the video data can be easily and reliably compressed and decompressed.

  A video / audio data synchronous reproduction apparatus according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic configuration diagram of an apparatus for synchronously reproducing sound data / moving image data according to an embodiment of the present invention.

  In FIG. 1, 11 is an acoustic data input unit, 12 is a moving image data input unit, 13 is a data section dividing unit, 14 is a parameter time-series instruction unit, 15 is a speech rate conversion unit, and 16 is an acoustic data output unit. , 17 is an acoustic video synchronization data output unit, 18 is a video data reconstruction unit, and 19 is a video data output unit.

  When video data in which audio and moving images are integrated is provided from the acoustic data input unit 11 and the moving image data input unit 12, both the acoustic data and the moving image data are based on a certain standard in the data section dividing unit 13. Divided. As the division method, the simplest method may be divided at equal intervals, but is not limited to this, and various criteria such as scene breaks in the video and the appearance / exit of a specific character are used. Can be divided.

  Further, the parameter time series instruction unit 14 extracts the parameter time series from the given acoustic data and moving picture data, or the user inputs the parameter time series, so that both the acoustic data and the moving picture data are stored. Set a parameter time series that determines how to synchronize.

  Next, based on the parameter time series set by the parameter time series instructing unit 14, the speech speed is converted by the speech speed converting unit 15 for each section divided by the data section dividing unit 13, and the acoustic video synchronization data output unit In 17, the audio video synchronization data corresponding to the speed change is output.

  The speaking speed conversion unit 15 converts the speaking speed but does not change the voice level. That is, the pitch of the sound is not converted. Furthermore, the speech speed need not always be the same in the same section, and can be expanded and compressed within the same section.

  In addition, as the acoustic moving image synchronization data, a correspondence table indicating which time of the original sound data is the data at each time of the sound data whose speech speed is converted is output. By clarifying such correspondence, the moving image data can be compressed and expanded so as to be synchronized with the moving image data corresponding to the time of the original sound data.

  Then, the moving image data reconstruction unit 18 performs compression / decompression processing such as thinning out moving image frames or conversely overlapping moving image frames on the basis of the correspondence table, thereby generating moving image data synchronized with the sound data.

  The sound data is output from the sound data output unit 16 as new sound data in a state converted by the speech speed conversion unit 15, and is reconstructed by the moving image data reconstructing unit 18 in a state of being synchronized with this. The moving image data is output from the moving image data output unit 19.

  Next, FIG. 2 shows an example of an apparatus for synchronously reproducing sound data / moving image data according to an embodiment of the present invention. In FIG. 2, 21 indicates an input frame dividing unit, 22 indicates an output frame length determining unit, and 23 indicates an acoustic data reconstruction unit.

  In FIG. 2, when video data in which audio and moving images are integrated is given, both audio data and moving image data are divided by the data section dividing unit 13 based on a certain standard. As the division method, the simplest method may be divided at equal intervals, but is not limited to this, and various criteria such as scene breaks in the video and the appearance / exit of a specific character are used. Can be divided.

  As a reference for section division, a method of detecting a scene change point and dividing data from one scene change point to the next scene change point as one section can be considered. The scene change point means that the connection between the video frames has been greatly lost. Hereinafter, a scene change point detection method will be described with an MPEG1 data stream as an example. However, the moving image data is not limited to the MPEG1 data stream, and the scene change point detection method is not limited to the following method.

  In the MPEG1 data stream, a GOP (Group Of) composed of an intra-frame encoded frame (I-Picture), an inter-frame encoded frame (P-Picture), and an inter-frame interpolated frame (B-Picture) in order to increase compression efficiency. (Pictures) layer. Each frame is compressed for each 8 × 8 macroblock.

  There are various types of macroblocks. Intra macroblocks that are intra-coded (hereinafter referred to as “I-MB”) and forward prediction macroblocks that are used for forward prediction (hereinafter referred to as “P-”). MB ”), backward prediction macroblocks used for backward prediction (hereinafter referred to as“ B-MB ”), and bidirectional prediction macroblocks used for bidirectional prediction (hereinafter referred to as“ D-MB ”). The total number of four macroblocks in a frame is used for scene change point detection.

  Further, the average value of the motion vectors obtained from each prediction macroblock in the frame is also used. Here, the “motion vector” means a difference vector from a certain scene to a certain scene.

  Specifically, considering the scene change point as a point where the connection between frames has been greatly lost, it can be expected that it will be difficult to obtain a motion vector in the frames before and after the scene change point. Inevitably, a rapid increase in I-MB and a rapid decrease in D-MB occur. Further, an increase in P-MB and a decrease in B-MB are expected in the frame immediately before the scene change point, and a decrease in P-MB and an increase in B-MB are expected in the frame immediately after the scene change point. A scene change point is estimated by defining an evaluation function considering these features.

  Next, the parameter time series instructing unit 14 extracts the parameter time series based on the given acoustic data and moving picture data, or the user inputs the parameter time series so that both the acoustic data and the moving picture data are obtained. Set a parameter time series that determines how to synchronize.

  Various methods for extracting parameter time series are also conceivable. Here, a method of extracting from only sound data, a method of extracting from only moving image data, and a method of extracting using both sound data and moving image data will be described.

  First, as a method of extracting the parameter time series from only the acoustic data, a method of extracting based on the strength of the acoustic power can be considered. This is a method of determining a parameter time series according to a criterion such as that important information is included in a loud part and less important information is included in a low part. By setting the sound power of the character's speaking voice large in the video, this is an effective method for synchronizing the sound data and the moving image data in accordance with the words spoken by the character. Similarly, a method of extracting the parameter time series according to the pitch (pitch) of the sound can be considered.

  Further, in order to distinguish sound sources, a method of extracting a parameter time series based on a sound spectrum is also conceivable. As the sound spectrum, an FFT (Fast Fourier Transform) spectrum, an LPC (Linear Predictive Coding) spectrum, and the like are conceivable. However, the present invention is not limited to this, and any type of sound source may be specified. good. By this method, for example, the parameter time series is determined based on a criterion such as evaluating the importance of the section where the main character is speaking high and evaluating the importance of the section where the other supporting character is speaking low. be able to.

  Next, as a method of extracting the parameter time series from only the moving image data, a method of extracting based on a motion vector (difference vector) can be considered. This is to determine a parameter time series by obtaining scene change points according to the degree of change in motion vector and arranging scene change intervals in time series.

  Finally, as a method for extracting parameter time series using both sound data and moving image data, weighting is added to the parameter time series obtained by both the method of extracting only from sound data and the method of extracting only from moving image data. Thus, a method for extracting an optimal parameter time series can be considered. The weighting may be set in advance by a program or the like, or may be input by the user according to the characteristics of the video data.

  The parameter time series may be input by the user through an input medium such as a keyboard. In this case, since the parameter time series itself is input as a step function, smoothing is performed to smooth parameter changes. As shown in FIG. 3, by changing the input step function to a gradual change, it is possible to expect an effect of smoothly connecting audio data without creating a completely silent area.

  Next, the input frame dividing unit 21 divides the acoustic data into equally spaced input frames. The interval between the input frames is shorter than the data interval determined by the data interval dividing unit 13.

  The output frame length determination unit 22 then sets each output frame length based on the value of the parameter time series so that the sum of the output frame lengths in each data section is (original data section length) * (compression / decompression rate). To decide. As the simplest method, a method of making the output frame length proportional to the parameter time series can be considered.

  FIG. 4 is a diagram showing a typical example of a method for obtaining the output frame length in proportion to the parameter time series. In FIG. 4, 41 indicates the input frame length of the acoustic data, 42 indicates the output frame length when the acoustic data is reproduced slowly, and 43 indicates the output frame length when the acoustic data is reproduced fast.

  4A shows acoustic data divided by the input frame dividing unit 21 at equal intervals. First, in the case where the acoustic data is proportional to the parameter time series and is reproduced later, the frame length 41 divided at equal intervals as shown in (b) is multiplied by the proportional constant of the parameter time series, The output frame length 42 is obtained. Accordingly, the output frame length is expanded for each frame, and the moving image data is also expanded to the determined new output frame length. Therefore, as long as audio and video are synchronized within each frame before decompression, the audio and video are synchronized even after decompression.

  Similarly, when the acoustic data is proportional to the parameter time series and is reproduced more quickly, as shown in (c), the frame length 41 divided at equal intervals is multiplied by a proportional constant of the parameter time series. The output frame length 43 is obtained. Therefore, compression to the output frame length is performed for each frame, and the moving image data is also expanded to the obtained new output frame length. Therefore, as long as the audio and video are synchronized within each frame before compression, the audio and video are synchronized even after compression.

  Then, the acoustic data reconstruction unit 23 cuts out acoustic data of only the output frame length from each input frame, and connects after shifting the position so that the joints are smooth. As a method for obtaining a position where the joint is smooth, various methods such as obtaining a position having the strongest cross-correlation can be considered. However, the present invention is not limited to this, and any method can be used as long as the sound can be heard smoothly.

  In addition, the acoustic video synchronization data outputs a correspondence table indicating at which time in the original acoustic data the data at each time of the acoustic data whose speech speed has been converted. By clarifying such correspondence, the moving image data can be compressed and expanded so as to be synchronized with the moving image data corresponding to the time of the original sound data.

  FIG. 5 shows an example of the correspondence table output as the audio video synchronization data. In FIG. 5, the time stamp of the acoustic data before compression / expansion and the time stamp of the acoustic data after compression / expansion are recorded in the correspondence table. Therefore, for example, it is understood that the acoustic data is expanded in the division category “1”, and the segment end time stamp is extended by 15 seconds. Similarly, it can be seen that the acoustic data is compressed in the divided section “2”, and the section end time stamp is advanced by 20 seconds.

  By compressing and expanding the moving image data in accordance with the increase / decrease of these time stamps, the audio data and the moving image data can be synchronized for each division. Therefore, by using the required synchronization time of audio data and video data as the division point, it is possible to compress and decompress the video without fail at the time when synchronization is required. It becomes.

  That is, the moving image data reconstruction unit 18 creates moving image data synchronized with the sound data by performing processing such as thinning out moving image frames or conversely overlapping moving image frames based on the correspondence table. .

  The video data decompression / compression method can be adjusted by duplicating or deleting the video frame of a scene where the character's movement is relatively slow. , Or a method of thinning out. Note that the present invention is not limited to these methods.

  As described above, according to the present embodiment, in the case where video data composed of synchronized audio data and moving image data is compressed and expanded, even when precise synchronization is required, The synchronization can be surely and easily performed, and it can be reproduced as a natural image for the viewer.

  Next, a description will be given of the flow of processing of a program that implements the acoustic data / moving picture data synchronization reconstruction apparatus according to the embodiment of the present invention. FIG. 6 shows a flowchart of processing of a program that realizes the synchronous reconstruction apparatus for audio data / moving image data according to the embodiment of the present invention.

  In FIG. 6, first, acoustic data / moving image data to be compressed / expanded is input (step S61). Next, the input data is divided into sections based on a certain standard such as each scene change point (step S62).

  Further, the parameter time series is extracted based on the input sound data / moving picture data, or the parameter time series is inputted by the user (step S63).

  Then, after the sound data is divided into equal intervals for each division (step S64), an output frame length for each input frame is obtained based on the determined parameter time series (step S65). In order to synchronize both the audio data and the video data in accordance with the output frame length, as the audio video synchronization data output, the data at each time of the compressed and decompressed audio data is at which time in the original audio data. A correspondence table indicating whether the data is data is output (step S66).

  Then, based on the correspondence table, both the audio data and the moving image data are reconstructed (step S67) and output as video data integrated with the audio data and the moving image data (step S68).

  As shown in the example of the recording medium shown in FIG. 7, the recording medium storing the program for realizing the audio data / moving picture data synchronous reconstruction apparatus according to the embodiment of the present invention is a CD-ROM or a floppy disk. Not only a portable recording medium but also any other storage device provided at the end of a communication line, or a recording medium such as a hard disk or RAM of a computer. When a program is executed, the program is loaded and executed on the main memory. Is done.

  In addition, as shown in the example of the recording medium shown in FIG. 7, a recording medium on which the acoustic moving picture synchronization data generated by the acoustic data / moving picture data synchronization reconstruction apparatus according to the embodiment of the present invention is recorded as a CD. -Not only a portable recording medium such as a ROM or a floppy disk but also any other storage device provided at the end of a communication line, a recording medium such as a hard disk or a RAM of a computer, for example, acoustic data according to the present invention -It is read by a computer when using a synchronous reconstruction device for moving image data.

FIG. 1 is a diagram illustrating the principle of an audio / video data synchronization reconstruction apparatus according to an embodiment of the invention 1 is a schematic configuration diagram of a synchronous reconstruction apparatus for audio data / moving image data according to an embodiment of the present invention. Example of smoothing step input Explanatory diagram of how to determine the output frame length Example of correspondence table that is audio video synchronization data output Flowchart of processing in a synchronous reconstruction apparatus for audio data / moving image data according to an embodiment of the present invention Example of recording medium

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Acoustic data input part 12 Movie data input part 13 Data area division | segmentation part 14 Parameter time series instruction | indication part 15 Speech speed conversion part 16 Acoustic data output part 17 Acoustic moving image synchronous data output part 18 Movie data reconstruction part 19 Movie data output part 21 Input frame division unit 22 Output frame length determination unit 23 Acoustic data reconstruction unit 31 Step input 32 Smoothed step input 41 Input frame 42 Output frame length (long)
43 Output frame length (short)
71 Line-destination storage device 72 Portable recording medium such as CD-ROM or floppy disk 72-1 CD-ROM
72-2 Floppy disk 73 Computer 74 Recording medium such as RAM / hard disk on computer

Claims (7)

An audio data / video data synchronous reconstruction device that compresses and decompresses a pair of data composed of audio data and video data synchronized with each other,
A data section dividing unit that divides a series of audio data and a series of moving image data into a plurality of sections;
A parameter time series instructing unit for instructing a parameter time series for determining the degree of compression and expansion of the sound data and moving image data for each of the sections divided by the data section dividing unit;
An input frame dividing unit that further divides the acoustic data divided by the data section dividing unit into equally spaced input frames;
An output for determining the output frame length of the acoustic data by multiplying the length of the input frame divided at equal intervals by the input frame dividing unit by the proportional constant of the parameter time series determined by the parameter time series instruction unit A frame length determination unit;
Outputs a correspondence table of synchronization points between the acoustic data before compression and decompression and the acoustic data after compression and decompression for the acoustic data compressed and decompressed based on the parameter time series determined by the parameter time series instruction unit An audio video synchronization data output unit,
An acoustic data reconstruction unit that cuts out and connects only the acoustic data of the output frame length from each input frame;
A sound data / moving image data synchronous reconstruction device comprising: a moving image data reconstructing unit that compresses / decompresses the moving image data based on the correspondence table.   The apparatus for synchronous reconstruction of acoustic data / moving image data according to claim 1, wherein the parameter time series instruction unit inputs the parameter time series by a user's hand.   The acoustic data / moving image data synchronous reconstruction apparatus according to claim 1, wherein the parameter time-series instruction unit extracts the parameter time-series by estimating the importance of each section from the acoustic data.   The acoustic data / moving image data synchronous reconstruction apparatus according to claim 1, wherein the parameter time series instruction unit extracts the parameter time series by estimating the importance of each section from the moving image data.   The acoustic data / moving image data synchronous reconstruction apparatus according to claim 1, wherein the data section dividing unit divides the moving image data into sections according to the contents thereof. A method for synchronous reconstruction of audio data / video data that compresses / decompresses a pair of data composed of audio data and video data synchronized with each other,
Dividing the acoustic data and the video data into a plurality of sections;
Instructing a parameter time series for determining the degree of compression / expansion of the sound data and the moving image data for each divided section;
Dividing the divided acoustic data into input frames that are equally spaced;
Determining the output frame length by multiplying the length of the divided input frame by the proportional constant of the parameter time series;
A step of outputting a correspondence table of synchronization points between the acoustic data before compression and decompression and the acoustic data after compression and decompression for the acoustic data to be compressed and decompressed based on the determined parameter time series;
Cutting out and connecting the acoustic data of the output frame length from each input frame; and
A method for synchronously reconstructing audio data / moving image data, comprising the step of compressing / decompressing the moving image data based on the correspondence table. A computer-readable recording medium recording a program for causing a computer to execute a process of compressing and decompressing a pair of data composed of sound data and moving image data synchronized with each other,
The program is
Dividing the acoustic data and the moving image data into a plurality of sections;
Instructing a parameter time series for determining the degree of compression / expansion of the sound data and the moving image data for each divided section;
Dividing the divided acoustic data into input frames that are equally spaced;
Determining the output frame length by multiplying the length of the divided input frame by the proportional constant of the parameter time series;
A step of outputting a correspondence table of synchronization points between the acoustic data before compression and decompression and the acoustic data after compression and decompression for the acoustic data to be compressed and decompressed based on the determined parameter time series;
Cutting out and connecting the acoustic data of the output frame length from each input frame; and
A computer-readable recording medium that causes a computer to execute a process of compressing and decompressing the moving image data based on the correspondence table.

Images