JP3377463B2 - Video / audio gap correction system, method and recording medium - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects a shift between a moving image and a sound in a moving image / voice having a moving image and a sound (voice or sound, etc.), and corrects the shift. More specifically, the present invention relates to a system, a method, and a recording medium. More specifically, a medium that handles moving images, for example, a video / audio recording medium such as a television, a VTR, and a DVD, a video / audio processing computer, a medical device, or a communication device such as a videophone. The present invention relates to a suitable video / audio shift correction system, method, and recording medium.

[0002]

2. Description of the Related Art When a video, that is, a video having a moving image and a sound is processed by an analog device or a digital processing device, for example, when only the moving image is processed, a difference occurs between the moving image and the sound. . As a method for correcting such a shift, there is a proposal such as Japanese Patent Application No. 8-296216. In this proposal, the reference signals of both the moving image and the sound are superimposed before the program starts, and the shift between the moving image and the sound is corrected.

[0003]

In the above-mentioned proposal, there is a problem that it is difficult to measure and correct the deviation during broadcasting because the moving image and audio reference signals of the main line are superimposed before the program broadcasting.

Further, when editing a material (video) recorded in a VTR (video recording / reproducing apparatus), a reference signal must be superimposed on each fragmentary material. It has been a great effort for the user to synchronize the image and the sound.

Therefore, in view of the above-mentioned point, an object of the present invention is a video image in which a synchronizing process between a moving image in a video image / sound and a sound (sound or sound) can be executed by a simple operation even during broadcasting. / To provide an audio shift correction system, method and recording medium.

[0006]

In order to achieve such an object, the invention of claim 1 is a video / audio shift correction for correcting a temporal shift between an input moving image and an input sound. In the system, first detecting means for detecting a generation timing of a first sound of the sound source from a motion of the sound source included in the moving image of the input, and second detecting means of the sound source from the sound of the input. Second detection means for detecting the generation timing;
Measuring means for measuring a time difference between the generation timing of the first sound detected by the first detection means and the generation timing of the second sound detected by the second detection means; It is characterized by comprising signal processing means for synchronizing the moving image of the input and the voice of the input based on the measured time lag.

According to a second aspect of the present invention, in the video / audio shift correction system according to the first aspect, the first detecting means calculates a motion vector of the motion of the sound source, and the calculated result of the motion vector is obtained. The generation timing of the first sound in the moving image is detected based on the above.

According to a third aspect of the present invention, in the video / audio shift correction system according to the second aspect, the motion vector is calculated in at least one of a vertical direction and a horizontal direction.

According to a fourth aspect of the present invention, in the video / audio shift correction system according to any one of the first to third aspects, the sound source is a lip.

According to a fifth aspect of the present invention, in the video / audio shift correction system according to the fourth aspect, the second detecting means determines the generation timing of the second sound from the unvoiced / voiced / silent section of the voice. It is characterized by detecting.

According to a sixth aspect of the present invention, in the video / audio shift correction system according to any of the first to third aspects, the sound source is a plurality of objects that generate sound by collision. To do.

According to a seventh aspect of the present invention, in the video / audio shift correction system according to the sixth aspect, the second detecting means detects the generation timing of the second sound from the sound / silence section of the sound. It is characterized by doing.

According to an eighth aspect of the present invention, in the video / audio shift correction system according to any one of the first to seventh aspects, the first sound generation timing and the second sound generation timing are plural. It is characterized in that it is detected once, and the time difference is determined by the measuring means based on the generation timings of the plurality of detected first and second sounds.

According to a ninth aspect of the present invention, in a video / audio shift correction method for correcting a temporal shift between an input moving image and an input sound, a sound source included in the input moving image. Detecting the generation timing of the first sound of the sound source from the movement of the sound source, detecting the second generation timing of the sound source from the input sound, detecting the generation timing of the detected first sound and the detected timing of the first sound. It is characterized in that a time lag between the generation timing of the second sound is measured, and the moving image of the input and the voice of the input are synchronized based on the measured time lag.

According to a tenth aspect of the present invention, a program for correcting a time lag between an input moving image and an input sound is recorded, and the program is stored in a recording medium executed by a computer. Is the first sound source of the sound source from the motion of the sound source included in the input moving image.
Detection step of detecting the sound generation timing of the sound source, a second detection step of detecting the second generation timing of the sound source from the input sound, and a first detection step detected by the first detection step. Measuring step for measuring the temporal shift between the sound generation timing of the second sound and the second sound generation timing detected by the second detecting step, and the input based on the measured time shift. And a signal processing step of synchronizing the moving image with the input voice.

[0016]

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings.

FIG. 1 shows the contents of processing in the embodiment of the present invention.

(A) For a video / audio shift correction system, a moving image signal, for example, HDTV, NTSC, P
In this case, an audio signal of a moving image signal in the form of AL or the like and a sound accompanying the moving image signal is input. In this example, a moving image in which the movement of the lips is photographed as when uttering (for example, a moving image photographed by a bust shot such as news) will be described. It should be noted that it is possible to input a moving image signal in a video of a human playing a percussion instrument such as a tambourine or castanet, or even a moving image signal indicating that a sound is generated by hitting. In addition, this embodiment will be described by taking a high-definition signal as an example.

(B) Block 1 indicates a portion (detection window) for detecting a motion vector of a moving image, which is necessary for measuring a shift between the moving image and sound. The inventor of the present application has found that there is a peculiar movement when the sound is generated in the sound generation source, in this case, the movement of the lips, and the peculiar movement is detected by analyzing the moving image. The concept of motion vectors (described below) is used for this analysis.

By comparing the sound generation timing obtained from the moving image analysis with the sound (or sound) generation timing obtained from the sound signal, the mutual deviation is measured. Based on the measured shift, either the moving image or the sound is delayed to synchronize them.

(C) In the present embodiment, in order to correct the deviation of the video (moving image + sound) in real time, the moving image is displayed on the touch panel display of the analog image processing device, and the displayed moving image is displayed. An image processing device cuts out a moving image in the vicinity of the inside sound source image, for example, a block of 200 dots vertically and 350 dots horizontally. The clipped moving image is converted to analog to digital by the A / D converter in block 2 in RGB 4: 4: 4 format.
Since the number of dots (pixels) of the moving image data to be measured is reduced by removing the moving image portion of the background of the sound source by this cutout processing, it is possible to shorten the moving image analysis processing time described below.

(D) The motion vector of the lips, the motion vector of the percussion instrument, and the motion image generated by tapping from the motion image A / D-converted in block 2 (in the case of a motion image generated by tapping, the tapped part ( For example, pre-processing for detecting a motion vector of (hand) is performed in block 3. In the present embodiment, movements for generating sounds of the various sound sources in the moving image will be collectively referred to as sound generation patterns. As a specific preprocessing, a luminance signal for motion vector detection by a matrix is created by a signal processor or the like.

(E) Next, an initial deviation vector based on the gradient method (for example, candidate vector: 8 types; block size 8)
Iterative gradient method using (pixels, 8 lines) (for example, maximum number of iterations: 2 times, block size: 8 pixels, 8 lines)
Then, the motion vector related to the sound generation pattern of the sound source is obtained in blocks 4 and 5. For the gradient method and the iterative gradient method, see Japanese Patent Application No. 9-249134, Journal of Television Society Vol 48, No 1, pp 84-94 (1994) and NHK Giken R & D, No 26, September 1993.

(F) Motion vector amount calculation processing block 6
At, the amounts of the vertical motion vector and the horizontal motion vector of the sound source are obtained. When the cut-out moving image shows a motion that produces a sound, at least one of these vector amounts has a certain value or more. Therefore, one motion vector is obtained by summing these vertical and horizontal vectors, and the magnitude of this vector is used to determine whether or not sound is generated. The signal processor compares the calculated vector amount with a constant value to determine whether or not the clipped moving image has a sound generation pattern.

(G) The input audio signal is converted to analog digital (A / D) by the A / D converter in block 7.
To be converted. More specifically, A / D conversion is performed by sampling the audio signal with a quantization bit number of 16 bits and a sampling frequency of 48 kHz.

(H) The A / D converted voice is divided into unvoiced / voiced / voiceless sections by a signal processor. As the signal processing for the unvoiced / voiced / unvoiced section, a well-known technique used in digital voice processing may be used. Regarding an acoustic signal of a percussion instrument or the like, it may be simply detected that the volume of the sound is equal to or larger than a threshold value, more specifically, that a sound having a frequency peculiar to the percussion instrument is generated.

(I) Voice (sound) in the divided voiced section
The stand is then detected in block 9. By performing the above-described processes (h) and (i) by the signal processor, the sound (unvoiced / voiced or voiced) / silent section is identified, and the timing of occurrence of the pitch is detected.

(J) When the time difference between the sound generation timing obtained from the moving image by the block 6 and the sound generation timing obtained from the sound signal by the processing of the block 9 is calculated, the moving image signal and the sound are calculated. The deviation from the signal can be calculated (measured). This processing is performed by the signal processor in block 10.

(K) The input voice signal is a block 11
At, a delay element or the like delays the shift amount calculated in block 10. It is also possible to display the measured value in the block 10 and manually set the delay amount of the block 11 based on the displayed value. As a result, the moving image signal and the audio signal are synchronized and output from the video / audio shift correction system.

FIG. 2 shows the above-described motion vector, more specifically, a method of defining the motion vector of the lip up / down method. In the silent state, that is, in the state where nothing is uttered, the mouth is tied, so the vector at the center of the lips at this time is used as the starting point, and the vector in the direction in which the upper lip rises (the mouth is opened) is defined as the "vertical motion vector". and ", represented by a. This vector a has a starting point of 0
The upward direction is a (+) vector as a (zero) vector. The downward direction is a (-) vector.

FIG. 3 shows a method of defining a horizontal motion vector of the lips. The left end or the right end of the lips in the silent state is used as a starting point, and a direction in which this starting point extends outward is referred to as a “horizontal motion vector” and is represented by b . When the starting point is a 0 vector, the outer direction is a (+) vector and the inner direction is a (-) vector.

Here, the motion vector S (t) at each time (t) is

[0033]

## EQU00001 ## S (t) = a (t) + b (t) is defined.

FIG. 4 shows the relationship between the voice call point and the size of the lip motion vector │S (t) │ in a state where the moving image and the voice are synchronized when the announcer speaks a news sentence. . In FIG. 4, it will be apparent that in the lip motion vector, for example, the voice waveform is also generated from the voice-up point in response to the increase of the 0 vector from the A point.

FIG. 5 shows a case where a shift occurs between the moving image and the sound. Generally, the processing time of the image system is longer than the processing time of the audio system, so that the (moving) image is delayed by the shift time T as compared with the sound as shown in FIG. The time from the voice call point A to the next voice call point B is several seconds during normal utterance, while the time gap between the moving image and the voice is a time length of several frames (screen). One frame corresponds to 1/30 second.

In order to measure the time difference between the video and the audio, the voice call point in FIG. 5 can be measured by only one point, but depending on the utterance state, the accuracy may not be secured by measuring only one point. is there.

Therefore, a processing method for increasing the measurement accuracy will be described with reference to FIG.

FIG. 6 shows a voice waveform when a shift occurs,
The relationship between motion vectors is shown. Voice-up points A, B,
C, D, E and deviation time T with respect to the moving image of each point
_A , T _B , T _C , T _D , and T _E are measured according to the above-described deviation correction method. The magnitude relationship of the measured deviation time is

[0039]

## EQU2 ## When T _A ≤T _E ≤T _B ≤T _C ≤T _D , T _B and T excluding the maximum value T _D and the minimum value T _A
The average value T _AVE of _C and T _E may be calculated by a signal processor and used as the time lag between the moving image and the sound. In addition,

[0040]

## _EQU3 ## T _AVE = (T _B + T _C + T _E ) / 3. Using the average value T _AVE as the measurement result of the deviation,
If the input voice signal is delayed by the delay element in the correction processing block 11, the moving image and the voice can be synchronized.

In the above example, an analog image processing device,
Although it is composed of an A / D converter, a signal processor, a delay element, etc., when handling moving images and audio (acoustic) signals in digital form, a general-purpose computer such as a personal computer or a workstation corrects video / audio deviation. The system can be realized. Since the configuration of a general-purpose computer is well known, its processing procedure is shown in FIGS. 7 and 8. Figure 7
For convenience of description, the processing procedure in FIG. 8 and FIG. 8 use functional expressions, but in reality, the processing language is stored in a recording medium in a program language executable by the CPU and executed. A hard disk, a ROM, or the like can be used as a recording medium for storage purposes. Furthermore, floppy disk, CD-R
It may be mounted on the recording device (storage device) in the general-purpose computer from a portable recording medium such as OM. In FIG. 7, a general-purpose computer inputs a moving image and a sound signal. In the case of analog signals, analog-digital conversion is performed by a video card or the like. The moving images and audio signals corresponding to a plurality of frames are temporarily stored in the internal memory of the device (step S10). The temporarily stored moving image or the still image of the head portion is displayed on the display, and the cutout region of the image is designated with the mouse or the like (step S20). The image of the area specified by the range is cut out from the temporarily stored moving image (a plurality of still images), and the cut out image is stored in another area of the memory. The above-described preprocessing is executed on the cut out image signal (step S3).
0 → S40). A motion vector and its magnitude are calculated for the pre-processed video signal (step S5).
0). The vertical and horizontal motion vectors obtained as the calculation result are analyzed, and the points corresponding to A and B in FIG. 5 are detected. The data indicating this point is temporarily stored in the memory of the apparatus, and the utterance timing of the sound indicated by the image is detected (steps S60 → S70).

Next, the voice signal for a plurality of frames temporarily stored in the memory is used to detect the unvoiced / voiced / silent section, and then the voice-up point, that is, the voice generation point is detected ( Steps S80 → S90).

The time difference between the sound generation point detected from the moving image stored in the memory and the sound generation point detected from the sound is calculated (measured) (step S).
100). The temporarily stored input voice is delayed by the calculated time, and the input and temporarily stored moving image signal and voice signal are externally output (step S11).
0).

If necessary, the delayed audio signal and moving image signal may be stored in the hard disk of the apparatus (step S120).

After that, the above-mentioned processing is executed every time a moving image is input in units of a plurality of frames. Note that step S
Only the process of 20 is executed initially, and is omitted after the region (detection window) to be cut out is set.

In addition to the above embodiments, the following modes can be implemented.

1) In the above-mentioned embodiment, the explanation has been given mainly with respect to the voice, but if an object is accompanied by a motion in the generation of the sound, the timing of the sound generation can be detected from the moving image.

2) The video / audio shift correction system may be appropriately configured according to the form of the signal. Also a card,
Of course, it may be configured in the form of an IC chip or the like, and may be incorporated in another device, for example, a communication device.

3) In the above-described embodiment, the motion vector is detected in both the vertical direction and the horizontal direction, but when the motion direction of the sound source is specified in the vertical direction or the horizontal direction in advance, only that specific direction is detected. It suffices to calculate the motion vector.

4) The movement in which the sound source produces sound differs depending on the type of the sound source. Therefore, if a change in motion vector is prepared as a characteristic pattern for each type of sound source, the type of sound source is identified by comparing a plurality of characteristic patterns with a characteristic pattern extracted from a moving image to be synchronized. You can also Also, an unspecified number of moving images can be synchronized.

5) In the above-described embodiment, the image (data) of the range-designated area is cut out from the still images forming the moving image. However, even if the difference between two adjacent still images is calculated. It is possible to acquire image data of a sound source that moves.

6) As a sound source, if there is a characteristic peculiar to the movement of the sound of a single object, the sound generation timing can be detected from the moving image. For example, castanets, tambourines and hands, When sound is generated by two objects colliding with each other, such as a drum, a drumstick, and a hand clapping motion, the moving directions of the objects change in opposite directions. It is possible to detect the sound generation timing.

[0053]

As described above, according to claims 1, 9 and
According to the tenth aspect of the invention, by detecting the generation timing of the (first) sound from the motion of the sound source included in the moving image in the video / audio signal, the detection is performed from the sound itself (second
The difference between the sound and the video / audio can be measured by comparing with the sound generation timing. Therefore, it is possible to correct the deviation between the moving image and the sound in real time even in the video (moving image + sound) during broadcasting. In addition, claims 1, 9 and
Since the series of processes according to the tenth invention can be automated, the operation labor of the user can be significantly reduced.

According to the second aspect of the present invention, by calculating the motion vector of the sound source, it is possible to detect the characteristics of the motion of the sound source that generates the sound, and to detect the sound generation timing from the moving image.

According to the third to seventh aspects of the invention, sound is generated from the movement of the lips for the voice. For example, in the case of a percussion instrument, the generation of sound is detected from an image showing the movement of a colliding object, and the input sound is detected. With respect to the above, since the sound generation timing is distinguished by distinguishing between the voiced section and the silent section, human operation is not required, and automatic processing is possible.

In the invention of claim 8, the deviation time between the moving image and the sound is measured a plurality of times, and the delay time for actually correcting the deviation is determined from the measurement result, whereby the measurement accuracy of the delay time is improved.

[Brief description of drawings]

FIG. 1 is a block diagram showing the processing contents of an embodiment of the present invention.

FIG. 2 is an explanatory diagram showing vertical motion vectors of lips.

FIG. 3 is an explanatory diagram showing a horizontal motion vector of a lip.

FIG. 4 is a waveform diagram showing a relationship between a voice call and a motion vector when a moving image and a sound are synchronized with each other.

FIG. 5 is a waveform diagram showing a relationship between a voice call and a motion vector when a moving image is delayed.

FIG. 6 is a waveform diagram showing a relationship between a voice call and a motion vector when a shift occurs between a moving image and a sound.

FIG. 7 is a flowchart showing a processing procedure of a general-purpose computer according to the embodiment of the present invention.

FIG. 8 is a flowchart showing a processing procedure of a general-purpose computer according to the embodiment of the present invention.

[Explanation of symbols]

1 Processing to specify the motion vector detection window (block) 2,7 A / D conversion processing (block) 3 Pre-processing (block) 4 Motion detection processing (block) 5 Vector detection allocation processing (block) 6 Motion vector amount detection processing (block) 8 Unvoiced / voiced / unvoiced segmentation processing (block) 9 Voice detection processing (block) 10 Moving image / sound shift detection (measurement) processing (block) 11 Voice delay processing (block)

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Sadaharu Okada 2-2-1 Jinnan 2-chome, Shibuya-ku, Tokyo Inside the Japan Broadcasting Corporation Broadcasting Center (56) Reference JP-A-9-163333 (JP, A) JP-A 8-23530 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/222

Claims (10)

(57) [Claims] 1. A video / audio shift correction system for correcting a temporal shift between an input moving image and an input sound, wherein a sound source is converted from a motion of a sound source included in the input moving image. Detected by the first detection means, second detection means for detecting the second generation timing of the sound source from the input sound, and second detection means for detecting the second generation timing of the sound source from the input sound. Measuring means for measuring the time difference between the generation timing of the first sound and the generation timing of the second sound detected by the second detecting means, and the measured time difference. A video / audio shift correction system characterized by comprising signal processing means for synchronizing the input moving image with the input audio based on the above. 2. The video / audio shift correction system according to claim 1, wherein the first detecting means calculates a motion vector of the motion of the sound source, and the motion vector of the motion vector is calculated based on the calculation result of the motion vector. A video / audio shift correction system characterized by detecting the timing of generation of a first sound. 3. The video / audio shift correction system according to claim 2, wherein the motion vector is calculated in at least one of a vertical direction and a horizontal direction. . 4. The video / audio shift correction system according to claim 1, wherein the sound source is a lip. 5. The video / audio shift correction system according to claim 4, wherein the second detection unit is a voice unvoiced device.
A video / audio shift correction system, characterized in that the generation timing of the second sound is detected from a voiced / silent section. 6. The video / audio shift correction system according to claim 1, wherein the sound source is a plurality of objects that generate sound due to a collision. Correction system. 7. The video / audio shift correction system according to claim 6, wherein the second detection unit is configured to detect the presence / absence of sound.
A video / audio shift correction system, characterized in that the generation timing of the second sound is detected from a silent section. 8. The video / audio shift correction system according to claim 1, wherein the generation timing of the first sound and the generation timing of the second sound are detected a plurality of times. A plurality of first and second detected
The video / audio shift correction system, wherein the measuring unit determines the temporal shift based on the timing of sound generation. 9. A video / audio shift correction method for correcting a temporal shift between an input moving image and an input sound, wherein a sound source is converted from a motion of a sound source included in the input moving image. Detecting the generation timing of the first sound, detecting the second generation timing of the sound source from the input sound, and detecting the generation timing of the detected first sound and the detected second sound. A video / audio shift correction method characterized by measuring a temporal shift from a generation timing and synchronizing the input moving image and the input voice based on the measured temporal shift. 10. A recording medium for recording a program for correcting a temporal shift between an input moving image and an input sound, the program being executed by a computer, wherein the program is the input A first detecting step of detecting a generation timing of a first sound of the sound source from a motion of the sound source included in the moving image; and a second detecting step of detecting a second generation timing of the sound source from the input sound. 2 detection step, and a time lag between the generation timing of the first sound detected by the first detection step and the generation timing of the second sound detected by the second detection step. A recording medium comprising a measuring step of measuring and a signal processing step of synchronizing the moving image of the input and the sound of the input based on the measured time lag. .